20 May
2021
20 May
'21
1:32 p.m.
Hi, we're running 15.2.7 and our cluster is warning us about LARGE_OMAP_OBJECTS (1
large omap objects).
Here is what the distribution looks like for the bucket in question, and as you can see
all but 3 of the keys reside in shard 2.
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.0 1
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.8 0
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.9 0
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.7 0
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.1 0
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.4 0
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.3 1
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.2 262384
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.6 0
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.5 0
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.12 0
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.10 1
.dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.11 0
osd_deep_scrub_large_omap_object_key_threshold is set to 200000 by default, hence the
warning observed for this bucket.
Dynamic resharding is enabled, and the bucket is not in the process of being resharded.
Versioning not in use for this bucket, so we're not affected by
https://tracker.ceph.com/issues/46456.
Can anyone help us understand why all the keys are getting mapped to a singe shard? Is
there a bug here, or is this expected behaviour?
Could it be related to the fact that the bucket contains large multipart uploads? (Object
names look like this:)
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5900
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5901
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5902
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5903
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5904
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5905
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5906
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5907
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5908
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5909
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7152
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7153
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7154
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7155
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7156
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7157
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7158
_multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7159