Sorry to sound clueless but no matter what I search for on El Goog I can't figure out how to answer the question as to whether dynamic sharding is enabled in our environment. It's not configured as true in the config files, but it is the default. Is there a radosgw-admin command to determine whether or not it's enabled in the running environment? Thanks, -Drew -----Original Message----- From: Benoît Knecht <bknecht@protonmail.ch> Sent: Thursday, March 4, 2021 11:46 AM To: Drew Weaver <drew.weaver@thenap.com> Cc: 'ceph-users@ceph.io' <ceph-users@ceph.io> Subject: Re: [ceph-users] Resolving LARGE_OMAP_OBJECTS Hi Drew, On Thursday, March 4th, 2021 at 15:18, Drew Weaver <drew.weaver@thenap.com> wrote:

Howdy, the dashboard on our cluster keeps showing LARGE_OMAP_OBJECTS.

I went through this document

https://urldefense.proofpoint.com/v2/url?u=https-3A__www.suse.com_support_kb_doc_-3Fid-3D000019698&d=DwIFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=CNkAvwXr2_ctEE9c_biQmD6shezowIzUC32IvquGvmE&s=gzTWzc9eJy5J-DuCuctcLib_g1evf_bHZZ0uhOhy4ak&e=

I've found that we have a total of 5 buckets, each one is owned by a different user.

Do you have dynamic sharding enabled? If so, hitting the large OMAP object threshold is a bit suspicious, as resharding should keep each shard below the threshold. Did you look into your logs to find out which object is affected, and it's the number of keys or its size triggering the warning (you can grep for "Large omap object found" in /var/log/ceph/ceph.log)? From the name of the object, you can figure out which bucket is affected, and you can look for it in the default.rgw.buckets.index pool and see if the index keys are evenly distributed among the shards. -- Ben

Hi, Only 2 of the buckets are really used: "buckets": [ { "bucket": "test", "tenant": "", "num_objects": 968107, "num_shards": 16, "objects_per_shard": 60506, "fill_status": "OK" } ] "buckets": [ { "bucket": "prototype", "tenant": "", "num_objects": 4633533, "num_shards": 98, "objects_per_shard": 47280, "fill_status": "OK" } ] In the log it shows this: cluster [WRN] Large omap object found. Object: 7:3019f91b:::.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.213.0:head PG: 7.d89f980c (7.c) Key count: 206795 Size (bytes): 46499042 pool id 7 is default.rgw.buckets.index The entry in the log doesn't match any of the bucket ids but this one exists: "id": "2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637", Thanks to a RedHat KB article that mentioned checking for stale instances I ran this: # radosgw-admin reshard stale-instances list | wc -l 713 This command: ``` for obj in $(rados -p default.rgw.buckets.index ls | grep 2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637); do printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys $obj | wc -l) done ``` returns this: -bash: command substitution: line 4: syntax error: unexpected end of file I figured perhaps you were using ``` to denote code so I tried running it without that and also on one line and neither of those did the trick. Is that just bash? Thanks so much for all of your help thus far. -Drew -----Original Message----- From: Benoît Knecht <bknecht@protonmail.ch> Sent: Friday, March 5, 2021 12:00 PM To: Drew Weaver <drew.weaver@thenap.com> Cc: 'ceph-users@ceph.io' <ceph-users@ceph.io> Subject: RE: [ceph-users] Resolving LARGE_OMAP_OBJECTS On Friday, March 5th, 2021 at 15:20, Drew Weaver <drew.weaver@thenap.com> wrote:

Sorry to sound clueless but no matter what I search for on El Goog I can't figure out how to answer the question as to whether dynamic sharding is enabled in our environment.

It's not configured as true in the config files, but it is the default.

Is there a radosgw-admin command to determine whether or not it's enabled in the running environment?

If `rgw_dynamic_resharding` is not explicitly set to `false` in your environment, I think we can assume dynamic resharding is enabled. And if any of your buckets have more than one shard and you didn't reshard them manually, you'll know for sure dynamic resharding is working; you can check the number of shards on a bucket with `radosgw-admin bucket stats --bucket=<name>`, there's a `num_shards` field. You can also check with `radosgw-admin bucket limit check` if any of your buckets are about to be resharded. Assuming dynamic resharding is enabled and none of your buckets are about to be resharded, I would then find out which object has too many OMAP keys by grepping the logs. The name of the object will contain the bucket ID (also found in the output of `radosgw-admin bucket stats`), so you'll know which bucket is causing the issue. And you can check how many OMAP keys are in each shard of that bucket index using ``` for obj in $(rados -p default.rgw.buckets.index ls | grep eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4); do printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys $obj | wc -l) done ``` (where `eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4` is your bucket ID). If the number of keys are very uneven amongst the shards, there's probably an issue that needs to be addressed. If you they are relatively even but slightly above the warning threshold, it's probably a versioned bucket, and it should be safe to simply increase the threshold. Cheers, -- Ben

Response inline: On Fri, Mar 5, 2021 at 11:00 AM Benoît Knecht <bknecht@protonmail.ch> wrote:

On Friday, March 5th, 2021 at 15:20, Drew Weaver <drew.weaver@thenap.com> wrote:

...

Sorry to sound clueless but no matter what I search for on El Goog I can't figure out how to answer the question as to whether dynamic sharding is enabled in our environment.

It's not configured as true in the config files, but it is the default.

Is there a radosgw-admin command to determine whether or not it's enabled in the running environment?

If `rgw_dynamic_resharding` is not explicitly set to `false` in your environment, I think we can assume dynamic resharding is enabled. And if any of your buckets have more than one shard and you didn't reshard them manually, you'll know for sure dynamic resharding is working; you can check the number of shards on a bucket with `radosgw-admin bucket stats --bucket=<name>`, there's a `num_shards` field. You can also check with `radosgw-admin bucket limit check` if any of your buckets are about to be resharded.

Assuming dynamic resharding is enabled and none of your buckets are about to be resharded, I would then find out which object has too many OMAP keys by grepping the logs. The name of the object will contain the bucket ID (also found in the output of `radosgw-admin bucket stats`), so you'll know which bucket is causing the issue. And you can check how many OMAP keys are in each shard of that bucket index using

``` for obj in $(rados -p default.rgw.buckets.index ls | grep eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4); do printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys $obj | wc -l) done ```

(where `eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4` is your bucket ID). If the number of keys are very uneven amongst the shards, there's probably an issue that needs to be addressed. If you they are relatively even but slightly above the warning threshold, it's probably a versioned bucket, and it should be safe to simply increase the threshold.

As this is somewhat relevant, jumping in here... we're seeing the same "large omap objects" warning and this is only happening with versioned buckets/objects. Looking through logs, we can find a few instances: cluster 2021-03-29T14:22:12.822291+0000 osd.55 (osd.55) 1074 : cluster [WRN] Large omap object found. Object: 18:7004a547:::.dir.d99b34b6-5e94-4b64-a189-e23a3fabd712.326812.1.10:head PG: 18.e2a5200e (18.e) Key count: 264199 Size (bytes): 107603375 We check the bucket (we do have dynamic sharding enabled): "num_shards": 23, "num_objects": 1524017 Doing the math, something seems off with that key count (23 shards with 1.52 million objects shouldn't be 260k+ a shard). We check: root@ceph01:~# rados -p res22-vbo1a.rgw.buckets.index listomapkeys .dir.d99b34b6-5e94-4b64-a189-e23a3fabd712.326812.1.10 | wc -l 264239 Sure enough, it is more than 200,000, just as the alert indicates. However, why did it not reshard further? Here's the kicker - we _only_ see this with versioned buckets/objects. I don't see anything in the documentation that indicates this is a known issue with sharding, but perhaps there is something going on with versioned buckets/objects. Is there any clarity here/suggestions on how to deal with this? It sounds like you expect this behavior with versioned buckets, so we must be missing something. root@ceph01:~# ceph config get osd rgw_dynamic_resharding true root@ceph01:~# ceph config get osd rgw_max_objs_per_shard 100000 root@ceph01:~# ceph config get osd rgw_max_dynamic_shards 1999 root@ceph01:~# Config should be sharding further based on the key counts in each of the shards. I checked all 23 shards and they are all ~260,000 keys. Thanks, David

Cheers,

-- Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Ben, That was beyond helpful. Thank you so much for the thoughtful and detailed explanation. That should definitely be added to the documentation, until/unless the dynamic resharder/sharder handle this case (if there is even desire to do so) with versioned objects. Respectfully, David On Tue, Mar 30, 2021 at 12:21 AM Benoît Knecht <bknecht@protonmail.ch> wrote:

Hi David,

On Tuesday, March 30th, 2021 at 00:50, David Orman <ormandj@corenode.com> wrote:

...

Sure enough, it is more than 200,000, just as the alert indicates. However, why did it not reshard further? Here's the kicker - we only see this with versioned buckets/objects. I don't see anything in the documentation that indicates this is a known issue with sharding, but perhaps there is something going on with versioned buckets/objects. Is there any clarity here/suggestions on how to deal with this? It sounds like you expect this behavior with versioned buckets, so we must be missing something.

The issue with versioned buckets is that each object is associated with at least 4 index entries, with 2 additional index entries for each version of the object. Dynamic resharding is based on the number of objects, not the number of index entries, and it counts each version of an object as an object, so the biggest discrepancy between number of objects and index entries happens when there's only one version of each object (factor of 4), and it tends to a factor of two as the number of versions per object increases to infinity. But there's one more special case. When you delete an versioned object, it also creates two more index entries, but those are not taken into account by dynamic resharding. Therefore, the absolute worst case is when there was a single version of each object, and all the objects have been deleted. In that case, there's 6 index entries for each object counted by dynamic resharding, i.e. a factor of 6.

So one way to "solve" this issue is to set `osd_deep_scrub_large_omap_object_key_threshold=600000`, which (with the default `rgw_max_objs_per_shard=100000`) will guarantee that dynamic resharding will kick in before you get a large omap object warning even in the worst case scenario for versioned buckets. If you're not comfortable having that many keys per omap object, you could instead decrease `rgw_max_objs_per_shard`.

Cheers,

-- Ben