5 Mar
2021
5 Mar
'21
10:52 p.m.
Sorry for multi-reply, I got that command to run:
for obj in $(rados -p default.rgw.buckets.index ls | grep
2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637); do printf "%-60s %7d\n" $obj
$(rados -p default.rgw.buckets.index listomapkeys $obj | wc -l); done;
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.0 10423
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.15 10445
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.3 10542
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.6 10511
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.13 10414
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.12 10479
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.2 10486
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.5 10448
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.4 10470
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.8 10474
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.1 10470
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.10 10411
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.7 10445
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.14 10413
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.9 10356
.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.11 10410
-----Original Message-----
From: Benoît Knecht <bknecht(a)protonmail.ch>
Sent: Friday, March 5, 2021 12:00 PM
To: Drew Weaver <drew.weaver(a)thenap.com>
Cc: 'ceph-users(a)ceph.io' <ceph-users(a)ceph.io>
Subject: RE: [ceph-users] Resolving LARGE_OMAP_OBJECTS
On Friday, March 5th, 2021 at 15:20, Drew Weaver <drew.weaver(a)thenap.com> wrote:
Sorry to sound clueless but no matter what I search
for on El Goog I can't figure out how to answer the question as to whether dynamic
sharding is enabled in our environment.
It's not configured as true in the config files, but it is the default.
Is there a radosgw-admin command to determine whether or not it's enabled in the
running environment?
If `rgw_dynamic_resharding` is not explicitly set to `false` in your environment, I think
we can assume dynamic resharding is enabled. And if any of your buckets have more than one
shard and you didn't reshard them manually, you'll know for sure dynamic
resharding is working; you can check the number of shards on a bucket with `radosgw-admin
bucket stats --bucket=<name>`, there's a `num_shards` field. You can also check
with `radosgw-admin bucket limit check` if any of your buckets are about to be
resharded.
Assuming dynamic resharding is enabled and none of your buckets are about to be resharded,
I would then find out which object has too many OMAP keys by grepping the logs. The name
of the object will contain the bucket ID (also found in the output of `radosgw-admin
bucket stats`), so you'll know which bucket is causing the issue. And you can check
how many OMAP keys are in each shard of that bucket index using
```
for obj in $(rados -p default.rgw.buckets.index ls | grep
eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4); do
printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys
$obj | wc -l) done ```
(where `eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4` is your bucket ID). If the number
of keys are very uneven amongst the shards, there's probably an issue that needs to be
addressed. If you they are relatively even but slightly above the warning threshold,
it's probably a versioned bucket, and it should be safe to simply increase the
threshold.
Cheers,
--
Ben