Hi,
Only 2 of the buckets are really used:
"buckets": [
{
"bucket": "test",
"tenant": "",
"num_objects": 968107,
"num_shards": 16,
"objects_per_shard": 60506,
"fill_status": "OK"
}
]
"buckets": [
{
"bucket": "prototype",
"tenant": "",
"num_objects": 4633533,
"num_shards": 98,
"objects_per_shard": 47280,
"fill_status": "OK"
}
]
In the log it shows this:
cluster [WRN] Large omap object found. Object:
7:3019f91b:::.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.213.0:head PG: 7.d89f980c
(7.c) Key count: 206795 Size (bytes): 46499042
pool id 7 is default.rgw.buckets.index
The entry in the log doesn't match any of the bucket ids but this one exists:
"id": "2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637",
Thanks to a RedHat KB article that mentioned checking for stale instances I ran this:
# radosgw-admin reshard stale-instances list | wc -l
713
This command:
```
for obj in $(rados -p default.rgw.buckets.index ls | grep
2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637); do
printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys
$obj | wc -l) done ```
returns this:
-bash: command substitution: line 4: syntax error: unexpected end of file
I figured perhaps you were using ``` to denote code so I tried running it without that and
also on one line and neither of those did the trick.
Is that just bash?
Thanks so much for all of your help thus far.
-Drew
-----Original Message-----
From: Benoît Knecht <bknecht(a)protonmail.ch>
Sent: Friday, March 5, 2021 12:00 PM
To: Drew Weaver <drew.weaver(a)thenap.com>
Cc: 'ceph-users(a)ceph.io' <ceph-users(a)ceph.io>
Subject: RE: [ceph-users] Resolving LARGE_OMAP_OBJECTS
On Friday, March 5th, 2021 at 15:20, Drew Weaver <drew.weaver(a)thenap.com> wrote:
Sorry to sound clueless but no matter what I search
for on El Goog I can't figure out how to answer the question as to whether dynamic
sharding is enabled in our environment.
It's not configured as true in the config files, but it is the default.
Is there a radosgw-admin command to determine whether or not it's enabled in the
running environment?
If `rgw_dynamic_resharding` is not explicitly set to `false` in your environment, I think
we can assume dynamic resharding is enabled. And if any of your buckets have more than one
shard and you didn't reshard them manually, you'll know for sure dynamic
resharding is working; you can check the number of shards on a bucket with `radosgw-admin
bucket stats --bucket=<name>`, there's a `num_shards` field. You can also check
with `radosgw-admin bucket limit check` if any of your buckets are about to be
resharded.
Assuming dynamic resharding is enabled and none of your buckets are about to be resharded,
I would then find out which object has too many OMAP keys by grepping the logs. The name
of the object will contain the bucket ID (also found in the output of `radosgw-admin
bucket stats`), so you'll know which bucket is causing the issue. And you can check
how many OMAP keys are in each shard of that bucket index using
```
for obj in $(rados -p default.rgw.buckets.index ls | grep
eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4); do
printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys
$obj | wc -l) done ```
(where `eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4` is your bucket ID). If the number
of keys are very uneven amongst the shards, there's probably an issue that needs to be
addressed. If you they are relatively even but slightly above the warning threshold,
it's probably a versioned bucket, and it should be safe to simply increase the
threshold.
Cheers,
--
Ben