Not all Bucket Shards being used
Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler
1. I recommend that you *not* issue another bucket reshard until you figure out what’s going on. 2. Which version of Ceph are you using? 3. Can you issue a `radosgw-admin metadata get bucket:<bucket-name>` so we can verify what the current marker is? 4. After you resharded previously, did you get command-line output along the lines of: 2023-07-24T13:33:50.867-0400 7f10359f2a80 1 execute INFO: reshard of bucket “<bucket-name>" completed successfully Eric (he/him) P.S. It’s likely obvious, but in the above replace <bucket-name> with the actual bucket name.
On Jul 18, 2023, at 10:18 AM, Christian Kugler <syphdias+ceph@gmail.com> wrote:
Hi,
I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef!
From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182
The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833
Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage).
Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again.
# rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111
Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect.
How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97.
Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone?
Best, Christian Kugler _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eric,
1. I recommend that you *not* issue another bucket reshard until you figure out what’s going on.
Thanks, noted!
2. Which version of Ceph are you using? 17.2.5 I wanted to get the Cluster to Health OK before upgrading. I didn't see anything that led me to believe that an upgrade could fix the reshard issue.
3. Can you issue a `radosgw-admin metadata get bucket:<bucket-name>` so we can verify what the current marker is?
# radosgw-admin metadata get bucket:sql20 { "key": "bucket:sql20", "ver": { "tag": "_hGhtgzjcWY9rO9JP7YlWzt8", "ver": 3 }, "mtime": "2023-07-12T15:56:55.226784Z", "data": { "bucket": { "name": "sql20", "marker": "3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9", "bucket_id": "3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1", "tenant": "", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" } }, "owner": "S3user", "creation_time": "2023-04-26T09:22:01.681646Z", "linked": "true", "has_bucket_info": "false" } }
4. After you resharded previously, did you get command-line output along the lines of: 2023-07-24T13:33:50.867-0400 7f10359f2a80 1 execute INFO: reshard of bucket “<bucket-name>" completed successfully
I think so, at least for the second reshard. But I wouldn't bet my life on it. I fear I might have missed an error on the first one since I have done a radosgw-admin bucket reshard so often and never seen it fail. Christian
Thank you for the information, Christian. When you reshard the bucket id is updated (with most recent versions of ceph, a generation number is incremented). The first bucket id matches the bucket marker, but after the first reshard they diverge. The bucket id is in the names of the currently used bucket index shards. You’re searching for the marker, which means you’re finding older bucket index shards. Change your commands to these: # rados -p raum.rgw.buckets.index ls \ |grep 3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1 \ |sort -V # rados -p raum.rgw.buckets.index ls \ |grep 3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' When you refer to the “second zone”, what do you mean? Is this cluster using multisite? If and only if your answer is “no”, then it’s safe to remove old bucket index shards. Depending on the version of ceph running when reshard was run, they were either intentionally left behind (earlier behavior) or removed automatically (later behavior). Eric (he/him)
On Jul 25, 2023, at 6:32 AM, Christian Kugler <syphdias+ceph@gmail.com> wrote:
Hi Eric,
1. I recommend that you *not* issue another bucket reshard until you figure out what’s going on.
Thanks, noted!
2. Which version of Ceph are you using? 17.2.5 I wanted to get the Cluster to Health OK before upgrading. I didn't see anything that led me to believe that an upgrade could fix the reshard issue.
3. Can you issue a `radosgw-admin metadata get bucket:<bucket-name>` so we can verify what the current marker is?
# radosgw-admin metadata get bucket:sql20 { "key": "bucket:sql20", "ver": { "tag": "_hGhtgzjcWY9rO9JP7YlWzt8", "ver": 3 }, "mtime": "2023-07-12T15:56:55.226784Z", "data": { "bucket": { "name": "sql20", "marker": "3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9", "bucket_id": "3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1", "tenant": "", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" } }, "owner": "S3user", "creation_time": "2023-04-26T09:22:01.681646Z", "linked": "true", "has_bucket_info": "false" } }
4. After you resharded previously, did you get command-line output along the lines of: 2023-07-24T13:33:50.867-0400 7f10359f2a80 1 execute INFO: reshard of bucket “<bucket-name>" completed successfully
I think so, at least for the second reshard. But I wouldn't bet my life on it. I fear I might have missed an error on the first one since I have done a radosgw-admin bucket reshard so often and never seen it fail.
Christian
Thank you for the information, Christian. When you reshard the bucket id is updated (with most recent versions of ceph, a generation number is incremented). The first bucket id matches the bucket marker, but after the first reshard they diverge.
This makes a lot of sense and explains why the large omap objects do not go away. It is the old shards that are too big.
The bucket id is in the names of the currently used bucket index shards. You’re searching for the marker, which means you’re finding older bucket index shards.
Change your commands to these:
# rados -p raum.rgw.buckets.index ls \ |grep 3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1 \ |sort -V
# rados -p raum.rgw.buckets.index ls \ |grep 3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l'
I don't think the outputs are very interesting here. They are as expected: - 131 lines of rados objects (omap) - each omap contains about 70k keys (below the 100k limit).
When you refer to the “second zone”, what do you mean? Is this cluster using multisite? If and only if your answer is “no”, then it’s safe to remove old bucket index shards. Depending on the version of ceph running when reshard was run, they were either intentionally left behind (earlier behavior) or removed automatically (later behavior).
Yes, this cluster uses multisite. It is one realm, one zonegroup with two zones (bidirectional sync). Ceph warns about resharding on the non-metadata zone. So I did not do that and only resharded on the metadata zone. The resharding was done using a radosgw-admin v16.2.6 on a ceph cluster running v17.2.5. Is there a way to get rid of the old (big) shards without breaking something? Christian
Even after upgrading to Reef and enabling resharding on the multi-site cluster. The large omap objects did not go away. Today I noticed in the Squid release notes two new commands: - radosgw-admin bucket check olh [--fix] - radosgw-admin bucket check unlinked [--fix] Two questions: 1. Is there a chance these commands find the objects and delete them or is this referring to S3 objects rather than unused RADOS OMAP objects? 2. Is this safe to run on a multi-site cluster? Christian
participants (2)
-
Christian Kugler
-
J. Eric Ivancich