18 Jul
2023
18 Jul
'23
6:48 p.m.
Hi,
I have trouble with large OMAP files in a cluster in the RGW index pool. Some
background information about the cluster: There is CephFS and RBD usage on the
main cluster but for this issue I think only S3 is interesting.
There is one realm, one zonegroup with two zones which have a bidirectional sync
set up. Since this does not allow for autoresharding we have to do it by hand in
this cluster – looking forward to Reef!
From the logs:
cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 :
cluster [WRN] Large omap object found. Object:
34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head
PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182
The offending bucket looks like this:
# radosgw-admin bucket stats \
| jq '.[] | select(.marker
=="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9")
|"\(.num_shards) \(.usage["rgw.main"].num_objects)"'
-r
131 9463833
Last week the number of objects was about 12 million. Which is why I reshareded
the offending bucket twice, I think. Once to 129 and the second time to 131
because I wanted some leeway (or lieway? scnr, Sage).
Unfortunately, even after a week the objects were still to big (the log line
above is quite recent), so I looked into it again.
# rados -p raum.rgw.buckets.index ls \
|grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \
|sort -V
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10
# rados -p raum.rgw.buckets.index ls \
|grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \
|sort -V \
|xargs -IOMAP sh -c \
'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l'
1013854
1011007
1012287
1011232
1013565
998262
1012777
1012713
1012230
1010690
997111
Apparently, only 11 shards are in use. This would explain why the "Key usage"
(from the log line) is about ten times higher than I would expect.
How can I deal with this issue?
One thing I could try to fix this would be to reshard to a lower number, but I
am not sure if there are any risks associated with "downsharding". After that I
could reshard to something like 97. Or I could directly "downshard" to 97.
Also, the second zone has a similar problem, but as the error messsage lets me
know, this would be a bad idea. Will it just take more time until the sharding
is transferred to the seconds zone?
Best,
Christian Kugler
24 Jul
24 Jul
10:21 p.m.
1. I recommend that you *not* issue another bucket reshard until you figure out what’s
going on.
2. Which version of Ceph are you using?
3. Can you issue a `radosgw-admin metadata get bucket:<bucket-name>` so we can
verify what the current marker is?
4. After you resharded previously, did you get command-line output along the lines of:
2023-07-24T13:33:50.867-0400 7f10359f2a80 1 execute INFO: reshard of bucket
“<bucket-name>" completed successfully
Eric
(he/him)
P.S. It’s likely obvious, but in the above replace <bucket-name> with the actual
bucket name.
On Jul 18, 2023, at 10:18 AM, Christian Kugler
<syphdias+ceph(a)gmail.com> wrote:
Hi,
I have trouble with large OMAP files in a cluster in the RGW index pool. Some
background information about the cluster: There is CephFS and RBD usage on the
main cluster but for this issue I think only S3 is interesting.
There is one realm, one zonegroup with two zones which have a bidirectional sync
set up. Since this does not allow for autoresharding we have to do it by hand in
this cluster – looking forward to Reef!
From the logs:
cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 :
cluster [WRN] Large omap object found. Object:
34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head
PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182
The offending bucket looks like this:
# radosgw-admin bucket stats \
| jq '.[] | select(.marker
=="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9")
|"\(.num_shards) \(.usage["rgw.main"].num_objects)"'
-r
131 9463833
Last week the number of objects was about 12 million. Which is why I reshareded
the offending bucket twice, I think. Once to 129 and the second time to 131
because I wanted some leeway (or lieway? scnr, Sage).
Unfortunately, even after a week the objects were still to big (the log line
above is quite recent), so I looked into it again.
# rados -p raum.rgw.buckets.index ls \
|grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \
|sort -V
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10
# rados -p raum.rgw.buckets.index ls \
|grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \
|sort -V \
|xargs -IOMAP sh -c \
'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l'
1013854
1011007
1012287
1011232
1013565
998262
1012777
1012713
1012230
1010690
997111
Apparently, only 11 shards are in use. This would explain why the "Key usage"
(from the log line) is about ten times higher than I would expect.
How can I deal with this issue?
One thing I could try to fix this would be to reshard to a lower number, but I
am not sure if there are any risks associated with "downsharding". After that
I
could reshard to something like 97. Or I could directly "downshard" to 97.
Also, the second zone has a similar problem, but as the error messsage lets me
know, this would be a bad idea. Will it just take more time until the sharding
is transferred to the seconds zone?
Best,
Christian Kugler
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 Jul
25 Jul
3:02 p.m.
Hi Eric,
1. I recommend that you *not* issue another bucket
reshard until you figure out what’s going on.
Thanks, noted!
2. Which version of Ceph are you using?
17.2.5
I wanted to get the Cluster to Health OK before upgrading. I didn't
see anything that led me to believe that an upgrade could fix the
reshard issue.
3. Can you issue a `radosgw-admin metadata get
bucket:<bucket-name>` so we can verify what the current marker is?
# radosgw-admin metadata get bucket:sql20
{
"key": "bucket:sql20",
"ver": {
"tag": "_hGhtgzjcWY9rO9JP7YlWzt8",
"ver": 3
},
"mtime": "2023-07-12T15:56:55.226784Z",
"data": {
"bucket": {
"name": "sql20",
"marker":
"3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9",
"bucket_id":
"3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1",
"tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"owner": "S3user",
"creation_time": "2023-04-26T09:22:01.681646Z",
"linked": "true",
"has_bucket_info": "false"
}
}
4. After you resharded previously, did you get
command-line output along the lines of:
2023-07-24T13:33:50.867-0400 7f10359f2a80 1 execute INFO: reshard of bucket
“<bucket-name>" completed successfully
I think so, at least for the second reshard. But I wouldn't bet my
life on it. I fear I might have missed an error on the first one since
I have done a radosgw-admin bucket reshard so often and never seen it
fail.
Christian
28 Jul
28 Jul
11 p.m.
Thank you for the information, Christian. When you reshard the bucket id is updated (with
most recent versions of ceph, a generation number is incremented). The first bucket id
matches the bucket marker, but after the first reshard they diverge.
The bucket id is in the names of the currently used bucket index shards. You’re searching
for the marker, which means you’re finding older bucket index shards.
Change your commands to these:
# rados -p raum.rgw.buckets.index ls \
|grep 3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1 \
|sort -V
# rados -p raum.rgw.buckets.index ls \
|grep 3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1 \
|sort -V \
|xargs -IOMAP sh -c \
'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l'
When you refer to the “second zone”, what do you mean? Is this cluster using multisite? If
and only if your answer is “no”, then it’s safe to remove old bucket index shards.
Depending on the version of ceph running when reshard was run, they were either
intentionally left behind (earlier behavior) or removed automatically (later behavior).
Eric
(he/him)
On Jul 25, 2023, at 6:32 AM, Christian Kugler
<syphdias+ceph(a)gmail.com> wrote:
Hi Eric,
1. I recommend that you *not* issue another
bucket reshard until you figure out what’s going on.
Thanks, noted!
2. Which version of Ceph are you using?
17.2.5
I wanted to get the Cluster to Health OK before upgrading. I didn't
see anything that led me to believe that an upgrade could fix the
reshard issue.
3. Can you issue a `radosgw-admin metadata get
bucket:<bucket-name>` so we can verify what the current marker is?
# radosgw-admin metadata get bucket:sql20
{
"key": "bucket:sql20",
"ver": {
"tag": "_hGhtgzjcWY9rO9JP7YlWzt8",
"ver": 3
},
"mtime": "2023-07-12T15:56:55.226784Z",
"data": {
"bucket": {
"name": "sql20",
"marker":
"3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9",
"bucket_id":
"3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1",
"tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"owner": "S3user",
"creation_time": "2023-04-26T09:22:01.681646Z",
"linked": "true",
"has_bucket_info": "false"
}
}
4. After you resharded previously, did you get
command-line output along the lines of:
2023-07-24T13:33:50.867-0400 7f10359f2a80 1 execute INFO: reshard of bucket
“<bucket-name>" completed successfully
I think so, at least for the second reshard. But I wouldn't bet my
life on it. I fear I might have missed an error on the first one since
I have done a radosgw-admin bucket reshard so often and never seen it
fail.
Christian
2 Aug
2 Aug
2:06 p.m.
Thank you for the information, Christian. When you
reshard the bucket id is updated (with most recent versions of ceph, a generation number
is incremented). The first bucket id matches the bucket marker, but after the first
reshard they diverge.
This makes a lot of sense and explains why the large omap objects do
not go away. It is the old shards that are too big.
The bucket id is in the names of the currently used
bucket index shards. You’re searching for the marker, which means you’re finding older
bucket index shards.
Change your commands to these:
# rados -p raum.rgw.buckets.index ls \
|grep 3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1 \
|sort -V
# rados -p raum.rgw.buckets.index ls \
|grep 3caabb9a-4e3b-4b8a-8222-34c33dd63210.10648356.1 \
|sort -V \
|xargs -IOMAP sh -c \
'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l'
I don't think the outputs are very interesting here. They are as expected:
- 131 lines of rados objects (omap)
- each omap contains about 70k keys (below the 100k limit).
When you refer to the “second zone”, what do you mean?
Is this cluster using multisite? If and only if your answer is “no”, then it’s safe to
remove old bucket index shards. Depending on the version of ceph running when reshard was
run, they were either intentionally left behind (earlier behavior) or removed
automatically (later behavior).
Yes, this cluster uses multisite. It is one realm, one zonegroup with
two zones (bidirectional sync).
Ceph warns about resharding on the non-metadata zone. So I did not do
that and only resharded on the metadata zone.
The resharding was done using a radosgw-admin v16.2.6 on a ceph
cluster running v17.2.5.
Is there a way to get rid of the old (big) shards without breaking something?
Christian
292
days inactive
307
days old
4 comments
2 participants
participants (2)
-
Christian Kugler -
J. Eric Ivancich