[RGW - octopus] too many omapkeys on versioned bucket
Hi, I have one bucket that showed up with a large omap warning, but the amount of objects in the bucket, does not align with the amount of omap keys. The bucket is sharded to get rid of the "large omapkeys" warning. I've counted all the omapkeys of one bucket and it came up with 33.383.622 (rados -p INDEXPOOL listomapkeys INDEXOBJECT | wc -l) I've checked the amount of actual rados objects and it came up with 17.095.877 (rados -p DATAPOOL ls | grep BUCKETMARKER | wc -l) I've checked the bucket index and it came up with 16.738.482 (radosgw-admin bi list --bucket BUCKET | grep -F '"idx":' | wc -l) I have tried to fix it with radosgw-admin bucket check --check-objects --fix --bucket BUCKET but this did not change anything. Is this a known bug or might there be something else going on. How can I investigate further? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
So here is some more weirdness: I've piped a list of all omapkeys into a file: (dedacted customer data with placeholders in <>) # grep -aFn '<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>' omapkeys_list 9844:<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE> 9845:<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>v913<SECOND_PART> 88010:�1000_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE><SECOND_PART> 128981:�1001_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE> # grep -aFn '<PARENT>/<FOLDER><FIRST_PART_OF_FILE><SECOND_PART>' omapkeys_list <Returns nothing> # vim omapkeys_list +88010 (copy pasted from terminal) <80>1000_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>^@<SECOND_PART> Any idea what this is? Am Mo., 13. Feb. 2023 um 13:57 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi, I have one bucket that showed up with a large omap warning, but the amount of objects in the bucket, does not align with the amount of omap keys. The bucket is sharded to get rid of the "large omapkeys" warning.
I've counted all the omapkeys of one bucket and it came up with 33.383.622 (rados -p INDEXPOOL listomapkeys INDEXOBJECT | wc -l) I've checked the amount of actual rados objects and it came up with 17.095.877 (rados -p DATAPOOL ls | grep BUCKETMARKER | wc -l) I've checked the bucket index and it came up with 16.738.482 (radosgw-admin bi list --bucket BUCKET | grep -F '"idx":' | wc -l)
I have tried to fix it with radosgw-admin bucket check --check-objects --fix --bucket BUCKET but this did not change anything.
Is this a known bug or might there be something else going on. How can I investigate further?
Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
I've tried it the other way around and let cat give out all escaped chars and the did the grep: # cat -A omapkeys_list | grep -aFn '<PARENT>/<FOLDER><FIRST_PART_OF_FILE>' 9844:<PARENT>/<FOLDER><FIRST_PART_OF_FILE>$ 9845:<PARENT>/<FOLDER><FIRST_PART_OF_FILE>^@v913^@<SECOND_PART>$ 88010:M-^@1000_<PARENT>/<FOLDER><FIRST_PART_OF_FILE>^@<SECOND_PART>$ 128981:M-^@1001_<PARENT>/<FOLDER><FIRST_PART_OF_FILE>$ Did anyone ever saw something like this? Am Mo., 13. Feb. 2023 um 14:31 Uhr schrieb Boris Behrens <bb@kervyn.de>:
So here is some more weirdness: I've piped a list of all omapkeys into a file: (dedacted customer data with placeholders in <>)
# grep -aFn '<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>' omapkeys_list 9844:<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE> 9845:<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>v913<SECOND_PART> 88010:�1000_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE><SECOND_PART> 128981:�1001_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>
# grep -aFn '<PARENT>/<FOLDER><FIRST_PART_OF_FILE><SECOND_PART>' omapkeys_list <Returns nothing>
# vim omapkeys_list +88010 (copy pasted from terminal) <80>1000_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>^@<SECOND_PART>
Any idea what this is?
Am Mo., 13. Feb. 2023 um 13:57 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi, I have one bucket that showed up with a large omap warning, but the amount of objects in the bucket, does not align with the amount of omap keys. The bucket is sharded to get rid of the "large omapkeys" warning.
I've counted all the omapkeys of one bucket and it came up with 33.383.622 (rados -p INDEXPOOL listomapkeys INDEXOBJECT | wc -l) I've checked the amount of actual rados objects and it came up with 17.095.877 (rados -p DATAPOOL ls | grep BUCKETMARKER | wc -l) I've checked the bucket index and it came up with 16.738.482 (radosgw-admin bi list --bucket BUCKET | grep -F '"idx":' | wc -l)
I have tried to fix it with radosgw-admin bucket check --check-objects --fix --bucket BUCKET but this did not change anything.
Is this a known bug or might there be something else going on. How can I investigate further?
Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
On Mon, Feb 13, 2023 at 8:41 AM Boris Behrens <bb@kervyn.de> wrote:
I've tried it the other way around and let cat give out all escaped chars and the did the grep:
# cat -A omapkeys_list | grep -aFn '<PARENT>/<FOLDER><FIRST_PART_OF_FILE>' 9844:<PARENT>/<FOLDER><FIRST_PART_OF_FILE>$ 9845:<PARENT>/<FOLDER><FIRST_PART_OF_FILE>^@v913^@<SECOND_PART>$ 88010:M-^@1000_<PARENT>/<FOLDER><FIRST_PART_OF_FILE>^@<SECOND_PART>$ 128981:M-^@1001_<PARENT>/<FOLDER><FIRST_PART_OF_FILE>$
Did anyone ever saw something like this?
Am Mo., 13. Feb. 2023 um 14:31 Uhr schrieb Boris Behrens <bb@kervyn.de>:
So here is some more weirdness: I've piped a list of all omapkeys into a file: (dedacted customer data with placeholders in <>)
# grep -aFn '<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>' omapkeys_list 9844:<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE> 9845:<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>v913<SECOND_PART> 88010:�1000_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE><SECOND_PART> 128981:�1001_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>
# grep -aFn '<PARENT>/<FOLDER><FIRST_PART_OF_FILE><SECOND_PART>' omapkeys_list <Returns nothing>
# vim omapkeys_list +88010 (copy pasted from terminal) <80>1000_<PARENT>/<FOLDER>/<FIRST_PART_OF_FILE>^@<SECOND_PART>
Any idea what this is?
Am Mo., 13. Feb. 2023 um 13:57 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi, I have one bucket that showed up with a large omap warning, but the amount of objects in the bucket, does not align with the amount of omap keys. The bucket is sharded to get rid of the "large omapkeys" warning.
I've counted all the omapkeys of one bucket and it came up with 33.383.622 (rados -p INDEXPOOL listomapkeys INDEXOBJECT | wc -l) I've checked the amount of actual rados objects and it came up with 17.095.877 (rados -p DATAPOOL ls | grep BUCKETMARKER | wc -l) I've checked the bucket index and it came up with 16.738.482 (radosgw-admin bi list --bucket BUCKET | grep -F '"idx":' | wc -l)
I have tried to fix it with radosgw-admin bucket check --check-objects --fix --bucket BUCKET but this did not change anything.
Is this a known bug or might there be something else going on. How can I investigate further?
Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
hi Boris, the bucket index is more complicated for versioned buckets than normal ones. i wrote a high-level summary of this in https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/#s3-object-versioni... each object version may have additional keys starting with 1000_ and 1001_. the keys starting with 1000_ are sorted by time (most recent version first), and the 1001_ keys correspond to the ‘olh' entry. the output of `radosgw-admin bi list` should distinguish between these index entry types using the names "plain", "instance", and "olh" it's hard to tell from your email whether there's anything wrong, but i hope this helps with your debugging
participants (2)
-
Boris Behrens
-
Casey Bodley