Hello, since some time I've been investigating problems causing buckets' index corruption. In my case it's been because of numerous bugs related to index resharding and bucket lifecycle policies. One of those bugs present in versions prior to 14.2.8 made the index omapkeys' names contain unicode NULL which manifests like this: root@mach0122:~/mkw # radosgw-admin bi list --bucket=sysa-user-logs|grep idx |tail -5 "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031521.gz.u6_qPM0X2mXQRXD0hEfRK2dCc4dD4El\u0000.4", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031600.gz.bvZdIu5KnsAQyXoa_ZWpix-BYD-yvcz\u0000.15", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031600.gz.wX9jHyKnJ21w5DYL_rdfTsb7zH3tUEa\u0000.10", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031601.gz.K6GhZfNzcqDCcjkrn5GskRWS8ufXSbO\u0000.8", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031601.gz.vEr4VCu0QU07te_RHNJ9Wi1cb8tsiYq\u0000.7", I am unable to remove them by rmomapkey. Also I tried to use the method https://github.com/ceph/ceph/blob/8c63b26fe88bb02d894705cb1beec289668fb43d/s... but it fails with: File "./idx_removal.py", line 96, in remove_key iocontext.remove_omap_keys(write_op, key) File "rados.pyx", line 516, in rados.requires.wrapper.validate_func (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:4992) File "rados.pyx", line 3607, in rados.Ioctx.remove_omap_keys (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:45059) File "rados.pyx", line 543, in rados.cstr_list (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:5665) File "rados.pyx", line 539, in rados.cstr (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:5463) TypeError: keys must be a string Does anyone know what else can be done to remove such omapkey? For the obvious reasons I can't use the clear_omap method... Kind regards, Maks Kowalik