Hi everybody, we upgraded our containerized Red Hat Pacific cluster to the latest Quincy release (Community Edition). The upgrade itself went fine, the cluster is HEALTH_OK, all daemons run the upgraded version: ---- %< ---- $ ceph -s cluster: id: 68675a58-cf09-4ebd-949c-b9fcc4f2264e health: HEALTH_OK services: mon: 5 daemons, quorum node02,node03,node04,node05,node01 (age 25h) mgr: node03.ztlair(active, since 25h), standbys: node01.koymku, node04.uvxgvp, node02.znqnhg, node05.iifmpc osd: 408 osds: 408 up (since 22h), 408 in (since 7d) rgw: 19 daemons active (19 hosts, 1 zones) data: pools: 11 pools, 8481 pgs objects: 236.99M objects, 544 TiB usage: 1.6 PiB used, 838 TiB / 2.4 PiB avail pgs: 8385 active+clean 79 active+clean+scrubbing+deep 17 active+clean+scrubbing io: client: 42 MiB/s rd, 439 MiB/s wr, 2.15k op/s rd, 1.64k op/s wr --- $ ceph versions | jq .overall { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 437 } ---- >% ---- After all the daemons were upgraded we started noticing some RGW buckets which are inaccessible. s3cmd failed with NoSuchKey: ---- %< ---- $ s3cmd la -l ERROR: S3 error: 404 (NoSuchKey) ---- >% ---- The buckets still exists according to "radosgw-admin bucket list". Out of the ~600 buckets, 13 buckets are unaccessible at the moment: ---- %< ---- $ radosgw-admin bucket radoslist --tenant xy --uid xy --bucket xy 2024-04-03T12:13:40.607+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed 2024-04-03T12:13:40.609+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed ---- >% ---- The affected buckets are comparatively large, around 4 - 7 TB, but not all buckets of that size are affected. Using "rados -p rgw.buckets.data ls" it seems like all the objects are still there, although "rados -p rgw.buckets.data get objectname -" only prints unusable (?) binary data, even for objects of intact buckets. Overall we're facing around 60 TB of customer data which are just gone at the moment. Is there a way to recover from this situation or further narrowing down the root cause of the problem? Kind regards, Lorenz