Hi all, I'm currently experience some strange behavior in our cluster: the dashboards object gateway "buckets" submenu is broken and I'm getting 503 errors (however, "Users" and "Daemons" work flawlessly). Looking into the mgr log gives me following error: 2020-07-24T12:38:12.695+0200 7f42150f3700 0 [dashboard ERROR rest_client] RGW REST API failed GET req status: 404 2020-07-24T12:38:12.843+0200 7f42130ef700 0 [dashboard ERROR request] [10.1.0.133:38454] [GET] [500] [0.039s] [eugen] [513.0B] /api/rgw/bucket/91e22800581543b5be4654f7b9b0c7cc_102020-07-24T12:38:12.843+0200 7f42130ef700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "00a3c78a-d96f-4f8b-b3c4-f24eac99f4a1"} '] So it took me to the point to look into buckets and I got this list with weird bucket names (even with dash, but it's actually not allowed): root@ceph1-40-10~# radosgw-admin buckets list [ "12345", "deployment", "91e22800581543b5be4654f7b9b0c7cc_5", "91e22800581543b5be4654f7b9b0c7cc_6", "91e22800581543b5be4654f7b9b0c7cc_11", "91e22800581543b5be4654f7b9b0c7cc_8", "91e22800581543b5be4654f7b9b0c7cc_3", "91e22800581543b5be4654f7b9b0c7cc_17", [...] ] If I try to perform some operations on these buckets, I get an error: root@ceph1-40-10~# radosgw-admin bucket rm --bucket=91e22800581543b5be4654f7b9b0c7cc_15 --purge-objects 2020-07-28T12:40:05.392+0200 7f500024a080 -1 ERROR: unable to remove bucket(2) No such file or directory I'm able to change the owner and even rename it, but anything else is ending in the same error above. This also seems to break the buckets functionality in the dashboard. My question is: what are those buckets, where do they come from and what is stored in there? And why one is not able to perform any ops on them? Any ideas? Best, eugen