28 Jul
2020
28 Jul
'20
12:10 p.m.
Hi all,
I'm currently experience some strange behavior in our cluster: the dashboards object
gateway "buckets" submenu is broken and I'm getting 503 errors (however,
"Users" and "Daemons" work flawlessly). Looking into the mgr log gives
me following error:
2020-07-24T12:38:12.695+0200 7f42150f3700 0 [dashboard ERROR rest_client] RGW REST API
failed GET req status: 404
2020-07-24T12:38:12.843+0200 7f42130ef700 0 [dashboard ERROR request] [10.1.0.133:38454]
[GET] [500] [0.039s] [eugen] [513.0B]
/api/rgw/bucket/91e22800581543b5be4654f7b9b0c7cc_102020-07-24T12:38:12.843+0200
7f42130ef700 0 [dashboard ERROR request] [b'{"status": "500 Internal
Server Error", "detail": "The server encountered an unexpected
condition which prevented it from fulfilling the request.", "request_id":
"00a3c78a-d96f-4f8b-b3c4-f24eac99f4a1"}
']
So it took me to the point to look into buckets and I got this list with weird bucket
names (even with dash, but it's actually not allowed):
root@ceph1-40-10~# radosgw-admin buckets list
[
"12345",
"deployment",
"91e22800581543b5be4654f7b9b0c7cc_5",
"91e22800581543b5be4654f7b9b0c7cc_6",
"91e22800581543b5be4654f7b9b0c7cc_11",
"91e22800581543b5be4654f7b9b0c7cc_8",
"91e22800581543b5be4654f7b9b0c7cc_3",
"91e22800581543b5be4654f7b9b0c7cc_17",
[...]
]
If I try to perform some operations on these buckets, I get an error:
root@ceph1-40-10~# radosgw-admin bucket rm --bucket=91e22800581543b5be4654f7b9b0c7cc_15
--purge-objects
2020-07-28T12:40:05.392+0200 7f500024a080 -1 ERROR: unable to remove bucket(2) No such
file or directory
I'm able to change the owner and even rename it, but anything else is ending in the
same error above. This also seems to break the buckets functionality in the dashboard.
My question is: what are those buckets, where do they come from and what is stored in
there? And why one is not able to perform any ops on them?
Any ideas?
Best,
eugen