Hey you all, so today our certificates got renewed and we suddenly have the problem that every ls returns a 404
# rclone ls s3-nichtmuk: 2026/08/17 16:49:58 NOTICE: Failed to ls: operation error S3: ListBuckets, https response error StatusCode: 404, RequestID: tx00000d5116646fe615090-006a831f96-885923856-rub-z, HostID: 885923856-rub-z-rub-zg, api error NoSuchBucket: UnknownError
We are not sure what caused it but one likely culprit is a test with a placement rule we created and removed a while ago. We created a test placement and later removed it, but still got a user that is in that placement and because ceph still things that the user exists we get an error on the rgw startup
Aug 17 16:43:03 my-hostname radosgw[49228]: deferred set uid:gid to 167:167 (ceph:ceph) Aug 17 16:43:03 my-hostname radosgw[49228]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo), process radosgw, pid 8 Aug 17 16:43:03 my-hostname radosgw[49228]: framework: beast Aug 17 16:43:03 my-hostname radosgw[49228]: framework conf key: port, val: 8001 Aug 17 16:43:03 my-hostname radosgw[49228]: init_numa not setting numa affinity Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Realm: rub-realm (4d725f78-42cc-4ec1-8a0e-feff5800bdfb) Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: ZoneGroup: rub-zg (4170a834-cbe5-4cee-b4f9-83a47941c5a3) Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Zone: rub-z (99d273b2-e74c-432f-94de-1a73886419e8) Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: using period configuration: bcf5d281-9eb3-4a13-a011-7cc94c040cdf:19 Aug 17 16:43:04 my-hostname radosgw[49228]: all 8 watchers are set, enabling cache Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: garbage collection: start Aug 17 16:43:04 my-hostname radosgw[49228]: rgw object expirer Worker thread: object expiration: start Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: garbage collection: stop Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: watch_reload::DEDUP_WATCH_OBJ exists Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: dedup_bg::watch_reload(): Started watching DEDUP_WATCH_OBJ::watch_handle=94372625010048 Aug 17 16:43:04 my-hostname radosgw[49228]: framework: beast Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key Aug 17 16:43:04 my-hostname radosgw[49228]: starting handler: beast Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on 0.0.0.0:8001 Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on [::]:8001 Aug 17 16:43:05 my-hostname radosgw[49228]: set uid:gid to 167:167 (ceph:ceph) Aug 17 16:43:05 my-hostname radosgw[49228]: mgrc service_daemon_register rgw.885923856 metadata {arch=x86_64,ceph_release=tentacle,ceph_version=ceph version 20.2.0 (69f84cc2651aa259a15bc 192ddaabd3baba07489) tentacle (stable - RelWithDebInfo),ceph_version_short=20.2.0,container_hostname=my-hostname,container_image=quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b 6dac688bcc77369b06009b5830fa8d86,cpu=Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz,distro=centos,distro_description=CentOS Stream 9,distro_version=9,frontend_config#0=beast port=8001,frontend_type#0=beast,hostname=rub-01-osd.c eph-2.it-services.ruhr-uni-bochum.de,id=rub-rgw.rub-01-osd.ckfkhi,kernel_description=#1 SMP Thu May 18 10:27:05 EDT 2023,kernel_version=4.18.0-477.13.1.el8_8.x86_64,mem_swap_kb=16777212,mem_total_kb=131086224,num_handles=1, os=Linux,pid=8,realm_id=4d725f78-42cc-4ec1-8a0e-feff5800bdfb,realm_name=rub-realm,zone_id=99d273b2-e74c-432f-94de-1a73886419e8,zone_name=rub-z,zonegroup_id=4170a834-cbe5-4cee-b4f9-83a47941c5a3,zonegroup_name=rub-zg} Aug 17 16:43:05 my-hostname radosgw[49228]: rgw period pusher: No zones to update Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: ERROR: could not sync bucket stats: ret=-5 Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: ERROR: failed user stats sync, ret=-5 Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: ERROR: sync_owner() failed, owner=rub-fowi-katzen ret=-5 Aug 17 16:43:06 my-hostname radosgw[49228]: rgw object expirer Worker thread: object expiration: stop Aug 17 16:43:22 my-hostname radosgw[49228]: ====== starting new request req=0x7fd2a608b310 request_id=tx0000000924606841dd39d-006a831e0a-885923856-rub-z =====
(set rgw_debug to 5/5 for this)
Aug 17 16:49:54 my-hostname radosgw[49228]: beast: 0x7fd2a608b310: 127.0.0.1 - rub-bucket [17/Aug/2026:14:49:54.826 +0000] "GET /sfb1491-f7?max-keys=1000&prefix= HTTP/1.1" 404 255 - "rclone/v1.74.3" - latency=0.001000081s Aug 17 16:49:58 my-hostname radosgw[49228]: ====== starting new request req=0x7fd2a608b310 request_id=tx00000d5116646fe615090-006a831f96-885923856-rub-z ===== Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 0.000000000s getting op 0 Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 0.000000000s s3:list_bucket verifying requester Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 0.001000082s s3:list_bucket normalizing buckets and tenants Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 0.001000082s s3:list_bucket init permissions Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 0.001000082s s3:list_bucket op status=0 Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 0.001000082s s3:list_bucket http status=404 Aug 17 16:49:58 my-hostname radosgw[49228]: ====== req done req=0x7fd2a608b310 op=list_bucket bucket=our-s3-host status=0 http_status=404 latency=0.
We are not sure how to remove the user or debug the problem right now
# rubgw user rm --uid=rub-fowi-katzen --purge-data could not remove user: unable to remove user, unable to delete user data 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: cls_bucket_list_unordered: error in rgw_rados_operate (bucket list op), r=-2 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: list_objects_unordered cls_bucket_list_unordered returned -2 for :FOO[99d273b2-e74c-432f-94de-1a73886419e8.791780109.1])
Any help or hints would be greatly appreciated Kind regards, Thomas -- Thomas Schneider IT.SERVICES Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 Bochum Telefon: +49 234 32 23939 http://www.it-services.rub.de/