- ceph-users - lists.ceph.io

cephx key rotation - blue green rollout
by mudman 19 Aug '26

19 Aug '26

Hi, I'm noticing that ceph does not have dual validity for keys. Is that correct? Because of this, I am thinking to use a blue green process to accomplish cephx key rotation. The caps of the existing client can be cloned with something like the below snippet, watch out for typos... And then clients can do rollout with new client+credentials with reduced impact. AUTH_OUT=$(ceph auth get "$SRC" 2>&1) || { echo “ERROR: 'ceph auth get $SRC' failed. Does the client exist?" >&2 exit 1 } CAP_ARGS=() while IFS= read -r line; do # Match: caps <daemon> = "<perms>" if [[ "$line" =~ ^[[:space:]]*caps[[:space:]]+([a-z]+)[[: space:]]*=[[:space:]]*\"(.*)\"[[:space:]]*$ ]]; then daemon="${BASH_REMATCH[1]}" perms="${BASH_REMATCH[2]}" CAP_ARGS+=("$daemon" “$perms” ) fi done <<< "“$AUTH_OUT" if [[ ${#CAP_ARGS[@]} -eq 0 ]]; then echo “ERROR: No caps found for $SRC. Refusing to create a client with no capabilities." >&2 exit 1 fi echo "Caps to apply to $NEW:" for (( i=0; i<${#CAP_ARGS[@]}; i+=2 )); do printf " %-6s = \"%s\"\n" "${CAP_ARGS[i]}" "${CAP_ARGS[i+1]}" done echo "" echo "Creating $NEW ..." ceph auth get-or-create "$NEW" "${CAP_ARGS[@]}" -o "$OUTFILE"

1 0

Re: Bucket listing suddenly 404
by Eugen Block 19 Aug '26

19 Aug '26

I feel like rgw_dns_name popped up more often recently, not sure if here on this list or on slack. I don't have anything to back that feeling up, though. But it's great that you figured it out. Zitat von Thomas Schneider <ceph-users(a)ceph.io>: > Ok it's completely solved. > I too late realized that suddenly when calling the s3 gateway > instead of seeing the default xml I saw our domain as Bucket. > I _somehow_ after nearly 10 years of running the rgw had to set > rgw_dns_name (again?) > I'm too tired right now to analyze it any further. > Hope this helps _someone_ in the future. > > On 2026-08-17 19:18, Thomas Schneider wrote: >> We "solved" the sync_owner() problem by deleting all Buckets owned >> by the user by changing the placement >>> # rubgw is radosgw-admin with realm/zone/zonegroup set >>> $rubgw metadata get "bucket.instance:$bucket:$meta" > $bucket-old.json >>> cp $bucket-old.json $bucket-new.json >>> sed -i 's/"placement_rule": "[^"]*"/"placement_rule": >>> "default-placement"/g' $bucket-new.json >>> $rubgw metadata put "bucket.instance:$bucket:$meta" >>> --infile=$bucket-new.json >> >> and then deleting the buckets, but the 404 problem when trying to >> list/access buckets still exist :/ >> >> I'm all out of ideas right now. >> >> On 2026-08-17 16:57, Thomas Schneider wrote: >>> Hey you all, >>> >>> so today our certificates got renewed and we suddenly have the >>> problem that every ls returns a 404 >>>> # rclone ls s3-nichtmuk: >>>> 2026/08/17 16:49:58 NOTICE: Failed to ls: operation error S3: >>>> ListBuckets, https response error StatusCode: 404, RequestID: >>>> tx00000d5116646fe615090-006a831f96-885923856-rub-z, HostID: >>>> 885923856-rub-z-rub-zg, api error NoSuchBucket: UnknownError >>> >>> We are not sure what caused it but one likely culprit is a test >>> with a placement rule we created and removed a while ago. >>> We created a test placement and later removed it, but still got a >>> user that is in that placement and because ceph still things that >>> the user exists we get an error on the rgw startup >>> >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: deferred set uid:gid >>>> to 167:167 (ceph:ceph) >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: ceph version 20.2.0 >>>> (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - >>>> RelWithDebInfo), process radosgw, pid 8 >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: framework: beast >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: framework conf key: >>>> port, val: 8001 >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: init_numa not setting >>>> numa affinity >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Realm: >>>> rub-realm (4d725f78-42cc-4ec1-8a0e-feff5800bdfb) >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: ZoneGroup: >>>> rub-zg (4170a834-cbe5-4cee-b4f9-83a47941c5a3) >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Zone: >>>> rub-z (99d273b2-e74c-432f-94de-1a73886419e8) >>>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: using >>>> period configuration: bcf5d281-9eb3-4a13-a011-7cc94c040cdf:19 >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: all 8 watchers are >>>> set, enabling cache >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: >>>> garbage collection: start >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw object expirer >>>> Worker thread: object expiration: start >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: >>>> garbage collection: stop >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: >>>> watch_reload::DEDUP_WATCH_OBJ exists >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: >>>> dedup_bg::watch_reload(): Started watching >>>> DEDUP_WATCH_OBJ::watch_handle=94372625010048 >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework: beast >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: >>>> ssl_certificate, val: config://rgw/cert/$realm/$zone.crt >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: >>>> ssl_private_key, val: config://rgw/cert/$realm/$zone.key >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: starting handler: beast >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on >>>> 0.0.0.0:8001 >>>> Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on >>>> [::]:8001 >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: set uid:gid to >>>> 167:167 (ceph:ceph) >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: mgrc >>>> service_daemon_register rgw.885923856 metadata >>>> {arch=x86_64,ceph_release=tentacle,ceph_version=ceph version >>>> 20.2.0 (69f84cc2651aa259a15bc >>>> 192ddaabd3baba07489) tentacle (stable - >>>> RelWithDebInfo),ceph_version_short=20.2.0,container_hostname=my-hostname,container_image=quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b >>>> 6dac688bcc77369b06009b5830fa8d86,cpu=Intel(R) Xeon(R) Silver 4114 >>>> CPU @ 2.20GHz,distro=centos,distro_description=CentOS Stream >>>> 9,distro_version=9,frontend_config#0=beast >>>> port=8001,frontend_type#0=beast,hostname=rub-01-osd.c >>>> eph-2.it-services.ruhr-uni-bochum.de,id=rub-rgw.rub-01-osd.ckfkhi,kernel_description=#1 SMP Thu May 18 10:27:05 EDT >>>> 2023,kernel_version=4.18.0-477.13.1.el8_8.x86_64,mem_swap_kb=16777212,mem_total_kb=131086224,num_handles=1, >>>> os=Linux,pid=8,realm_id=4d725f78-42cc-4ec1-8a0e-feff5800bdfb,realm_name=rub-realm,zone_id=99d273b2-e74c-432f-94de-1a73886419e8,zone_name=rub-z,zonegroup_id=4170a834-cbe5-4cee-b4f9-83a47941c5a3,zonegroup_name=rub-zg} >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw period pusher: No >>>> zones to update >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>>> ERROR: could not sync bucket stats: ret=-5 >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>>> ERROR: failed user stats sync, ret=-5 >>>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>>> ERROR: sync_owner() failed, owner=rub-fowi-katzen ret=-5 >>>> Aug 17 16:43:06 my-hostname radosgw[49228]: rgw object expirer >>>> Worker thread: object expiration: stop >>>> Aug 17 16:43:22 my-hostname radosgw[49228]: ====== starting new >>>> request req=0x7fd2a608b310 >>>> request_id=tx0000000924606841dd39d-006a831e0a-885923856-rub-z ===== >>> >>> >>> (set rgw_debug to 5/5 for this) >>>> Aug 17 16:49:54 my-hostname radosgw[49228]: beast: >>>> 0x7fd2a608b310: 127.0.0.1 - rub-bucket [17/Aug/2026:14:49:54.826 >>>> +0000] "GET /sfb1491-f7?max-keys=1000&prefix= HTTP/1.1" 404 255 - >>>> "rclone/v1.74.3" - latency=0.001000081s >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: ====== starting new >>>> request req=0x7fd2a608b310 >>>> request_id=tx00000d5116646fe615090-006a831f96-885923856-rub-z ===== >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.000000000s getting op 0 >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.000000000s s3:list_bucket verifying >>>> requester >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.001000082s s3:list_bucket normalizing >>>> buckets and tenants >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.001000082s s3:list_bucket init permissions >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.001000082s s3:list_bucket op status=0 >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: req >>>> 15353165059784265872 0.001000082s s3:list_bucket http status=404 >>>> Aug 17 16:49:58 my-hostname radosgw[49228]: ====== req done >>>> req=0x7fd2a608b310 op=list_bucket bucket=our-s3-host status=0 >>>> http_status=404 latency=0. >>> >>> >>> We are not sure how to remove the user or debug the problem right now >>>> # rubgw user rm --uid=rub-fowi-katzen --purge-data >>>> could not remove user: unable to remove user, unable to delete user data >>>> 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: >>>> cls_bucket_list_unordered: error in rgw_rados_operate (bucket >>>> list op), r=-2 >>>> 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: >>>> list_objects_unordered cls_bucket_list_unordered returned -2 for >>>> :FOO[99d273b2-e74c-432f-94de-1a73886419e8.791780109.1]) >>> >>> Any help or hints would be greatly appreciated >>> >>> Kind regards, >>> Thomas > > -- > Thomas Schneider > IT.SERVICES > Wissenschaftliche Informationsversorgung > Ruhr-Universität Bochum | 44780 Bochum > Telefon: +49 234 32 23939 > http://www.it-services.rub.de/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Affects of high pg count on OSD peering
by Eugen Block 18 Aug '26

18 Aug '26

Hi, the IO should not be blocked for the entire pool but only for those repeering PGs. Warnings during that phase are expected. Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement. One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds). That could (or probably would) increase network traffic significantly, so it's a trade-off. If you have a test cluster with similar behavior, test it there first. Regards, Eugen Zitat von Elias Carter via ceph-users <ceph-users(a)ceph.io>: > We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and > upwards). > > Something that I have noticed is that CRUSH changes (marking an OSD as out, > or a new OSD joining the cluster) can cause many hundreds of PGs to remap > and peer. During this time we get PG_AVAILABILITY health check warnings as > the OSDs need to peer with their new PGs before they can serve IOs. RADOS > tail latency suffers because IOs sent to a peering PG are blocked until > peering completes. > > I believe that the more PGs we have, the more PGs need to peer, and the > longer IOs are blocked for the pool. > > Does anyone have suggestions on how to deal with PG peering blocking IOs? > > I was thinking about gradually re-weighting OSDs instead of marking them > out or in, but that can result in multiple redundant data moves. > > Elias > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Bucket listing suddenly 404
by Thomas Schneider 17 Aug '26

17 Aug '26

Ok it's completely solved. I too late realized that suddenly when calling the s3 gateway instead of seeing the default xml I saw our domain as Bucket. I _somehow_ after nearly 10 years of running the rgw had to set rgw_dns_name (again?) I'm too tired right now to analyze it any further. Hope this helps _someone_ in the future. On 2026-08-17 19:18, Thomas Schneider wrote: > We "solved" the sync_owner() problem by deleting all Buckets owned by > the user by changing the placement >> # rubgw is radosgw-admin with realm/zone/zonegroup set >> $rubgw metadata get "bucket.instance:$bucket:$meta" > $bucket-old.json >> cp $bucket-old.json $bucket-new.json >> sed -i 's/"placement_rule": "[^"]*"/"placement_rule": >> "default-placement"/g' $bucket-new.json >> $rubgw metadata put "bucket.instance:$bucket:$meta" >> --infile=$bucket-new.json > > and then deleting the buckets, but the 404 problem when trying to > list/access buckets still exist :/ > > I'm all out of ideas right now. > > On 2026-08-17 16:57, Thomas Schneider wrote: >> Hey you all, >> >> so today our certificates got renewed and we suddenly have the problem >> that every ls returns a 404 >>> # rclone ls s3-nichtmuk: >>> 2026/08/17 16:49:58 NOTICE: Failed to ls: operation error S3: >>> ListBuckets, https response error StatusCode: 404, RequestID: >>> tx00000d5116646fe615090-006a831f96-885923856-rub-z, HostID: >>> 885923856-rub-z-rub-zg, api error NoSuchBucket: UnknownError >> >> We are not sure what caused it but one likely culprit is a test with a >> placement rule we created and removed a while ago. >> We created a test placement and later removed it, but still got a user >> that is in that placement and because ceph still things that the user >> exists we get an error on the rgw startup >> >>> Aug 17 16:43:03 my-hostname radosgw[49228]: deferred set uid:gid to >>> 167:167 (ceph:ceph) >>> Aug 17 16:43:03 my-hostname radosgw[49228]: ceph version 20.2.0 >>> (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - >>> RelWithDebInfo), process radosgw, pid 8 >>> Aug 17 16:43:03 my-hostname radosgw[49228]: framework: beast >>> Aug 17 16:43:03 my-hostname radosgw[49228]: framework conf key: port, >>> val: 8001 >>> Aug 17 16:43:03 my-hostname radosgw[49228]: init_numa not setting >>> numa affinity >>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Realm: >>> rub-realm (4d725f78-42cc-4ec1-8a0e-feff5800bdfb) >>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: ZoneGroup: >>> rub-zg (4170a834-cbe5-4cee-b4f9-83a47941c5a3) >>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Zone: >>> rub-z (99d273b2-e74c-432f-94de-1a73886419e8) >>> Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: using period >>> configuration: bcf5d281-9eb3-4a13-a011-7cc94c040cdf:19 >>> Aug 17 16:43:04 my-hostname radosgw[49228]: all 8 watchers are set, >>> enabling cache >>> Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: >>> garbage collection: start >>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw object expirer Worker >>> thread: object expiration: start >>> Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: >>> garbage collection: stop >>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: >>> watch_reload::DEDUP_WATCH_OBJ exists >>> Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: >>> dedup_bg::watch_reload(): Started watching >>> DEDUP_WATCH_OBJ::watch_handle=94372625010048 >>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework: beast >>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: >>> ssl_certificate, val: config://rgw/cert/$realm/$zone.crt >>> Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: >>> ssl_private_key, val: config://rgw/cert/$realm/$zone.key >>> Aug 17 16:43:04 my-hostname radosgw[49228]: starting handler: beast >>> Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on >>> 0.0.0.0:8001 >>> Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on >>> [::]:8001 >>> Aug 17 16:43:05 my-hostname radosgw[49228]: set uid:gid to 167:167 >>> (ceph:ceph) >>> Aug 17 16:43:05 my-hostname radosgw[49228]: mgrc >>> service_daemon_register rgw.885923856 metadata >>> {arch=x86_64,ceph_release=tentacle,ceph_version=ceph version 20.2.0 >>> (69f84cc2651aa259a15bc >>> 192ddaabd3baba07489) tentacle (stable - >>> RelWithDebInfo),ceph_version_short=20.2.0,container_hostname=my-hostname,container_image=quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b >>> 6dac688bcc77369b06009b5830fa8d86,cpu=Intel(R) Xeon(R) Silver 4114 CPU >>> @ 2.20GHz,distro=centos,distro_description=CentOS Stream >>> 9,distro_version=9,frontend_config#0=beast >>> port=8001,frontend_type#0=beast,hostname=rub-01-osd.c >>> eph-2.it-services.ruhr-uni-bochum.de,id=rub-rgw.rub-01-osd.ckfkhi,kernel_description=#1 >>> SMP Thu May 18 10:27:05 EDT >>> 2023,kernel_version=4.18.0-477.13.1.el8_8.x86_64,mem_swap_kb=16777212,mem_total_kb=131086224,num_handles=1, >>> os=Linux,pid=8,realm_id=4d725f78-42cc-4ec1-8a0e-feff5800bdfb,realm_name=rub-realm,zone_id=99d273b2-e74c-432f-94de-1a73886419e8,zone_name=rub-z,zonegroup_id=4170a834-cbe5-4cee-b4f9-83a47941c5a3,zonegroup_name=rub-zg} >>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw period pusher: No >>> zones to update >>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>> ERROR: could not sync bucket stats: ret=-5 >>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>> ERROR: failed user stats sync, ret=-5 >>> Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: >>> ERROR: sync_owner() failed, owner=rub-fowi-katzen ret=-5 >>> Aug 17 16:43:06 my-hostname radosgw[49228]: rgw object expirer Worker >>> thread: object expiration: stop >>> Aug 17 16:43:22 my-hostname radosgw[49228]: ====== starting new >>> request req=0x7fd2a608b310 >>> request_id=tx0000000924606841dd39d-006a831e0a-885923856-rub-z ===== >> >> >> (set rgw_debug to 5/5 for this) >>> Aug 17 16:49:54 my-hostname radosgw[49228]: beast: 0x7fd2a608b310: >>> 127.0.0.1 - rub-bucket [17/Aug/2026:14:49:54.826 +0000] "GET >>> /sfb1491-f7?max-keys=1000&prefix= HTTP/1.1" 404 255 - >>> "rclone/v1.74.3" - latency=0.001000081s >>> Aug 17 16:49:58 my-hostname radosgw[49228]: ====== starting new >>> request req=0x7fd2a608b310 >>> request_id=tx00000d5116646fe615090-006a831f96-885923856-rub-z ===== >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.000000000s getting op 0 >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.000000000s s3:list_bucket verifying requester >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.001000082s s3:list_bucket normalizing buckets and tenants >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.001000082s s3:list_bucket init permissions >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.001000082s s3:list_bucket op status=0 >>> Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 >>> 0.001000082s s3:list_bucket http status=404 >>> Aug 17 16:49:58 my-hostname radosgw[49228]: ====== req done >>> req=0x7fd2a608b310 op=list_bucket bucket=our-s3-host status=0 >>> http_status=404 latency=0. >> >> >> We are not sure how to remove the user or debug the problem right now >>> # rubgw user rm --uid=rub-fowi-katzen --purge-data >>> could not remove user: unable to remove user, unable to delete user >>> data >>> 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: >>> cls_bucket_list_unordered: error in rgw_rados_operate (bucket list >>> op), r=-2 >>> 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: >>> list_objects_unordered cls_bucket_list_unordered returned -2 for >>> :FOO[99d273b2-e74c-432f-94de-1a73886419e8.791780109.1]) >> >> Any help or hints would be greatly appreciated >> >> Kind regards, >> Thomas -- Thomas Schneider IT.SERVICES Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 Bochum Telefon: +49 234 32 23939 http://www.it-services.rub.de/

1 0

Bucket listing suddenly 404
by Thomas Schneider 17 Aug '26

17 Aug '26

Hey you all, so today our certificates got renewed and we suddenly have the problem that every ls returns a 404 > # rclone ls s3-nichtmuk: > 2026/08/17 16:49:58 NOTICE: Failed to ls: operation error S3: > ListBuckets, https response error StatusCode: 404, RequestID: > tx00000d5116646fe615090-006a831f96-885923856-rub-z, HostID: > 885923856-rub-z-rub-zg, api error NoSuchBucket: UnknownError We are not sure what caused it but one likely culprit is a test with a placement rule we created and removed a while ago. We created a test placement and later removed it, but still got a user that is in that placement and because ceph still things that the user exists we get an error on the rgw startup > Aug 17 16:43:03 my-hostname radosgw[49228]: deferred set uid:gid to > 167:167 (ceph:ceph) > Aug 17 16:43:03 my-hostname radosgw[49228]: ceph version 20.2.0 > (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo), process radosgw, pid 8 > Aug 17 16:43:03 my-hostname radosgw[49228]: framework: beast > Aug 17 16:43:03 my-hostname radosgw[49228]: framework conf key: port, > val: 8001 > Aug 17 16:43:03 my-hostname radosgw[49228]: init_numa not setting numa > affinity > Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Realm: > rub-realm (4d725f78-42cc-4ec1-8a0e-feff5800bdfb) > Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: ZoneGroup: rub-zg > (4170a834-cbe5-4cee-b4f9-83a47941c5a3) > Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: Zone: rub-z > (99d273b2-e74c-432f-94de-1a73886419e8) > Aug 17 16:43:03 my-hostname radosgw[49228]: rgw main: using period > configuration: bcf5d281-9eb3-4a13-a011-7cc94c040cdf:19 > Aug 17 16:43:04 my-hostname radosgw[49228]: all 8 watchers are set, > enabling cache > Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: garbage > collection: start > Aug 17 16:43:04 my-hostname radosgw[49228]: rgw object expirer Worker > thread: object expiration: start > Aug 17 16:43:04 my-hostname radosgw[49228]: garbage collection: garbage > collection: stop > Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: > watch_reload::DEDUP_WATCH_OBJ exists > Aug 17 16:43:04 my-hostname radosgw[49228]: rgw main: > dedup_bg::watch_reload(): Started watching > DEDUP_WATCH_OBJ::watch_handle=94372625010048 > Aug 17 16:43:04 my-hostname radosgw[49228]: framework: beast > Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: > ssl_certificate, val: config://rgw/cert/$realm/$zone.crt > Aug 17 16:43:04 my-hostname radosgw[49228]: framework conf key: > ssl_private_key, val: config://rgw/cert/$realm/$zone.key > Aug 17 16:43:04 my-hostname radosgw[49228]: starting handler: beast > Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on > 0.0.0.0:8001 > Aug 17 16:43:04 my-hostname radosgw[49228]: frontend listening on > [::]:8001 > Aug 17 16:43:05 my-hostname radosgw[49228]: set uid:gid to 167:167 > (ceph:ceph) > Aug 17 16:43:05 my-hostname radosgw[49228]: mgrc > service_daemon_register rgw.885923856 metadata > {arch=x86_64,ceph_release=tentacle,ceph_version=ceph version 20.2.0 > (69f84cc2651aa259a15bc > 192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo),ceph_version_short=20.2.0,container_hostname=my-hostname,container_image=quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b > 6dac688bcc77369b06009b5830fa8d86,cpu=Intel(R) Xeon(R) Silver 4114 CPU @ > 2.20GHz,distro=centos,distro_description=CentOS Stream > 9,distro_version=9,frontend_config#0=beast > port=8001,frontend_type#0=beast,hostname=rub-01-osd.c > eph-2.it-services.ruhr-uni-bochum.de,id=rub-rgw.rub-01-osd.ckfkhi,kernel_description=#1 > SMP Thu May 18 10:27:05 EDT > 2023,kernel_version=4.18.0-477.13.1.el8_8.x86_64,mem_swap_kb=16777212,mem_total_kb=131086224,num_handles=1, > os=Linux,pid=8,realm_id=4d725f78-42cc-4ec1-8a0e-feff5800bdfb,realm_name=rub-realm,zone_id=99d273b2-e74c-432f-94de-1a73886419e8,zone_name=rub-z,zonegroup_id=4170a834-cbe5-4cee-b4f9-83a47941c5a3,zonegroup_name=rub-zg} > Aug 17 16:43:05 my-hostname radosgw[49228]: rgw period pusher: No zones > to update > Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: > ERROR: could not sync bucket stats: ret=-5 > Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: > ERROR: failed user stats sync, ret=-5 > Aug 17 16:43:05 my-hostname radosgw[49228]: rgw user sync thread: > ERROR: sync_owner() failed, owner=rub-fowi-katzen ret=-5 > Aug 17 16:43:06 my-hostname radosgw[49228]: rgw object expirer Worker > thread: object expiration: stop > Aug 17 16:43:22 my-hostname radosgw[49228]: ====== starting new request > req=0x7fd2a608b310 > request_id=tx0000000924606841dd39d-006a831e0a-885923856-rub-z ===== (set rgw_debug to 5/5 for this) > Aug 17 16:49:54 my-hostname radosgw[49228]: beast: 0x7fd2a608b310: > 127.0.0.1 - rub-bucket [17/Aug/2026:14:49:54.826 +0000] "GET > /sfb1491-f7?max-keys=1000&prefix= HTTP/1.1" 404 255 - "rclone/v1.74.3" > - latency=0.001000081s > Aug 17 16:49:58 my-hostname radosgw[49228]: ====== starting new > request req=0x7fd2a608b310 > request_id=tx00000d5116646fe615090-006a831f96-885923856-rub-z ===== > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.000000000s getting op 0 > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.000000000s s3:list_bucket verifying requester > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.001000082s s3:list_bucket normalizing buckets and tenants > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.001000082s s3:list_bucket init permissions > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.001000082s s3:list_bucket op status=0 > Aug 17 16:49:58 my-hostname radosgw[49228]: req 15353165059784265872 > 0.001000082s s3:list_bucket http status=404 > Aug 17 16:49:58 my-hostname radosgw[49228]: ====== req done > req=0x7fd2a608b310 op=list_bucket bucket=our-s3-host status=0 > http_status=404 latency=0. We are not sure how to remove the user or debug the problem right now > # rubgw user rm --uid=rub-fowi-katzen --purge-data > could not remove user: unable to remove user, unable to delete user > data > 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: > cls_bucket_list_unordered: error in rgw_rados_operate (bucket list op), > r=-2 > 2026-08-17T14:56:53.248+0000 7ffbf3d418c0 0 ERROR: > list_objects_unordered cls_bucket_list_unordered returned -2 for > :FOO[99d273b2-e74c-432f-94de-1a73886419e8.791780109.1]) Any help or hints would be greatly appreciated Kind regards, Thomas -- Thomas Schneider IT.SERVICES Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 Bochum Telefon: +49 234 32 23939 http://www.it-services.rub.de/

1 1

RelWithDebInfo in 20.2.3 Tentacle release (ceph orch package)
by Boris 17 Aug '26

17 Aug '26

Hi, is this intentional? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Ceph Developer Summit Vampire recordings
by Anthony Middleton 14 Aug '26

14 Aug '26

Hey friends, Here are the recordings from the Developer Summit for Vampire: https://www.youtube.com/playlist?list=PLAYbnnO0qf9U Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: [tentacle 20.2.3] How to configure ceph dashboard in a multi subnet deployment
by Alexander Harvey 14 Aug '26

14 Aug '26

I had considerable difficulty getting mgmt-gateway to work here with lots of changes to the YAML and many re-deploys of the dashboard. There are still some problems accessing Prometheus and Grafana that I have not figured out how to solve. The YAML ended up looking like this in the end: --- service_type: ingress service_id: ingress-mgmt-gw service_name: ingress.ingress-mgmt-gw placement: label: mgmt spec: backend_service: mgmt-gateway first_virtual_router_id: 50 keepalive_only: true virtual_ip: .... --- service_type: mgmt-gateway service_name: mgmt-gateway placement: label: mgmt spec: ssl: true ssl_cert: '-----BEGIN CERTIFICATE----- -----END CERTIFICATE----- ' ssl_key: '-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY----- ' --- Alexander Harvey Senior Systems Engineer SAA Infrastructure Team University of Vermont -- From: wodel youchi via ceph-users <ceph-users(a)ceph.io> Date: Thursday, 2026 August 13 at 18:07 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] [tentacle 20.2.3] How to configure ceph dashboard in a multi subnet deployment Hi, In my ceph deployment, I have 3 subnets : mgmt subnet : 172.16.2.0/24 ceph public subnet : 10.10.0.0/16 ceph cluster subnet : 10.20.0.0/16 dashboard is reachable from 10.10.0.0/16 subnet the public ceph. I want to be able to reach it from the mgmt network. I've tried to install keepalived and haproxy to do that, but when I access the mgmt VIP, I'am redirected to the mgr ip on the public subnet. Regards _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: RGW: cannot PUT data into S3-like bucket
by Jan Marek 14 Aug '26

14 Aug '26

Hello, problem was in mtu settings on haproxy host. It has set mtu to 9000, whole CEPH cluster have mtu 9000, but client had mtu 1500. Problem was in the fragmentation. When I set on haproxy host mtu to 1500, problem disappeared. Sincerely Jan Marek Dne st, srp 12, 2026 at 05:12:20 CEST napsal(a) Jan Marek via ceph-users: > Hello, > > I've tried to connect directly to RGW and everythink works fine. > Thus problem is in haproxy, or it's configuration. > > Sincerely > Jan Marek > > Dne st, srp 12, 2026 at 12:51:24 CEST napsal(a) Janne Johansson: > > after upgrade from ceph 19.2.5 to 20.2.2 (and now I'm on 20.2.3), > > I have big problem to write any bigger blocks of data to S3 > > buckets. > > After upgrade to 20.2.2, everythink PUT requests gets 502 HTTP > > code. > > Today I've tried this: > > 1) I've created new bucket > > 2) I set this bucket for rustic (rustic init) with user, which is > > Owner of this buckets. 'rustic init' was fine. > > 3) I've tried to do backup: 'rustic backup'. No one block of > > backup data was written. > > > > I would certainly try a manual S3 client and validate if you can or not put > > objects into buckets, also reading the RGW logs from when this is erroring out > > seems like a very good idea. > > -- > > May the most significant bit of your life be positive. > > -- > Ing. Jan Marek > správce IT > -- Ing. Jan Marek správce IT Jihočeská univerzita v Českých Budějovicích Centum informačních technologií Branišovská 1645/31a, 370 05 České Budějovice T/ +420 38 903 2080 M/ +420 604 660 121 E/ jmarek(a)jcu.cz www.jcu.cz

1 0

[tentacle 20.2.3] How to configure ceph dashboard in a multi subnet deployment
by wodel youchi 13 Aug '26

13 Aug '26

Hi, In my ceph deployment, I have 3 subnets : mgmt subnet : 172.16.2.0/24 ceph public subnet : 10.10.0.0/16 ceph cluster subnet : 10.20.0.0/16 dashboard is reachable from 10.10.0.0/16 subnet the public ceph. I want to be able to reach it from the mgmt network. I've tried to install keepalived and haproxy to do that, but when I access the mgmt VIP, I'am redirected to the mgr ip on the public subnet. Regards

1 0