- ceph-users - lists.ceph.io

Re: [squid] buckets not in bucket stats in multisite setup when create in a non master zonegroup
by Casey Bodley 03 Feb '26

03 Feb '26

thanks Boris, there was some previous discussion in related issue https://tracker.ceph.com/issues/74231, and i've proposed a fix for that On Tue, Feb 3, 2026 at 6:07 AM Boris via ceph-users <ceph-users(a)ceph.io> wrote: > > created a tracker with additional information: > https://tracker.ceph.com/issues/74729 > > Am Di., 3. Feb. 2026 um 01:51 Uhr schrieb Boris <bb(a)kervyn.de>: > > > Hi, > > > > we just encountered a strange problem: > > > > When we run radosgw-admin bucket stats on the master zonegroup we get some > > of these errors > > error getting bucket stats bucket=rregergf ret=-2 > > > > This is the debug_rgw=20 output: > > error getting bucket stats bucket=rregergf > > ret=2026-02-03T00:50:32.347+0000 722007981a80 20 cls_bucket_head: > > CLSRGWIssueGetDirHeader() returned -2 > > -2 > > failure: (2) No such file or directory: > > > > But the bucketname is listed in a bucket list command. > > > > After some digging we found that the bucket was created after the upgrade > > to squid in a non master zonegroup. > > > > Doing the radosgw-admin bucket stats command in any zonegroup, except the > > master, yields results. > > > > We checked, and it is every bucket created after the upgrade, that was not > > created in the master zonegroup. > > > > We only sync metadata between the zonegrouops, and I just ran a fresh > > metadata sync on all zonegroups. > > > > Any ideas what this could be? > > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

ACT NOW! Ceph Days Raleigh CFP Closes Feb. 16th
by Anthony Middleton 03 Feb '26

03 Feb '26

Hey Ceph Friends, Just a quick reminder that the CFP for Ceph Days Raleigh is open! Please submit your proposal by February 16. You can do it here: https://docs.google.com/forms/d/e/1FAIpQLSfHUyLahd9EFAIJCF0niveMhszt_F5lgTU… If you have any questions, feel free to reach out to Laura Flores <lflores(a)ibm.com> or me <anthony(a)amicmid.com>. Thanks! -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Question is then why this is happening only on s3 cluster and not also on NO s3 cluster while write io during recovery/backfilling is actually much higher than on s3? On Tue, Feb 3, 2026 at 3:27 PM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > I concur with Robert's statement, having HDDs only could explain what > you're describing. Not sure where you get your number from (10 DB devices > max per NVMe), but the docs [0] state to not have more than 15 OSDs per > NVMe: > > > DB/WAL offload (optional) > > 1x SSD partition per HDD OSD 4-5x HDD OSDs per DB/WAL SATA SSD <= 15 HDD > OSDs per DB/WAL NVMe SSD > But you're correct about the SPOF, if one NVMe dies, all OSDs that have > their DB/WAL on that NVMe die as well. > > [0] > > https://docs.ceph.com/en/latest/start/hardware-recommendations/#minimum-har… > > Am Di., 3. Feb. 2026 um 15:00 Uhr schrieb Rok Jaklič via ceph-users < > ceph-users(a)ceph.io>: > > > We have 28 OSDs per host and we can only have 2 NVMe per host (one being > > used for OS) ... and if I remember correctly there is max 10 OSDs/NVMe > > recommended, that's why we decided to go just for HDD based clusters at > the > > beinging. > > > > We have 2 clusters this way, one being for HPC (no radosgw/s3) and other > > for "users" (radosgw/s3), running over 4 years now ... works ok, > > performance is ok, just we have this problem where we have to do gentle > > reweight of a failed OSDs. > > > > Thanks for the info, we will consider NVMe ... although then there is > SPOF > > for those OSDs which have DB on NVMe? > > > > Rok > > > > On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > > > Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: > > > > On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- > > > > users(a)ceph.io <mailto:ceph-users@ceph.io>> wrote: > > > > > > > >> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > > > >> > > > >> > 2 OSD(s) experiencing slow operations in > BlueStore > > > >> > 2 OSD(s) experiencing stalled read in db device > of > > > >> BlueFS > > > >> > > > >> Are your OSDs HDD only? > > > > > > > > > > > Yes. > > > > > > > > Does not affect users much. Usually those messages appear when we are > > > > reweighting and changing failed disks. > > > > > > These HDDs will be maxed out with the recovery work and cannot serve > > > anything else any more. > > > > > > I have seen HDD only clusters going into the "spiral of death" because > > > the HDDs cannot answer fast enough. OSDs randomly dropping out making > > > the whole system unstable. > > > > > > The RocksDB is such a random IO application that it is not suitable for > > > HDDs. It should always be put on flash storage (SSD/NVMe). > > > > > > Regards > > > -- > > > Robert Sander > > > Linux Consultant > > > > > > Heinlein Consulting GmbH > > > Schwedter Str. 8/9b, 10119 Berlin > > > > > > https://www.heinlein-support.de > > > > > > Tel: +49 30 405051 - 0 > > > Fax: +49 30 405051 - 19 > > > > > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Eugen Block 03 Feb '26

03 Feb '26

I concur with Robert's statement, having HDDs only could explain what you're describing. Not sure where you get your number from (10 DB devices max per NVMe), but the docs [0] state to not have more than 15 OSDs per NVMe: > DB/WAL offload (optional) 1x SSD partition per HDD OSD 4-5x HDD OSDs per DB/WAL SATA SSD <= 15 HDD OSDs per DB/WAL NVMe SSD But you're correct about the SPOF, if one NVMe dies, all OSDs that have their DB/WAL on that NVMe die as well. [0] https://docs.ceph.com/en/latest/start/hardware-recommendations/#minimum-har… Am Di., 3. Feb. 2026 um 15:00 Uhr schrieb Rok Jaklič via ceph-users < ceph-users(a)ceph.io>: > We have 28 OSDs per host and we can only have 2 NVMe per host (one being > used for OS) ... and if I remember correctly there is max 10 OSDs/NVMe > recommended, that's why we decided to go just for HDD based clusters at the > beinging. > > We have 2 clusters this way, one being for HPC (no radosgw/s3) and other > for "users" (radosgw/s3), running over 4 years now ... works ok, > performance is ok, just we have this problem where we have to do gentle > reweight of a failed OSDs. > > Thanks for the info, we will consider NVMe ... although then there is SPOF > for those OSDs which have DB on NVMe? > > Rok > > On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < > ceph-users(a)ceph.io> wrote: > > > Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: > > > On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- > > > users(a)ceph.io <mailto:ceph-users@ceph.io>> wrote: > > > > > >> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > > >> > > >> > 2 OSD(s) experiencing slow operations in BlueStore > > >> > 2 OSD(s) experiencing stalled read in db device of > > >> BlueFS > > >> > > >> Are your OSDs HDD only? > > > > > > > > Yes. > > > > > > Does not affect users much. Usually those messages appear when we are > > > reweighting and changing failed disks. > > > > These HDDs will be maxed out with the recovery work and cannot serve > > anything else any more. > > > > I have seen HDD only clusters going into the "spiral of death" because > > the HDDs cannot answer fast enough. OSDs randomly dropping out making > > the whole system unstable. > > > > The RocksDB is such a random IO application that it is not suitable for > > HDDs. It should always be put on flash storage (SSD/NVMe). > > > > Regards > > -- > > Robert Sander > > Linux Consultant > > > > Heinlein Consulting GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > https://www.heinlein-support.de > > > > Tel: +49 30 405051 - 0 > > Fax: +49 30 405051 - 19 > > > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

We have 28 OSDs per host and we can only have 2 NVMe per host (one being used for OS) ... and if I remember correctly there is max 10 OSDs/NVMe recommended, that's why we decided to go just for HDD based clusters at the beinging. We have 2 clusters this way, one being for HPC (no radosgw/s3) and other for "users" (radosgw/s3), running over 4 years now ... works ok, performance is ok, just we have this problem where we have to do gentle reweight of a failed OSDs. Thanks for the info, we will consider NVMe ... although then there is SPOF for those OSDs which have DB on NVMe? Rok On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: > > On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- > > users(a)ceph.io <mailto:ceph-users@ceph.io>> wrote: > > > >> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > >> > >> > 2 OSD(s) experiencing slow operations in BlueStore > >> > 2 OSD(s) experiencing stalled read in db device of > >> BlueFS > >> > >> Are your OSDs HDD only? > > > > > Yes. > > > > Does not affect users much. Usually those messages appear when we are > > reweighting and changing failed disks. > > These HDDs will be maxed out with the recovery work and cannot serve > anything else any more. > > I have seen HDD only clusters going into the "spiral of death" because > the HDDs cannot answer fast enough. OSDs randomly dropping out making > the whole system unstable. > > The RocksDB is such a random IO application that it is not suitable for > HDDs. It should always be put on flash storage (SSD/NVMe). > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Yes. Does not affect users much. Usually those messages appear when we are reweighting and changing failed disks. Rok On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > > > 2 OSD(s) experiencing slow operations in BlueStore > > 2 OSD(s) experiencing stalled read in db device of BlueFS > > Are your OSDs HDD only? > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Robert Sander 03 Feb '26

03 Feb '26

Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > 2 OSD(s) experiencing slow operations in BlueStore > 2 OSD(s) experiencing stalled read in db device of BlueFS Are your OSDs HDD only? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Rok Jaklič 03 Feb '26

03 Feb '26

Probably I did not use the right words to explain current situtaion... --- Cluster starts to recover/backfilling once I set crush reweight of a failed osd to 0 (here s3 stops responding after some time). If I set it to a little bit lower than the previous value, then s3 does not stop working after some time. For example, right now its io says it is in a recovery state? (that is why I said it is in recovering state): Every 2.0s: ceph -s ctplmon1.arnes.si: Tue Feb 3 13:55:19 2026 cluster: id: 0a6e5422-ac75-4093-af20-528ee00cc847 health: HEALTH_ERR 2 OSD(s) experiencing slow operations in BlueStore 2 OSD(s) experiencing stalled read in db device of BlueFS 5 scrub errors Possible data damage: 3 pgs inconsistent services: mon: 3 daemons, quorum ctplmon1,ctplmon3,ctplmon2 (age 10d) mgr: ctplmon1(active, since 10d), standbys: ctplmon3 mds: 1/1 daemons up osd: 240 osds: 240 up (since 42h), 240 in (since 42h); 13 remapped pgs rgw: 6 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 6353 pgs objects: 360.61M objects, 421 TiB usage: 708 TiB used, 623 TiB / 1.3 PiB avail pgs: 1544002/1802985000 objects misplaced (0.086%) 6328 active+clean 13 active+remapped+backfilling 9 active+clean+scrubbing+deep 3 active+clean+inconsistent io: client: 97 KiB/s rd, 1.2 KiB/s wr, 18 op/s rd, 3 op/s wr recovery: 110 MiB/s, 90 objects/s --- I do not use orchestrator. Crush failure domain is set to host and EC 3+2 is used (historical reasons), even though we would like to change this to 8+3 in the near future. For every new failed disk different PGs are probably used, not the same? Rok On Tue, Feb 3, 2026 at 1:35 PM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > So the cluster is currently recovering and then you wanted to remove > another OSD? I'm just trying to get the full picture here. Let's assume the > cluster is backfilling because you removed one OSD (and it's not done yet), > then you decide to remove another one. If we assume a replicated pool (size > 3, min_size 2), this could cause inactive PGs until backfill has finished. > Depending on the failure domain (host?) you need to be careful when > removing OSDs from multiple failure domains at the same time. That's why > some background here would help to better understand what really happens. > And does it happen every time you replace a disk? Does it always impact the > same PGs? Does the cluster report inactive PGs for more than a few seconds? > > I was asking for RGW logs during that outage, not the "normal" behaviour > (starting new request), so some error logs. The reshardlogs messages are > also nothing unusual. > > Regarding the replacement procedure, it looks like you don't use the > orchestrator, right? > > > Am Di., 3. Feb. 2026 um 10:30 Uhr schrieb Rok Jaklič via ceph-users < > ceph-users(a)ceph.io>: > > > ceph -s is in recovery state, moving around 200-300MB/s of data. > > > > [root@ctplmon1 ~]# ceph osd ok-to-stop 239 > > > > > {"ok_to_stop":true,"osds":[239],"num_ok_pgs":94,"num_not_ok_pgs":0,"ok_become_degraded":["9.e","9.40","9.4a","9.4e","9.69","9.7a","9.87","9.bc","9.c5"," > > 9.cf > > > > > ","9.dd","9.128","9.172","9.1b6","9.1c7","9.227","9.244","9.246","9.24e","9.2b3","9.2d3","9.2ee","9.2f1","9.31c","9.335","9.361","9.364","9.38b","9.3b0","9.3d9","9.408","9.41b","9.483","9.49a","9.4c1","9.4dc","9.4ea","9.4fd","9.553","9.554","9.55b","9.57a","9.586","9.5c7","9.61a","9.627","9.68b","9.69d","9.711","9.71d","9.737","9.784","10.12","10.82","10.18c","10.1c7","10.290","10.310","10.390","10.434","10.4aa","10.4e1","10.525","10.558","10.57f","10.5e2","10.5e7","10.62f","10.736","10.842","10.8d9","10.904","10.964","10.977","10.9eb","10.a1a","10.a40","10.b80","10.bbf","10.be2","10.c7c","10.c9b","10.cb0","10.cb3","10.d7a","10.dc8","10.dcf","10.dde","10.e5b","10.e6d","10.e78","10.ece","10.ef1","10.f60"]} > > > > rgw logs is usually something like: > > 2026-02-03T09:48:32.050+0100 7f34ff13a640 1 ====== starting new request > > req=0x7f34b8f0c640 ===== > > 2026-02-03T09:48:35.901+0100 7f34cb0d2640 1 ====== starting new request > > req=0x7f34b8e8b640 ===== > > 2026-02-03T09:49:41.408+0100 7f34e00fc640 1 ====== starting new request > > req=0x7f34b8d08640 ===== > > 2026-02-03T09:49:46.267+0100 7f35511de640 1 ====== starting new request > > req=0x7f34b8c87640 ===== > > > > while I can see also: > > 2026-02-03T09:49:50.912+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000000 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.919+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000002 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.926+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000004 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.934+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000006 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.953+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000008 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.960+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000010 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.968+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000012 to be held by another RGW process; > > skipping for now > > 2026-02-03T09:49:50.975+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > > found lock on reshard.0000000014 to be held by another RGW process; > > skipping for now > > > > but do not know if it is related. > > > > ... > > > > Should I use some different strategy to "replace" failed disks? > > > > Right now I am doing something like: > > ceph osd crush reweight osd.{osdid} 4 3 2 1 ... in steps > > > > while ! ceph osd safe-to-destroy osd.{osdid} ; do sleep 10 ; done > > > > systemctl stop ceph-osd@{id} > > ceph osd down osd.{id} > > ceph osd destroy {id} --yes-i-really-mean-it > > > > ceph-volume lvm prepare --osd-id {id} --data /dev/sdX > > ceph-volume lvm list > > ceph-volume lvm activate {id} {osd fsid} > > > > > > > > On Tue, Feb 3, 2026 at 10:21 AM Eugen Block <eblock82(a)gmail.com> wrote: > > > > > Hi, > > > > > > what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs > > > reveal? What's the 'ceph status' during that time? > > > > > > Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < > > > ceph-users(a)ceph.io>: > > > > > >> Hi, > > >> > > >> we currently have 240 OSDs in the cluster, but when I set the reweight > > of > > >> just one failed OSD to 0 (from 5.45798), radosgw stops working for > > >> some reason. > > >> > > >> If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), > > it > > >> works ok. > > >> > > >> Any ideas why? > > >> > > >> Running on reef ceph version 18.2.7. > > >> > > >> Kind regards, > > >> Rok > > >> _______________________________________________ > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: radosgw (s3) stops when setting weight to 0 on one OSD
by Eugen Block 03 Feb '26

03 Feb '26

So the cluster is currently recovering and then you wanted to remove another OSD? I'm just trying to get the full picture here. Let's assume the cluster is backfilling because you removed one OSD (and it's not done yet), then you decide to remove another one. If we assume a replicated pool (size 3, min_size 2), this could cause inactive PGs until backfill has finished. Depending on the failure domain (host?) you need to be careful when removing OSDs from multiple failure domains at the same time. That's why some background here would help to better understand what really happens. And does it happen every time you replace a disk? Does it always impact the same PGs? Does the cluster report inactive PGs for more than a few seconds? I was asking for RGW logs during that outage, not the "normal" behaviour (starting new request), so some error logs. The reshardlogs messages are also nothing unusual. Regarding the replacement procedure, it looks like you don't use the orchestrator, right? Am Di., 3. Feb. 2026 um 10:30 Uhr schrieb Rok Jaklič via ceph-users < ceph-users(a)ceph.io>: > ceph -s is in recovery state, moving around 200-300MB/s of data. > > [root@ctplmon1 ~]# ceph osd ok-to-stop 239 > > {"ok_to_stop":true,"osds":[239],"num_ok_pgs":94,"num_not_ok_pgs":0,"ok_become_degraded":["9.e","9.40","9.4a","9.4e","9.69","9.7a","9.87","9.bc","9.c5"," > 9.cf > > ","9.dd","9.128","9.172","9.1b6","9.1c7","9.227","9.244","9.246","9.24e","9.2b3","9.2d3","9.2ee","9.2f1","9.31c","9.335","9.361","9.364","9.38b","9.3b0","9.3d9","9.408","9.41b","9.483","9.49a","9.4c1","9.4dc","9.4ea","9.4fd","9.553","9.554","9.55b","9.57a","9.586","9.5c7","9.61a","9.627","9.68b","9.69d","9.711","9.71d","9.737","9.784","10.12","10.82","10.18c","10.1c7","10.290","10.310","10.390","10.434","10.4aa","10.4e1","10.525","10.558","10.57f","10.5e2","10.5e7","10.62f","10.736","10.842","10.8d9","10.904","10.964","10.977","10.9eb","10.a1a","10.a40","10.b80","10.bbf","10.be2","10.c7c","10.c9b","10.cb0","10.cb3","10.d7a","10.dc8","10.dcf","10.dde","10.e5b","10.e6d","10.e78","10.ece","10.ef1","10.f60"]} > > rgw logs is usually something like: > 2026-02-03T09:48:32.050+0100 7f34ff13a640 1 ====== starting new request > req=0x7f34b8f0c640 ===== > 2026-02-03T09:48:35.901+0100 7f34cb0d2640 1 ====== starting new request > req=0x7f34b8e8b640 ===== > 2026-02-03T09:49:41.408+0100 7f34e00fc640 1 ====== starting new request > req=0x7f34b8d08640 ===== > 2026-02-03T09:49:46.267+0100 7f35511de640 1 ====== starting new request > req=0x7f34b8c87640 ===== > > while I can see also: > 2026-02-03T09:49:50.912+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000000 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.919+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000002 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.926+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000004 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.934+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000006 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.953+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000008 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.960+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000010 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.968+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000012 to be held by another RGW process; > skipping for now > 2026-02-03T09:49:50.975+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock > found lock on reshard.0000000014 to be held by another RGW process; > skipping for now > > but do not know if it is related. > > ... > > Should I use some different strategy to "replace" failed disks? > > Right now I am doing something like: > ceph osd crush reweight osd.{osdid} 4 3 2 1 ... in steps > > while ! ceph osd safe-to-destroy osd.{osdid} ; do sleep 10 ; done > > systemctl stop ceph-osd@{id} > ceph osd down osd.{id} > ceph osd destroy {id} --yes-i-really-mean-it > > ceph-volume lvm prepare --osd-id {id} --data /dev/sdX > ceph-volume lvm list > ceph-volume lvm activate {id} {osd fsid} > > > > On Tue, Feb 3, 2026 at 10:21 AM Eugen Block <eblock82(a)gmail.com> wrote: > > > Hi, > > > > what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs > > reveal? What's the 'ceph status' during that time? > > > > Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < > > ceph-users(a)ceph.io>: > > > >> Hi, > >> > >> we currently have 240 OSDs in the cluster, but when I set the reweight > of > >> just one failed OSD to 0 (from 5.45798), radosgw stops working for > >> some reason. > >> > >> If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), > it > >> works ok. > >> > >> Any ideas why? > >> > >> Running on reef ceph version 18.2.7. > >> > >> Kind regards, > >> Rok > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

[squid] buckets not in bucket stats in multisite setup when create in a non master zonegroup
by Boris 03 Feb '26

03 Feb '26

Hi, we just encountered a strange problem: When we run radosgw-admin bucket stats on the master zonegroup we get some of these errors error getting bucket stats bucket=rregergf ret=-2 This is the debug_rgw=20 output: error getting bucket stats bucket=rregergf ret=2026-02-03T00:50:32.347+0000 722007981a80 20 cls_bucket_head: CLSRGWIssueGetDirHeader() returned -2 -2 failure: (2) No such file or directory: But the bucketname is listed in a bucket list command. After some digging we found that the bucket was created after the upgrade to squid in a non master zonegroup. Doing the radosgw-admin bucket stats command in any zonegroup, except the master, yields results. We checked, and it is every bucket created after the upgrade, that was not created in the master zonegroup. We only sync metadata between the zonegrouops, and I just ran a fresh metadata sync on all zonegroups. Any ideas what this could be?

1 1