Probably I did not use the right words to explain current situtaion... --- Cluster starts to recover/backfilling once I set crush reweight of a failed osd to 0 (here s3 stops responding after some time). If I set it to a little bit lower than the previous value, then s3 does not stop working after some time. For example, right now its io says it is in a recovery state? (that is why I said it is in recovering state): Every 2.0s: ceph -s ctplmon1.arnes.si: Tue Feb 3 13:55:19 2026 cluster: id: 0a6e5422-ac75-4093-af20-528ee00cc847 health: HEALTH_ERR 2 OSD(s) experiencing slow operations in BlueStore 2 OSD(s) experiencing stalled read in db device of BlueFS 5 scrub errors Possible data damage: 3 pgs inconsistent services: mon: 3 daemons, quorum ctplmon1,ctplmon3,ctplmon2 (age 10d) mgr: ctplmon1(active, since 10d), standbys: ctplmon3 mds: 1/1 daemons up osd: 240 osds: 240 up (since 42h), 240 in (since 42h); 13 remapped pgs rgw: 6 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 6353 pgs objects: 360.61M objects, 421 TiB usage: 708 TiB used, 623 TiB / 1.3 PiB avail pgs: 1544002/1802985000 objects misplaced (0.086%) 6328 active+clean 13 active+remapped+backfilling 9 active+clean+scrubbing+deep 3 active+clean+inconsistent io: client: 97 KiB/s rd, 1.2 KiB/s wr, 18 op/s rd, 3 op/s wr recovery: 110 MiB/s, 90 objects/s --- I do not use orchestrator. Crush failure domain is set to host and EC 3+2 is used (historical reasons), even though we would like to change this to 8+3 in the near future. For every new failed disk different PGs are probably used, not the same? Rok On Tue, Feb 3, 2026 at 1:35 PM Eugen Block via ceph-users < ceph-users@ceph.io> wrote:

So the cluster is currently recovering and then you wanted to remove another OSD? I'm just trying to get the full picture here. Let's assume the cluster is backfilling because you removed one OSD (and it's not done yet), then you decide to remove another one. If we assume a replicated pool (size 3, min_size 2), this could cause inactive PGs until backfill has finished. Depending on the failure domain (host?) you need to be careful when removing OSDs from multiple failure domains at the same time. That's why some background here would help to better understand what really happens. And does it happen every time you replace a disk? Does it always impact the same PGs? Does the cluster report inactive PGs for more than a few seconds?

I was asking for RGW logs during that outage, not the "normal" behaviour (starting new request), so some error logs. The reshardlogs messages are also nothing unusual.

Regarding the replacement procedure, it looks like you don't use the orchestrator, right?

Am Di., 3. Feb. 2026 um 10:30 Uhr schrieb Rok Jaklič via ceph-users < ceph-users@ceph.io>:

...

ceph -s is in recovery state, moving around 200-300MB/s of data.

[root@ctplmon1 ~]# ceph osd ok-to-stop 239

{"ok_to_stop":true,"osds":[239],"num_ok_pgs":94,"num_not_ok_pgs":0,"ok_become_degraded":["9.e","9.40","9.4a","9.4e","9.69","9.7a","9.87","9.bc","9.c5","

...

9.cf

","9.dd","9.128","9.172","9.1b6","9.1c7","9.227","9.244","9.246","9.24e","9.2b3","9.2d3","9.2ee","9.2f1","9.31c","9.335","9.361","9.364","9.38b","9.3b0","9.3d9","9.408","9.41b","9.483","9.49a","9.4c1","9.4dc","9.4ea","9.4fd","9.553","9.554","9.55b","9.57a","9.586","9.5c7","9.61a","9.627","9.68b","9.69d","9.711","9.71d","9.737","9.784","10.12","10.82","10.18c","10.1c7","10.290","10.310","10.390","10.434","10.4aa","10.4e1","10.525","10.558","10.57f","10.5e2","10.5e7","10.62f","10.736","10.842","10.8d9","10.904","10.964","10.977","10.9eb","10.a1a","10.a40","10.b80","10.bbf","10.be2","10.c7c","10.c9b","10.cb0","10.cb3","10.d7a","10.dc8","10.dcf","10.dde","10.e5b","10.e6d","10.e78","10.ece","10.ef1","10.f60"]}

...

rgw logs is usually something like: 2026-02-03T09:48:32.050+0100 7f34ff13a640 1 ====== starting new request req=0x7f34b8f0c640 ===== 2026-02-03T09:48:35.901+0100 7f34cb0d2640 1 ====== starting new request req=0x7f34b8e8b640 ===== 2026-02-03T09:49:41.408+0100 7f34e00fc640 1 ====== starting new request req=0x7f34b8d08640 ===== 2026-02-03T09:49:46.267+0100 7f35511de640 1 ====== starting new request req=0x7f34b8c87640 =====

while I can see also: 2026-02-03T09:49:50.912+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock found lock on reshard.0000000000 to be held by another RGW process; skipping for now 2026-02-03T09:49:50.919+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock found lock on reshard.0000000002 to be held by another RGW process; skipping for now 2026-02-03T09:49:50.926+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock found lock on reshard.0000000004 to be held by another RGW process; skipping for now 2026-02-03T09:49:50.934+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock found lock on reshard.0000000006 to be held by another RGW process; skipping for now 2026-02-03T09:49:50.953+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock found lock on reshard.0000000008 to be held by another RGW process; skipping for now 2026-02-03T09:49:50.960+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock found lock on reshard.0000000010 to be held by another RGW process; skipping for now 2026-02-03T09:49:50.968+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock found lock on reshard.0000000012 to be held by another RGW process; skipping for now 2026-02-03T09:49:50.975+0100 7f35bf2db640 0 INFO: RGWReshardLock::lock found lock on reshard.0000000014 to be held by another RGW process; skipping for now

but do not know if it is related.

...

Should I use some different strategy to "replace" failed disks?

Right now I am doing something like: ceph osd crush reweight osd.{osdid} 4 3 2 1 ... in steps

while ! ceph osd safe-to-destroy osd.{osdid} ; do sleep 10 ; done

systemctl stop ceph-osd@{id} ceph osd down osd.{id} ceph osd destroy {id} --yes-i-really-mean-it

ceph-volume lvm prepare --osd-id {id} --data /dev/sdX ceph-volume lvm list ceph-volume lvm activate {id} {osd fsid}

On Tue, Feb 3, 2026 at 10:21 AM Eugen Block <eblock82@gmail.com> wrote:

...

Hi,

what does 'ceph osd ok-to-stop <OSD_ID>' show? What do the rgw logs reveal? What's the 'ceph status' during that time?

Am Di., 3. Feb. 2026 um 10:18 Uhr schrieb Rok Jaklič via ceph-users < ceph-users@ceph.io>:

...

Hi,

we currently have 240 OSDs in the cluster, but when I set the reweight of just one failed OSD to 0 (from 5.45798), radosgw stops working for some reason.

If I do "gentle reweight" in steps of 1 for example (from 5.4 to 4.4), it works ok.

Any ideas why?

Running on reef ceph version 18.2.7.

Kind regards, Rok _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io