the IO should not be blocked for the entire pool but only for those repeering PGs
Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on
Yes, the issue is that when running 600 PGs per OSD a high fraction of PGs re-peer simultaneously. this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement. Thank you for the confirmation. We run Ceph within kubernetes (rook) and my plan is to write an operator which will handle gradually reweighting OSDs as they are added/removed from the cluster. Rook allows setting the initial OSD weight, so my plan is to set the initial OSD weights to 0 and then have the operator gradually reweight the OSDs to 1.
One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds).
Thank you; I will try that out. We have a mix of 100G and 400G networks, so I think the increased ping frequency will be ok. I will try it out on our staging clusters first and report back if there are any interesting findings. Elias On Tue, Aug 18, 2026 at 2:21 AM Eugen Block <ceph-users@ceph.io> wrote:
Hi,
the IO should not be blocked for the entire pool but only for those repeering PGs. Warnings during that phase are expected. Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement.
One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds). That could (or probably would) increase network traffic significantly, so it's a trade-off. If you have a test cluster with similar behavior, test it there first.
Regards, Eugen
Zitat von Elias Carter via ceph-users <ceph-users@ceph.io>:
We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and upwards).
Something that I have noticed is that CRUSH changes (marking an OSD as out, or a new OSD joining the cluster) can cause many hundreds of PGs to remap and peer. During this time we get PG_AVAILABILITY health check warnings as the OSDs need to peer with their new PGs before they can serve IOs. RADOS tail latency suffers because IOs sent to a peering PG are blocked until peering completes.
I believe that the more PGs we have, the more PGs need to peer, and the longer IOs are blocked for the pool.
Does anyone have suggestions on how to deal with PG peering blocking IOs?
I was thinking about gradually re-weighting OSDs instead of marking them out or in, but that can result in multiple redundant data moves.
Elias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io