Oh, in your first message you wrote you have 300 PGs per OSD, which I didn’t find that high. Now you wrote 600 PGs per OSD, which is it? 😁 600 sounds like a lot indeed, that could be one of the downsides of having too many PGs per OSD. Some time ago there was a thread here on the list about the ancient recommendation of 100 per OSD and nobody seemed to know the exact reason. Several people (including me) mentioned good experiences with around 300 to 400 PGs per OSD. Maybe you found the max value 😅 Can you share some details about the OSDs (size)? Zitat von Elias Carter <elias@dropbox.com>:
the IO should not be blocked for the entire pool but only for those repeering PGs
Yes, the issue is that when running 600 PGs per OSD a high fraction of PGs re-peer simultaneously.
Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement.
Thank you for the confirmation. We run Ceph within kubernetes (rook) and my plan is to write an operator which will handle gradually reweighting OSDs as they are added/removed from the cluster. Rook allows setting the initial OSD weight, so my plan is to set the initial OSD weights to 0 and then have the operator gradually reweight the OSDs to 1.
One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds).
Thank you; I will try that out. We have a mix of 100G and 400G networks, so I think the increased ping frequency will be ok. I will try it out on our staging clusters first and report back if there are any interesting findings.
Elias
On Tue, Aug 18, 2026 at 2:21 AM Eugen Block <ceph-users@ceph.io> wrote:
Hi,
the IO should not be blocked for the entire pool but only for those repeering PGs. Warnings during that phase are expected. Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement.
One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds). That could (or probably would) increase network traffic significantly, so it's a trade-off. If you have a test cluster with similar behavior, test it there first.
Regards, Eugen
Zitat von Elias Carter via ceph-users <ceph-users@ceph.io>:
We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and upwards).
Something that I have noticed is that CRUSH changes (marking an OSD as out, or a new OSD joining the cluster) can cause many hundreds of PGs to remap and peer. During this time we get PG_AVAILABILITY health check warnings as the OSDs need to peer with their new PGs before they can serve IOs. RADOS tail latency suffers because IOs sent to a peering PG are blocked until peering completes.
I believe that the more PGs we have, the more PGs need to peer, and the longer IOs are blocked for the pool.
Does anyone have suggestions on how to deal with PG peering blocking IOs?
I was thinking about gradually re-weighting OSDs instead of marking them out or in, but that can result in multiple redundant data moves.
Elias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io