Can you share these values? osd_max_pg_per_osd_hard_ratio mon_max_pg_per_osd Zitat von Elias Carter <elias@dropbox.com>:
Oh, in your first message you wrote you have 300 PGs per OSD, which I didn’t find that high. Now you wrote 600 PGs per OSD, which is it?
My mistake, our target is 300 PGs per OSD and the maximum is 600 PGs per OSD. Currently, our clusters are hovering around the target of 300 PGs per OSD.
Can you share some details about the OSDs (size)?
We are running 14TiB NVMe SSD drives (PCIe 4). Our largest cluster right now has approximately 400 OSDs.
Regarding tuning osd_heartbeat_interval, I did gradually lower it from 6 to 1 and found that packets per second increased by about 50% on the OSDs. Although we are only seeing tens of thousands of packets per second per OSD, these nodes have NICs rated for 100 million packets per second, so we are not even close to hitting limits there. So far I don't see any cluster instability caused by setting osd_heartbeat_interval to 1 in our clusters. My assumption is that the number of heartbeat packets will increase quadratically with the number of OSDs in the cluster (given 300 PGs/OSD there is a good likelihood that any one OSD peers with almost every other OSD?).
Elias
On Tue, Aug 18, 2026 at 1:15 PM Eugen Block <eblock@nde.ag> wrote:
Oh, in your first message you wrote you have 300 PGs per OSD, which I didn’t find that high. Now you wrote 600 PGs per OSD, which is it? 😁 600 sounds like a lot indeed, that could be one of the downsides of having too many PGs per OSD. Some time ago there was a thread here on the list about the ancient recommendation of 100 per OSD and nobody seemed to know the exact reason. Several people (including me) mentioned good experiences with around 300 to 400 PGs per OSD. Maybe you found the max value 😅
Can you share some details about the OSDs (size)?
Zitat von Elias Carter <elias@dropbox.com>:
the IO should not be blocked for the entire pool but only for those repeering PGs
Yes, the issue is that when running 600 PGs per OSD a high fraction of PGs re-peer simultaneously.
Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement.
Thank you for the confirmation. We run Ceph within kubernetes (rook) and my plan is to write an operator which will handle gradually reweighting OSDs as they are added/removed from the cluster. Rook allows setting the initial OSD weight, so my plan is to set the initial OSD weights to 0 and then have the operator gradually reweight the OSDs to 1.
One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds).
Thank you; I will try that out. We have a mix of 100G and 400G networks, so I think the increased ping frequency will be ok. I will try it out on our staging clusters first and report back if there are any interesting findings.
Elias
On Tue, Aug 18, 2026 at 2:21 AM Eugen Block <ceph-users@ceph.io> wrote:
Hi,
the IO should not be blocked for the entire pool but only for those repeering PGs. Warnings during that phase are expected. Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement.
One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds). That could (or probably would) increase network traffic significantly, so it's a trade-off. If you have a test cluster with similar behavior, test it there first.
Regards, Eugen
Zitat von Elias Carter via ceph-users <ceph-users@ceph.io>:
We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and upwards).
Something that I have noticed is that CRUSH changes (marking an OSD as out, or a new OSD joining the cluster) can cause many hundreds of PGs to remap and peer. During this time we get PG_AVAILABILITY health check warnings as the OSDs need to peer with their new PGs before they can serve IOs. RADOS tail latency suffers because IOs sent to a peering PG are blocked until peering completes.
I believe that the more PGs we have, the more PGs need to peer, and the longer IOs are blocked for the pool.
Does anyone have suggestions on how to deal with PG peering blocking IOs?
I was thinking about gradually re-weighting OSDs instead of marking them out or in, but that can result in multiple redundant data moves.
Elias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io