Hi Wannes, The 33% threshold is not applied against the raw total number of OSDs in the cluster. It is applied against the OSD’s heartbeat peer set. With the cluster network VLAN missing, public-network connectivity to MONs could still work, while OSD heartbeats, replication, recovery, and peering over the cluster network failed. -- Alex Gorbachev ISS/Storcium On Wed, Jun 24, 2026 at 1:25 AM Wannes Smet <smet@icsense.com> wrote:
Hi Alex,
Lessons learned are always valuable information to share! Thanks.
Pretty sure I'm missing out on a subtlety here. I can't really put my finger on it why only osd_heartbeat_min_healthy_ratio caused this, so some questions from my side.
At first glance I'd think 50% is still larger than 33%? I also assume OSDs can be peers even if they never were able to heartbeat each other before due to crush.
Maybe this happened because that peering sets aren't perfectly balanced? So you might have gotten slightly below that 50%, but still above 33%. Meaning you're <17% away from 33%, "critical mass". But then some additional peering/activity or the like happened due to the "shock change", and the whole effectively dropped below 33% cluster wide due to some cascading effects?
Thanks,
Wannes ------------------------------ *From:* Alex Gorbachev via ceph-users <ceph-users@ceph.io> *Sent:* Tuesday, June 23, 2026 18:10 *To:* ceph-users <ceph-users@ceph.io> *Subject:* [ceph-users] PSA: be aware of osd_heartbeat_min_healthy_ratio
osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers an OSD must be able to reach before it can accurately mark itself up or avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD heartbeat decisions are made ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers an OSD must be able to reach before it can accurately mark itself up or avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD heartbeat decisions are made independently of monitor quorum health. An OSD can begin reporting peers as down based on heartbeat failures even when monitor quorum remains healthy. I was adding a second rack of OSD servers to an existing installation, assuming that the worst that could happen is that those will be unreachable. I also missed setting up the cluster_network VLAN on one side (in the 4x100GbE bond).
I added all the new OSDs at once. The moment that happened, the existing OSDs saw 50% unreachable, and all production OSDs promptly shut down.
The take away in my recommendation is to add OSDs in such a way that they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to fix connectivity issues before dealing with down OSDs, as I experienced. -- Alex Gorbachev ISS/Storcium _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io