Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs?
It seems that it's not enabled by default. This plausibly could have warned after hosts were added, before OSD deployment, assuming the hosts had already been added to cephadm and the check was enabled. And one more question: could 'osd_crush_initial_weight = 0' have
prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting.
I doubt this would have helped because the CRUSH weight just controls data placement, but osd heartbeat comes at a lower level than that, and that would have fired anyway. -- Alex Gorbachev ISS/Storcium On Wed, Jun 24, 2026 at 5:50 AM Eugen Block via ceph-users < ceph-users@ceph.io> wrote:
Hi,
since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs?
It seems that it's not enabled by default.
And one more question: could 'osd_crush_initial_weight = 0' have prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting.
Thanks, Eugen
Zitat von Alex Gorbachev via ceph-users <ceph-users@ceph.io>:
osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers an OSD must be able to reach before it can accurately mark itself up or avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD heartbeat decisions are made independently of monitor quorum health. An OSD can begin reporting peers as down based on heartbeat failures even when monitor quorum remains healthy. I was adding a second rack of OSD servers to an existing installation, assuming that the worst that could happen is that those will be unreachable. I also missed setting up the cluster_network VLAN on one side (in the 4x100GbE bond).
I added all the new OSDs at once. The moment that happened, the existing OSDs saw 50% unreachable, and all production OSDs promptly shut down.
The take away in my recommendation is to add OSDs in such a way that they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to fix connectivity issues before dealing with down OSDs, as I experienced. -- Alex Gorbachev ISS/Storcium _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Zitat von Alex Gorbachev <ag@iss-integration.com>:
since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs?
It seems that it's not enabled by default.
no, this is not enabled by default. Although the ls output might suggest otherwise (see my thread from yesterday: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/H4X57MJJPFMQ...)
This plausibly could have warned after hosts were added, before OSD deployment, assuming the hosts had already been added to cephadm and the check was enabled.
And one more question: could 'osd_crush_initial_weight = 0' have
prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting.
I doubt this would have helped because the CRUSH weight just controls data placement, but osd heartbeat comes at a lower level than that, and that would have fired anyway.
Yeah, that makes sense I think.
-- Alex Gorbachev ISS/Storcium
On Wed, Jun 24, 2026 at 5:50 AM Eugen Block via ceph-users < ceph-users@ceph.io> wrote:
Hi,
since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs?
It seems that it's not enabled by default.
And one more question: could 'osd_crush_initial_weight = 0' have prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting.
Thanks, Eugen
Zitat von Alex Gorbachev via ceph-users <ceph-users@ceph.io>:
osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers an OSD must be able to reach before it can accurately mark itself up or avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD heartbeat decisions are made independently of monitor quorum health. An OSD can begin reporting peers as down based on heartbeat failures even when monitor quorum remains healthy. I was adding a second rack of OSD servers to an existing installation, assuming that the worst that could happen is that those will be unreachable. I also missed setting up the cluster_network VLAN on one side (in the 4x100GbE bond).
I added all the new OSDs at once. The moment that happened, the existing OSDs saw 50% unreachable, and all production OSDs promptly shut down.
The take away in my recommendation is to add OSDs in such a way that they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to fix connectivity issues before dealing with down OSDs, as I experienced. -- Alex Gorbachev ISS/Storcium _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Alex Gorbachev
-
Eugen Block