Hi, since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs? It seems that it's not enabled by default. And one more question: could 'osd_crush_initial_weight = 0' have prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting. Thanks, Eugen Zitat von Alex Gorbachev via ceph-users <ceph-users@ceph.io>:

osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers an OSD must be able to reach before it can accurately mark itself up or avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD heartbeat decisions are made independently of monitor quorum health. An OSD can begin reporting peers as down based on heartbeat failures even when monitor quorum remains healthy. I was adding a second rack of OSD servers to an existing installation, assuming that the worst that could happen is that those will be unreachable. I also missed setting up the cluster_network VLAN on one side (in the 4x100GbE bond).

I added all the new OSDs at once. The moment that happened, the existing OSDs saw 50% unreachable, and all production OSDs promptly shut down.

The take away in my recommendation is to add OSDs in such a way that they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to fix connectivity issues before dealing with down OSDs, as I experienced. -- Alex Gorbachev ISS/Storcium _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io