Hi, I think this is why the mon-osd interaction requires a certain number of osd to report another osd as down/unavailable: https://docs.ceph.com/en/latest/rados/configuration/mon-osd-interaction/#osd... The default value for mon_osd_reporter_subtree_level is host, and the default value for mon_osd_min_down_reporters is 2 (values taken from one of our cluster, no override in ceph config). So it requires reports from two osds in different hosts to consider another osd as down. This should not be case in the reported situation unless setting osd_fast_fail_on_connection_refused<https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-osd_fast_fail_on_connection_refused>=true changes this behaviour. Best regards, Burkhard Linke On 24.11.23 11:09, Janne Johansson wrote:
Hi Denis,
I would agree with you that a single misconfigured host should not take out healthy hosts under any circumstances. I'm not sure if your incident is actually covered by the devs comments, it is quite possible that you observed an unintended side effect that is a bug in handling the connection error. I think the intention is to shut down fast the OSDs with connection refused (where timeouts are not required) and not other OSDs. No, this has been true for a long while and has happened to me multiple times. Any kind of fault where an OSD can talk to at least one mon, and then for any of multiple reasons do not respond or connect to other OSDs lead to this. The OSD comes up, can hold 0 data and have a lifetime of 5 seconds, and then it claims it can't talk to some 5-10-15 other OSDs, and rats on them to the mon which then
Den fre 24 nov. 2023 kl 10:25 skrev Frank Schilder<frans@dtu.dk>: proceeds to believe this new OSD and flaps those other OSDs, which then get to reconnect and the mon logs that "osdmap says I am down but I am not". This goes on until you kill the bad OSD or fix the network. If you boot up a host with many OSDs, they all pick 5-10-15 OSDs to shoot down, so you quickly get annoying errors at a large scale if you ever misconfigure a new host in the least possible way.
There should be some better kind of validation that the new (and in my case at least, often empty) OSD is not at fault and that the other hundreds of working OSDs are in fact not gone at all before causing this kind of confusion.