Hi, Yeah, so that is a bit of a clickbait title, but I do think it's true. In the hope this might be useful for someone, or some cluster, I decided to write this down. So what happened? First some context: subject at hand is a Ceph pacific cluster that has been running fine for over nine years (Went through 12 -> 13 -> 14 -> 15 -> 16 with many point releases in between). All OSDs were recreated in 16.2.11 to make use of the RocksDB column families without the need to do resharding (which would have taken forever with the amount of OMAP we have). This cluster has been provisioned with a home build Ansible playbook based on the manual procedure [1], and is basically an automation of all these manual steps. For OSDs we used the "short form" variant [2]. The first issue we encountered was when we added new storage nodes / OSDs to the cluster. The deployment of the first storage node took longer than expected (PCIe NVMe devices that were in use for OS were not correctly filtered and turned up as eligible for OSD creation). After we fixed that we could succesfully proceed. However, the first OSD that was added hit the "out" timer (we did not have noout flag set) and was set "out" by the cluster. We issued the "ceph osd in osd.$id" command to get the OSD back in. The state for all new OSDs was "down" and "in" apart from this OSD, which was in state "down" and "out". This immediately resulted in a assert of the monitors (num_down_in_osds <= num_in_osds). Different monitors crashed in quick succession but the quorum was never lost, and three mons (out of five) were always online. It also did not impact production traffic as now data was stored on them yet. The "ceph osd in osd.$id" that was hanging (due to crashing monitors) was aborted and shortly after that the monitors stopped crashing. The storage node got rebooted and the OSDs were all "up" and "in", apart from one. But this time we could successfully set the osd "in" ("up" and "in" state). We tried to reproduce this in a (virtual) test cluster that is as close a match to production but did not manage to reproduce. We figured this might have been a race condition of some kind (while the command was issued new OSDs might also have been added). The storage nodes were rebalanced and the new nodes were now also taking production workload. Somewhat later we decided to test a change (disable bypass workqueue flags of the encrypted (dmcrypt) devices) and shutdown the OSDs. Immediately after that several mons hit the same assert again. Although the OSDs had gracefully shut itself down ... because of the asserting mons no new osdmaps were created and the peer OSDs were not informed that there peers were down. Thus all involved OSDs had to figure this out themselves which they directly did ... but of course this did result in a lot of slow OSD operations. The OSDs were quickly started again and only until all OSDs were "up" and "in" again the monitors stopped crashing. The monitors did not loose quorum although we were of course afraid that they would. As it turned out this was likely not a race condition we started to look into this further. A search query on Ceph's issue tracker [3] resulted in a lot of hits. Turned out this issues has been hit by clusters provisioned by ROOK, manual procedure (ours) and potentially others and that the issue had been seen on clusters from 16.x, 17.x as well as 19.x and 20.x. This note gives a good summary [4]. It was fixed by this PR [5] and backported to Squid and Tentacle. But the reason *why* we hit this issue was not clear to us. Until we saw this report [6] wherein a user reported that their cluster had a lot of their OSDs with a "NEW" bit and when they would set an OSD "out" and "in" this bit would disappear and so did their issues (not hitting the assert anymore). We dumped the osdmap of the cluster and found out that *all* of our OSDs had the "NEW" bit active apart from one ... the one osd that had been through the OSD "out" / "in" state before. We only expected a "NEW" bit to be enabled for truely new OSDs, not OSDs that have been in the cluster for many years already. We decided not to wait until we would hit a scenario like the following: - one or more OSDs go down (broken or otherwise unable to start again) - have the monitors hit the assert - unable to get out of this situation easily as at that point you cannot issue any ceph commands as the monitors are asserting and the OSDs that went down cannot be brought back up. - hit a lot of slow ops in the meantime until the monitors would set the OSDs down ... or out eventually by itself (down out timer) which could have taken a long time in our case - have the risk of the monitors loosing the quorum and experience unavailability for an extended period of time. We backported the fix to the OSDMap.cc and created our own packages in case we would need them. After that we went ahead and set every OSD "out" and "in" to get rid of the NEW bit and that worked fine. The problem did not trigger anymore, as expected. Today I dived a bit deeper in the OSDMap.cc code responsible for this behavior ... turns out that there is a quicker way to fix this. Relevant code snippet: for (const auto &weight : inc.new_weight) { set_weight(weight.first, weight.second); // if we are marking in, clear the AUTOOUT and NEW bits, and clear // xinfo old_weight. if (weight.second) { osd_state[weight.first] &= ~(CEPH_OSD_AUTOOUT | CEPH_OSD_NEW); osd_xinfo[weight.first].old_weight = 0; } } If you perform a osd reweight, like "ceph osd reweight osd.0 1" it will remove the NEW bit. This is a way quicker process than "out" / "in". We added a reweight step to get rid of the "NEW" bit in our playbook if we would need to provision / replace OSDs again in the future. The reason why we hit this issue now, and not ever before, with the NEW bit OSDs in them and the bug still present is still unclear. There must be some other condition(s) that we hit this time and not before. I've checked our Cephadm based clusters and those don't have any NEW bits present. I'm not sure what steps Cephadm executes under the hood when it creates new OSDs but I guess this involves a step were it performs a reweight. For clusters out there that have been manually provisioned in the past, or created with Ansible playbooks that follow a similar procedure there might be OSDs present with this NEW bit that sooner or later have the risk of hitting this assert (until they run a version in which this is fixed). I advise you to check your clusters and get rid of these NEW bits because AFAIK their only purpose is to have this bit set when they are indeed, new, and they might harm your cluster(s) in the future. Questions we still have: - What other condition(s) are needed to trigger this assert - Would the manual install procedure need to include this reweight step, or is there another way to have the NEW bit cleared? Gr. Stefan [1]: https://docs.ceph.com/en/latest/install/manual-deployment [2]: https://docs.ceph.com/en/latest/install/manual-deployment/#short-form [3]: https://tracker.ceph.com/search?page=3&q=ceph_assert%28num_down_in_osds+%3C%3D+num_in_osds&scope= [4]: https://tracker.ceph.com/issues/70869#note-9 [5]: https://github.com/ceph/ceph/pull/64009 [6]: https://tracker.ceph.com/issues/52535#note-12