Hi Raul, we had quite a similar issue last year. We removed the two failing MONs from the monmap, injected the reduced monmap into the surviving MON so it would have a quorum. After that the other daemons would start but we had to deal with large mon stores (I believe around 250 GB or so) during this phase. IIRC we had to prevent it from compacting too often during startup (mon_compact_on_start = false) and also added SSD storage for the MON store so the sync speed would increase. Eventually, we brought the cluster up into a healthy state and then added back the two crashed MONs. The root cause was that /var/ ran out of disk space. So in our case it definitely was not a bug. ;-) Hope this helps! Eugen Zitat von Raul H C Lopes <raul.cardoso.lopes@cern.ch>:
Dear CEPH dev team,
I have a CEPH cluster with three MONs two of each are down. When I try to start them they crash and
journalctl shows that they crashed and a core dump was created. Would that be a bug? Or a corrupt DB?
I have then a third MON that starts fine but when I get mon_status through the admin-socket I see
"quorum": []
"state": "probing"
because of that (I believe) I cannot use 'ceph orch' to create new MONs.
So my questions:
- Is there way that I can use 'ceph orch' to create new MONS?
- Can I just rsync the store.db from this running node to the crashing MON nodes?
- Or do I have to rebuild the store.db using a script as in
https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#m...
Regards,
Raul _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io