Alright, quorum is good. Do you have mgr keyrings in 'ceph auth ls' output? If they are present, is there a keyring file in /var/lib/ceph/{fsid}/mgr.{mgr}/ and does it match the auth output? Zitat von RIT Computer Science House <csh@rit.edu>:
One mon survived (it took us a while to find it since it was in a damaged state), and we have since been able to create a new second mon where an old mon was - quorum has been re-established. We are not able to use `ceph orch` now to deploy new mons though, it is giving us an error from the keyring.
The ceph commands were not working at the time of only one damaged mon, but now most ceph commands function normally with 2 mons.
On Wed, Aug 21, 2024 at 6:27 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
Is there any command-line history available to get at least some sort of history of events? Are all MONs down or has one survived? Could he have tried to change IP addresses or something? There's an old blog post [0] explaining how to clean up. And here's some more reading [1] how to modify a monmap in a cephadm managed cluster. I assume none of the ceph commands work, can you confirm?
[0] https://ceph.io/en/news/blog/2015/ceph-monitor-troubleshooting/ [1]
https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#rados-mon-r...
Zitat von RIT Computer Science House <csh@rit.edu>:
Hello,
Our cluster has become unresponsive after a teammate's work on the cluster. We are currently unable to get the full story on what he did to fully understand what is going on, and the only error we are able to see in any of the logs is the following: 2024-08-20T03:12:34.183+0000 7f3670246b80 -1 unable to read magic from mon data
Any help would be greatly appreciated! I can provide any information necessary to help debugging.
Thanks, Tyler System Administrator @ Computer Science House _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io