Hi, Also make sure every node has an _admin label. (ceph orch host label add <hostname> _admin. Verify with ceph orch host ls. Not proud of it, but I once chown-ed /var/lib/ceph on mulptiple nodes. Due to that, our Ceph cluster lost quorum because all the containers crashed. A learning experience I'll never forget #facepalm. I'll definitely think twice before I manupulate /var/lib/ceph ever again ;) . Only if your cluster is 100% test with no production workload on it: Why not try to shut down one node and see what happens on the surviving nodes. Do you get back to HEALTH_OK after you restart the downed node? It should and if it doesn't you're lucky to find out before you went in production. Also, do a "rolling" reboot of all of your nodes. Do you get back to HEALTH_OK after a reboot has finished? This for all the nodes? Anything else breaks if you shut down one node? Wannes. ________________________________ From: Eugen Block via ceph-users <ceph-users@ceph.io> Sent: Tuesday, July 21, 2026 22:31 To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: How to administer Ceph from any node without bootstrap dependency Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of the admin keyring via orchestrator and the _admin label [0]. That enables you to manage the cluster from any admin node. Not sure what you mean by "bootstrap file" though... If you'd synced the /var/lib/ceph directory, you'd overwrite all the daemons but one, including keyring etc. So please don't do that, that would break your cluster. Regards, Eugen [0] https://urldefense.com/v3/__https://docs.ceph.com/en/latest/cephadm/operations/*enabling-management-of-a-keyring-file__;Iw!!FtrhtPsWDhZ6tw!D5rYS6vuFBBX8RvD33YU7lOpN1vdG8KmtV0TK5INmTDXXlFnjmf5_usJ0XWfxusrzz59lNFi85ZaMG2J$[docs[.]ceph[.]com] Zitat von Michel Le Cocq via ceph-users <ceph-users@ceph.io>:
Hi all,
I am currently deploying a Ceph cluster with cephadm on 3 nodes with MON, MGR, and MDS roles. OSDs will be added later.
My goal is for these 3 nodes to be fully interchangeable, with the ability to administer the cluster from any of them without risk if a node fails.
I have identified that the solution involves synchronizing /var/lib/ceph/ (excluding the bootstrap file) across all nodes.
However:
- Is this the right approach to achieve this goal?
- Which tool should I use for reliable bidirectional synchronization?
- What are the potential risks (conflicts, corruption) and precautions to take?
Have you implemented such a configuration in production before?
Thank you for your advice.
-- Michel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io