understanding orchestration and cephadm
Good morning all, I'm experimenting with ceph orchestration and cephadm after using ceph-deploy for several years, and I have a hopefully simple question. I've converted a basic nautilus cluster over to cephadm+orchestration and I tried adding, then removing a monitor. However, when I removed the host using 'ceph orch host rm', it removed two mons. I may have missed something in the adoption/upgrade that has left the cluster in a bad state. Any advice/pointers/clarification would be of assistance. Details: A nautilus cluster with two mons (I know this is not correct for quorum), a mgr, and a handful of osds. I went though the adoption process and enabled the ceph orch backend. [root@osdev-ctrl2 ~]# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID mgr.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f e73c19b51a09 mon.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f a6bfc27221f0 mon.osdev-net1 osdev-net1 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f f66e2bef3d44 osd.0 osdev-stor1 running (17h) 50s ago 17h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f ac59dbdc267c ... [root@osdev-ctrl2 ~]# ceph orch status Backend: cephadm Available: True [root@osdev-ctrl2 ~]# ceph orch host ls HOST ADDR LABELS STATUS osdev-ctrl2 osdev-ctrl2 mon mgr osdev-net1 osdev-net1 mon osdev-stor1 osdev-stor1 osd [root@osdev-ctrl2 ~]# ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID mgr 1/1 9m ago 20h label:mgr docker.io/ceph/ceph:v15.2.10 5b724076c58f mon 2/2 9m ago 20h label:mon docker.io/ceph/ceph:v15.2.10 5b724076c58f I then added a new mon host: [root@osdev-ctrl2 ~]# ceph orch host add osdev-ctrl3 mon It did not spawn a mon container on osdev-ctrl3 until I defined the public network in the config: [root@osdev-ctrl2 ~]# ceph config set global public_network 10.10.10.0/24 At this point all is good with three running mon as expected. Now I wanted to delete the mon using [root@osdev-ctrl2 ~]# ceph orch host rm osdev-ctrl3 This had the effect of: 1. removing the osdev-ctrl3 mon from 'ceph orch ls' and 'ceph orch ps' 2. the mon on osdev-ctrl3 is still running, and is part of 'ceph -s' but reported as not managed by cephadm 3. (Big issue) the mon running on osdev-net1 was completely destroyed. Any ideas what is going on? Sorry for the long post, but I tried to be as clear as possible. -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam@uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566
Yoo have conflated "ceph orch host add" and "ceph orch host label add" This is not valid syntax: ceph orch host add osdev-ctrl3 mon The docs imply that the trailing "mon" would be ignored, since there is no 6th argument to that command. https://docs.ceph.com/en/latest/cephadm/host-management/ you have to run a separate ceph orch host label add osdev-ctrl3 mon ----- Original Message ----- From: "Gary Molenkamp" <molenkam@uwo.ca> To: "ceph-users" <ceph-users@ceph.io> Sent: Wednesday, March 31, 2021 8:12:53 AM Subject: [ceph-users] understanding orchestration and cephadm Good morning all, I'm experimenting with ceph orchestration and cephadm after using ceph-deploy for several years, and I have a hopefully simple question. I've converted a basic nautilus cluster over to cephadm+orchestration and I tried adding, then removing a monitor. However, when I removed the host using 'ceph orch host rm', it removed two mons. I may have missed something in the adoption/upgrade that has left the cluster in a bad state. Any advice/pointers/clarification would be of assistance. Details: A nautilus cluster with two mons (I know this is not correct for quorum), a mgr, and a handful of osds. I went though the adoption process and enabled the ceph orch backend. [root@osdev-ctrl2 ~]# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID mgr.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f e73c19b51a09 mon.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f a6bfc27221f0 mon.osdev-net1 osdev-net1 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f f66e2bef3d44 osd.0 osdev-stor1 running (17h) 50s ago 17h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f ac59dbdc267c ... [root@osdev-ctrl2 ~]# ceph orch status Backend: cephadm Available: True [root@osdev-ctrl2 ~]# ceph orch host ls HOST ADDR LABELS STATUS osdev-ctrl2 osdev-ctrl2 mon mgr osdev-net1 osdev-net1 mon osdev-stor1 osdev-stor1 osd [root@osdev-ctrl2 ~]# ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID mgr 1/1 9m ago 20h label:mgr docker.io/ceph/ceph:v15.2.10 5b724076c58f mon 2/2 9m ago 20h label:mon docker.io/ceph/ceph:v15.2.10 5b724076c58f I then added a new mon host: [root@osdev-ctrl2 ~]# ceph orch host add osdev-ctrl3 mon It did not spawn a mon container on osdev-ctrl3 until I defined the public network in the config: [root@osdev-ctrl2 ~]# ceph config set global public_network 10.10.10.0/24 At this point all is good with three running mon as expected. Now I wanted to delete the mon using [root@osdev-ctrl2 ~]# ceph orch host rm osdev-ctrl3 This had the effect of: 1. removing the osdev-ctrl3 mon from 'ceph orch ls' and 'ceph orch ps' 2. the mon on osdev-ctrl3 is still running, and is part of 'ceph -s' but reported as not managed by cephadm 3. (Big issue) the mon running on osdev-net1 was completely destroyed. Any ideas what is going on? Sorry for the long post, but I tried to be as clear as possible. -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam@uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Gary, It looks like everything you did is fine. I think the "problem" is that cephadm has/had some logic that tried to leave users with an odd number of monitors. I'm pretty sure this is why two of them were removed. This code has been removed in pacific, and should probably be backported to octopus. There is nothing wrong with an even number of mons. The only number you might want to avoid is 2 because a failure of either monitor will cause the cluster to lose quorum and become unavailable (quorum requires > N/2, which in a 2-mon case means both mons). As far as availability goes that is probably not ideal, but as far as durability goes, it's extremely useful to have a duplicate copy of the mon data so that losing a single disk doesn't destroy the cluster metadata (and require a complicated recovery process). In any case, generally speaking, nobody should worry about having an even number of monitors. Focus instead of getting >2 so you can tolerate at least one mon failure and keep the cluster running. On Wed, Mar 31, 2021 at 10:14 AM Gary Molenkamp <molenkam@uwo.ca> wrote:
A nautilus cluster with two mons (I know this is not correct for quorum), a mgr, and a handful of osds. I went though the adoption
Any number of monitors is correct. Less than 3 is not recommended. sage
Thanks Sage, I opted to move to an explicit placement map of candidate hostnames and a replica count rather than using labels. This is a testing cluster of VMs to experiment before updating the production system. The only reason I was starting with 2 on the test cluster is that my production ceph has a few mons that are custom built docker containers that will give some grief in the adoption process. My plan is to remove those mons before converting to cephadm. On further thought, it will probably be safer to add a few normal mons to the OSD servers before adopting to cephadm, to ensure I have more than 3 before the conversion. I like the work done with cephadm and orchestration as compared to the older ceph-deploy. I do have a few questions/challenges on OSD deployment that I will put in a new post. Cheers. Gary On 2021-03-31 5:00 p.m., Sage Weil wrote:
Hi Gary,
It looks like everything you did is fine. I think the "problem" is that cephadm has/had some logic that tried to leave users with an odd number of monitors. I'm pretty sure this is why two of them were removed.
This code has been removed in pacific, and should probably be backported to octopus.
There is nothing wrong with an even number of mons. The only number you might want to avoid is 2 because a failure of either monitor will cause the cluster to lose quorum and become unavailable (quorum requires > N/2, which in a 2-mon case means both mons). As far as availability goes that is probably not ideal, but as far as durability goes, it's extremely useful to have a duplicate copy of the mon data so that losing a single disk doesn't destroy the cluster metadata (and require a complicated recovery process).
In any case, generally speaking, nobody should worry about having an even number of monitors. Focus instead of getting >2 so you can tolerate at least one mon failure and keep the cluster running.
On Wed, Mar 31, 2021 at 10:14 AM Gary Molenkamp <molenkam@uwo.ca> wrote:
A nautilus cluster with two mons (I know this is not correct for quorum), a mgr, and a handful of osds. I went though the adoption Any number of monitors is correct. Less than 3 is not recommended.
sage
-- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam@uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566
participants (3)
-
Gary Molenkamp
-
Philip Brown
-
Sage Weil