31 Mar
2021
31 Mar
'21
7:42 p.m.
Good morning all,
I'm experimenting with ceph orchestration and cephadm after using
ceph-deploy for several years, and I have a hopefully simple question.
I've converted a basic nautilus cluster over to cephadm+orchestration
and I tried adding, then removing a monitor. However, when I removed the
host using 'ceph orch host rm', it removed two mons. I may have missed
something in the adoption/upgrade that has left the cluster in a bad
state. Any advice/pointers/clarification would be of assistance.
Details:
A nautilus cluster with two mons (I know this is not correct for
quorum), a mgr, and a handful of osds. I went though the adoption
process and enabled the ceph orch backend.
[root@osdev-ctrl2 ~]# ceph orch ps
NAME HOST STATUS REFRESHED
AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID
mgr.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10
docker.io/ceph/ceph:v15.2.10 5b724076c58f e73c19b51a09
mon.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10
docker.io/ceph/ceph:v15.2.10 5b724076c58f a6bfc27221f0
mon.osdev-net1 osdev-net1 running (18h) 50s ago 18h 15.2.10
docker.io/ceph/ceph:v15.2.10 5b724076c58f f66e2bef3d44
osd.0 osdev-stor1 running (17h) 50s ago 17h
15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f ac59dbdc267c
...
[root@osdev-ctrl2 ~]# ceph orch status
Backend: cephadm
Available: True
[root@osdev-ctrl2 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
osdev-ctrl2 osdev-ctrl2 mon mgr
osdev-net1 osdev-net1 mon
osdev-stor1 osdev-stor1 osd
[root@osdev-ctrl2 ~]# ceph orch ls
NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME
IMAGE ID
mgr 1/1 9m ago 20h label:mgr docker.io/ceph/ceph:v15.2.10
5b724076c58f
mon 2/2 9m ago 20h label:mon docker.io/ceph/ceph:v15.2.10
5b724076c58f
I then added a new mon host:
[root@osdev-ctrl2 ~]# ceph orch host add osdev-ctrl3 mon
It did not spawn a mon container on osdev-ctrl3 until I defined the
public network in the config:
[root@osdev-ctrl2 ~]# ceph config set global public_network 10.10.10.0/24
At this point all is good with three running mon as expected. Now I
wanted to delete the mon using
[root@osdev-ctrl2 ~]# ceph orch host rm osdev-ctrl3
This had the effect of:
1. removing the osdev-ctrl3 mon from 'ceph orch ls' and 'ceph orch
ps'
2. the mon on osdev-ctrl3 is still running, and is part of 'ceph
-s' but reported as not managed by cephadm
3. (Big issue) the mon running on osdev-net1 was completely destroyed.
Any ideas what is going on? Sorry for the long post, but I tried to be
as clear as possible.
--
Gary Molenkamp Computer Science/Science Technology Services
Systems Administrator University of Western Ontario
molenkam(a)uwo.ca http://www.csd.uwo.ca
(519) 661-2111 x86882 (519) 661-3566
1 Apr
1 Apr
12:43 a.m.
Yoo have conflated "ceph orch host add" and
"ceph orch host label add"
This is not valid syntax:
ceph orch host add osdev-ctrl3 mon
The docs imply that the trailing "mon" would be ignored, since there is no 6th
argument to that command.
https://docs.ceph.com/en/latest/cephadm/host-management/
you have to run a separate
ceph orch host label add osdev-ctrl3 mon
----- Original Message -----
From: "Gary Molenkamp" <molenkam(a)uwo.ca>
To: "ceph-users" <ceph-users(a)ceph.io>
Sent: Wednesday, March 31, 2021 8:12:53 AM
Subject: [ceph-users] understanding orchestration and cephadm
Good morning all,
I'm experimenting with ceph orchestration and cephadm after using
ceph-deploy for several years, and I have a hopefully simple question.
I've converted a basic nautilus cluster over to cephadm+orchestration
and I tried adding, then removing a monitor. However, when I removed the
host using 'ceph orch host rm', it removed two mons. I may have missed
something in the adoption/upgrade that has left the cluster in a bad
state. Any advice/pointers/clarification would be of assistance.
Details:
A nautilus cluster with two mons (I know this is not correct for
quorum), a mgr, and a handful of osds. I went though the adoption
process and enabled the ceph orch backend.
[root@osdev-ctrl2 ~]# ceph orch ps
NAME HOST STATUS REFRESHED
AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID
mgr.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10
docker.io/ceph/ceph:v15.2.10 5b724076c58f e73c19b51a09
mon.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10
docker.io/ceph/ceph:v15.2.10 5b724076c58f a6bfc27221f0
mon.osdev-net1 osdev-net1 running (18h) 50s ago 18h 15.2.10
docker.io/ceph/ceph:v15.2.10 5b724076c58f f66e2bef3d44
osd.0 osdev-stor1 running (17h) 50s ago 17h
15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f ac59dbdc267c
...
[root@osdev-ctrl2 ~]# ceph orch status
Backend: cephadm
Available: True
[root@osdev-ctrl2 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
osdev-ctrl2 osdev-ctrl2 mon mgr
osdev-net1 osdev-net1 mon
osdev-stor1 osdev-stor1 osd
[root@osdev-ctrl2 ~]# ceph orch ls
NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME
IMAGE ID
mgr 1/1 9m ago 20h label:mgr docker.io/ceph/ceph:v15.2.10
5b724076c58f
mon 2/2 9m ago 20h label:mon docker.io/ceph/ceph:v15.2.10
5b724076c58f
I then added a new mon host:
[root@osdev-ctrl2 ~]# ceph orch host add osdev-ctrl3 mon
It did not spawn a mon container on osdev-ctrl3 until I defined the
public network in the config:
[root@osdev-ctrl2 ~]# ceph config set global public_network 10.10.10.0/24
At this point all is good with three running mon as expected. Now I
wanted to delete the mon using
[root@osdev-ctrl2 ~]# ceph orch host rm osdev-ctrl3
This had the effect of:
1. removing the osdev-ctrl3 mon from 'ceph orch ls' and 'ceph orch
ps'
2. the mon on osdev-ctrl3 is still running, and is part of 'ceph
-s' but reported as not managed by cephadm
3. (Big issue) the mon running on osdev-net1 was completely destroyed.
Any ideas what is going on? Sorry for the long post, but I tried to be
as clear as possible.
--
Gary Molenkamp Computer Science/Science Technology Services
Systems Administrator University of Western Ontario
molenkam(a)uwo.ca http://www.csd.uwo.ca
(519) 661-2111 x86882 (519) 661-3566
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:30 a.m.
Hi Gary,
It looks like everything you did is fine. I think the "problem" is
that cephadm has/had some logic that tried to leave users with an odd
number of monitors. I'm pretty sure this is why two of them were
removed.
This code has been removed in pacific, and should probably be
backported to octopus.
There is nothing wrong with an even number of mons. The only number
you might want to avoid is 2 because a failure of either monitor will
cause the cluster to lose quorum and become unavailable (quorum
requires > N/2, which in a 2-mon case means both mons). As far as
availability goes that is probably not ideal, but as far as durability
goes, it's extremely useful to have a duplicate copy of the mon data
so that losing a single disk doesn't destroy the cluster metadata (and
require a complicated recovery process).
In any case, generally speaking, nobody should worry about having an
even number of monitors. Focus instead of getting >2 so you can
tolerate at least one mon failure and keep the cluster running.
On Wed, Mar 31, 2021 at 10:14 AM Gary Molenkamp <molenkam(a)uwo.ca> wrote:
A nautilus cluster with two mons (I know this is not
correct for
quorum), a mgr, and a handful of osds. I went though the adoption
Any number of monitors is correct. Less than 3 is not recommended.
sage
5 Apr
5 Apr
10:38 p.m.
Thanks Sage,
I opted to move to an explicit placement map of candidate hostnames and
a replica count rather than using labels. This is a testing cluster of
VMs to experiment before updating the production system.
The only reason I was starting with 2 on the test cluster is that my
production ceph has a few mons that are custom built docker containers
that will give some grief in the adoption process. My plan is to remove
those mons before converting to cephadm. On further thought, it will
probably be safer to add a few normal mons to the OSD servers before
adopting to cephadm, to ensure I have more than 3 before the conversion.
I like the work done with cephadm and orchestration as compared to the
older ceph-deploy. I do have a few questions/challenges on OSD
deployment that I will put in a new post.
Cheers.
Gary
On 2021-03-31 5:00 p.m., Sage Weil wrote:
Hi Gary,
It looks like everything you did is fine. I think the "problem" is
that cephadm has/had some logic that tried to leave users with an odd
number of monitors. I'm pretty sure this is why two of them were
removed.
This code has been removed in pacific, and should probably be
backported to octopus.
There is nothing wrong with an even number of mons. The only number
you might want to avoid is 2 because a failure of either monitor will
cause the cluster to lose quorum and become unavailable (quorum
requires > N/2, which in a 2-mon case means both mons). As far as
availability goes that is probably not ideal, but as far as durability
goes, it's extremely useful to have a duplicate copy of the mon data
so that losing a single disk doesn't destroy the cluster metadata (and
require a complicated recovery process).
In any case, generally speaking, nobody should worry about having an
even number of monitors. Focus instead of getting >2 so you can
tolerate at least one mon failure and keep the cluster running.
On Wed, Mar 31, 2021 at 10:14 AM Gary Molenkamp <molenkam(a)uwo.ca> wrote:
--
Gary Molenkamp Computer Science/Science Technology Services
Systems Administrator University of Western Ontario
molenkam(a)uwo.ca http://www.csd.uwo.ca
(519) 661-2111 x86882 (519) 661-3566
A nautilus cluster with two mons (I know this is
not correct for
quorum), a mgr, and a handful of osds. I went though the adoption
Any number of
monitors is correct. Less than 3 is not recommended.
sage
1138
days inactive
1143
days old
3 comments
3 participants
participants (3)
-
Gary Molenkamp -
Philip Brown -
Sage Weil