On Thu, Mar 5, 2020 at 8:05 PM Sage Weil <sage@newdream.net> wrote:
On Thu, 5 Mar 2020, Dan van der Ster wrote:
On Thu, Mar 5, 2020 at 4:42 PM Sage Weil <sage@newdream.net> wrote:
On Thu, 5 Mar 2020, Dan van der Ster wrote:
Hi Sage,
On Thu, Mar 5, 2020 at 3:22 PM Sage Weil <sage@newdream.net> wrote:
On Thu, 5 Mar 2020, Dan van der Ster wrote:
Hi all,
There's something broken in our env when we try to add new mons to existing clusters, confirmed on two clusters running mimic and nautilus. It's basically this issue https://tracker.ceph.com/issues/42830
In case something is wrong with our puppet manifests, I'm trying to doing it manually.
First we --mkfs the mon and start it, but as soon as the new mon starts synchronizing, the existing leader becomes unresponsive and an election is triggered.
Here's exactly what I'm doing:
# cd /var/lib/ceph/tmp/ # scp cephmon1:/var/lib/ceph/tmp/keyring.mon.cephmon1 keyring.mon.cephmon4 # ceph mon getmap -o monmap # ceph-mon --mkfs -i cephmon4 --monmap monmap --keyrin keyring.mon.cephmon4 --setuser ceph --setgroup ceph # vi /etc/ceph/ceph.conf <add the new mon to ceph.conf like this> [mon.cephmon4] host = cephmon4 mon addr = a.b.c.d:6790 # systemctl start ceph-mon@cephmon4
The log file on the new mon shows it start synchronizing, then immediately the CPU usage on the leader goes to 100% and elections start happening, and ceph health shows mon slow ops. perf top of the ceph-mon with 100% CPU is shown below [1]. On a small nautilus cluster, the new mon gets added withing a minute or so (but not cleanly -- the leader is unresponsive for quite awhile until the new mon joins). debug_mon=20 on the leader doesn't show anything very interesting. On our large mimic cluster we tried waiting more than 10 minutes -- suffering through several mon elections and 100% usage bouncing around between leaders -- until we gave up.
I'm pulling my hair out a bit on this -- it's really weird!
Can you try running a rocksdb compaction on the existing mons before adding the new one and see if that helps?
It doesn't help. I compacted the 3 mons in quorum then started a new one with debug mon & paxos = 20.
ceph-post-file: 9867d4ef-38cc-4ae7-9631-c6b86e666683
I stopped that new mon as soon as the sync source started spinning 100% and left the quorum.
Can you include the log from teh sync source too? That's presumably where the bug is.
Here's a different new mon and the leader, with debug_paxos & mon = 20:
ceph-post-file: 8db3d788-e266-4034-9d0c-4ee55eb1d055
Things start to go wrong at this line:
2020-03-05 19:37:35.697 7f5fe87e2700 10 mon.p05517715y58557@0(leader) e32 handle_sync mon_sync(get_chunk cookie 170322296835) v2
...which is the just before it tries to sync osd_snap.
I also included the output of ceph-monstore-tool dump-keys. There are really a lot of osd_snap keys!
Aha, I knew this sounded familiar! See https://github.com/ceph/ceph/pull/31581
We should backport this for the next nautilus...
Perfect.. thanks!! -- dan
sage
Thanks!
-- dan
Thanks! sage
-- Dan
s
Did anyone add a new mon to an existing large cluster recently, and it went smoothly?
Cheers, Dan
[1]
15.12% ceph-mon [.] MonitorDBStore::Transaction::encode 8.95% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::append 8.68% libceph-common.so.0 [.] ceph::buffer::v14_2_0::list::append 7.69% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::release 5.86% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::ptr _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io