On Thu, 5 Mar 2020, Dan van der Ster wrote: > Hi all, > > There's something broken in our env when we try to add new mons to > existing clusters, confirmed on two clusters running mimic and > nautilus. It's basically this issue > https://tracker.ceph.com/issues/42830 > > In case something is wrong with our puppet manifests, I'm trying to > doing it manually. > > First we --mkfs the mon and start it, but as soon as the new mon > starts synchronizing, the existing leader becomes unresponsive and an > election is triggered. > > Here's exactly what I'm doing: > > # cd /var/lib/ceph/tmp/ > # scp cephmon1:/var/lib/ceph/tmp/keyring.mon.cephmon1 keyring.mon.cephmon4 > # ceph mon getmap -o monmap > # ceph-mon --mkfs -i cephmon4 --monmap monmap --keyrin > keyring.mon.cephmon4 --setuser ceph --setgroup ceph > # vi /etc/ceph/ceph.conf <add the new mon to ceph.conf like this> > [mon.cephmon4] > host = cephmon4 > mon addr = a.b.c.d:6790 > # systemctl start ceph-mon@cephmon4 > > The log file on the new mon shows it start synchronizing, then > immediately the CPU usage on the leader goes to 100% and elections > start happening, and ceph health shows mon slow ops. perf top of the > ceph-mon with 100% CPU is shown below [1]. > On a small nautilus cluster, the new mon gets added withing a minute > or so (but not cleanly -- the leader is unresponsive for quite awhile > until the new mon joins). debug_mon=20 on the leader doesn't show > anything very interesting. > On our large mimic cluster we tried waiting more than 10 minutes -- > suffering through several mon elections and 100% usage bouncing around > between leaders -- until we gave up. > > I'm pulling my hair out a bit on this -- it's really weird! Can you try running a rocksdb compaction on the existing mons before adding the new one and see if that helps?

On Thu, 5 Mar 2020, Dan van der Ster wrote: > Hi Sage, > > On Thu, Mar 5, 2020 at 3:22 PM Sage Weil &lt;sage(a)newdream.net&gt; wrote: > > > > On Thu, 5 Mar 2020, Dan van der Ster wrote: > > > Hi all, > > > > > > There's something broken in our env when we try to add new mons to > > > existing clusters, confirmed on two clusters running mimic and > > > nautilus. It's basically this issue > > > https://tracker.ceph.com/issues/42830 > > > > > > In case something is wrong with our puppet manifests, I'm trying to > > > doing it manually. > > > > > > First we --mkfs the mon and start it, but as soon as the new mon > > > starts synchronizing, the existing leader becomes unresponsive and an > > > election is triggered. > > > > > > Here's exactly what I'm doing: > > > > > > # cd /var/lib/ceph/tmp/ > > > # scp cephmon1:/var/lib/ceph/tmp/keyring.mon.cephmon1 keyring.mon.cephmon4 > > > # ceph mon getmap -o monmap > > > # ceph-mon --mkfs -i cephmon4 --monmap monmap --keyrin > > > keyring.mon.cephmon4 --setuser ceph --setgroup ceph > > > # vi /etc/ceph/ceph.conf <add the new mon to ceph.conf like this> > > > [mon.cephmon4] > > > host = cephmon4 > > > mon addr = a.b.c.d:6790 > > > # systemctl start ceph-mon@cephmon4 > > > > > > The log file on the new mon shows it start synchronizing, then > > > immediately the CPU usage on the leader goes to 100% and elections > > > start happening, and ceph health shows mon slow ops. perf top of the > > > ceph-mon with 100% CPU is shown below [1]. > > > On a small nautilus cluster, the new mon gets added withing a minute > > > or so (but not cleanly -- the leader is unresponsive for quite awhile > > > until the new mon joins). debug_mon=20 on the leader doesn't show > > > anything very interesting. > > > On our large mimic cluster we tried waiting more than 10 minutes -- > > > suffering through several mon elections and 100% usage bouncing around > > > between leaders -- until we gave up. > > > > > > I'm pulling my hair out a bit on this -- it's really weird! > > > > Can you try running a rocksdb compaction on the existing mons before > > adding the new one and see if that helps? > > It doesn't help. I compacted the 3 mons in quorum then started a new > one with debug mon & paxos = 20. > > ceph-post-file: 9867d4ef-38cc-4ae7-9631-c6b86e666683 > > I stopped that new mon as soon as the sync source started spinning > 100% and left the quorum. Can you include the log from teh sync source too? That's presumably where the bug is.

On Thu, 5 Mar 2020, Dan van der Ster wrote: > On Thu, Mar 5, 2020 at 4:42 PM Sage Weil &lt;sage(a)newdream.net&gt; wrote: > > > > On Thu, 5 Mar 2020, Dan van der Ster wrote: > > > Hi Sage, > > > > > > On Thu, Mar 5, 2020 at 3:22 PM Sage Weil &lt;sage(a)newdream.net&gt; wrote: > > > > > > > > On Thu, 5 Mar 2020, Dan van der Ster wrote: > > > > > Hi all, > > > > > > > > > > There's something broken in our env when we try to add new mons to > > > > > existing clusters, confirmed on two clusters running mimic and > > > > > nautilus. It's basically this issue > > > > > https://tracker.ceph.com/issues/42830 > > > > > > > > > > In case something is wrong with our puppet manifests, I'm trying to > > > > > doing it manually. > > > > > > > > > > First we --mkfs the mon and start it, but as soon as the new mon > > > > > starts synchronizing, the existing leader becomes unresponsive and an > > > > > election is triggered. > > > > > > > > > > Here's exactly what I'm doing: > > > > > > > > > > # cd /var/lib/ceph/tmp/ > > > > > # scp cephmon1:/var/lib/ceph/tmp/keyring.mon.cephmon1 keyring.mon.cephmon4 > > > > > # ceph mon getmap -o monmap > > > > > # ceph-mon --mkfs -i cephmon4 --monmap monmap --keyrin > > > > > keyring.mon.cephmon4 --setuser ceph --setgroup ceph > > > > > # vi /etc/ceph/ceph.conf <add the new mon to ceph.conf like this> > > > > > [mon.cephmon4] > > > > > host = cephmon4 > > > > > mon addr = a.b.c.d:6790 > > > > > # systemctl start ceph-mon@cephmon4 > > > > > > > > > > The log file on the new mon shows it start synchronizing, then > > > > > immediately the CPU usage on the leader goes to 100% and elections > > > > > start happening, and ceph health shows mon slow ops. perf top of the > > > > > ceph-mon with 100% CPU is shown below [1]. > > > > > On a small nautilus cluster, the new mon gets added withing a minute > > > > > or so (but not cleanly -- the leader is unresponsive for quite awhile > > > > > until the new mon joins). debug_mon=20 on the leader doesn't show > > > > > anything very interesting. > > > > > On our large mimic cluster we tried waiting more than 10 minutes -- > > > > > suffering through several mon elections and 100% usage bouncing around > > > > > between leaders -- until we gave up. > > > > > > > > > > I'm pulling my hair out a bit on this -- it's really weird! > > > > > > > > Can you try running a rocksdb compaction on the existing mons before > > > > adding the new one and see if that helps? > > > > > > It doesn't help. I compacted the 3 mons in quorum then started a new > > > one with debug mon & paxos = 20. > > > > > > ceph-post-file: 9867d4ef-38cc-4ae7-9631-c6b86e666683 > > > > > > I stopped that new mon as soon as the sync source started spinning > > > 100% and left the quorum. > > > > Can you include the log from teh sync source too? That's presumably where > > the bug is. > > Here's a different new mon and the leader, with debug_paxos & mon = 20: > > ceph-post-file: 8db3d788-e266-4034-9d0c-4ee55eb1d055 > > Things start to go wrong at this line: > > 2020-03-05 19:37:35.697 7f5fe87e2700 10 mon.p05517715y58557@0(leader) > e32 handle_sync mon_sync(get_chunk cookie 170322296835) v2 > > ...which is the just before it tries to sync osd_snap. > > I also included the output of ceph-monstore-tool dump-keys. There are > really a lot of osd_snap keys! Aha, I knew this sounded familiar! See https://github.com/ceph/ceph/pull/31581 We should backport this for the next nautilus...