Hi Eugen! Thanks for the support, my response is inline On 14/04/26 13:03, Eugen Block via ceph-users wrote:
Hi,
I don't have a solution (yet?), but a couple of points to clarify.
- I assume your cluster isn't managed by cephadm, correct?
Correct! It's an old cluster initialy deployed with ceph-deploy in the Hammer days and upgraded since then. Right now is manually managed (but one of the targets at the end of the upgrade is to switch to cephadm).
- Can you share the output of 'ceph mon dump'?
Sure! # ceph mon dump epoch 18 fsid 5bed1bf6-123f-4597-aaaa-d77931228548 last_changed 2026-04-08T20:23:59.781295+0200 created 2018-01-09T10:20:36.351967+0100 min_mon_release 15 (octopus) election_strategy: 1 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 dumped monmap epoch 18
- There were a couple of reports back at O/P versions about MONs not joining the quorum. Sometimes it worked after hours, sometimes it didn't, and I'm still not aware of a real solution. One of our customers has been facing severe MON sync delay, sometimes taking around 40 minutes until a MON joined quorum. It turned out to be caused by some paxos default settings, requring the MONs to do a full sync instead of a diff sync. This has to do with millions of snapshots they have in their cluster. What made it worse was the MON's filesystems were on HDDs, increasing the sync time as well.
We didn't wait for a long time before stopping the new MON a regain "control" of the cluster, maybe 10 minutes max...
- Inspecting debug logs on the syncing MON could help finding the root cause.
With "syncing MON" you mean the MON which is just added to the cluster, right? Collecting this kind of logs is on the to-do list, but blocked because the "log problem" mentioned bellow.
Some more comments inline.
[cut]
We taught, among other things, to:
- Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is the only daemon which wasn't restarted and when queried ("ceph daemon mon.node1 mon_status") has in "extra_probe_peers" the IP addresses of the original removed nodes, which are the same IPs as the nodes we want to add. We aren't sure if this prevent the ceph-mon to join, because we don't understand when/what "extra_probe_peers" is doing.
According to the output above node1 is already out of the cluster, are you referring to node3?
- Remove the node1 from the cluster, set min_mon_release to Quincy and then try to add the new monitors.
Yes, sorry for the misunderstanding, I was referring to node3 in both instances.
We are unsure about both approaches because if they don't work we will have a "unstable" cluster with only 2 monitors and no much space to operate.
To "complicate" the matter we notice that on node4 and node5 the ceph- mon is not logging anymore to "/var/log/ceph/ceph*.log" even tough a "mon_cluster_log_to_file: true". If I search for open FD there is nothing in /var/log/ceph for the running ceph-mon...
Is there something in journalctl?
We increased the 'debug_mon' to 20 ('ceph daemon mon.node4 config set debug_mon 20') and we check it with 'ceph daemon mon.node4 config show' but the output of 'journalctl -u ceph-mon@mon.node4` is not more verbose then before, the last line is from 2:00 this morning and it's related to sudo: apr 14 02:35:40 node4 sudo[881890]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/md127 Iztok