Hi, comments inline. On 15/04/26 09:31, Eugen Block via ceph-users wrote: [cut]
- Can you share the output of 'ceph mon dump'?
Sure!
# ceph mon dump epoch 18 fsid 5bed1bf6-123f-4597-aaaa-d77931228548 last_changed 2026-04-08T20:23:59.781295+0200 created 2018-01-09T10:20:36.351967+0100 min_mon_release 15 (octopus) election_strategy: 1 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 dumped monmap epoch 18
So your cluster currently has quorum with 2 out of 3 MONs on Quincy already, correct? Restarting the mon on node3 shouldn't be dangerous, I think. You'd still have quorum with node4 and node5, even if node3 wouldn't be able to join. If it doesn't join as Octopus, you could still upgrade it and retry with Quincy.
Yes. I was just reluctant to do so because in the case it doesn't join our room to manoeuvre will be reduced. Plus the fact that node3 is the only daemon which is still logging something...
What I'm wondering about, why did you remove the MONs you wanted to upgrade from Ceph? Is that recommended somewhere? Maybe you need to run 'ceph mon add <name> <addr>' as well to let node1 or node2 join the monmap again? I'm not sure, it has been a long time doing this kind of stuff. ;-) But I don't think I ever removed a MON before upgrading it, not even when reinstalling the OS.
No, we didn't follow any recommendations, it seemed reasonable to remove a monitor before reinstalling the OS, a fresh start looked better than a backup/restore of '/var/lib/ceph/'. Maybe we were wrong. For the next upgrades (node3-5) we could think not to do that and see what it happens.
Just for completeness sake, could you also share 'ceph -s' and 'ceph versions'?
Sure! # ceph -s cluster: id: 5bed1bf6-123f-4597-aaa-d77931228548 health: HEALTH_OK services: mon: 3 daemons, quorum node3,node4,node5 (age 6d) mgr: node1(active, since 6d), standbys: node2 osd: 852 osds: 849 up (since 2w), 849 in (since 8w) data: pools: 11 pools, 13825 pgs objects: 447.25M objects, 1.7 PiB usage: 5.0 PiB used, 3.7 PiB / 8.7 PiB avail pgs: 13817 active+clean 8 active+clean+scrubbing+deep io: client: 3.0 MiB/s rd, 53 MiB/s wr, 35 op/s rd, 1.05k op/s wr cache: 0 op/s promote # ceph versions { "mon": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 1, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2 }, "mgr": { "ceph version 17.2.9 (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 }, "osd": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 179, "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670 }, "mds": {}, "overall": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 180, "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2, "ceph version 17.2.9 (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 } }
- There were a couple of reports back at O/P versions about MONs not joining the quorum. Sometimes it worked after hours, sometimes it didn't, and I'm still not aware of a real solution. One of our customers has been facing severe MON sync delay, sometimes taking around 40 minutes until a MON joined quorum. It turned out to be caused by some paxos default settings, requring the MONs to do a full sync instead of a diff sync. This has to do with millions of snapshots they have in their cluster. What made it worse was the MON's filesystems were on HDDs, increasing the sync time as well.
We didn't wait for a long time before stopping the new MON a regain "control" of the cluster, maybe 10 minutes max...
- Inspecting debug logs on the syncing MON could help finding the root cause.
With "syncing MON" you mean the MON which is just added to the cluster, right? Collecting this kind of logs is on the to-do list, but blocked because the "log problem" mentioned bellow.
Yes, that's what I meant. The missing logs are a problem, of course. It does sound familiar, but I don't have a good idea right now except for looking at directory and file permissions, apparmor/selinux things, dmesg, stuff like that.
Yep. My biggest fear right now is that lack of logging from Quincy monitor daemons could mean that they don't work properly and if we restart the Octopus one (or it doesn't want to rejoin the quorum) we will have a not functioning cluster. We need to investigate further into it. Iztok