Re: Upgrade problem from Octopus to Quincy

15 Apr 2026


      Hi,

Zitat von Iztok Gregori via ceph-users <ceph-users@ceph.io>:
...
Hi,
comments inline.
On 15/04/26 09:31, Eugen Block via ceph-users wrote:
[cut]
...
...
...
- Can you share the output of 'ceph mon dump'?
Sure!
# ceph mon dump
epoch 18
fsid 5bed1bf6-123f-4597-aaaa-d77931228548
last_changed 2026-04-08T20:23:59.781295+0200
created 2018-01-09T10:20:36.351967+0100
min_mon_release 15 (octopus)
election_strategy: 1
0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3
1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4
2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5
dumped monmap epoch 18
So your cluster currently has quorum with 2 out of 3 MONs on Quincy  
already, correct? Restarting the mon on node3 shouldn't be  
dangerous, I think. You'd still have quorum with node4 and node5,  
even if node3 wouldn't be able to join. If it doesn't join as  
Octopus, you could still upgrade it and retry with Quincy.
Yes. I was just reluctant to do so because in the case it doesn't  
join our room to manoeuvre will be reduced. Plus the fact that node3  
is the only daemon which is still logging something...
oh I didn't realize there was only one left logging something. I  
thought only the newly to be joined MON didn't log. In that case I  
would also try to find the root cause for that.
...
...
What I'm wondering about, why did you remove the MONs you wanted to  
upgrade from Ceph? Is that recommended somewhere? Maybe you need to  
run 'ceph mon add <name> <addr>' as well to let node1 or node2 join  
the monmap again? I'm not sure, it has been a long time doing this  
kind of stuff. ;-) But I don't think I ever removed a MON before  
upgrading it, not even when reinstalling the OS.
No, we didn't follow any recommendations, it seemed reasonable to  
remove a monitor before reinstalling the OS, a fresh start looked  
better than a backup/restore of '/var/lib/ceph/'. Maybe we were  
wrong. For the next upgrades (node3-5) we could think not to do that  
and see what it happens.
I don't exactly remember, tbh, but I expect that it would suffice to  
have the /var/lib/ceph/mon/ceph-nodeX directory with the correct  
permissions present in order for it to re-sync with the other MONs.  
But please double-check that, I'm really not sure. If I have the time,  
I will try to test that in a lab environment.
...
...
Just for completeness sake, could you also share 'ceph -s' and  
'ceph versions'?
Sure!
# ceph -s
  cluster:
    id:     5bed1bf6-123f-4597-aaa-d77931228548
    health: HEALTH_OK
services:
    mon: 3 daemons, quorum node3,node4,node5 (age 6d)
    mgr: node1(active, since 6d), standbys: node2
    osd: 852 osds: 849 up (since 2w), 849 in (since 8w)
data:
    pools:   11 pools, 13825 pgs
    objects: 447.25M objects, 1.7 PiB
    usage:   5.0 PiB used, 3.7 PiB / 8.7 PiB avail
    pgs:     13817 active+clean
             8     active+clean+scrubbing+deep
io:
    client:   3.0 MiB/s rd, 53 MiB/s wr, 35 op/s rd, 1.05k op/s wr
    cache:    0 op/s promote
# ceph versions
{
    "mon": {
        "ceph version 15.2.16  
(d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 1,
        "ceph version 17.2.6  
(d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2
    },
    "mgr": {
        "ceph version 17.2.9  
(69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2
    },
    "osd": {
        "ceph version 15.2.16  
(d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 179,
        "ceph version 15.2.17  
(8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670
    },
    "mds": {},
    "overall": {
        "ceph version 15.2.16  
(d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 180,
        "ceph version 15.2.17  
(8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670,
        "ceph version 17.2.6  
(d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2,
        "ceph version 17.2.9  
(69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2
    }
}
...
...
...
- There were a couple of reports back at O/P versions about MONs  
not joining the quorum. Sometimes it worked after hours,  
sometimes it didn't, and I'm still not aware of a real solution.  
One of our customers has been facing severe MON sync delay,  
sometimes taking around 40 minutes until a MON joined quorum. It  
turned out to be caused by some paxos default settings, requring  
the MONs to do a full sync instead of a diff sync. This has to do  
with millions of snapshots they have in their cluster. What made  
it worse was the MON's filesystems were on HDDs, increasing the  
sync time as well.
We didn't wait for a long time before stopping the new MON a  
regain "control" of the cluster, maybe 10 minutes max...
...
- Inspecting debug logs on the syncing MON could help finding the  
root cause.
With "syncing MON" you mean the MON which is just added to the  
cluster, right? Collecting this kind of logs is on the to-do list,  
but blocked because the "log problem" mentioned bellow.
Yes, that's what I meant. The missing logs are a problem, of  
course. It does sound familiar, but I don't have a good idea right  
now except for looking at directory and file permissions,  
apparmor/selinux things, dmesg, stuff like that.
Yep. My biggest fear right now is that lack of logging from Quincy  
monitor daemons could mean that they don't work properly and if we  
restart the Octopus one (or it doesn't want to rejoin the quorum) we  
will have a not functioning cluster. We need to investigate further  
into it.
I totally understand your hesitation, before I'd do that I would also  
try to rule out anything I can. But at the same time it would be  
surprising, given your cluster shows a healthy quorum right now. But  
yeah, investigate further before moving forward is the safer choice  
here.
...
Iztok
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io

Eugen Block

tags

participants (1)