- ceph-users - lists.ceph.io

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 15 Apr '26

15 Apr '26

On 15/04/26 12:08, Eugen Block via ceph-users wrote: > Yeah, looks like it. I just compared the output with a fresh Tentacle > install, I only enabled log-to-file during bootstrap, so log_to_file is > true in my cluster as well. There's no leveldb_log_to_ceph_log option > anymore since the default has been rocksdb for a while now. I put 'log_to_journald' to true and journalctl started to log. A lot (with debug=20) :-) Now I'm ready to debug the joining problem. I will keep you posted. Thanks Iztok

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 15 Apr '26

15 Apr '26

Yeah, looks like it. I just compared the output with a fresh Tentacle install, I only enabled log-to-file during bootstrap, so log_to_file is true in my cluster as well. There's no leveldb_log_to_ceph_log option anymore since the default has been rocksdb for a while now. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi! > > On 15/04/26 11:11, Eugen Block via ceph-users wrote: >> Hi, >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> > > [cut] > >>>> So your cluster currently has quorum with 2 out of 3 MONs on >>>> Quincy already, correct? Restarting the mon on node3 shouldn't be >>>> dangerous, I think. You'd still have quorum with node4 and node5, >>>> even if node3 wouldn't be able to join. If it doesn't join as >>>> Octopus, you could still upgrade it and retry with Quincy. >>> >>> Yes. I was just reluctant to do so because in the case it doesn't >>> join our room to manoeuvre will be reduced. Plus the fact that >>> node3 is the only daemon which is still logging something... >> >> oh I didn't realize there was only one left logging something. I >> thought only the newly to be joined MON didn't log. In that case I >> would also try to find the root cause for that. > > It could easily be, because this is an old cluster which was > upgraded multiple times, that "default" log settings changed in the > meantime. If I query the daemon for 'mon_status' and search for > log_to I see the following: > > # ceph daemon mon.node4 config show | grep log_to > "clog_to_graylog": "false", > "clog_to_graylog_host": "127.0.0.1", > "clog_to_graylog_port": "12201", > "clog_to_monitors": "default=true", > "clog_to_syslog": "false", > "clog_to_syslog_facility": "default=daemon audit=local0", > "clog_to_syslog_level": "info", > "leveldb_log_to_ceph_log": "true", > "log_to_file": "true", > "log_to_graylog": "false", > "log_to_journald": "false", > "log_to_stderr": "false", > "log_to_syslog": "false", > "mon_cluster_log_to_file": "true", > "mon_cluster_log_to_graylog": "false", > "mon_cluster_log_to_graylog_host": "127.0.0.1", > "mon_cluster_log_to_graylog_port": "12201", > "mon_cluster_log_to_journald": "false", > "mon_cluster_log_to_stderr": "false", > "mon_cluster_log_to_syslog": "default=false", > "mon_cluster_log_to_syslog_facility": "daemon", > "mon_cluster_log_to_syslog_level": "info", > "rocksdb_log_to_ceph_log": "true", > > > If I understand correctly I don't see the log in journald because > '"log_to_journald": "false"' (in fact, on the Octopus node there is > no "log_to_journald" option). Is this the correct default behaviour? > > > >>>> What I'm wondering about, why did you remove the MONs you wanted >>>> to upgrade from Ceph? Is that recommended somewhere? Maybe you >>>> need to run 'ceph mon add <name> <addr>' as well to let node1 or >>>> node2 join the monmap again? I'm not sure, it has been a long >>>> time doing this kind of stuff. ;-) But I don't think I ever >>>> removed a MON before upgrading it, not even when reinstalling the >>>> OS. >>> >>> No, we didn't follow any recommendations, it seemed reasonable to >>> remove a monitor before reinstalling the OS, a fresh start looked >>> better than a backup/restore of '/var/lib/ceph/'. Maybe we were >>> wrong. For the next upgrades (node3-5) we could think not to do >>> that and see what it happens. >> >> I don't exactly remember, tbh, but I expect that it would suffice >> to have the /var/lib/ceph/mon/ceph-nodeX directory with the correct >> permissions present in order for it to re-sync with the other MONs. >> But please double-check that, I'm really not sure. If I have the >> time, I will try to test that in a lab environment. > > Thank you! I will check it before doing anything strange. > > BTW, I remember that another reason for the MONs removal was that we > needed to change the hostname (not the IP address) and we taught > that a fresh install would be better. > > [cut] > >>>>>> - Inspecting debug logs on the syncing MON could help finding >>>>>> the root cause. >>>>> >>>>> With "syncing MON" you mean the MON which is just added to the >>>>> cluster, right? Collecting this kind of logs is on the to-do >>>>> list, but blocked because the "log problem" mentioned bellow. >>>> >>>> Yes, that's what I meant. The missing logs are a problem, of >>>> course. It does sound familiar, but I don't have a good idea >>>> right now except for looking at directory and file permissions, >>>> apparmor/selinux things, dmesg, stuff like that. >>> >>> >>> Yep. My biggest fear right now is that lack of logging from Quincy >>> monitor daemons could mean that they don't work properly and if we >>> restart the Octopus one (or it doesn't want to rejoin the quorum) >>> we will have a not functioning cluster. We need to investigate >>> further into it. >> >> I totally understand your hesitation, before I'd do that I would >> also try to rule out anything I can. But at the same time it would >> be surprising, given your cluster shows a healthy quorum right now. >> But yeah, investigate further before moving forward is the safer >> choice here. >> > > Probably you are right. I'll be cautiuos. > > Thank you for your help! > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Changing EC profile
by Eugen Block 15 Apr '26

15 Apr '26

From my understanding the new placement will not be available if the period can't be updated. Your zonegroup/zone configuration seems off compared to the test cluster. Is there a default zonegroup/zone? Zitat von Rok Jaklič via ceph-users <ceph-users(a)ceph.io>: > So I've created a little plan on how to create a new profile and assign new > S3 users to EC 8+3 pool. Idea is that all new s3 users are using a new > pool, and old ones pool with EC 3+2 and data is intact. > > --- > ceph osd erasure-code-profile set ec-83-profile k=8 m=3 > crush-failure-domain=host > ceph osd pool create rgw.buckets.ec-83.data 256 erasure ec-83-profile > ceph osd pool application enable rgw.buckets.ec-83.data rgw > > radosgw-admin zonegroup placement add --placement-id=ec-83-placement > > radosgw-admin zone placement add \ > --placement-id ec-83-placement \ > --data-pool rgw.buckets.ec-83.data \ > --index-pool default.rgw.buckets.index \ > --data-extra-pool default.rgw.buckets.non-ec \ > --placement-index-type 0 > > radosgw-admin period update --commit > > # restart rgw > systemctl restart ceph-radosgw.target > # verify the new placement > radosgw-admin zone get | grep -A 20 "ec-83-placement" > > # for new s3 users, set the default placement > radosgw-admin user create --uid=<user> --tenant=<tenant> > --placement-id=ec-83-placement > --- > > It seems to work on "test cluster", however I get an error on: > > [root@a8-mon1 ~]# radosgw-admin period update --commit > cannot commit period: period does not have a master zone of a master > zonegroup > failed to commit period: (22) Invalid argument > > Do I actually need to run this command or is "rgw restart" enough? > > Any comments appriciated. > > Rok > > > On Fri, Apr 3, 2026 at 12:42 PM Rok Jaklič via ceph-users < > ceph-users(a)ceph.io> wrote: > >> Hi, >> >> we expanded the cluster to 12 nodes with 28 osds, but we are still running >> on EC 3+2, since we started cluster with 6 nodes. We would like to change >> it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much >> CPU/network capacities? Should I use isa/isa-l instead of jerasure? >> >> Can anyone share experience with 8+3? >> >> Kind regards, >> Rok >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 15 Apr '26

15 Apr '26

Hi! On 15/04/26 11:11, Eugen Block via ceph-users wrote: > Hi, > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > [cut] >>> So your cluster currently has quorum with 2 out of 3 MONs on Quincy >>> already, correct? Restarting the mon on node3 shouldn't be dangerous, >>> I think. You'd still have quorum with node4 and node5, even if node3 >>> wouldn't be able to join. If it doesn't join as Octopus, you could >>> still upgrade it and retry with Quincy. >> >> Yes. I was just reluctant to do so because in the case it doesn't join >> our room to manoeuvre will be reduced. Plus the fact that node3 is the >> only daemon which is still logging something... > > oh I didn't realize there was only one left logging something. I thought > only the newly to be joined MON didn't log. In that case I would also > try to find the root cause for that. It could easily be, because this is an old cluster which was upgraded multiple times, that "default" log settings changed in the meantime. If I query the daemon for 'mon_status' and search for log_to I see the following: # ceph daemon mon.node4 config show | grep log_to "clog_to_graylog": "false", "clog_to_graylog_host": "127.0.0.1", "clog_to_graylog_port": "12201", "clog_to_monitors": "default=true", "clog_to_syslog": "false", "clog_to_syslog_facility": "default=daemon audit=local0", "clog_to_syslog_level": "info", "leveldb_log_to_ceph_log": "true", "log_to_file": "true", "log_to_graylog": "false", "log_to_journald": "false", "log_to_stderr": "false", "log_to_syslog": "false", "mon_cluster_log_to_file": "true", "mon_cluster_log_to_graylog": "false", "mon_cluster_log_to_graylog_host": "127.0.0.1", "mon_cluster_log_to_graylog_port": "12201", "mon_cluster_log_to_journald": "false", "mon_cluster_log_to_stderr": "false", "mon_cluster_log_to_syslog": "default=false", "mon_cluster_log_to_syslog_facility": "daemon", "mon_cluster_log_to_syslog_level": "info", "rocksdb_log_to_ceph_log": "true", If I understand correctly I don't see the log in journald because '"log_to_journald": "false"' (in fact, on the Octopus node there is no "log_to_journald" option). Is this the correct default behaviour? >>> What I'm wondering about, why did you remove the MONs you wanted to >>> upgrade from Ceph? Is that recommended somewhere? Maybe you need to >>> run 'ceph mon add <name> <addr>' as well to let node1 or node2 join >>> the monmap again? I'm not sure, it has been a long time doing this >>> kind of stuff. ;-) But I don't think I ever removed a MON before >>> upgrading it, not even when reinstalling the OS. >> >> No, we didn't follow any recommendations, it seemed reasonable to >> remove a monitor before reinstalling the OS, a fresh start looked >> better than a backup/restore of '/var/lib/ceph/'. Maybe we were wrong. >> For the next upgrades (node3-5) we could think not to do that and see >> what it happens. > > I don't exactly remember, tbh, but I expect that it would suffice to > have the /var/lib/ceph/mon/ceph-nodeX directory with the correct > permissions present in order for it to re-sync with the other MONs. But > please double-check that, I'm really not sure. If I have the time, I > will try to test that in a lab environment. Thank you! I will check it before doing anything strange. BTW, I remember that another reason for the MONs removal was that we needed to change the hostname (not the IP address) and we taught that a fresh install would be better. [cut] >>>>> - Inspecting debug logs on the syncing MON could help finding the >>>>> root cause. >>>> >>>> With "syncing MON" you mean the MON which is just added to the >>>> cluster, right? Collecting this kind of logs is on the to-do list, >>>> but blocked because the "log problem" mentioned bellow. >>> >>> Yes, that's what I meant. The missing logs are a problem, of course. >>> It does sound familiar, but I don't have a good idea right now except >>> for looking at directory and file permissions, apparmor/selinux >>> things, dmesg, stuff like that. >> >> >> Yep. My biggest fear right now is that lack of logging from Quincy >> monitor daemons could mean that they don't work properly and if we >> restart the Octopus one (or it doesn't want to rejoin the quorum) we >> will have a not functioning cluster. We need to investigate further >> into it. > > I totally understand your hesitation, before I'd do that I would also > try to rule out anything I can. But at the same time it would be > surprising, given your cluster shows a healthy quorum right now. But > yeah, investigate further before moving forward is the safer choice here. > Probably you are right. I'll be cautiuos. Thank you for your help! Iztok

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 15 Apr '26

15 Apr '26

Hi, Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi, > > comments inline. > > On 15/04/26 09:31, Eugen Block via ceph-users wrote: > > [cut] > >>>> - Can you share the output of 'ceph mon dump'? >>> >>> Sure! >>> >>> # ceph mon dump >>> epoch 18 >>> fsid 5bed1bf6-123f-4597-aaaa-d77931228548 >>> last_changed 2026-04-08T20:23:59.781295+0200 >>> created 2018-01-09T10:20:36.351967+0100 >>> min_mon_release 15 (octopus) >>> election_strategy: 1 >>> 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 >>> 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 >>> 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 >>> dumped monmap epoch 18 >> >> So your cluster currently has quorum with 2 out of 3 MONs on Quincy >> already, correct? Restarting the mon on node3 shouldn't be >> dangerous, I think. You'd still have quorum with node4 and node5, >> even if node3 wouldn't be able to join. If it doesn't join as >> Octopus, you could still upgrade it and retry with Quincy. > > Yes. I was just reluctant to do so because in the case it doesn't > join our room to manoeuvre will be reduced. Plus the fact that node3 > is the only daemon which is still logging something... oh I didn't realize there was only one left logging something. I thought only the newly to be joined MON didn't log. In that case I would also try to find the root cause for that. >> What I'm wondering about, why did you remove the MONs you wanted to >> upgrade from Ceph? Is that recommended somewhere? Maybe you need to >> run 'ceph mon add <name> <addr>' as well to let node1 or node2 join >> the monmap again? I'm not sure, it has been a long time doing this >> kind of stuff. ;-) But I don't think I ever removed a MON before >> upgrading it, not even when reinstalling the OS. > > No, we didn't follow any recommendations, it seemed reasonable to > remove a monitor before reinstalling the OS, a fresh start looked > better than a backup/restore of '/var/lib/ceph/'. Maybe we were > wrong. For the next upgrades (node3-5) we could think not to do that > and see what it happens. I don't exactly remember, tbh, but I expect that it would suffice to have the /var/lib/ceph/mon/ceph-nodeX directory with the correct permissions present in order for it to re-sync with the other MONs. But please double-check that, I'm really not sure. If I have the time, I will try to test that in a lab environment. >> Just for completeness sake, could you also share 'ceph -s' and >> 'ceph versions'? > > Sure! > > # ceph -s > cluster: > id: 5bed1bf6-123f-4597-aaa-d77931228548 > health: HEALTH_OK > > services: > mon: 3 daemons, quorum node3,node4,node5 (age 6d) > mgr: node1(active, since 6d), standbys: node2 > osd: 852 osds: 849 up (since 2w), 849 in (since 8w) > > data: > pools: 11 pools, 13825 pgs > objects: 447.25M objects, 1.7 PiB > usage: 5.0 PiB used, 3.7 PiB / 8.7 PiB avail > pgs: 13817 active+clean > 8 active+clean+scrubbing+deep > > io: > client: 3.0 MiB/s rd, 53 MiB/s wr, 35 op/s rd, 1.05k op/s wr > cache: 0 op/s promote > > # ceph versions > { > "mon": { > "ceph version 15.2.16 > (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 1, > "ceph version 17.2.6 > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2 > }, > "mgr": { > "ceph version 17.2.9 > (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 > }, > "osd": { > "ceph version 15.2.16 > (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 179, > "ceph version 15.2.17 > (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670 > }, > "mds": {}, > "overall": { > "ceph version 15.2.16 > (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 180, > "ceph version 15.2.17 > (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670, > "ceph version 17.2.6 > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2, > "ceph version 17.2.9 > (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 > } > } > > >>>> - There were a couple of reports back at O/P versions about MONs >>>> not joining the quorum. Sometimes it worked after hours, >>>> sometimes it didn't, and I'm still not aware of a real solution. >>>> One of our customers has been facing severe MON sync delay, >>>> sometimes taking around 40 minutes until a MON joined quorum. It >>>> turned out to be caused by some paxos default settings, requring >>>> the MONs to do a full sync instead of a diff sync. This has to do >>>> with millions of snapshots they have in their cluster. What made >>>> it worse was the MON's filesystems were on HDDs, increasing the >>>> sync time as well. >>> >>> We didn't wait for a long time before stopping the new MON a >>> regain "control" of the cluster, maybe 10 minutes max... >>> >>>> - Inspecting debug logs on the syncing MON could help finding the >>>> root cause. >>> >>> With "syncing MON" you mean the MON which is just added to the >>> cluster, right? Collecting this kind of logs is on the to-do list, >>> but blocked because the "log problem" mentioned bellow. >> >> Yes, that's what I meant. The missing logs are a problem, of >> course. It does sound familiar, but I don't have a good idea right >> now except for looking at directory and file permissions, >> apparmor/selinux things, dmesg, stuff like that. > > > Yep. My biggest fear right now is that lack of logging from Quincy > monitor daemons could mean that they don't work properly and if we > restart the Octopus one (or it doesn't want to rejoin the quorum) we > will have a not functioning cluster. We need to investigate further > into it. I totally understand your hesitation, before I'd do that I would also try to rule out anything I can. But at the same time it would be surprising, given your cluster shows a healthy quorum right now. But yeah, investigate further before moving forward is the safer choice here. > Iztok > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Changing EC profile
by Rok Jaklič 15 Apr '26

15 Apr '26

So I've created a little plan on how to create a new profile and assign new S3 users to EC 8+3 pool. Idea is that all new s3 users are using a new pool, and old ones pool with EC 3+2 and data is intact. --- ceph osd erasure-code-profile set ec-83-profile k=8 m=3 crush-failure-domain=host ceph osd pool create rgw.buckets.ec-83.data 256 erasure ec-83-profile ceph osd pool application enable rgw.buckets.ec-83.data rgw radosgw-admin zonegroup placement add --placement-id=ec-83-placement radosgw-admin zone placement add \ --placement-id ec-83-placement \ --data-pool rgw.buckets.ec-83.data \ --index-pool default.rgw.buckets.index \ --data-extra-pool default.rgw.buckets.non-ec \ --placement-index-type 0 radosgw-admin period update --commit # restart rgw systemctl restart ceph-radosgw.target # verify the new placement radosgw-admin zone get | grep -A 20 "ec-83-placement" # for new s3 users, set the default placement radosgw-admin user create --uid=<user> --tenant=<tenant> --placement-id=ec-83-placement --- It seems to work on "test cluster", however I get an error on: [root@a8-mon1 ~]# radosgw-admin period update --commit cannot commit period: period does not have a master zone of a master zonegroup failed to commit period: (22) Invalid argument Do I actually need to run this command or is "rgw restart" enough? Any comments appriciated. Rok On Fri, Apr 3, 2026 at 12:42 PM Rok Jaklič via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > we expanded the cluster to 12 nodes with 28 osds, but we are still running > on EC 3+2, since we started cluster with 6 nodes. We would like to change > it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much > CPU/network capacities? Should I use isa/isa-l instead of jerasure? > > Can anyone share experience with 8+3? > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 15 Apr '26

15 Apr '26

Hi, comments inline. On 15/04/26 09:31, Eugen Block via ceph-users wrote: [cut] >>> - Can you share the output of 'ceph mon dump'? >> >> Sure! >> >> # ceph mon dump >> epoch 18 >> fsid 5bed1bf6-123f-4597-aaaa-d77931228548 >> last_changed 2026-04-08T20:23:59.781295+0200 >> created 2018-01-09T10:20:36.351967+0100 >> min_mon_release 15 (octopus) >> election_strategy: 1 >> 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 >> 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 >> 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 >> dumped monmap epoch 18 > > So your cluster currently has quorum with 2 out of 3 MONs on Quincy > already, correct? Restarting the mon on node3 shouldn't be dangerous, I > think. You'd still have quorum with node4 and node5, even if node3 > wouldn't be able to join. If it doesn't join as Octopus, you could still > upgrade it and retry with Quincy. Yes. I was just reluctant to do so because in the case it doesn't join our room to manoeuvre will be reduced. Plus the fact that node3 is the only daemon which is still logging something... > What I'm wondering about, why did you remove the MONs you wanted to > upgrade from Ceph? Is that recommended somewhere? Maybe you need to run > 'ceph mon add <name> <addr>' as well to let node1 or node2 join the > monmap again? I'm not sure, it has been a long time doing this kind of > stuff. ;-) But I don't think I ever removed a MON before upgrading it, > not even when reinstalling the OS. No, we didn't follow any recommendations, it seemed reasonable to remove a monitor before reinstalling the OS, a fresh start looked better than a backup/restore of '/var/lib/ceph/'. Maybe we were wrong. For the next upgrades (node3-5) we could think not to do that and see what it happens. > Just for completeness sake, could you also share 'ceph -s' and 'ceph > versions'? Sure! # ceph -s cluster: id: 5bed1bf6-123f-4597-aaa-d77931228548 health: HEALTH_OK services: mon: 3 daemons, quorum node3,node4,node5 (age 6d) mgr: node1(active, since 6d), standbys: node2 osd: 852 osds: 849 up (since 2w), 849 in (since 8w) data: pools: 11 pools, 13825 pgs objects: 447.25M objects, 1.7 PiB usage: 5.0 PiB used, 3.7 PiB / 8.7 PiB avail pgs: 13817 active+clean 8 active+clean+scrubbing+deep io: client: 3.0 MiB/s rd, 53 MiB/s wr, 35 op/s rd, 1.05k op/s wr cache: 0 op/s promote # ceph versions { "mon": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 1, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2 }, "mgr": { "ceph version 17.2.9 (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 }, "osd": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 179, "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670 }, "mds": {}, "overall": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 180, "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2, "ceph version 17.2.9 (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 } } >>> - There were a couple of reports back at O/P versions about MONs not >>> joining the quorum. Sometimes it worked after hours, sometimes it >>> didn't, and I'm still not aware of a real solution. One of our >>> customers has been facing severe MON sync delay, sometimes taking >>> around 40 minutes until a MON joined quorum. It turned out to be >>> caused by some paxos default settings, requring the MONs to do a full >>> sync instead of a diff sync. This has to do with millions of >>> snapshots they have in their cluster. What made it worse was the >>> MON's filesystems were on HDDs, increasing the sync time as well. >> >> We didn't wait for a long time before stopping the new MON a regain >> "control" of the cluster, maybe 10 minutes max... >> >>> - Inspecting debug logs on the syncing MON could help finding the >>> root cause. >> >> With "syncing MON" you mean the MON which is just added to the >> cluster, right? Collecting this kind of logs is on the to-do list, but >> blocked because the "log problem" mentioned bellow. > > Yes, that's what I meant. The missing logs are a problem, of course. It > does sound familiar, but I don't have a good idea right now except for > looking at directory and file permissions, apparmor/selinux things, > dmesg, stuff like that. Yep. My biggest fear right now is that lack of logging from Quincy monitor daemons could mean that they don't work properly and if we restart the Octopus one (or it doesn't want to rejoin the quorum) we will have a not functioning cluster. We need to investigate further into it. Iztok

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 15 Apr '26

15 Apr '26

Hi, some comments inline. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi Eugen! > > Thanks for the support, my response is inline > > On 14/04/26 13:03, Eugen Block via ceph-users wrote: >> Hi, >> >> I don't have a solution (yet?), but a couple of points to clarify. >> >> - I assume your cluster isn't managed by cephadm, correct? > > Correct! It's an old cluster initialy deployed with ceph-deploy in > the Hammer days and upgraded since then. Right now is manually > managed (but one of the targets at the end of the upgrade is to > switch to cephadm). > > >> - Can you share the output of 'ceph mon dump'? > > Sure! > > # ceph mon dump > epoch 18 > fsid 5bed1bf6-123f-4597-aaaa-d77931228548 > last_changed 2026-04-08T20:23:59.781295+0200 > created 2018-01-09T10:20:36.351967+0100 > min_mon_release 15 (octopus) > election_strategy: 1 > 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 > 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 > 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 > dumped monmap epoch 18 So your cluster currently has quorum with 2 out of 3 MONs on Quincy already, correct? Restarting the mon on node3 shouldn't be dangerous, I think. You'd still have quorum with node4 and node5, even if node3 wouldn't be able to join. If it doesn't join as Octopus, you could still upgrade it and retry with Quincy. What I'm wondering about, why did you remove the MONs you wanted to upgrade from Ceph? Is that recommended somewhere? Maybe you need to run 'ceph mon add <name> <addr>' as well to let node1 or node2 join the monmap again? I'm not sure, it has been a long time doing this kind of stuff. ;-) But I don't think I ever removed a MON before upgrading it, not even when reinstalling the OS. Just for completeness sake, could you also share 'ceph -s' and 'ceph versions'? >> - There were a couple of reports back at O/P versions about MONs >> not joining the quorum. Sometimes it worked after hours, sometimes >> it didn't, and I'm still not aware of a real solution. One of our >> customers has been facing severe MON sync delay, sometimes taking >> around 40 minutes until a MON joined quorum. It turned out to be >> caused by some paxos default settings, requring the MONs to do a >> full sync instead of a diff sync. This has to do with millions of >> snapshots they have in their cluster. What made it worse was the >> MON's filesystems were on HDDs, increasing the sync time as well. > > We didn't wait for a long time before stopping the new MON a regain > "control" of the cluster, maybe 10 minutes max... > >> - Inspecting debug logs on the syncing MON could help finding the >> root cause. > > With "syncing MON" you mean the MON which is just added to the > cluster, right? Collecting this kind of logs is on the to-do list, > but blocked because the "log problem" mentioned bellow. Yes, that's what I meant. The missing logs are a problem, of course. It does sound familiar, but I don't have a good idea right now except for looking at directory and file permissions, apparmor/selinux things, dmesg, stuff like that. >> >> Some more comments inline. >> > > [cut] > >>> >>> We taught, among other things, to: >>> >>> - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is >>> the only daemon which wasn't restarted and when queried ("ceph >>> daemon mon.node1 mon_status") has in "extra_probe_peers" the IP >>> addresses of the original removed nodes, which are the same IPs as >>> the nodes we want to add. We aren't sure if this prevent the >>> ceph-mon to join, because we don't understand when/what >>> "extra_probe_peers" is doing. >> >> According to the output above node1 is already out of the cluster, >> are you referring to node3? >> >>> - Remove the node1 from the cluster, set min_mon_release to Quincy >>> and then try to add the new monitors. > > Yes, sorry for the misunderstanding, I was referring to node3 in > both instances. > > >>> >>> We are unsure about both approaches because if they don't work we >>> will have a "unstable" cluster with only 2 monitors and no much >>> space to operate. >>> >>> To "complicate" the matter we notice that on node4 and node5 the >>> ceph- mon is not logging anymore to "/var/log/ceph/ceph*.log" even >>> tough a "mon_cluster_log_to_file: true". If I search for open FD >>> there is nothing in /var/log/ceph for the running ceph-mon... >> >> Is there something in journalctl? > > We increased the 'debug_mon' to 20 ('ceph daemon mon.node4 config > set debug_mon 20') and we check it with 'ceph daemon mon.node4 > config show' but the output of 'journalctl -u ceph-mon(a)mon.node4` is > not more verbose then before, the last line is from 2:00 this > morning and it's related to sudo: > > apr 14 02:35:40 node4 sudo[881890]: ceph : PWD=/ ; USER=root ; > COMMAND=/usr/sbin/smartctl -x --json=o /dev/md127 > > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Malte Stroem 14 Apr '26

14 Apr '26

Hi Marek, add storage to the cluster. Please show us: ceph -s Best, Malte On 4/14/26 17:53, Marek Szuba via ceph-users wrote: > Hello, > > After the latest run of system updates and reboots on our prototype Ceph > cluster (Ubuntu Jammy + cephadm-managed Ceph 20.2.0 on Podman), we have > ended up with something seriously weird. Specifically, at a glance the > cluster appears to be in service: > > * Ceph reports the health of the cluster as OK > * according to both 'ceph -s' and 'ceph orch ps', all the daemons which > should be up and running ARE up and running. In particular, all the > expected OSDs appear to be operational > * 'ceph -s' shows no recovery I/O > > However: > > * despite all of the above, 'ceph -s' reports 1448 remapped PGs and > about 24% objects classified as misplaced > * 'ceph pg ls remapped' does report a long list of PGs with misplaced > objects > * on said list, most if not all PGs seem (I haven't checked all 1448 > lines) to have 3 OSDs each listed under ACTING but only either 1 or 2 > listed under UP - but not in a consistent way, in that an OSD missing > from the UP list for one PG does appear on the UP lists of other PGs > > Any idea what might be going on here, and how to make it stop? We were > about to upgrade that cluster to 20.2.1 following the system updates, > however we are very much hesitant to do so while things look so odd in > case the upgrade makes things worse. >

1 0

Ceph User + Dev April Meetup
by Anthony Middleton 14 Apr '26

14 Apr '26

Hey all, The next Ceph User+Dev meetup is tomorrow, April 15th, at 14:00 UTC. Add your discussion topics here: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes I hope to see more people there! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0