- ceph-users - lists.ceph.io

Re: v20.2.2 merge window is now open!
by Alfonso Escribano Merino 09 Apr '26

09 Apr '26

1 0

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Gregory Farnum 08 Apr '26

08 Apr '26

Can you please talk more about how you’re running the cluster and what data is already on it? There’s nothing about growing a stretch cluster that naively would lead to this behavior, but it does have a lot more constraints than a regular Ceph cluster does about which OSDs it can use to go active with (because it needs a current copy in both sites, and it’s trying to keep 4 copies up to date rather than 3). -Greg On Mon, Apr 6, 2026 at 7:26 AM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > Are you using upmap-remapped to mitigate the thundering herd? And, if > you're using the PG autoscaler, to mitigate the thundering herd of backfill > that it adds? > > > On Apr 6, 2026, at 7:33 AM, Michal Strnad via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > Hi. > > > > We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 > OSDs and another 1000 OSDs to be added. We are concerned with issues > related to a large number of peering connections and remappings when a node > with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of > roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we > see reflected in the ceph status output before the cluster finally > stabilizes. > > > > In a classic (non‑stretch) Ceph cluster the convergence is noticeably > quicker (inactive PGs appear only for a very short interval). In our > stretch deployment, however, the window when PGs are inactive is > considerably longer, consistent with our observations. The network latency > between the two sites is consistently below 10 ms with no packet loss. > > > > We are currently trying to determine whether this impacts client‑side > read/write simulations via S3, but this has not yet been verified. Our next > step is to test with RBD, which should be more resilient and demonstrable. > Naturally, we would like to control the process in a way that minimizes the > impact on clients. Could you please confirm that the periods of PG > inactivity are short enough to prevent client visible glitches, or even > better, could you recommend an optimal protocol for adding new OSDs to a > cluster of this size to ensure a smooth convergence? Do you have any tested > or recommended timeout adjustments for the stretch configuration? > > > > Thank you for any feedback > > -- > > Michal > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Question Regarding Ceph Monitor Election Strategy Default Configuration
by Gregory Farnum 08 Apr '26

08 Apr '26

Mode 1 (classic) works well most of the time and is the default simply because it was first and has the most history. In mode 3 (connectivity) each monitor rates every other based on response rates to a periodic ping, and they use those scores to elect the monitor with the highest total score. Since this one is checking networking consistency and has a slight bias towards not changing the current leader, it makes sense that it would do better under a network loss scenario. Maybe someday we will make it the default, but there’s been no real reason to make the change so far. They ought to have no other differences, nor impact the operations of the rest of the Ceph cluster at all. -Greg On Mon, Apr 6, 2026 at 12:35 AM Christian Mauricio Narvaez Acaro via ceph-users <ceph-users(a)ceph.io> wrote: > Hi, > We are currently setting up and executing a testbench to analyze and > validate the resilience and robustness of the SEPATH hypervisor running > across the three nodes of a cluster (SEAPATH - LF Energy Wiki< > https://lf-energy.atlassian.net/wiki/spaces/SEAP/overview>). > As part of these tests, we evaluated the behavior of the Ceph service in a > simulated failure scenario. Using tc and netem, we introduced approximately > 40% random packet loss on the cluster network of one node, which resulted > in quorum instability. > > The main observations were as follows: > > * Continuous increase in election_epoch, indicating frequent monitor > re-elections. > * Intermittent quorum formation, with transitions between (1,2), > (1,3), (2,3), and (1,2,3). > * Repeated changes in quorum_leader_name, alternating mainly between > node 1 and node 2. > * A tendency for hypervisor1 (rank 0) to regain leadership whenever it > restored connectivity with a majority of nodes. > > > At the time of the test, we found that election_strategy was configured > with value 1. Based on our understanding, this behavior is consistent with > the classic election mechanism, where all monitors are treated equally and > the monitor with the lowest rank is typically selected as leader. > > To further analyze this behavior, we compared the two monitor election > strategies under the same induced failure condition. In both cases, we > captured TCP traffic on port 3300 over a 10-minute period using tshark. > With the classic strategy, repeated tests showed between 68 and 74 leader > changes, continuous quorum leave-and-rejoin events for the affected node, > and approximately double the traffic compared to the lossless scenario > (636Kbits/s versus 340Kbits/s). By contrast, with the connectivity-based > strategy, repeated tests showed only 1 to 4 leader changes, leadership > limited to node 2 and node 3, no rejoin of the affected node to the quorum, > and only an approximately 10% increase in TCP traffic compared to the > lossless case (437 Kbits/s versus 400 Kbits/s) > From a traffic perspective, this approach consumes less bandwidth and > significantly improves system stability during cluster network failures, > since node 1 is no longer repeatedly re-elected as leader while > experiencing network issues. > Based on the measurements we have collected and our comparison of the > election algorithms, strategy 3 appears to be the most suitable option for > our use case. We would therefore appreciate understanding whether there is > any additional consideration or technical reason for preferring strategy 1 > over strategy 3, which may explain why it is the default configuration. > Thanks in advance. > > Best regards, > > Christian Narváez A. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: v20.2.2 merge window is now open!
by Stefan Kooman 08 Apr '26

08 Apr '26

On 4/6/26 20:02, Patrick Donnelly via dev wrote: > Hello everyone, > > v20.2.1 has finally been released! > > The tentacle merge window is open again. Please follow this tracker > ticket for updates and processes for getting backporrts merged: > > https://tracker.ceph.com/issues/75878 I would like to have this in: https://tracker.ceph.com/issues/75267 I've successfully tested this on a patched 20.2.0. Thanks, Gr. Stefan

1 0

MGR stuck in starting state
by Iztok Gregori 08 Apr '26

08 Apr '26

Hi to all! First a little of background. We are in the process of a massive upgrade from Octopus to Quincy (during which there is also the upgrade (reinstall) from CentOS 7/Rocky 8 to Rocky 9). During the first phase we removed from the cluster 2 monitor/managers and reinstalled them with Rocky 9 and Ceph 17.2.9). The 3 remaining monitors/manager were: - 2 x Rocky 8 with 15.2.17 - 1 x CentOS 7 with 15.2.16 Then we upgraded Ceph from 15.2.17 to 17.9.6 on the 2 nodes with Rocky 8 and restarted the services. So to summarize we have: - 2 x MON Rocky 8 with 17.2.6 - 1 x MON CentOS 7 with 15.2.16 and - 1 x MGR CentOS 7 with 15.2.16 The next step was add the new (reinstalled) nodes as MON in the cluster we encounter the first problem. Once started the new ceph-mon daemon entered the cluster but all the commands ("ceph -s" for example) remained stuck. If we stopped the new ceph-mon daemon the commands restart to work (the status was in warning because the new ceph-mon was down). To regain the quorum we deleted the new ceph-mon. And here I think we made a mistake, we started the ceph-mgr daemons on the 2x Rocky 8 nodes. So we had: - 2 x MGR Rocky 8 with 17.2.6 (standby) - 1 x MGR CentOS 7 with 15.2.16 (active) We wrongly assume that the MGR was the culprit and we fail the active MGR. From then on the status of the active MGR is in starting and all the PGS are unknown: cluster: id: 5bed1bf6-123f-4597-b8b1-d77931228548 health: HEALTH_WARN Reduced data availability: 13825 pgs inactive services: mon: 3 daemons, quorum n-3-c7,n-2-r8,n-1-r8 (age 91m) mgr: n-1-r8 (active, starting, since 13m), standbys: n-2-r8, n-3-c7 osd: 852 osds: 849 up (since 8d), 849 in (since 7w) data: pools: 11 pools, 13825 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 13825 unknown From the logs we see > 2026-04-08T16:59:26.244+0200 7fe0596a1700 0 ms_deliver_dispatch: unhandled message 0x5583db0d0c60 mon_map magic: 0 v1 from mon.2 v2:172.19.240.35:3300/0 > 2026-04-08T16:59:26.807+0200 7fe0596a1700 1 mgr handle_mgr_map Activating! > 2026-04-08T16:59:26.808+0200 7fe0596a1700 1 mgr handle_mgr_map I am now activating > 2026-04-08T16:59:34.923+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:36.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:38.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:40.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:42.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:44.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:46.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:48.926+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:50.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:52.926+0200 7fdfd7e70700 1 mgr.server send_report Giving up on OSDs that haven't reported yet, sending potentially incomplete PG state to mon The mgr dump looks like this: > { > "epoch": 373, > "active_gid": 16955846717, > "active_name": "n-1-r8", > "active_addrs": { > "addrvec": [] > }, > "active_addr": "(unrecognized address family 0)/0", > "active_change": "2026-04-08T16:59:26.282185+0200", > "active_mgr_features": 4540138320759226367, > "available": false, > "standbys": [ > { > "gid": 16955784229, > "name": "n-2-r8", > "mgr_features": 4540138320759226367, > "available_modules": [ We think that "unrecognized address family" could be the problem, because we have a cluster/public network which is on a different interface and could be that is not recognized/binded on mgr startup. But we might be wrong. Somebody can give us some help? Thanks in advance Iztok Gregori -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A.

1 0

Re: v20.2.2 merge window is now open!
by Sascha Lucas 07 Apr '26

07 Apr '26

Hi Robert, On Tue, 7 Apr 2026, Robert Sander via ceph-users wrote: > It would be nice if https://tracker.ceph.com/issues/72696 could make it into > 20.2.2. Also having older clusters with DB devices on NVME. I thought this setups are not so uncommon? But good to know, not to upgrade to v19/v20 until fixed. This BUG seems not yet listed on the clyso's Known Bugs page[1]. There should be more transparency for normal ceph users, so that not everyone ends up with broken setups on upgrade. BUGs are uncomfortable, but not knowing them is sad. Thanks, Sascha. [1] https://docs.clyso.com/docs/kb/category/known-bugs/

1 0

Re: Changing EC profile
by Rok Jaklič 07 Apr '26

07 Apr '26

On Fri, Apr 3, 2026 at 4:06 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > >> * Are you looking to change the profile to get more usable capacity? > >> * Are you looking to change the profile to get more durability / > sustain more failures? > > Not necessarily, we would rather have a responsive rgw for example > during failures, so when there is backfill/recovery going on that rgw does > not stop responding. > > The wider profile may actually make that worse. > > Since you mention RGW and only mention HDDs, does that mean that your > index pool is on HDDs? There's a good chance that was a factor. HDDs have > very limited IOPS and all of the RGW pools that aren't bucket data really > should be on SSDs. I speculate that client ops, backfill, and RGW ops were > DoSing each other. > Forgot to add, index pools are on NVMEs. Some old OSD nodes have 2 NVMEe inside, newer ones still do not have, so we will have to consider adding NVMEs also. > An expedient way to add those would be something like > > > https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTRl12SIsjlOJTId1V6Z… > > > A PCI-e AIC that accepts 2x NVMe M.2 drives, assuming that your > motherboard can do PCIe bifurcation. You could populate each with like 2x > 1.9T Micron 7450 and place the RGW pools on those. For this purpose having > more OSDs is more beneficial than having larger OSDs. You could easily see > the size of your existing pools to help gauge what you need. > Thx. > > > It happened a few times. e.g. slow/faulty disk from on which reweight > was just set > > You aren't using "ceph osd reweight" or "ceph osd reweight-by-utilization" > are you? Those are legacies and should be avoided. You want the REWEIGHT > value on each OSD to be 1.0000 and to use the manager balancer. > > ceph config set mgr mgr/balancer/upmap_max_deviation 1 > Actually I am doing `ceph osd crush reweight X ...` when using it. But not for faulty OSDs. On faulty OSDs, I set the OSD to out, replace it, activate it. I am using following procedure: 1. systemctl stop ceph-osd@{id} 2. ceph osd down osd.{id} 3. ceph osd out osd.{id} 4. pocakas rebalance 5. ceph osd destroy {id} --yes-i-really-mean-it --- 6. replace disk... 7. ceph-volume lvm prepare --osd-id {id} --data /dev/sdX 8. ceph-volume lvm list 9. ceph-volume lvm activate {id} {osd fsid} Does that mean you're only using RGW? No RBD, no CephFS? This informs the > process by which you would migrate data between pools. If you don't have > capacity concerns, maybe migration isn't worth the trouble. Only RGW for now. We did test CephFS a little bit, but RGW suffices our needs currently. Rok

1 0

Re: v20.2.2 merge window is now open!
by Robert Sander 07 Apr '26

07 Apr '26

Hi Patrick, Am 06.04.26 um 8:03 PM schrieb Patrick Donnelly via ceph-users: > For discussion, please visit #ceph-upstream-releases-tentacle on > Slack, respond to this mail, or come to the "Ceph Release Engineering" > on Thursdays (see Ceph community calendar). It would be nice if https://tracker.ceph.com/issues/72696 could make it into 20.2.2. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Janne Johansson 07 Apr '26

07 Apr '26

> We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 > OSDs and another 1000 OSDs to be added. We are concerned with issues > related to a large number of peering connections and remappings when a > node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of > roughly 2000 PGs and can leave tens of minutes of PGs inactive > Could you please confirm that the > periods of PG inactivity are short enough to prevent client visible > glitches, or even better, could you recommend an optimal protocol for > adding new OSDs to a cluster of this size to ensure a smooth > convergence? Do you have any tested or recommended timeout adjustments > for the stretch configuration? You could ask the new OSDs to join with initial size of 0.01 (or 0.0001 but that has certain issues too) so that few/no PGs decides to move over, then increase size later on slowly. We used to set initial OSD crush weight to 0.0001 which works, but clients that check "max free space" think the cluster is almost full, so now we do 0.01 on existing clusters, which for some reason makes free-testing-clients not think the cluster is about to be full any second. Then stop the auto-balancer, have norebalance set, bump sizes one OSD at a time, sleep some 20-30 secs (in our case), bump next OSD and so on, and lastly run pgremapper to cancel backfills using upmap, then unset norebalance and start the balancer so it can un-upmap each such PG in an orderly fashion. -- May the most significant bit of your life be positive.

1 0

v20.2.2 merge window is now open!
by Patrick Donnelly 06 Apr '26

06 Apr '26

Hello everyone, v20.2.1 has finally been released! The tentacle merge window is open again. Please follow this tracker ticket for updates and processes for getting backporrts merged: https://tracker.ceph.com/issues/75878 For discussion, please visit #ceph-upstream-releases-tentacle on Slack, respond to this mail, or come to the "Ceph Release Engineering" on Thursdays (see Ceph community calendar). Thanks, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0