- ceph-users - lists.ceph.io

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Gregory Farnum 08 Apr '26

08 Apr '26

Can you please talk more about how you’re running the cluster and what data is already on it? There’s nothing about growing a stretch cluster that naively would lead to this behavior, but it does have a lot more constraints than a regular Ceph cluster does about which OSDs it can use to go active with (because it needs a current copy in both sites, and it’s trying to keep 4 copies up to date rather than 3). -Greg On Mon, Apr 6, 2026 at 7:26 AM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > Are you using upmap-remapped to mitigate the thundering herd? And, if > you're using the PG autoscaler, to mitigate the thundering herd of backfill > that it adds? > > > On Apr 6, 2026, at 7:33 AM, Michal Strnad via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > Hi. > > > > We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 > OSDs and another 1000 OSDs to be added. We are concerned with issues > related to a large number of peering connections and remappings when a node > with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of > roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we > see reflected in the ceph status output before the cluster finally > stabilizes. > > > > In a classic (non‑stretch) Ceph cluster the convergence is noticeably > quicker (inactive PGs appear only for a very short interval). In our > stretch deployment, however, the window when PGs are inactive is > considerably longer, consistent with our observations. The network latency > between the two sites is consistently below 10 ms with no packet loss. > > > > We are currently trying to determine whether this impacts client‑side > read/write simulations via S3, but this has not yet been verified. Our next > step is to test with RBD, which should be more resilient and demonstrable. > Naturally, we would like to control the process in a way that minimizes the > impact on clients. Could you please confirm that the periods of PG > inactivity are short enough to prevent client visible glitches, or even > better, could you recommend an optimal protocol for adding new OSDs to a > cluster of this size to ensure a smooth convergence? Do you have any tested > or recommended timeout adjustments for the stretch configuration? > > > > Thank you for any feedback > > -- > > Michal > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Question Regarding Ceph Monitor Election Strategy Default Configuration
by Gregory Farnum 08 Apr '26

08 Apr '26

Mode 1 (classic) works well most of the time and is the default simply because it was first and has the most history. In mode 3 (connectivity) each monitor rates every other based on response rates to a periodic ping, and they use those scores to elect the monitor with the highest total score. Since this one is checking networking consistency and has a slight bias towards not changing the current leader, it makes sense that it would do better under a network loss scenario. Maybe someday we will make it the default, but there’s been no real reason to make the change so far. They ought to have no other differences, nor impact the operations of the rest of the Ceph cluster at all. -Greg On Mon, Apr 6, 2026 at 12:35 AM Christian Mauricio Narvaez Acaro via ceph-users <ceph-users(a)ceph.io> wrote: > Hi, > We are currently setting up and executing a testbench to analyze and > validate the resilience and robustness of the SEPATH hypervisor running > across the three nodes of a cluster (SEAPATH - LF Energy Wiki< > https://lf-energy.atlassian.net/wiki/spaces/SEAP/overview>). > As part of these tests, we evaluated the behavior of the Ceph service in a > simulated failure scenario. Using tc and netem, we introduced approximately > 40% random packet loss on the cluster network of one node, which resulted > in quorum instability. > > The main observations were as follows: > > * Continuous increase in election_epoch, indicating frequent monitor > re-elections. > * Intermittent quorum formation, with transitions between (1,2), > (1,3), (2,3), and (1,2,3). > * Repeated changes in quorum_leader_name, alternating mainly between > node 1 and node 2. > * A tendency for hypervisor1 (rank 0) to regain leadership whenever it > restored connectivity with a majority of nodes. > > > At the time of the test, we found that election_strategy was configured > with value 1. Based on our understanding, this behavior is consistent with > the classic election mechanism, where all monitors are treated equally and > the monitor with the lowest rank is typically selected as leader. > > To further analyze this behavior, we compared the two monitor election > strategies under the same induced failure condition. In both cases, we > captured TCP traffic on port 3300 over a 10-minute period using tshark. > With the classic strategy, repeated tests showed between 68 and 74 leader > changes, continuous quorum leave-and-rejoin events for the affected node, > and approximately double the traffic compared to the lossless scenario > (636Kbits/s versus 340Kbits/s). By contrast, with the connectivity-based > strategy, repeated tests showed only 1 to 4 leader changes, leadership > limited to node 2 and node 3, no rejoin of the affected node to the quorum, > and only an approximately 10% increase in TCP traffic compared to the > lossless case (437 Kbits/s versus 400 Kbits/s) > From a traffic perspective, this approach consumes less bandwidth and > significantly improves system stability during cluster network failures, > since node 1 is no longer repeatedly re-elected as leader while > experiencing network issues. > Based on the measurements we have collected and our comparison of the > election algorithms, strategy 3 appears to be the most suitable option for > our use case. We would therefore appreciate understanding whether there is > any additional consideration or technical reason for preferring strategy 1 > over strategy 3, which may explain why it is the default configuration. > Thanks in advance. > > Best regards, > > Christian Narváez A. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: v20.2.2 merge window is now open!
by Stefan Kooman 08 Apr '26

08 Apr '26

On 4/6/26 20:02, Patrick Donnelly via dev wrote: > Hello everyone, > > v20.2.1 has finally been released! > > The tentacle merge window is open again. Please follow this tracker > ticket for updates and processes for getting backporrts merged: > > https://tracker.ceph.com/issues/75878 I would like to have this in: https://tracker.ceph.com/issues/75267 I've successfully tested this on a patched 20.2.0. Thanks, Gr. Stefan

1 0

MGR stuck in starting state
by Iztok Gregori 08 Apr '26

08 Apr '26

Hi to all! First a little of background. We are in the process of a massive upgrade from Octopus to Quincy (during which there is also the upgrade (reinstall) from CentOS 7/Rocky 8 to Rocky 9). During the first phase we removed from the cluster 2 monitor/managers and reinstalled them with Rocky 9 and Ceph 17.2.9). The 3 remaining monitors/manager were: - 2 x Rocky 8 with 15.2.17 - 1 x CentOS 7 with 15.2.16 Then we upgraded Ceph from 15.2.17 to 17.9.6 on the 2 nodes with Rocky 8 and restarted the services. So to summarize we have: - 2 x MON Rocky 8 with 17.2.6 - 1 x MON CentOS 7 with 15.2.16 and - 1 x MGR CentOS 7 with 15.2.16 The next step was add the new (reinstalled) nodes as MON in the cluster we encounter the first problem. Once started the new ceph-mon daemon entered the cluster but all the commands ("ceph -s" for example) remained stuck. If we stopped the new ceph-mon daemon the commands restart to work (the status was in warning because the new ceph-mon was down). To regain the quorum we deleted the new ceph-mon. And here I think we made a mistake, we started the ceph-mgr daemons on the 2x Rocky 8 nodes. So we had: - 2 x MGR Rocky 8 with 17.2.6 (standby) - 1 x MGR CentOS 7 with 15.2.16 (active) We wrongly assume that the MGR was the culprit and we fail the active MGR. From then on the status of the active MGR is in starting and all the PGS are unknown: cluster: id: 5bed1bf6-123f-4597-b8b1-d77931228548 health: HEALTH_WARN Reduced data availability: 13825 pgs inactive services: mon: 3 daemons, quorum n-3-c7,n-2-r8,n-1-r8 (age 91m) mgr: n-1-r8 (active, starting, since 13m), standbys: n-2-r8, n-3-c7 osd: 852 osds: 849 up (since 8d), 849 in (since 7w) data: pools: 11 pools, 13825 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 13825 unknown From the logs we see > 2026-04-08T16:59:26.244+0200 7fe0596a1700 0 ms_deliver_dispatch: unhandled message 0x5583db0d0c60 mon_map magic: 0 v1 from mon.2 v2:172.19.240.35:3300/0 > 2026-04-08T16:59:26.807+0200 7fe0596a1700 1 mgr handle_mgr_map Activating! > 2026-04-08T16:59:26.808+0200 7fe0596a1700 1 mgr handle_mgr_map I am now activating > 2026-04-08T16:59:34.923+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:36.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:38.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:40.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:42.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:44.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:46.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:48.926+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:50.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs > 2026-04-08T16:59:52.926+0200 7fdfd7e70700 1 mgr.server send_report Giving up on OSDs that haven't reported yet, sending potentially incomplete PG state to mon The mgr dump looks like this: > { > "epoch": 373, > "active_gid": 16955846717, > "active_name": "n-1-r8", > "active_addrs": { > "addrvec": [] > }, > "active_addr": "(unrecognized address family 0)/0", > "active_change": "2026-04-08T16:59:26.282185+0200", > "active_mgr_features": 4540138320759226367, > "available": false, > "standbys": [ > { > "gid": 16955784229, > "name": "n-2-r8", > "mgr_features": 4540138320759226367, > "available_modules": [ We think that "unrecognized address family" could be the problem, because we have a cluster/public network which is on a different interface and could be that is not recognized/binded on mgr startup. But we might be wrong. Somebody can give us some help? Thanks in advance Iztok Gregori -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A.

1 0

Re: v20.2.2 merge window is now open!
by Sascha Lucas 07 Apr '26

07 Apr '26

Hi Robert, On Tue, 7 Apr 2026, Robert Sander via ceph-users wrote: > It would be nice if https://tracker.ceph.com/issues/72696 could make it into > 20.2.2. Also having older clusters with DB devices on NVME. I thought this setups are not so uncommon? But good to know, not to upgrade to v19/v20 until fixed. This BUG seems not yet listed on the clyso's Known Bugs page[1]. There should be more transparency for normal ceph users, so that not everyone ends up with broken setups on upgrade. BUGs are uncomfortable, but not knowing them is sad. Thanks, Sascha. [1] https://docs.clyso.com/docs/kb/category/known-bugs/

1 0

Re: Changing EC profile
by Rok Jaklič 07 Apr '26

07 Apr '26

On Fri, Apr 3, 2026 at 4:06 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > >> * Are you looking to change the profile to get more usable capacity? > >> * Are you looking to change the profile to get more durability / > sustain more failures? > > Not necessarily, we would rather have a responsive rgw for example > during failures, so when there is backfill/recovery going on that rgw does > not stop responding. > > The wider profile may actually make that worse. > > Since you mention RGW and only mention HDDs, does that mean that your > index pool is on HDDs? There's a good chance that was a factor. HDDs have > very limited IOPS and all of the RGW pools that aren't bucket data really > should be on SSDs. I speculate that client ops, backfill, and RGW ops were > DoSing each other. > Forgot to add, index pools are on NVMEs. Some old OSD nodes have 2 NVMEe inside, newer ones still do not have, so we will have to consider adding NVMEs also. > An expedient way to add those would be something like > > > https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTRl12SIsjlOJTId1V6Z… > > > A PCI-e AIC that accepts 2x NVMe M.2 drives, assuming that your > motherboard can do PCIe bifurcation. You could populate each with like 2x > 1.9T Micron 7450 and place the RGW pools on those. For this purpose having > more OSDs is more beneficial than having larger OSDs. You could easily see > the size of your existing pools to help gauge what you need. > Thx. > > > It happened a few times. e.g. slow/faulty disk from on which reweight > was just set > > You aren't using "ceph osd reweight" or "ceph osd reweight-by-utilization" > are you? Those are legacies and should be avoided. You want the REWEIGHT > value on each OSD to be 1.0000 and to use the manager balancer. > > ceph config set mgr mgr/balancer/upmap_max_deviation 1 > Actually I am doing `ceph osd crush reweight X ...` when using it. But not for faulty OSDs. On faulty OSDs, I set the OSD to out, replace it, activate it. I am using following procedure: 1. systemctl stop ceph-osd@{id} 2. ceph osd down osd.{id} 3. ceph osd out osd.{id} 4. pocakas rebalance 5. ceph osd destroy {id} --yes-i-really-mean-it --- 6. replace disk... 7. ceph-volume lvm prepare --osd-id {id} --data /dev/sdX 8. ceph-volume lvm list 9. ceph-volume lvm activate {id} {osd fsid} Does that mean you're only using RGW? No RBD, no CephFS? This informs the > process by which you would migrate data between pools. If you don't have > capacity concerns, maybe migration isn't worth the trouble. Only RGW for now. We did test CephFS a little bit, but RGW suffices our needs currently. Rok

1 0

Re: v20.2.2 merge window is now open!
by Robert Sander 07 Apr '26

07 Apr '26

Hi Patrick, Am 06.04.26 um 8:03 PM schrieb Patrick Donnelly via ceph-users: > For discussion, please visit #ceph-upstream-releases-tentacle on > Slack, respond to this mail, or come to the "Ceph Release Engineering" > on Thursdays (see Ceph community calendar). It would be nice if https://tracker.ceph.com/issues/72696 could make it into 20.2.2. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Janne Johansson 07 Apr '26

07 Apr '26

> We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 > OSDs and another 1000 OSDs to be added. We are concerned with issues > related to a large number of peering connections and remappings when a > node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of > roughly 2000 PGs and can leave tens of minutes of PGs inactive > Could you please confirm that the > periods of PG inactivity are short enough to prevent client visible > glitches, or even better, could you recommend an optimal protocol for > adding new OSDs to a cluster of this size to ensure a smooth > convergence? Do you have any tested or recommended timeout adjustments > for the stretch configuration? You could ask the new OSDs to join with initial size of 0.01 (or 0.0001 but that has certain issues too) so that few/no PGs decides to move over, then increase size later on slowly. We used to set initial OSD crush weight to 0.0001 which works, but clients that check "max free space" think the cluster is almost full, so now we do 0.01 on existing clusters, which for some reason makes free-testing-clients not think the cluster is about to be full any second. Then stop the auto-balancer, have norebalance set, bump sizes one OSD at a time, sleep some 20-30 secs (in our case), bump next OSD and so on, and lastly run pgremapper to cancel backfills using upmap, then unset norebalance and start the balancer so it can un-upmap each such PG in an orderly fashion. -- May the most significant bit of your life be positive.

1 0

v20.2.2 merge window is now open!
by Patrick Donnelly 06 Apr '26

06 Apr '26

Hello everyone, v20.2.1 has finally been released! The tentacle merge window is open again. Please follow this tracker ticket for updates and processes for getting backporrts merged: https://tracker.ceph.com/issues/75878 For discussion, please visit #ceph-upstream-releases-tentacle on Slack, respond to this mail, or come to the "Ceph Release Engineering" on Thursdays (see Ceph community calendar). Thanks, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

v20.2.1 Tentacle released
by Yuri Weinstein 06 Apr '26

06 Apr '26

This is the first minor release in the Tentacle series. We recommend that all users update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-1-tentacle-released/ Notable Changes --------------- OSD / BlueStore --------------- * EC Recovery: Fixed a length calculation bug in erase_after_ro_offset() that caused empty shards to retain data, leading to shard_size >= tobj_size assertion failures when recovering small objects in EC pools. * BlueFS Volume Selector: Updated the BlueFS volume selector to properly account for file size changes when recovering the WAL in envelope mode. * BlueFS: Fixed a bug where stat() missed the actual file size update after indexing WAL envelope files. Monitor (mon) ------------- * Fast EC Restrictions: Denied the ability to enable EC optimizations ("fast EC") for non-4K-aligned chunk sizes. Unaligned chunk sizes handled by fast EC perform poorly and suffer from bugs, so attempts to force this configuration are now rejected. * Peering: Ensured ceph pg repeer proposes a correctly sized pg temp, as optimized EC cannot cope with mismatched sizes. * NVMeoF Gateway: Added a new nvme-gw listeners command to display all existing listeners (including auto-listeners) inside a pool/group. * NVMeoF Failover: Overhauled the NVMeoF Gateway fast-failover logic. Beacon timeouts are now evaluated within prepare_beacon to support shorter intervals, and the mechanism for detecting monitor slowness was improved. librbd & rbd-mirror ------------------- * RBD: Introduced a new ``RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT`` policy for ``rbd_lock_acquire()``. This is a low-level interface intended to allow a peer to grab exclusive lock manually for short periods of time with other peers pausing their activity and waiting for the lock to be released rather than instantly aborting I/O and returning an error. It's possible to switch from ``RBD_LOCK_MODE_EXCLUSIVE`` to ``RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT`` policy and vice versa even if the lock is already held. Ceph Object Gateway (RGW) ------------------------- * Multi-Part Operations: Fixed conditional validation handling in MultiWrite, Delete, and MultiDelete workflows. mgr/dashboard ------------- * UI Navigation: Redesigned the main landing page; the "Dashboard" navigation item was renamed to "Overview" and uses a new carbonized productive card layout. * NVMeoF Management: Added the nvmeof get_subsystems CLI command, fixed JSON output indentation for NVMeoF CLI commands, and reverted the server_addr API parameter back to traddr for consistency. * Hosts View: Fixed a bug causing the IP addresses of hosts to be hidden on the Hosts page due to an issue with fact merging. * Forms & Modals: Standardized forms onto the Carbon Design System, including the pools form, service form, multi-site realm token export modal, delete zone modal, and password change forms. * Form Validation: Generalized form error handling and validations using a new cdValidate directive. mgr/cephadm ----------- * Monitoring Stack: Bumped the default container image versions for the monitoring stack: Prometheus to v3.6.0, Node-exporter to v1.9.1, Alertmanager to v0.28.1, and Grafana to v12.2.0. Security Changes ---------------- * Monitoring Stack Images: Updated Prometheus, Alertmanager, and Grafana container image versions, picking up upstream security and stability fixes. Configuration Changes --------------------- * ``bluefs_check_volume_selector_on_mount``: The previous bluefs_check_volume_selector_on_umount debug setting was renamed and repurposed. It now checks for volume selector inconsistencies on both mount and unmount phases. * ``mon_nvmeofgw_beacon_grace``: The default grace period before marking a gateway as failed has been reduced from 10 seconds to 7 seconds for faster failover. * ``nvmeof_mon_client_tick_period``: The default beacon tick interval has been lowered from 2 seconds to 1 second. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-1-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 6a49aff47758778a5f5951e731d437c317f72fb2

1 0