- ceph-users - lists.ceph.io

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Anthony D'Atri 06 Apr '26

06 Apr '26

Are you using upmap-remapped to mitigate the thundering herd? And, if you're using the PG autoscaler, to mitigate the thundering herd of backfill that it adds? > On Apr 6, 2026, at 7:33 AM, Michal Strnad via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi. > > We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and another 1000 OSDs to be added. We are concerned with issues related to a large number of peering connections and remappings when a node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we see reflected in the ceph status output before the cluster finally stabilizes. > > In a classic (non‑stretch) Ceph cluster the convergence is noticeably quicker (inactive PGs appear only for a very short interval). In our stretch deployment, however, the window when PGs are inactive is considerably longer, consistent with our observations. The network latency between the two sites is consistently below 10 ms with no packet loss. > > We are currently trying to determine whether this impacts client‑side read/write simulations via S3, but this has not yet been verified. Our next step is to test with RBD, which should be more resilient and demonstrable. Naturally, we would like to control the process in a way that minimizes the impact on clients. Could you please confirm that the periods of PG inactivity are short enough to prevent client visible glitches, or even better, could you recommend an optimal protocol for adding new OSDs to a cluster of this size to ensure a smooth convergence? Do you have any tested or recommended timeout adjustments for the stretch configuration? > > Thank you for any feedback > -- > Michal > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Michal Strnad 06 Apr '26

06 Apr '26

Hi. We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and another 1000 OSDs to be added. We are concerned with issues related to a large number of peering connections and remappings when a node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we see reflected in the ceph status output before the cluster finally stabilizes. In a classic (non‑stretch) Ceph cluster the convergence is noticeably quicker (inactive PGs appear only for a very short interval). In our stretch deployment, however, the window when PGs are inactive is considerably longer, consistent with our observations. The network latency between the two sites is consistently below 10 ms with no packet loss. We are currently trying to determine whether this impacts client‑side read/write simulations via S3, but this has not yet been verified. Our next step is to test with RBD, which should be more resilient and demonstrable. Naturally, we would like to control the process in a way that minimizes the impact on clients. Could you please confirm that the periods of PG inactivity are short enough to prevent client visible glitches, or even better, could you recommend an optimal protocol for adding new OSDs to a cluster of this size to ensure a smooth convergence? Do you have any tested or recommended timeout adjustments for the stretch configuration? Thank you for any feedback -- Michal

1 0

Question Regarding Ceph Monitor Election Strategy Default Configuration
by Christian Mauricio Narvaez Acaro 06 Apr '26

06 Apr '26

Hi, We are currently setting up and executing a testbench to analyze and validate the resilience and robustness of the SEPATH hypervisor running across the three nodes of a cluster (SEAPATH - LF Energy Wiki<https://lf-energy.atlassian.net/wiki/spaces/SEAP/overview>). As part of these tests, we evaluated the behavior of the Ceph service in a simulated failure scenario. Using tc and netem, we introduced approximately 40% random packet loss on the cluster network of one node, which resulted in quorum instability. The main observations were as follows: * Continuous increase in election_epoch, indicating frequent monitor re-elections. * Intermittent quorum formation, with transitions between (1,2), (1,3), (2,3), and (1,2,3). * Repeated changes in quorum_leader_name, alternating mainly between node 1 and node 2. * A tendency for hypervisor1 (rank 0) to regain leadership whenever it restored connectivity with a majority of nodes. At the time of the test, we found that election_strategy was configured with value 1. Based on our understanding, this behavior is consistent with the classic election mechanism, where all monitors are treated equally and the monitor with the lowest rank is typically selected as leader. To further analyze this behavior, we compared the two monitor election strategies under the same induced failure condition. In both cases, we captured TCP traffic on port 3300 over a 10-minute period using tshark. With the classic strategy, repeated tests showed between 68 and 74 leader changes, continuous quorum leave-and-rejoin events for the affected node, and approximately double the traffic compared to the lossless scenario (636Kbits/s versus 340Kbits/s). By contrast, with the connectivity-based strategy, repeated tests showed only 1 to 4 leader changes, leadership limited to node 2 and node 3, no rejoin of the affected node to the quorum, and only an approximately 10% increase in TCP traffic compared to the lossless case (437 Kbits/s versus 400 Kbits/s) From a traffic perspective, this approach consumes less bandwidth and significantly improves system stability during cluster network failures, since node 1 is no longer repeatedly re-elected as leader while experiencing network issues. Based on the measurements we have collected and our comparison of the election algorithms, strategy 3 appears to be the most suitable option for our use case. We would therefore appreciate understanding whether there is any additional consideration or technical reason for preferring strategy 1 over strategy 3, which may explain why it is the default configuration. Thanks in advance. Best regards, Christian Narváez A.

1 0

Re: Cephadm woes on EVPN loopback network interfaces, detects wrong IPs
by Gregor Radtke 05 Apr '26

05 Apr '26

Martin, in case this is still an issue (apparently it is april already...): I am assuming here you have your /32 on lo and run IS-IS/openfabric/eBGP unnumbered on your eth[0-9]. In this case, i faced a similar issue a while back and, while i was not able to identify the actual root cause, my gut feeling told me that ceph would simply look which interface the default route would point to, and then take that interface's IP address. Which might be entirely wrong in setups like these... You might want to try putting the same /32 (same for /128 v6 if applicable) on both lo and your eth[0-9]. This probably feels entirely wrong (it does to me!), but i was successful in resolving this exact issue in this way. I came to think about this solution, as i vaguely remember the proxmox folks documenting a full-mesh / routed setup where the same IP + /24 (in their example) mask would be put on multiple interfaces at the same time. This might have had the same idea in mind, but i cannot find it anymore (and, just in case, shame on me, not them..). Be careful however, if you do eBGP unnumbered, your next-hops might get signalled w/ LLA + GLA as a result. This is what happened to me with PVE 9.1.5 (FRR 10.4.1) talking towards JunOS. JunOS would then erronuously pick the GLA as next-hop, resulting in the entire route becoming invalid. My interim fix is to set the next-hop to the peer address in the import policy, and carefully avoiding 24.4R1-S3 on MX (this release lets you set the next-hop to the LLA, but then still deems the next-hop to be unusable. Have not found a matching PR yet, but it works in e.g. 24.4R2-S2 and -S3). Cheers & happy Easter! Gregor P.S. (Configuring the same IP/32 on lo and eth[0-9] also helps with systemd-networksomething to behave more nicely, because it doesn't feel obligated to wait for connectivity on the eth interfaces _based on its weird (default) understanding of connectivity_. Can be fine-tuned with systemd-networkd options...) On 13.03.26 19:43, Martin Gerhard Loschwitz via ceph-users wrote: > Folks, > > I have a setup here where a server has a single /32 IP from 10.1.0.0/16. It has a perfectly normal IP address and is reachable from within the local network thanks to EVPN. cephadm however only detects nonsense: > > { > "fe80::/64": { > "data1": [ > "fe80::425b:7fff:fec8:c956" > ], > "loopback0": [ > "fe80::1c54:fbff:fe6a:51e7" > ] > } > } > > How can I set the networks the machine has manually so that it becomes usable for MON placement? At this time, ceph-mgr does not deploy additional MONs, for instance, because it claims that the host network does not match the public network, while I assume the host network is directly read from the „cephadm list-networks“ output. > > Thank you very much in advance > Martin >

1 0

Re: tentacle 20.2.1 QE validation status
by Yuri Weinstein 03 Apr '26

03 Apr '26

We have not done the Sepia LRC upgrade to 20.2.1 yet. Do we want to do it for this release? Neha, Patrick? On Wed, Apr 1, 2026 at 3:37 PM Laura Flores via dev <dev(a)ceph.io> wrote: > The only outstanding issue (https://tracker.ceph.com/issues/75570 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…>) > is not a blocker to the release. We found that it is due to an > overaggressive thrasher in the stress-split upgrade tests. > > More details of the analysis are on the ticket. > > Thanks, > Laura > > *Laura Flores* > Software Engineer; Ceph Storage > 1 708 738 8804 Mobile > She/Her/Hers > > IBM > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Wednesday, April 1, 2026 3:41 PM > *To:* Laura Flores <lflores(a)ibm.com> > *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io>; Adam King < > adking(a)redhat.com>; Sridhar Seshasayee <Sridhar.Seshasayee(a)ibm.com> > *Subject:* [EXTERNAL] Re: tentacle 20.2.1 QE validation status > > Laura, ack Thx for the update! On Wed, Apr 1, 2026 at 9: 45 AM Laura > Flores via dev <dev@ ceph. io> wrote: Hi Yuri, The orch team is still > investigating the issue that was found in the upgrade suite (https: > //tracker. ceph. com/issues/75570). > Laura, ack > Thx for the update! > > On Wed, Apr 1, 2026 at 9:45 AM Laura Flores via dev <dev(a)ceph.io> wrote: > > Hi Yuri, > > The orch team is still investigating the issue that was found in the > upgrade suite (https://tracker.ceph.com/issues/75570 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…>). > We are leaning toward it being a testing/environmental failure, but we > would like a bit more time to make a concrete consensus. Someone will try > to update for sure by the end of today. > > Thanks, > Laura > > *Laura Flores* > Software Engineer; Ceph Storage > 1 708 738 8804 Mobile > She/Her/Hers > > IBM > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Wednesday, April 1, 2026 10:12 AM > *To:* Yuri Weinstein <yweinste(a)redhat.com> > *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io>; Adam King < > adking(a)redhat.com>; Sridhar Seshasayee <Sridhar.Seshasayee(a)ibm.com> > *Subject:* [EXTERNAL] Re: tentacle 20.2.1 QE validation status > > We got all approvals and are ready to start building. Please speak up now > if anything else is needed. On Wed, Apr 1, 2026 at 7: 54 AM Yuri Weinstein > via dev <dev@ ceph. io> wrote: In a separate email Venky replied that > failures in orch is > We got all approvals and are ready to start building. > Please speak up now if anything else is needed. > > On Wed, Apr 1, 2026 at 7:54 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > In a separate email Venky replied that failures in orch is a known issue > > "This is a known issue and is tracked here > > https://tracker.ceph.com/issues/64502 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…> > > Redo ^^" > > Adam King, I'd like to assume orch is good to go. > Please confirm. > > On Tue, Mar 31, 2026 at 4:43 PM Yuri Weinstein via dev <dev(a)ceph.io> > wrote: > > We are still waiting for orch approval. > > On Tue, Mar 31, 2026 at 12:26 PM Adam Emerson <aemerson(a)redhat.com> wrote: > > On 22/03/2026, Yuri Weinstein via dev wrote: > > We have merged several important PRs and reran all suites. > > > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/74838#note-4 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…> > > > > Seeking reviews and approval for all suites, please see runs on build 2 > > RGW Approved. > >

1 0

Re: Changing EC profile
by Rok Jaklič 03 Apr '26

03 Apr '26

On Fri, Apr 3, 2026 at 2:31 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > > > we expanded the cluster to 12 nodes with 28 osds > > 2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged > (Proxmox?) > 28 osds per node. > > > but we are still running on EC 3+2, since we started cluster with 6 > nodes. > > There are ways to fake failure domains (hosts, racks, etc) when you *know* > that a cluster will be expanding, albeit with increased risk until it does. > We use host failure domain for now. > > We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable > or will 8+3 take too much > > CPU/network capacities? > > That depends on several things. > > * Are you looking to change the profile to get more usable capacity? > * Are you looking to change the profile to get more durability / sustain > more failures? > Not necessarily, we would rather have a responsive rgw for example during failures, so when there is backfill/recovery going on that rgw does not stop responding. It happened a few times. e.g. slow/faulty disk from on which reweight was just set, was the reason that rgw stopped responding, since some queue filled up ... because ceph was unable to read from faulty pgs or smth... > * What is your hardware like? CPU, drive size/model, network links > There is a mix of: - Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 20Gb network bonds - AMD EPYC 9254 24-Core Processor with 50Gb network bonds with mostly old 6T Seagate ST6000NM0095 drives, but we are just in the process rotating them. > * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny > files? > RGW, S3 users, backup data, nextcloud storage or just storage for placing files. Mostly tiny files, however backups with >100GB or even TB files can be found also. usage: 775 TiB used, 1.0 PiB / 1.8 PiB avail * Which Ceph release are you running? > > Reef 18.2.8 Rok

2 1

Re: Changing EC profile
by Anthony D'Atri 03 Apr '26

03 Apr '26

> we expanded the cluster to 12 nodes with 28 osds 2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged (Proxmox?) > but we are still running on EC 3+2, since we started cluster with 6 nodes. There are ways to fake failure domains (hosts, racks, etc) when you *know* that a cluster will be expanding, albeit with increased risk until it does. > We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much > CPU/network capacities? That depends on several things. * Are you looking to change the profile to get more usable capacity? * Are you looking to change the profile to get more durability / sustain more failures? * What is your hardware like? CPU, drive size/model, network links * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny files? * Which Ceph release are you running? > Should I use isa/isa-l instead of jerasure? For new pools I would suggest that. The two are very similar, but ISA is maintained while jerasure is abandonware. > Can anyone share experience with 8+3? It goes to 11 ;) That said, props for having k+m < the number of failure domains. This gives you better RAS, and lets the size of those failure domains vary (somewhat) without stranding capacity. Having k+m=failure domains is a very common ... well I won't go so far as to call it a mistake, but let's say suboptimal strategy. I helped someone with 10 dense HDD nodes running 8+2. Adding one more (and later more) node made a big difference. If you are storing mostly very small files/ RGW objects, 8+3 will "waste" a high percentage of underlying storage. https://markhpc.github.io/2019/11/21/BlueFS-Space-Amplification.html If there are a few small files mixed in with much bigger ones, that becomes less of a concern. This space amplification drove the reduction of the default min_alloc_size from 64KB to 4KB in the O/P era. If you are running Tentacle and the pool has the ec optimization flag turned on, this space amp will be much reduced since RADOS will no longer add null padding. A wide EC profile will make scrubs and recovery slower due to the gridlock effect -- scrubs and recovery rely on reservations, and there will be more collision and thus queueing. A wide EC profile will result in slower writes. Your media and workload are factors in whether or not that's a problem for you. Whether your data is on CephFS, RGW, and/or RBD informs the actual process for moving. Please share more detail. There has been talk of being able to do this transparently with Umbrella, but it's hearsay until it's merged. > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Changing EC profile
by Rok Jaklič 03 Apr '26

03 Apr '26

Hi, we expanded the cluster to 12 nodes with 28 osds, but we are still running on EC 3+2, since we started cluster with 6 nodes. We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much CPU/network capacities? Should I use isa/isa-l instead of jerasure? Can anyone share experience with 8+3? Kind regards, Rok

1 0

Re: Q1 2026 Ceph Foundation Newsletter
by Matthias Ferdinand 03 Apr '26

03 Apr '26

On Thu, Apr 02, 2026 at 09:01:04PM +0000, Anthony Middleton via ceph-users wrote: > Read the full newsletter here: https://t.ly/Q1-2026-Newsletter for users of w3m, curl, Tor-Browser: https://ceph.io/en/news/blog/2026/Q1-community-newsletter/ (avoids js-based redirects etc.)

1 0

Q1 2026 Ceph Foundation Newsletter
by Anthony Middleton 02 Apr '26

02 Apr '26

Hey Ceph friends, Hope you're doing well! The Q1 2026 Ceph Foundation Newsletter is out, and it’s packed with updates from the first quarter. We've made great strides, from new governance charters to cost-saving initiatives, all of which shape how Ceph is built and supported. Highlights include: - Outcomes from recent CSC and Foundation Board meetings - How OVHcloud savings are benefiting our community - Our new event plans beyond Cephalocon - Opportunities to get involved If you're interested in Ceph’s future and want to contribute, this is worth a read. Read the full newsletter here: https://t.ly/Q1-2026-Newsletter Feel free to reach out with any questions or concerns. Your feedback about the community is always appreciated. Best, Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0