- ceph-users - lists.ceph.io

v20.2.1 Tentacle released
by Yuri Weinstein 06 Apr '26

06 Apr '26

This is the first minor release in the Tentacle series. We recommend that all users update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-1-tentacle-released/ Notable Changes --------------- OSD / BlueStore --------------- * EC Recovery: Fixed a length calculation bug in erase_after_ro_offset() that caused empty shards to retain data, leading to shard_size >= tobj_size assertion failures when recovering small objects in EC pools. * BlueFS Volume Selector: Updated the BlueFS volume selector to properly account for file size changes when recovering the WAL in envelope mode. * BlueFS: Fixed a bug where stat() missed the actual file size update after indexing WAL envelope files. Monitor (mon) ------------- * Fast EC Restrictions: Denied the ability to enable EC optimizations ("fast EC") for non-4K-aligned chunk sizes. Unaligned chunk sizes handled by fast EC perform poorly and suffer from bugs, so attempts to force this configuration are now rejected. * Peering: Ensured ceph pg repeer proposes a correctly sized pg temp, as optimized EC cannot cope with mismatched sizes. * NVMeoF Gateway: Added a new nvme-gw listeners command to display all existing listeners (including auto-listeners) inside a pool/group. * NVMeoF Failover: Overhauled the NVMeoF Gateway fast-failover logic. Beacon timeouts are now evaluated within prepare_beacon to support shorter intervals, and the mechanism for detecting monitor slowness was improved. librbd & rbd-mirror ------------------- * RBD: Introduced a new ``RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT`` policy for ``rbd_lock_acquire()``. This is a low-level interface intended to allow a peer to grab exclusive lock manually for short periods of time with other peers pausing their activity and waiting for the lock to be released rather than instantly aborting I/O and returning an error. It's possible to switch from ``RBD_LOCK_MODE_EXCLUSIVE`` to ``RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT`` policy and vice versa even if the lock is already held. Ceph Object Gateway (RGW) ------------------------- * Multi-Part Operations: Fixed conditional validation handling in MultiWrite, Delete, and MultiDelete workflows. mgr/dashboard ------------- * UI Navigation: Redesigned the main landing page; the "Dashboard" navigation item was renamed to "Overview" and uses a new carbonized productive card layout. * NVMeoF Management: Added the nvmeof get_subsystems CLI command, fixed JSON output indentation for NVMeoF CLI commands, and reverted the server_addr API parameter back to traddr for consistency. * Hosts View: Fixed a bug causing the IP addresses of hosts to be hidden on the Hosts page due to an issue with fact merging. * Forms & Modals: Standardized forms onto the Carbon Design System, including the pools form, service form, multi-site realm token export modal, delete zone modal, and password change forms. * Form Validation: Generalized form error handling and validations using a new cdValidate directive. mgr/cephadm ----------- * Monitoring Stack: Bumped the default container image versions for the monitoring stack: Prometheus to v3.6.0, Node-exporter to v1.9.1, Alertmanager to v0.28.1, and Grafana to v12.2.0. Security Changes ---------------- * Monitoring Stack Images: Updated Prometheus, Alertmanager, and Grafana container image versions, picking up upstream security and stability fixes. Configuration Changes --------------------- * ``bluefs_check_volume_selector_on_mount``: The previous bluefs_check_volume_selector_on_umount debug setting was renamed and repurposed. It now checks for volume selector inconsistencies on both mount and unmount phases. * ``mon_nvmeofgw_beacon_grace``: The default grace period before marking a gateway as failed has been reduced from 10 seconds to 7 seconds for faster failover. * ``nvmeof_mon_client_tick_period``: The default beacon tick interval has been lowered from 2 seconds to 1 second. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-1-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 6a49aff47758778a5f5951e731d437c317f72fb2

1 0

Re: ACME for the mgmt-gateway
by Redouane Kachach 06 Apr '26

06 Apr '26

Hi Matthias, Currently, the mgmt-gateway (and the rest of cephadm services) don't support ACME integration. Users must manually provide/update the external certificates. So as I see it it's not a "lack of documentation", in this case ACME is not a supported option. That said, I think this can be achieved using the current implementation without any extra code changes in cephadm. The key insight is that cephadm's certmgr already has all the necessary components: it monitors cert expiry, and when it detects a certificate has changed in the store, it automatically triggers a reconfig of the mgmt-gateway. So the only missing piece for externally-managed certs is the renewal trigger itself. For that, "ceph orch certmgr cert ls --filter-by status=expiring" can be used to detect expiring certificates and kick off the renewal process. The procedure can be: 1. Set certificate_source: reference in the mgmt-gateway spec. This tells certmgr the cert is externally owned and makes it accept pushes via "ceph orch certmgr cert set." 2. Run certbot (or your tool of choice) with DNS-01 (or HTTP) challenge which avoids any port-80 conflict with mgmt-gateway already owning the HTTPS endpoint. 3. A simple daily systemd timer runs a script that: - Queries ceph orch certmgr cert ls --filter-by "status=expiring" to check if renewal is needed - If yes, calls cerbot renew procedure - Pushes the new cert+key into certmgr via ceph "orch certmgr cert set / key set" - cephadm's serve loop detects the cert change on its next cycle and reconfigures mgmt-gateway automatically (no manual restart needed). 4. For security, the script should use a dedicated client keyring with caps restricted to only the two certmgr write commands. Best, Redouane. On Mon, Mar 30, 2026 at 2:27 PM Matthias Riße via ceph-users < ceph-users(a)ceph.io> wrote: > Hey all! > > I've just managed to get the mgmt-gateway working on our Ceph Cluster > with cephadm, which is quite cool to have as a single HA entrypoint. I > am surprised about the lack of documentation to get ACME-signed > certificates into / in front of the mgmt-gateway. > > Is anyone doing this already? If yes, what's your setup? I am wondering > if it is easier to put certbot, lego, or similar into a "custom > container service" and update the mgmt-gateway service with renewed > certs from there, or run an additional caddy instance (also as a custom > container service?) as a reverse proxy in front of the mgmt-gateway and > let it handle ACME. > > Kind regards > Matthias > > -- > > --------------------------------------------------------------------------------------------- > > --------------------------------------------------------------------------------------------- > Forschungszentrum Jülich GmbH > 52425 Jülich > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Dr. Stephanie Bauer (stellvertretende Vorsitzende), > Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers > > --------------------------------------------------------------------------------------------- > > --------------------------------------------------------------------------------------------- > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Anthony D'Atri 06 Apr '26

06 Apr '26

Are you using upmap-remapped to mitigate the thundering herd? And, if you're using the PG autoscaler, to mitigate the thundering herd of backfill that it adds? > On Apr 6, 2026, at 7:33 AM, Michal Strnad via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi. > > We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and another 1000 OSDs to be added. We are concerned with issues related to a large number of peering connections and remappings when a node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we see reflected in the ceph status output before the cluster finally stabilizes. > > In a classic (non‑stretch) Ceph cluster the convergence is noticeably quicker (inactive PGs appear only for a very short interval). In our stretch deployment, however, the window when PGs are inactive is considerably longer, consistent with our observations. The network latency between the two sites is consistently below 10 ms with no packet loss. > > We are currently trying to determine whether this impacts client‑side read/write simulations via S3, but this has not yet been verified. Our next step is to test with RBD, which should be more resilient and demonstrable. Naturally, we would like to control the process in a way that minimizes the impact on clients. Could you please confirm that the periods of PG inactivity are short enough to prevent client visible glitches, or even better, could you recommend an optimal protocol for adding new OSDs to a cluster of this size to ensure a smooth convergence? Do you have any tested or recommended timeout adjustments for the stretch configuration? > > Thank you for any feedback > -- > Michal > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Stretch Ceph cluster - prolonged inactive PGs after bulk OSD deployment
by Michal Strnad 06 Apr '26

06 Apr '26

Hi. We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and another 1000 OSDs to be added. We are concerned with issues related to a large number of peering connections and remappings when a node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we see reflected in the ceph status output before the cluster finally stabilizes. In a classic (non‑stretch) Ceph cluster the convergence is noticeably quicker (inactive PGs appear only for a very short interval). In our stretch deployment, however, the window when PGs are inactive is considerably longer, consistent with our observations. The network latency between the two sites is consistently below 10 ms with no packet loss. We are currently trying to determine whether this impacts client‑side read/write simulations via S3, but this has not yet been verified. Our next step is to test with RBD, which should be more resilient and demonstrable. Naturally, we would like to control the process in a way that minimizes the impact on clients. Could you please confirm that the periods of PG inactivity are short enough to prevent client visible glitches, or even better, could you recommend an optimal protocol for adding new OSDs to a cluster of this size to ensure a smooth convergence? Do you have any tested or recommended timeout adjustments for the stretch configuration? Thank you for any feedback -- Michal

1 0

Question Regarding Ceph Monitor Election Strategy Default Configuration
by Christian Mauricio Narvaez Acaro 06 Apr '26

06 Apr '26

Hi, We are currently setting up and executing a testbench to analyze and validate the resilience and robustness of the SEPATH hypervisor running across the three nodes of a cluster (SEAPATH - LF Energy Wiki<https://lf-energy.atlassian.net/wiki/spaces/SEAP/overview>). As part of these tests, we evaluated the behavior of the Ceph service in a simulated failure scenario. Using tc and netem, we introduced approximately 40% random packet loss on the cluster network of one node, which resulted in quorum instability. The main observations were as follows: * Continuous increase in election_epoch, indicating frequent monitor re-elections. * Intermittent quorum formation, with transitions between (1,2), (1,3), (2,3), and (1,2,3). * Repeated changes in quorum_leader_name, alternating mainly between node 1 and node 2. * A tendency for hypervisor1 (rank 0) to regain leadership whenever it restored connectivity with a majority of nodes. At the time of the test, we found that election_strategy was configured with value 1. Based on our understanding, this behavior is consistent with the classic election mechanism, where all monitors are treated equally and the monitor with the lowest rank is typically selected as leader. To further analyze this behavior, we compared the two monitor election strategies under the same induced failure condition. In both cases, we captured TCP traffic on port 3300 over a 10-minute period using tshark. With the classic strategy, repeated tests showed between 68 and 74 leader changes, continuous quorum leave-and-rejoin events for the affected node, and approximately double the traffic compared to the lossless scenario (636Kbits/s versus 340Kbits/s). By contrast, with the connectivity-based strategy, repeated tests showed only 1 to 4 leader changes, leadership limited to node 2 and node 3, no rejoin of the affected node to the quorum, and only an approximately 10% increase in TCP traffic compared to the lossless case (437 Kbits/s versus 400 Kbits/s) From a traffic perspective, this approach consumes less bandwidth and significantly improves system stability during cluster network failures, since node 1 is no longer repeatedly re-elected as leader while experiencing network issues. Based on the measurements we have collected and our comparison of the election algorithms, strategy 3 appears to be the most suitable option for our use case. We would therefore appreciate understanding whether there is any additional consideration or technical reason for preferring strategy 1 over strategy 3, which may explain why it is the default configuration. Thanks in advance. Best regards, Christian Narváez A.

1 0

Re: Cephadm woes on EVPN loopback network interfaces, detects wrong IPs
by Gregor Radtke 05 Apr '26

05 Apr '26

Martin, in case this is still an issue (apparently it is april already...): I am assuming here you have your /32 on lo and run IS-IS/openfabric/eBGP unnumbered on your eth[0-9]. In this case, i faced a similar issue a while back and, while i was not able to identify the actual root cause, my gut feeling told me that ceph would simply look which interface the default route would point to, and then take that interface's IP address. Which might be entirely wrong in setups like these... You might want to try putting the same /32 (same for /128 v6 if applicable) on both lo and your eth[0-9]. This probably feels entirely wrong (it does to me!), but i was successful in resolving this exact issue in this way. I came to think about this solution, as i vaguely remember the proxmox folks documenting a full-mesh / routed setup where the same IP + /24 (in their example) mask would be put on multiple interfaces at the same time. This might have had the same idea in mind, but i cannot find it anymore (and, just in case, shame on me, not them..). Be careful however, if you do eBGP unnumbered, your next-hops might get signalled w/ LLA + GLA as a result. This is what happened to me with PVE 9.1.5 (FRR 10.4.1) talking towards JunOS. JunOS would then erronuously pick the GLA as next-hop, resulting in the entire route becoming invalid. My interim fix is to set the next-hop to the peer address in the import policy, and carefully avoiding 24.4R1-S3 on MX (this release lets you set the next-hop to the LLA, but then still deems the next-hop to be unusable. Have not found a matching PR yet, but it works in e.g. 24.4R2-S2 and -S3). Cheers & happy Easter! Gregor P.S. (Configuring the same IP/32 on lo and eth[0-9] also helps with systemd-networksomething to behave more nicely, because it doesn't feel obligated to wait for connectivity on the eth interfaces _based on its weird (default) understanding of connectivity_. Can be fine-tuned with systemd-networkd options...) On 13.03.26 19:43, Martin Gerhard Loschwitz via ceph-users wrote: > Folks, > > I have a setup here where a server has a single /32 IP from 10.1.0.0/16. It has a perfectly normal IP address and is reachable from within the local network thanks to EVPN. cephadm however only detects nonsense: > > { > "fe80::/64": { > "data1": [ > "fe80::425b:7fff:fec8:c956" > ], > "loopback0": [ > "fe80::1c54:fbff:fe6a:51e7" > ] > } > } > > How can I set the networks the machine has manually so that it becomes usable for MON placement? At this time, ceph-mgr does not deploy additional MONs, for instance, because it claims that the host network does not match the public network, while I assume the host network is directly read from the „cephadm list-networks“ output. > > Thank you very much in advance > Martin >

1 0

Re: tentacle 20.2.1 QE validation status
by Yuri Weinstein 03 Apr '26

03 Apr '26

We have not done the Sepia LRC upgrade to 20.2.1 yet. Do we want to do it for this release? Neha, Patrick? On Wed, Apr 1, 2026 at 3:37 PM Laura Flores via dev <dev(a)ceph.io> wrote: > The only outstanding issue (https://tracker.ceph.com/issues/75570 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…>) > is not a blocker to the release. We found that it is due to an > overaggressive thrasher in the stress-split upgrade tests. > > More details of the analysis are on the ticket. > > Thanks, > Laura > > *Laura Flores* > Software Engineer; Ceph Storage > 1 708 738 8804 Mobile > She/Her/Hers > > IBM > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Wednesday, April 1, 2026 3:41 PM > *To:* Laura Flores <lflores(a)ibm.com> > *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io>; Adam King < > adking(a)redhat.com>; Sridhar Seshasayee <Sridhar.Seshasayee(a)ibm.com> > *Subject:* [EXTERNAL] Re: tentacle 20.2.1 QE validation status > > Laura, ack Thx for the update! On Wed, Apr 1, 2026 at 9: 45 AM Laura > Flores via dev <dev@ ceph. io> wrote: Hi Yuri, The orch team is still > investigating the issue that was found in the upgrade suite (https: > //tracker. ceph. com/issues/75570). > Laura, ack > Thx for the update! > > On Wed, Apr 1, 2026 at 9:45 AM Laura Flores via dev <dev(a)ceph.io> wrote: > > Hi Yuri, > > The orch team is still investigating the issue that was found in the > upgrade suite (https://tracker.ceph.com/issues/75570 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…>). > We are leaning toward it being a testing/environmental failure, but we > would like a bit more time to make a concrete consensus. Someone will try > to update for sure by the end of today. > > Thanks, > Laura > > *Laura Flores* > Software Engineer; Ceph Storage > 1 708 738 8804 Mobile > She/Her/Hers > > IBM > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Wednesday, April 1, 2026 10:12 AM > *To:* Yuri Weinstein <yweinste(a)redhat.com> > *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io>; Adam King < > adking(a)redhat.com>; Sridhar Seshasayee <Sridhar.Seshasayee(a)ibm.com> > *Subject:* [EXTERNAL] Re: tentacle 20.2.1 QE validation status > > We got all approvals and are ready to start building. Please speak up now > if anything else is needed. On Wed, Apr 1, 2026 at 7: 54 AM Yuri Weinstein > via dev <dev@ ceph. io> wrote: In a separate email Venky replied that > failures in orch is > We got all approvals and are ready to start building. > Please speak up now if anything else is needed. > > On Wed, Apr 1, 2026 at 7:54 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > In a separate email Venky replied that failures in orch is a known issue > > "This is a known issue and is tracked here > > https://tracker.ceph.com/issues/64502 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…> > > Redo ^^" > > Adam King, I'd like to assume orch is good to go. > Please confirm. > > On Tue, Mar 31, 2026 at 4:43 PM Yuri Weinstein via dev <dev(a)ceph.io> > wrote: > > We are still waiting for orch approval. > > On Tue, Mar 31, 2026 at 12:26 PM Adam Emerson <aemerson(a)redhat.com> wrote: > > On 22/03/2026, Yuri Weinstein via dev wrote: > > We have merged several important PRs and reran all suites. > > > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/74838#note-4 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Furldefense.proofpoi…> > > > > Seeking reviews and approval for all suites, please see runs on build 2 > > RGW Approved. > >

1 0

Re: Changing EC profile
by Rok Jaklič 03 Apr '26

03 Apr '26

On Fri, Apr 3, 2026 at 2:31 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > > > we expanded the cluster to 12 nodes with 28 osds > > 2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged > (Proxmox?) > 28 osds per node. > > > but we are still running on EC 3+2, since we started cluster with 6 > nodes. > > There are ways to fake failure domains (hosts, racks, etc) when you *know* > that a cluster will be expanding, albeit with increased risk until it does. > We use host failure domain for now. > > We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable > or will 8+3 take too much > > CPU/network capacities? > > That depends on several things. > > * Are you looking to change the profile to get more usable capacity? > * Are you looking to change the profile to get more durability / sustain > more failures? > Not necessarily, we would rather have a responsive rgw for example during failures, so when there is backfill/recovery going on that rgw does not stop responding. It happened a few times. e.g. slow/faulty disk from on which reweight was just set, was the reason that rgw stopped responding, since some queue filled up ... because ceph was unable to read from faulty pgs or smth... > * What is your hardware like? CPU, drive size/model, network links > There is a mix of: - Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 20Gb network bonds - AMD EPYC 9254 24-Core Processor with 50Gb network bonds with mostly old 6T Seagate ST6000NM0095 drives, but we are just in the process rotating them. > * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny > files? > RGW, S3 users, backup data, nextcloud storage or just storage for placing files. Mostly tiny files, however backups with >100GB or even TB files can be found also. usage: 775 TiB used, 1.0 PiB / 1.8 PiB avail * Which Ceph release are you running? > > Reef 18.2.8 Rok

2 1

Re: Changing EC profile
by Anthony D'Atri 03 Apr '26

03 Apr '26

> we expanded the cluster to 12 nodes with 28 osds 2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged (Proxmox?) > but we are still running on EC 3+2, since we started cluster with 6 nodes. There are ways to fake failure domains (hosts, racks, etc) when you *know* that a cluster will be expanding, albeit with increased risk until it does. > We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much > CPU/network capacities? That depends on several things. * Are you looking to change the profile to get more usable capacity? * Are you looking to change the profile to get more durability / sustain more failures? * What is your hardware like? CPU, drive size/model, network links * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny files? * Which Ceph release are you running? > Should I use isa/isa-l instead of jerasure? For new pools I would suggest that. The two are very similar, but ISA is maintained while jerasure is abandonware. > Can anyone share experience with 8+3? It goes to 11 ;) That said, props for having k+m < the number of failure domains. This gives you better RAS, and lets the size of those failure domains vary (somewhat) without stranding capacity. Having k+m=failure domains is a very common ... well I won't go so far as to call it a mistake, but let's say suboptimal strategy. I helped someone with 10 dense HDD nodes running 8+2. Adding one more (and later more) node made a big difference. If you are storing mostly very small files/ RGW objects, 8+3 will "waste" a high percentage of underlying storage. https://markhpc.github.io/2019/11/21/BlueFS-Space-Amplification.html If there are a few small files mixed in with much bigger ones, that becomes less of a concern. This space amplification drove the reduction of the default min_alloc_size from 64KB to 4KB in the O/P era. If you are running Tentacle and the pool has the ec optimization flag turned on, this space amp will be much reduced since RADOS will no longer add null padding. A wide EC profile will make scrubs and recovery slower due to the gridlock effect -- scrubs and recovery rely on reservations, and there will be more collision and thus queueing. A wide EC profile will result in slower writes. Your media and workload are factors in whether or not that's a problem for you. Whether your data is on CephFS, RGW, and/or RBD informs the actual process for moving. Please share more detail. There has been talk of being able to do this transparently with Umbrella, but it's hearsay until it's merged. > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Changing EC profile
by Rok Jaklič 03 Apr '26

03 Apr '26

Hi, we expanded the cluster to 12 nodes with 28 osds, but we are still running on EC 3+2, since we started cluster with 6 nodes. We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much CPU/network capacities? Should I use isa/isa-l instead of jerasure? Can anyone share experience with 8+3? Kind regards, Rok

1 0