October 2026 - ceph-users

S3 Select usage ahead of proposed deprecation
by Matthew H 16 Oct '26

16 Oct '26

Hi Team, PR #72337<https://github.com/ceph/ceph/pull/72337> proposes deprecating S3 Select in RGW and disabling it by default. Before we consider merging it, is anyone currently using this feature or aware of users who rely on it? If so, please share your use cases so we can understand the impact. Thanks!

2 2

umbrella 21.2.0 QE validation status
by Yuri Weinstein 15 Oct '26

15 Oct '26

Details of this release are summarized here: https://tracker.ceph.com/issues/76756#note-11 Release Notes - https://github.com/ceph/ceph/pull/70240 LRC upgrade - TBD Build 1 (umbrella-release 8ecd7c9aefe4cd9fecac47ea39487ed7700c1342): reruns of the failed/dead jobs are scheduled; some are still running. PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: Laura rados: Radek, Nitzan, Laura, Afreen (dashboard), Travis (rook) rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane, Guillaume rbd: Ilya krbd: Ilya nvmeof: PASSED upgrade: Laura, Patrick upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: PASSED upgrade-clients:client-upgrade-squid-umbrella-umbrella: PASSED perf-basic: PASSED crimson-rados: Matan powercycle: PASSED TIA

1 0

help me understand "slow heartbeats" warnings
by Janne Johansson 15 Oct '26

15 Oct '26

I see this at times, 2026-10-15T06:33:57.184296+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 3038.405ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:33:57.184319+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 3038.335ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:07.332951+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 2981.937ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:07.332975+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 2982.992ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:35.591255+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 2038.587ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:35.591280+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 2039.430ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:50.182523+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1284.844ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:50.182545+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1284.781ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:59.363750+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1228.156ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:59.363774+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1228.295ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:35:07.363735+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1220.582ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:35:07.363771+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1220.705ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:35:21.546156+0000 mon.mon-1 [INF] Health check cleared: OSD_SLOW_PING_TIME_BACK (was: Slow OSD heartbeats on back (longest 1220.582ms)) 2026-10-15T06:35:21.546210+0000 mon.mon-1 [INF] Health check cleared: OSD_SLOW_PING_TIME_FRONT (was: Slow OSD heartbeats on front (longest 1220.705ms) These lines are 10-15-20 seconds apart, and in that time span, the "longest" is lowering by .. a few hundred milliseconds. So over the course of two minutes something has gone from 3000 ms to under 1000. Now if this was a queue then it seems weird that it did not sink slower, assuming that the involved parts are working off items on the queue for minutes. If they were NOT working off the queue, one would assume it would grow by one second per second, like when you get 1 ops are blocked > 3276 sec on osd.21 and nothing is improving the situation, then that delay does grow by the second, totally reasonable. But the slow ops feels like a mystery to me, how it can be so specific in milliseconds while often working itself downwards so slowly that it takes literal minutes to get down below the warning limit. So what does this "Slow OSD heartbeats" actually measure at the time of these log lines? I often get them during expansion of the cluster or rolling OSD restarts, so its not necessarily even a strict error condition either. -- May the most significant bit of your life be positive.

2 3

Tentacle: rgw zonegroup modify failed: Module 'rgw' is not enabled
by Eugen Block 15 Oct '26

15 Oct '26

Hi all, today we stumbled across an rgw issue. This issue [0] (and PR [1]) seems to address it but to me it looks like not all cases are covered. - cephadm deployment - Ceph v20.2.4 - Ubuntu hosts (22.04) - no multi-site For a specific reason we gave a customer the instructions to pre-create a realm, zonegroup and zone before applying the rgw service. It worked well in previous versions (Squid < 19.2.6, haven't tested other versions yet). Now on 20.2.4 we try the same procedure (create realm, zonegroup and zone beforehand) with this rgw spec: service_type: rgw service_id: s3 service_name: s3 placement: label: rgw spec: ssl: true rgw_frontend_port: 7480 generate_cert: true wildcard_enabled: true zonegroup_hostnames: - s3.example.com What we didn't have in previous tests was the requirement to use a generated certificate (generate_cert). And we currently don't have our own certificates in this case (I know, we could just generate some and probably will). Anyway, applying this spec results in working RGW daemons (they successfully show ListAllMyBuckets) but this Ceph warning can't be cleared: [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): rgw.s3 rgw.s3: rgw zonegroup modify failed: Module 'rgw' is not enabled (required by command 'rgw zonegroup modify'): use `ceph mgr module enable rgw` to enable it retval: -95 Is the mentioned fix incomplete? Or how could we utilize generate_cert without issuing a health warning? I guess the simple workaround would be to create our own certs and use the same procedure as before. Thanks, Eugen [0] https://tracker.ceph.com/issues/75264 [1] https://github.com/ceph/ceph/pull/67598

1 0

Octopus PG not going active
by Michael Thomas 15 Oct '26

15 Oct '26

The background: We are preparing to migrate our Octopus cluster to Quincy (as a first step on the way to Tentacle). The primary use of the cluster is cephfs, with separate data pools on HDD and SSD. During the migration we accidentally reformatted 6 OSDs in a SSD 6+2 EC pool. Fortunately, all PGs recovered, except for one. Three of the lost OSDs contributed to this PG, so we are accepting that it is gone and any files that were placed on this PG are lost. We identified the lost files with cephfs-data-scan and found that all can be recovered from our non-ceph backup. The problem: Now that we've determined that all lost data can be recovered from backup, we'd like to get the PG active again so we can start this recovery. Unfortunately, the PG is stuck in the incomplete state. We've tried a few things to get it going again, some of which may have made matters worse: ceph osd reweight $osd 0.0 ceph osd lost $osd This left the PG blocked from peering due to 'history_les_bound'. After about an hour we tried: ceph tell $osd config set osd_find_best_info_ignore_history_les 1 This didn't get it peering again, so a couple of hours later we tried forcing a repeer: ceph pg repeer 8.7b That didn't get it going either. Over the next couple of days we then tried a combination of repeer and restarting OSDs. During this we tried doing a force-create on the pg: ceph osd force-create-pg 8.7b However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions? --Mike

3 5

Simultaneous increasing numbers of PGs and OSDs
by Loïc Tortay 14 Oct '26

14 Oct '26

Hello, For one of our CephFS clusters, we need to increase the number of PGs in the main data pool from 8192 to 16384. We have also received new nodes to increase the capacity of this cluster, an almost 50% increase in capacity/number of nodes. We plan to use "upmap-remapped.py" as described in https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ and https://docs.clyso.com/docs/kb/rados/pg-splitting-with-upmap-remapped/ Is is safe/OK to combine the two operations ? That is: - disable rebalance/the balancer - "apply" the OSDs YAML, the new hosts are already added to the cluster - run upmap-remapped.py a few times until the number of misplaced objects is low - increase the number of PGs in the pool - run upmap-remapped.py until "pg_num" has reached the new value - re-enable rebalance/the balancer Or is it safer/recommended to do this in two distinct operations ? Loïc. -- | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

2 1

Autoscaler second-guessing itself on number of PGs required for pool
by Martin Andersen 13 Oct '26

13 Oct '26

Hi all, We have a 19.2.4 cluster with a Cinder pool (autoscale=warn) that keeps asking for num_pgs to be adjusted. This was recently set to 2048 (from 4096), and over this weekend it wanted to increase it again to 4096 - this was also done. However, before the rebalance has even finished, it again informs that the num_pgs value is now too high, and asks for it to be reduced. This is within the span of under one hour. ceph health detail HEALTH_WARN 1 pools have too few placement groups [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool cinder-volumes has 2048 placement groups, should have 4096 ceph osd pool set cinder-volumes pg_num 4096 set pool 4 pg_num to 4096 ceph health detail HEALTH_WARN 1 pools have too many placement groups [WRN] POOL_TOO_MANY_PGS: 1 pools have too many placement groups Pool cinder-volumes has 4096 placement groups, should have 2048 Anyone else experiencing this behavior? We can obviously set the autoscaler to off for this pool, but there seems to either be a misconfiguration or errant default setting that should be addressed. This is a long-running pool that has had no recent config changes to pg assignment. Also, for reference - ceph config dump | grep pg global advanced mon_max_pg_per_osd 250 global advanced mon_pg_warn_max_object_skew 11.000000 global advanced mon_pg_warn_min_per_osd 10 global advanced mon_target_pg_per_osd 101 mgr advanced mgr/pg_autoscaler/log_level info --Martin

2 2

v21.1.2 Umbrella RC2 released
by Yuri Weinstein 12 Oct '26

12 Oct '26

This is the third release candidate for Umbrella. All community feedback is welcome. It is our intention that this is the final release candidate for Umbrella. If no blockers are identified, we plan to release Umbrella around October 21st. Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie) Containers are built on a Rocky 10 base image, available at: https://quay.io/repository/ceph/ceph?tab=tags Please note: the full draft release notes for v21.2.0 are available at https://github.com/ceph/ceph/pull/70240 Compatibility requirements for this release: - CPU: Umbrella containers are based on el10 and require a CPU supporting x86-64-v3. - Python: Python 3.9+ is required on all host operating systems for cephadm. Notable changes from 21.1.1, the second release candidate: Security / CephX: - auth/cephx: harden service ticket verification (pr#72126) - cephadm: rotate OSD/MDS keyrings at upgrade time and based on health warnings (pr#72240) - rgw: bind GCM object keys to the bucket marker (pr#71038) - mgr/rbd_support: make schedule phase offset calculation FIPS compliant (pr#71167) RADOS / OSD: - OSDMap::calc_pg_maps: fix balancing for some important corner cases (pr#71916) - osd/scheduler: classify subOp reads by op priority for mClock, and bound high-priority drain to prevent starvation (pr#69761, pr#72179) - mgr/osd/mon: remove pg_upmap_primary mappings for PG merge targets (pr#71050) - osd/PeeringState: don't clear the PG rebuild latch on same-primary interval restart (pr#70990) - Reverted the ProtocolV2 connection speedup and netsplit grace-period changes (pr#71957) RGW: - Compress/decompress objects during lifecycle transitions (pr#70687) - Multisite: redirect to the endpoint of the bucket's zonegroup (pr#71524) - Honor CopyObject tagging-directive (pr#70681); read user usage log in S3 GetUsage (pr#71102) - S3: reject an empty x-amz-server-side-encryption header (pr#71536); allow completing a multipart upload without SSE-C headers (pr#71841) - Fix conditional writes on versioning-suspended buckets (pr#72048) - Fix use-after-free in shard_io async_reads (pr#71136); infinite loop in role-list pagination (pr#71312) - radosgw-admin: 'account rm --purge-data' removes OIDC by account id (pr#71532); 'bi list' honors --max-entries (pr#71043) cephadm / Orchestrator: - certmgr: support fullchain PEM (pr#71513) - Allow removing OSDs stuck in error state (pr#71729); zap raw OSD devices on rm-cluster (pr#71730); "ok-to-stop" check for OSD redeploy/reconfig (pr#72032) - Fix staggered upgrade (pr#69724); hardware monitoring shows "NO DATA" for missing categories (pr#71725); node-proxy no longer crashes on empty Redfish logout (pr#71727) Dashboard: - Angular 20 upgrade and HTML linter fixes (pr#70871) - Skip RGW daemons whose frontend is unreachable (pr#72047) CephFS: - cephfs-mirror bug-fix backports (pr#70957) - Reverted "mds: enable osdc op throttles" (pr#72186) - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.2.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: b6f98ed07f44274e393138ffbae442b7a3f7ec84 - Distribution matrix: https://docs.ceph.com/en/latest/start/os-recommendations/

1 0

Ceph Community Survey 2026
by Anthony Middleton 12 Oct '26

12 Oct '26

Hey Ceph friends, It's Cephalopod Awareness Week, and we're taking a cue from the octopus. The Ceph Foundation needs to hear from every arm of the community. The 2026 Ceph Community Survey is now open and covers: - Foundation priorities for the next three years - What would make it easier to contribute to Ceph - Cephalocon and Ceph Days - How you use AI in your Ceph work An octopus’s eight arms sense and act independently but move it in one direction. The Ceph community works similarly: developers, operators, vendors, and users contribute distinct strengths, advancing the project together. Your answers shape how the Foundation allocates its resources, including the events we run and the tools we build. It takes about 10 minutes. Take the survey: https://t.ly/Ceph-Survey-2026 Open until: November 13th, 2026 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph> Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

v19.2.7 Squid released
by Yuri Weinstein 12 Oct '26

12 Oct '26

This is the seventh backport release in the Squid series. We recommend that all deployments update to this release. ## Notable Changes CephFS ------ - Fixed a crash caused by an invalid iterator in `_readdir_cache_cb`. - Fixed a deadlock in the client when the OSD is full. - Fixed client I/O stalling due to a buffer list exceeding `INT_MAX`. - Fixed client cache trimming removing dentries still attached to an open directory inode. - Fixed `join_fscid` being incorrectly reset for an active MDS during filesystem removal. - MDS no longer path-traverses a damaged dirfrag. - Fixed rank 0 being marked damaged if stopping fails after log flush and trim. - Fixed an incorrect snap check for directories with parent snaps. - Added a `ceph.dir.subvolume` get vxattr. - `mgr/snap_schedule` now handles volume deletion correctly. - Fixed `fs status` JSON output. - `mgr/vol` now handles a failed non-atomic operation correctly. RADOS ----- - Removed `pg_upmap_primary` mappings for PG merge targets. - Added `pg-upmap-primary` support to `clean_pg_upmaps`. - Fixed a Neorados `CephContext` leak. - Fixed a BlueStore onode-cache issue causing stalls. - Stretch mode: added a max bucket weight diff threshold. - `mgr` now handles `SIGTERM`/`SIGINT` in standby to avoid a spurious `CEPHADM_FAILED_DAEMON`. RBD (RADOS Block Device) ------------------------ - It's possible to specify the source cluster's `mon_host` and `key` for `native` format migration via the migration spec now. This eliminates the dependency on a `<cluster-name>.conf` file in a known location, which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Fixed a use-after-free in trash purge on image open error. - Fixed a use-after-free releasing object map locks during deep copy. - Fixed memory leaks in PWL cache discard operations. - Fixed a race between `dispatch_deferred_writes()` and `~AbstractWriteLog()` in the SSD PWL cache. - Fixed strict weak ordering in `rbd-mirror`'s `PeerSpec::operator<`. - `rbd-mirror` now prunes obsolete primary mirror snapshots after relocation. - Fixed `mgr/rbd_support` perf iostat pool-spec filtering by data pool. RGW (RADOS Gateway) ------------------- - Fixed conditional Delete, MultiDelete, and Put handling. - `x-amz-content-sha256` is no longer required in signed headers when verifying client auth. - The AWS v4 signer now always includes the `x-amz-content-sha256` header when signing. - `rgw_gc_max_objs` is now startup-only. - Added a warning against changing `rgw_lc_max_objs` once lifecycle is in use. - `x-amz-expiration` is now reported only for the current object version. - Multisite: bucket `obj_lock` settings are now synced. - Fixed a race between bucket reshard and bucket index update. - Fixed a crash on realm reload. - Fixed a frontend crash in `abort_early()` on client disconnect. - Fixed the lifecycle transition of encrypted multipart objects. - Content-Type is no longer always required to be present in the signature. - Fixed a shutdown hang caused by a race between the async processor and multisite sync threads. - Fixed cloud-tier multipart resume starting at part number 0. - `complete-multipart` now returns an ETag header on success. - Fixed a crash when an admin tries to assume a nonexistent role. ceph-volume ----------- - Fixed raw activation when the device path is stale. - Fixed OSD re-deployment issues with disk selection filters and DB devices. - Detects rotational media under dm-crypt for workqueue bypass. - Includes LVM mapper devices in `get_devices()`. - Retries `lvs` after an empty result or a "devices file is missing" error. - Skips virtual CD-ROM devices during inventory. Dashboard --------- - Fixed a pool profile issue. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v19-2-7-squid-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 9ec952b9f9668e9fa97e29a4c2b51e951674df2a

1 0

BlueFS spilling over to SLOW even when DB has a lot of free space
by Jeremi-Ernst Avenant 12 Oct '26

12 Oct '26

Good day We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions. However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left. We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why? Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again. We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs. Below are some of our servers and the DB used / SLOW used. Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

3 4

Ceph Days Chicago CFP Reminder
by Anthony Middleton 09 Oct '26

09 Oct '26

Hey Ceph Users, Just one more week to submit your talk(s) for Ceph Days Chicago 2026! Join us in Chicago to share your Ceph experience, technical insights, or community story with developers, operators, and enthusiasts at this in-person event. CFP deadline: Friday, Oct 15 Event date: Monday, Nov 16 Submit your talk here: https://t.ly/CDChicago26 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

v21.1.1 Umbrella (RC1) released
by Patrick Donnelly 09 Oct '26

09 Oct '26

This is the second release candidate for Umbrella. All community feedback is welcome. Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie) Containers are built on a Rocky 10 base image, available at: https://quay.io/repository/ceph/ceph?tab=tags Please note: the full draft release notes for v21.2.0 are available at https://github.com/ceph/ceph/pull/70240 Notable changes from 21.1.0, the first release candidate: - Fixes for CVE-2025-30156, CVE-2026-39944,CVE-2026-50152, and CVE-2026-54330 included. (Please see upgrade notes for v20.2.4/v19.2.6.) - MGR: The default values of ``mon_target_pg_per_osd`` and ``mon_max_pg_per_osd`` have been increased from 100 and 250 to 200 and 500, respectively. These values enhance the PG autoscaler's ability to calculate reasonable ``pg_num`` values for pools under its control. These in turn increase parallelism and performance, and improve the ``balancer`` module's ability to acheive uniform OSD utilization. [This concomitant PR](https://github.com/ceph/ceph/pull/67694) ensures that upgrades are not delayed or disrupted by PG splitting as a result of this change. Clusters with effective ``bluestore_osd_memory_target`` values smaller than the default 4GiB may wish to explicitly set the previous default values in the central config store before upgrading. This will reduce the potential for increased memory starvation. Clusters with SSDs split into more than two OSDs may also wish to pin their current values in advance, or redeploy those SSDs with at most two OSDs per. - RGW: In multisite deployments, zone endpoints configured in the zonegroup (e.g. ``https://zone-a.example.com``) can now be resolved to all IP addresses returned by DNS, with inter-zone traffic distributed across them using round-robin and per-IP health tracking. This enables DNS-based service discovery for inter-zone traffic without an external load balancer. This feature is disabled by default: to opt in, set ``rgw_rest_conn_connect_to_resolved_ips = true``. The per-IP retry-after-failure timeout is controlled by ``rgw_rest_conn_ip_fail_timeout_secs`` (default: 2 seconds). Deployments that work around libcurl's single-address behavior by repeating the same endpoint multiple times in zone configuration should review that setup before enabling, as round-robin distribution makes such duplication unnecessary. - RGW: The rgw_gc_max_deferred and rgw_gc_max_deferred_entries_size options have been removed, as they did not do anything. - Updated the default for rgw_gc_max_queue_size to account for the extra space from removing rgw_gc_max_deferred_entries_size. - RGW: The default value of ``rgw_thread_pool_size`` has been reduced from 512 to 128. Benchmarks showed that the lower thread count improves throughput and reduces latency on NVMe and HDD based environments. Users with specific workload requirements can tune this value; see also ``rgw_max_concurrent_requests``. - CephFS Mirroring: The ``fs mirror peer status`` admin socket command reports additional per-directory sync metrics (sync mode, throughput, crawl and data-sync queue timing, bytes/files progress, and ETA). Output is grouped under ``metrics/<mirrored-dir-path>/peer/<peer-uuid>``. For more information, see https://tracker.ceph.com/issues/73453 - CephFS Mirroring: Per-directory snapshot sync progress is exposed as labeled perf counters in the ``cephfs_mirror_directory`` group (``counter dump`` on the mirror daemon admin socket, exportable via ``ceph-exporter``). Counters mirror ``fs mirror peer status`` fields (directory state, current sync, last sync, and snap summary) and are labeled by peer UUID and mirrored directory path. See https://tracker.ceph.com/issues/73457 - RADOS: PG autoscaler allows overlapping roots. Each root receives a PG target based on its OSDs, with OSDs shared across multiple roots contributing proportionally less to each root's allocation. - CephFS Mirroring: Introduced snapshot checkpoints feature that allows users to mark specific snapshots as important milestones and track their replication status to remote sites. Checkpoints automatically detect if a snapshot has already been synced and provide visibility into disaster recovery readiness. The feature includes four new CLI commands: ``ceph fs snapshot mirror checkpoint add`` to mark a snapshot, ``ceph fs snapshot mirror checkpoint now`` to checkpoint the latest snapshot, ``ceph fs snapshot mirror checkpoint ls`` to list all checkpoints with their status (created/complete/failed), and ``ceph fs snapshot mirror checkpoint remove`` to remove a checkpoint. Checkpoint metadata is persistent across daemon restarts. This feature is useful for compliance auditing, application consistency verification, and SLA tracking. For more information, see https://tracker.ceph.com/issues/73454 - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.1.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: 815bc6b6f9b4e0fd98daed552ec8e7123a0a8f84 - Distribution matrix: https://docs.ceph.com/en/latest/start/os-recommendations/ -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

5 7

alertmanager connecting to secure mail server?
by listy 09 Oct '26

09 Oct '26

Hi guys. Is there a way to add an external CA so ceph could use secure mail? I get: ... [WRN] ALERTS_SMTP_ERROR: unable to send alert email [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:998) ... many thanks, L.

1 0

Performance Related Issue
by Vishnu Bhaskar 09 Oct '26

09 Oct '26

Hi, I am currently working on building an *all-flash Ceph setup for high-performance workloads*, and I am testing the performance using *8K write I/O*. I have tested the *same hardware* with PetaSAN as well. Since PetaSAN is Ceph-based, I expected the performance to be relatively similar, but I am seeing a significant difference. Performance Observed With *PetaSAN*, I am getting approximately *120K IOPS* from the iSCSI workload. With my standalone *Ceph setup*, I am getting only around *30K IOPS* from the same type of workload. Test Environment - All-flash storage - 100 GbE network - Jumbo frames enabled on both setups - FIO workload running over an *iSCSI share* - 8K write workload - Same/similar hardware used for both PetaSAN and Ceph - Comparing the performance of PetaSAN vs. my Ceph deployment I also checked the OSD-level benchmark to make sure the OSD itself is not showing a significant performance difference. PetaSAN / OSD Benchmark For 4K: root@node3:~# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.054923286, "bytes_per_sec": 223730240.75799105, "iops": 54621.640810056408 } For 8K: root@node3:~# ceph tell osd.0 bench 12288000 8192 4194304 100 { "bytes_written": 12288000, "blocksize": 8192, "elapsed_sec": 0.028371557, "bytes_per_sec": 433109821.92482424, "iops": 52869.851309182646 } Standalone Ceph OSD Benchmark For 4K: root@c100node2:~# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.060264142, "bytes_per_sec": 203902347.10385489, "iops": 49780.846460902074 } For 8K: root@c100node2:~# ceph tell osd.0 bench 12288000 8192 4194304 100 { "bytes_written": 12288000, "blocksize": 8192, "elapsed_sec": 0.031226269, "bytes_per_sec": 393514832.01531374, "iops": 48036.478517494354 } Interestingly, the OSD-level benchmark results are fairly close between the two environments (~48–53K IOPS), but the *end-to-end iSCSI FIO performance differs significantly*, with PetaSAN reaching around *120K IOPS* while my Ceph setup reaches only around *30K IOPS*. Any suggestions on how to isolate where the additional ~90K IOPS difference is coming from would be very helpful Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

3 2

We dodged a bullet
by Stefan Kooman 07 Oct '26

07 Oct '26

Hi, Yeah, so that is a bit of a clickbait title, but I do think it's true. In the hope this might be useful for someone, or some cluster, I decided to write this down. So what happened? First some context: subject at hand is a Ceph pacific cluster that has been running fine for over nine years (Went through 12 -> 13 -> 14 -> 15 -> 16 with many point releases in between). All OSDs were recreated in 16.2.11 to make use of the RocksDB column families without the need to do resharding (which would have taken forever with the amount of OMAP we have). This cluster has been provisioned with a home build Ansible playbook based on the manual procedure [1], and is basically an automation of all these manual steps. For OSDs we used the "short form" variant [2]. The first issue we encountered was when we added new storage nodes / OSDs to the cluster. The deployment of the first storage node took longer than expected (PCIe NVMe devices that were in use for OS were not correctly filtered and turned up as eligible for OSD creation). After we fixed that we could succesfully proceed. However, the first OSD that was added hit the "out" timer (we did not have noout flag set) and was set "out" by the cluster. We issued the "ceph osd in osd.$id" command to get the OSD back in. The state for all new OSDs was "down" and "in" apart from this OSD, which was in state "down" and "out". This immediately resulted in a assert of the monitors (num_down_in_osds <= num_in_osds). Different monitors crashed in quick succession but the quorum was never lost, and three mons (out of five) were always online. It also did not impact production traffic as now data was stored on them yet. The "ceph osd in osd.$id" that was hanging (due to crashing monitors) was aborted and shortly after that the monitors stopped crashing. The storage node got rebooted and the OSDs were all "up" and "in", apart from one. But this time we could successfully set the osd "in" ("up" and "in" state). We tried to reproduce this in a (virtual) test cluster that is as close a match to production but did not manage to reproduce. We figured this might have been a race condition of some kind (while the command was issued new OSDs might also have been added). The storage nodes were rebalanced and the new nodes were now also taking production workload. Somewhat later we decided to test a change (disable bypass workqueue flags of the encrypted (dmcrypt) devices) and shutdown the OSDs. Immediately after that several mons hit the same assert again. Although the OSDs had gracefully shut itself down ... because of the asserting mons no new osdmaps were created and the peer OSDs were not informed that there peers were down. Thus all involved OSDs had to figure this out themselves which they directly did ... but of course this did result in a lot of slow OSD operations. The OSDs were quickly started again and only until all OSDs were "up" and "in" again the monitors stopped crashing. The monitors did not loose quorum although we were of course afraid that they would. As it turned out this was likely not a race condition we started to look into this further. A search query on Ceph's issue tracker [3] resulted in a lot of hits. Turned out this issues has been hit by clusters provisioned by ROOK, manual procedure (ours) and potentially others and that the issue had been seen on clusters from 16.x, 17.x as well as 19.x and 20.x. This note gives a good summary [4]. It was fixed by this PR [5] and backported to Squid and Tentacle. But the reason *why* we hit this issue was not clear to us. Until we saw this report [6] wherein a user reported that their cluster had a lot of their OSDs with a "NEW" bit and when they would set an OSD "out" and "in" this bit would disappear and so did their issues (not hitting the assert anymore). We dumped the osdmap of the cluster and found out that *all* of our OSDs had the "NEW" bit active apart from one ... the one osd that had been through the OSD "out" / "in" state before. We only expected a "NEW" bit to be enabled for truely new OSDs, not OSDs that have been in the cluster for many years already. We decided not to wait until we would hit a scenario like the following: - one or more OSDs go down (broken or otherwise unable to start again) - have the monitors hit the assert - unable to get out of this situation easily as at that point you cannot issue any ceph commands as the monitors are asserting and the OSDs that went down cannot be brought back up. - hit a lot of slow ops in the meantime until the monitors would set the OSDs down ... or out eventually by itself (down out timer) which could have taken a long time in our case - have the risk of the monitors loosing the quorum and experience unavailability for an extended period of time. We backported the fix to the OSDMap.cc and created our own packages in case we would need them. After that we went ahead and set every OSD "out" and "in" to get rid of the NEW bit and that worked fine. The problem did not trigger anymore, as expected. Today I dived a bit deeper in the OSDMap.cc code responsible for this behavior ... turns out that there is a quicker way to fix this. Relevant code snippet: for (const auto &weight : inc.new_weight) { set_weight(weight.first, weight.second); // if we are marking in, clear the AUTOOUT and NEW bits, and clear // xinfo old_weight. if (weight.second) { osd_state[weight.first] &= ~(CEPH_OSD_AUTOOUT | CEPH_OSD_NEW); osd_xinfo[weight.first].old_weight = 0; } } If you perform a osd reweight, like "ceph osd reweight osd.0 1" it will remove the NEW bit. This is a way quicker process than "out" / "in". We added a reweight step to get rid of the "NEW" bit in our playbook if we would need to provision / replace OSDs again in the future. The reason why we hit this issue now, and not ever before, with the NEW bit OSDs in them and the bug still present is still unclear. There must be some other condition(s) that we hit this time and not before. I've checked our Cephadm based clusters and those don't have any NEW bits present. I'm not sure what steps Cephadm executes under the hood when it creates new OSDs but I guess this involves a step were it performs a reweight. For clusters out there that have been manually provisioned in the past, or created with Ansible playbooks that follow a similar procedure there might be OSDs present with this NEW bit that sooner or later have the risk of hitting this assert (until they run a version in which this is fixed). I advise you to check your clusters and get rid of these NEW bits because AFAIK their only purpose is to have this bit set when they are indeed, new, and they might harm your cluster(s) in the future. Questions we still have: - What other condition(s) are needed to trigger this assert - Would the manual install procedure need to include this reweight step, or is there another way to have the NEW bit cleared? Gr. Stefan [1]: https://docs.ceph.com/en/latest/install/manual-deployment [2]: https://docs.ceph.com/en/latest/install/manual-deployment/#short-form [3]: https://tracker.ceph.com/search?page=3&q=ceph_assert%28num_down_in_osds+%3C… [4]: https://tracker.ceph.com/issues/70869#note-9 [5]: https://github.com/ceph/ceph/pull/64009 [6]: https://tracker.ceph.com/issues/52535#note-12

3 2

Google Summer Internship recording
by Anthony Middleton 07 Oct '26

07 Oct '26

Hey Ceph friends, In case you missed it, here's the recording of yesterday's Google Summer of Code intern showcase: https://youtu.be/jUrze3sZMhA Project descriptions are on the event landing page: https://ceph.io/en/community/events/2026/GSoC2026/ Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Segmentation fault adding a disk
by Alfredo De Luca 07 Oct '26

07 Oct '26

Hi all. I have installed in a few RPI the microceph (ceph-version: 19.2.3-0ubuntu0.24.04.3; microceph-git: d95f07173f). Now 2 RPI are ubuntu based OS (Ubuntu 24.04.5 LTS) and 1 is with Debian 13 (Trixie). All works well except when I need to add a disk on debian which cause me the following segmentation fault: *# microceph disk add /dev/disk/by-id/nvme-eui.00000000000000000000004380201825 --wipe+-----------------------------------------------------------+---------+| PATH | STATUS |+-----------------------------------------------------------+---------+| /dev/disk/by-id/nvme-eui.00000000000000000000004380201825 | Failure |+-----------------------------------------------------------+---------+Error: failed to bootstrap OSD: Failed to run: ceph-osd --mkfs --no-mon-config -i 34: signal: segmentation fault* I can see the disk from the OS level of course *# nvme list * */dev/nvme0n1 /dev/ng0n1 ASBE44380201825 Acer SSD FA200 1TB 0x1 1.00 TB / 1.00 TB 512 B + 0 B SN15536 * # lsblk nvme0n1 259:0 0 931.5G 0 disk Any idea what the issue would be? Cheers -- */Alfredo*

1 0

19.2.7 QE validation status
by Yuri Weinstein 06 Oct '26

06 Oct '26

Details of this release are summarized here: https://tracker.ceph.com/issues/78483#note-7 Release Notes (drafts) - https://github.com/ceph/ceph/pull/71539 https://github.com/ceph/ceph.io/pull/1198 LRC/SRC upgrade - TBD (Patrick,Dan / The Core Team) PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: Laura rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson, Eric fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/quincy-x: PASSED upgrade/reef-x: PASSED upgrade/telemetry-upgrade: PASSED upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: FAILED (suspect https://tracker.ceph.com/issues/79948) upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED powercycle: PASSED TIA

2 2

Massive mon slow-ops when restarting mon daemons
by Alex Walender 06 Oct '26

06 Oct '26

Hey there! I’m currently in the process of upgrading the production cluster from v18.2.8 -> v19.2.6 via „ceph orch upgrade“. I tested this upgrade in our staging environment and everything went great. On our production environment I encounter some worrisome issues with the mon daemons. The cluster itself: 5x MON Daemons 5x MGR Daemons 1500x OSD - 2 larger Pools with each 16k in PG. … so a somewhat larger cluster. The orchestrator started re-deploying the MGR, which went fine. During the MON redeployment, the cluster went into warning with messages like „34846 slow ops. mon.mon1 has slow ops“. The counter climbed rapidly. Only way to stop this was to restart the MON in question. Sometimes I had to restart the MON multiple times. Afterwards the cluster is healthy again….until I restart any MON. The symptoms return until restarting fixes it again. I tried to dump the mon_ops_in_flight, but the command just hangs (probably because of the large error count). Filtering the journal for the issues: ``` journalctl -g "oldest is" --since "3 hours ago" -u ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c(a)mon.partnach <mailto:ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c@mon.partnach> Sep 14 09:45:05 partnach conmon[1735]: 2026-09-14T07:45:05.331+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18286 slow ops, oldest is log(1000 entries from seq 2 at 2026-08-26T04:51:47.819315+0000) Sep 14 09:45:10 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:10 partnach conmon[1735]: 2026-09-14T07:45:10.827+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:15 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:15 partnach conmon[1735]: 2026-09-14T07:45:15.924+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:21 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:21 partnach conmon[1735]: 2026-09-14T07:45:21.224+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:26 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:26 partnach conmon[1735]: 2026-09-14T07:45:26.700+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:32 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:32 partnach conmon[1735]: 2026-09-14T07:45:32.432+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:37 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:37 partnach conmon[1735]: 2026-09-14T07:45:37.568+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:43 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) Sep 14 09:45:43 partnach conmon[1735]: 2026-09-14T07:45:43.292+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) ``` I’m not quite sure what the lock is here. The mon tries to store some old cluster log entries from various dates? Checked the underlying network fabric, CPU load, clock skew, compacted the rocksdb manually on each mon ect. All is fine. I also set "ceph config set global mon_cluster_log_level info“ **before** the 10th of September. Luckily the upgrade has finished with the MON daemons and is now re-deploying all OSDs. Does anyone know whats going on? Some kind of old hanging Log-Entries? Best Regards, Alex Walender --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

3 2

20.2.4 Upgrade questions
by Jay 06 Oct '26

06 Oct '26

Looks like similar things are already being discussed on this list, but I'm not sure, so here goes. Q1 - I've updated all my service keys (mon, osd, etc) to the new type, and the "health detail" lines for AUTH_INSECURE_SERVICE_KEY_TYPE is gone, but after two days I still have the insecure rotating key messages: [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes I thought those would go away after a few hours? I stopped each process while I upgraded its keys, and I've since rebooted each host, but it looks like the old keys are still rotating or something? Is this a sign that something's gone wrong? Q2 - The guide [0] said it shouldn't be necessary to manually update the keyring files for the monitors. I assumed that meant the monitors would update their own keyrings, but they never did, so I did the import command for those as well. What was the expectation there? Should the keyrings have been rewritten, or do monitors now store their key info elsewhere? Q3 - OSD Lockbox? I saw the other thread and the linked script[1], so I think that answers my questions, but to be clear, a totally manual process would pretty much be: stop osd.{id} ceph auth rotate --key-type=aes256k client.osd-lockbox.{uuid} | tee keyring lvchange --addtag ceph.cephx_lockbox_secret={verbatim key from keyring} vg/lv start osd.{id} And that should pretty much do the trick? Starting the osd fires off ceph-volume activate, which rewrites the keyring in the osd tmpfs, so I don't think a person would need to do any manual file manipulation? Also a comment, I think the guide for updating OSDs is incorrect when the OSDs are dmcrypt. It says this: ceph-volume lvm list $OSD_ID --format json and find the lv_path field ceph-bluestore-tool --dev $DEV_PATH set-label-key --key osd_key -v $COPIED_KEYRING but when the OSD is encrypted, the lv_path refers to the encrypted block device, which doesn't have the bluestore metadata on it. What worked for me was the block_uuid field, and finding that entry under /dev/mapper, like this: # ceph-volume lvm list 0 --format json |grep '"ceph.block_uuid"' "ceph.block_uuid": "GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij", # ceph-bluestore-tool --dev /dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij show-label { "/dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij": { "osd_uuid": "013e9cce-66da-4b30-99a0-6e144665167e", ... Also the bit about set-label-key and $COPIED_KEYRING should be clear that it's going to write the value of $COPIED_KEYRING nto the bluestore metadata, so if you give it the value "keyring", then the stored ceph key will be "keyring". Grabbing the value from the "key = value" stanza of the keyring worked a lot better for me. 0 - <https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#upgrad…> 1 - <https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate/src/branch/main/ceph…> -- Jay ceph(a)tsuraan.net

3 6

Ceph Developer Meetup Wednesday, Oct 7th, at 8 am PDT
by Anthony Middleton 05 Oct '26

05 Oct '26

Hey Ceph Devs, The next Ceph Developer Meetup is this Wednesday, Oct 7th, at 8 am PDT / 15:00 UTC. Add your discussion topics or general questions to the agenda here: https://tracker.ceph.com/projects/ceph/wiki/CDM_07_OCT_2026 Please reach out to Laura Flores with any questions about this session. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph Community: Google Summer of Code internet showcase
by Anthony Middleton 05 Oct '26

05 Oct '26

Happy Monday, everyone! Just a reminder that the Ceph Community: Google Summer of Code internet showcase is tomorrow, Oct 6th, at 8 am PDT/11 am EDT. During this session, we'll hear from 3 mentees who spent their summer tackling real-world Ceph problems: - RGW Admin UX & Docs — auto-generated radosgw-admin docs - Kafka Security — Kerberos/JWT/mTLS support for RGW's Kafka notifications - Ceph Dashboard — completing the Dashboard's move to Carbon Each mentee will walk through their project, the challenges they faced, how they overcame them, and what they'd tell the next GSoC contributor. 🕗 8:00 am PDT / 11:00 am EDT Join here: https://meet.jit.si/CephGSOC2026 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph OSD Jaeger tracing: errors and no end-to-end traces
by Kathirvel T 05 Oct '26

05 Oct '26

Hi Ceph community, We enabled the native Jaeger tracing on the OSDs of our Ceph 19.2.3 cluster to get distributed tracing. Issues: 1. Every OSD continuously logs this following error, [Error] File: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/src/recordable.cc:95[TRACE JAEGER Exporter] SetAttribute() failed, attribute type not supported 2. In Jaeger we only see two standalone spans per op (op requested/created and enqueue). They are not connected to each other or to any client-side trace. Questions 1. Is the SetAttribute() error a known issue, and is there a fix or workaround? 2. What is the intended use of these two spans? Is there a way to get full end-to-end traces in current releases? If anyone has faced this or knows a fix, please let us know. Thanks & Regards, Kathirvel T Associate R&D Engineer 9025610912 mailto:manoj.kumar@zybisys.com kathirvel.t(a)zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 2

Ceph Tech Talk Recording
by Anthony Middleton 01 Oct '26

01 Oct '26

Hey Ceph friends, Here is the recording of yesterday's Ceph Tech Talk: https://youtu.be/7lY-yJNkPhc Thank you to Anthony D'Atri for leading the talk, and to everyone who joined and contributed to the discussion. Our next tech talk is scheduled for *Monday, Oct 19th at 12 pm EDT* and will feature Tadas Sutkaitis of VEXXHOST. Tadas will present on *OIDC Web Identity for Ceph RGW*. More details to follow! Hope to see you there! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph MDS issues!
by Erich Weiler 01 Oct '26

01 Oct '26

Hey All, I’m hoping someone can help with an issue I’m having… I have Ceph 19.2.3 on Rocky 9.3, and I have a CephFS filesystem. It’s been working for years OK, through several upgrades. Just today it went offline, with all MDS daemons (3 of them) stuck in “replay” forever. I tried restarting them, etc, but nothing helped. I tried manually failing ranks 2 and 3 to make it a simple one MDS daemon, then set "ceph fs set slugfs max_mds 1" and it seems to be stuck in “resolve” state after restarting that one daemon: [root@pr-md-01 ~]# ceph fs status slugfs - 30 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve slugfs.pr-md-01.umxlxt 679k 557k 66.4k 0 1 failed 2 failed POOL TYPE USED AVAIL cephfs_metadata metadata 797G 3026G cephfs_md_and_data data 0 36.2T cephfs_data data 1449T 72.5T MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@pr-md-01 ~]# cephadm enter --name mds.slugfs.pr-md-01.umxlxt -- ceph daemon /var/run/ceph/ceph-mds.slugfs.pr-md-01.umxlxt.asok status Inferring fsid 58bde08a-d7ed-11ee-9098-506b4b4da440 { "cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", "whoami": 0, "id": 19345540, "want_state": "up:resolve", "state": "up:resolve", "fs_name": "slugfs", "resolve_status": { "resolve_gather": "", "resolve_ack_gather": "" }, "rank_uptime": 1538.6363653870001, "mdsmap_epoch": 745780, "osdmap_epoch": 76803, "osdmap_epoch_barrier": 0, "uptime": 1538.823382218 } Ranks 1 and 2 are manually failed for now just to try to get things going again. I ran these commands to remove the “failed” ranks (which I probably should not have done, in retrospect): ceph mds fail slugfs:1 --yes-i-really-mean-it ceph mds fail slugfs:2 --yes-i-really-mean-it ceph mds rmfailed slugfs:1 --yes-i-really-mean-it ceph mds rmfailed slugfs:2 --yes-i-really-mean-it Any ideas how I can get the MDS back? I am up against a wall… I did dump the journals. I think I just need to get ranks 1 and 2 back online in addition to rank 0, then get the MDS's to move from "resolving" to "up"... Thanks for any advice!! cheers, erich

2 3

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Frank Ederveen 01 Oct '26

01 Oct '26

> If you upgrade to 20.2.4 the key rotation for the Ceph services will > be done by the orchestrator automatically. But only for the Ceph > services, no client keyrings will be touched by cephadm. So you can > safely upgrade and fix the CVEs but can still use clients with the aes > format keys. I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both. It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains. All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd. Is there going to be a new release soon, or some scripting which fixes things? health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ... Thanks, Frank ________________________________ From: Eugen Block <ceph-users(a)ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. Zitat von Brent Kennedy <ceph-users(a)ceph.io>: > Question, sort of related. If upgrading from squid to tentacle, > will it automatically try to rotate keys are part of the cluster > upgrade or is that still a manual process? We would like to avoid > the key rotation for now as we have some proxmox nodes using ceph > storage in some of the clusters. > > Regards, > -Brent > > Existing Clusters: > US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > > > > -----Original Message----- > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Saturday, August 29, 2026 10:38 AM > To: Saif Mohammad <samdto987(a)gmail.com> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Clarification on Client Key Rotation > During Upgrade to 19.2.6/20.2.4 > > Hello, > > On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: >> >> Hi, >> >> While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these >> versions introduce the secure key type, I have a clarification >> regarding client key rotation. >> For the client.admin key, we need to rotate the key and then >> import/update the keyring. >> For the other client keys, do we only need to rotate the key and >> verify that the key has changed? Is there no need to import/update >> the keyring for the other clients, since there is no local keyring >> available? > > Step (9) of [1] says: > > "Then copy and import the key to each machine using that client.$ID > credential." > > [1] > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 6