October 2026 - ceph-users

CephFS erasure-coded default data pool
by Eugen Block 05 Nov '26

05 Nov '26

Hi, just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community. Thanks Eugen

6 7

Adding a HDD "Tier" for low-perf storage
by Javier Charne 05 Nov '26

05 Nov '26

Hello! Since NVME drives become quite expensive, we are thinking about adding a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS) Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon 6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04 Total OSDs: 423 x 3.5 TB ~ (1.9 PB) We want to add 5 servers, each with: - 12 x 20 TB HDD, - 4 NVME 2TB for WAL+DB, - 2 SSD for operative system, - 256 GB RAM, - 24 CPU - 4 x 25 Gbps NIC interfaces. One pool with replica 3 and another with EC 3+2 Please feel free to share any comments or experiences you may have on this topic—they are more than welcome. Thanks in advance! Javier.-

4 5

Massive mon slow-ops when restarting mon daemons
by Alex Walender 02 Nov '26

02 Nov '26

Hey there! I’m currently in the process of upgrading the production cluster from v18.2.8 -> v19.2.6 via „ceph orch upgrade“. I tested this upgrade in our staging environment and everything went great. On our production environment I encounter some worrisome issues with the mon daemons. The cluster itself: 5x MON Daemons 5x MGR Daemons 1500x OSD - 2 larger Pools with each 16k in PG. … so a somewhat larger cluster. The orchestrator started re-deploying the MGR, which went fine. During the MON redeployment, the cluster went into warning with messages like „34846 slow ops. mon.mon1 has slow ops“. The counter climbed rapidly. Only way to stop this was to restart the MON in question. Sometimes I had to restart the MON multiple times. Afterwards the cluster is healthy again….until I restart any MON. The symptoms return until restarting fixes it again. I tried to dump the mon_ops_in_flight, but the command just hangs (probably because of the large error count). Filtering the journal for the issues: ``` journalctl -g "oldest is" --since "3 hours ago" -u ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c(a)mon.partnach <mailto:ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c@mon.partnach> Sep 14 09:45:05 partnach conmon[1735]: 2026-09-14T07:45:05.331+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18286 slow ops, oldest is log(1000 entries from seq 2 at 2026-08-26T04:51:47.819315+0000) Sep 14 09:45:10 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:10 partnach conmon[1735]: 2026-09-14T07:45:10.827+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:15 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:15 partnach conmon[1735]: 2026-09-14T07:45:15.924+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:21 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:21 partnach conmon[1735]: 2026-09-14T07:45:21.224+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:26 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:26 partnach conmon[1735]: 2026-09-14T07:45:26.700+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:32 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:32 partnach conmon[1735]: 2026-09-14T07:45:32.432+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:37 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:37 partnach conmon[1735]: 2026-09-14T07:45:37.568+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:43 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) Sep 14 09:45:43 partnach conmon[1735]: 2026-09-14T07:45:43.292+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) ``` I’m not quite sure what the lock is here. The mon tries to store some old cluster log entries from various dates? Checked the underlying network fabric, CPU load, clock skew, compacted the rocksdb manually on each mon ect. All is fine. I also set "ceph config set global mon_cluster_log_level info“ **before** the 10th of September. Luckily the upgrade has finished with the MON daemons and is now re-deploying all OSDs. Does anyone know whats going on? Some kind of old hanging Log-Entries? Best Regards, Alex Walender --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

4 4

Umbrella 21.2.0 bad MGR memory leak
by Chris Palmer 02 Nov '26

02 Nov '26

I've just upgraded a small production light-usage cephadm cluster from Tentacle to Umbrella. The active MGR now seems to be growing by about 1GB per hour though. I can't see anything wrong in the mgr logs. The only other sign of trouble is that a lot of metrics are not displaying in Grafana (mainly OSD-related I think - haven't checked it thoroughly yet). I don't know if that is related to the leak. I just had to fail the MGR to get some memory back. It immediately released about 2.5GB, but is still tracking up gently while in standby. Help... (Just adding a large amount of swap...) Thanks, Chris

3 4

v21.2.0 Umbrella released
by Yuri Weinstein 29 Oct '26

29 Oct '26

Umbrella is the 21st stable release of Ceph. This is the first stable release of Ceph Umbrella. We recommend that all users update to this release. Warnings - Umbrella container images are based on Rocky Linux 10 (el10), which requires a CPU supporting the x86-64-v3 microarchitecture level. Deployments on hosts with older CPUs will not be able to run the containerized daemons. - Python 3.9 or newer is required on all hosts for cephadm support. Notable Changes RADOS / OSD / Erasure Coding - Optimized ("Fast") Erasure Coding: a substantial new EC I/O path. - EC Direct Reads - CLS support for EC reads - EC OMAP support - Add Fast EC CRC-checking for data and first parity - Numerous correctness fixes to the new path (truncate+write planning, PDW vs RMW selection, snapshot scrub handling, peering/recovery corner cases) Ships with health warnings for deprecated EC plugins/techniques and for Blaum-Roth EC profiles whose w+1 isn't prime; non-4k-aligned chunk sizes are now rejected for optimized EC. - Security: CephX authentication hardening -- new AES256KRB5 cipher (replacing weaker legacy key types as the default), online key rotation for mon/OSD/MDS daemons during upgrades, and fixes for CVE-2025-30156 (AES-CBC misuse enabling authentication bypass), CVE-2026-39944, CVE-2026-50152, and CVE-2026-54330 (https://github.com/ceph/ceph/pull/71172) - Monitor: new optional RocksDB backup/restore mechanism (mon_backup_interval, ceph tell mon.* backup, offline restore via ceph-mon --restore-backup). - New availability score feature for Monitor/OSD (tracking + health warnings + config toggle), off by default. - Stretch clusters: tiebreaker mon can now be optional, added a max bucket-weight-diff threshold, a health warning for colocated monitors, and a restriction on changing the election strategy after stretch mode is enabled. - mgr/DaemonServer: ok-to-upgrade now aggregates and globally sorts OSDs across CRUSH child buckets (fixing inaccurate per-chassis/rack ordering), and upgrades can now be scoped to a CRUSH bucket. - pg-upmap-primary support and improved pg-upmap computing speed. - PG autoscaler: default mon_target_pg_per_osd increased 100 -> 200 and mon_max_pg_per_osd increased 250 -> 500, letting the autoscaler compute more parallelism-friendly pg_num values and improving the balancer's ability to reach uniform OSD utilization. Existing pools may autoscale to more PGs after upgrading as a result; pg_autoscale_during_upgrade (https://github.com/ceph/ceph/pull/67694) paces that splitting so it doesn't disrupt the upgrade itself. Clusters with a smaller-than-4GiB effective bluestore_osd_memory_target, or SSDs split into more than two OSDs, may want to pin the old values before upgrading (https://github.com/ceph/ceph/pull/69452) - BlueStore: BlueFS Spillover Cleaner evolution, faster allocation recovery, static/runtime object-fragmentation tracking, full-device discard on mkfs now default, several spanning-blob/onode-cache-stall/reallocation-corruption fixes, and device-type-specific allocation policy. - mClock scheduler refactor. Default HDD OSD shard config changed as an interim fix for inconsistent client/recovery throughput with multiple shards: osd_op_num_shards_hdd 5 -> 1, osd_op_num_threads_per_shard_hdd 1 -> 5 ( https://tracker.ceph.com/issues/66289) - Crimson/Seastore: continued heavy investment -- erasure coding support landed in Crimson (including recovery), PG splitting, a new 2Q cache replacement algorithm, broad coroutine conversions, and a long tail of correctness/perf fixes across the object-data, LBA/omap-btree, and cleaner subsystems. RGW (RADOS Gateway) - Full object dedup -- tech preview. - Bucket logging -- new feature: journal-mode logging, EC-pool support, admin commands, and new per-bucket lifecycle/logging performance monitoring that closes a long-standing observability gap around lifecycle processing. - Cloud transition / cloud restore: restore from Glacier/tape, admin CLI for restore list/status, restore-event notifications, persistent restore state, per-bucket target options; lifecycle, cloud-restore, and cloud-transition processing converted to coroutines for a significant performance improvement. - Encryption: added AES-256-GCM SSE support; CopyObject now supported for encrypted objects. - IAM/Account: GetAccountSummary, account-wide PublicAccessBlock, Admin REST APIs for account quota, S3 Object Ownership (disable ACLs), legacy Keystone admin-token requirement removed, Keystone roles injected into IAM policy. - STS/OIDC: GetCallerIdentity, JWT/JWKS signature validation, OIDC multisite support. - SigV4: fixed a regression (present in 19.2.6/20.2.4) that rejected requests with an unsigned Content-Type, breaking presigned PUT uploads; x-amz-content-sha256 is no longer required in SignedHeaders for S3 requests ( https://github.com/ceph/ceph/pull/71363) - Multisite: numerous sync-correctness fixes (full-sync infinite loop, segfault during startup, forwarded awsv4/UNSIGNED-PAYLOAD requests, bucket-sync full-sync locking, zonegroup rename handling). - D4N (Redis-backed) cache: continued build-out (bucket check_empty, connection pooling/pipelining, dirty-entry handling, crash fixes). - s3-tests vendored directly into the repo (src/test/rgw/s3-tests); QA suites run them from there instead of an external branch reference. - Timestamp-based epochs for versioned objects; global CORS rule support; rate-limit response with Retry-After; standalone RGW build (without RadosStore) via a new config-store abstraction -- a significant architectural refactor of rgw_zone/realm-watching code. - Lua scripting: new postAuth hook, fresh VM per background script execution, package-management fixes. Dashboard - Large-scale UI migration to the IBM Carbon design system across most pages (RGW, pools, OSD flags, multisite, zone/zonegroup, services, upgrade page, and more). - NVMe/TCP (NVMeoF) management significantly expanded: multi-step subsystem/gateway/namespace creation wizards, TLS/mTLS, DHCHAP auth, KMIP support, consistency-group CRUD APIs, and many CLI/UI alignment fixes. - SMB cluster management support added. - RGW role management realigned with Carbon + API routing fixes; global RGW tenant "Roles" tab removed in favor of per-account roles. - CephFS mirroring wizard/entity management UI. - OAuth2 SSO support (config UI + backend prerequisites). - Angular upgraded to v19. cephadm / Orchestrator - Extended certificate management: cephadm now provides comprehensive certmgr integration across managed services including mgmt-gateway, monitoring stack, NFS, and RGW. Services support auto-generated and renewed cephadm-signed certificates, user-provided certificates, and certificate-store references. TLS PEM handling is now standardized with support for full certificate chains and automatic separation of certificates and private keys, simplifying certificate management across services. - SMB/Samba cluster support (the managed service type itself shipped in Tentacle): new remote-control client tool, per-share QoS/bandwidth rate limiting, custom ports, keybridge/FSCrypt-based share configuration, and support for CephFS shares served from an external Ceph cluster. - NVMe-oF gateway spec gained many fields (TLS, KMIP, Prometheus frequency, namespace limits, group assignment). - NFS-Ganesha: QoS support, active-active/keepalived-only ingress mode, TLS configuration support, RDMA handling during upgrades. - Crimson/seastore OSD deployment support added to cephadm and ceph-volume. - cephadm now supports CRUSH bucket-aware upgrades using the new ok-to-upgrade command, enabling faster and safe concurrent OSD upgrades while preserving data availability. CephFS / MDS - FSCrypt (client-side encryption) userspace implementation. - Multi-threaded cephfs-mirror, plus a blockdiff-based sync mechanism for regular-file transfers. - cephfs-mirror: new mirroring checkpoints (CLI + cluster-wide status tracking), mutable CephFS snapshot metadata (new libcephfs/pybind API), and substantially expanded per-directory/per-peer sync metrics. - New pybind/cephfs bindings: utimensat, chownat, symlinkat, readlinkat, statxat, mkdirat; non-recursive rmtree()/cptree(). - Subvolume group name-mangling support; new subvolume performance/utilization metrics; clone-source info retained in subvolume info. - Root-directory fragmentation disallowed; session-reclaim blocklisting fix; several shutdown-hang and scrub-pinning fixes. ceph-volume - Migrated internal config/sysinfo representation to dataclasses; device detection switched to udev data instead of shelling out to LVM/udevadm in several code paths (perf and reliability win). - Added seastore OSD support; fixed inventory failures on hosts with RAID-mirrored system volumes, cdrom devices, and missing /dev/vg/lv paths. Build / Packaging / QA Infrastructure - Ubuntu 24.04 (Noble) and Rocky Linux 10 added as supported distros across build and QA; CMake bumped to require 3.22.1 and gained CMake-4 support; C++ standard raised to C++23. - The Ceph container's base OS image has changed to rockylinux:10. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v21-2-0-umbrella-released/ Getting Ceph - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.2.0.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: b060042deb05d33e6cc358970cfb8221240bcf3e

6 10

Ceph talks at OpenInfra Days North America in Chicago
by Anthony Middleton 29 Oct '26

29 Oct '26

Hey Ceph friends, Our friends at OpenInfra are hosting OpenInfra Days North America in Chicago on Tuesday, November 17, the day after Ceph Days Chicago <https://ceph.io/en/community/events/2026/ceph-days-chicago/>. They’d love to have the Ceph community join them, and they’re offering a 25% discount with code: *Ceph25.* A few sessions that may interest the Ceph community: - Operating Ceph at Scale: Minimizing Recovery and Failure Impact on Production Workloads <https://openinfra-day-northamerica-2026.sessionize.com/session/1329050> - Beyond VMware: Engineering a Sovereign Cloud Using OpenStack & Ceph <https://openinfra-day-northamerica-2026.sessionize.com/session/1306356> - Closing the CephFS Trust Gap: Hypervisor-Mediated Storage in OpenStack <https://openinfra-day-northamerica-2026.sessionize.com/session/1315069> *Event page:* https://openinfra.org/days/northamerica *Schedule:* https://openinfra-day-northamerica-2026.sessionize.com/ *Registration:* https://linuxfoundation.regfox.com/openinfra-day-north-america Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

RBD: Enabling Mirror snapshots?
by Oliver Freyermuth 29 Oct '26

29 Oct '26

Dear Cephalopodians, long time no write — please bear with the painful question about an Octopus cluster (upgrade planned) for which we'd like to switch from RBD Mirroring via journaling to snapshot-based mirroring (both clusters involved are "latest" Octopus). The RBD volumes are used via libvirt/QEMU(KVM), using librbd. We have first switched the pool mode over to "image" mode: rbd mirror pool enable rbd image (we were using pool before) to be able to use the snapshot mode for individual images. Then, after disabling mirroring completely for an image: rbd mirror image disable rbd/myvm-disk1 which works fine and disabled the journaling and journal-based mirroring, we try to switch to mirror snapshots: rbd --debug-rbd=6 mirror image enable rbd/myvm-disk1 snapshot Sadly, this reveals the messages appended in full below, the critical part (only visible with debug level >=5) seems to be: handle_acquire_lock: unable to acquire exclusive lock Naturally, the running VMs are holding an exclusive lock on the images, and we are also doing regular "normal" snapshots daily. Are we correct to interpret this as follows? * Journal-mirroring can be toggled any time, no exclusive lock required. * Enabling snapshot-based mirroring requires an exclusive lock(?), i.e. shutting down the affected VM? Or is there a better way (breaking the lock seems unsafe)? Is this expected behaviour, or a bug (and / or is this different in later Ceph versions), or are we missing a step / some configuration, or just misinterpreting the output? It would of course be great operationally if we could get away without a VM shutdown to toggle this :-). Furthermore, if somebody has experience with mirror snapshots in general, any experience is welcome. For reference, we are right now using journal-based mirroring and in parallel performing daily (regular) snapshots with fstrim/fsfreeze before each snapshot (via the qemu guest agent) and backing these up with Benji, so we have two backup paths in place (mirroring plus Benji to a separate system). We plan to also perform (manual/scripted) mirror snapshots in parallel to regular snapshots to profit from the advantages of the trim+freeze+snapshot approach :-). Cheers, Oliver ------------- 2026-10-28T13:15:22.838+0100 7fafca678500 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 ImageRequestWQ: ictx=0x55720aba5e20 2026-10-28T13:15:22.849+0100 7fafa8ff9700 5 librbd::ImageCtx: 0x55720aba5e20: disabling zero-copy writes 2026-10-28T13:15:22.852+0100 7fafa8ff9700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 block_writes: 0x55720aba5e20, num=1 2026-10-28T13:15:22.852+0100 7faf9bfff700 5 librbd::cache::WriteAroundObjectDispatch: 0x7faf94006020 init: 2026-10-28T13:15:22.852+0100 7faf9bfff700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 register_object_dispatch: object_dispatch_layer=1 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 SimpleSchedulerObjectDispatch: ictx=0x55720aba5e20 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 init: 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 register_object_dispatch: object_dispatch_layer=4 2026-10-28T13:15:22.857+0100 7fafa8ff9700 5 librbd::Operations: 0x55720ab986c0 snap_create: snap_name=.mirror.primary.919fa522-7d50-49c6-8b9d-e7926ade404a.ef881760-f1cf-4fcf-8fe5-cf491359bf4a 2026-10-28T13:15:22.859+0100 7fafa8ff9700 5 librbd::managed_lock::AcquireRequest: 0x7faf94007a40 handle_break_lock: lock owner is still alive 2026-10-28T13:15:22.859+0100 7faf9bfff700 5 librbd::ManagedLock: 0x7faf8c0150b8 handle_acquire_lock: unable to acquire exclusive lock 2026-10-28T13:15:22.860+0100 7fafa8ff9700 5 librbd::Watcher: 0x7faf8c003030 notifications_blocked: blocked=0 2026-10-28T13:15:25.126+0100 7faf9bfff700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7faf8c01a600 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7faf9bfff700 -1 librbd::mirror::EnableRequest: 0x55720aa7aa30 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7fafca678500 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7fafca678500 5 librbd::Watcher: 0x7faf8c003030 block_notifies: blocked_count=1 2026-10-28T13:15:25.126+0100 7faf9bfff700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 shut_down: shut_down: in_flight=0 2026-10-28T13:15:25.126+0100 7faf9bfff700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 unblock_writes: 0x55720aba5e20, num=0 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::ObjectDispatch: 0x55720abe98a0 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::cache::WriteAroundObjectDispatch: 0x7faf94006020 shut_down: -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

2 8

October Ceph Foundation Board Meeting
by Anthony Middleton 28 Oct '26

28 Oct '26

Hey Ceph friends, The October Ceph Foundation Board meeting is now available on our YouTube channel. You can watch it here: https://youtu.be/H0yKb_5DAkQ. If you’re interested in contributing to the Ceph project, please feel free to reach out. I’d be happy to help you find a way to get involved. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Segmentation fault adding a disk
by Alfredo De Luca 28 Oct '26

28 Oct '26

Hi all. I have installed in a few RPI the microceph (ceph-version: 19.2.3-0ubuntu0.24.04.3; microceph-git: d95f07173f). Now 2 RPI are ubuntu based OS (Ubuntu 24.04.5 LTS) and 1 is with Debian 13 (Trixie). All works well except when I need to add a disk on debian which cause me the following segmentation fault: *# microceph disk add /dev/disk/by-id/nvme-eui.00000000000000000000004380201825 --wipe+-----------------------------------------------------------+---------+| PATH | STATUS |+-----------------------------------------------------------+---------+| /dev/disk/by-id/nvme-eui.00000000000000000000004380201825 | Failure |+-----------------------------------------------------------+---------+Error: failed to bootstrap OSD: Failed to run: ceph-osd --mkfs --no-mon-config -i 34: signal: segmentation fault* I can see the disk from the OS level of course *# nvme list * */dev/nvme0n1 /dev/ng0n1 ASBE44380201825 Acer SSD FA200 1TB 0x1 1.00 TB / 1.00 TB 512 B + 0 B SN15536 * # lsblk nvme0n1 259:0 0 931.5G 0 disk Any idea what the issue would be? Cheers -- */Alfredo*

1 1

Reflinks coming?
by Jay 27 Oct '26

27 Oct '26

I've been periodically checking on the reflink ticket[0] for a few years, and I just noticed that it's been updated with a target version of v22.0.0. Is that real? If so, is there anywhere I can read up on the implementation, plans, etc? 0 - <https://tracker.ceph.com/issues/1680> -- Jay ceph(a)tsuraan.net

3 3

20.2.4 Upgrade questions
by Jay 23 Oct '26

23 Oct '26

Looks like similar things are already being discussed on this list, but I'm not sure, so here goes. Q1 - I've updated all my service keys (mon, osd, etc) to the new type, and the "health detail" lines for AUTH_INSECURE_SERVICE_KEY_TYPE is gone, but after two days I still have the insecure rotating key messages: [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes I thought those would go away after a few hours? I stopped each process while I upgraded its keys, and I've since rebooted each host, but it looks like the old keys are still rotating or something? Is this a sign that something's gone wrong? Q2 - The guide [0] said it shouldn't be necessary to manually update the keyring files for the monitors. I assumed that meant the monitors would update their own keyrings, but they never did, so I did the import command for those as well. What was the expectation there? Should the keyrings have been rewritten, or do monitors now store their key info elsewhere? Q3 - OSD Lockbox? I saw the other thread and the linked script[1], so I think that answers my questions, but to be clear, a totally manual process would pretty much be: stop osd.{id} ceph auth rotate --key-type=aes256k client.osd-lockbox.{uuid} | tee keyring lvchange --addtag ceph.cephx_lockbox_secret={verbatim key from keyring} vg/lv start osd.{id} And that should pretty much do the trick? Starting the osd fires off ceph-volume activate, which rewrites the keyring in the osd tmpfs, so I don't think a person would need to do any manual file manipulation? Also a comment, I think the guide for updating OSDs is incorrect when the OSDs are dmcrypt. It says this: ceph-volume lvm list $OSD_ID --format json and find the lv_path field ceph-bluestore-tool --dev $DEV_PATH set-label-key --key osd_key -v $COPIED_KEYRING but when the OSD is encrypted, the lv_path refers to the encrypted block device, which doesn't have the bluestore metadata on it. What worked for me was the block_uuid field, and finding that entry under /dev/mapper, like this: # ceph-volume lvm list 0 --format json |grep '"ceph.block_uuid"' "ceph.block_uuid": "GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij", # ceph-bluestore-tool --dev /dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij show-label { "/dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij": { "osd_uuid": "013e9cce-66da-4b30-99a0-6e144665167e", ... Also the bit about set-label-key and $COPIED_KEYRING should be clear that it's going to write the value of $COPIED_KEYRING nto the bluestore metadata, so if you give it the value "keyring", then the stored ceph key will be "keyring". Grabbing the value from the "key = value" stanza of the keyring worked a lot better for me. 0 - <https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#upgrad…> 1 - <https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate/src/branch/main/ceph…> -- Jay ceph(a)tsuraan.net

4 8

umbrella 21.2.0 QE validation status
by Yuri Weinstein 23 Oct '26

23 Oct '26

Details of this release are summarized here: https://tracker.ceph.com/issues/76756#note-11 Release Notes - https://github.com/ceph/ceph/pull/70240 LRC upgrade - TBD Build 1 (umbrella-release 8ecd7c9aefe4cd9fecac47ea39487ed7700c1342): reruns of the failed/dead jobs are scheduled; some are still running. PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: Laura rados: Radek, Nitzan, Laura, Afreen (dashboard), Travis (rook) rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane, Guillaume rbd: Ilya krbd: Ilya nvmeof: PASSED upgrade: Laura, Patrick upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: PASSED upgrade-clients:client-upgrade-squid-umbrella-umbrella: PASSED perf-basic: PASSED crimson-rados: Matan powercycle: PASSED TIA

1 1

BlueFS spilling over to SLOW even when DB has a lot of free space
by Jeremi-Ernst Avenant 22 Oct '26

22 Oct '26

Good day We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions. However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left. We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why? Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again. We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs. Below are some of our servers and the DB used / SLOW used. Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

3 5

EC plugin ISA-L production ready?
by Eugen Block 22 Oct '26

22 Oct '26

Hi *, since Tentacle the new default EC plugin is ISA-L, I wanted to ask if anyone is using it in production (maybe for several years already)? I searched tracker for issues with the cauchy technique and found several (haven't read all of them), sometimes in combination with FastEC (which isn't safe to use yet from my understanding), but for example this doesn't mention the technique: https://tracker.ceph.com/issues/74943 and it's not backported to Tentacle yet. So my current impression is that ISA may not be fully production ready, is it? Would it be safer to stick to jerasure for now although it's not being maintained anymore? One specific EC profile I'm dealing with is k=5 m=7 which can be created with "isa + cauchy" or "jerasure + reed_sol_van" but is refused by "isa + reed_sol_van". I'd appreciate any comments! Thanks! Eugen

2 2

Ceph User / Dev Meetup Tomorrow!
by Anthony Middleton 20 Oct '26

20 Oct '26

Hey Ceph friends, The next Ceph User + Dev Meetup is tomorrow, Wednesday, Oct. 21, at 7:00 AM PDT / 14:00 UTC. The agenda is still open, so bring your questions and topics to help move the conversation forward. RSVP here: https://t.ly/Oct-Ceph-UersDev See you there! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Creating unfound objects on purpose
by Thomas Erbesdobler 19 Oct '26

19 Oct '26

Hi all, do you know about a reliable way to create unfound objects? I recently had a couple unfound objects on a production cluster after a group of osds crashed, and I would like to reproduce this state and experiment with it in my development environment. But therefore I first need to get unfound objects ;-). In all cases I've used ceph 19.2.3 with bluestore OSDs. What I tried so far: Attempt 1: * create a replicated pg with 3 shards, min_size=1 (same as in the production system at the time of the crash) * stop all but one osd, set norecover and nobackfill * write to a part of an object * start a second osd, wait until it boots and peers, stop the first osd * the object is indeed unfound, but I cannot run `ceph pg ... mark_unfound_lost ...` on it, since the osds who might have the object have not been queried * but when I destroy and recreate the first osd, the object becomes available again after recovery and reads the new content ~> recovery must have happened in some way Attempt 2: * same pg as in attempt 1 * stop all but one osd * write a lot to the pg (or rather its pool, that is, to create the data I've actually used the pool for rbd) * start one more osd of the pg and wait until recovery starts, then stop the first osd * the pg goes down, but no unfound objects have been created Basically I tried to follow the sequence sketched in https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#unf… . So, I guess it is more complicated to create unfound objects, maybe one requires to cause the "correct" osd crashes such that the pg's log somehow gets out of sync with the actual data stored on the osd? Any ideas would be welcome. Regards, Thomas

3 2

Oct Ceph Tech Talk
by Anthony Middleton 16 Oct '26

16 Oct '26

Hey Ceph friends, Just a reminder that our next Ceph Tech Talk is Monday, Oct 19th, at 12 pm EDT. The session will also feature four live demos covering Ceph RGW, Terraform/OpenTofu, S3 access, Kubernetes pod identity, and integration with VEXXHOST Inc.’s <https://www.linkedin.com/company/vexxhost/posts/?feedView=all> Sunrise OpenStack dashboard. RSVP and join the session here: https://www.linkedin.com/events/cephtechtalk-oidcwebidentityfor751324473649… Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

help me understand "slow heartbeats" warnings
by Janne Johansson 16 Oct '26

16 Oct '26

I see this at times, 2026-10-15T06:33:57.184296+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 3038.405ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:33:57.184319+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 3038.335ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:07.332951+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 2981.937ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:07.332975+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 2982.992ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:35.591255+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 2038.587ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:35.591280+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 2039.430ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:50.182523+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1284.844ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:50.182545+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1284.781ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:59.363750+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1228.156ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:59.363774+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1228.295ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:35:07.363735+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1220.582ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:35:07.363771+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1220.705ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:35:21.546156+0000 mon.mon-1 [INF] Health check cleared: OSD_SLOW_PING_TIME_BACK (was: Slow OSD heartbeats on back (longest 1220.582ms)) 2026-10-15T06:35:21.546210+0000 mon.mon-1 [INF] Health check cleared: OSD_SLOW_PING_TIME_FRONT (was: Slow OSD heartbeats on front (longest 1220.705ms) These lines are 10-15-20 seconds apart, and in that time span, the "longest" is lowering by .. a few hundred milliseconds. So over the course of two minutes something has gone from 3000 ms to under 1000. Now if this was a queue then it seems weird that it did not sink slower, assuming that the involved parts are working off items on the queue for minutes. If they were NOT working off the queue, one would assume it would grow by one second per second, like when you get 1 ops are blocked > 3276 sec on osd.21 and nothing is improving the situation, then that delay does grow by the second, totally reasonable. But the slow ops feels like a mystery to me, how it can be so specific in milliseconds while often working itself downwards so slowly that it takes literal minutes to get down below the warning limit. So what does this "Slow OSD heartbeats" actually measure at the time of these log lines? I often get them during expansion of the cluster or rolling OSD restarts, so its not necessarily even a strict error condition either. -- May the most significant bit of your life be positive.

2 4

S3 Select usage ahead of proposed deprecation
by Matthew H 16 Oct '26

16 Oct '26

Hi Team, PR #72337<https://github.com/ceph/ceph/pull/72337> proposes deprecating S3 Select in RGW and disabling it by default. Before we consider merging it, is anyone currently using this feature or aware of users who rely on it? If so, please share your use cases so we can understand the impact. Thanks!

2 2

Tentacle: rgw zonegroup modify failed: Module 'rgw' is not enabled
by Eugen Block 15 Oct '26

15 Oct '26

Hi all, today we stumbled across an rgw issue. This issue [0] (and PR [1]) seems to address it but to me it looks like not all cases are covered. - cephadm deployment - Ceph v20.2.4 - Ubuntu hosts (22.04) - no multi-site For a specific reason we gave a customer the instructions to pre-create a realm, zonegroup and zone before applying the rgw service. It worked well in previous versions (Squid < 19.2.6, haven't tested other versions yet). Now on 20.2.4 we try the same procedure (create realm, zonegroup and zone beforehand) with this rgw spec: service_type: rgw service_id: s3 service_name: s3 placement: label: rgw spec: ssl: true rgw_frontend_port: 7480 generate_cert: true wildcard_enabled: true zonegroup_hostnames: - s3.example.com What we didn't have in previous tests was the requirement to use a generated certificate (generate_cert). And we currently don't have our own certificates in this case (I know, we could just generate some and probably will). Anyway, applying this spec results in working RGW daemons (they successfully show ListAllMyBuckets) but this Ceph warning can't be cleared: [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): rgw.s3 rgw.s3: rgw zonegroup modify failed: Module 'rgw' is not enabled (required by command 'rgw zonegroup modify'): use `ceph mgr module enable rgw` to enable it retval: -95 Is the mentioned fix incomplete? Or how could we utilize generate_cert without issuing a health warning? I guess the simple workaround would be to create our own certs and use the same procedure as before. Thanks, Eugen [0] https://tracker.ceph.com/issues/75264 [1] https://github.com/ceph/ceph/pull/67598

1 0

Octopus PG not going active
by Michael Thomas 15 Oct '26

15 Oct '26

The background: We are preparing to migrate our Octopus cluster to Quincy (as a first step on the way to Tentacle). The primary use of the cluster is cephfs, with separate data pools on HDD and SSD. During the migration we accidentally reformatted 6 OSDs in a SSD 6+2 EC pool. Fortunately, all PGs recovered, except for one. Three of the lost OSDs contributed to this PG, so we are accepting that it is gone and any files that were placed on this PG are lost. We identified the lost files with cephfs-data-scan and found that all can be recovered from our non-ceph backup. The problem: Now that we've determined that all lost data can be recovered from backup, we'd like to get the PG active again so we can start this recovery. Unfortunately, the PG is stuck in the incomplete state. We've tried a few things to get it going again, some of which may have made matters worse: ceph osd reweight $osd 0.0 ceph osd lost $osd This left the PG blocked from peering due to 'history_les_bound'. After about an hour we tried: ceph tell $osd config set osd_find_best_info_ignore_history_les 1 This didn't get it peering again, so a couple of hours later we tried forcing a repeer: ceph pg repeer 8.7b That didn't get it going either. Over the next couple of days we then tried a combination of repeer and restarting OSDs. During this we tried doing a force-create on the pg: ceph osd force-create-pg 8.7b However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions? --Mike

3 5

Simultaneous increasing numbers of PGs and OSDs
by Loïc Tortay 14 Oct '26

14 Oct '26

Hello, For one of our CephFS clusters, we need to increase the number of PGs in the main data pool from 8192 to 16384. We have also received new nodes to increase the capacity of this cluster, an almost 50% increase in capacity/number of nodes. We plan to use "upmap-remapped.py" as described in https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ and https://docs.clyso.com/docs/kb/rados/pg-splitting-with-upmap-remapped/ Is is safe/OK to combine the two operations ? That is: - disable rebalance/the balancer - "apply" the OSDs YAML, the new hosts are already added to the cluster - run upmap-remapped.py a few times until the number of misplaced objects is low - increase the number of PGs in the pool - run upmap-remapped.py until "pg_num" has reached the new value - re-enable rebalance/the balancer Or is it safer/recommended to do this in two distinct operations ? Loïc. -- | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

2 1

Autoscaler second-guessing itself on number of PGs required for pool
by Martin Andersen 13 Oct '26

13 Oct '26

Hi all, We have a 19.2.4 cluster with a Cinder pool (autoscale=warn) that keeps asking for num_pgs to be adjusted. This was recently set to 2048 (from 4096), and over this weekend it wanted to increase it again to 4096 - this was also done. However, before the rebalance has even finished, it again informs that the num_pgs value is now too high, and asks for it to be reduced. This is within the span of under one hour. ceph health detail HEALTH_WARN 1 pools have too few placement groups [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool cinder-volumes has 2048 placement groups, should have 4096 ceph osd pool set cinder-volumes pg_num 4096 set pool 4 pg_num to 4096 ceph health detail HEALTH_WARN 1 pools have too many placement groups [WRN] POOL_TOO_MANY_PGS: 1 pools have too many placement groups Pool cinder-volumes has 4096 placement groups, should have 2048 Anyone else experiencing this behavior? We can obviously set the autoscaler to off for this pool, but there seems to either be a misconfiguration or errant default setting that should be addressed. This is a long-running pool that has had no recent config changes to pg assignment. Also, for reference - ceph config dump | grep pg global advanced mon_max_pg_per_osd 250 global advanced mon_pg_warn_max_object_skew 11.000000 global advanced mon_pg_warn_min_per_osd 10 global advanced mon_target_pg_per_osd 101 mgr advanced mgr/pg_autoscaler/log_level info --Martin

2 2

v21.1.2 Umbrella RC2 released
by Yuri Weinstein 12 Oct '26

12 Oct '26

This is the third release candidate for Umbrella. All community feedback is welcome. It is our intention that this is the final release candidate for Umbrella. If no blockers are identified, we plan to release Umbrella around October 21st. Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie) Containers are built on a Rocky 10 base image, available at: https://quay.io/repository/ceph/ceph?tab=tags Please note: the full draft release notes for v21.2.0 are available at https://github.com/ceph/ceph/pull/70240 Compatibility requirements for this release: - CPU: Umbrella containers are based on el10 and require a CPU supporting x86-64-v3. - Python: Python 3.9+ is required on all host operating systems for cephadm. Notable changes from 21.1.1, the second release candidate: Security / CephX: - auth/cephx: harden service ticket verification (pr#72126) - cephadm: rotate OSD/MDS keyrings at upgrade time and based on health warnings (pr#72240) - rgw: bind GCM object keys to the bucket marker (pr#71038) - mgr/rbd_support: make schedule phase offset calculation FIPS compliant (pr#71167) RADOS / OSD: - OSDMap::calc_pg_maps: fix balancing for some important corner cases (pr#71916) - osd/scheduler: classify subOp reads by op priority for mClock, and bound high-priority drain to prevent starvation (pr#69761, pr#72179) - mgr/osd/mon: remove pg_upmap_primary mappings for PG merge targets (pr#71050) - osd/PeeringState: don't clear the PG rebuild latch on same-primary interval restart (pr#70990) - Reverted the ProtocolV2 connection speedup and netsplit grace-period changes (pr#71957) RGW: - Compress/decompress objects during lifecycle transitions (pr#70687) - Multisite: redirect to the endpoint of the bucket's zonegroup (pr#71524) - Honor CopyObject tagging-directive (pr#70681); read user usage log in S3 GetUsage (pr#71102) - S3: reject an empty x-amz-server-side-encryption header (pr#71536); allow completing a multipart upload without SSE-C headers (pr#71841) - Fix conditional writes on versioning-suspended buckets (pr#72048) - Fix use-after-free in shard_io async_reads (pr#71136); infinite loop in role-list pagination (pr#71312) - radosgw-admin: 'account rm --purge-data' removes OIDC by account id (pr#71532); 'bi list' honors --max-entries (pr#71043) cephadm / Orchestrator: - certmgr: support fullchain PEM (pr#71513) - Allow removing OSDs stuck in error state (pr#71729); zap raw OSD devices on rm-cluster (pr#71730); "ok-to-stop" check for OSD redeploy/reconfig (pr#72032) - Fix staggered upgrade (pr#69724); hardware monitoring shows "NO DATA" for missing categories (pr#71725); node-proxy no longer crashes on empty Redfish logout (pr#71727) Dashboard: - Angular 20 upgrade and HTML linter fixes (pr#70871) - Skip RGW daemons whose frontend is unreachable (pr#72047) CephFS: - cephfs-mirror bug-fix backports (pr#70957) - Reverted "mds: enable osdc op throttles" (pr#72186) - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.2.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: b6f98ed07f44274e393138ffbae442b7a3f7ec84 - Distribution matrix: https://docs.ceph.com/en/latest/start/os-recommendations/

1 0

Ceph Community Survey 2026
by Anthony Middleton 12 Oct '26

12 Oct '26

Hey Ceph friends, It's Cephalopod Awareness Week, and we're taking a cue from the octopus. The Ceph Foundation needs to hear from every arm of the community. The 2026 Ceph Community Survey is now open and covers: - Foundation priorities for the next three years - What would make it easier to contribute to Ceph - Cephalocon and Ceph Days - How you use AI in your Ceph work An octopus’s eight arms sense and act independently but move it in one direction. The Ceph community works similarly: developers, operators, vendors, and users contribute distinct strengths, advancing the project together. Your answers shape how the Foundation allocates its resources, including the events we run and the tools we build. It takes about 10 minutes. Take the survey: https://t.ly/Ceph-Survey-2026 Open until: November 13th, 2026 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph> Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

v19.2.7 Squid released
by Yuri Weinstein 12 Oct '26

12 Oct '26

This is the seventh backport release in the Squid series. We recommend that all deployments update to this release. ## Notable Changes CephFS ------ - Fixed a crash caused by an invalid iterator in `_readdir_cache_cb`. - Fixed a deadlock in the client when the OSD is full. - Fixed client I/O stalling due to a buffer list exceeding `INT_MAX`. - Fixed client cache trimming removing dentries still attached to an open directory inode. - Fixed `join_fscid` being incorrectly reset for an active MDS during filesystem removal. - MDS no longer path-traverses a damaged dirfrag. - Fixed rank 0 being marked damaged if stopping fails after log flush and trim. - Fixed an incorrect snap check for directories with parent snaps. - Added a `ceph.dir.subvolume` get vxattr. - `mgr/snap_schedule` now handles volume deletion correctly. - Fixed `fs status` JSON output. - `mgr/vol` now handles a failed non-atomic operation correctly. RADOS ----- - Removed `pg_upmap_primary` mappings for PG merge targets. - Added `pg-upmap-primary` support to `clean_pg_upmaps`. - Fixed a Neorados `CephContext` leak. - Fixed a BlueStore onode-cache issue causing stalls. - Stretch mode: added a max bucket weight diff threshold. - `mgr` now handles `SIGTERM`/`SIGINT` in standby to avoid a spurious `CEPHADM_FAILED_DAEMON`. RBD (RADOS Block Device) ------------------------ - It's possible to specify the source cluster's `mon_host` and `key` for `native` format migration via the migration spec now. This eliminates the dependency on a `<cluster-name>.conf` file in a known location, which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Fixed a use-after-free in trash purge on image open error. - Fixed a use-after-free releasing object map locks during deep copy. - Fixed memory leaks in PWL cache discard operations. - Fixed a race between `dispatch_deferred_writes()` and `~AbstractWriteLog()` in the SSD PWL cache. - Fixed strict weak ordering in `rbd-mirror`'s `PeerSpec::operator<`. - `rbd-mirror` now prunes obsolete primary mirror snapshots after relocation. - Fixed `mgr/rbd_support` perf iostat pool-spec filtering by data pool. RGW (RADOS Gateway) ------------------- - Fixed conditional Delete, MultiDelete, and Put handling. - `x-amz-content-sha256` is no longer required in signed headers when verifying client auth. - The AWS v4 signer now always includes the `x-amz-content-sha256` header when signing. - `rgw_gc_max_objs` is now startup-only. - Added a warning against changing `rgw_lc_max_objs` once lifecycle is in use. - `x-amz-expiration` is now reported only for the current object version. - Multisite: bucket `obj_lock` settings are now synced. - Fixed a race between bucket reshard and bucket index update. - Fixed a crash on realm reload. - Fixed a frontend crash in `abort_early()` on client disconnect. - Fixed the lifecycle transition of encrypted multipart objects. - Content-Type is no longer always required to be present in the signature. - Fixed a shutdown hang caused by a race between the async processor and multisite sync threads. - Fixed cloud-tier multipart resume starting at part number 0. - `complete-multipart` now returns an ETag header on success. - Fixed a crash when an admin tries to assume a nonexistent role. ceph-volume ----------- - Fixed raw activation when the device path is stale. - Fixed OSD re-deployment issues with disk selection filters and DB devices. - Detects rotational media under dm-crypt for workqueue bypass. - Includes LVM mapper devices in `get_devices()`. - Retries `lvs` after an empty result or a "devices file is missing" error. - Skips virtual CD-ROM devices during inventory. Dashboard --------- - Fixed a pool profile issue. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v19-2-7-squid-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 9ec952b9f9668e9fa97e29a4c2b51e951674df2a

1 0

Ceph Days Chicago CFP Reminder
by Anthony Middleton 09 Oct '26

09 Oct '26

Hey Ceph Users, Just one more week to submit your talk(s) for Ceph Days Chicago 2026! Join us in Chicago to share your Ceph experience, technical insights, or community story with developers, operators, and enthusiasts at this in-person event. CFP deadline: Friday, Oct 15 Event date: Monday, Nov 16 Submit your talk here: https://t.ly/CDChicago26 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

v21.1.1 Umbrella (RC1) released
by Patrick Donnelly 09 Oct '26

09 Oct '26

This is the second release candidate for Umbrella. All community feedback is welcome. Packages built for: - Ubuntu 22.04 - Ubuntu 24.04 - CentOS 9 - Rocky Linux 10 - Debian 12 (Bookworm) - Debian 13 (Trixie) Containers are built on a Rocky 10 base image, available at: https://quay.io/repository/ceph/ceph?tab=tags Please note: the full draft release notes for v21.2.0 are available at https://github.com/ceph/ceph/pull/70240 Notable changes from 21.1.0, the first release candidate: - Fixes for CVE-2025-30156, CVE-2026-39944,CVE-2026-50152, and CVE-2026-54330 included. (Please see upgrade notes for v20.2.4/v19.2.6.) - MGR: The default values of ``mon_target_pg_per_osd`` and ``mon_max_pg_per_osd`` have been increased from 100 and 250 to 200 and 500, respectively. These values enhance the PG autoscaler's ability to calculate reasonable ``pg_num`` values for pools under its control. These in turn increase parallelism and performance, and improve the ``balancer`` module's ability to acheive uniform OSD utilization. [This concomitant PR](https://github.com/ceph/ceph/pull/67694) ensures that upgrades are not delayed or disrupted by PG splitting as a result of this change. Clusters with effective ``bluestore_osd_memory_target`` values smaller than the default 4GiB may wish to explicitly set the previous default values in the central config store before upgrading. This will reduce the potential for increased memory starvation. Clusters with SSDs split into more than two OSDs may also wish to pin their current values in advance, or redeploy those SSDs with at most two OSDs per. - RGW: In multisite deployments, zone endpoints configured in the zonegroup (e.g. ``https://zone-a.example.com``) can now be resolved to all IP addresses returned by DNS, with inter-zone traffic distributed across them using round-robin and per-IP health tracking. This enables DNS-based service discovery for inter-zone traffic without an external load balancer. This feature is disabled by default: to opt in, set ``rgw_rest_conn_connect_to_resolved_ips = true``. The per-IP retry-after-failure timeout is controlled by ``rgw_rest_conn_ip_fail_timeout_secs`` (default: 2 seconds). Deployments that work around libcurl's single-address behavior by repeating the same endpoint multiple times in zone configuration should review that setup before enabling, as round-robin distribution makes such duplication unnecessary. - RGW: The rgw_gc_max_deferred and rgw_gc_max_deferred_entries_size options have been removed, as they did not do anything. - Updated the default for rgw_gc_max_queue_size to account for the extra space from removing rgw_gc_max_deferred_entries_size. - RGW: The default value of ``rgw_thread_pool_size`` has been reduced from 512 to 128. Benchmarks showed that the lower thread count improves throughput and reduces latency on NVMe and HDD based environments. Users with specific workload requirements can tune this value; see also ``rgw_max_concurrent_requests``. - CephFS Mirroring: The ``fs mirror peer status`` admin socket command reports additional per-directory sync metrics (sync mode, throughput, crawl and data-sync queue timing, bytes/files progress, and ETA). Output is grouped under ``metrics/<mirrored-dir-path>/peer/<peer-uuid>``. For more information, see https://tracker.ceph.com/issues/73453 - CephFS Mirroring: Per-directory snapshot sync progress is exposed as labeled perf counters in the ``cephfs_mirror_directory`` group (``counter dump`` on the mirror daemon admin socket, exportable via ``ceph-exporter``). Counters mirror ``fs mirror peer status`` fields (directory state, current sync, last sync, and snap summary) and are labeled by peer UUID and mirrored directory path. See https://tracker.ceph.com/issues/73457 - RADOS: PG autoscaler allows overlapping roots. Each root receives a PG target based on its OSDs, with OSDs shared across multiple roots contributing proportionally less to each root's allocation. - CephFS Mirroring: Introduced snapshot checkpoints feature that allows users to mark specific snapshots as important milestones and track their replication status to remote sites. Checkpoints automatically detect if a snapshot has already been synced and provide visibility into disaster recovery readiness. The feature includes four new CLI commands: ``ceph fs snapshot mirror checkpoint add`` to mark a snapshot, ``ceph fs snapshot mirror checkpoint now`` to checkpoint the latest snapshot, ``ceph fs snapshot mirror checkpoint ls`` to list all checkpoints with their status (created/complete/failed), and ``ceph fs snapshot mirror checkpoint remove`` to remove a checkpoint. Checkpoint metadata is persistent across daemon restarts. This feature is useful for compliance auditing, application consistency verification, and SLA tracking. For more information, see https://tracker.ceph.com/issues/73454 - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for ``native`` format migration via the migration spec now. This eliminates the dependency on ``<cluster-name>.conf`` file in a known location which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.1.1.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: 815bc6b6f9b4e0fd98daed552ec8e7123a0a8f84 - Distribution matrix: https://docs.ceph.com/en/latest/start/os-recommendations/ -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

5 7

alertmanager connecting to secure mail server?
by listy 09 Oct '26

09 Oct '26

Hi guys. Is there a way to add an external CA so ceph could use secure mail? I get: ... [WRN] ALERTS_SMTP_ERROR: unable to send alert email [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:998) ... many thanks, L.

1 0

Performance Related Issue
by Vishnu Bhaskar 09 Oct '26

09 Oct '26

Hi, I am currently working on building an *all-flash Ceph setup for high-performance workloads*, and I am testing the performance using *8K write I/O*. I have tested the *same hardware* with PetaSAN as well. Since PetaSAN is Ceph-based, I expected the performance to be relatively similar, but I am seeing a significant difference. Performance Observed With *PetaSAN*, I am getting approximately *120K IOPS* from the iSCSI workload. With my standalone *Ceph setup*, I am getting only around *30K IOPS* from the same type of workload. Test Environment - All-flash storage - 100 GbE network - Jumbo frames enabled on both setups - FIO workload running over an *iSCSI share* - 8K write workload - Same/similar hardware used for both PetaSAN and Ceph - Comparing the performance of PetaSAN vs. my Ceph deployment I also checked the OSD-level benchmark to make sure the OSD itself is not showing a significant performance difference. PetaSAN / OSD Benchmark For 4K: root@node3:~# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.054923286, "bytes_per_sec": 223730240.75799105, "iops": 54621.640810056408 } For 8K: root@node3:~# ceph tell osd.0 bench 12288000 8192 4194304 100 { "bytes_written": 12288000, "blocksize": 8192, "elapsed_sec": 0.028371557, "bytes_per_sec": 433109821.92482424, "iops": 52869.851309182646 } Standalone Ceph OSD Benchmark For 4K: root@c100node2:~# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.060264142, "bytes_per_sec": 203902347.10385489, "iops": 49780.846460902074 } For 8K: root@c100node2:~# ceph tell osd.0 bench 12288000 8192 4194304 100 { "bytes_written": 12288000, "blocksize": 8192, "elapsed_sec": 0.031226269, "bytes_per_sec": 393514832.01531374, "iops": 48036.478517494354 } Interestingly, the OSD-level benchmark results are fairly close between the two environments (~48–53K IOPS), but the *end-to-end iSCSI FIO performance differs significantly*, with PetaSAN reaching around *120K IOPS* while my Ceph setup reaches only around *30K IOPS*. Any suggestions on how to isolate where the additional ~90K IOPS difference is coming from would be very helpful Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

3 2

We dodged a bullet
by Stefan Kooman 08 Oct '26

08 Oct '26

Hi, Yeah, so that is a bit of a clickbait title, but I do think it's true. In the hope this might be useful for someone, or some cluster, I decided to write this down. So what happened? First some context: subject at hand is a Ceph pacific cluster that has been running fine for over nine years (Went through 12 -> 13 -> 14 -> 15 -> 16 with many point releases in between). All OSDs were recreated in 16.2.11 to make use of the RocksDB column families without the need to do resharding (which would have taken forever with the amount of OMAP we have). This cluster has been provisioned with a home build Ansible playbook based on the manual procedure [1], and is basically an automation of all these manual steps. For OSDs we used the "short form" variant [2]. The first issue we encountered was when we added new storage nodes / OSDs to the cluster. The deployment of the first storage node took longer than expected (PCIe NVMe devices that were in use for OS were not correctly filtered and turned up as eligible for OSD creation). After we fixed that we could succesfully proceed. However, the first OSD that was added hit the "out" timer (we did not have noout flag set) and was set "out" by the cluster. We issued the "ceph osd in osd.$id" command to get the OSD back in. The state for all new OSDs was "down" and "in" apart from this OSD, which was in state "down" and "out". This immediately resulted in a assert of the monitors (num_down_in_osds <= num_in_osds). Different monitors crashed in quick succession but the quorum was never lost, and three mons (out of five) were always online. It also did not impact production traffic as now data was stored on them yet. The "ceph osd in osd.$id" that was hanging (due to crashing monitors) was aborted and shortly after that the monitors stopped crashing. The storage node got rebooted and the OSDs were all "up" and "in", apart from one. But this time we could successfully set the osd "in" ("up" and "in" state). We tried to reproduce this in a (virtual) test cluster that is as close a match to production but did not manage to reproduce. We figured this might have been a race condition of some kind (while the command was issued new OSDs might also have been added). The storage nodes were rebalanced and the new nodes were now also taking production workload. Somewhat later we decided to test a change (disable bypass workqueue flags of the encrypted (dmcrypt) devices) and shutdown the OSDs. Immediately after that several mons hit the same assert again. Although the OSDs had gracefully shut itself down ... because of the asserting mons no new osdmaps were created and the peer OSDs were not informed that there peers were down. Thus all involved OSDs had to figure this out themselves which they directly did ... but of course this did result in a lot of slow OSD operations. The OSDs were quickly started again and only until all OSDs were "up" and "in" again the monitors stopped crashing. The monitors did not loose quorum although we were of course afraid that they would. As it turned out this was likely not a race condition we started to look into this further. A search query on Ceph's issue tracker [3] resulted in a lot of hits. Turned out this issues has been hit by clusters provisioned by ROOK, manual procedure (ours) and potentially others and that the issue had been seen on clusters from 16.x, 17.x as well as 19.x and 20.x. This note gives a good summary [4]. It was fixed by this PR [5] and backported to Squid and Tentacle. But the reason *why* we hit this issue was not clear to us. Until we saw this report [6] wherein a user reported that their cluster had a lot of their OSDs with a "NEW" bit and when they would set an OSD "out" and "in" this bit would disappear and so did their issues (not hitting the assert anymore). We dumped the osdmap of the cluster and found out that *all* of our OSDs had the "NEW" bit active apart from one ... the one osd that had been through the OSD "out" / "in" state before. We only expected a "NEW" bit to be enabled for truely new OSDs, not OSDs that have been in the cluster for many years already. We decided not to wait until we would hit a scenario like the following: - one or more OSDs go down (broken or otherwise unable to start again) - have the monitors hit the assert - unable to get out of this situation easily as at that point you cannot issue any ceph commands as the monitors are asserting and the OSDs that went down cannot be brought back up. - hit a lot of slow ops in the meantime until the monitors would set the OSDs down ... or out eventually by itself (down out timer) which could have taken a long time in our case - have the risk of the monitors loosing the quorum and experience unavailability for an extended period of time. We backported the fix to the OSDMap.cc and created our own packages in case we would need them. After that we went ahead and set every OSD "out" and "in" to get rid of the NEW bit and that worked fine. The problem did not trigger anymore, as expected. Today I dived a bit deeper in the OSDMap.cc code responsible for this behavior ... turns out that there is a quicker way to fix this. Relevant code snippet: for (const auto &weight : inc.new_weight) { set_weight(weight.first, weight.second); // if we are marking in, clear the AUTOOUT and NEW bits, and clear // xinfo old_weight. if (weight.second) { osd_state[weight.first] &= ~(CEPH_OSD_AUTOOUT | CEPH_OSD_NEW); osd_xinfo[weight.first].old_weight = 0; } } If you perform a osd reweight, like "ceph osd reweight osd.0 1" it will remove the NEW bit. This is a way quicker process than "out" / "in". We added a reweight step to get rid of the "NEW" bit in our playbook if we would need to provision / replace OSDs again in the future. The reason why we hit this issue now, and not ever before, with the NEW bit OSDs in them and the bug still present is still unclear. There must be some other condition(s) that we hit this time and not before. I've checked our Cephadm based clusters and those don't have any NEW bits present. I'm not sure what steps Cephadm executes under the hood when it creates new OSDs but I guess this involves a step were it performs a reweight. For clusters out there that have been manually provisioned in the past, or created with Ansible playbooks that follow a similar procedure there might be OSDs present with this NEW bit that sooner or later have the risk of hitting this assert (until they run a version in which this is fixed). I advise you to check your clusters and get rid of these NEW bits because AFAIK their only purpose is to have this bit set when they are indeed, new, and they might harm your cluster(s) in the future. Questions we still have: - What other condition(s) are needed to trigger this assert - Would the manual install procedure need to include this reweight step, or is there another way to have the NEW bit cleared? Gr. Stefan [1]: https://docs.ceph.com/en/latest/install/manual-deployment [2]: https://docs.ceph.com/en/latest/install/manual-deployment/#short-form [3]: https://tracker.ceph.com/search?page=3&q=ceph_assert%28num_down_in_osds+%3C… [4]: https://tracker.ceph.com/issues/70869#note-9 [5]: https://github.com/ceph/ceph/pull/64009 [6]: https://tracker.ceph.com/issues/52535#note-12

3 2

Google Summer Internship recording
by Anthony Middleton 07 Oct '26

07 Oct '26

Hey Ceph friends, In case you missed it, here's the recording of yesterday's Google Summer of Code intern showcase: https://youtu.be/jUrze3sZMhA Project descriptions are on the event landing page: https://ceph.io/en/community/events/2026/GSoC2026/ Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

19.2.7 QE validation status
by Yuri Weinstein 06 Oct '26

06 Oct '26

Details of this release are summarized here: https://tracker.ceph.com/issues/78483#note-7 Release Notes (drafts) - https://github.com/ceph/ceph/pull/71539 https://github.com/ceph/ceph.io/pull/1198 LRC/SRC upgrade - TBD (Patrick,Dan / The Core Team) PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: Laura rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson, Eric fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/quincy-x: PASSED upgrade/reef-x: PASSED upgrade/telemetry-upgrade: PASSED upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: FAILED (suspect https://tracker.ceph.com/issues/79948) upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED powercycle: PASSED TIA

2 2

Ceph Developer Meetup Wednesday, Oct 7th, at 8 am PDT
by Anthony Middleton 06 Oct '26

06 Oct '26

Hey Ceph Devs, The next Ceph Developer Meetup is this Wednesday, Oct 7th, at 8 am PDT / 15:00 UTC. Add your discussion topics or general questions to the agenda here: https://tracker.ceph.com/projects/ceph/wiki/CDM_07_OCT_2026 Please reach out to Laura Flores with any questions about this session. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph Community: Google Summer of Code internet showcase
by Anthony Middleton 05 Oct '26

05 Oct '26

Happy Monday, everyone! Just a reminder that the Ceph Community: Google Summer of Code internet showcase is tomorrow, Oct 6th, at 8 am PDT/11 am EDT. During this session, we'll hear from 3 mentees who spent their summer tackling real-world Ceph problems: - RGW Admin UX & Docs — auto-generated radosgw-admin docs - Kafka Security — Kerberos/JWT/mTLS support for RGW's Kafka notifications - Ceph Dashboard — completing the Dashboard's move to Carbon Each mentee will walk through their project, the challenges they faced, how they overcame them, and what they'd tell the next GSoC contributor. 🕗 8:00 am PDT / 11:00 am EDT Join here: https://meet.jit.si/CephGSOC2026 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph OSD Jaeger tracing: errors and no end-to-end traces
by Kathirvel T 05 Oct '26

05 Oct '26

Hi Ceph community, We enabled the native Jaeger tracing on the OSDs of our Ceph 19.2.3 cluster to get distributed tracing. Issues: 1. Every OSD continuously logs this following error, [Error] File: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/src/recordable.cc:95[TRACE JAEGER Exporter] SetAttribute() failed, attribute type not supported 2. In Jaeger we only see two standalone spans per op (op requested/created and enqueue). They are not connected to each other or to any client-side trace. Questions 1. Is the SetAttribute() error a known issue, and is there a fix or workaround? 2. What is the intended use of these two spans? Is there a way to get full end-to-end traces in current releases? If anyone has faced this or knows a fix, please let us know. Thanks & Regards, Kathirvel T Associate R&D Engineer 9025610912 mailto:manoj.kumar@zybisys.com kathirvel.t(a)zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 2

Ceph Tech Talk Recording
by Anthony Middleton 01 Oct '26

01 Oct '26

Hey Ceph friends, Here is the recording of yesterday's Ceph Tech Talk: https://youtu.be/7lY-yJNkPhc Thank you to Anthony D'Atri for leading the talk, and to everyone who joined and contributed to the discussion. Our next tech talk is scheduled for *Monday, Oct 19th at 12 pm EDT* and will feature Tadas Sutkaitis of VEXXHOST. Tadas will present on *OIDC Web Identity for Ceph RGW*. More details to follow! Hope to see you there! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Ceph MDS issues!
by Erich Weiler 01 Oct '26

01 Oct '26

Hey All, I’m hoping someone can help with an issue I’m having… I have Ceph 19.2.3 on Rocky 9.3, and I have a CephFS filesystem. It’s been working for years OK, through several upgrades. Just today it went offline, with all MDS daemons (3 of them) stuck in “replay” forever. I tried restarting them, etc, but nothing helped. I tried manually failing ranks 2 and 3 to make it a simple one MDS daemon, then set "ceph fs set slugfs max_mds 1" and it seems to be stuck in “resolve” state after restarting that one daemon: [root@pr-md-01 ~]# ceph fs status slugfs - 30 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 resolve slugfs.pr-md-01.umxlxt 679k 557k 66.4k 0 1 failed 2 failed POOL TYPE USED AVAIL cephfs_metadata metadata 797G 3026G cephfs_md_and_data data 0 36.2T cephfs_data data 1449T 72.5T MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) [root@pr-md-01 ~]# cephadm enter --name mds.slugfs.pr-md-01.umxlxt -- ceph daemon /var/run/ceph/ceph-mds.slugfs.pr-md-01.umxlxt.asok status Inferring fsid 58bde08a-d7ed-11ee-9098-506b4b4da440 { "cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", "whoami": 0, "id": 19345540, "want_state": "up:resolve", "state": "up:resolve", "fs_name": "slugfs", "resolve_status": { "resolve_gather": "", "resolve_ack_gather": "" }, "rank_uptime": 1538.6363653870001, "mdsmap_epoch": 745780, "osdmap_epoch": 76803, "osdmap_epoch_barrier": 0, "uptime": 1538.823382218 } Ranks 1 and 2 are manually failed for now just to try to get things going again. I ran these commands to remove the “failed” ranks (which I probably should not have done, in retrospect): ceph mds fail slugfs:1 --yes-i-really-mean-it ceph mds fail slugfs:2 --yes-i-really-mean-it ceph mds rmfailed slugfs:1 --yes-i-really-mean-it ceph mds rmfailed slugfs:2 --yes-i-really-mean-it Any ideas how I can get the MDS back? I am up against a wall… I did dump the journals. I think I just need to get ranks 1 and 2 back online in addition to rank 0, then get the MDS's to move from "resolving" to "up"... Thanks for any advice!! cheers, erich

2 3

Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4
by Frank Ederveen 01 Oct '26

01 Oct '26

> If you upgrade to 20.2.4 the key rotation for the Ceph services will > be done by the orchestrator automatically. But only for the Ceph > services, no client keyrings will be touched by cephadm. So you can > safely upgrade and fix the CVEs but can still use clients with the aes > format keys. I have upgraded two cephadm managed test clusters from 20.2.1 to 20.2.4 with limited success on both. It allows new and old keys, and the default is to hand out new keys, but all of my clients (osd, rgw, admin, bootstrap, crash) are still using the insecure key. I ran the upgrade again with cephadm which did restart all the OSD, but the issue remains. All previous upgrades have gone very easily, this one.. not so much. Manually rotating and copying keys around is not an option for clusters with > 1k osd. Is there going to be a new release soon, or some scripting which fixes things? health: HEALTH_WARN 87 auth client entities with insecure key types Monitors are configured to allow auth using insecure key types Monitors are configured to allow creation of insecure key types ... [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 87 auth client entities with insecure key types entity client.admin using insecure key type: aes entity client.bootstrap-mds using insecure key type: aes entity client.bootstrap-mgr using insecure key type: aes entity client.bootstrap-osd using insecure key type: aes entity client.bootstrap-rbd using insecure key type: aes entity client.bootstrap-rbd-mirror using insecure key type: aes entity client.bootstrap-rgw using insecure key type: aes entity client.crash.node1-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node2-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node3-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node4-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node5-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node6-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node7-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.crash.node8-bceph1-int-nvan.dev-mydomain using insecure key type: aes entity client.osd-lockbox.02f209c1-ade4-492a-8a65-22fd21a71365 using insecure key type: aes entity client.osd-lockbox.0cae9ed5-2976-4047-9628-f90d53b3e577 using insecure key type: aes entity client.osd-lockbox.1656cd37-7cd9-4ade-a396-5d973c866711 using insecure key type: aes entity client.osd-lockbox.1d17a243-d4a5-4f7d-9aeb-88e14dc45812 using insecure key type: aes ... Thanks, Frank ________________________________ From: Eugen Block <ceph-users(a)ceph.io> Sent: 31 August 2026 07:44:50 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Clarification on Client Key Rotation During Upgrade to 19.2.6/20.2.4 If you upgrade to 20.2.4 the key rotation for the Ceph services will be done by the orchestrator automatically. But only for the Ceph services, no client keyrings will be touched by cephadm. So you can safely upgrade and fix the CVEs but can still use clients with the aes format keys. Zitat von Brent Kennedy <ceph-users(a)ceph.io>: > Question, sort of related. If upgrading from squid to tentacle, > will it automatically try to rotate keys are part of the cluster > upgrade or is that still a manual process? We would like to avoid > the key rotation for now as we have some proxmox nodes using ceph > storage in some of the clusters. > > Regards, > -Brent > > Existing Clusters: > US Production(HDD): Squid 19.2.3 Cephadm with 11 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > UK Production(HDD): Squid 19.2.3 Cephadm with 20 osd servers, 5 > mons, 4 rgw, 2 iscsigw, 2 mds > US Production(SSD): Squid 19.2.3 Cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > UK Production(SSD): Squid 19.2.3 cephadm with 6 osd servers, 5 mons, > 4 rgw, 2 mds > > > > -----Original Message----- > From: Patrick Donnelly <ceph-users(a)ceph.io> > Sent: Saturday, August 29, 2026 10:38 AM > To: Saif Mohammad <samdto987(a)gmail.com> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Clarification on Client Key Rotation > During Upgrade to 19.2.6/20.2.4 > > Hello, > > On Fri, Aug 28, 2026 at 10:20 PM Saif Mohammad <ceph-users(a)ceph.io> wrote: >> >> Hi, >> >> While upgrading to Squid 19.2.6 or Tentacle 20.2.4, since these >> versions introduce the secure key type, I have a clarification >> regarding client key rotation. >> For the client.admin key, we need to rotate the key and then >> import/update the keyring. >> For the other client keys, do we only need to rotate the key and >> verify that the key has changed? Is there no need to import/update >> the keyring for the other clients, since there is no local keyring >> available? > > Step (9) of [1] says: > > "Then copy and import the key to each machine using that client.$ID > credential." > > [1] > https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#cephx-… > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 6