- ceph-users - lists.ceph.io

Vote for the W Ceph release name!
by Patrick Donnelly 02 Nov '26

02 Nov '26

Please give your +1 here: https://pad.ceph.com/p/w -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

CSC minutes 2026-11-02
by Yaarit Hatuka 02 Nov '26

02 Nov '26

* Working groups and motions * should we adopt a new policy for LLM? * see: https://github.com/ceph/ceph/pull/72565 * moving https://tracker.ceph.com/issues/82128 to a vote. * this is a copy ofhttps://tracker.ceph.com/issues/79745 which did not pass due to lack of votes * other Voting Active motions: * https://tracker.ceph.com/issues/82119 * https://tracker.ceph.com/issues/82118 * https://tracker.ceph.com/issues/81735 - moved to a vote * PR: https://github.com/ceph/ceph/pull/72649 * W Name decision: https://pad.ceph.com/p/w * vote (by adding your +1) * share with the user,dev lists * Vampire dates * branching at the end of January * do not merge big changes right before branching * removing centos9, jammy from main * Squid last release * go through QA tickets * merge window closes November 11 (community calendar is up to date) * Umbrella v21.2.1 aligns with squid * X release names: [https://pad.ceph.com/p/x](https://pad.ceph.com/p/x) * add names * no need to +1 yet * [Joseph M] - (to be discussed next week) Draft PR created [https://github.com/ceph/ceph/pull/73371](https://github.com/ceph/ceph/pull/… (commit<https://github.com/ceph/ceph/pull/73371](https://github.com/ceph/ceph/pull/…> sha 8c048607addd8e23def31bca14b0559222f15d94) for automatic member removal policy * unfortunately on PTO to will miss meeting * Hoping to use this draft to get feedback and make tweaks before putting this to a vote asap to reduce election and governance overheads

1 0

v20.2.5 Tentacle released
by Yuri Weinstein 02 Nov '26

02 Nov '26

This is the fifth minor release in the Tentacle series. We recommend that all users update to this release. Notable Changes Security - CephX: hardened service ticket verification. RGW (RADOS Gateway) - Added AES-256-GCM support for server-side encryption. - Added support for a global CORS rule, and CORS rules are now matched the way Amazon S3 matches them. - Added radosgw-admin bucket suspend and unsuspend commands. - Added --shard-id support to radosgw-admin gc process and gc list. - radosgw-admin bilog trim now requires --yes-i-really-mean-it. - STS Lite is deprecated for Tentacle. - Fixed conditional writes on versioning-suspended buckets. - Fixed SigV4 signature handling so Content-Type is no longer always required in the signature and x-amz-content-sha256 need not be a signed header. - S3 GetUsage now reads the user usage log. - Added Keystone scope information to ops logging. - Fixed several multisite issues: bucket object-lock sync, bucket reshard racing with index updates, and bilog autotrim on archive zones and stale bucket instance metadata. RADOS / OSD / BlueStore - BlueStore: fixed unsharing blobs on remove, stray spanning blobs produced by ExtentMap::reshard, onode cache stalls, and expanded space being added to the allocator after allocmap recovery. - BlueStore: BlueFS spillover cleaner improvements. - Autoscaler: PG autoscaling is now allowed for overlapped CRUSH roots, and PG distribution is fixed. - Scrub now auto-corrects accounting-only stat mismatches. - mClock: bounded the high_priority queue drain to prevent starvation. - pg_upmap_primary mappings are now removed for PG merge targets. - The read balancer no longer aborts on float rounding. - Reverted the msg/async/ProtocolV2 connection-speedup and netsplit grace period change. CephFS - MDS now persists the session auth_name in the ESession journal event. - The client accounts for mixed quotas in statfs. - cephfs-journal-tool can now scan and flush the journal in batches. - Fixed ceph fs perf stats handling of the json-pretty output format. RBD - It's possible to specify the source cluster's mon_host and key for native format migration via the migration spec now. This eliminates the dependency on a <cluster-name>.conf file in a known location, which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Fixed a use-after-free in trash purge on image open error. - Fixed list_snaps() handling of non-existent groups in pybind/rbd. cephadm / NVMe-oF - Added a command to send signals to daemons. - Upgrades onto hosts whose CPU cannot run the target release are now blocked. - NVMe-oF gateways are disabled and enabled around host maintenance. - Added CNC and Prometheus frequency fields to the NVMe-oF spec file, --keep-connections to the NVMe-oF CLI, and --force to the NVMe-oF CLI delete kmip server command. - Fixed several NVMe-oF gateway monitor issues, including stretched-cluster relocate and beacon timeouts for gateways in the Created state. Dashboard - Upgraded Angular (19, then 20.3.29). - Backported the notification panel, overview page fixes, and a batch of other dashboard fixes. - OAuth2 SSO configuration fixes. Platform and Packaging - Ubuntu 24.04 is now a supported distro and container host. - Builds now use FUSE 3, and the C standard moved from C99 to C11. - s390x CRC assembly no longer requires an ELF executable stack. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-5-tentacle-released/ Getting Ceph - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-20.2.5.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: ed27c686770872022f4c1de04f6b0ac5d5384ded

3 5

Massive mon slow-ops when restarting mon daemons
by Alex Walender 02 Nov '26

02 Nov '26

Hey there! I’m currently in the process of upgrading the production cluster from v18.2.8 -> v19.2.6 via „ceph orch upgrade“. I tested this upgrade in our staging environment and everything went great. On our production environment I encounter some worrisome issues with the mon daemons. The cluster itself: 5x MON Daemons 5x MGR Daemons 1500x OSD - 2 larger Pools with each 16k in PG. … so a somewhat larger cluster. The orchestrator started re-deploying the MGR, which went fine. During the MON redeployment, the cluster went into warning with messages like „34846 slow ops. mon.mon1 has slow ops“. The counter climbed rapidly. Only way to stop this was to restart the MON in question. Sometimes I had to restart the MON multiple times. Afterwards the cluster is healthy again….until I restart any MON. The symptoms return until restarting fixes it again. I tried to dump the mon_ops_in_flight, but the command just hangs (probably because of the large error count). Filtering the journal for the issues: ``` journalctl -g "oldest is" --since "3 hours ago" -u ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c(a)mon.partnach <mailto:ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c@mon.partnach> Sep 14 09:45:05 partnach conmon[1735]: 2026-09-14T07:45:05.331+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18286 slow ops, oldest is log(1000 entries from seq 2 at 2026-08-26T04:51:47.819315+0000) Sep 14 09:45:10 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:10 partnach conmon[1735]: 2026-09-14T07:45:10.827+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:15 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:15 partnach conmon[1735]: 2026-09-14T07:45:15.924+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:21 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:21 partnach conmon[1735]: 2026-09-14T07:45:21.224+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:26 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:26 partnach conmon[1735]: 2026-09-14T07:45:26.700+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:32 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:32 partnach conmon[1735]: 2026-09-14T07:45:32.432+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:37 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:37 partnach conmon[1735]: 2026-09-14T07:45:37.568+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:43 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) Sep 14 09:45:43 partnach conmon[1735]: 2026-09-14T07:45:43.292+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) ``` I’m not quite sure what the lock is here. The mon tries to store some old cluster log entries from various dates? Checked the underlying network fabric, CPU load, clock skew, compacted the rocksdb manually on each mon ect. All is fine. I also set "ceph config set global mon_cluster_log_level info“ **before** the 10th of September. Luckily the upgrade has finished with the MON daemons and is now re-deploying all OSDs. Does anyone know whats going on? Some kind of old hanging Log-Entries? Best Regards, Alex Walender --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

4 4

Umbrella 21.2.0 bad MGR memory leak
by Chris Palmer 02 Nov '26

02 Nov '26

I've just upgraded a small production light-usage cephadm cluster from Tentacle to Umbrella. The active MGR now seems to be growing by about 1GB per hour though. I can't see anything wrong in the mgr logs. The only other sign of trouble is that a lot of metrics are not displaying in Grafana (mainly OSD-related I think - haven't checked it thoroughly yet). I don't know if that is related to the leak. I just had to fail the MGR to get some memory back. It immediately released about 2.5GB, but is still tracking up gently while in standby. Help... (Just adding a large amount of swap...) Thanks, Chris

3 4

v21.2.0 Umbrella released
by Yuri Weinstein 29 Oct '26

29 Oct '26

Umbrella is the 21st stable release of Ceph. This is the first stable release of Ceph Umbrella. We recommend that all users update to this release. Warnings - Umbrella container images are based on Rocky Linux 10 (el10), which requires a CPU supporting the x86-64-v3 microarchitecture level. Deployments on hosts with older CPUs will not be able to run the containerized daemons. - Python 3.9 or newer is required on all hosts for cephadm support. Notable Changes RADOS / OSD / Erasure Coding - Optimized ("Fast") Erasure Coding: a substantial new EC I/O path. - EC Direct Reads - CLS support for EC reads - EC OMAP support - Add Fast EC CRC-checking for data and first parity - Numerous correctness fixes to the new path (truncate+write planning, PDW vs RMW selection, snapshot scrub handling, peering/recovery corner cases) Ships with health warnings for deprecated EC plugins/techniques and for Blaum-Roth EC profiles whose w+1 isn't prime; non-4k-aligned chunk sizes are now rejected for optimized EC. - Security: CephX authentication hardening -- new AES256KRB5 cipher (replacing weaker legacy key types as the default), online key rotation for mon/OSD/MDS daemons during upgrades, and fixes for CVE-2025-30156 (AES-CBC misuse enabling authentication bypass), CVE-2026-39944, CVE-2026-50152, and CVE-2026-54330 (https://github.com/ceph/ceph/pull/71172) - Monitor: new optional RocksDB backup/restore mechanism (mon_backup_interval, ceph tell mon.* backup, offline restore via ceph-mon --restore-backup). - New availability score feature for Monitor/OSD (tracking + health warnings + config toggle), off by default. - Stretch clusters: tiebreaker mon can now be optional, added a max bucket-weight-diff threshold, a health warning for colocated monitors, and a restriction on changing the election strategy after stretch mode is enabled. - mgr/DaemonServer: ok-to-upgrade now aggregates and globally sorts OSDs across CRUSH child buckets (fixing inaccurate per-chassis/rack ordering), and upgrades can now be scoped to a CRUSH bucket. - pg-upmap-primary support and improved pg-upmap computing speed. - PG autoscaler: default mon_target_pg_per_osd increased 100 -> 200 and mon_max_pg_per_osd increased 250 -> 500, letting the autoscaler compute more parallelism-friendly pg_num values and improving the balancer's ability to reach uniform OSD utilization. Existing pools may autoscale to more PGs after upgrading as a result; pg_autoscale_during_upgrade (https://github.com/ceph/ceph/pull/67694) paces that splitting so it doesn't disrupt the upgrade itself. Clusters with a smaller-than-4GiB effective bluestore_osd_memory_target, or SSDs split into more than two OSDs, may want to pin the old values before upgrading (https://github.com/ceph/ceph/pull/69452) - BlueStore: BlueFS Spillover Cleaner evolution, faster allocation recovery, static/runtime object-fragmentation tracking, full-device discard on mkfs now default, several spanning-blob/onode-cache-stall/reallocation-corruption fixes, and device-type-specific allocation policy. - mClock scheduler refactor. Default HDD OSD shard config changed as an interim fix for inconsistent client/recovery throughput with multiple shards: osd_op_num_shards_hdd 5 -> 1, osd_op_num_threads_per_shard_hdd 1 -> 5 ( https://tracker.ceph.com/issues/66289) - Crimson/Seastore: continued heavy investment -- erasure coding support landed in Crimson (including recovery), PG splitting, a new 2Q cache replacement algorithm, broad coroutine conversions, and a long tail of correctness/perf fixes across the object-data, LBA/omap-btree, and cleaner subsystems. RGW (RADOS Gateway) - Full object dedup -- tech preview. - Bucket logging -- new feature: journal-mode logging, EC-pool support, admin commands, and new per-bucket lifecycle/logging performance monitoring that closes a long-standing observability gap around lifecycle processing. - Cloud transition / cloud restore: restore from Glacier/tape, admin CLI for restore list/status, restore-event notifications, persistent restore state, per-bucket target options; lifecycle, cloud-restore, and cloud-transition processing converted to coroutines for a significant performance improvement. - Encryption: added AES-256-GCM SSE support; CopyObject now supported for encrypted objects. - IAM/Account: GetAccountSummary, account-wide PublicAccessBlock, Admin REST APIs for account quota, S3 Object Ownership (disable ACLs), legacy Keystone admin-token requirement removed, Keystone roles injected into IAM policy. - STS/OIDC: GetCallerIdentity, JWT/JWKS signature validation, OIDC multisite support. - SigV4: fixed a regression (present in 19.2.6/20.2.4) that rejected requests with an unsigned Content-Type, breaking presigned PUT uploads; x-amz-content-sha256 is no longer required in SignedHeaders for S3 requests ( https://github.com/ceph/ceph/pull/71363) - Multisite: numerous sync-correctness fixes (full-sync infinite loop, segfault during startup, forwarded awsv4/UNSIGNED-PAYLOAD requests, bucket-sync full-sync locking, zonegroup rename handling). - D4N (Redis-backed) cache: continued build-out (bucket check_empty, connection pooling/pipelining, dirty-entry handling, crash fixes). - s3-tests vendored directly into the repo (src/test/rgw/s3-tests); QA suites run them from there instead of an external branch reference. - Timestamp-based epochs for versioned objects; global CORS rule support; rate-limit response with Retry-After; standalone RGW build (without RadosStore) via a new config-store abstraction -- a significant architectural refactor of rgw_zone/realm-watching code. - Lua scripting: new postAuth hook, fresh VM per background script execution, package-management fixes. Dashboard - Large-scale UI migration to the IBM Carbon design system across most pages (RGW, pools, OSD flags, multisite, zone/zonegroup, services, upgrade page, and more). - NVMe/TCP (NVMeoF) management significantly expanded: multi-step subsystem/gateway/namespace creation wizards, TLS/mTLS, DHCHAP auth, KMIP support, consistency-group CRUD APIs, and many CLI/UI alignment fixes. - SMB cluster management support added. - RGW role management realigned with Carbon + API routing fixes; global RGW tenant "Roles" tab removed in favor of per-account roles. - CephFS mirroring wizard/entity management UI. - OAuth2 SSO support (config UI + backend prerequisites). - Angular upgraded to v19. cephadm / Orchestrator - Extended certificate management: cephadm now provides comprehensive certmgr integration across managed services including mgmt-gateway, monitoring stack, NFS, and RGW. Services support auto-generated and renewed cephadm-signed certificates, user-provided certificates, and certificate-store references. TLS PEM handling is now standardized with support for full certificate chains and automatic separation of certificates and private keys, simplifying certificate management across services. - SMB/Samba cluster support (the managed service type itself shipped in Tentacle): new remote-control client tool, per-share QoS/bandwidth rate limiting, custom ports, keybridge/FSCrypt-based share configuration, and support for CephFS shares served from an external Ceph cluster. - NVMe-oF gateway spec gained many fields (TLS, KMIP, Prometheus frequency, namespace limits, group assignment). - NFS-Ganesha: QoS support, active-active/keepalived-only ingress mode, TLS configuration support, RDMA handling during upgrades. - Crimson/seastore OSD deployment support added to cephadm and ceph-volume. - cephadm now supports CRUSH bucket-aware upgrades using the new ok-to-upgrade command, enabling faster and safe concurrent OSD upgrades while preserving data availability. CephFS / MDS - FSCrypt (client-side encryption) userspace implementation. - Multi-threaded cephfs-mirror, plus a blockdiff-based sync mechanism for regular-file transfers. - cephfs-mirror: new mirroring checkpoints (CLI + cluster-wide status tracking), mutable CephFS snapshot metadata (new libcephfs/pybind API), and substantially expanded per-directory/per-peer sync metrics. - New pybind/cephfs bindings: utimensat, chownat, symlinkat, readlinkat, statxat, mkdirat; non-recursive rmtree()/cptree(). - Subvolume group name-mangling support; new subvolume performance/utilization metrics; clone-source info retained in subvolume info. - Root-directory fragmentation disallowed; session-reclaim blocklisting fix; several shutdown-hang and scrub-pinning fixes. ceph-volume - Migrated internal config/sysinfo representation to dataclasses; device detection switched to udev data instead of shelling out to LVM/udevadm in several code paths (perf and reliability win). - Added seastore OSD support; fixed inventory failures on hosts with RAID-mirrored system volumes, cdrom devices, and missing /dev/vg/lv paths. Build / Packaging / QA Infrastructure - Ubuntu 24.04 (Noble) and Rocky Linux 10 added as supported distros across build and QA; CMake bumped to require 3.22.1 and gained CMake-4 support; C++ standard raised to C++23. - The Ceph container's base OS image has changed to rockylinux:10. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v21-2-0-umbrella-released/ Getting Ceph - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-21.2.0.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: b060042deb05d33e6cc358970cfb8221240bcf3e

6 10

Ceph talks at OpenInfra Days North America in Chicago
by Anthony Middleton 29 Oct '26

29 Oct '26

Hey Ceph friends, Our friends at OpenInfra are hosting OpenInfra Days North America in Chicago on Tuesday, November 17, the day after Ceph Days Chicago <https://ceph.io/en/community/events/2026/ceph-days-chicago/>. They’d love to have the Ceph community join them, and they’re offering a 25% discount with code: *Ceph25.* A few sessions that may interest the Ceph community: - Operating Ceph at Scale: Minimizing Recovery and Failure Impact on Production Workloads <https://openinfra-day-northamerica-2026.sessionize.com/session/1329050> - Beyond VMware: Engineering a Sovereign Cloud Using OpenStack & Ceph <https://openinfra-day-northamerica-2026.sessionize.com/session/1306356> - Closing the CephFS Trust Gap: Hypervisor-Mediated Storage in OpenStack <https://openinfra-day-northamerica-2026.sessionize.com/session/1315069> *Event page:* https://openinfra.org/days/northamerica *Schedule:* https://openinfra-day-northamerica-2026.sessionize.com/ *Registration:* https://linuxfoundation.regfox.com/openinfra-day-north-america Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

RBD: Enabling Mirror snapshots?
by Oliver Freyermuth 29 Oct '26

29 Oct '26

Dear Cephalopodians, long time no write — please bear with the painful question about an Octopus cluster (upgrade planned) for which we'd like to switch from RBD Mirroring via journaling to snapshot-based mirroring (both clusters involved are "latest" Octopus). The RBD volumes are used via libvirt/QEMU(KVM), using librbd. We have first switched the pool mode over to "image" mode: rbd mirror pool enable rbd image (we were using pool before) to be able to use the snapshot mode for individual images. Then, after disabling mirroring completely for an image: rbd mirror image disable rbd/myvm-disk1 which works fine and disabled the journaling and journal-based mirroring, we try to switch to mirror snapshots: rbd --debug-rbd=6 mirror image enable rbd/myvm-disk1 snapshot Sadly, this reveals the messages appended in full below, the critical part (only visible with debug level >=5) seems to be: handle_acquire_lock: unable to acquire exclusive lock Naturally, the running VMs are holding an exclusive lock on the images, and we are also doing regular "normal" snapshots daily. Are we correct to interpret this as follows? * Journal-mirroring can be toggled any time, no exclusive lock required. * Enabling snapshot-based mirroring requires an exclusive lock(?), i.e. shutting down the affected VM? Or is there a better way (breaking the lock seems unsafe)? Is this expected behaviour, or a bug (and / or is this different in later Ceph versions), or are we missing a step / some configuration, or just misinterpreting the output? It would of course be great operationally if we could get away without a VM shutdown to toggle this :-). Furthermore, if somebody has experience with mirror snapshots in general, any experience is welcome. For reference, we are right now using journal-based mirroring and in parallel performing daily (regular) snapshots with fstrim/fsfreeze before each snapshot (via the qemu guest agent) and backing these up with Benji, so we have two backup paths in place (mirroring plus Benji to a separate system). We plan to also perform (manual/scripted) mirror snapshots in parallel to regular snapshots to profit from the advantages of the trim+freeze+snapshot approach :-). Cheers, Oliver ------------- 2026-10-28T13:15:22.838+0100 7fafca678500 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 ImageRequestWQ: ictx=0x55720aba5e20 2026-10-28T13:15:22.849+0100 7fafa8ff9700 5 librbd::ImageCtx: 0x55720aba5e20: disabling zero-copy writes 2026-10-28T13:15:22.852+0100 7fafa8ff9700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 block_writes: 0x55720aba5e20, num=1 2026-10-28T13:15:22.852+0100 7faf9bfff700 5 librbd::cache::WriteAroundObjectDispatch: 0x7faf94006020 init: 2026-10-28T13:15:22.852+0100 7faf9bfff700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 register_object_dispatch: object_dispatch_layer=1 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 SimpleSchedulerObjectDispatch: ictx=0x55720aba5e20 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 init: 2026-10-28T13:15:22.854+0100 7fafa8ff9700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 register_object_dispatch: object_dispatch_layer=4 2026-10-28T13:15:22.857+0100 7fafa8ff9700 5 librbd::Operations: 0x55720ab986c0 snap_create: snap_name=.mirror.primary.919fa522-7d50-49c6-8b9d-e7926ade404a.ef881760-f1cf-4fcf-8fe5-cf491359bf4a 2026-10-28T13:15:22.859+0100 7fafa8ff9700 5 librbd::managed_lock::AcquireRequest: 0x7faf94007a40 handle_break_lock: lock owner is still alive 2026-10-28T13:15:22.859+0100 7faf9bfff700 5 librbd::ManagedLock: 0x7faf8c0150b8 handle_acquire_lock: unable to acquire exclusive lock 2026-10-28T13:15:22.860+0100 7fafa8ff9700 5 librbd::Watcher: 0x7faf8c003030 notifications_blocked: blocked=0 2026-10-28T13:15:25.126+0100 7faf9bfff700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7faf8c01a600 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7faf9bfff700 -1 librbd::mirror::EnableRequest: 0x55720aa7aa30 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7fafca678500 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument 2026-10-28T13:15:25.126+0100 7fafca678500 5 librbd::Watcher: 0x7faf8c003030 block_notifies: blocked_count=1 2026-10-28T13:15:25.126+0100 7faf9bfff700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 shut_down: shut_down: in_flight=0 2026-10-28T13:15:25.126+0100 7faf9bfff700 5 librbd::io::ImageRequestWQ: 0x55720aa770f0 unblock_writes: 0x55720aba5e20, num=0 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::ObjectDispatcher: 0x55720abe9650 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::ObjectDispatch: 0x55720abe98a0 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::io::SimpleSchedulerObjectDispatch: 0x7faf8c011f50 shut_down: 2026-10-28T13:15:25.129+0100 7faf9bfff700 5 librbd::cache::WriteAroundObjectDispatch: 0x7faf94006020 shut_down: -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

2 8

October Ceph Foundation Board Meeting
by Anthony Middleton 28 Oct '26

28 Oct '26

Hey Ceph friends, The October Ceph Foundation Board meeting is now available on our YouTube channel. You can watch it here: https://youtu.be/H0yKb_5DAkQ. If you’re interested in contributing to the Ceph project, please feel free to reach out. I’d be happy to help you find a way to get involved. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Segmentation fault adding a disk
by Alfredo De Luca 28 Oct '26

28 Oct '26

Hi all. I have installed in a few RPI the microceph (ceph-version: 19.2.3-0ubuntu0.24.04.3; microceph-git: d95f07173f). Now 2 RPI are ubuntu based OS (Ubuntu 24.04.5 LTS) and 1 is with Debian 13 (Trixie). All works well except when I need to add a disk on debian which cause me the following segmentation fault: *# microceph disk add /dev/disk/by-id/nvme-eui.00000000000000000000004380201825 --wipe+-----------------------------------------------------------+---------+| PATH | STATUS |+-----------------------------------------------------------+---------+| /dev/disk/by-id/nvme-eui.00000000000000000000004380201825 | Failure |+-----------------------------------------------------------+---------+Error: failed to bootstrap OSD: Failed to run: ceph-osd --mkfs --no-mon-config -i 34: signal: segmentation fault* I can see the disk from the OS level of course *# nvme list * */dev/nvme0n1 /dev/ng0n1 ASBE44380201825 Acer SSD FA200 1TB 0x1 1.00 TB / 1.00 TB 512 B + 0 B SN15536 * # lsblk nvme0n1 259:0 0 931.5G 0 disk Any idea what the issue would be? Cheers -- */Alfredo*

1 1