November 2025 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

cephadm without containers
by Adrian Sevcenco 10 Apr '26

10 Apr '26

Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine) If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way? I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm Thank you! Adrian

11 20

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

reef 18.2.8 QE validation status
by Yuri Weinstein 13 Feb '26

13 Feb '26

Note: this is an attempt to test and publish post EOL reef point release Details of this release are summarized here: https://tracker.ceph.com/issues/73906#note-1 Release Notes - TBD LRC upgrade - N/A distro => jammy focal centos9 windows bookworm Seeking approvals/reviews for: smoke - Laura approved? rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? (rados is also running on build 2 == build 1 + https://github.com/ceph/ceph/pull/66056 per Igor's request) rgw - Adam E approved? fs - Venky approved? orch - Adam King approved? rbd - Ilya approved? krbd - Ilya approved? upgrade/pacific-x (reef) - can this be deprecated? Josh? Neha? (same as in 18.2.5) upgrade/quincy-x (reef) - Laura, Prashant please take a look. (same as in 18.2.5) powercycle - Brad approved? ceph-volume - see orch, Guillaume approved? Notably upgrades suites show same results as for 18.2.5 (tested in March 2025) To compare, see 18.2.5 tests https://tracker.ceph.com/issues/70563#note-1 TIA

4 5

Debian Packages for Trixie
by Andrew 09 Feb '26

09 Feb '26

Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew

6 6

Re: Object replication issue between zones for RGW Account owned buckets
by Ashok Kumar M 08 Jan '26

08 Jan '26

Hi, I’m currently using Ceph version 19.2.3 and have a setup with two zones (zone-a and zone-b) under a single zonegroup (zonegroup-a) and realm (realm-a). When I replicate objects from bucket-a (in zone-a) to bucket-b (in zone-b) for buckets owned by a regular user (ex: my-user), the replication works successfully. However, replication does not work for buckets owned by the RGW Account (ex: RGW84754594545284760). In this case: There are no errors shown in the sync status on either zone. The replication status of the objects in the source bucket remains in a “PENDING” state indefinitely. Has anyone else encountered a similar issue, or is there any known workaround for this? I would really appreciate any kind of advice or guidance regarding this. Thanks, Ashok M Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 1

Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 07 Jan '26

07 Jan '26

Hi, this is kind of a follow-up on a two years old thread [0] and I wanted to raise some awareness for the corresponding tracker [1]. Back then we managed to limit the impact on the mon store performance with some paxos configs, but now the OSDs are impacted as well when there are new ones created: Each newly created OSD process grows in RAM usage to around 140 GB within a few minutes, easily causing oom killers on hosts if multiple OSDs are created at once. The residual RAM usage drops to the memory target after it successfully booted. The reason are the purged_snaps that are loaded during OSD boot (snap_mapper.record_purged_snaps purged_snaps), two years ago the customer had more than 42 million purged_snap entries. I don't know how many there are today, I don't have access myself, but I'll try to get a current number. Anyway, the only way to safely create OSDs is one by one, maybe two at once depending on the host's RAM capacity. An automated (unattended) OSD deployment is currently not possible. Unfortunately, the new tool [2] doesn't seem to work as expected, at least in my test cluster it didn't have any impact on the number of purged_snaps in the mon store. That's why we haven't tried it on the customer cluster(s) yet. How do other operators/admins/users deal with this kind of scenario? Having many snapshots can't be a corner case, but I can't remember having read anything like this on the list(s). I'd appreciate any comments, although I'm aware that everybody is probably busy travelling to Cephalocon. ;-) Thanks! Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZEMGKBLMERE… [1] https://tracker.ceph.com/issues/64519 [2] https://github.com/ceph/ceph/pull/57548

1 1

Removing a specific object.
by quaglio＠bol.com.br 01 Dec '25

01 Dec '25

Hi, I needed to remove a disk from a specific pool and now I'm facing a problem that I can't solve. There is a pool that has 1 "unfound" object. Furthermore, recovery is not happening for that PG because of this object. pg 3.792 is active+forced_recovery+recovery_unfound+undersized+degraded+remapped, acting [69], 1 unfound There is no problem removing this object. How do I delete it? Thanks Rafael.

1 1

Persistent “PGs not scrubbed in time” on Reef 18.2.7 cluster — seeking tuning guidance
by Zachary Wright-Faust 30 Nov '25

30 Nov '25

Hi all, I’m running a 10-node Ceph cluster (Reef 18.2.7) that has been having persistent issues keeping PG scrubs up to date as we migrate workloads from our previous production cluster. This environment provides CephFS only. The main CephFS data pool is an EC 4+2 pool backed by HDD OSDs, and the metadata pool is a REP 3 pool on SSD OSDs. The hardware is relatively old (mostly Dell PowerEdge R630s with SAS-2 attached JBODs). This cluster was deployed as an interim platform while we rebuild our production environment following unrecoverable MDS corruption. The scrub backlog started to accumulate as soon as we began migrating data onto this interim cluster, and although I’ve been able to mitigate it at times, it has been growing again now that the majority of our clients have shifted over to it. Below is the current cluster status: > cluster: > id: 150735d4-4449-11f0-a328-a0369ff049b4 > health: HEALTH_WARN > 126 pgs not scrubbed in time > > services: > mon: 5 daemons, quorum > sio-nautiloid01,sio-nautiloid02,sio-nautiloid03,sio-nautiloid07,sio-nautiloid06 > (age 4w) > mgr: sio-nautiloid02.jbayyi(active, since 5h), standbys: > sio-nautiloid01.ssmngh > mds: 1/1 daemons up, 1 standby > osd: 248 osds: 248 up (since 4w), 248 in (since 9w) > > data: > volumes: 1/1 healthy > pools: 6 pools, 8417 pgs > objects: 1.04G objects, 1.3 PiB > usage: 1.9 PiB used, 1.7 PiB / 3.6 PiB avail > pgs: 6913 active+clean > 669 active+clean+scrubbing+deep > 408 active+clean+scrubbing > 405 active+clean+snaptrim_wait > 22 active+clean+snaptrim > > io: > client: 37 MiB/s rd, 4.3 MiB/s wr, 1.11k op/s rd, 53 op/s wr > > Current scrub-related configuration: mgr advanced > osd_deep_scrub_interval 5184000.000000 > > mgr advanced > osd_scrub_max_interval 2419200.000000 > > mgr advanced > osd_scrub_min_interval 1209600.000000 > > osd advanced > osd_deep_scrub_interval 4838400.000000 > > osd advanced > osd_deep_scrub_randomize_ratio 0.150000 > > osd advanced > osd_max_scrubs 3 > > osd advanced > osd_scrub_chunk_max 25 > > osd advanced > osd_scrub_load_threshold 0.900000 > > osd advanced > osd_scrub_max_interval 2419200.000000 > > osd advanced > osd_scrub_min_interval 1209600.000000 > > osd advanced > osd_scrub_priority 10 > > osd advanced > osd_scrub_sleep 0.100000 > These intervals mirror what we had configured on our previous production cluster (Quincy 17.2.7). I introduced osd_scrub_sleep after observing high commit and apply latency in ceph osd perf. I’ve also experimented with higher scrub priority and a higher load threshold, but the number of overdue scrubs continues to grow. At this point I’m looking for guidance on: - Additional tuning options that might help scrub throughput - Whether my current interval choices are counterproductive - How to identify the primary bottlenecks (OSD load, disk latency, stride size, concurrency, etc.) Any insights would be greatly appreciated. Thanks, Zach

1 0

Re: cephadm without containers
by Anthony D'Atri 30 Nov '25

30 Nov '25

> Thus displacing the problem from developers to the admins as now admins have to maintain the whole container support environment, including the "slightly varying library packages etc." This whole exercise is pointless as Ceph cluster storage nodes are natively dedicated to Ceph There are LOTS of converged clusters, notably Proxmox and Rook within K8s. > thus the bare-metal machine is the real Ceph container. Just install the specific OS supported by Ceph Fleets don’t care for that approach. They don’t want to have to maintain automation and package and provisioning systems for multiple OSes, multiply the infosec burden, etc. With Podman and log-to-file, in addition to the cephadm orchestrator, one can still manage individual daemons with systemctl. > and you've got your Ceph "container" made out of metal. > > You can make an argument that monitor and/or manager daemons could run in containers to allow for migration to alternative nodes. Indeed, this is a big advantage. > Properly sized Ceph clusters come with a number of dedicated monitor nodes anyway In the Luminous days I felt that way, since mons were subtle and quick to anger. I liked to decouple them from OSDs, which were more likely to need node reboots. I no longer due. I’ve had a pre-cephadm cluster with 5x mon/RGW nodes, two of which broke, and it took me *18 months* to convince the TAC to fix them. The cluster was one more failure from an outage, where with a converged cluster the full quorum would be maintained automagically. Defining the service spec to abstract based on a “mon” host label, and judiciously applying that label to one or at most two nodes per rack / failure domain maintains prudent anti-affinity. > therefore migration usually means add a new monitor node to the cluster and remove the old one, thus making the monitor containers point moot, too. Adding a new node to the cluster is far from trivial when the DC is overseas and unstaffed. It would also mean having a warm / hot spare node, which with your reasoning would be a different spec than the OSD nodes. The fewer chassis types in a DC the better, from an interchangeability, servicing, and sparing perspective. And, yes, I’ve run fleets of as many as 15000 nodes. ymmv. — aad

1 0

[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag
by Reto Gysi 29 Nov '25

29 Nov '25

Hi I tried to enable the new ec optimisation flag on one of the erasure encoded pools for rbd on my recently upgraded to Tentacle 20.2.0 version ceph cluster. ceph osd pool set rbd_ecpool allow_ec_optimizations true At first everything seemed normal. After a while when I went back to the computer I saw that osd.1 and osd.3 were always crashing. After a reboot I then saw errors and warnings about inconsistent pg's. OSD_SCRUB_ERRORS: 116116 scrub errors OSD_TOO_MANY_REPAIRS: Too many repaired reads on 4 OSDs PG_DAMAGED: Possible data damage: 3 pgs inconsistent I tried to run a manual ceph pg repair <pgid> on the affected pg's. While the affected pg's disappeared from the inconsistent pg list after a deep-scrub+repair new pgid appeared on the incosistent pg list and the amount of scrub errors increased. When I run a manual repair on a pgid, I see a lot of errors in the pg's primary osd log like: - 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39(237963'11927123 client.40141308.0:1785630 dirty s 4194304 uv 11910481 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39 : size 1400832 != size 1396736 from auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d39(237963'11927123 client.40141308.0:1785630 dirty s 4194304 uv 11910481 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45(238043'11948425 client.40186030.0:433080 dirty s 4194304 uv 11933347 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45 : size 1400832 != size 1396736 from auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d45(238043'11948425 client.40186030.0:433080 dirty s 4194304 uv 11933347 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : candidate size 1400832 info size 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.b shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : candidate size 1400832 info siz e 1396736 mismatch 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 3(2) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : size 1400832 != size 1396736 f rom auth oi 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d(238319'11972769 client.40186030.0:1294524 dirty|data_digest s 4194304 uv 11955052 dd 2cef26e9 alloc_hint [0 0 0]) 2025-11-29T11:48:10.251+0000 7f7328665640 -1 log_channel(cluster) log [ERR] : 90.bs0 shard 15(1) soid 90:d0bc306e:::rbd_data.7.6f5c183c7dc42b.0000000000001331:2d5d : size 1400832 != size 1396736 The size mismatch always seem to be the same: 1400832 != size 1396736 And looking at the rbd namespace like example rbd_data.7.6f5c183c7dc42b It looks like so far that images that have been in use/mounted during enablement of allow_ec_pool_optimization are affected. So my questions are: 1. Are these errors expected/explainable? 2. Will a manual pg repair on all the pools pgids eventually fix the problem? 3. Is there any chance that data has been lost/inconsistent after pg repair on all the pools pgids? Thanks & best regards, Reto

1 1

v20.2.0 Tentacle released
by Yuri Weinstein 28 Nov '25

28 Nov '25

We're very happy to announce the first stable release of the Tentacle series. We express our gratitude to all members of the Ceph community who contributed by proposing pull requests, testing this release, providing feedback, and offering valuable suggestions. We would like to especially thank some of our community members who helped us test upgrades for the pre-released version of 20.2.0. Your feedback and effort are greatly appreciated! Note from the Ceph Infrastructure Team: Part of our Standard Operating Procedure for Ceph releases is typically to upgrade the Ceph cluster in the Ceph lab to the latest version before announcing the release. This is a production cluster, which we colloquially call the Long Running Cluster (LRC), as it has existed and been updated for the past decade or even longer. For this release, we have decided to hold off on updating the LRC until we have migrated some of the labs from Red Hat data centers to IBM. We plan to update the LRC to Tentacle in the next calendar year. We believe this decision should not be viewed as a reflection of our confidence in the project, but rather as an attempt to focus on other priorities before the migration. ---Highlights--- CephFS * Directories may now be configured with case-insensitive or normalized directory entry names. * Modifying the FS setting variable ``max_mds`` when a cluster is unhealthy now requires users to pass the confirmation flag (``--yes-i-really-mean-it``). * ``EOPNOTSUPP`` (Operation not supported) is now returned by the CephFS FUSE client for ``fallocate`` for the default case (i.e. ``mode == 0``). Crimson * SeaStore Tech Preview: SeaStore object store is now deployable alongside Crimson-OSD, mainly for early testing and experimentation. Community feedback is encouraged to help with future improvements. Dashboard * Support has been added for NVMe/TCP gateway groups and multiple namespaces, multi-cluster management, OAuth 2.0 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. Integrated SMB support * Ceph clusters now offer an SMB Manager module that works like the existing NFS subsystem. The new SMB support allows the Ceph cluster to automatically create Samba-backed SMB file shares connected to CephFS. The ``smb`` module can configure both basic Active Directory domain or standalone user authentication. The Ceph cluster can host one or more virtual SMB clusters which can be truly clustered using Samba's CTDB technology. The ``smb`` module requires a cephadm-enabled Ceph cluster and deploys container images provided by the ``samba-container`` project. The Ceph dashboard can be used to configure SMB clusters and shares. A new ``cephfs-proxy`` daemon is automatically deployed to improve scalability and memory usage when connecting Samba to CephFS. MGR * Users now have the ability to force-disable always-on modules. * The ``restful`` and ``zabbix`` modules (deprecated since 2020) have been officially removed. RADOS * FastEC: Long-anticipated performance and space amplification optimizations are added for erasure-coded pools. * BlueStore: Improved compression and a new, faster WAL (write-ahead-log). * Data Availability Score: Users can now track a data availability score for each pool in their cluster. * OMAP: All components have been switched to the faster OMAP iteration interface, which improves RGW bucket listing and scrub operations. RBD * New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats. * There is now support for RBD namespace remapping while mirroring between Ceph clusters. * Several commands related to group and group snap info were added or improved, and ``rbd device map`` command now defaults to ``msgr2``. RGW * Added support for S3 ``GetObjectAttributes``. * For compatibility with AWS S3, ``LastModified`` timestamps are now truncated to the second. Note that during upgrade, users may observe these timestamps moving backwards as a result. * Bucket resharding now does most of its processing before it starts to block write operations. This should significantly reduce the client-visible impact of resharding on large buckets. * The User Account feature introduced in Squid provides first-class support for IAM APIs and policy. Our preliminary STS support was based on tenants, and exposed some IAM APIs to admins only. This tenant-level IAM functionality is now deprecated in favor of accounts. While we'll continue to support the tenant feature itself for namespace isolation, the following features will be removed no sooner than the V release: - Tenant-level IAM APIs including CreateRole, PutRolePolicy and PutUserPolicy, - Use of tenant names instead of accounts in IAM policy documents, - Interpretation of IAM policy without cross-account policy evaluation, - S3 API support for cross-tenant names such as `Bucket='tenant:bucketname'` - STS Lite and `sts:GetSessionToken`. We encourage you to read the full release notes at https://ceph.io/en/news/blog/2025/v20-2-0-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_20.2.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 69f84cc2651aa259a15bc192ddaabd3baba07489 Did you know? Every Ceph release is built and tested on resources funded directly by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now https://ceph.io/en/foundation/.

6 7

cephadm cluster - logging redudancy
by Francesco Di Nucci 27 Nov '25

27 Nov '25

Hi all, I am deploying a new cluster with cephadm, I used "--with-centralized-logging" option at bootstrap and it set up graphana, loki and prometheus with count: 1 as placement. Now I'd like to increase redundancy/availability of the logging part, so that even if the host that has one of these services goes down the logging keeps on working; are there any guidelines/suggestions on this? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

1 0

Meaning of RGW sync status full sync: objects completed
by Adam Prycki 26 Nov '25

26 Nov '25

Hello, I have a question about rgw synchronization mechanic. When we are trying to initialize full bucket sync on existing bucket, or when try to synchonize delete bucket we see "full sync" in bucket sync status on our clusters. For example: realm b6ca44da-921e-42da-b377-80e9fae83b61 (s3-TEST.kmd4.pl) zonegroup c41b36c6-2031-407b-bab4-36cd0b3e56da (poznan-1) zone 91a17d0c-84a0-4706-a40d-1b6df1996a10 (zone-poznan-1-kielman) bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) current time 2025-11-24T13:11:48Z source zone 8e5c54a9-7eb2-4521-b327-081a1a647205 (zone-poznan-1-pcss-bst) source bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) full sync: 15221490 objects completed In this case we have deleted bucket with broken synchronization on the secondary zone. After RGW was started up again synchronization should have tarted from zero. Bucket 1d2848a2-9ca7-402c-9976-a6dc775547bb has 1163263 10MiB s3 objects. bucket sync status is right now at 15221490 and still increasing. That's 13x bigger number than object count in this bucket. In case of some other bucket we saw that "full sync: objects completed" number grew to about 20x s3 object count in the bucket. One bucket with 10 000 000 objects has running full sync has "full sync: 605952706 objects completed" Can someone explain what this number means and why it gets so big? How big it's supposed to get? Can we predict how much of the synchronization is "done" from this number? Best regards Adam Prycki

1 0

Ceph Developer Meetup next week
by Anthony Middleton 26 Nov '25

26 Nov '25

Hey Ceph friends, Please join us for the* Ceph Developer Meetup next week on December 3 at 11 am EST / 8 am PDT*. The agenda is in the meetup announcement. Please *add your topics* in advance. *RSVP *here: https://www.meetup.com/ceph-user-group/events/312181316/?utm_medium=referra…

1 0

RGW Signature does not match
by GLE, Vivien 26 Nov '25

26 Nov '25

Hi, I'm trying to setup a VM RGW in a cephadm managed test cluster. Everything is working fine except 2 things: - RGW didn't show up in the object section of the ceph dashboard and this message shows up The Object Gateway Service is not configured Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx000002bfe4c78d8a' b'4b666-006926c06c-42063466-<zone>","HostId":"42063466-<zone>-<groupzone>"}') The daemon is active in ceph -s and other functionalities are working fine I read that it might be a date error, as my hosts are on CET, but the containers are set to UTC is there a way to correct that ? - The second thing is that I need to radosgw-admin --name client.ceph-rgw (which is working fine) instead of only radosgw-admin otherwise it seems that radosgw-admin didn't find my ceph.conf and output is looking like this : -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin (2) No such file or directory failed to fetch mon config (--no-mon-config to skip) If I rename my client.keyring file this output is still showing up => failed to fetch mon config (--no-mon-config to skip) cat /etc/ceph/ceph.conf [global] fsid = <fsid> mon_host = [v2:192.168.219.12:3300/0,v1:192.168.219.12:6789/0] [v2:192.168.219.14:3300/0,v1:192.168.219.14:6789/0] [v2:192.168.219.15:3300/0,v1:192.168.219.15:6789/0] auth_client_required = cephx auth_service_required = cephx auth_supported = cephx [client.ceph-rgw] host = ceph-rgw rgw_dns_name = ceph-rgw keyring = /etc/ceph/ceph.client.ceph-rgw.keyring rgw_frontends = "beast endpoint=192.168.219.19:8080 ssl_endpoint=192.168.219.19:8443 ssl_certificate=/etc/ceph/certs/ceph-rgw.pem" log_file = /var/log/ceph/ceph-rgw-ceph-rgw.log rgw_thread_pool_size = 512 cat /etc/ceph/ceph.client.ceph-rgw.keyring [client.ceph-rgw] key = <key> ceph-radosgw(a)ceph-rgw.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; preset: enabled) Active: active (running) since Tue 2025-11-25 16:33:08 CET; 19h ago Main PID: 90607 (radosgw) Tasks: 621 Memory: 114.6M (peak: 115.1M) CPU: 2min 2.665s CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw(a)ceph-rgw.service └─90607 /usr/bin/radosgw -f --cluster ceph --name client.ceph-rgw --setuser ceph --setgroup ceph Thanks Vivien

1 0

Changing the crush rule of an existing pool
by Huseyin Cotuk 26 Nov '25

26 Nov '25

Hello Cephers, I just want to get confirmation on two things. I'm going to change the crush rule of an existing pool so that it uses SSDs instead of HDDs. 1. Will changing the rule cause any problems with the current workload? 2. Does Ceph manage the migration of data currently in use when the existing pool's rule is changed? If so, does it also delete the data on the old disks after the migration? Thanks in advance. BR, Huseyin Cotuk hcotuk(a)gmail.com

1 0

New Cluster Ceph
by GLE, Vivien 25 Nov '25

25 Nov '25

Hi, We plan to buy hardware for a new cluster ceph and would like some approbation on what we choose. 6 nodes DELL 6715 + 6 Powervault MD2412 enclosure For each node => 1 CPU AMD EPYC 9475F 3,65 GHz, 48C/96T, 256M Cache (400 W) RAM 16Gox16 = 256Go 4x 100GB NVIDIA MELLANOX 100GB HBA465e (externe, 22,5GB/s) 2 NVMe mixed 6,4To DBWAL 6 NVMe mixed 6,4To 5 NVMe read 15,36To for each enclosure => 12 HDD 20To HDD will be a replica 3 pool with the dbwal NVME mixed and read will be 2 differents pools (we will test replica 3 and EC to see which performance/storage efficency satisfy us the most) This cluster will be mostly use to store block (VM proxmox/kubernetes) and S3. The crushmap will be like so : 3 rooms with 2 nodes per room. So the point of failure can be for replica 3 => rooms and if we use EC 4+2 => host We saw that the thread needs for NVMe OSD are very expensive, does this CPU good enough to carry them ? MON and MGR will be spread across the cluster and RGW on virtual machine Thanks for your answer ! Vivien

3 5

Tentacle: bootstrap fails setting orch backend
by Eugen Block 25 Nov '25

25 Nov '25

Hi, I tried bootstrapping a fresh test cluster on a virtual machine (openSUSE Leap 16), and it fails: ---snip--- Setting orchestrator backend to cephadm... Non-zero exit code 95 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v20.2.0 -e NODE_NAME=tentacle -v /var/log/ceph/c31e1b94-c921-11f0-b1cf-fa163e7037b6:/var/log/ceph:z -v /tmp/ceph-tmpqdv5t793:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp615joe66:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.0 orch set backend cephadm /usr/bin/ceph: stderr Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date /usr/bin/ceph: stderr Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v20.2.0 -e NODE_NAME=tentacle -v /var/log/ceph/c31e1b94-c921-11f0-b1cf-fa163e7037b6:/var/log/ceph:z -v /tmp/ceph-tmpqdv5t793:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp615joe66:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.0 orch set backend cephadm: Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it ---snip--- Setting it afterwards succeeds, but then you end up with a semi-broken cluster: [ceph: root@tentacle /]# ceph orch set backend cephadm [ceph: root@tentacle /]# ceph orch host ls 0 hosts in cluster I created a tracker without checking if anything similar already exists: https://tracker.ceph.com/issues/73974 I haven't tried bootstrapping with other distros yet, have others experienced something similar on a different OS? Thanks, Eugen

1 1

CSC Meeting Notes: 2025-11-24
by Adam Emerson 24 Nov '25

24 Nov '25

For the meeting on Monday, 24 November 2025. S390/PPC Testing Upstream? ========================== Radoslaw noted a recent PR (https://github.com/ceph/ceph/pull/64894) that is S390/Power specific. We'd like to merge it, it looks reasonable. In general, we don't test S390 and PowerPC upstream. Since we're moving the lab, is this on our plate/plans at all? Greg points out that we don't just lack machines, we don't really have anything in place to support those architectures upstream. Perhaps we should, but we'd need to convince some corporation to support it and donate hardware. The answer for new architectures is always that we are happy to accept hardware, we will not make your github checks required, and you have to do the work. (Especially since many of them are very expensive.) Ilya says that IBM has some virtual machines that can be carved out. Several upstream projects are using that to test on S390. It's hardware made available by IBM in a cloud-like setting. Probably not sufficient for our needed testing. We might put a github check there on a low frequency, maybe a couple times a week on main. Someone needs to watch it and address its output for it to be any use. We would need continuous time investment to do the work. Casey thinks downstreams should set up nightly/weekly builds on their own hardware and fix things when they break. building/testing upstream puts the maintenance burden on us. Lab Shutdown - 15 December ========================== David Galloway reminds us that 15 December is when we will start shutting the Sepia lab down. A truck will be arriving two days later to pick everything up and take it tou Poughkeepsie. It is increasingly looking like we will not have access to the stuff in Poughkeepsie at the end of the month as we were told. So it is looking increasingly like there might just be a time where the upstream lab is not available. Floated the idea of pushing this back through the management chain and that did not go well. Unfortunately things did not work out and there will be some downtime. Hopefully the downtime will last no more than a month. Will likely go to Poughkeepsie to to the wiring. While the physical network infrastructure is there and ready, the main networking person there is on medical leave and has not been backfilled yet. There is also Security Stuff to deal with. There are a number of hurdles. It may be feasible to run some Jenkins builders in the cloud. Release order documentation =========================== From Joseph Mundackal (ref: https://ceph-storage.slack.com/archives/C05R975114L/p1763681398991599) Since we were doing a reef point release even though it's EoL, got asked what our criteria for deciding what release to do next were. Do we have that documented anywhere? According to Ilya, the reef release was a result of our n − 2 policy and tentacle being delayed. Expecting the delay, we did the work to get the PRs together and put a Reef out. The listed EoL date is when we expect the next major release to come out. Joseph asked if there would be interest in a PR to resolve the confusion between listed EoL date and actual release activity. Ilya suggested removing the column for expected EoL and replacing it with an explanatory note and agrees the current situation is probably confusing to users. Joseph will raise a PR to update the documentation. Reef Release ============ Yuri did a set of runs for the Reef release and sent an email to the leads asking to approve Erroneous Spam Marking ====================== Laura notes that when you find one of the emails in your spam filter you can mark it as Not Spam. Greg says you need to set up a folder and tell it never to send emails to that list to Spam. David Galloway mentions a few possible tweaks for deliverability next year.

1 0

BLUEFS_SPILLOVER after Reef upgrade
by Matthew Hutchinson 24 Nov '25

24 Nov '25

Hello everyone, I hope you are doing well. My cluster is in a similar situation to the one described in this email thread: <https://www.mail-archive.com/search?l=ceph-users@ceph.io&q=subject:%22%5C%5 Bceph%5C-users%5C%5D+Re%5C%3A+BLUEFS_SPILLOVER+after+Reef+upgrade%22&o=newes t> https://www.mail-archive.com/search?l=ceph-users@ceph.io&q=subject:%22%5C%5B ceph%5C-users%5C%5D+Re%5C%3A+BLUEFS_SPILLOVER+after+Reef+upgrade%22&o=newest This cluster was originally deployed on Octopus and has since been upgraded to the latest version of Reef. We are now seeing the same BLUEFS_SPILLOVER behavior described in that thread. I was wondering if anyone else has encountered this issue and managed to resolve it without having to rebuild the OSDs. Thank you in advance for any guidance or suggestions you can share. Matthew Hutchinson Solution Specialist 1-866-594-7199 ext. 818 <mailto:mhutchinson@45drives.com> mhutchinson(a)45drives.com <http://www.45drives.com/> www.45drives.com

1 0

CLI pool creation with application
by Eugen Block 24 Nov '25

24 Nov '25

Hi, I do most of my Ceph stuff on the command line, although the dashboard can be quite helpful with a lot of stuff, too. But when I create a new pool on a newly deployed cluster I always have to enable the application manually. And since many of those systems are just for testing something quickly (and maybe throw them away afterwards), they aren't attached to any sort of automation (virtual machines in almost air-gapped environments). So I was wondering if there's a chance to add a CLI option to directly set the application during pool creation. The dashboard has that option, but in the background it also just sets the application after pool creation: log [INF] : from='mgr.5307102 192.168.168.50:0/604015211' entity='mgr.host1.vhylmr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "test-appl", "app": "rbd", "yes_i_really_mean_it": true}]': finished Is this something I could add to my wish list? :-) Thanks, Eugen

1 0

Meaning of RGW sync status "full sync: objects completed"
by Adam Prycki 24 Nov '25

24 Nov '25

Hello, I have a question about rgw synchronization mechanic. When we are trying to initialize full bucket sync on existing bucket, or when try to synchonize delete bucket we see "full sync" in bucket sync status on our clusters. For example: realm b6ca44da-921e-42da-b377-80e9fae83b61 (s3-TEST.kmd4.pl) zonegroup c41b36c6-2031-407b-bab4-36cd0b3e56da (poznan-1) zone 91a17d0c-84a0-4706-a40d-1b6df1996a10 (zone-poznan-1-kielman) bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) current time 2025-11-24T13:11:48Z source zone 8e5c54a9-7eb2-4521-b327-081a1a647205 (zone-poznan-1-pcss-bst) source bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) full sync: 15221490 objects completed In this case we have deleted bucket with broken synchronization on the secondary zone. After RGW was started up again synchronization should have tarted from zero. Bucket 1d2848a2-9ca7-402c-9976-a6dc775547bb has 1163263 10MiB s3 objects. bucket sync status is right now at 15221490 and still increasing. That's 13x bigger number than object count in this bucket. In case of some other bucket we saw that "full sync: objects completed" number grew to about 20x s3 object count in the bucket. Can someone explain what this number means and why it gets so big? How big it's supposed to get? Can we predict how much of the synchronization is "done" from this number? Best regards Adam Prycki

1 0

wrong lvs attribut
by Albert Shih 24 Nov '25

24 Nov '25

Hi everyone, I got a very strange issue, after a reboot the lvs attribut of a disk are missing «ao» osd-block-2d30e765-d02e-4dd1-9eee-16c67e539263 ceph-1efb4c4d-f7ac-4511-a593-20db66acaf0d -wi-ao---- 2.91t osd-block-bab12e42-4bbd-4225-a120-bfff72990a90 ceph-24377b3a-1bcc-45f3-8f6c-e7cabb0c212d -wi------- 3.49t osd-block-13630348-cf13-4387-8476-c13ee0e14633 ceph-3c335428-129b-4787-9195-0b67eb6dfba6 -wi-ao---- 2.91t osd-block-187e637e-c235-4e36-99eb-849c30561b09 ceph-8bde9b87-167b-4da8-9447-004ecdfa6a4c -wi-ao---- 2.91t so of course the osd daemon is unable to start. It's not the first time I got this issue, it's not a big deal because (at least until now) a reboot solve the problem. But anyone have any clue why sometime after a reboot the disk (nvme ssd) are in non active and non open state ? Hardware issue ? (I can't remember if it was the same osd). Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: lun. 24 nov. 2025 13:20:53 CET

1 0

Best way to remove a node from cluster temp to replace faulty network cable
by gagan tiwari 24 Nov '25

24 Nov '25

Hi Guys, We are getting an increasing number of RX errors on the ports on some of the nodes in the cluster including mon , mds and osd nodes In an attempt to fix it , we plan to replace SFPs / cables on these nodes. I think I will need to temp remove those nodes one by one from the cluster and replace the network cables / SFPs and add them back to cluster So, plz let me know the best / safest way to proceed. Thanks, Gagan

4 4

CephFS: large omap object
by Philipp Hocke 21 Nov '25

21 Nov '25

Hi, last week we had a "large omap" warning in one of our clusters. The affected object was 025-11-09T23:31:24.799+0000 7f70131b3640 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 2:ddb71591:::mds1_openfiles.d:head PG: 2.89a8edbb (2.b) Key count: 200001 Size (bytes): 12122520 I wanted to get a deeper understanding on what triggers this specific issue and had a look at the code - I'm not that deep into C++, so please, correct me if I'm wrong. If I understand correctly, LARGE_OMAP warnings are only generated (and cleared) during deep-scrubs. If an object is found with more keys than osd_deep_scrub_large_omap_object_key_threshold, it will trigger the warning. In this case mds1_openfiles.d had a Key count of 200001 while this specific option got deep-scrubbed and that triggered the warning The MDS should create a new openfiles fragment when the threshold is reached. My question is: How exactly does the MDS end up with an openfiles segment with more than 200000? Shouldn't the MDS create a new segment as soon as the previous one reaches the configured limit? If this is considered a bug, I'm happy to open a report. Best regards Philipp -- Philipp Hocke Leiter Linux Systemadministration / Head Of TechOps CM4ALL GmbH Im Mediapark 6A - 50670 Köln / Cologne Phone +49-(0)221-6601-0 <https://outlook.office.com/mail/sentitems> Fax +49-(0)221-6601-1011 <https://outlook.office.com/mail/sentitems> E-Mail: philipp.hocke(a)cm4all.com Internet: www.cm4all.com

2 2

Rados data export tool from ERR and unfound object cluster
by Özkan Göksu 21 Nov '25

21 Nov '25

Hello folks. I got a failed cluster recovery job and the cluster is: 2PB EC 8+2 pool used as RGW S3 storage. (14.2.16 nautilus) I fixed most of the issues and rescued a lot of failed OSD's but some OSD's are not recoverable and the cluster size is huge, recovery takes time. Fixing all the issues is impossible because missing OSD's and running PG repair take weeks. I did my best and the current state is enough for starting data export. I only need to export the healthy data first and then I'm gonna destroy-rebuild the cluster. I tried to export data by using rclone and it runs good at the beginning but when the radosgw service hits an unfound object it slows down and it freezes. Because of this issue I can not export data by using rclone+radosgw I'm using these parameters for the export: > rclone copy --files-from=object-list.txt --progress --checkers=8 > --transfers=8 --no-traverse \ > --no-update-modtime --ignore-checksum --ignore-size --no-gzip-encoding > --s3-no-check-bucket \ > --buffer-size=1G --low-level-retries=1 --retries=1 --retries-sleep=100ms > --timeout=30s \ > --contimeout=30s --ignore-errors --ignore-existing > --log-file=export-rclone.log --log-level=INFO Current state is: > # ceph health detail (14.2.16 nautilus) > HEALTH_ERR 3303 large omap objects; 1460729/1238267531 objects unfound > (0.118%); Possible data damage: 2 pgs recovery_unfound, 1 pg > backfill_unfound; Degraded data redundancy: 5487055/12315635595 objects > degraded (0.045%), 7 pgs degraded, 7 pgs undersized; 1987 pgs not > deep-scrubbed in time; 1824 pgs not scrubbed in time > LARGE_OMAP_OBJECTS 3303 large omap objects > 165 large objects found in pool 'xyz.rgw.log' > 3138 large objects found in pool 'xyz.rgw.buckets.index' > Search the cluster log for 'Large omap object found' for more details. > OBJECT_UNFOUND 1460729/1238267531 objects unfound (0.118%) > pg 56.73a has 99125 unfound objects > pg 56.470 has 572028 unfound objects > pg 56.3a9 has 564371 unfound objects > pg 56.346 has 7871 unfound objects > pg 56.36 has 1 unfound objects > pg 56.1e2 has 108269 unfound objects > pg 56.204 has 109064 unfound objects > PG_DAMAGED Possible data damage: 2 pgs recovery_unfound, 1 pg > backfill_unfound > pg 56.36 is active+backfill_unfound+undersized+degraded+remapped, > acting [166,36,7,46,152,193,128,223,80,2147483647], 1 unfound > pg 56.3a9 is active+recovery_unfound+undersized+degraded+remapped, > acting [153,69,177,156,49,167,2,2147483647,2147483647,31], 564371 unfound > pg 56.470 is active+recovery_unfound+undersized+degraded+remapped, > acting [135,2147483647,2147483647,176,37,85,65,48,183,224], 572028 unfound > PG_DEGRADED Degraded data redundancy: 5487055/12315635595 objects degraded > (0.045%), 7 pgs degraded, 7 pgs undersized > pg 56.36 is stuck undersized for 47661.320358, current state > active+backfill_unfound+undersized+degraded+remapped, last acting > [166,36,7,46,152,193,128,223,80,2147483647] > pg 56.1e2 is stuck undersized for 47660.312165, current state > active+recovering+undersized+degraded+remapped, last acting > [2147483647,127,77,140,167,195,220,131,113,31] > pg 56.204 is stuck undersized for 47660.325926, current state > active+recovering+undersized+degraded+remapped, last acting > [2147483647,42,173,17,64,189,22,113,136,158] > pg 56.346 is stuck undersized for 47660.335077, current state > active+recovering+undersized+degraded+remapped, last acting > [63,108,53,140,132,2147483647,183,171,0,98] > pg 56.3a9 is stuck undersized for 4403604.202577, current state > active+recovery_unfound+undersized+degraded+remapped, last acting > [153,69,177,156,49,167,2,2147483647,2147483647,31] > pg 56.470 is stuck undersized for 47661.762140, current state > active+recovery_unfound+undersized+degraded+remapped, last acting > [135,2147483647,2147483647,176,37,85,65,48,183,224] > pg 56.73a is stuck undersized for 47656.126741, current state > active+recovering+undersized+degraded+remapped, last acting > [2147483647,189,85,155,170,106,40,25,222,121] > PG_NOT_DEEP_SCRUBBED 1987 pgs not deep-scrubbed in time I considered "bucket check" but each bucket has 10M objects and 100TB of data. It will take days-weeks and it may fail because of the cluster state. > "rgw.main": { > "size": 106443540756257, > "size_actual": 106463698989056, > "size_utilized": 106443540756257, > "size_kb": 103948770270, > "size_kb_actual": 103968456044, > "size_kb_utilized": 103948770270, > "num_objects": 9855872 > }, I'm thinking of writing a rados exporter script and I need to do this: 1. Read bucket index and have a list of objects. 2. Check the bucket data by each object and eliminate the unfound objects and generate a clean list with healthy objects 3. Export the data directly from rados to an NFS share by combining multipart data. My questions are: 1. Are there any other ways to use radosgw+rclone for fast export from ERROR state cluster? 2. If not, then do you know any rados export tool I can use? I really need your advice. What is the best approach to this? Thank you.

1 0

November Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 20 Nov '25

20 Nov '25

Hey Folks, We will be having a Ceph science/research/big cluster call on Tuesday November 25th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting.If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Virtual event details: November 25th, 2025 9am Central US 15:00 UTC 4pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_20251125 Pad Index: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ See you there! Kevin

1 0

Ceph Community Manager Roundup Email
by Anthony Middleton 20 Nov '25

20 Nov '25

Hello, Ceph Friends! I hope this week is treating you well. Here’s a quick roundup of what’s new in the Ceph community and how you can get involved! *Tentacle (v20.2.0) Is Officially Released!* Ceph Tentacle is out in the wild! A huge thank you to everyone who contributed code, testing, docs, and feedback. This release reflects the hard work and collaboration of so many community members. Congratulations to all of you! Read the full release notes and upgrade guidance: *https://t.ly/CephTentacle20.2 <https://t.ly/CephTentacle20.2>* *Cephalocon 2025 Recordings Are Live* Miss a session in Vancouver? Want to rewatch your favorite talks? All Cephalocon recordings are now available on YouTube. Watch the playlist: *https://t.ly/WatchCephalocon25 <https://t.ly/WatchCephalocon25>* *New Office Hours with Radoslaw Zarzynski* Radoslaw Zarzynski, Software Engineering Expert at IBM, is launching a new biweekly *RADOS Office Hour. *This is an informal space to collaborate on pull requests, ask questions, walk through tricky parts of the codebase, or even do some pair programming. *When:* Every other Friday, 08:00–10:00 AM PT *Where:* Jitsi *Agenda:* Show up and hack together *Calendar:* Add it here: https://calendar.google.com/calendar/b/1?cid=OXRzOWM3bHQ3dTF2aWMyaWp2dnFxbG… *Join the New Ceph Subreddit* Our previous subreddit was banned (long story…), so we’ve launched a new home on Reddit. Join us at *r/ceph_storage*: https://www.reddit.com/r/ceph_storage/ If you have experience getting subreddits reinstated, please reach out: *community-manager(a)ceph.io <community-manager(a)ceph.io>* *New Funding Request Form* We’ve replaced the old email workflow with a much simpler *Asana Funding Request Form*. Your request now goes straight to the Community Manager + Operations Team for faster review. Submit requests here: *https://form.asana.com/?k=7aCHVRhp0x1Ga1nOCXlckQ&d=9283783873717 <https://form.asana.com/?k=7aCHVRhp0x1Ga1nOCXlckQ&d=9283783873717>* (As always, not all requests can be approved, but every submission will receive a detailed response.) *We Need Your Voice: Ceph Community Engagement Survey* This 2-minute survey helps us understand what’s working and what needs improvement across the community. I worked with survey expert Laura Flores to make responses more straightforward to process. Your input matters more than ever. Take the survey: *https://t.ly/Engagement-Survey25 <https://t.ly/Engagement-Survey25>* Thanks for being part of the Ceph community and helping shape what comes next. If you ever have ideas, questions, or just want to chat, feel free to reach out to me. Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

ATTN: Regarding Tentacle packages for Ubuntu 24.04
by Casey Bodley 20 Nov '25

20 Nov '25

The recent Tentacle v20.2.0 release includes packages for Ubuntu 24.04. We've become aware of significant issues in ceph-mgr related to python 3.12 and later, tracked in https://tracker.ceph.com/issues/73930. We don't yet have test coverage for Ubuntu 24.04, so didn't catch this in time for the release. The only workaround for this issue is to deploy ceph-mgr instances as packages on Ubuntu 22.04 or CentOS Stream 9, or as containers using cephadm. For this reason, we've removed all Ubuntu 24.04 packages from 20.2.0. We plan to fully test and support Ubuntu 24.04 and Rocky 10 in a point release.

1 0

mons with "low on available space" message
by filip Mutterer 20 Nov '25

20 Nov '25

Hi! its about my Lab setup. As I didn't have a thought about the space requirements for the root file system I want to know if there is a rule by thumb how much space a cpeh mon might need? rightnow I only have the "whoami" container running in rook-ceph so it does consume nearly no space in the OSD. But how far can I go with only 1.7 GB left on my Raspberry Pi's sdcard? Greetings filip

5 6

Something changed between RHEL 9.6 and 9.7 - unable to activate HDD OSDs with DB/WAL on NVMe
by Torkil Svensgaard 20 Nov '25

20 Nov '25

Hi We just upgraded a host running containerized squid from RHEL 9.6 to RHEL 9.7 and it failed to activate HDD OSDs with DB/WAL on NVMe when running the 9.7 kernel. SATA SSD OSDs were not affected. We rebooted it on the 9.6 kernel and everything worked again. Anyone know what changed? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 1

cephfs purge queue
by Massimo Sgaravatto 20 Nov '25

20 Nov '25

Dear all In our cephfs installation I saw a couple of times a discrepancy between the "ceph df" and the "du -sh" outputs (see also this thread: https://lists.ceph.io\ /hyperkitty/list/ceph-users(a)ceph.io/thread/BBED4ADXO3CE4FYLCNUWB4OML6N6CTZU/ ). I have the feeling that the problem is because of a lot of deletions done by some users (I know that some of them in their jobs use to write from time to time a checkpoint file and then delete the previous checkpoint file) Trying to reproduce the problem, I am running a script with 30 parallel threads where each thread: - writes a 40 GB file - sleeps 5 secs - deletes the file produced in the previous iteration After some hours I have been able to reproduce the issue. Right now "ceph df" shows a usage of ~ 11 TB (~ 33 TB considering the replica 3) while "du -sh" shows a usage of about 2.7 TB. We have 2 active (and 1 standby) MDS instances. In one MDS I see: [root@ceph-mds-01 ~]# ceph --admin-daemon /run/ceph/ceph-mds.ceph-mds-01.asok perf dump | jq '.["purge_queue"]' { "pq_executing_ops": 10240, "pq_executing_ops_high_water": 13202, "pq_executing": 1, "pq_executing_high_water": 16, "pq_executed_ops": 74781922, "pq_executed": 924254, "pq_item_in_journal": 26986 } while in the second one: [root@ceph-mds-02 ~]# ceph --admin-daemon /run/ceph/ceph-mds.ceph-mds-02.asok perf dump | jq '.["purge_queue"]' { "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executed_ops": 0, "pq_executed": 0, "pq_item_in_journal": 0 } I am using the default values for filer_max_purge_ops and "mds_max_purge*: [root@ceph-mds-01 ~]# ceph daemon /run/ceph/ceph-mds.ceph-mds-01.asok config show | grep filer_max_purge "filer_max_purge_ops": "10", [root@ceph-mds-01 ~]# ceph daemon /run/ceph/ceph-mds.ceph-mds-01.asok config show | grep mds_max_purge "mds_max_purge_files": "64", "mds_max_purge_ops": "8192", "mds_max_purge_ops_per_pg": "0.500000", mds_cache_memory_limit is set to 32 GiB and doing a: # ceph daemon mds.<mds> perf dump | grep mds_co_bytes I see for the 2 MDS instances, , respectively: 8237145057 3479698 We are running ceph reef (we will soon update to squid) Should I try to increase filer_max_purge_ops ? Thanks a lot Cheers, Massimo

3 3

Ceph containers image version to pull them in a local registry
by wodel youchi 19 Nov '25

19 Nov '25

Hi, When deploying ceph, several images are used beside ceph itself, like prometheus, grafana, ...etc. When using a local registry, how to get the version of those images to pull them and prepare the registry? Regards.

2 1

ceph.conf on every node
by Albert Shih 19 Nov '25

19 Nov '25

Hi, Just like to know if it's a bad idea (like something going to bite me in few year) to have a copy of the /etc/ceph/ceph.conf from the master copy on every node of the cluster (install with cephadm). I «need» the ceph.conf to able to get the fsid on every node. Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 18 nov. 2025 10:40:25 CET

6 10

Re: Ceph Steering Committee Meeting 2025-11-03
by Neha Ojha 18 Nov '25

18 Nov '25

Thank you for testing 20.2.0! Can you please share which version you upgraded this cluster from? I assume this was a test environment. Thanks, Neha On Mon, Nov 10, 2025 at 2:49 AM Lukasz Borek <lukasz(a)borek.org.pl> wrote: > > Thanks. > > 8 nodes NVME cluster with EC4+2 upgraded. No problem observed. > > Let me know if you want me to grab some logs. > > I will do some testing later this week. > > Thanks. > > NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID > ceph-exporter.fast-cephbackup-mon1 fast-cephbackup-mon1 running (117m) 9m ago 2w 9349k - 20.2.0 524f3da27646 4de9da93f7dc > ceph-exporter.fast-cephbackup-mon2 fast-cephbackup-mon2 running (117m) 9m ago 2w 9118k - 20.2.0 524f3da27646 456692e98ef9 > ceph-exporter.fast-cephbackup-mon3 fast-cephbackup-mon3 running (117m) 9m ago 2w 9172k - 20.2.0 524f3da27646 20c6764da039 > ceph-exporter.fast-cephbackup-osd1 fast-cephbackup-osd1 running (117m) 9m ago 2w 22.3M - 20.2.0 524f3da27646 6e5f442f5dde > ceph-exporter.fast-cephbackup-osd2 fast-cephbackup-osd2 running (117m) 8m ago 2w 22.3M - 20.2.0 524f3da27646 b341d7dd4390 > ceph-exporter.fast-cephbackup-osd3 fast-cephbackup-osd3 running (116m) 8m ago 2w 22.4M - 20.2.0 524f3da27646 1240b6cc66f7 > ceph-exporter.fast-cephbackup-osd4 fast-cephbackup-osd4 running (116m) 6m ago 2w 22.3M - 20.2.0 524f3da27646 df81ed883912 > ceph-exporter.fast-cephbackup-osd5 fast-cephbackup-osd5 running (116m) 6m ago 2w 22.2M - 20.2.0 524f3da27646 1c089585b35d > ceph-exporter.fast-cephbackup-osd6 fast-cephbackup-osd6 running (116m) 4m ago 2w 22.5M - 20.2.0 524f3da27646 3be6d4d8b1da > ceph-exporter.fast-cephbackup-osd7 fast-cephbackup-osd7 running (116m) 3m ago 2w 22.4M - 20.2.0 524f3da27646 cb690c2e1936 > ceph-exporter.fast-cephbackup-osd8 fast-cephbackup-osd8 running (115m) 81s ago 2w 22.5M - 20.2.0 524f3da27646 187336418c0d > crash.fast-cephbackup-mon1 fast-cephbackup-mon1 running (2h) 9m ago 4w 7625k - 20.2.0 524f3da27646 5bc9c4e61b24 > crash.fast-cephbackup-mon2 fast-cephbackup-mon2 running (2h) 9m ago 3w 7629k - 20.2.0 524f3da27646 927fc4633bf8 > crash.fast-cephbackup-mon3 fast-cephbackup-mon3 running (2h) 9m ago 3w 7633k - 20.2.0 524f3da27646 b1d1ad6e5904 > crash.fast-cephbackup-osd1 fast-cephbackup-osd1 running (2h) 9m ago 3w 7671k - 20.2.0 524f3da27646 a07c9ba448b0 > crash.fast-cephbackup-osd2 fast-cephbackup-osd2 running (2h) 8m ago 3w 7646k - 20.2.0 524f3da27646 6ddae0f43fb0 > crash.fast-cephbackup-osd3 fast-cephbackup-osd3 running (2h) 8m ago 3w 7650k - 20.2.0 524f3da27646 41f2c7c9fd33 > crash.fast-cephbackup-osd4 fast-cephbackup-osd4 running (2h) 6m ago 3w 7646k - 20.2.0 524f3da27646 b4953c96bd1b > crash.fast-cephbackup-osd5 fast-cephbackup-osd5 running (2h) 6m ago 3w 7650k - 20.2.0 524f3da27646 ad2a9d34e155 > crash.fast-cephbackup-osd6 fast-cephbackup-osd6 running (2h) 4m ago 3w 7646k - 20.2.0 524f3da27646 89eaa53086c4 > crash.fast-cephbackup-osd7 fast-cephbackup-osd7 running (2h) 3m ago 3w 7646k - 20.2.0 524f3da27646 87fd08e6b274 > crash.fast-cephbackup-osd8 fast-cephbackup-osd8 running (2h) 81s ago 3w 7638k - 20.2.0 524f3da27646 881408f9d76d > mgr.fast-cephbackup-mon1.xfamca fast-cephbackup-mon1 *:8443,9283,8765 running (3h) 9m ago 2w 507M - 20.2.0 524f3da27646 59f0778832bd > mgr.fast-cephbackup-mon2.gzasaj fast-cephbackup-mon2 *:8443,9283,8765 running (3h) 9m ago 3w 865M - 20.2.0 524f3da27646 862a55610d55 > mgr.fast-cephbackup-mon3.trmjnk fast-cephbackup-mon3 *:8443,9283,8765 running (3h) 9m ago 3w 508M - 20.2.0 524f3da27646 7c6c8271e0b2 > mon.fast-cephbackup-mon1 fast-cephbackup-mon1 running (3h) 9m ago 2w 442M 2048M 20.2.0 524f3da27646 b3701a59d934 > mon.fast-cephbackup-mon2 fast-cephbackup-mon2 running (3h) 9m ago 3w 440M 2048M 20.2.0 524f3da27646 c70c96d6f220 > mon.fast-cephbackup-mon3 fast-cephbackup-mon3 running (3h) 9m ago 3w 444M 2048M 20.2.0 524f3da27646 e0465231d5bb > osd.0 fast-cephbackup-osd5 running (2h) 6m ago 3w 14.2G 21.7G 20.2.0 524f3da27646 77d1125f3ded > osd.1 fast-cephbackup-osd3 running (2h) 8m ago 3w 15.6G 21.7G 20.2.0 524f3da27646 6224a7873350 > osd.2 fast-cephbackup-osd6 running (2h) 4m ago 3w 12.6G 21.7G 20.2.0 524f3da27646 173a847e73bf > osd.3 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.8G 21.7G 20.2.0 524f3da27646 515a1b402559 > osd.4 fast-cephbackup-osd2 running (2h) 8m ago 3w 12.6G 21.7G 20.2.0 524f3da27646 555e60b3d272 > osd.5 fast-cephbackup-osd1 running (2h) 9m ago 3w 14.6G 21.7G 20.2.0 524f3da27646 15ec0a8364a1 > osd.6 fast-cephbackup-osd8 running (2h) 81s ago 3w 14.4G 21.7G 20.2.0 524f3da27646 cefe12a100b3 > osd.7 fast-cephbackup-osd4 running (2h) 6m ago 3w 13.7G 21.7G 20.2.0 524f3da27646 5ac5aa3645f7 > osd.8 fast-cephbackup-osd5 running (2h) 6m ago 3w 13.9G 21.7G 20.2.0 524f3da27646 3d9b4fc53385 > osd.9 fast-cephbackup-osd3 running (2h) 8m ago 3w 14.9G 21.7G 20.2.0 524f3da27646 8bd7b4154925 > osd.10 fast-cephbackup-osd6 running (2h) 4m ago 3w 14.0G 21.7G 20.2.0 524f3da27646 037877a08541 > osd.11 fast-cephbackup-osd1 running (2h) 9m ago 3w 14.2G 21.7G 20.2.0 524f3da27646 58659b91ef03 > osd.12 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.3G 21.7G 20.2.0 524f3da27646 c9aec438a8e4 > osd.13 fast-cephbackup-osd8 running (2h) 81s ago 3w 15.1G 21.7G 20.2.0 524f3da27646 d60262f8462d > osd.14 fast-cephbackup-osd2 running (2h) 8m ago 3w 15.8G 21.7G 20.2.0 524f3da27646 9bba8fe67e06 > osd.15 fast-cephbackup-osd4 running (2h) 6m ago 3w 16.2G 21.7G 20.2.0 524f3da27646 847b77855b6f > osd.16 fast-cephbackup-osd5 running (2h) 6m ago 3w 13.3G 21.7G 20.2.0 524f3da27646 736932c6d76a > osd.17 fast-cephbackup-osd3 running (2h) 8m ago 3w 15.9G 21.7G 20.2.0 524f3da27646 4092581a7fe4 > osd.18 fast-cephbackup-osd6 running (2h) 4m ago 3w 15.2G 21.7G 20.2.0 524f3da27646 64c154a002aa > osd.19 fast-cephbackup-osd1 running (2h) 9m ago 3w 15.3G 21.7G 20.2.0 524f3da27646 00e58326a1ef > osd.20 fast-cephbackup-osd8 running (2h) 81s ago 3w 15.6G 21.7G 20.2.0 524f3da27646 22bb3db279d5 > osd.21 fast-cephbackup-osd2 running (2h) 8m ago 3w 15.0G 21.7G 20.2.0 524f3da27646 cc6c38738efd > osd.22 fast-cephbackup-osd4 running (2h) 6m ago 3w 14.5G 21.7G 20.2.0 524f3da27646 2d13e252b00e > osd.23 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.1G 21.7G 20.2.0 524f3da27646 f9e17e6a58fa > osd.24 fast-cephbackup-osd5 running (2h) 6m ago 3w 14.3G 21.7G 20.2.0 524f3da27646 0f8197209d12 > osd.25 fast-cephbackup-osd3 running (2h) 8m ago 3w 14.1G 21.7G 20.2.0 524f3da27646 5589f732c8e7 > osd.26 fast-cephbackup-osd6 running (2h) 4m ago 3w 13.2G 21.7G 20.2.0 524f3da27646 936ef5adcd5b > osd.27 fast-cephbackup-osd5 running (2h) 6m ago 3w 12.3G 21.7G 20.2.0 524f3da27646 0241c249ebab > osd.28 fast-cephbackup-osd3 running (2h) 8m ago 3w 14.4G 21.7G 20.2.0 524f3da27646 0daec562fbee > osd.29 fast-cephbackup-osd6 running (2h) 4m ago 3w 14.3G 21.7G 20.2.0 524f3da27646 1c9dd693d8e0 > osd.30 fast-cephbackup-osd8 running (2h) 81s ago 3w 13.9G 21.7G 20.2.0 524f3da27646 c76825580158 > osd.31 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.2G 21.7G 20.2.0 524f3da27646 4e49578cf239 > osd.32 fast-cephbackup-osd1 running (2h) 9m ago 3w 14.9G 21.7G 20.2.0 524f3da27646 5e49d50a8b93 > osd.33 fast-cephbackup-osd2 running (2h) 8m ago 3w 12.9G 21.7G 20.2.0 524f3da27646 ad90aff39abd > osd.34 fast-cephbackup-osd4 running (2h) 6m ago 3w 15.1G 21.7G 20.2.0 524f3da27646 c5849cdceb23 > osd.35 fast-cephbackup-osd5 running (2h) 6m ago 3w 15.3G 21.7G 20.2.0 524f3da27646 6c870213ae6a > osd.36 fast-cephbackup-osd3 running (2h) 8m ago 3w 11.0G 21.7G 20.2.0 524f3da27646 04ebd6dc1162 > osd.37 fast-cephbackup-osd6 running (2h) 4m ago 3w 13.7G 21.7G 20.2.0 524f3da27646 f700c3738d68 > osd.38 fast-cephbackup-osd1 running (2h) 9m ago 3w 15.5G 21.7G 20.2.0 524f3da27646 1669493ed49c > osd.39 fast-cephbackup-osd8 running (2h) 81s ago 3w 11.9G 21.7G 20.2.0 524f3da27646 bf9c144c445a > osd.40 fast-cephbackup-osd2 running (2h) 8m ago 3w 15.1G 21.7G 20.2.0 524f3da27646 539b78b9c0e8 > osd.41 fast-cephbackup-osd7 running (2h) 3m ago 3w 10.2G 21.7G 20.2.0 524f3da27646 be6466b9a097 > osd.42 fast-cephbackup-osd4 running (2h) 6m ago 3w 15.0G 21.7G 20.2.0 524f3da27646 a5e9904a0bb4 > osd.43 fast-cephbackup-osd5 running (2h) 6m ago 3w 16.0G 21.7G 20.2.0 524f3da27646 e2f62ba83020 > osd.44 fast-cephbackup-osd6 running (2h) 4m ago 3w 15.2G 21.7G 20.2.0 524f3da27646 a8072fa05fa7 > osd.45 fast-cephbackup-osd3 running (2h) 8m ago 3w 14.6G 21.7G 20.2.0 524f3da27646 01d56e9c8a62 > osd.46 fast-cephbackup-osd1 running (2h) 9m ago 3w 11.4G 21.7G 20.2.0 524f3da27646 59cbbb9896ba > osd.47 fast-cephbackup-osd2 running (2h) 8m ago 3w 15.0G 21.7G 20.2.0 524f3da27646 d93301e82e43 > osd.48 fast-cephbackup-osd8 running (2h) 81s ago 3w 14.7G 21.7G 20.2.0 524f3da27646 f61d43751328 > osd.49 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.4G 21.7G 20.2.0 524f3da27646 52447edc6a9b > osd.50 fast-cephbackup-osd4 running (2h) 6m ago 3w 14.9G 21.7G 20.2.0 524f3da27646 49b6112a27d8 > osd.51 fast-cephbackup-osd5 running (2h) 6m ago 3w 14.0G 21.7G 20.2.0 524f3da27646 9229368e0e16 > osd.52 fast-cephbackup-osd6 running (2h) 4m ago 3w 14.5G 21.7G 20.2.0 524f3da27646 eb677be81e9b > osd.53 fast-cephbackup-osd3 running (2h) 8m ago 3w 15.6G 21.7G 20.2.0 524f3da27646 d5de2b9fff4c > osd.54 fast-cephbackup-osd7 running (2h) 3m ago 3w 15.3G 21.7G 20.2.0 524f3da27646 fa076a6e94d0 > osd.55 fast-cephbackup-osd1 running (2h) 9m ago 3w 14.4G 21.7G 20.2.0 524f3da27646 3c1d2089252d > osd.56 fast-cephbackup-osd2 running (2h) 8m ago 3w 14.0G 21.7G 20.2.0 524f3da27646 b29067e714b9 > osd.57 fast-cephbackup-osd8 running (2h) 81s ago 3w 14.1G 21.7G 20.2.0 524f3da27646 a9a5be72ba34 > osd.58 fast-cephbackup-osd4 running (2h) 6m ago 3w 15.4G 21.7G 20.2.0 524f3da27646 abbf11f1e2ea > osd.59 fast-cephbackup-osd7 running (2h) 3m ago 3w 13.6G 21.7G 20.2.0 524f3da27646 ca8df0031018 > osd.60 fast-cephbackup-osd1 running (2h) 9m ago 3w 15.1G 21.7G 20.2.0 524f3da27646 74738c3d930d > osd.61 fast-cephbackup-osd2 running (2h) 8m ago 3w 17.4G 21.7G 20.2.0 524f3da27646 098aa2c69b0f > osd.62 fast-cephbackup-osd8 running (2h) 81s ago 2w 11.6G 21.7G 20.2.0 524f3da27646 113f92904b8f > osd.63 fast-cephbackup-osd4 running (2h) 6m ago 3w 15.2G 21.7G 20.2.0 524f3da27646 70dff6c0798d > rgw.label.fast-cephbackup-osd1.oyqdoe fast-cephbackup-osd1 *:8000 running (2h) 9m ago 3w 648M - 20.2.0 524f3da27646 d83719a402d5 > rgw.label.fast-cephbackup-osd2.lzxiwi fast-cephbackup-osd2 *:8000 running (2h) 8m ago 3w 554M - 20.2.0 524f3da27646 49914023f796 > rgw.label.fast-cephbackup-osd3.bzbawb fast-cephbackup-osd3 *:8000 running (2h) 8m ago 2w 545M - 20.2.0 524f3da27646 81dce830901e > rgw.label.fast-cephbackup-osd4.epcdwr fast-cephbackup-osd4 *:8000 running (2h) 6m ago 3w 595M - 20.2.0 524f3da27646 b7b04e7728cc > rgw.label.fast-cephbackup-osd5.fhzzju fast-cephbackup-osd5 *:8000 running (2h) 6m ago 2w 542M - 20.2.0 524f3da27646 845d6fcf68b6 > rgw.label.fast-cephbackup-osd6.fztzxe fast-cephbackup-osd6 *:8000 running (119m) 4m ago 2w 648M - 20.2.0 524f3da27646 45bb95f840d7 > rgw.label.fast-cephbackup-osd7.asrxjx fast-cephbackup-osd7 *:8000 running (119m) 3m ago 2w 853M - 20.2.0 524f3da27646 0e6bae7f8a14 > rgw.label.fast-cephbackup-osd8.avbbsq fast-cephbackup-osd8 *:8000 running (119m) 81s ago 3w 584M - 20.2.0 524f3da27646 bfd48ec8d147 > > > On Sun, 9 Nov 2025 at 23:31, Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Hello >> >> 20.2.0 image is pre-released >> Please don't share it >> https://quay.io/repository/yuriw/ceph?tab=tags >> v20.2.0-20251104 >> >> Thx for helping! >> >> >> On Sun, Nov 9, 2025 at 1:02 PM Neha Ojha <nojha(a)redhat.com> wrote: >>> >>> Thank you, Lukasz! @yuri can you please share the pre-release image? >>> >>> Thanks, >>> Neha >>> >>> On Sun, Nov 9, 2025 at 12:49 PM Lukasz Borek <lukasz(a)borek.org.pl> wrote: >>>> >>>> Hi, >>>> >>>> Happy to help with testing. I have not yet production cluster with EC that can be upgraded. >>>> >>>> Let me know image detail. >>>> >>>> Łukasz Borek >>>> lukasz(a)borek.org.pl >>>> >>>> >>>> W dniu pt., 7 lis 2025 o 21:12 Neha Ojha <nojha(a)redhat.com> napisał(a): >>>>> >>>>> Special thanks to @Stefan Kooman for testing upgrades for 20.2.0! He >>>>> has successfully upgraded 2 clusters including one with RGW >>>>> mutli-site. >>>>> >>>>> Thanks, >>>>> Neha >>>>> >>>>> >>>>> On Thu, Nov 6, 2025 at 12:31 PM Neha Ojha <nojha(a)redhat.com> wrote: >>>>> > >>>>> > Hi everyone, >>>>> > >>>>> > On Mon, Nov 3, 2025 at 9:28 AM Casey Bodley <cbodley(a)redhat.com> wrote: >>>>> > > >>>>> > > we covered two topics today that generated a lot of good discussion, >>>>> > > but neither reached a consensus: >>>>> > > >>>>> > > AI Code contributions >>>>> > > * current lack of clear policy on submissions containing AI-generated >>>>> > > code. do we want one? what are the concerns/risks? how do we enforce? >>>>> > > * requires further discussion >>>>> > > >>>>> > > Tentacle updates? >>>>> > > * merge conflicts in https://github.com/ceph/ceph/pull/66096 >>>>> > > * lrc upgrade? >>>>> > > ** very risky with pending lab move next week. after the move, lrc may >>>>> > > not be available for upgrade until 2026? >>>>> > > ** should this block the release or not? requires further discussion >>>>> > >>>>> > Are there any community members who would be willing to help test >>>>> > Tentacle 20.2.0 upgrades on their test/staging clusters? This will >>>>> > immensely help us build more confidence in the stability of the >>>>> > release and expedite GA'ing long-awaited features like Fast EC! >>>>> > >>>>> > Thanks, >>>>> > Neha >>>>> > >>>>> > >>>>> > > _______________________________________________ >>>>> > > ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> > > >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > Łukasz Borek > lukasz(a)borek.org.pl

2 3

Ceph kernel page cache behaviour
by Domhnall McGuigan 18 Nov '25

18 Nov '25

Hi all, I've been investigating a problem with tests of our database processing jobs running on cephfs apparently reading several times more data than actually exists in the working set. I haven't had much luck diagnosing the cause so far, but one possibility I'd at least like to rule out is that the kernel cephfs client is discarding pages with cached reads too aggressively and so we're redundantly reading small chunks of data over and over again. Our access patterns are quite unusual -- we read small chunks (perhaps a few hundred kilobytes) from files, then close the file handles before reopening and doing more reads. I found a 10 year old thread indicating that cephfs would outright clear page caches for files in situations like this: https://ceph-users.ceph.narkive.com/o2vJ4rWf/cephfs-and-page-cache That behaviour was patched out long ago, but I'd like to know whether aggressive page cache eviction like this is still known to be an issue with certain access patterns, even if just to eliminate it as a possibility. We're hosting ceph on the same machines as our database which further makes me wonder if some kind of cache pressure could be involved. Regards, Domhnall *********************************************************************************************************************************************************************** This email, its contents and any files attached are a confidential communication and are intended only for the named addressees indicated in the message. If you are not the named addressee or if you have received this email in error, you may not, without the consent of KX, copy, use or rely on any information or attachments in any way. Please notify the sender by return email and delete it from your email system. Unless separately agreed, KX does not accept any responsibility for the accuracy or completeness of the contents of this email or its attachments. Please note that any views, opinion or advice contained in this communication are those of the sending individual and not those of KX and KX shall have no liability whatsoever in relation to this communication (or its content) unless separately agreed. ***********************************************************************************************************************************************************************

2 1

[PSA] MDS running on big-endian machines will break clients operating on fragmented directories
by Patrick Donnelly 17 Nov '25

17 Nov '25

Greetings, For anyone running or planning to run an MDS on a big-endian architecture, please be aware that when directories are fragmented (an automatic process when they become large) that the MDS will send byte-swapped fragment identifiers to clients. This confuses the client and can result in it entering an infinite loop trying to read the contents of a directory. The issue is tracked here: https://tracker.ceph.com/issues/73792 The fix includes code to automatically correct any persisted byte-swapped fragment identifiers. We expect it to be backported to Tentacle and Squid as soon as the fix is in the main branch. Regards, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Ceph Network Design
by Alexander Leutz 14 Nov '25

14 Nov '25

Hi Ceph Users, I need to understand the Ceph network design and the way it communicates with different partners. Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x: ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24 After this I only see this as paramter in the GUI screen, under host / ceph / configuration in the section Configuration Database Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host). Is this correct? I thought, all storage traffic needs to be isolated in my storage network 10.10.x.x - no IPs in the mgmt network for ceph. Or do I have the wrong understanding of the ceph network design? Best regards, Enjoy your weekend, Beste Grüße, Schönes Wochenende, Alexander Concat AG IT Solutions | Berliner Ring 127 - 129 | 64625 Bensheim | www.concat.de alexander.leutz(a)concat.de | Tel.: +49 6251 7026-143 | Mobil: +49 151 18856009 | Fax: +49 6251 7026-356 Hauptsitz Bensheim a. d. Bergstraße | Vorstand Olaf von Heyer Aufsichtsratsvorsitzender Tim Patronik | Amtsgericht Darmstadt (HRB 25326)

4 5

New user with some questions...
by Matthias Riße 14 Nov '25

14 Nov '25

Hey all! I am evaluating Ceph at work and have some questions to people with more experience with it, to avoid making dumb mistakes... I hope this is the right place to ask. So, to give some context, we have 5 machines with about 400 TiB of raw storage total in the form of 8 HDDs per host. The hardware is not all the same, two nodes have a rather low core count which would be dedicated entirely to Ceph, the other three I would like to use in a HCI fashion to run VMs off of Ceph. I'd like to at least be able to take one node at a time out of rotation for maintenance purposes. I'd be using cephadm for the setup, which in my testing works really well. As for the questions: 1. 3x replication is a tough sell IMO. For better storage utilization I am looking at EC. 2+2 would be an obvious choice, and seems to be rather performant. But what about MSR with higher values, e.g. 5+3 with 2 OSDs per host across 4 hosts? Or taken to more of an extreme, 17+7 with 6 OSDs across 4 hosts? This should also allow me to take one host out for maintenance at a time, but has a better storage utilization. Is this a stupid idea? (16+8 might make more sense, to also be able to sustain a disk failure while in maintenance.) 2. I've read that I can update the crush rule of EC pools after the fact, to change both the failure domain as well as the device class of the pool. What about changing k, m, or the plugin type? My understanding is that this is not supported, but Ceph didn't stop me from doing it and it seems to do /something/ when those values are changed? 3. Right now we are using libvirt with qcow2 images on local storage. I know that with Ceph the commonly recommended way would be to use RBD instead, but we have an existing proprietary tape archive for backup purposes whose official client can to my knowledge only do file-based backups, and around which we already have a system to back up live snapshots of VMs. How bad of an idea would it actually be to use qcow2 on top of (kernel- or fuse-mounted?) CephFS? So far it seems to perform on par with RBD in my testing, but both also seem to fully saturate the single OSDs per host I am testing with anyway. 4. In v20 there seems to be a new ec_optimizations feature for ec pools used in CephFS or RBD. Are those a good idea with this kind of (large) qcow2 image workload on top of CephFS? 5. Speaking of v20, while it is not yet the latest "active" version extrapolating the releases suggests that it could soon become that. Should I be waiting for it / start a new cluster with v20 now already? Thanks for any insights you can give! Kind regards Matthias Riße -- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

4 8

Re: Large OMAP & Bucket Not found
by Joo Jaemin 14 Nov '25

14 Nov '25

I integrated ceph with keystone. bucket name "aaa/bbb" means "<keystone user-id>/<bucket-name>" I can't change the bucket name '/'

2 1

How to increase the performance of rgw
by Szabo, Istvan (Agoda) 14 Nov '25

14 Nov '25

Hi, We have multiple gateways behind haproxy but seems like some user can max out some value in rgw which makes the request super slow like 54 sec for a 7kb object: 2025-11-13T01:16:07.367+0700 7f9bb27fc640 1 beast: 0x7f9d24402710: 1.1.1.1 - user [13/Nov/2025:01:15:12.683 +0700] "PUT /002b92966f0b HTTP/1.1" 200 7754 - "aws-sdk-java/1.12.201 Linux/5.15.0-143-generic OpenJDK_64-Bit_Server_VM/21.0.9+10-LTS java/21.0.9 scala/2.12.18 kotlin/1.3.50 vendor/Eclipse_Adoptium cfg/retry-mode/legacy" - latency=54.683818817s The server itself on the network card/CPU/memory didn't spike, on the backend osds I don't see high latency either so would be cusrious which values should be tuned to increase performance for individual gateway. Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 2

Large OMAP & Bucket Not found
by Jaemin Joo 14 Nov '25

14 Nov '25

Hi all, I saw several "Large OMAP Objects" in "ceph -s" And I found bucket "Not found" error about I'm using the bucket, when I executed "rados-admin bucket stats --bucket="755d/abc" e.g.) radosgw-admin bucket stats --bucket="755d/abc" error getting bucket stats bucket=abc ret=-2 Failure (2) no such file or directory I guess that there is the index mapping problem after "reshard" fails. I tried "radosgw-admin bucket check --bucket=755d/abc" --fix". and "radosgw-admin bilog trim --bucket="bucket_name" --bucket-id="bucket_id" but it doesn't work. Is there another solution or diagnostic way?

2 1

ceph-volume /dev/mapper not found even if it exists
by Francesco Di Nucci 13 Nov '25

13 Nov '25

Hi, I have provisioned a Ceph cluster with cephadm and I'm having issues with "ceph-volume inventory" on OSD nodes. It fails with errors like # ceph-volume inventory --> RuntimeError: /dev/mapper/ceph--009df43a--5864--43ca--9507--59a9c1207e08-osd--block--410f9c5c--c83b--4208--9060--fffd4829ae96 not found. Even though that device exists under /dev/mapper I've tried rebooting and node creation with "vgscan --mknodes" but the result is still the same, but nothing. Has anyone encountered the same errors? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

3 4

Are osd-secondary and osd-tertiary copys parallel writes?
by jsterr＠proxforge.de 12 Nov '25

12 Nov '25

Hello Ceph-Users, I could not find a information about this in the docs: Are osd-secondary and osd-tertiary parallel writes? Thanks for the help. Greetings -- Jonas Sterr

3 2

Ceph cluster design advice
by Gustavo Garcia Rondina 12 Nov '25

12 Nov '25

Hello Ceph community, We are provisioning a new Ceph cluster and would appreciate design advice. Below is our current hardware: - 13 data servers each with 18 x 24 TB HDD and 2 x 3.5 TB SSD (6 servers in rack A, 7 in rack B) - 3 control servers each with 2 x 960 GB SSD and 4 x 1.9 TB SSD (2 in rack A, 1 in rack B) The planned topology is: - A single CephFS backed by a data pool - The 3 control servers will run MON/MGR/MDS and other daemons. The 13 data servers will host OSDs and some extra MONs as needed. Current idea (constraints: we cannot add hardware) is on each data server we will create a RAID1 from the two 3.5 TB SSDs and carve it into a 500 GB virtual disk for the OS and a 3 TB virtual disk for DB/WAL (Bluestore). Each HDD will be an OSD, giving 18 x 24 TB = 432 TB raw HDD per data node. 1) Is it acceptable to host WAL/DB on a RAID1 virtual disk made from the two SSDs, or is that a bad idea for performance/reliability? 2) Is 3 TB for DB/WAL per data node (~167 GB per OSD) likely to be sufficient for 18 x 24 TB OSDs, or should we expect to need more DB space given CephFS metadata/object counts? (We've seen a 2% rule cited, that would be ~8.6 TB per node which is much larger) 3) Alternative: keep WAL/DB on the HDDs (i.e., do not separate), and use the 3 TB RAID1 SSD area plus the control servers' SSDs (4 x 1.9 TB each) as SSD OSDs for the CephFS metadata pool. Would that be a reasonable approach? 4) If we do SSD-backed metadata OSDs with 3x replication for the metadata pool, we were thinking of CRUSH domains: domain1 = 6 data servers in rack A (6 x 3 TB SSD OSDs), domain2 = 7 data servers in rack B (7 x 3 TB), domain3 = 3 control servers (12 x 1.9 TB SSD OSDs). Any pitfalls with that CRUSH layout? Any suggestions would be highly appreciated. Thank you, Gustavo

2 5

pools nearfull (because cephfs is lazy in releasing space for deleted files ?)
by Massimo Sgaravatto 12 Nov '25

12 Nov '25

Dear all We have a portion of a Cephfs file system that maps to a ceph pool called cephfs_data_ssd. If I perform a "du -sh" on this portion of the file system, I see that the value matches the "STORED" field of the "ceph df" output for the cephfs_data_ssd pool. So far so good. I set a quota of 4.5 TB for this file system area. During the weekend, this pool (and other pools of the same device class) became nearfull. A "ceph df" showed that the problem was indeed in the the cephfs_data_ssd pool, with a reported usage of 7 TiB of data (21 TiB in replica 3): cephfs_data_ssd 62 32 7.1 TiB 2.02M 21 TiB 89.06 898 GiB This sounds strange to me because I set a quota of 4.5 TB in that area, and because a "du -sh" of the relevant directory showed a usage of 600 GB. When I lowered the disk quota from 4.5 TB to 600 GB, the jobs writing in that area failed (because of disk quota exceeded) and after a while the space was released. The only explanation I can think of is that, as far as I understand, cephfs can take a while to release the space for deleted files (https://docs.ceph.com/en/reef/dev/delayed-delete/) This would also be consistent with the fact that it looks like some jobs were performing a lot of writes and deletions (they kept writing a ~ 5GB checkpoint file, and deleting the previous one after each iteration). How can I understand from the log files if this was indeed the problem ? Or do you have some other possible explanations for this problem ? And, most important, how can I prevent scenarios such as this one ? Thanks, Massimo

3 8

OSD idname already in use
by Francesco Di Nucci 10 Nov '25

10 Nov '25

Hi all, I'm deploying a Ceph cluster with cephadm and I'm having some issues with OSD deploy - they fail with errors like "/bin/podman: stderr stderr: WARNING: adding device /dev/sde with idname naa.5000c500940453db which is already used for /dev/sds", how should I prevent this? (Might it be related to usage of storage arrays?) A sample log is below Related to this - When monitoring logs with "ceph -W cephadm", the operations are retried continously, is "ceph mgr module disable cephadm" enough to disable them while debugging? How can I list the operations orch is trying to apply and in case stop/delete them? SAMPLE LOG 2025-11-04T15:11:13.632450+0000 mgr.my-monmgr01.dospbf [ERR] Failed to apply osd.osd02 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: osd02 service_name: osd.osd02 placement: hosts: - my-osd02.example.com spec: data_devices: size: '4TB:' filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec -e NODE_NAME=my-osd02.example.com -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd02 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/run/ceph:z -v /var/log/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/log/ceph:z -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5hr75pjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpt9txcizn:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec lvm batch --no-auto /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdn /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd /bin/podman: stderr --> passed data devices: 20 physical, 0 LVM /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 6065211e-ba14-4c47-b786-0c4ab3dd6f12 /bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-e90f3c54-857f-41ef-9a2e-5635c64c4365 /dev/sde /bin/podman: stderr stderr: WARNING: adding device /dev/sde with idname naa.5000c500866fdcdb which is already used for /dev/sdq. /bin/podman: stderr stdout: Physical volume "/dev/sde" successfully created. /bin/podman: stderr stdout: Volume group "ceph-e90f3c54-857f-41ef-9a2e-5635c64c4365" successfully created /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.17 --yes-i-really-mean-it /bin/podman: stderr stderr: purged osd.17 /bin/podman: stderr --> No OSD identified by "17" was found among LVM-based OSDs. /bin/podman: stderr --> Proceeding to check RAW-based OSDs. /bin/podman: stderr No OSD were found. Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5581, in <module> File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5569, in main File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 409, in _infer_config File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 437, in _infer_image File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec -e NODE_NAME=my-osd02.example.com -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd02 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/run/ceph:z -v /var/log/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/log/ceph:z -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5hr75pjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpt9txcizn:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec lvm batch --no-auto /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdn /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec -e NODE_NAME=my-osd02.example.com -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd02 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/run/ceph:z -v /var/log/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/log/ceph:z -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5hr75pjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpt9txcizn:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec lvm batch --no-auto /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdn /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd /bin/podman: stderr --> passed data devices: 20 physical, 0 LVM /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 6065211e-ba14-4c47-b786-0c4ab3dd6f12 /bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-e90f3c54-857f-41ef-9a2e-5635c64c4365 /dev/sde /bin/podman: stderr stderr: WARNING: adding device /dev/sde with idname naa.5000c500866fdcdb which is already used for /dev/sdq. /bin/podman: stderr stdout: Physical volume "/dev/sde" successfully created. /bin/podman: stderr stdout: Volume group "ceph-e90f3c54-857f-41ef-9a2e-5635c64c4365" successfully created /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.17 --yes-i-really-mean-it /bin/podman: stderr stderr: purged osd.17 /bin/podman: stderr --> No OSD identified by "17" was found among LVM-based OSDs. /bin/podman: stderr --> Proceeding to check RAW-based OSDs. /bin/podman: stderr No OSD were found. Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5581, in <module> File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5569, in main File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 409, in _infer_config File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 437, in _infer_image File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec -e NODE_NAME=my-osd02.example.com -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd02 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/run/ceph:z -v /var/log/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/log/ceph:z -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5hr75pjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpt9txcizn:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec lvm batch --no-auto /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdn /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd 2025-11-04T15:11:30.613025+0000 mgr.my-monmgr01.dospbf [INF] Detected new or changed devices on my-osd02.example.com Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

2 2

Ceph Steering Committee Meeting 2025-11-03
by Casey Bodley 10 Nov '25

10 Nov '25

we covered two topics today that generated a lot of good discussion, but neither reached a consensus: AI Code contributions * current lack of clear policy on submissions containing AI-generated code. do we want one? what are the concerns/risks? how do we enforce? * requires further discussion Tentacle updates? * merge conflicts in https://github.com/ceph/ceph/pull/66096 * lrc upgrade? ** very risky with pending lab move next week. after the move, lrc may not be available for upgrade until 2026? ** should this block the release or not? requires further discussion

4 4

Weekly Ceph Community Newsletter
by Anthony Middleton 07 Nov '25

07 Nov '25

Hey Ceph Friends, I hope you're all doing well after last week's exciting time in Vancouver for Cephalocon. Videos are coming soon! As a first-time attending Cephalocon, the highlight of the event was meeting so many wonderful community members in person. I felt welcomed into the community, and I look forward to continuing to be a part of it. I have a few updates and opportunities for you to get involved. Let's dive in! Your Voice Matters: Take Our Community Survey! At last week's Cephalocon, Matthew Leonard, Chair of the Ceph Foundation, delivered the Foundation Update covering the good, the bad, and the ugly. He shared insights into how the Foundation fosters innovation and why YOUR participation matters. We need your help to shape the future of the Foundation! Take just *3 minutes* to complete our Ceph Community Engagement Survey. Please share with us what's working, what's not, and what needs improvement. This is your Foundation, and it's only as strong as the people who build it together. Survey link: https://t.ly/Engagement-Survey25 New Slack Channel Alert: #go-ceph Calling all Go developers! We've launched a new *#go-ceph* Slack channel <https://join.slack.com/t/ceph-storage/shared_invite/zt-32hkefbs5-f6qZDZLd5U…>. This is the definitive way to interact with Ceph in Go, and it will only become more important as our Go ecosystem grows. Join the conversation and connect with fellow Gophers working with Ceph! *Do you have suggestions for other Slack channels? Message or email me anytime!* Help Us Build Our Reddit Community We're seeking moderators to help manage our new Ceph subreddit <https://www.reddit.com/r/ceph_storage/>! After our previous subreddit was unfortunately banned due to a bot attack, we're starting fresh and need passionate community members to help keep things running smoothly. Interested in being part of the mod team? *Reach out to me directly* and let's chat! Share Your Story: Ceph Contributors Intake Form Along with the new subreddit, there are many ways to get involved with the Foundation, and here's how to do it. If you're interested in becoming more engaged with the Ceph Community, we have an intake form to help start the process. This form is brief and will let us understand some of the community's needs. The success of our Foundation depends on the support and involvement of our community. Complete the intake form here: https://t.ly/contributors Quick Action Items: - ✅ Complete the Community Engagement Survey <https://t.ly/Engagement-Survey25> (3 minutes!) - ✅ Join the new *#go-ceph* Slack channel - ✅ Contact me to become a Reddit moderator - ✅ Fill out the Contributors Intake Form <https://t.ly/contributors> Thanks for being a fantastic part of the Ceph community! Together, we're building something special. Stay awesome, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph> *Questions, suggestions, or just want to chat? Don't hesitate to reach out!*

1 0

How important is the "default" data pool being replicated for CephFS
by Mikael Öhman 07 Nov '25

07 Nov '25

The recommendations for cephfs is to make a replicated default data pool, and adding any EC data pools using layouts: https://docs.ceph.com/en/latest/cephfs/createfs/ > If erasure-coded pools are planned for file system data, it is best to configure the default as a replicated pool to improve small-object write and read performance when updating backtraces. I have an cephfs that unfortunately wasn't set up like this: they just made an EC pool on the slow HDDs as the default, which sounds like the worst case scenario to me. I would like to add a NVME data pool to this ceph fs, but recommended gives me pause on if i should instead go through the hassle of creating a new cephfs and migrating all users. I've tried to run some mdtest with small 1k files to see if i could measure this difference, but speed is about the same in my relatively small tests so far. I'm also not sure what impact I should realistically expect here. I don't even know if creating files counts as "updating backtraces", so my testing might just be pointless. I guess my core question is; just how important is this suggestion to keep the default data pool on replicated NVME? Setup: 14 hosts x 42 HDD + 3 NVMEs for db/wal 2*2x25 GbitE bonds 12 hosts x 10 NVME. 2*2x100 GbitE bonds Old CephFS setup: - metadata: replicated NVME - data-pools: EC 10+2 on HDD (i plan to add a EC NVME pool here via layouts) New CephFS setup as recommended: - metadata: replicated NVME - data-pools: replicated NVME (default), EC 8+2 on HDD via layout, EC 8+2 on NVME via layout. Ceph 18.2.7 Best regards, Mikael

4 5

HDD replacements with shared NVME lvm for DB - what am I missing here
by Mikael Öhman 07 Nov '25

07 Nov '25

My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per NVME). It's all using ceph orch, containerized setup, using LVMs, so it's probably the most conventional HDD based setup one can do. I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0 But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup: ceph orch apply -i osd_spec_osd02.yml --dry-run ... ################ OSDSPEC PREVIEWS ################ +---------+----------+-------------+---------------------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+-------------+---------------------+----+-----+ |osd |osd_spec |mimer-osd02 |/dev/mapper/mpathau |- |- | +---------+----------+-------------+---------------------+----+-----+ which is the worst outcome, so I have to set all specs to be unmanaged, since otherwise they do the wrong thing automatically. So, most of https://docs.ceph.com/en/squid/cephadm/services/osd/ can just be ignored, since it won't work. Instead I'm stuck with this long complicated procedure, ensuring all specs are unmanaged, recreate the db lvm manually making up a UUID for it so that it matches the rest of the osds, then manually enter a shell, copy over the client.bootstrap-osd keyring, run this huge ceph-volume lvm prepare --bluestore --no-systemd --osd-id 12345 --data /dev/mapper/mpathxx --block.db /dev/ceph-2e401d48-931b-4529-88c0-d36424560xxx/osd-db-61264cc4-0f40-458b-a4d1-08709b919xxx manually adding the daemon, starting it. I don't love this procedure and it's hard to explain all the things that can go wrong to new staff. I really miss the basic step by step of how to perform the most common operation; replacing a hdd with a ssd db partition. I have tried things with --replace and --no-destroy (a flag that isn't documented at all), but I can't understand from the documentation what it wants from me when I have separate DB partitions. -------- A bit of a tangent, butI also don't understand "REJECT REASONS" from ceph orch device ls Insufficient space (<10 extents) on vgs, LVM detected Why even bother looking at extents on VGs if the presence of LVM by itself already rejects the drive. In my case, it does detect that nvme0n1 has sufficient space on the vgs: mimer-osd02 /dev/mapper/mpathau hdd 12.7T Yes 10m ago mimer-osd02 /dev/nvme0n1 ssd KCM61VUL800G_7170A009TM38 745G No 10m ago Has a FileSystem, LVM detected mimer-osd02 /dev/nvme1n1 ssd KCM61VUL800G_7170A004TM38 745G No 10m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected mimer-osd02 /dev/nvme2n1 ssd KCM61VUL800G_7170A007TM38 745G No 10m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected but, "LVM detected" already rejects it. Checking the code (src/ceph-volume/ceph_volume/util_device.py): self.available_lvm, self.rejected_reasons_lvm = self._check_lvm_reject_reasons() self.available_raw, self.rejected_reasons_raw = self._check_raw_reject_reasons() self.available = self.available_lvm and self.available_raw self.rejected_reasons = list(set(self.rejected_reasons_lvm + self.rejected_reasons_raw)) So apparently, the device must be available both as RAW (i.e. have no vgs) and also have enough space on its LVMs (i.e. have enough vgs). This code doesn't make any sense at all. In addition, is also checks disk.blkdid for any TYPE field @property def has_fs(self) -> bool: self.load_blkid_api() return 'TYPE' in self.blkid_api and for a device in lvm it would have something like {'UUID': '3OWtic-3RCI-G3g5-7M6D-pKpb-eMJe-X0zgif', 'TYPE': 'LVM2_member'} so again it doesn't make any sense to ever have _check_lvm_reject_reasons, since it must never be a LVM2_member anyway. Best regards, Mikael

4 7

[reef-18.2.7] radosgw token cache for keystone not working
by Boris 07 Nov '25

07 Nov '25

Hi, I am currently debugging a problem that the radosgw keystone token cache seems not to work properly. Or at all. I tried to debug it and attached the rgw_debug log set to 10. I've truncated to only show the part from "No stored secret string, cache miss" until the request is done. The failed request hits a rate limit on the keystone which currently takes around 2k answered requests per minute. Any ideas what I did wrong? * All requests were done within 10 seconds and were only an ls to show buckets. * This particular RGW only took my requests during testing. * We didn't set any timeouts or special cache configs in ceph * system time is correct First request worked instantly: req 8122732607072897744 0.106001295s s3:list_buckets No stored secret string, cache miss [4.0K blob data] req 8122732607072897744 0.315003842s s3:list_buckets s3 keystone: validated token: 8144848695793469:user-9XGYcbFNUVTQ expires: 1762266594 req 8122732607072897744 0.315003842s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee$a13f0472be744104ad1f64bb2855cdee : hit (negative entry) req 8122732607072897744 0.315003842s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee : hit (requested=0x13, cached=0x13) req 8122732607072897744 0.315003842s s3:list_buckets normalizing buckets and tenants req 8122732607072897744 0.315003842s s->object=<NULL> s->bucket= req 8122732607072897744 0.315003842s s3:list_buckets init permissions req 8122732607072897744 0.315003842s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee : hit (requested=0x13, cached=0x13) req 8122732607072897744 0.315003842s s3:list_buckets recalculating target req 8122732607072897744 0.315003842s s3:list_buckets reading permissions req 8122732607072897744 0.315003842s s3:list_buckets init op req 8122732607072897744 0.315003842s s3:list_buckets verifying op mask req 8122732607072897744 0.315003842s s3:list_buckets verifying op permissions req 8122732607072897744 0.315003842s s3:list_buckets verifying op params req 8122732607072897744 0.315003842s s3:list_buckets pre-executing req 8122732607072897744 0.315003842s s3:list_buckets check rate limiting req 8122732607072897744 0.315003842s s3:list_buckets executing req 8122732607072897744 0.315003842s s3:list_buckets completing req 8122732607072897744 0.315003842s cache get: name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) req 8122732607072897744 0.315003842s s3:list_buckets op status=0 req 8122732607072897744 0.315003842s s3:list_buckets http status=200 ====== req done req=0x74659e51b6f0 op status=0 http_status=200 latency=0.315003842s ====== 2nd request failed req 10422983006485317789 0.061000749s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.keys+05917cf2ee9d4fdea8baf6a3348ca33a : hit (negative entry) req 10422983006485317789 0.061000749s s3:list_buckets error reading user info, uid=05917cf2ee9d4fdea8baf6a3348ca33a can't authenticate req 10422983006485317789 0.061000749s s3:list_buckets Failed the auth strategy, reason=-5 failed to authorize request WARNING: set_req_state_err err_no=5 resorting to 500 req 10422983006485317789 0.061000749s cache get: name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) req 10422983006485317789 0.061000749s s3:list_buckets op status=0 req 10422983006485317789 0.061000749s s3:list_buckets http status=500 ====== req done req=0x74659e51b6f0 op status=0 http_status=500 latency=0.061000749s ====== 3rd requests went through again req 13123970335019889535 0.000000000s s3:list_buckets No stored secret string, cache miss [250B blob data] req 13123970335019889535 0.204002500s s3:list_buckets s3 keystone: validated token: 8144848695793469:user-9XGYcbFNUVTQ expires: 1762266602 req 13123970335019889535 0.204002500s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee$a13f0472be744104ad1f64bb2855cdee : hit (negative entry) req 13123970335019889535 0.204002500s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee : hit (requested=0x13, cached=0x13) req 13123970335019889535 0.204002500s s3:list_buckets normalizing buckets and tenants req 13123970335019889535 0.204002500s s->object=<NULL> s->bucket= req 13123970335019889535 0.204002500s s3:list_buckets init permissions req 13123970335019889535 0.204002500s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee : hit (requested=0x13, cached=0x13) req 13123970335019889535 0.204002500s s3:list_buckets recalculating target req 13123970335019889535 0.204002500s s3:list_buckets reading permissions req 13123970335019889535 0.204002500s s3:list_buckets init op req 13123970335019889535 0.204002500s s3:list_buckets verifying op mask req 13123970335019889535 0.204002500s s3:list_buckets verifying op permissions req 13123970335019889535 0.204002500s s3:list_buckets verifying op params req 13123970335019889535 0.204002500s s3:list_buckets pre-executing req 13123970335019889535 0.204002500s s3:list_buckets check rate limiting req 13123970335019889535 0.204002500s s3:list_buckets executing req 13123970335019889535 0.204002500s s3:list_buckets completing req 13123970335019889535 0.204002500s cache get: name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) req 13123970335019889535 0.204002500s s3:list_buckets op status=0 req 13123970335019889535 0.204002500s s3:list_buckets http status=200 ====== req done req=0x74659e51b6f0 op status=0 http_status=200 latency=0.204002500s ====== -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 15

Snaptrim flood
by Lukasz Gomulka 06 Nov '25

06 Nov '25

Dear all, In the previous year, we were forced to migrate all data from one cluster due to a critical issue with snaptrimq length, which caused the cluster to become unstable. Unfortunately, we are now experiencing the same problem again. We've been flooded with a massive queue of snaptrims despite a massive lack of snapshot deletion. Within seconds, every PG received over 2 million snaps to delete. It looks like some kind of database corruption. 1.1 SNAPTRIMQ_LEN 2264067 Last Year it was Octopus cluster (15.2.17). Now it is cluster in Quincy (17.2.7). Originally cluster was created in version 14.2.16 and upgraded -> 15.2.17 -> 17.2.7. Below I attached background/history in chronological order from earliest. I have also attach logs from snaptrimming osd in text file. ---------- BACKGROUND: All was good: > 2025-10-16T21:30:09.776167+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47659 : cluster [DBG] pgmap v22191: 8193 pgs: 2 active+clean+scrubbing+deep, 8191 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 491 MiB/s rd, 359 MiB/s wr, 20.99k op/s First laggy pg: > 2025-10-16T21:30:11.789813+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47661 : cluster [DBG] pgmap v22192: 8193 pgs: 4 active+clean+laggy, 2 active+clean+scrubbing+deep, 8187 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 328 MiB/s rd, 247 MiB/s wr, 14.01k op/s First hearbeat problem: > 2025-10-16T21:30:21.893+0000 7f578efd7700 -1 osd.yyy 602287 heartbeat_check: no reply from xyx.xyz.xyz.xyz:6802 osd.xxx since back 2025-10-16T21:29:57.827954+0000 front 2025-10-16T21:29:57.828552+0000 (oldest deadline 2025-10-16T21:30:21.328590+0000) First report of failed osd: > 2025-10-16T21:30:21.841714+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235394 : cluster [DBG] osd.16 reported failed by osd.182 First failed osd: > 2025-10-16T21:30:21.841681+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235393 : cluster [INF] osd.15 failed (root=default,rack=rack123,host=host-123) (2 reporters from different host after 23.000420 >= grace 20.000000) Host down: > 2025-10-16T21:30:22.914804+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235451 : cluster [WRN] Health check failed: 1 host (24 osds) down (OSD_HOST_DOWN) Some osd up again: > 2025-10-16T21:30:22.892201+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235450 : cluster [DBG] osd.22 failure report canceled by osd.64 > 2025-10-16T21:30:21.904+0000 7f4c358ca700 1 mon.mon-04-<cluster-fsid>@1(peon).osd e602288 e602288: 283 total, 266 up, 283 in Cluster during recovering: > 2025-10-16T21:30:23.865481+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47668 : cluster [DBG] pgmap v22200: 8193 pgs: 34 peering, 648 stale+active+clean, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 676 active+clean+laggy, 60 active+undersized+degraded, 6773 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 4.7 MiB/s rd, 45 MiB/s wr, 1.28k op/s; 372351/152450688 objects degraded (0.244%) Again all down: > 2025-10-16T21:30:23.936587+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235453 : cluster [DBG] osdmap e602290: 283 total, 259 up, 283 in ... few osds were flapping up and down x few times (about 5) (259->266->259...->259) ... Finally problem was born: >2025-10-16T21:30:34.405+0000 7fe5613fc700 0 log_channel(cluster) log [WRN] : Health check failed: snap trim queue for 4133 pg(s) >= 1024 (mon_osd_snap_trim_queue_warn_on) (PG_SLOW_SNAP_TRIMMING) During problem our monitors with joy were compacting rocksdb: > 2025-10-16T21:30:26.224+0000 7f4c310c1700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting (mon-04-<cluster-fsid>.caas.ovh.net) > 2025-10-16T21:30:26.820+0000 7f0fefa75700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting (mon-05-<cluster-fsid>.caas.ovh.net) ---------- Of course we ended with 2M+ snaps to remove on every pg. Additionally after few minutes we saw such operation on mon: > 2025-10-16T21:37:11.434+0000 7fe5613fc700 -1 mon.mon-01-<cluster-fsid>@0(leader) e10 get_health_metrics reporting 1 slow ops, oldest is pool_op(delete unmanaged snap pool 1 tid 7113677 name v602484) Do you have any idea what might be the reason for this issue and how to address it? Have you encountered a similar problem elsewhere? Currently, we are running snaptrim, but it may take several weeks to complete, and during this time, the cluster is severely impacted. Maybe do You have any idea what to check next, where can we found a clue. Thank You for any help, Best Regards, Lukasz Lucki Gomulka

2 4

squid 19.2.3 - RGW datalog shard 36 missing on secondary
by Daan van Gorkum 05 Nov '25

05 Nov '25

Hi all, Recently I had to replace a whole cluster that was a secondary zone of the rgw realm and zonegroup "main-store". I removed the original zone and added the new one. The sync started (although initially very slow) and now I'm up to about 85% of data replicated to the new cluster. (Data) sync seems to have stalled now (and based on the logs of the primary/master it seems to request the same objects over and over) and the RGW sync instance is spitting out these errors: debug 2025-11-05T15:01:57.165+0000 7fd9d6929640 -1 rgw data changes log: int rgw::cls::fifo::FIFO::push(const DoutPrefixProvider*, const std::vector<ceph::buffer::v15_2_0::list>&, optional_yield):1446 canceled too many times, giving up: tid=88894 debug 2025-11-05T15:01:57.165+0000 7fd9d6929640 -1 rgw data changes log: virtual int RGWDataChangesFIFO::push(const DoutPrefixProvider*, int, RGWDataChangesBE::entries&&, optional_yield): unable to push to FIFO: data_log.36: (125) Operation canceled debug 2025-11-05T15:01:57.165+0000 7fd9d6929640 -1 rgw data changes log: ERROR: svc.cls->timelog.add() returned -125 debug 2025-11-05T15:01:57.165+0000 7fd9d6929640 0 rgw data changes log: ERROR: RGWDataChangesLog::renew_entries returned error r=-125 and debug 2025-11-05T15:01:42.638+0000 7fd9e5146640 -1 rgw async rados processor: virtual int RGWDataChangesFIFO::push(const DoutPrefixProvider*, int, ceph::real_time, const string&, ceph::bu ffer::v15_2_0::list&&, optional_yield): unable to push to FIFO: data_log.36: (125) Operation canceled I figured out this has to do with datalog shard 36 and I can indeed not see the status: bash-5.1$ radosgw-admin datalog status --rgw-realm main-store --shard-id 36 [ { "marker": "", "last_update": "0.000000" } ] 2025-11-05T15:04:10.622+0000 7f3bbf941900 -1 int rgw::cls::fifo::{anonymous}::get_part_info(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, rados::cls::fifo::part_header*, uint64_t, optional_yield):328 fifo::op::GET_PART_INFO failed r=-2 tid=2 2025-11-05T15:04:10.622+0000 7f3bbf941900 -1 int rgw::cls::fifo::FIFO::get_part_info(const DoutPrefixProvider*, int64_t, rados::cls::fifo::part_header*, optional_yield):2023 get_part_info failed: r=-2 tid=2 2025-11-05T15:04:10.622+0000 7f3bbf941900 -1 virtual int RGWDataChangesFIFO::get_info(const DoutPrefixProvider*, int, RGWDataChangesLogInfo*, optional_yield): unable to get part info: data_log.36/6: (2) No such file or directory When checking the rados pool I do see a "data_log.36" object: bash-5.1$ rados -p backup-store-ams.rgw.log ls | grep -E 'data_log.36' data_log.36 And the size seems similar to a neighbor object that's working fine: bash-5.1$ rados -p backup-store-ams.rgw.log stat data_log.36 # Bad backup-store-ams.rgw.log/data_log.36 mtime 2025-10-25T23:47:04.000000+0000, size 156 bash-5.1$ rados -p backup-store-ams.rgw.log stat data_log.35 # Good backup-store-ams.rgw.log/data_log.35 mtime 2025-11-01T12:27:43.000000+0000, size 156 I should add that currently the cluster has a "13 large omap objects" warning activated, but I was thinking this was part of the full resync and sharding happens after completion, it's unclear of this error is related to this warning at all. I've been going through the ceph FIFO source but I've not been able to pinpoint it. The primary cluster is still running OMAP datalog type. Any help/insights would be greatly appreciated! Daan

1 0

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 05 Nov '25

05 Nov '25

Hi Eugen, I have already tried as the below method I stated before. I got the current osdmap and tried to set-osdmap via ceph-object-store-tool. But the command failed with error: ceph osd getmap 72555 > /tmp/osd_map_72555 CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-11/ --op set-osdmap --file /tmp/osd_map_72555 osdmap (#-1:9c8e9ef2:::osdmap.72555:0#) does not exist. https://www.mail-archive.com/ceph-users@ceph.io/msg11545.html BR, Huseyin hcotuk(a)gmail.com

4 4

crc errors on 4 node cluster
by Roland Giesler 05 Nov '25

05 Nov '25

From https://forum.proxmox.com/threads/hyperconverged-cluster-logging-seemingly-… We have 4 nodes (dual Xeon CPU's, 256G RAM, 4 NVMe SSD's, 4 HDD's and dual Melanox 25Gb/s sfp's) in a cluster. I have started noticing apparently random crc errors in the osd logs. Node B, osd.6 2025-10-23T10:32:59.808+0200 7f22a75bf700 0 bad crc in data 3330350463 != exp 677417498 from v1:192.168.131.4:0/3121668685 192.168.131.4 is node D Node B, osd.7 2025-10-23T09:35:12.995+0200 7fbcdbcd7700 0 bad crc in data 3922083958 != exp 3479198006 from v1:192.168.131.2:0/2732728486 192.168.131.2 is node B, which is the node osd.7 is on. and so there are others on other nodes and osd's. From what I understand this means that data copied from some other osd to this one that logs the error fails the crc test. However, I have taken (as a test) one of these ssd's out of the cluster and it tests just fine. I put it back and no crc errors are logged for it. Question: Can something else we causing this? A network connector? It seems pretty random so me, so how can I trace this sources of this? It was suggested to me by die vendor tech support that turning offloading off on the nic: /ethtool -K vmbr0 rx off tx off tso off gso off gro off lro off./ That doesn't seem to make a difference however. The next step would be to check for a MTU/jumbo mismatch. Some details: ceph version 17.2.7 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 62.29883 root default -3 15.17824 host FT1-NodeA 2 hdd 1.86029 osd.2 up 1.00000 1.00000 3 hdd 1.86029 osd.3 up 1.00000 1.00000 4 hdd 1.86029 osd.4 up 1.00000 1.00000 5 hdd 1.86589 osd.5 up 1.00000 1.00000 0 ssd 0.93149 osd.0 up 1.00000 1.00000 28 ssd 3.30690 osd.28 up 1.00000 1.00000 29 ssd 3.49309 osd.29 up 1.00000 1.00000 -7 16.10413 host FT1-NodeB 10 hdd 1.86029 osd.10 up 1.00000 1.00000 11 hdd 1.86029 osd.11 up 1.00000 1.00000 26 hdd 1.86029 osd.26 up 1.00000 1.00000 27 hdd 1.86029 osd.27 up 1.00000 1.00000 6 ssd 0.93149 osd.6 up 1.00000 1.00000 7 ssd 0.93149 osd.7 up 1.00000 1.00000 25 ssd 3.49309 osd.25 up 1.00000 1.00000 41 ssd 3.30690 osd.41 up 1.00000 1.00000 -10 15.84383 host FT1-NodeC 14 hdd 1.59999 osd.14 up 1.00000 1.00000 15 hdd 1.86029 osd.15 up 1.00000 1.00000 16 hdd 1.86029 osd.16 up 1.00000 1.00000 17 hdd 1.86029 osd.17 up 1.00000 1.00000 8 ssd 0.93149 osd.8 up 1.00000 1.00000 9 ssd 0.93149 osd.9 up 1.00000 1.00000 24 ssd 3.49309 osd.24 up 1.00000 1.00000 43 ssd 3.30690 osd.43 up 1.00000 1.00000 -13 15.17264 host FT1-NodeD 20 hdd 1.86029 osd.20 up 1.00000 1.00000 21 hdd 1.86029 osd.21 up 1.00000 1.00000 22 hdd 1.86029 osd.22 up 1.00000 1.00000 23 hdd 1.86029 osd.23 up 1.00000 1.00000 12 ssd 3.30690 osd.12 up 1.00000 1.00000 13 ssd 0.93149 osd.13 up 1.00000 1.00000 19 ssd 3.49309 osd.19 up 1.00000 1.00000 The spinners have their RocksDB on the nvme's, for extra performance, but I have not noticed any crc errors prior to installing the 4TB Samsungs in each node. Suggestions are welcome. thanks Roland

3 3

PSA: pad.ceph.com may have lost updates since ~10/27
by Patrick Donnelly 04 Nov '25

04 Nov '25

It was discovered the CSC notes pad is missing recent updates from this week's meeting. Please share if you have also observed this in one of the pads you use. This is probably related to the recent migration to AWS. https://lists.ceph.io/hyperkitty/list/sepia@ceph.io/thread/ECODNJ2EJH35QWUJ… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

demons being stray?
by lejeczek 04 Nov '25

04 Nov '25

Hi guys. Is there a way to "clean" those up, both orderly & not manners would be ok. -> $ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.02539 root default -3 0.34180 host podster1 9 hdd 0.04880 osd.9 up 1.00000 1.00000 10 hdd 0.29300 osd.10 up 1.00000 1.00000 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 up 1.00000 1.00000 4 hdd 0.29300 osd.4 up 1.00000 1.00000 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 up 1.00000 1.00000 5 hdd 0.29300 osd.5 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000 3 0 osd.3 down 0 1.00000 6 0 osd.6 down 0 1.00000 7 0 osd.7 down 0 1.00000 8 0 osd.8 down 0 1.00000 ID CLASS WEIGHT TYPE NAME -1 1.02539 root default -3 0.34180 host podster1 9 hdd 0.04880 osd.9 10 hdd 0.29300 osd.10 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 4 hdd 0.29300 osd.4 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 5 hdd 0.29300 osd.5 -> $ ceph node ls ... "osd": { "podster1.mine.priv": [ 0, 3, 6, 7, 8, 9, 10 ], "podster2.mine.priv": [ 2, 4 ], "podster3.mine.priv": [ 1, 5 ] }, ... many thanks, L.

2 4

Mismatching Ceph package versions running alongside
by Demian Romeijn 04 Nov '25

04 Nov '25

Hi ceph-users, We're currently in the process of reviewing our current Ceph S3/FS cluster and were debating on adjusting Ceph versions. At this very moment we are running Ubuntu 22.04 with the official Ceph packages from download-ceph-com repository (Reef 18.2.4), however we'd like to change this out for Proxmox VE Ceph, primarily because of the extra bug-fixes & cherry picks that the regular Ceph packages sometimes lack. Also for the built-in support for ZFS which we'd like to change our boot-disks to. Our question at the moment is; 'Can we run these Ceph versions together, probably mismatched between different Reef versions (e.g. Ceph official 18.2.4 - PveCeph 18.2.7)'. Unfortunately we've not got any extra hardware laying around that we could prepare and simply `ceph-volume lvm activate all`, it has to be migrated node-per-node. Is this hypothetically possible to do without any major implecations? Thanks in advance! - Demian

1 0

Pool total sizes increase when deleteing RBDs.
by Konold, Martin 03 Nov '25

03 Nov '25

Hi, I experienced unexpected behaviour. When entirely deleting all rbds on a ceph pool (3/2) not only the used space decreased as expected but the total size of the pool increased significantly. IMHO it is caused by wrong calculations when osds are of slightly different sizes. Kind regards, --martin konold ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

1 0