August 2025 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

Debian Packages for Trixie
by Andrew 09 Feb '26

09 Feb '26

Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew

6 6

DriveGroup Spec question
by Robert Sander 17 Oct '25

17 Oct '25

Hi, we have OSD nodes currently consisting of two 605GB SSDs and six 18TB HDDs. The hosts have room for twelve HDDs. We created a drivegroup spec that looks like this: spec: block_db_size: 100GB data_devices: rotational: true size: '18TB:' db_devices: rotational: false size: '550GB:650GB' db_slots: 6 Initially this creates 6 OSDs with their RocksDB+WAL on the SSDs, 3 each which is nice for load balancing. But when we add another HDD it gets a 17.9TB data volume and a 100GB DB volume, both on the HDD: sdm 8:192 0 18T 0 disk ├─ceph--846e1a59--aff6--4ef8--9b71--de7241531677-osd--block--026e8cef--123d--47d9--9b30--211f94edf96c 252:16 0 17.9T 0 lvm └─ceph--846e1a59--aff6--4ef8--9b71--de7241531677-osd--db--88c47d0b--f5c6--4cec--8909--c5f8036ca459 252:17 0 100G 0 lvm I would have assumed that the remaining 305GB on the SSDs would be used. How do we achieve this? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 9

tls certs per manager - does it work?
by lejeczek 16 Oct '25

16 Oct '25

Hi guys. I thought I had a problem with restarting dashboard - following in docs on ssl certs. But ! I might have other issue - restarting dashboard does what is expected of - but, with these: -> $ ceph dashboard set-ssl-certificate-key podster2.mine.priv -i /root/podster2.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster2.mine.priv -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key podster1.mine.priv -i /root/podster1.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster1.mine.priv -i /root/podster1.mine.priv.crt If I go to podster2 I get podster1's cert, perhaps these do not do anything, If I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster2.mine.priv.key then podster1 has podster2's cert - naturally - but (weirdly?) if go to podster2 (by FQDN) then I get redirected to IP (as in URL) of when I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster1.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster1.mine.priv.key and I go to podster2 then, no redirection, "only" warning about cert being of podster1's, podster1 has a good cert! If I do (again) - having podster1 have "correct" cert, with no per-node/manager cert - per-node/mgr cert, as shown earlier, then.. again, podster1 has its correct cert, podster2 has podster1's cert (no ! redirection to IP though). How much of an "issue" this might be - given the fact that I deployed anew cluster twice (lab) - and this reproduces each time. Or, I'm missing some obvious, big picture - and it's simply bad luck, deploying cluster twice. ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) all thoughts are much appreciated. many thanks, L.

2 9

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Per-RBD-image stats
by William David Edwards 18 Sep '25

18 Sep '25

Hi, What do people use to store per-RBD-image I/O stats? To retrospectively see which VM was keeping a Ceph cluster busy in terms of I/O, I'd like to track `rbd perf image iostat` and `rbd perf image iotop`-like metrics. I've looked at the LibreNMS plugin [1], Prometheus metrics in the manager [2], and Ceph ingestion in Elasticsearch [3]. None of them seem to export per-RBD metrics. I can't find much information about this use case on the mailing list either, though it's hard for me to imagine I'm the only one with this want. How do other users handle this? William David Edwards [1]: https://github.com/librenms/librenms-agent/blob/master/agent-local/ceph [2]: https://docs.ceph.com/en/latest/mgr/prometheus/ [3]: https://www.elastic.co/docs/reference/integrations/ceph

5 4

Subject : Account-Level API Support in Ceph RGW for Production Use
by Dhivya G 17 Sep '25

17 Sep '25

Dear Ceph Community, I hope this message finds you well. We are currently working on integrating Ceph RGW into our system and have encountered a challenge regarding account-level management via API. At present, we are using radosgw-admin commands programmatically within our code to manage and retrieve account-level information. While this approach works, we understand that relying on radosgw-admin is not recommended for production environments due to its direct interaction with the backend and potential impact on performance and stability. We have explored the available documentation and interfaces but could not find a dedicated Account-level API (e.g., for managing users, retrieving account metadata, etc.) that can be safely used in production over RESTful protocols (such as S3 or Swift-compatible APIs). We are seeking clarification on the following points: Is there an officially supported Account API in Ceph RGW that allows account-level operations via REST or other production-safe interfaces? If not, is there a roadmap or recommended alternative for performing these actions in a production-ready manner? Are there any best practices or tools from the community that can bridge this gap? We would greatly appreciate any guidance, documentation, or recommendations you can share . Best regards Dhivya G https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

3 2

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

Re: snap-schedule not running
by Sophonet 04 Sep '25

04 Sep '25

Quick update / follow-up question: In the mgr logs I have seen that the schedule is indeed running, but snapshots cannot be created due to an error when cephfs.LibCephFS.mkdir ist called (Operation not permitted). Apparently, the scheduler is not using client.admin as ID (for the keyring when connecting to the target filesystem), since "ceph auth get client.admin" shows "caps mds = allow *". I have not found any documentation about setting the client.ID for a schedule. Any hints how the permission error can be avoided? Thanks, sophonet (I am posting this update for the third time now, hoping that the two earlier attempts that did not show up in the list are not just delayed...)

2 3

squid 19.2.2 - added disk does not reflect in available capacity on pools
by Steven Vacaroaia 03 Sep '25

03 Sep '25

Hi, I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers ) My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data ) It seems it is not happening ...yet ?! Is it because the peering is still going ? Many thanks Steven Below are outputs from ceph -s ceph df ceph osd df tree cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore 1 failed cephadm daemon(s) 569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time 39 daemons have recently crashed services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi mds: 19/19 daemons up, 7 standby osd: 181 osds: 181 up (since 4d), 181 in (since 14h); 2770 remapped pgs flags noautoscale data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%) 4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling io: client: 84 B/s rd, 2.8 KiB/s wr, 0 op/s rd, 1 op/s wr recovery: 384 MiB/s, 128 objects/s --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54 nvme_class 293 TiB 293 TiB 216 GiB 216 GiB 0.07 ssd_class 587 TiB 424 TiB 163 TiB 163 TiB 27.80 TOTAL 2.2 PiB 1.5 PiB 742 TiB 742 TiB 32.64 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 0 93 TiB ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1 107 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 4 KiB 48 GiB 5.2 TiB 71.71 2.20 422 up osd.107 110 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 4 KiB 45 GiB 5.3 TiB 71.30 2.18 421 up osd.110 116 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 4 KiB 43 GiB 5.3 TiB 71.50 2.19 424 up osd.116 123 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 14 KiB 44 GiB 5.5 TiB 70.20 2.15 414 up osd.123 130 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 56 GiB 4.2 TiB 76.99 2.36 451 up osd.130 137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 2 nvme_class 13.97260 1.00000 14 TiB 14 GiB 906 MiB 3.9 GiB 8.9 GiB 14 TiB 0.10 0.00 300 up osd.2 8 nvme_class 13.97260 1.00000 14 TiB 7.2 GiB 846 MiB 2.5 GiB 3.9 GiB 14 TiB 0.05 0.00 293 up osd.8 15 nvme_class 13.97260 1.00000 14 TiB 11 GiB 887 MiB 3.3 GiB 6.9 GiB 14 TiB 0.08 0.00 288 up osd.15 23 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 6 KiB 14 GiB 5.2 TiB 26.28 0.81 106 up osd.23 30 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.4 GiB 5.0 TiB 27.73 0.85 114 up osd.30 37 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 6 KiB 15 GiB 5.2 TiB 26.28 0.80 104 up osd.37 44 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.3 GiB 5.1 TiB 27.34 0.84 111 up osd.44 52 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 4 KiB 15 GiB 5.1 TiB 26.35 0.81 104 up osd.52 59 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.79 0.85 107 up osd.59 66 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 4 KiB 14 GiB 5.1 TiB 26.29 0.81 104 up osd.66 72 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.8 TiB 4 KiB 15 GiB 5.1 TiB 26.68 0.82 107 up osd.72 79 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.1 TiB 27.05 0.83 111 up osd.79 86 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 4 KiB 10 GiB 5.0 TiB 28.49 0.87 116 up osd.86 93 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 5.5 GiB 5.1 TiB 27.29 0.84 112 up osd.93 100 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.19 0.86 114 up osd.100 -15 345.49374 - 345 TiB 106 TiB 104 TiB 8.7 GiB 523 GiB 239 TiB 30.69 0.94 - host ceph-host-2 115 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 7 KiB 43 GiB 5.9 TiB 68.17 2.09 402 up osd.115 122 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 2 KiB 45 GiB 5.4 TiB 70.72 2.17 420 up osd.122 134 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 12 KiB 52 GiB 4.7 TiB 74.54 2.28 440 up osd.134 141 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 54 GiB 4.3 TiB 76.58 2.35 447 up osd.141 144 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 4 KiB 56 GiB 4.5 TiB 75.77 2.32 443 up osd.144 146 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 7 KiB 52 GiB 4.4 TiB 76.13 2.33 449 up osd.146 159 hdd_class 18.19040 1.00000 18 TiB 203 GiB 200 GiB 10 KiB 2.4 GiB 18 TiB 1.09 0.03 6 up osd.159 176 hdd_class 18.19040 1.00000 18 TiB 135 GiB 133 GiB 6 KiB 1.4 GiB 18 TiB 0.72 0.02 4 up osd.176 177 hdd_class 18.19040 1.00000 18 TiB 74 GiB 73 GiB 6 KiB 1.7 GiB 18 TiB 0.40 0.01 2 up osd.177 178 hdd_class 18.19040 1.00000 18 TiB 238 GiB 236 GiB 7 KiB 2.3 GiB 18 TiB 1.28 0.04 5 up osd.178 179 hdd_class 18.19040 1.00000 18 TiB 76 GiB 75 GiB 6 KiB 828 MiB 18 TiB 0.41 0.01 2 up osd.179 180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180 6 nvme_class 13.97260 1.00000 14 TiB 11 GiB 903 MiB 2.9 GiB 7.0 GiB 14 TiB 0.08 0.00 282 up osd.6 13 nvme_class 13.97260 1.00000 14 TiB 9.4 GiB 1.1 GiB 2.7 GiB 5.6 GiB 14 TiB 0.07 0.00 300 up osd.13 20 nvme_class 13.97260 1.00000 14 TiB 11 GiB 862 MiB 3.1 GiB 7.2 GiB 14 TiB 0.08 0.00 288 up osd.20 26 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.58 0.88 108 up osd.26 32 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.21 0.86 107 up osd.32 41 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.23 0.86 113 up osd.41 48 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 11 KiB 16 GiB 5.0 TiB 28.19 0.86 114 up osd.48 53 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 15 GiB 5.0 TiB 28.21 0.86 107 up osd.53 61 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.22 0.86 112 up osd.61 68 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.19 0.86 107 up osd.68 74 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.56 0.88 114 up osd.74 82 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.79 0.85 106 up osd.82 89 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.85 0.85 108 up osd.89 95 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 11 KiB 16 GiB 5.0 TiB 28.58 0.88 112 up osd.95 104 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.83 0.85 108 up osd.104 -13 345.49374 - 345 TiB 108 TiB 106 TiB 9.5 GiB 423 GiB 238 TiB 31.16 0.95 - host ceph-host-3 114 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 4 KiB 42 GiB 6.3 TiB 66.07 2.02 389 up osd.114 121 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 9 KiB 44 GiB 5.9 TiB 67.77 2.08 400 up osd.121 127 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 14 KiB 48 GiB 4.7 TiB 74.32 2.28 437 up osd.127 133 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 54 GiB 4.0 TiB 78.33 2.40 461 up osd.133 140 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 2 KiB 50 GiB 4.4 TiB 76.20 2.33 447 up osd.140 143 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 50 GiB 4.4 TiB 76.39 2.34 446 up osd.143 154 hdd_class 18.19040 1.00000 18 TiB 281 GiB 279 GiB 9 KiB 2.3 GiB 18 TiB 1.51 0.05 8 up osd.154 158 hdd_class 18.19040 1.00000 18 TiB 725 GiB 722 GiB 20 KiB 3.0 GiB 17 TiB 3.89 0.12 22 up osd.158 163 hdd_class 18.19040 1.00000 18 TiB 696 GiB 692 GiB 22 KiB 3.2 GiB 18 TiB 3.73 0.11 22 up osd.163 167 hdd_class 18.19040 1.00000 18 TiB 458 GiB 455 GiB 15 KiB 2.4 GiB 18 TiB 2.46 0.08 11 up osd.167 171 hdd_class 18.19040 1.00000 18 TiB 788 GiB 785 GiB 25 KiB 3.1 GiB 17 TiB 4.23 0.13 23 up osd.171 175 hdd_class 18.19040 1.00000 18 TiB 518 GiB 515 GiB 22 KiB 3.0 GiB 18 TiB 2.78 0.09 15 up osd.175 4 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 1.1 GiB 2.8 GiB 5.2 GiB 14 TiB 0.06 0.00 283 up osd.4 11 nvme_class 13.97260 1.00000 14 TiB 10 GiB 848 MiB 3.0 GiB 6.4 GiB 14 TiB 0.07 0.00 289 up osd.11 19 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 922 MiB 3.6 GiB 4.6 GiB 14 TiB 0.06 0.00 289 up osd.19 25 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.8 GiB 5.0 TiB 28.42 0.87 113 up osd.25 31 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.05 0.86 110 up osd.31 38 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.1 GiB 5.1 TiB 27.34 0.84 110 up osd.38 45 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.1 GiB 5.0 TiB 27.74 0.85 108 up osd.45 51 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 14 GiB 5.0 TiB 28.16 0.86 111 up osd.51 58 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 12 GiB 5.1 TiB 27.03 0.83 105 up osd.58 65 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.04 0.83 109 up osd.65 73 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.1 TiB 27.03 0.83 109 up osd.73 80 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.9 GiB 5.0 TiB 28.41 0.87 110 up osd.80 87 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.8 GiB 5.0 TiB 28.42 0.87 114 up osd.87 94 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.10 0.86 114 up osd.94 101 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.2 GiB 5.0 TiB 28.07 0.86 113 up osd.101 -11 345.49374 - 345 TiB 107 TiB 105 TiB 8.2 GiB 439 GiB 239 TiB 30.95 0.95 - host ceph-host-4 108 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 4 KiB 43 GiB 6.1 TiB 67.05 2.05 394 up osd.108 113 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 7 KiB 47 GiB 6.3 TiB 65.92 2.02 387 up osd.113 120 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 2 KiB 40 GiB 6.2 TiB 66.60 2.04 389 up osd.120 126 hdd_class 18.43300 0.95001 18 TiB 15 TiB 15 TiB 7 KiB 58 GiB 3.6 TiB 80.37 2.46 473 up osd.126 132 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 52 GiB 4.1 TiB 78.00 2.39 457 up osd.132 139 hdd_class 18.43300 1.00000 18 TiB 15 TiB 14 TiB 14 KiB 53 GiB 3.9 TiB 78.87 2.42 463 up osd.139 151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151 156 hdd_class 18.19040 1.00000 18 TiB 205 GiB 203 GiB 12 KiB 1.9 GiB 18 TiB 1.10 0.03 6 up osd.156 161 hdd_class 18.19040 1.00000 18 TiB 853 GiB 849 GiB 4 KiB 3.9 GiB 17 TiB 4.58 0.14 26 up osd.161 165 hdd_class 18.19040 1.00000 18 TiB 569 GiB 566 GiB 22 KiB 2.7 GiB 18 TiB 3.05 0.09 15 up osd.165 169 hdd_class 18.19040 1.00000 18 TiB 891 GiB 888 GiB 24 KiB 3.3 GiB 17 TiB 4.78 0.15 25 up osd.169 173 hdd_class 18.19040 1.00000 18 TiB 591 GiB 589 GiB 24 KiB 2.6 GiB 18 TiB 3.17 0.10 19 up osd.173 3 nvme_class 13.97260 1.00000 14 TiB 10 GiB 929 MiB 2.8 GiB 6.3 GiB 14 TiB 0.07 0.00 283 up osd.3 10 nvme_class 13.97260 1.00000 14 TiB 8.7 GiB 872 MiB 2.6 GiB 5.3 GiB 14 TiB 0.06 0.00 271 up osd.10 17 nvme_class 13.97260 1.00000 14 TiB 9.5 GiB 904 MiB 2.8 GiB 5.8 GiB 14 TiB 0.07 0.00 298 up osd.17 24 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.3 GiB 5.0 TiB 28.11 0.86 112 up osd.24 33 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.41 0.87 113 up osd.33 40 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.8 GiB 5.0 TiB 28.44 0.87 115 up osd.40 47 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 14 GiB 5.1 TiB 27.03 0.83 106 up osd.47 55 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.22 0.86 108 up osd.55 60 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.06 0.83 111 up osd.60 67 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.3 GiB 5.1 TiB 27.28 0.84 110 up osd.67 75 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 109 up osd.75 83 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 16 GiB 5.1 TiB 27.05 0.83 105 up osd.83 88 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.0 GiB 5.0 TiB 28.42 0.87 109 up osd.88 98 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.7 GiB 5.0 TiB 28.41 0.87 114 up osd.98 103 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.7 GiB 5.0 TiB 28.07 0.86 107 up osd.103 -3 290.92255 - 291 TiB 104 TiB 103 TiB 8.7 GiB 452 GiB 187 TiB 35.81 1.10 - host ceph-host-5 105 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 53 GiB 5.3 TiB 71.10 2.18 417 up osd.105 109 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 7 KiB 47 GiB 5.1 TiB 72.56 2.22 428 up osd.109 117 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 6 KiB 47 GiB 5.3 TiB 71.49 2.19 422 up osd.117 124 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 52 GiB 5.2 TiB 71.87 2.20 424 up osd.124 129 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 2 KiB 50 GiB 4.4 TiB 75.91 2.33 449 up osd.129 136 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 11 KiB 49 GiB 4.9 TiB 73.30 2.25 432 up osd.136 181 hdd_class 18.19040 1.00000 18 TiB 109 GiB 108 GiB 7 KiB 846 MiB 18 TiB 0.59 0.02 3 up osd.181 182 hdd_class 18.19040 1.00000 18 TiB 356 GiB 354 GiB 12 KiB 2.6 GiB 18 TiB 1.91 0.06 11 up osd.182 183 hdd_class 18.19040 1.00000 18 TiB 155 GiB 154 GiB 10 KiB 1.8 GiB 18 TiB 0.83 0.03 5 up osd.183 1 nvme_class 13.97260 1.00000 14 TiB 12 GiB 875 MiB 3.2 GiB 7.7 GiB 14 TiB 0.08 0.00 290 up osd.1 7 nvme_class 13.97260 1.00000 14 TiB 7.4 GiB 1.2 GiB 2.6 GiB 3.7 GiB 14 TiB 0.05 0.00 285 up osd.7 14 nvme_class 13.97260 1.00000 14 TiB 11 GiB 837 MiB 3.0 GiB 7.6 GiB 14 TiB 0.08 0.00 290 up osd.14 21 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.2 GiB 5.1 TiB 27.24 0.83 106 up osd.21 28 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.1 TiB 27.08 0.83 105 up osd.28 35 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.05 0.83 104 up osd.35 42 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.9 GiB 5.0 TiB 27.72 0.85 107 up osd.42 49 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.19 0.86 112 up osd.49 56 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.2 GiB 5.0 TiB 28.10 0.86 107 up osd.56 63 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.10 0.86 111 up osd.63 70 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.4 GiB 5.0 TiB 28.01 0.86 112 up osd.70 77 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 109 up osd.77 84 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 104 up osd.84 90 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.42 0.87 114 up osd.90 96 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 104 up osd.96 -9 345.49374 - 345 TiB 107 TiB 106 TiB 9.7 GiB 446 GiB 238 TiB 31.02 0.95 - host ceph-host-6 111 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 6 KiB 45 GiB 6.4 TiB 65.30 2.00 384 up osd.111 118 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 11 KiB 52 GiB 6.0 TiB 67.51 2.07 397 up osd.118 128 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 12 KiB 50 GiB 4.4 TiB 75.96 2.33 445 up osd.128 135 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 48 GiB 4.7 TiB 74.68 2.29 440 up osd.135 142 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 56 GiB 4.1 TiB 77.54 2.38 454 up osd.142 145 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 55 GiB 4.4 TiB 75.98 2.33 445 up osd.145 153 hdd_class 18.19040 1.00000 18 TiB 169 GiB 167 GiB 7 KiB 1.7 GiB 18 TiB 0.90 0.03 5 up osd.153 157 hdd_class 18.19040 1.00000 18 TiB 488 GiB 486 GiB 14 KiB 2.7 GiB 18 TiB 2.62 0.08 15 up osd.157 162 hdd_class 18.19040 1.00000 18 TiB 440 GiB 437 GiB 17 KiB 2.6 GiB 18 TiB 2.36 0.07 11 up osd.162 166 hdd_class 18.19040 1.00000 18 TiB 852 GiB 849 GiB 24 KiB 3.3 GiB 17 TiB 4.57 0.14 25 up osd.166 170 hdd_class 18.19040 1.00000 18 TiB 473 GiB 470 GiB 22 KiB 2.5 GiB 18 TiB 2.54 0.08 14 up osd.170 174 hdd_class 18.19040 1.00000 18 TiB 673 GiB 670 GiB 20 KiB 3.0 GiB 18 TiB 3.61 0.11 21 up osd.174 5 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 846 MiB 2.9 GiB 5.3 GiB 14 TiB 0.06 0.00 276 up osd.5 12 nvme_class 13.97260 1.00000 14 TiB 12 GiB 873 MiB 3.5 GiB 7.8 GiB 14 TiB 0.08 0.00 296 up osd.12 18 nvme_class 13.97260 1.00000 14 TiB 15 GiB 857 MiB 3.3 GiB 11 GiB 14 TiB 0.10 0.00 288 up osd.18 27 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.5 GiB 5.0 TiB 28.44 0.87 114 up osd.27 34 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.11 0.86 113 up osd.34 39 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.09 0.86 108 up osd.39 46 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.5 GiB 5.1 TiB 27.65 0.85 107 up osd.46 54 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.0 GiB 5.1 TiB 27.70 0.85 107 up osd.54 62 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.5 GiB 5.0 TiB 28.49 0.87 109 up osd.62 69 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.2 GiB 5.0 TiB 28.08 0.86 113 up osd.69 76 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.78 0.85 111 up osd.76 81 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.1 GiB 5.0 TiB 28.46 0.87 115 up osd.81 91 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.0 TiB 27.82 0.85 108 up osd.91 97 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.0 GiB 5.0 TiB 28.47 0.87 109 up osd.97 102 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.11 0.86 108 up osd.102 -5 345.49374 - 345 TiB 106 TiB 105 TiB 8.8 GiB 435 GiB 239 TiB 30.73 0.94 - host ceph-host-7 106 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 2 KiB 47 GiB 5.8 TiB 68.70 2.10 406 up osd.106 112 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 6 KiB 51 GiB 5.8 TiB 68.47 2.10 403 up osd.112 119 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 54 GiB 5.7 TiB 69.31 2.12 407 up osd.119 125 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 4 KiB 45 GiB 5.7 TiB 68.98 2.11 410 up osd.125 131 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 57 GiB 4.2 TiB 77.12 2.36 454 up osd.131 138 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 58 GiB 4.1 TiB 77.85 2.38 456 up osd.138 150 hdd_class 18.19040 1.00000 18 TiB 133 GiB 131 GiB 7 KiB 1.5 GiB 18 TiB 0.71 0.02 4 up osd.150 155 hdd_class 18.19040 1.00000 18 TiB 48 GiB 46 GiB 9 KiB 2.0 GiB 18 TiB 0.26 0.01 2 up osd.155 160 hdd_class 18.19040 1.00000 18 TiB 873 GiB 869 GiB 4 KiB 3.6 GiB 17 TiB 4.69 0.14 27 up osd.160 164 hdd_class 18.19040 1.00000 18 TiB 850 GiB 847 GiB 4 KiB 3.2 GiB 17 TiB 4.57 0.14 25 up osd.164 168 hdd_class 18.19040 1.00000 18 TiB 696 GiB 693 GiB 19 KiB 3.2 GiB 18 TiB 3.74 0.11 17 up osd.168 172 hdd_class 18.19040 1.00000 18 TiB 852 GiB 848 GiB 2 KiB 3.7 GiB 17 TiB 4.57 0.14 25 up osd.172 0 nvme_class 13.97260 1.00000 14 TiB 10 GiB 858 MiB 2.8 GiB 6.3 GiB 14 TiB 0.07 0.00 293 up osd.0 9 nvme_class 13.97260 1.00000 14 TiB 9.3 GiB 847 MiB 2.9 GiB 5.6 GiB 14 TiB 0.07 0.00 281 up osd.9 16 nvme_class 13.97260 1.00000 14 TiB 10 GiB 897 MiB 3.1 GiB 6.3 GiB 14 TiB 0.07 0.00 288 up osd.16 22 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.2 GiB 5.1 TiB 27.28 0.84 107 up osd.22 29 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.42 0.87 112 up osd.29 36 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.08 0.86 109 up osd.36 43 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.42 0.87 115 up osd.43 50 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 5.6 GiB 5.1 TiB 27.28 0.84 111 up osd.50 57 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.3 GiB 5.1 TiB 27.32 0.84 112 up osd.57 64 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.45 0.87 113 up osd.64 71 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.46 0.87 109 up osd.71 78 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.08 0.83 105 up osd.78 85 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.9 GiB 5.0 TiB 28.50 0.87 114 up osd.85 92 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.41 0.87 109 up osd.92 99 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.77 0.85 112 up osd.99 TOTAL 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17

2 6

[CEPH-MDS] directory access blocked while looping subtree empty export
by 何钧 30 Aug '25

30 Aug '25

Hi, all Our cluster has encountered a CephFS metadata-related failure, which causes user requests to access a specific directory (/danlu/ProjectStorage/, with approximately 1.4 billion objects) to get stuck. In the MDS logs, there is a recurring loop of printouts for all dir_frags of certain directories. This issue may be caused by frequent changes we made to the filesystem subtree export pin. Can it be resolved by resetting the MDS MAP using the command ceph fs reset <fs name> --yes-i-really-mean-it? 1. Failure Timeline 1.0 Our CephFS is deployed with multiple MDS nodes. We manually control MDS load balancing using static pins, with a total of 11 nodes (Rank 0–10). Among them, MDS Rank 4 (pinned to /danlu/ProjectStorage/) bears a relatively large volume of metadata. 1.1 Abnormal user access was detected, and slow METADATA occurred on MDS 4. 1.2 MDS 4 was restarted. During the MDS recovery process (replay or rejoin phase), excessive memory usage occurred, exceeding the physical machine’s memory limit (384 GB) and causing repeated OOM (Out-of-Memory) errors. 1.3 New nodes (Rank 11, 12, 13) were added, but the OOM issue persisted. 1.4 MDS Rank 4 was replaced with a server with larger memory. It started normally and entered the active state. 1.5 When traversing the extended attributes of files under Rank 4, it was found that the /danlu/ProjectStorage/3990/ directory contained approximately 80 million objects. For load balancing purposes, the /danlu/ProjectStorage/3990 subtree was statically pinned to the newly added Rank 11 node. 1.6 During the export of data from Rank 4 to Rank 11, MDS 4 restarted automatically because the mds_beacon_grace parameter was set too short (120 seconds). Subsequently, both Rank 4 and Rank 11 entered an abnormal state (stuck in rejoin or clientreplay). 1.7 By configuring the following parameters, Rank 4 and Rank 11 returned to the active state (all parameters were reverted later), but the /danlu/ProjectStorage/ directory still failed to provide services: mds_beacon_grace=3600 mds_bal_interval=0 mds_wipe_sessions true mds_replay_unsafe_with_closed_session true 1.8 Attempted to adjust the static pin of /danlu/ProjectStorage/ from Rank 4 to Rank 11. 1.9 Current stable state: Checking the process status and logs of Rank 4 and Rank 11 shows that the md_submit thread has a 100% CPU utilization rate. Logs indicate that Rank 4 is exporting the /danlu/ProjectStorage/3990 subtree to Rank 11, but Rank 11 reports an empty export error and routes the request back to Rank 4. 2. FS Status ceph fs status dl_cephfs - 186 clients ========= +------+--------+---------------------------+---------------+-------+-------+ | Rank | State  |            MDS            |    Activity   |  dns  |  inos | +------+--------+---------------------------+---------------+-------+-------+ |  0   | active |        ceph-prod-45       | Reqs:    5 /s |  114k |  113k | |  1   | active |        ceph-prod-46       | Reqs:    2 /s |  118k |  117k | |  2   | active |        ceph-prod-47       | Reqs:   11 /s | 5124k | 5065k | |  3   | active |        ceph-prod-10       | Reqs:  117 /s | 2408k | 2396k | |  4   | active |        ceph-prod-01       | Reqs:    0 /s | 31.5k | 26.9k | |  5   | active |        ceph-prod-02       | Reqs:    0 /s | 80.0k | 78.9k | |  6   | active |        ceph-prod-11       | Reqs:    2 /s | 1145k | 1145k | |  7   | active |        ceph-prod-57       | Reqs:    1 /s |  169k |  168k | |  8   | active |        ceph-prod-44       | Reqs:   33 /s | 10.1M | 10.1M | |  9   | active |        ceph-prod-20       | Reqs:    4 /s |  196k |  195k | |  10  | active |        ceph-prod-43       | Reqs:    2 /s | 1758k | 1751k | |  11  | active |        ceph-prod-48       | Reqs:    0 /s | 2879k | 2849k | |  12  | active |        ceph-prod-60       | Reqs:    0 /s | 1875  | 1738  | |  13  | active | fuxi-aliyun-ceph-res-tmp3 | Reqs:    3 /s | 80.8k | 60.3k | +------+--------+---------------------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ |       Pool      |   type   |  used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 2109G | 2741G | |   cephfs_data   |   data   | 1457T |  205T | +-----------------+----------+-------+-------+ +--------------------------+ |       Standby MDS        | +--------------------------+ | fuxi-aliyun-ceph-res-tmp | +--------------------------+ 3. MDS Ops 3.1 MDS.4 {     "description": "rejoin:client.92313222:258",     "initiated_at": "2025-08-30 05:24:02.110611",     "age": 9389.4134921530003,     "duration": 9389.4136841620002,     "type_data": {         "flag_point": "dispatched",         "reqid": "client.92313222:258",         "op_type": "no_available_op_found",         "events": [             {                 "time": "2025-08-30 05:24:02.110611",                 "event": "initiated"             },             {                 "time": "2025-08-30 05:24:02.110611",                 "event": "header_read"             },             {                 "time": "2025-08-30 05:24:02.110610",                 "event": "throttled"             },             {                 "time": "2025-08-30 05:24:02.110626",                 "event": "all_read"             },             {                 "time": "2025-08-30 05:24:02.110655",                 "event": "dispatched"             }         ]     } } 3.2 MDS.11 {     "ops": [         {             "description": "rejoin:client.87797615:61578323",             "initiated_at": "2025-08-29 23:36:22.696648",             "age": 38670.621505474002,             "duration": 38670.621529709002,             "type_data": {                 "flag_point": "dispatched",                 "reqid": "client.87797615:61578323",                 "op_type": "no_available_op_found",                 "events": [                     {                         "time": "2025-08-29 23:36:22.696648",                         "event": "initiated"                     },                     {                         "time": "2025-08-29 23:36:22.696648",                         "event": "header_read"                     },                     {                         "time": "2025-08-29 23:36:22.696646",                         "event": "throttled"                     },                     {                         "time": "2025-08-29 23:36:22.696659",                         "event": "all_read"                     },                     {                         "time": "2025-08-29 23:36:23.303274",                         "event": "dispatched"                     }                 ]             }         }     ],     "num_ops": 1 } 4. MDS Logs 4.1 mds.4 --- Start of Loop --- 2025-08-29 21:42:35.749 7fcfa3d45700  7 mds.4.migrator export_dir [dir 0x1000340af7a.00001* /danlu/ProjectStorage/ [2,head] auth{0=3,11=2} pv=173913808 v=173913806 cv=0/0 dir_auth=4 ap=2+1 state=1610613760|exporting f(v3168 91=0+91) n(v15319930 rc2025-07-17 14:10:02.703809 b6931715137150 1953080=1912827+40253) hs=1+0,ss=0+0 dirty=1 | request=1 child=1 subtree=1 subtreetemp=0 replicated=1 dirty=1 authpin=1 0x558fb40f8500] to 11 2025-08-29 21:42:35.749 7fcfa3d45700  7 mds.4.migrator already exporting 2025-08-29 21:42:35.749 7fcfa3d45700  7 mds.4.migrator export_dir [dir 0x1000340af7a.00110* /danlu/ProjectStorage/ [2,head] auth{0=3,11=2} pv=173913735 v=173913732 cv=0/0 dir_auth=4 ap=3+3 state=1610613760|exporting f(v3168 102=0+102) n(v15319926 rc2025-07-24 13:53:23.519020 b448426565839 422059=420844+1215) hs=3+0,ss=0+0 dirty=1 | request=1 child=1 subtree=1 subtreetemp=0 replicated=1 dirty=1 authpin=1 0x558fb40f8a00] to 11 2025-08-29 21:42:35.749 7fcfa3d45700  7 mds.4.migrator already exporting 2025-08-29 21:42:35.749 7fcfa3d45700  7 mds.4.migrator export_dir [dir 0x1000340af7a.00111* /danlu/ProjectStorage/ [2,head] auth{0=3,1=2,2=2,10=2,11=2,13=2} pv=173913731 v=173913730 cv=0/0 dir_auth=4 ap=2+1 state=1610613760|exporting f(v3168 102=0+102) n(v15319926 rc2025-08-27 12:17:12.186080 b25441350802673 324852174=320337481+4514693) hs=2+0,ss=0+0 | request=1 child=1 subtree=1 subtreetemp=0 replicated=1 dirty=1 authpin=1 0x558fb40f8f00] to 11 2025-08-29 21:42:35.749 7fcfa3d45700  7 mds.4.migrator already exporting 2025-08-29 21:42:35.749 7fcfa3d45700  7 mds.4.migrator export_dir [dir 0x1000340af7a.01001* /danlu/ProjectStorage/ [2,head] auth{0=3,1=2,2=2,3=2,6=2,7=2,8=3,9=2,10=2,11=2,13=2} pv=173914304 v=173914298 cv=0/0 dir_auth=4 ap=4+21 state=1610613760|exporting f(v3168 101=0+101) n(v15319926 rc2025-08-28 10:44:08.113080 b96173795872233 155213668=143249296+11964372) hs=2+0,ss=0+0 dirty=2 | request=1 child=1 subtree=1 subtreetemp=0 replicated=1 dirty=1 authpin=1 0x558fb40f9400] to 11 2025-08-29 21:42:35.749 7fcfa3d45700  7 mds.4.migrator already exporting ... ommited many already exporting for other dir_frags... --- End of Loop --- 4.2 mds.11 --- Start of Loop --- 2025-08-29 21:42:31.991 7f6ab9899700  2 mds.11.cache Memory usage:  total 50859980, rss 25329048, heap 331976, baseline 331976, 0 / 2849528 inodes have caps, 0 caps, 0 caps per inode 2025-08-29 21:42:31.991 7f6ab9899700  7 mds.11.server recall_client_state: min=100 max=1048576 total=0 flags=0xa 2025-08-29 21:42:31.991 7f6ab9899700  7 mds.11.server recalled 0 client caps. 2025-08-29 21:42:31.991 7f6abc09e700  0 log_channel(cluster) log [WRN] : 2 slow requests, 0 included below; oldest blocked for > 30395.764013 secs 2025-08-29 21:42:31.995 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1895) from client.91614761 2025-08-29 21:42:32.703 7f6abe0a2700  5 mds.ceph-prod-01 handle_mds_map old map epoch 1656906 <= 1656906, discarding 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1743) from client.91533406 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1737) from client.76133646 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1739) from client.91667655 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1895) from client.92253020 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1895) from client.92348807 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1895) from client.91480971 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1751) from client.87797615 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1739) from client.68366166 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 1390) from client.66458847 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 506) from client.91440537 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 309) from client.91117710 2025-08-29 21:42:32.703 7f6abe0a2700  3 mds.11.server handle_client_session client_session(request_renewcaps seq 283) from client.92192971 2025-08-29 21:42:32.703 7f6abe0a2700  7 mds.11.locker handle_file_lock a=nudge on (inest mix->lock(2) g=4 dirty) from mds.4 [inode 0x5002aada391 [...2,head] /danlu/ProjectStorage/1352/L36/screenTouchHtml/2025-08-27/ auth{4=2,10=3} v621993 pv621997 ap=4 dirtyparent f(v1 m2025-08-28 10:37:07.785514 1=0+1) n(v2 rc2025-08-28 10:37:07.809515 b29740 4=1+3) (inest mix->lock(2) g=4 dirty) (ifile mix->lock(2) w=1 flushing scatter_wanted) (iversion lock) | dirtyscattered=2 lock=1 importing=0 dirfrag=1 dirtyrstat=0 dirtyparent=1 replicated=1 dirty=1 authpin=1 0x5579af8b5c00] 2025-08-29 21:42:32.703 7f6abe0a2700  7 mds.11.locker handle_file_lock trying nudge on (inest mix->lock(2) g=4 dirty) on [inode 0x5002aada391 [...2,head] /danlu/ProjectStorage/1352/L36/screenTouchHtml/2025-08-27/ auth{4=2,10=3} v621993 pv621997 ap=4 dirtyparent f(v1 m2025-08-28 10:37:07.785514 1=0+1) n(v2 rc2025-08-28 10:37:07.809515 b29740 4=1+3) (inest mix->lock(2) g=4 dirty) (ifile mix->lock(2) w=1 flushing scatter_wanted) (iversion lock) | dirtyscattered=2 lock=1 importing=0 dirfrag=1 dirtyrstat=0 dirtyparent=1 replicated=1 dirty=1 authpin=1 0x5579af8b5c00] 2025-08-29 21:42:32.991 7f6ab9899700  7 mds.11.cache trim bytes_used=6GB limit=32GB reservation=0.05% count=0 2025-08-29 21:42:32.991 7f6ab9899700  7 mds.11.cache trim_lru trimming 0 items from LRU size=2879068 mid=1961726 pintail=0 pinned=76602 2025-08-29 21:42:32.991 7f6ab9899700  7 mds.11.cache trim_lru trimmed 0 items 2025-08-29 21:42:32.991 7f6ab9899700  7 mds.11.migrator export_empty_import [dir 0x5002aef450c.100110000001011* /danlu/ProjectStorage/3990/scrapy_html/ [2,head] auth{4=2} v=228873 cv=0/0 dir_auth=11 state=1073741824 f(v1 m2025-06-10 11:04:10.789649 1091=1091+0) n(v8 rc2025-06-10 11:04:10.789649 b133525839 1091=1091+0) hs=0+0,ss=0+0 | subtree=1 subtreetemp=0 replicated=1 0x557627968000] 2025-08-29 21:42:32.991 7f6ab9899700  7 mds.11.migrator  really empty, exporting to 4 2025-08-29 21:42:32.991 7f6ab9899700  7 mds.11.migrator exporting to mds.4 empty import [dir 0x5002aef450c.100110000001011* /danlu/ProjectStorage/3990/scrapy_html/ [2,head] auth{4=2} v=228873 cv=0/0 dir_auth=11 state=1073741824 f(v1 m2025-06-10 11:04:10.789649 1091=1091+0) n(v8 rc2025-06-10 11:04:10.789649 b133525839 1091=1091+0) hs=0+0,ss=0+0 | subtree=1 subtreetemp=0 replicated=1 0x557627968000] 2025-08-29 21:42:32.991 7f6ab9899700  7 mds.11.migrator export_dir [dir 0x5002aef450c.100110000001011* /danlu/ProjectStorage/3990/scrapy_html/ [2,head] auth{4=2} v=228873 cv=0/0 dir_auth=11 state=1073741824 f(v1 m2025-06-10 11:04:10.789649 1091=1091+0) n(v8 rc2025-06-10 11:04:10.789649 b133525839 1091=1091+0) hs=0+0,ss=0+0 | subtree=1 subtreetemp=0 replicated=1 0x557627968000] to 4 ... ommited many export_empty_import for other dir_frags... --- End of Loop, and all export_empty_imports above will be printed --- 5. Cluster informations Version: v1.42.21 Current Config: [global] ...cluster IPs omitted # update on Sep,26 2021 rbd_cache = false rbd_op_threads = 4 mon_osd_nearfull_ratio = 0.89 mds_beacon_grace = 300 mon_max_pg_per_osd = 300 [mon] mon_allow_pool_delete = true mon osd allow primary affinity = true mon_osd_cache_size = 40000 rocksdb_cache_size = 5368709120 debug_mon = 5/5 mon_lease_ack_timeout_factor=4 mon_osd_min_down_reporters = 3 [mds] mds_cache_memory_limit = 34359738368 #mds_cache_memory_limit = 23622320128 mds_bal_min_rebalance = 1000 mds_log_max_segments = 256 mds_session_blacklist_on_evict = false mds_session_blacklist_on_timeout = false mds_cap_revoke_eviction_timeout = 300 [osd] # update on Sep,26 2021 osd_op_threads = 32 osd_op_num_shards = 16 osd_op_num_threads_per_shard_hdd = 1 osd_op_num_threads_per_shard_ssd = 2 #rbd_op_threads = 16 osd_disk_threads = 16 filestore_op_threads = 16 osd_scrub_min_interval=1209600 osd_scrub_max_interval=2592000 osd_deep_scrub_interval=2592000 osd_scrub_begin_hour = 23 osd_scrub_end_hour = 18 osd_scrub_chunk_min = 5 osd_scrub_chunk_max = 25 osd_max_scrubs=2 osd_op_queue_cut_off = high osd_max_backfills = 4 bluefs_buffered_io = true #bluestore_fsck_quick_fix_threads = 2 osd_max_pg_per_osd_hard_ratio = 6 osd_crush_update_on_start = false osd_scrub_during_recovery = true osd_repair_during_recovery = true bluestore_cache_trim_max_skip_pinned = 10000 osd_heartbeat_grace = 60 osd_heartbeat_interval = 60 osd_op_thread_suicide_timeout = 2000 osd_op_thread_timeout = 90 [client] rgw cache enabled = true rgw cache lru size = 100000 rgw thread pool size = 4096 rgw_max_concurrent_requests = 4096 rgw override bucket index max shards = 32 rgw lifecycle work time = 00:00-24:00 Best Regards, Jun He

1 0

snap-schedule not running
by Sophonet 29 Aug '25

29 Aug '25

Hi list, following the official documentation on https://docs.ceph.com/en/reef/cephfs/snap-schedule/ or other sources like https://www.pivert.org/cephfs-snapshots/ I have tried to set up a snap-schedule for a cephfs without subvolumes on one of its hosts. The command "ceph fs snap-schedule add /PATH 1h --fs=FSNAME" succeeds, and "ceph fs snap-schedule status ..." returns a scheduled snapshot, but it never gets executed, so the entries "first", "last" etc. are always null, even after having activated the schedule manually with "ceph fs snap-schedule activate /PATH --fs=FSNAME". In addition to setting the interval to one hour I have also provided a start time in oder to reduce the time waiting for the first sync. The log files in /var/log/ceph/*, e.g. the ones with mds, do not contain any errors. "ceph fs status" reports all filesystems with green status "active" and there are no other ceph health issues reported. I am running ceph squid on proxmox 9 (Debian Bookworm). Any idea how to continue debugging, where to look for problems (e.g. services not being turned on etc.)? It would be great to use snap-schedules since the interface, also for retention time, is exactly what I need. Are there any working configurations besides the info on docs. ceph.com that can be used as blueprints? Thanks, sophonet

1 0

Re: ceph-users Digest, Vol 134, Issue 88
by renjianxinlover 29 Aug '25

29 Aug '25

hi, faced with a strange problem, one of multiple mds keeps in resolve state. following logs repeat ``` ... 2025-08-29 20:01:39.582 7f8656fe3700 1 mds.ceph-prod-60 Updating MDS map to version 1655087 from mon.3 2025-08-29 20:01:39.582 7f8656fe3700 10 mds.ceph-prod-60 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2025-08-29 20:01:39.582 7f8656fe3700 10 mds.ceph-prod-60 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2025-08-29 20:01:39.582 7f8656fe3700 10 mds.ceph-prod-60 my gid is 91681643 2025-08-29 20:01:39.582 7f8656fe3700 10 mds.ceph-prod-60 map says I am mds.12.1646436 state up:resolve 2025-08-29 20:01:39.582 7f8656fe3700 10 mds.ceph-prod-60 msgr says I am [v2:7.33.104.23:6988/1418417955,v1:7.33.104.23:6989/1418417955] 2025-08-29 20:01:39.582 7f8656fe3700 10 mds.ceph-prod-60 handle_mds_map: handling map as rank 12 2025-08-29 20:01:39.606 7f86527da700 10 mds.12.cache cache not ready for trimming 2025-08-29 20:01:40.606 7f86527da700 10 mds.12.cache cache not ready for trimming 2025-08-29 20:01:40.698 7f86547de700 5 mds.beacon.ceph-prod-60 Sending beacon up:resolve seq 6940 2025-08-29 20:01:40.698 7f86597e8700 5 mds.beacon.ceph-prod-60 received beacon reply up:resolve seq 6940 rtt 0 2025-08-29 20:01:41.606 7f86527da700 10 mds.12.cache cache not ready for trimming ... ``` mds status look like ``` ========= +------+---------+---------------------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+---------+---------------------------+---------------+-------+-------+ | 0 | active | ceph-prod-45 | Reqs: 0 /s | 113k | 112k | | 1 | active | ceph-prod-46 | Reqs: 0 /s | 114k | 113k | | 2 | active | ceph-prod-47 | Reqs: 50 /s | 3967k | 3924k | | 3 | active | ceph-prod-10 | Reqs: 14 /s | 2402k | 2391k | | 4 | active | ceph-prod-02 | Reqs: 0 /s | 31.6k | 27.0k | | 5 | active | ceph-prod-48 | Reqs: 0 /s | 357k | 356k | | 6 | active | ceph-prod-11 | Reqs: 0 /s | 1144k | 1144k | | 7 | active | ceph-prod-57 | Reqs: 0 /s | 168k | 168k | | 8 | active | ceph-prod-44 | Reqs: 30 /s | 5007k | 5007k | | 9 | active | ceph-prod-20 | Reqs: 0 /s | 195k | 195k | | 10 | active | ceph-prod-43 | Reqs: 0 /s | 1757k | 1750k | | 11 | active | ceph-prod-01 | Reqs: 0 /s | 2879k | 2849k | | 12 | resolve | ceph-prod-60 | | 652 | 655 | | 13 | active | fuxi-aliyun-ceph-res-tmp3 | Reqs: 0 /s | 79.9k | 59.6k | +------+---------+---------------------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 2110G | 2738G | | cephfs_data | data | 1457T | 205T | +-----------------+----------+-------+-------+ ``` can anyone help? thanks. ---- Replied Message ---- | From | <ceph-users-request(a)ceph.io> | | Date | 8/28/2025 16:54 | | To | <ceph-users(a)ceph.io> | | Subject | ceph-users Digest, Vol 134, Issue 88 | Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics: 1. Re: OSD crc errors: Faulty SSD? (Anthony D'Atri) 2. Re: OSD crc errors: Faulty SSD? (Igor Fedotov) 3. Debian Packages for Trixie (Andrew) ---------------------------------------------------------------------- Date: Wed, 27 Aug 2025 10:08:16 -0400 From: Anthony D'Atri <anthony.datri(a)gmail.com> Subject: [ceph-users] Re: OSD crc errors: Faulty SSD? To: Roland Giesler <roland(a)giesler.za.net> Cc: ceph-users(a)ceph.io Message-ID: <400EC8C4-0B0D-4FEE-ADB5-1F355E9ABB19(a)gmail.com> Content-Type: text/plain; charset=utf-8 Please: * Identify the SSD involved * Look for messages in `dmesg` and `/var/log/{syslog/messages}` for that device * `smartctl -a /dev/xxxx` * If you you got the device from Dell or HP, look for a firmware update. Are you setting non-default RocksDB options? On Aug 27, 2025, at 10:01 AM, Roland Giesler <roland(a)giesler.za.net> wrote: I have relatively new Samsung Enterprise NVMe in a node that is generating the following error: 2025-08-26T15:56:43.870+0200 7fe8ac968700 0 bad crc in data 3326000616 != exp 1246001655 fromv1:192.168.131.4:0/1799093090 2025-08-26T16:03:54.757+0200 7fe8ad96a700 0 bad crc in data 3195468789 != exp 4291467912 fromv1:192.168.131.3:0/315398791 2025-08-26T16:17:34.160+0200 7fe8ad96a700 0 bad crc in data 1471079732 != exp 1408597599 fromv1:192.168.131.3:0/315398791 2025-08-26T16:33:34.035+0200 7fe8ad96a700 0 bad crc in data 724234454 != exp 3110238891 fromv1:192.168.131.3:0/315398791 2025-08-26T16:36:34.265+0200 7fe8ad96a700 0 bad crc in data 96649884 != exp 3724606899 fromv1:192.168.131.3:0/315398791 2025-08-26T16:40:34.395+0200 7fe8ad96a700 0 bad crc in data 1554359919 != exp 1420125995 fromv1:192.168.131.3:0/315398791 2025-08-26T16:54:18.323+0200 7fe8ad169700 0 bad crc in data 362320144 != exp 1850249930 fromv1:192.168.131.1:0/1316652062 This is ceph osd.40. More details from the log: 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415016487, "job": 447787, "event": "table_file_deletion", "file_number": 389377} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415016839, "job": 447787, "event": "table_file_deletion", "file_number": 389359} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415017245, "job": 447787, "event": "table_file_deletion", "file_number": 389342} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415017633, "job": 447787, "event": "table_file_deletion", "file_number": 389276} 2025-08-26T17:00:15.041+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415047481, "job": 447787, "event": "table_file_deletion", "file_number": 389254} 2025-08-26T17:00:15.045+0200 7fe8a06fe700 4 rocksdb: (Original Log Time 2025/08/26-17:00:15.047592) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do 2025-08-26T17:04:35.776+0200 7fe899ee2700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2025-08-26T17:04:35.776+0200 7fe899ee2700 4 rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 24012063.5 total, 600.0 interval Cumulative writes: 10G writes, 37G keys, 10G commit groups, 1.0 writes per commit group, ingest: 19073.99 GB, 0.81 MB/s Cumulative WAL: 10G writes, 4860M syncs, 2.14 writes per sync, written: 19073.99 GB, 0.81 MB/s Cumulative stall: 00:00:0.000H:M:S, 0.0 percent Interval writes: 159K writes, 546K keys, 159K commit groups, 1.0 writes per commit group, ingest: 202.96 MB, 0.34 MB/s Interval WAL: 159K writes, 76K syncs, 2.10 writes per sync, written: 0.20 MB, 0.34 MB/s Interval stall: 00:00:0.000H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.4 2.4 0.0 1.0 0.0 25.1 98.33 49.99 19074 0.005 0 0 L1 2/0 137.69 MB 1.0 299.7 2.4 297.3 297.9 0.7 0.0 123.5 59.6 59.2 5152.28 4745.40 4769 1.080 8130M 47M L2 7/0 410.80 MB 0.2 1.3 0.2 1.1 1.2 0.1 0.3 5.8 79.4 72.1 17.00 15.59 4 4.250 32M 3822K Sum 9/0 548.48 MB 0.0 301.0 2.6 298.4 301.5 3.1 0.3 125.0 58.5 58.6 5267.61 4810.98 23847 0.221 8162M 51M Int 0/0 0.00 KB 0.0 0.1 0.0 0.1 0.1 0.0 0.0 3194.3 48.0 47.9 2.87 2.72 2 1.437 5633K 2173 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 301.0 2.6 298.4 299.1 0.7 0.0 0.0 59.6 59.3 5169.28 4761.00 4773 1.083 8162M 51M High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.4 2.4 0.0 0.0 0.0 25.1 98.33 49.99 19073 0.005 0 0 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 32.8 0.00 0.00 1 0.001 0 0 This gets repeated many times until finally all seems well again. This a few minutes later the problem repeats itself. Is this a faulty ssd causing this? Environment: # pveversion: pve-manager/7.4-19/f98bf8d4 (running kernel: 5.15.131-2-pve) # ceph version 17.2.7 (29dffbfe59476a6bb5363cf5cc629089b25654e3) quincy (stable) _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------ Date: Wed, 27 Aug 2025 17:22:19 +0300 From: Igor Fedotov <igor.fedotov(a)croit.io> Subject: [ceph-users] Re: OSD crc errors: Faulty SSD? To: Roland Giesler <roland(a)giesler.za.net>, ceph-users(a)ceph.io Message-ID: <b32252ed-1eaa-4c68-8d1c-d38ad158bae4(a)croit.io> Content-Type: text/plain; charset=UTF-8; format=flowed Hi Roland, this looks like a messenger error to me. Hence it's rather a transport/networking issue not data-at-rest one. Thanks, Igor On 8/27/2025 5:01 PM, Roland Giesler wrote: I have relatively new Samsung Enterprise NVMe in a node that is generating the following error: 2025-08-26T15:56:43.870+0200 7fe8ac968700 0 bad crc in data 3326000616 != exp 1246001655 fromv1:192.168.131.4:0/1799093090 2025-08-26T16:03:54.757+0200 7fe8ad96a700 0 bad crc in data 3195468789 != exp 4291467912 fromv1:192.168.131.3:0/315398791 2025-08-26T16:17:34.160+0200 7fe8ad96a700 0 bad crc in data 1471079732 != exp 1408597599 fromv1:192.168.131.3:0/315398791 2025-08-26T16:33:34.035+0200 7fe8ad96a700 0 bad crc in data 724234454 != exp 3110238891 fromv1:192.168.131.3:0/315398791 2025-08-26T16:36:34.265+0200 7fe8ad96a700 0 bad crc in data 96649884 != exp 3724606899 fromv1:192.168.131.3:0/315398791 2025-08-26T16:40:34.395+0200 7fe8ad96a700 0 bad crc in data 1554359919 != exp 1420125995 fromv1:192.168.131.3:0/315398791 2025-08-26T16:54:18.323+0200 7fe8ad169700 0 bad crc in data 362320144 != exp 1850249930 fromv1:192.168.131.1:0/1316652062 This is ceph osd.40. More details from the log: 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415016487, "job": 447787, "event": "table_file_deletion", "file_number": 389377} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415016839, "job": 447787, "event": "table_file_deletion", "file_number": 389359} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415017245, "job": 447787, "event": "table_file_deletion", "file_number": 389342} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415017633, "job": 447787, "event": "table_file_deletion", "file_number": 389276} 2025-08-26T17:00:15.041+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415047481, "job": 447787, "event": "table_file_deletion", "file_number": 389254} 2025-08-26T17:00:15.045+0200 7fe8a06fe700 4 rocksdb: (Original Log Time 2025/08/26-17:00:15.047592) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do 2025-08-26T17:04:35.776+0200 7fe899ee2700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2025-08-26T17:04:35.776+0200 7fe899ee2700 4 rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 24012063.5 total, 600.0 interval Cumulative writes: 10G writes, 37G keys, 10G commit groups, 1.0 writes per commit group, ingest: 19073.99 GB, 0.81 MB/s Cumulative WAL: 10G writes, 4860M syncs, 2.14 writes per sync, written: 19073.99 GB, 0.81 MB/s Cumulative stall: 00:00:0.000H:M:S, 0.0 percent Interval writes: 159K writes, 546K keys, 159K commit groups, 1.0 writes per commit group, ingest: 202.96 MB, 0.34 MB/s Interval WAL: 159K writes, 76K syncs, 2.10 writes per sync, written: 0.20 MB, 0.34 MB/s Interval stall: 00:00:0.000H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.4 2.4 0.0 1.0 0.0 25.1 98.33 49.99 19074 0.005 0 0 L1 2/0 137.69 MB 1.0 299.7 2.4 297.3 297.9 0.7 0.0 123.5 59.6 59.2 5152.28 4745.40 4769 1.080 8130M 47M L2 7/0 410.80 MB 0.2 1.3 0.2 1.1 1.2 0.1 0.3 5.8 79.4 72.1 17.00 15.59 4 4.250 32M 3822K Sum 9/0 548.48 MB 0.0 301.0 2.6 298.4 301.5 3.1 0.3 125.0 58.5 58.6 5267.61 4810.98 23847 0.221 8162M 51M Int 0/0 0.00 KB 0.0 0.1 0.0 0.1 0.1 0.0 0.0 3194.3 48.0 47.9 2.87 2.72 2 1.437 5633K 2173 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 301.0 2.6 298.4 299.1 0.7 0.0 0.0 59.6 59.3 5169.28 4761.00 4773 1.083 8162M 51M High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.4 2.4 0.0 0.0 0.0 25.1 98.33 49.99 19073 0.005 0 0 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 32.8 0.00 0.00 1 0.001 0 0 This gets repeated many times until finally all seems well again. This a few minutes later the problem repeats itself. Is this a faulty ssd causing this? Environment: # pveversion: pve-manager/7.4-19/f98bf8d4 (running kernel: 5.15.131-2-pve) # ceph version 17.2.7 (29dffbfe59476a6bb5363cf5cc629089b25654e3) quincy (stable) _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------ Date: Thu, 28 Aug 2025 18:52:05 +1000 From: Andrew <andrew(a)donehue.net> Subject: [ceph-users] Debian Packages for Trixie To: ceph-users(a)ceph.io Message-ID: <e81c6fed-ff9f-483a-acc0-23d0de5db012(a)donehue.net> Content-Type: text/plain; charset=UTF-8; format=flowed Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 134, Issue 88 *******************************************

1 0

OSD crc errors: Faulty SSD?
by Roland Giesler 27 Aug '25

27 Aug '25

I have relatively new Samsung Enterprise NVMe in a node that is generating the following error: 2025-08-26T15:56:43.870+0200 7fe8ac968700 0 bad crc in data 3326000616 != exp 1246001655 fromv1:192.168.131.4:0/1799093090 2025-08-26T16:03:54.757+0200 7fe8ad96a700 0 bad crc in data 3195468789 != exp 4291467912 fromv1:192.168.131.3:0/315398791 2025-08-26T16:17:34.160+0200 7fe8ad96a700 0 bad crc in data 1471079732 != exp 1408597599 fromv1:192.168.131.3:0/315398791 2025-08-26T16:33:34.035+0200 7fe8ad96a700 0 bad crc in data 724234454 != exp 3110238891 fromv1:192.168.131.3:0/315398791 2025-08-26T16:36:34.265+0200 7fe8ad96a700 0 bad crc in data 96649884 != exp 3724606899 fromv1:192.168.131.3:0/315398791 2025-08-26T16:40:34.395+0200 7fe8ad96a700 0 bad crc in data 1554359919 != exp 1420125995 fromv1:192.168.131.3:0/315398791 2025-08-26T16:54:18.323+0200 7fe8ad169700 0 bad crc in data 362320144 != exp 1850249930 fromv1:192.168.131.1:0/1316652062 This is ceph osd.40. More details from the log: 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415016487, "job": 447787, "event": "table_file_deletion", "file_number": 389377} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415016839, "job": 447787, "event": "table_file_deletion", "file_number": 389359} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415017245, "job": 447787, "event": "table_file_deletion", "file_number": 389342} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415017633, "job": 447787, "event": "table_file_deletion", "file_number": 389276} 2025-08-26T17:00:15.041+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415047481, "job": 447787, "event": "table_file_deletion", "file_number": 389254} 2025-08-26T17:00:15.045+0200 7fe8a06fe700 4 rocksdb: (Original Log Time 2025/08/26-17:00:15.047592) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do 2025-08-26T17:04:35.776+0200 7fe899ee2700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2025-08-26T17:04:35.776+0200 7fe899ee2700 4 rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 24012063.5 total, 600.0 interval Cumulative writes: 10G writes, 37G keys, 10G commit groups, 1.0 writes per commit group, ingest: 19073.99 GB, 0.81 MB/s Cumulative WAL: 10G writes, 4860M syncs, 2.14 writes per sync, written: 19073.99 GB, 0.81 MB/s Cumulative stall: 00:00:0.000H:M:S, 0.0 percent Interval writes: 159K writes, 546K keys, 159K commit groups, 1.0 writes per commit group, ingest: 202.96 MB, 0.34 MB/s Interval WAL: 159K writes, 76K syncs, 2.10 writes per sync, written: 0.20 MB, 0.34 MB/s Interval stall: 00:00:0.000H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.4 2.4 0.0 1.0 0.0 25.1 98.33 49.99 19074 0.005 0 0 L1 2/0 137.69 MB 1.0 299.7 2.4 297.3 297.9 0.7 0.0 123.5 59.6 59.2 5152.28 4745.40 4769 1.080 8130M 47M L2 7/0 410.80 MB 0.2 1.3 0.2 1.1 1.2 0.1 0.3 5.8 79.4 72.1 17.00 15.59 4 4.250 32M 3822K Sum 9/0 548.48 MB 0.0 301.0 2.6 298.4 301.5 3.1 0.3 125.0 58.5 58.6 5267.61 4810.98 23847 0.221 8162M 51M Int 0/0 0.00 KB 0.0 0.1 0.0 0.1 0.1 0.0 0.0 3194.3 48.0 47.9 2.87 2.72 2 1.437 5633K 2173 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 301.0 2.6 298.4 299.1 0.7 0.0 0.0 59.6 59.3 5169.28 4761.00 4773 1.083 8162M 51M High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.4 2.4 0.0 0.0 0.0 25.1 98.33 49.99 19073 0.005 0 0 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 32.8 0.00 0.00 1 0.001 0 0 This gets repeated many times until finally all seems well again. This a few minutes later the problem repeats itself. Is this a faulty ssd causing this? Environment: # pveversion: pve-manager/7.4-19/f98bf8d4 (running kernel: 5.15.131-2-pve) # ceph version 17.2.7 (29dffbfe59476a6bb5363cf5cc629089b25654e3) quincy (stable)

3 2

Preventing device zapping while replacing faulty drive (Squid 19.2.2)
by Dmitrijs Demidovs 27 Aug '25

27 Aug '25

Hi List! We have Squid 19.2.2. It is cephadm/docker based deployment (recently upgraded from Pacific 16.2.15). We are using 8 SAS drives for Block and 2 SSD drives for DB on every OSD Host. Problem: One of the SAS Block drives failed on OSD Host and we need to replace it. When our Ceph cluster was running on Pacific, we usually performed drive replacement using this steps: 1) Edit -> MARK osd.xx as OUT [re-balancing starts, wait until it is completed] 2) Edit -> MARK osd.xx as DOWN 3) Edit -> DELETE osd.xx [put check on "Preserve OSD ID" and "Yes, I am sure"] 4) Edit -> DESTROY osd.xx 5) Edit -> PURGE osd.xx [re-balancing starts, wait until it is completed] 6) Set "noout" and "norebalance" flags. Put OSD Host in maintenance mode. Shutdown OSD Host. Replace failed drive. Start OSD Host. 7) Wipe old DB Logical Volume (LV) [dd if=/dev/zero of=/dev/ceph-xxx/osd-db-xxx bs=1M count=10 conv=fsync]. 8) Wipe new Block disk. Destroy old DB LV. Wait for automatic discovery and creation of new osd.xx instance. In Pacific 16.2.15, after execution of PURGE command, Ceph just removed old osd.xx instance from cluster without deletion/zapping of DB and Block LVs. Now in Squid 19.2.2 we see what Ceph behaves differently. Execution of step 3 (Edit -> DELETE osd.xx) automatically executes DESTROY and PURGE, and after that Ceph automatically performs zapping and deletion of DB LV and Block LV! And after that it's automatic discovery finds "clean" SAS disk + free space on SSD drive and happily forms new osd.xx instance form failed drive what we need to replace :) Questions: 1) What is correct procedure how-to replace failed Block drive in Ceph Squid 19.2.2? 2) Is it possible to disable Zapping? 3) Is it possible to temporary disable automatic discovery of new drives for OSD service? P.S. Here is our Pacement Specification for OSD service: [ceph: root@ceph-mon12 /]# ceph orch ls osd osd.dashboard-admin-1633624229976 --export service_type: osd service_id: dashboard-admin-1633624229976 service_name: osd.dashboard-admin-1633624229976 placement: host_pattern: '*' spec: data_devices: rotational: true db_devices: rotational: false db_slots: 4 filter_logic: AND objectstore: bluestore Logs from Ceph: 12/8/25 08:54 AM [INF] Cluster is now healthy 12/8/25 08:54 AM [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded) 12/8/25 08:54 AM [WRN] Health check update: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 12/8/25 08:54 AM [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 12/8/25 08:54 AM [WRN] Health check failed: Degraded data redundancy: 12/121226847 objects degraded (0.000%), 2 pgs degraded (PG_DEGRADED) 12/8/25 08:54 AM [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 12/8/25 08:54 AM [INF] osd.33 [v2:10.10.10.105:6824/297218036,v1:10.10.10.105:6825/297218036] boot 12/8/25 08:54 AM [WRN] OSD bench result of 1909.305644 IOPS is not within the threshold limit range of 50.000000 IOPS and 500.000000 IOPS for osd.33. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. 12/8/25 08:53 AM [INF] Deploying daemon osd.33 on ceph-osd15 12/8/25 08:53 AM [INF] Found osd claims for drivegroup dashboard-admin-1633624229976 -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Found osd claims -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Detected new or changed devices on ceph-osd15 12/8/25 08:52 AM [INF] Successfully zapped devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Zapping devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Successfully destroyed old osd.33 on ceph-osd15; ready for replacement 12/8/25 08:52 AM [INF] Successfully removed osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Removing key for osd.33 12/8/25 08:52 AM [INF] Removing daemon osd.33 from ceph-osd15 -- ports []

4 3

cannot reach telemetry.ceph.com -> HEALTH_WARN after cluster upgrade to 19.2.3
by Björn Lässig 27 Aug '25

27 Aug '25

Hi, Does anyone know who to contact to report problems with telemetry.ceph.com? After a reasonably successful update of my cluster, I received a HEALTH_WARN [Telemetry requires re-opt-in] even though I had already re-approved telemetry. While trying to understand the problem, I stumbled across the fact that although the host telemetry.ceph.com has an IPv4 and an IPv6 address, I cannot reach it because IPv6 only has a dropping firewall or something similar attached to it. Now, I can configure a proxy that also reaches the legacy Internet, but it also tried to reach IPv6 first, so that currently no IPv6-enabled Ceph cluster should be visible in the telemetry. My workaround would now be to overlay telemetry.ceph.com on the DNS cache... but that's just a hack. ----- raw shell output for reproduction ----- > host telemetry.ceph.com telemetry.ceph.com has address 8.43.84.137 telemetry.ceph.com has IPv6 address 2620:52:3:3:201a:4aff:feab:8409 > telnet 2620:52:3:3:201a:4aff:feab:8409 80 Trying 2620:52:3:3:201a:4aff:feab:8409... telnet: Unable to connect to remote host: Connection timed out > telnet 8.43.84.137 80 Trying 8.43.84.137... Connected to 8.43.84.137. Escape character is '^]'. get / HTTP/1.0 HTTP/1.1 302 Found Location: https://telemetry.ceph.com/

2 2

Add previously OSD to a new installation of a 3 nodes Proxmox CEPH
by Gilberto Ferreira 27 Aug '25

27 Aug '25

Hi I have 3 nodes Proxmox Cluster with CEPH, and after a crash, I have to reinstall Proxmox from scratch, along with Ceph. OSD are intact. I already did ceph-volume lvm activate --all and the OSD appears with ceph-volum lvm list and I got a folder with the name of the OSD under /var/lib/ceph/osd. However is not appear in ceph osd tree or ceph -s or even in the web gui. Is there any way to re-add this OSD to Proxmox CEPH? Thanks a lot for any help. Best Regards --- Gilbert

3 19

MGR does not generate prometheus config for ceph-exporter
by Adam Prycki 27 Aug '25

27 Aug '25

Hello, I have issue with multiple of our clusters. It doesn't seem to be versions specific and just happens on our clusters which have been running for a while and have been through some version upgrades. In current cephadm versions prometheus is configured via service discovery from MGR. Relevant prometheus config looks like this: - job_name: 'ceph-exporter' honor_labels: true http_sd_configs: - url: http://X.X.X.X:8765/sd/prometheus/sd-config?service=ceph-exporter Issue is that mgr doesn't return any hosts. Just an empty array []. Prometheus doesn't scrap ceph-exporter and some of our grafana graphs are empty. :/ Other URLs in this config file work fine and generate valid sd configs. Has anyone else experienced this? Does anyone know how to fix this? Best regards Adam Prycki

2 1

Ceph Steering Committee Meeting Minutes for 2025-08-25
by Patrick Donnelly 26 Aug '25

26 Aug '25

Short meeting yesterday: * [Yuri] Tentacle Update - [Laura] maybe an NVMe blocker PR - https://github.com/ceph/ceph/pull/64779 + [Ilya] Does not believe this is a blocker. * [Patrick] Cloud spend has been increasing: we need to move more services local. Probably will be done next year after the lab infrastructure is relocated. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

NVME-oF only with quay.ceph.io/ceph-ci/ceph:squid-nvmeof ?
by Robert Sander 25 Aug '25

25 Aug '25

Hi, I have created a cluster with the container image quay.ceph.io/ceph-ci/ceph:squid-nvmeof and deployed an NVMe-oF service. This was successful. I am able to connect from a Linux host with the nvme-cli. I do see /dev/nvme0n1 and can fdisk it and format /dev/nvme0n1p1, mount the filesystem and run some IO on it. I can then stop one of the 4 NVMe-oF daemons that are running as listeners and the iniator re-connects to another one. Great. Is it true that currently this image is the only one where NVMe-oF works? The version is reported as 19.2.0-935-g0006f0dd. How much does this image deviate from the quay.io/ceph/ceph:v19.2.3? Does the whole cluster need to run with quay.ceph.io/ceph-ci/ceph:squid-nvmeof or is it sufficient to run the NVMe-oF service with quay.io/ceph/nvmeof:1.4 ? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Large file downloads (~2TB) from CephFS drop at ~100 GB
by Mevludin Blazevic 25 Aug '25

25 Aug '25

Hello all, I am experiencing a strange issue with downloads from a CephFS-backed environment and would like to ask if anyone has seen similar behavior. When downloading large files (via wget or even directly through Firefox), the transfer speed is fine until the file size reaches roughly 100 GB. At that point, the session gets disconnected. This happens consistently, independent of the bandwidth available (e.g., whether the link runs at ~80 Mbit/s or ~230 Mbit/s). The only difference is how quickly the 100 GB “border” is reached. After the disconnect, the download can usually be resumed, but the cutoff is very repeatable. Is there a configuration that needs to be done on the ceph site? Best, Mevludin

4 6

After disk failure not deep scrubbed pgs started to increase (ceph quincy)
by Szabo, Istvan (Agoda) 25 Aug '25

25 Aug '25

Hi, After a disk failure, the number of PGs not deep scrubbed started to increase (Ceph Quincy). Here is an example PG: https://gist.githubusercontent.com/Badb0yBadb0y/afe9b26fbae72c79c9430585bf4… Normally, when this happens, I observe some stuck operations and increased latency on certain OSDs (restarting the affected OSD usually resolves the issue, and the not deep-scrubbed PGs get scrubbed within a day). However, this time I don’t see any obvious problems. Could you please help review the attached not deep-scrubbed PG and let me know if you notice anything unusual? For context, the disk was re-added yesterday and rebalance is still in progress, but the not deep-scrubbed alerts started appearing before the disk was added back. Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 4

For posterity: cct->_conf->osd_fast_shutdown_timeout OSD errors / Run Full Recovery from ONodes (might take a while) during Reef 18.2.1 to 18.2.7 upgrade
by Anthony D'Atri 25 Aug '25

25 Aug '25

Posting this for posterity, in case someone runs into it down the line and finds it in the archives when trying to figure out what the heck is going on. On a Reef 18.2.1 cluster, when nodes reboot, some OSDs experience the below: "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/osd/OSD.cc: In function 'int OSD::shutdown()' thread 7f1286b61700 time 2025-08-24T19:57:36.343925+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/osd/OSD.cc: 4495: FAILED ceph_assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)\n”, which seems to lead to the below at OSD startup: 2025-08-25T00:04:29.669+0000 7faa68c20740 1 freelist _read_cfg 2025-08-25T00:04:29.881+0000 7faa68c20740 1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2025-08-25T00:04:29.881+0000 7faa68c20740 0 bluestore(/var/lib/ceph/osd/ceph-29) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) … After some research my understanding is that the root cause is addressed in 18.2.6. My sense is that after this 18.2.1 cluster is completely on 18.2.7 the problem should fade. During the upgrade, however, the instances of it complicate the process and require archiving crashes, waiting an hour or two (20TB spinners) for the abovedescribed full recovery at OSD startup, etc. I’ve found that doubling the default timeout: ceph config set global osd_fast_shutdown_timeout 30 is making a dramatic difference. After setting the above, the upgrade is progressing nicely as expected.

1 0

Ceph Roundup: Umbrella Summit, Windows Survey, Cephalocon & More!
by Anthony Middleton 22 Aug '25

22 Aug '25

Hey Cephers, Welcome to my first *Ceph Roundup! *This is your one-stop update for everything happening in our community. I plan to send this type of email weekly, or as often as needed. Here's what's making waves this week: *Ceph Developer Summit: Umbrella* The Summit runs from *Sept 3–12 (virtual)*. Join the conversation, shape the next release, and collaborate with fellow Cephers! Schedule coming soon! Read more here: https://ceph.io/en/community/events/2025/ceph-developer-summit-umbrella/ *Windows Support for Ceph – Community Survey* We're exploring Windows support! Your input helps guide this effort. Survey -https://forms.gle/ME2tqW12zavCorDd8 *Cephalocon 2025 Registration is OPEN!* Get ready for our flagship event. Secure your spot and meet Cephers from around the world. Register now - https://events.linuxfoundation.org/cephalocon/register/ *Cephalocon Sponsorship Opportunities* Want to showcase your brand at the biggest Ceph event of the year? Sponsorships are still available until September 31st! Learn more - https://events.linuxfoundation.org/cephalocon/sponsor/ That's it for this week! Share your experience with Ceph in a message below. Did you learn anything new? What Ceph project are you starting next week? Your insights help us keep Ceph thriving. Stay squiddy, --- Anthony Middleton Ceph Community Manager 🐙 LinkedIn <https://www.linkedin.com/in/middleton-anthony/>| BlueSky <https://bsky.app/profile/anthonymicmidd.bsky.social> community-manager(a)ceph.io

1 0

Problems afte update Debian bullseye to trixie and quinct to reef
by Christian Peters 22 Aug '25

22 Aug '25

Hi all, not sure it's ok to ask questions here as we are only a little Theater where I manage the data of our sound and video files. I set up a ceph cluster as second backup location on bullseye with quincy which worked flawless. All runs smooth but I thought I would have to do a step forward to get on track. Yes I know: never change a running system. :-D After updating all three nodes I forget to set the ceph packages on hold so they were updated to reef with trixie. No way to downgrade ...I tried a lot....! So I went forward. After all all 3 nodes are back online running reef my cluster seems to work, my cephfs share is reachable by my client machine, backups are working. This is my ceph status output: cluster: id: c5986871-ab9f-4365-8113-a1854fa07786 health: HEALTH_WARN 15 mgr modules have failed dependencies services: mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 19h) mgr: ceph1(active, since 18h), standbys: ceph2, ceph3 mds: 1/1 daemons up, 2 standby osd: 24 osds: 24 up (since 19h), 24 in (since 22M) data: volumes: 1/1 healthy pools: 3 pools, 1041 pgs objects: 3.34M objects, 11 TiB usage: 32 TiB used, 56 TiB / 87 TiB avail pgs: 1041 active+clean Problem: dashboard is not working any more and I have this erre/warn: 15 mgr modules have failed dependencies The cluster was made without podman/containers. ceph-crash gives me a flood of this errors: ... WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T11:51:45.411885Z_738ee242-f7bd-4f6c-bd90-d96a08e06bf6 as client.admin failed: 2025-08-22T06:55:50.234+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.234+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff728064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff728068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff72cea8410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.crash.ceph1 failed: 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81ac0640c8) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81ac0686c0) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81b3ea8410) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.crash failed: 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5b9c064068) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5b9c068540) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5ba2ea8410) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.admin failed: 2025-08-22T06:55:50.478+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.478+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc844064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc844068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc8494fd410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.crash.ceph1 failed: 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d40640c8) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d40686c0) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d9aa8410) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.crash failed: 2025-08-22T06:55:50.642+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.642+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a30064068) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a30068540) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a37aa8410) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.admin failed: 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa800064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa800068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa805ea8410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) ...... In /etc/ceph I have ceph.client.admin.keyring ceph.conf rbdmap Maye there is a way to fix this but if it's to complicated I will think about a complete re-install. I'm not an admin but an amateur with some linux knowledge. If there is hope and someone could support to solev this I would be very happy! Regards, Christian Peters

2 1

new osd's with custom device-class sata not being used by pool?
by jbuburuzlist 22 Aug '25

22 Aug '25

Hello I just added more nodes to quincy cluster(existing drives are SAS). The new nodes are sata drives. The problem is after creating a new replicated crush rule and applying to a new pool. The ceph cluster is still using osd with "hdd" class and not using sata drives. I tried a number of crush rules using replicated and EC and I cannot get the pools to use sata osds. Here is crush rule for sata: { "rule_id": 9, "rule_name": "rbdsatareplicated", "type": 1, "steps": [ { "op": "take", "item": -33, "item_name": "default~sata" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } I see in the crush rule "default~sata". I am wondering if this "default~" is causing me problems? Just another note. After I added the new sata nodes, my first crush rule used ec instead of replicated. I was getting errors when I tried to create a image on the ec pool "qemu-img error rbd create: Operation not supported on new pool". So I tried a test pool using replicated instead. Currently on the cluster I can create any number of pools and images if I use replicated/ec and the SAS drives which have a class of "hdd". Hope this makes sense? Here is some additional details: My existing nodes used SAS and are labelled: ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1122.64075 root default -3 34.03793 host node-01 0 hdd 2.26920 osd.0 up 1.00000 1.00000 1 hdd 2.26920 osd.1 up 1.00000 1.00000 2 hdd 2.26920 osd.2 up 1.00000 1.00000 ..etc After adding SATA nodes I have: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1122.64075 root default -3 34.03793 host node-01 0 hdd 2.26920 osd.0 up 1.00000 1.00000 1 hdd 2.26920 osd.1 up 1.00000 1.00000 2 hdd 2.26920 osd.2 up 1.00000 1.00000 .... ... -34 133.78070 host node-20 141 sata 11.14839 osd.141 up 1.00000 1.00000 142 sata 11.14839 osd.142 up 1.00000 1.00000 143 sata 11.14839 osd.143 up 1.00000 1.00000 144 sata 11.14839 osd.144 up 1.00000 1.00000 145 sata 11.14839 osd.145 up 1.00000 1.00000 146 sata 11.14839 osd.146 up 1.00000 1.00000 ... Thanks jerry

2 1

mclock scheduler on 19.2.1
by Curt 22 Aug '25

22 Aug '25

Hello all, I'm sure this has been discussed before, but I can't seem to find it. I know on older versions of Ceph there was an issue with mclock having no recovery and switching to wpq fixed it. Is this still an issue with 19.2.1? I recently ran into this bug <https://tracker.ceph.com/issues/70390>and various issues with it. In order to help recovery I set norebalance flag, so it would focus solely on undersized PGs. The issue I'm seeing though is sometimes recovering will show nothing despite having X active+undersized+remapped+backfilling. Sometimes restarting a few OSD's will fix the issue and it will start again. I'm tempted to switch to wpq, but that would mean having to slowly restart each OSD, which with undersized would cause IO to stop while some OSD's are restarted. Wanted to get others' thoughts before making the change. Thanks, Curt

2 3

Problems after update Debian bullseye to trixie and quincy to reef
by info＠hcpeters.de 22 Aug '25

22 Aug '25

Hi all, not sure it's ok to ask questions here as we are only a little Theater where I manage the data of our sound and video files. I set up a ceph cluster as second backup location on bullseye with quincy which worked flawless. All runs smooth but I thought I would have to do a step forward to get on track. Yes I know: never change a running system. :-D After updating all three nodes I forget to set the ceph packages on hold so they were updated to reef with trixie. No way to downgrade ...I tried a lot....! So I went forward. After all all 3 nodes are back online running reef my cluster seems to work, my cephfs share is reachable by my client machine, backups are working. This is my ceph status output: cluster: id: c5986871-ab9f-4365-8113-a1854fa07786 health: HEALTH_WARN 15 mgr modules have failed dependencies services: mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 19h) mgr: ceph1(active, since 18h), standbys: ceph2, ceph3 mds: 1/1 daemons up, 2 standby osd: 24 osds: 24 up (since 19h), 24 in (since 22M) data: volumes: 1/1 healthy pools: 3 pools, 1041 pgs objects: 3.34M objects, 11 TiB usage: 32 TiB used, 56 TiB / 87 TiB avail pgs: 1041 active+clean Problem: dashboard is not working any more and I have this erre/warn: 15 mgr modules have failed dependencies The cluster was made without podman/containers. ceph-crash gives me a flood of this errors: ... WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T11:51:45.411885Z_738ee242-f7bd-4f6c-bd90-d96a08e06bf6 as client.admin failed: 2025-08-22T06:55:50.234+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.234+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff728064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff728068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff72cea8410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.crash.ceph1 failed: 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81ac0640c8) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81ac0686c0) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81b3ea8410) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.crash failed: 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5b9c064068) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5b9c068540) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5ba2ea8410) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.admin failed: 2025-08-22T06:55:50.478+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.478+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc844064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc844068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc8494fd410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.crash.ceph1 failed: 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d40640c8) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d40686c0) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d9aa8410) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.crash failed: 2025-08-22T06:55:50.642+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.642+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a30064068) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a30068540) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a37aa8410) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.admin failed: 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa800064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa800068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa805ea8410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) ...... In /etc/ceph I have ceph.client.admin.keyring ceph.conf rbdmap Maye there is a way to fix this but if it's to complicated I will think about a complete re-install. I'm not an admin but an amateur with some linux knowledge. If there is hope and someone could support to solev this I would be very happy! Regards, Christian Peters

3 7

smartctl failed with error -22
by Robert Sander 22 Aug '25

22 Aug '25

Hi, On a new cluster with version 19.2.3 the device health metrics only show a smartctl error: { "20250821-000313": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "ata", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } } The device in question (like all the other in the cluster) is a Samsung MZ7L37T6 SATA SSD. What is happening here? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

5 8

[v19.2.3] All OSDs are not created with a managed spec
by Gilles Mocellin 22 Aug '25

22 Aug '25

On Hi Cephers, I'm building a new Squid cluster with cephadm on Ubuntu 24.04. After expanding my cluster in the Dashboard (adding my 7 hosts), I choose throughput_optimized proflie wich create a generic spec for hybrid HDD/SSD : service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore The cluster is for a LAB environment, on each of the 7 nodes, I have 17 HDD SAS 1.2TB drives and 1 SSD SAS Enterprise 400GB drive. On my first try, only 28 OSD where created (out of the 119), the others appeared as down, but they won't start, I didn't find systemd units created on the hosts. But, the VGs and LVs where created, there are 17 LVs on the SSD for WAL/DB of the 17 HDD (yes, small : 29GB). On my second try, it creates 72 OSDs, still it stops, and never tries to continue, or re-create the down OSDs. I didn't manage to find them, but it seems I saw some OSD creation timeout in the logs... What can I do to have my missing OSD created ? I tried restarting, redeploying the orch OSD service, but it only restart/redeploy le OSDs it has already created...

6 17

[v19.2.3] Zapped OSD are not recreated with DB device
by Gilles Mocellin 22 Aug '25

22 Aug '25

Hi, Having timeout problems with my cluster, I try to finsh, recreate OSDs that failed. My config is hybrid with 17 HDD and 1 SSD by server. My OSD spec is the standard : service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 objectstore: bluestore encrypted: true filter_logic: AND When I remove an OSD with : ceph orch osd rm ID --zap It is finaly re-created, but without DB device. And I can see the ceph-volume commands : Zapping : cephadm ['--image', 'quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec', '--timeout', '2395', 'ceph-volume', '--fsid', '8ec7575a-7de5-11f0-a78a-246e96bd90a4', '--', 'lvm', 'zap', '--osd-id', '82', '--destroy'] Recreate : cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=throughput_optimized', '--image', 'quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec', '--timeout', '2395', 'ceph-volume', '--fsid', '8ec7575a-7de5-11f0-a78a-246e96bd90a4', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/sde', '--dmcrypt', '--osd-ids', '82', '--yes', '--no-systemd'] I think it should have the argument --db-devices ? I can see that option when the OSD spec was applied, it launch for all drives on the host : cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=throughput_optimized', '--image', 'quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec', '--timeout', '2395', 'ceph-volume', '--fsid', '8ec7575a-7de5-11f0-a78a-246e96bd90a4', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/sda', '/dev/sdb', '/dev/sdc', '/dev/sdd', '/dev/sde', '/dev/sdf', '/dev/sdg', '/dev/sdh', '/dev/sdi', '/dev/sdj', '/dev/sdk', '/dev/sdl', '/dev/sdm', '/dev/sdn', '/dev/sdo', '/dev/sdp', '/dev/sdq', '--db-devices', '/dev/sdr', '--dmcrypt', '--yes', '--no-systemd'] There is the --db-devices argument. Is it a bug or I do it wrong ? I have several OSD that are now without DB device, I want to recreate them with the DB device. And also, I want that every time a new drive is created, it will be with a DB device, or fails if no room, but not create without DB device if my OSD SPEC tells it.

2 5

Re: Disk failure (with osds failure) cause 'unrelated|different' osd device to crash
by Wissem MIMOUNA - Ceph Users 21 Aug '25

21 Aug '25

Hi , The issue here is that one nvme failed so related osds ( let's say osd.1 , osd.2 , osd.3 and osd.4 ) all crashes and in the meantime another osd ( osd.108 ) on the same host crashes too (2 times with 2 different failed nvme ) on the same assert_line and assert_file as below ) . So why this happened just one the nvme failed? Is they are related ? I can,'t have a clear vue ... Regards On 8/21/25 15:57, Anthony D'Atri wrote: > >> We had a ceph cluster version 18.2.7 deployed with four osds per nvme device. > Post-Quincy the benefits of multiple OSDs per NVMe device are mostly obviated. > > Or are you saying that you have HDD OSDs that offload WAL+DB? > >> Two weeks ago, we lost one hard drive ( so 4 osds ) , just after the osds had crashed , we had another "healthy" osd which crashed ( few minutes after the initial hard drive failure ) . >> >> Again , this week we lost another hard drive ( lost 4 osds ) and again the same osd crashed too ( few minutes later ) . >> >> Here below the crash information , the same crash information for both crashes : >> >> >> ceph crash info 2025-08-21T00:31:13.929426Z_576e6e42-7c6b-49b8-90f1-9d51730f8ac2 >> { >> "assert_condition": "r == 0", >> "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/os/bluestore/BlueStore.cc", >> "assert_func": "void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)", >> "assert_line": 12944, >> "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)' thread 7f5fba6a3640 time 2025-08-21T00:31:13.926894+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/os/bluestore/BlueStore.cc: 12944: FAILED ceph_assert(r == 0)\n", >> "assert_thread_name": "bstore_kv_sync", >> "backtrace": [ >> "/lib64/libc.so.6(+0x3ebf0) [0x7f5fce5a5bf0]", >> "/lib64/libc.so.6(+0x8bf5c) [0x7f5fce5f2f5c]", >> "raise()", >> "abort()", >> "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x169) [0x560bc2d0e44d]", >> "/usr/bin/ceph-osd(+0x3cc5ae) [0x560bc2d0e5ae]", >> "/usr/bin/ceph-osd(+0x3b8864) [0x560bc2cfa864]", >> "(BlueStore::_kv_sync_thread()+0x1073) [0x560bc32bb623]", >> "/usr/bin/ceph-osd(+0x8fd3b1) [0x560bc323f3b1]", >> "/lib64/libc.so.6(+0x8a21a) [0x7f5fce5f121a]", >> "/lib64/libc.so.6(+0x10f290) [0x7f5fce676290]" >> ], >> "ceph_version": "18.2.7", >> "crash_id": "2025-08-21T00:31:13.929426Z_576e6e42-7c6b-49b8-90f1-9d51730f8ac2", >> "entity_name": "osd.108", >> "os_id": "centos", >> "os_name": "CentOS Stream", >> "os_version": "9", >> "os_version_id": "9", >> "process_name": "ceph-osd", >> "stack_sig": "0080731b49e5583e6d168903c1ea7df8bc2caded6b5e24ee4381077d54b045e2", >> "timestamp": "2025-08-21T00:31:13.929426Z", >> "utsname_hostname": "pub1-cephosd-9", >> "utsname_machine": "x86_64", >> "utsname_release": "6.1.0-31-amd64", >> "utsname_sysname": "Linux", >> "utsname_version": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07)" >> >> >> Regards >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Disk failure (with osds failure) cause 'unrelated|different' osd device to crash
by Wissem MIMOUNA - Ceph Users 21 Aug '25

21 Aug '25

Hi everyone , We had a ceph cluster version 18.2.7 deployed with four osds per nvme device. Two weeks ago, we lost one hard drive ( so 4 osds ) , just after the osds had crashed , we had another "healthy" osd which crashed ( few minutes after the initial hard drive failure ) . Again , this week we lost another hard drive ( lost 4 osds ) and again the same osd crashed too ( few minutes later ) . Here below the crash information , the same crash information for both crashes : ceph crash info 2025-08-21T00:31:13.929426Z_576e6e42-7c6b-49b8-90f1-9d51730f8ac2 { "assert_condition": "r == 0", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/os/bluestore/BlueStore.cc", "assert_func": "void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)", "assert_line": 12944, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)' thread 7f5fba6a3640 time 2025-08-21T00:31:13.926894+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/os/bluestore/BlueStore.cc: 12944: FAILED ceph_assert(r == 0)\n", "assert_thread_name": "bstore_kv_sync", "backtrace": [ "/lib64/libc.so.6(+0x3ebf0) [0x7f5fce5a5bf0]", "/lib64/libc.so.6(+0x8bf5c) [0x7f5fce5f2f5c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x169) [0x560bc2d0e44d]", "/usr/bin/ceph-osd(+0x3cc5ae) [0x560bc2d0e5ae]", "/usr/bin/ceph-osd(+0x3b8864) [0x560bc2cfa864]", "(BlueStore::_kv_sync_thread()+0x1073) [0x560bc32bb623]", "/usr/bin/ceph-osd(+0x8fd3b1) [0x560bc323f3b1]", "/lib64/libc.so.6(+0x8a21a) [0x7f5fce5f121a]", "/lib64/libc.so.6(+0x10f290) [0x7f5fce676290]" ], "ceph_version": "18.2.7", "crash_id": "2025-08-21T00:31:13.929426Z_576e6e42-7c6b-49b8-90f1-9d51730f8ac2", "entity_name": "osd.108", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-osd", "stack_sig": "0080731b49e5583e6d168903c1ea7df8bc2caded6b5e24ee4381077d54b045e2", "timestamp": "2025-08-21T00:31:13.929426Z", "utsname_hostname": "pub1-cephosd-9", "utsname_machine": "x86_64", "utsname_release": "6.1.0-31-amd64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07)" Regards

1 0

Scub VS Deep Scrub
by Alex 20 Aug '25

20 Aug '25

I'm trying to understand which settings control light scrubbing and which control deep scrubs. `osd_scrub_begin_hour` and ` osd_scrub_end_hour` are for deep scrubs as far as I can tell even though the documentation just calls them scrubs. What about `osd_scrub_min_interval` and `osd_scrub_max_interval` for example, are they for light or deep scrubs or do these settings control both? The documentation [ https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/ ] is unclear except for around 6 cases in which "deep scrub", "deep-scrub", and "deep--scrub" are all used interchangeably.

1 0

Squid on 24.04 does not have a Release file
by Devender Singh 20 Aug '25

20 Aug '25

Dear all Getting below error, any fix for this? # apt update Hit:1 https://repo.zabbix.com/zabbix/6.4/ubuntu noble InRelease Hit:2 http://security.ubuntu.com/ubuntu noble-security InRelease Ign:3 https://download.ceph.com/debian-squid noble InRelease Err:4 https://download.ceph.com/debian-squid noble Release 404 Not Found [IP: 158.69.68.124 443] Hit:5 http://archive.ubuntu.com/ubuntu noble InRelease Hit:6 http://archive.ubuntu.com/ubuntu noble-updates InRelease Hit:7 http://archive.ubuntu.com/ubuntu noble-backports InRelease Reading package lists... Done E: The repository 'https://download.ceph.com/debian-squid noble Release' does not have a Release file. N: Updating from such a repository can't be done securely, and is therefore disabled by default. N: See apt-secure(8) manpage for repository creation and user configuration details. # cat /etc/os-release PRETTY_NAME="Ubuntu 24.04.3 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.3 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble LOGO=ubuntu-logo # cat /etc/apt/sources.list.d/ceph.list deb https://download.ceph.com/debian-squid/ noble main Regards Dev

3 4

[bluestore] How to deal with free fragmentation
by Florent Carli 20 Aug '25

20 Aug '25

Hello, I’m running Ceph 19.2.3 with cephadm on a 3-node cluster, and I recently started seeing the following warning related to free fragmentation: HEALTH_WARN 3 OSD(s) [WRN] BLUESTORE_FREE_FRAGMENTATION: 3 OSD(s) osd.0 0.802713 osd.1 0.803276 osd.2 0.803870 I haven’t been able to find clear guidance on how to address this issue. Are there any recommended approaches or best practices for reducing BlueStore free space fragmentation? Thanks in advance for your help! Florent.

4 3

Issue with Ceph RBD Incremental Backup (import-diff failure)
by Vishnu Bhaskar 20 Aug '25

20 Aug '25

I was trying to back up volumes created in Ceph using *rbd export* and *rbd import*. For incremental backups, I used *rbd export-diff*. I was able to take the full backup using *rbd export*, but when I tried to import the incremental backup using *rbd import-diff*, I encountered an error: Steps I followed to take the backup: # cd volume-ee6f1396-7082-4475-bc0e-d7005b3331db/ # rbd snap create volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db@snap1-$(date +%Y%m%dT%H%M) # rbd export volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db@snap1-20250820T1227 snap1-20250820T1227.rbd2 --export-format --------- Write data in to the volume # rbd snap create volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db@snap2-$(date +%Y%m%dT%H%M) # rbd snap ls volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db SNAPID NAME SIZE PROTECTED TIMESTAMP 202 snap1-20250820T1227 1 GiB Wed Aug 20 12:27:16 2025 203 snap2-20250820T1231 1 GiB Wed Aug 20 12:31:08 2025 # rbd export-diff volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db@snap2-20250820T1231 --from-snap snap1-20250820T1227 snap2-20250820T1231.rbdiff # ls -lh total 782M -rw-r--r-- 1 adminuser adminuser 600M Aug 20 12:30 snap1-20250820T1227.rbd2 -rw-r--r-- 1 adminuser adminuser 182M Aug 20 12:37 snap2-20250820T1231.rbdiff # rbd import snap1-20250820T1227.rbd2 volumes/vm1 Importing image: 100% complete...done. # rbd import-diff snap2-20250820T1231.rbdiff volumes/vm1 start snapshot 'snap1-20250820T1227' does not exist in the image, aborting Importing image diff: 0% complete...failed. rbd: import-diff failed: (22) Invalid argument Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

1 0

MAX_AVAIL becomes 0 bytes when setting osd crush weight to low value.
by Justin Mammarella 19 Aug '25

19 Aug '25

Hello, We’re seeing the MAX_AVAIL value in ceph df instantaneously drop to 0 Bytes / 100% full when specific osds have their crush weight set to low values. The osds are otherwise healthy, and ceph osd df does not show their utilization to be above 70%. ceph version 19.2.2 CLASS SIZE AVAIL USED RAW USED %RAW USED mf1hdd 19 PiB 8.9 PiB 10 PiB 10 PiB 53.83 to POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL mf1fs_data 1 16384 6.8 PiB 2.51G 10 PiB 100.00 0 B We’re running a 9+3 EC pool. This cluster has 1139 osds / 46 host. We’re in the process of downsizing the cluster and draining nodes via crush reweight is part of our normal operations. It happened once a few weeks ago, and we isolated the issue to the weight on a single osd. Now it’s happening during rebalance on multiple osds, at some point the movement of PGs triggers an edge case causing the MAX AVAIL calculation to fail if the crush weight is too low. Example crush weights 1418 mf1hdd 0.02000 osd.1418 up 1.00000 1.00000 1419 mf1hdd 0.02000 osd.1419 up 0 1.00000 2110 mf1hdd 0.02000 osd.2110 up 1.00000 1.00000 2111 mf1hdd 0.02000 osd.2111 up 0 1.00000 2112 nvmemeta 0.02000 osd.2112 up 1.00000 1.00000 Any ideas before I file a bug report? Thankyou

2 4

/var/lib/ceph/crash/posted does not exist
by lejeczek 19 Aug '25

19 Aug '25

Hi guys. I'm missing: /var/lib/ceph/crash/posted on each node, one node where 'mon' fails: -> $ ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum podster3,podster2 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon mon.podster1.mine.priv on podster1.mine.priv is in error state [WRN] MON_DOWN: 1/3 mons down, quorum podster3,podster2 mon.podster1.mine.priv (rank 0) addr [v2:10.1.1.61:3300/0,v1:10.1.1.61:6789/0] is down (out of quorum) started logging: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create Is this expected - on newly deployed cluster with 'cephadm' - that this path does not exist, with 'crash' module already enabled? Is this the SOP that user creates (also perhaps do other necessary) this path manually in order for 'crash' to be functional? many thanks, L.

3 5

Re: Follow-up on Ceph RGW Account-level API
by William Edwards 19 Aug '25

19 Aug '25

> Op 19 aug 2025 om 09:40 heeft Pavithraa AG <pavithraa.ag(a)zybisys.com> het volgende geschreven: > > Hi ceph community, > Are you, by any chance, using an LLM to write your messages? It would save everyone a lot of reading to strip the redundant parts. > > Thank you for sharing the link to the AdminOps API documentation [1]. We have reviewed it thoroughly; however, we could not find endpoints specifically related to account-level operations. > > In particular, we are looking for REST API equivalents for the following actions, so that they can be used safely in production without relying on radosgw-admin commands: > > Account creation > > > Account quota update > > > Account user management (add/remove/list users under an account) > > > Retrieve account details > > > > Could you please confirm: > > Whether there are existing REST API endpoints for these actions (possibly under a different section of the documentation)? > > > If not, whether there is a roadmap or recommended alternative for implementing these account-level operations in a production environment? > > > > Any additional guidance, examples, or references from the community would be greatly appreciated. > > > > > > > > > > > > > > > > Pavithraa AG > > Associate Software Engineer > > mailto:pavithraa.ag@zybisys.com > > https://www.zybisys.com > > > > > > https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys > https://www.facebook.com/Zybisys/ > > > > > > > > > > > > > > > > > > > > > > > > ---- On Mon, 11 Aug 2025 12:48:13 +0530 Dhivya G < mailto:dhivya.g@zybisys.com > wrote --- > > > > Dear Ceph Community, > > I hope this message finds you well. > > We are currently working on integrating Ceph RGW into our system and have encountered a challenge regarding account-level management via API. > > At present, we are using radosgw-admin commands programmatically within our code to manage and retrieve account-level information. While this approach works, we understand that relying on radosgw-admin is not recommended for production environments due to its direct interaction with the backend and potential impact on performance and stability. > > We have explored the available documentation and interfaces but could not find a dedicated Account-level API (e.g., for managing users, retrieving account metadata, etc.) that can be safely used in production over RESTful protocols (such as S3 or Swift-compatible APIs). > > We are seeking clarification on the following points: > > Is there an officially supported Account API in Ceph RGW that allows account-level operations via REST or other production-safe interfaces? > > > If not, is there a roadmap or recommended alternative for performing these actions in a production-ready manner? > > > Are there any best practices or tools from the community that can bridge this gap? > > > > We would greatly appreciate any guidance, documentation, or recommendations you can share . > > > > > > > > Best regards > > > > Dhivya G > > > > > > > > > > > https://www.facebook.com/Zybisys/ > Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. > > > > E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email." > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Follow-up on Ceph RGW Account-level API
by Pavithraa AG 19 Aug '25

19 Aug '25

Hi ceph community, Thank you for sharing the link to the AdminOps API documentation [1]. We have reviewed it thoroughly; however, we could not find endpoints specifically related to account-level operations. In particular, we are looking for REST API equivalents for the following actions, so that they can be used safely in production without relying on radosgw-admin commands: Account creation Account quota update Account user management (add/remove/list users under an account) Retrieve account details Could you please confirm: Whether there are existing REST API endpoints for these actions (possibly under a different section of the documentation)? If not, whether there is a roadmap or recommended alternative for implementing these account-level operations in a production environment? Any additional guidance, examples, or references from the community would be greatly appreciated. Pavithraa AG Associate Software Engineer mailto:pavithraa.ag@zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ ---- On Mon, 11 Aug 2025 12:48:13 +0530 Dhivya G < mailto:dhivya.g@zybisys.com > wrote --- Dear Ceph Community, I hope this message finds you well. We are currently working on integrating Ceph RGW into our system and have encountered a challenge regarding account-level management via API. At present, we are using radosgw-admin commands programmatically within our code to manage and retrieve account-level information. While this approach works, we understand that relying on radosgw-admin is not recommended for production environments due to its direct interaction with the backend and potential impact on performance and stability. We have explored the available documentation and interfaces but could not find a dedicated Account-level API (e.g., for managing users, retrieving account metadata, etc.) that can be safely used in production over RESTful protocols (such as S3 or Swift-compatible APIs). We are seeking clarification on the following points: Is there an officially supported Account API in Ceph RGW that allows account-level operations via REST or other production-safe interfaces? If not, is there a roadmap or recommended alternative for performing these actions in a production-ready manner? Are there any best practices or tools from the community that can bridge this gap? We would greatly appreciate any guidance, documentation, or recommendations you can share . Best regards Dhivya G https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 1

Re: v19.2.3 Squid released
by Justin Owen 18 Aug '25

18 Aug '25

Hello, Is there any update on this CentOS Stream vs Enterprise Linux dependency issue? Wondering if there is any sort of workaround or a bug that I can follow for the progress. We're using OL9 as our container host, but can cannot update the ceph-common tools or ceph-volume packages (which have some bug fixes we'd like) to match the 19.2.3 of the containers. Thanks Justin

1 0

crash - auth: unable to find a keyring ... (2) No such file or directory
by lejeczek 18 Aug '25

18 Aug '25

Hi guys. I have one crash - containerized cluster - which upon start logs: ... INFO:ceph-crash:pinging cluster to exercise our key 2025-08-18T14:59:59.663+0000 7fe9106bb640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-18T14:59:59.663+0000 7fe9106bb640 -1 AuthRegistry(0x7fe908069670) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-18T14:59:59.664+0000 7fe9106bb640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-18T14:59:59.664+0000 7fe9106bb640 -1 AuthRegistry(0x7fe9106b9ff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-18T14:59:59.665+0000 7fe90e430640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2025-08-18T14:59:59.665+0000 7fe90dc2f640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] In that 'crash' container I have: -> $ ll /etc/ceph/ total 16 -rw-------. 1 ceph ceph 81 Aug 6 13:26 ceph.client.crash.podster1.keyring -rw-------. 1 ceph ceph 253 Aug 6 13:26 ceph.conf -rw-------. 1 root root 151 Aug 18 04:20 ceph.keyring -rw-r--r--. 1 root root 92 Jul 17 03:53 rbdmap then: -> $ ceph auth get client.admin [client.admin] key = AQD.... caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" which key is in: /etc/ceph/ceph.keyring on this 'crash' What is happening here can anybody suggest/tell? Other 'crashes' remain that-error-free - but I wonder if that is only because they have had nothing to do yet. This one 'crash' has - on the same node - monitor failing. many thanks, L.

1 0

Ceph upgrade OSD unsafe to stop
by GLE, Vivien 18 Aug '25

18 Aug '25

Hi, I'm trying to update my cluster (19.2.2 -> 19.2.3), mon and mgr upgrade goes well but I had some issue with OSD : Upgrade: unsafe to stop osd(s) at this time (165 PGs are or would become offline) Cluster is in health_ok All pools are replica 3 and pgs active+clean autoscaler is off following the ceph docs Does ceph osd ok-to-stop lead to lost data ? The only rules used in the cluster is replicated_rule : root@ceph-monitor-1:/# ceph osd crush rule dump replicated_rule { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "inist" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } root@ceph-monitor-1:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 6.63879 root inist -15 0.90970 host ceph-monitor-1 6 hdd 0.90970 osd.6 up 1.00000 1.00000 -21 5.72910 datacenter bat1 -20 2.72910 room room01 -19 2.72910 row left -18 2.72910 rack 10 -3 0.90970 host ceph-node-1 2 hdd 0.90970 osd.2 up 1.00000 1.00000 -5 0.90970 host ceph-node-2 1 hdd 0.90970 osd.1 up 1.00000 1.00000 -9 0.90970 host ceph-node-3 5 hdd 0.90970 osd.5 up 1.00000 1.00000 -36 3.00000 room room03 -35 3.00000 row left06 -34 3.00000 rack 08 -7 1.00000 host ceph-node-4 0 hdd 1.00000 osd.0 up 1.00000 1.00000 -13 1.00000 host ceph-node-5 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph-node-6 4 hdd 1.00000 osd.4 up 1.00000 1.00000 Thanks ! Vivien

4 5

ceph.io certificate expired :-/
by Dan O'Brien 17 Aug '25

17 Aug '25

Looks like the Let's Encrypt auto-renew caught a snag. Issued On Monday, May 19, 2025 at 6:27:52 AM Expires On Sunday, August 17, 2025 at 6:27:51 AM

1 0

Replicas
by Shawn Heil 16 Aug '25

16 Aug '25

I recently set up a 5-node Proxmox/Ceph cluster with 10 OSDs each node for a total for 50 nodes (approximately 135TB). All the nodes have 768GB RAM and a single 72-Core (144 thread) Epyx CPU. I set the ceph pool size to: 3 and the minimum size: 2. The performance seemed good and everything was happy in cephland! The other day I spoke to a ceph consultant and he recommended I change the pool size from 3/2 to 2/1. He cited several valid points: more usable storage space, faster rebuilds, better performance... I followed his advice and changed it. The benchmark performance was about the same, but the recovery time when I took a node down was improved. I really do like the idea of the extra storage space! So now I am confused whether I should leave it or go back to 3 replicas. Anyone have any thoughts or compelling reasons I should leave it or change it back? Shawn Shawn Heil Phone: (608) 836-7041 | Direct: 6084102333 Email: shawn.heil(a)brucecompany.com<mailto:shawn.heil@brucecompany.com> | Website: www.brucecompany.com<http://www.brucecompany.com> [cid:SocialLink_Facebook_32x32_24bd5ab0-6667-4cda-9838-e5caaf24a024.png]Find us on Facebook<https://www.facebook.com/pages/The-Bruce-Company/113279807065?ref=hl> [cid:BruceCompanyHomepageLogo(2025)_429ac410-efd6-49ca-8739-0317666483cd.jpg]

3 2

OSD's are Moving back from custom bucket...
by Devender Singh 16 Aug '25

16 Aug '25

Hello all Why my buckets are moving back when restarting osd…how to make them persistent.. After move command crush shows as below… host custom { id -9 # do not change unnecessarily id -10 class hdd # do not change unnecessarily # weight 0.01469 alg straw2 hash 0 # rjenkins1 item osd.3 weight 0.00490 item osd.4 weight 0.00490 item osd.5 weight 0.00490 } root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd crush move osd.3 host=custom root=default ceph osd crush move osd.4 host=custom root=default ceph osd crush move osd.5 host=custom root=default no need to move item id 3 name 'osd.3' to location {host=custom,root=default} in crush map no need to move item id 4 name 'osd.4' to location {host=custom,root=default} in crush map no need to move item id 5 name 'osd.5' to location {host=custom,root=default} in crush map root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 up 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph orch daemon restart osd.5 Scheduled to restart osd.5 on host 'ceph-01.tinihub.com' root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 up 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01469 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -1 0.08789 root default -5 0.02930 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.00980 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 -1 0.09279 root default -5 0.03419 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 5 hdd 0.00490 osd.5 down 0 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000 root@ceph-01:/var/lib/ceph/51b1bcd8-3527-11f0-b51b-dd58cdde2818/osd.5# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.00980 host custom 3 hdd 0.00490 osd.3 up 0 1.00000 4 hdd 0.00490 osd.4 up 0 1.00000 -1 0.09279 root default -5 0.03419 host ceph-01 2 hdd 0.02930 osd.2 up 1.00000 1.00000 5 hdd 0.00490 osd.5 up 0 1.00000 -3 0.02930 host ceph-02 1 hdd 0.02930 osd.1 up 1.00000 1.00000 -7 0.02930 host ceph-03 0 hdd 0.02930 osd.0 up 1.00000 1.00000

2 1

How to test run the new EC Erasure Coding Enhancements Optimizations?
by Chris Lawsonn 15 Aug '25

15 Aug '25

Hi, I am new ignorant user and want to practice test any of the new EC Optimizations that are being worked on for tentacle. How can I test install and setup using any of the new code that includes the EC Optimizations? What would be easiest steps to standup a test environment and how to run pre-release code? Thanks

5 5

August User / Dev Meeting
by Anthony Middleton 15 Aug '25

15 Aug '25

Hey everyone, Just a quick update: there won't be a User + Dev meeting on Wednesday, August 20th, since many team leads are busy. Our next meeting will be on Wednesday, September 17th. Start adding topics to the agenda beforehand: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes. Please continue to ask questions and share your concerns on the #ceph-devel Slack and mailing list. Have a great weekend! Best, Anthony Middleton Ceph Community Manager community-manager(a)ceph.io

1 0

Re: Default firewall zone
by Sake Ceph 15 Aug '25

15 Aug '25

Found a solution: sudo firewall-cmd --set-default-zone=ZONE_NAME Now need to find a way to make sure the company default aren't being applied again. > Op 14-08-2025 23:48 CEST schreef Anthony D'Atri <anthony.datri(a)gmail.com>: > > > There are all kinds of firewalls in use, so this would be difficult. > > > On Aug 14, 2025, at 1:41 AM, Sake Ceph <ceph(a)paulusma.eu> wrote: > > > > Is it possible to set the firewall zone to use for the orchestrator/cephadm to use? If we don't provide the zone when configuring the firewall, the default zone is drop in our company (don't ask me why). > > > > Kind regards, > > Sake > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

SuSE stops building Ceph packages for its distributions
by Robert Sander 15 Aug '25

15 Aug '25

Hi, SuSE will not build even the Ceph client packages for its distributions any more: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org/threa… https://www.suse.com/releasenotes/x86_64/SUSE-SLES/15-SP6/index.html#jsc-PE… This means that this distribution cannot consume Ceph storage any more, if no other party will build these packages. If you are using SuSE Linux Enterprise Server or OpenSuSE as a client for Ceph storage you need to plan for this. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 4

Safe Procedure to Increase PG Number in Cache Pool
by Vishnu Bhaskar 15 Aug '25

15 Aug '25

Hi Team, I have an OpenStack Ceph cluster where the Cinder volumes pool uses a cache pool named *volumes_cache*. Due to performance issues, I observed that the cache pool has fewer PGs and a high PG-to-object ratio, approximately 10x higher than normal. For instance, one PG contains around 35,000 objects. I am planning to increase the PG number for the cache pool. For the base pool, I was able to increase the PG count without any issues. However, for the cache pool, I encountered a warning indicating that a force argument is required. In my lab environment, I found that changing the cache pool mode to *none* allowed me to modify the PG number. Since this is my production setup, I would like to know if there is a safe and recommended procedure to increase the PG number of a *cache pool* without impacting the environment. Kindly advise on the best approach. Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

3 5

tentacle 20.1.0 RC QE validation status
by Yuri Weinstein 14 Aug '25

14 Aug '25

Details of this release are summarized here: https://tracker.ceph.com/issues/72316#note-1 Release Notes - TBD LRC upgrade - TBD Dev Leads, please review the list of suites for completeness, as this is a new release. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venly orch: Adam King rbd: Ilya krbd: Ilya upgrade/squid-x: Laura upgrade/reef-x: Laura upgrade/telemetry-upgrade: PASSED upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: N/A upgrade-clients:client-upgrade-squid-tentacle-tentacle: N/A perf-basic: PASSED crimson-rados: PASSED powercycle: PASSED ceph-volume - Guillaume TIA

8 11

Default firewall zone
by Sake Ceph 14 Aug '25

14 Aug '25

Is it possible to set the firewall zone to use for the orchestrator/cephadm to use? If we don't provide the zone when configuring the firewall, the default zone is drop in our company (don't ask me why). Kind regards, Sake

1 0

Backup Best Practices
by Anthony Fecarotta 14 Aug '25

14 Aug '25

Hello, I'm curious how the community handles backup and disaster recovery planning with Ceph in production environments. Does running Ceph fundamentally change your backup strategy compared to traditional storage solutions? Specifically, I'm interested in: - Are you backing up at the RBD level, or relying on application-level backups? - How do you handle cross-site replication for disaster recovery? - Any tools or practices you'd recommend for automated backup scheduling? Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

5 6

Re: cluster without quorum
by Eugen Block 13 Aug '25

13 Aug '25

Hi, I think the ceph-users list is more appropriate for your request, I removed the dev list from the recipients If I understand correctly, you should still have one surviving monitor, is that correct? The quickest way to restore the quorum would probably be to edit the monmap (make a backup) by removing the dead monitors and startup the surviving monitor. This would give you a single-mon cluster, from there you could then redeploy the other monitors and restore OSD functionality. > I tried reinstalling the system on one node and recreating a monitor > using the data taken from /var/lib/ceph/osd/, but I don't see the > OSDs. This is the situation… Is this a cephadm managed cluster? In that case the OSDs have their data directory in /var/lib/ceph/{CLUSTER_FSID}/osd.{OSD_ID}/. If it's not cephadm managed, you'll need to provide more information with details and logs etc. Regards, Eugen Zitat von FRANCESCO NAPOLI <francesco.napoli(a)cnr.it>: > Good morning, Ceph was installed on Ubuntu Server 24.04 with > cephadm. The cluster consists of 4 nodes; the first one also acted > as a monitor in the initial state, but as all the others were added, > in addition to being OSD hosts, they also acted as monitors. These > machines had the major flaw of not having the operating system disk > in RAID, and after 20 days, 3 nodes failed. The OSD disks are > fine... I tried reinstalling the system on one node and recreating a > monitor using the data taken from /var/lib/ceph/osd/, but I don't > see the OSDs. This is the situation… > thanks for the support. > > -- > Napoli Francesco > Gruppo Reti e Sistemi Informativi > Istituto di Fisiologia Clinica - CNR > Tel. 050.315.2344

1 0

Problem deploying ceph 19.2.3 on Rocky linux 9
by wodel youchi 13 Aug '25

13 Aug '25

Hi, I am having issues trying to deploy ceph 19.2.3 on rocky linux 9. 1 - I cannot install ceph-common : it needs openssl 3.4, rocky 9 offers only openssl 3.2 2 - I bootstraped my cluster like this : cephadm bootstrap \ --mon-ip 10.200.15.10 \ --cluster-network 10.210.0.0/16 \ --initial-dashboard-user admin \ --initial-dashboard-password 'admin' \ --ssh-private-key /root/.ssh/id_rsa \ --ssh-public-key /root/.ssh/id_rsa.pub it did bootstrap, but when I try adding other hosts, I cannot, it has something to do with ssh. I did copy id.rsa.pub on the other nodes via ssh-copy-id, and I can ssh without password to them. but I cannot ssh to them via cephadm shell. and when i try to add a host I get : Aug 11 15:29:06 hyper01 ceph-mgr[35633]: [cephadm ERROR cephadm.ssh] Traceback (most recent call last): File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for return fut.result() asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log yield File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[ self.mgr.tkey.name], File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect return await asyncio.wait_for( File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError Aug 11 15:29:06 hyper01 ceph-mgr[35633]: log_channel(cephadm) log [ERR] : Traceback (most recent call last): File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for return fut.result() asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log yield File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[ self.mgr.tkey.name], File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect return await asyncio.wait_for( File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: 2025-08-11T14:29:06.509+0000 7f27b44ed640 -1 log_channel(cephadm) log [ERR] : Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Traceback (most recent call last): Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: await options.waiter Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: asyncio.exceptions.CancelledError Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: During handling of the above exception, another exception occurred: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Traceback (most recent call last): Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: return fut.result() Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: asyncio.exceptions.CancelledError Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: The above exception was the direct cause of the following exception: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Traceback (most recent call last): Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: yield Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: return await asyncio.wait_for( Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: raise exceptions.TimeoutError() from exc Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: asyncio.exceptions.TimeoutError Aug 11 15:29:06 hyper01 ceph-mgr[35633]: [cephadm ERROR orchestrator._interface] Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed Traceback (most recent call last): File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for return fut.result() asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log yield File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[ self.mgr.tkey.name], File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect return await asyncio.wait_for( File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1775, in _check_valid_addr out, err, code = self.wait_async(CephadmServe(self)._run_cephadm( File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/serve.py", line 1595, in _run_cephadm await self.mgr.ssh._remote_connection(host, addr) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _remote_connection raise File "/lib64/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 214, in redirect_log raise HostConnectionError( cephadm.ssh.HostConnectionError: Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3502, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3251, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1800, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1789, in _check_valid_addr raise OrchestratorError(str(e)) orchestrator._interface.OrchestratorError: Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed Aug 11 15:29:06 hyper01 ceph-mgr[35633]: log_channel(cephadm) log [ERR] : Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed Traceback (most recent call last): File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for return fut.result() asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log yield File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[ self.mgr.tkey.name], File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect return await asyncio.wait_for( File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1775, in _check_valid_addr out, err, code = self.wait_async(CephadmServe(self)._run_cephadm( File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/serve.py", line 1595, in _run_cephadm await self.mgr.ssh._remote_connection(host, addr) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _remote_connection raise File "/lib64/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 214, in redirect_log raise HostConnectionError( cephadm.ssh.HostConnectionError: Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3502, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3251, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1800, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1789, in _check_valid_addr raise OrchestratorError(str(e)) orchestrator._interface.OrchestratorError: Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed Regards.

2 1

Performance scaling issue with multi-SSD (CrimsonOSD/Seastore)
by Ki-taek Lee 13 Aug '25

13 Aug '25

Hello Ceph community, I am evaluating Crimson OSD + Seastore performance for potential deployment in a distributed storage environment. With BlueStore, I have been able to achieve satisfying performance levels in my FIO tests for 4K random read/write IOPS. However, when testing Crimson OSD + Seastore, I observed that 4K random read/write IOPS do not scale as expected when increasing the number of SSDs/OSDs. The performance plateaus beyond a certain point or is much lower than expected. (See attached test results.) Test Environment: - Cluster: 8 clients, 1 OSD - Hardware: 40-core CPUs, 377 GiB DRAM - Image SHA (quay.io): e0543089a9e9cae97999761059eaccdf6bb22e9e - Configuration parameters: osd_memory_target = 34359738368 crimson_osd_scheduler_concurrency = 0 seastore_max_concurrent_transactions = 16 crimson_osd_obc_lru_size = 8192 seastore_cache_lru_size = 16G seastore_obj_data_write_amplification = 4 seastore_journal_batch_capacity = 1024 seastore_journal_batch_flush_size = 256M seastore_journal_iodepth_limit = 16 seastore_journal_batch_preferred_fullness = 0.8 seastore_segment_size = 128M seastore_device_size = 512G seastore_block_create = true seastore_default_object_metadata_reservation = 1073741824 rbd_cache = false rbd_cache_writethrough_until_flush = true rbd_op_threads = 16 Replication policy: - 4096 PGs, no replication (only 1 copy) Test Results: 1 SSD test (varying number of allocated CPUs, alien threads = 26-29, 36-39): num CPU | 4k randread | 4k randwrite | Allocated CPU sets 2 | 126772 | 14830 | 0-1 4 | 107860 | 16451 | 0-3 6 | 113741 | 17019 | 0-5 8 | 132060 | 16099 | 0-7 SSD scaling test (2 CPUs per SSD): OSD CPU mapping: OSD.0 (0-1), OSD.1 (10-11), OSD.2 (2-3), OSD.3 (12-13), ..., OSD.15 (34-35), Alien threads (26-29, 36-39) num SSD | 4k randread | 4k randwrite 4 | 861273 | 22360 8 | 1022793 | 22786 12 | 1019161 | 21211 16 | 927570 | 20502 SSD scaling test (1 CPU per SSD): OSD CPU mapping: OSD.0 (0), OSD.1 (10), OSD.2 (2), OSD.3 (12), ..., OSD.15 (24), Alien CPUs: 1, 11, 3, 13, ..., 15, 25 num SSD | 4k randread | 4k randwrite 4 | 936685 | 13730 8 | 1048204 | 18259 12 | 922727 | 23078 16 | 987838 | 30792 Questions: 1. Since Seastore is still under active development, are there any known unresolved performance issues that could explain this scaling behavior? 2. Are there recommended tuning parameters for improving small-block read scalability in multi-SSD configurations? 3. Regarding alien threads, are there best practices for CPU pinning or NUMA-aware placement that have shown measurable improvements? 4. Any additional guidance for maximizing IOPS with Crimson OSD + Seastore would be greatly appreciated. My goal is to be ready to switch from BlueStore to Crimson + Seastore after it becomes stable and shows reasonable performance compared to BlueStore, so I’d like to understand the current limitations and tuning opportunities. Thank you, Ki-taek Lee

4 3

Windows support for Ceph
by Anthony Middleton 12 Aug '25

12 Aug '25

The Ceph Steering Committee (CSC) is evaluating Windows support for Ceph. Please answer the question(s) in the survey below to help guide the CSC's decision. Survey link - https://forms.gle/b6WBry3i6JuLrVfP9 *The deadline to complete this survey is August 25th, 2025. We appreciate your participation.* If you have any questions, please contact Anthony Middleton at community-manager(a)ceph.io.

1 0

Ceph subreddit banned?
by Philipp Hocke 12 Aug '25

12 Aug '25

Hi everyone, I just noticed that the Ceph subreddit has been banned: "This community was banned for violating Rule 2. You agree that use of this site constitutes acceptance of Reddit’s User Agreement and acknowledge our Privacy Policy." Does anyone know what's going on? Best regards Philipp -- Philipp Hocke Leiter Linux Systemadministration / Head Of TechOps CM4ALL GmbH Im Mediapark 6A - 50670 Köln / Cologne Phone +49-(0)221-6601-0 <https://outlook.office.com/mail/sentitems> Fax +49-(0)221-6601-1011 <https://outlook.office.com/mail/sentitems> E-Mail: philipp.hocke(a)cm4all.com Internet: www.cm4all.com

2 1

Announcing go-ceph v0.32.0
by Sven Anderson 12 Aug '25

12 Aug '25

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.32.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

Debugging OSD cache thrashing
by Hector Martin 11 Aug '25

11 Aug '25

Hi all, I have a small 3-node cluster (4 HDD + 1 SSD OSD per node, ceph version 19.2.2) and noticed that during snaptrim ops (?), the OSDs max out CPU usage and cluster performance plummets. Most of the CPU usage was accounted as "system", while actual disk I/O usage was low, so that didn't sound right. I perf traced the system, and found that most of the usage is in __arch_copy_to_user in the kernel, in read() syscalls. That sounds like the OSD is thrashing buffered block device reads which the kernel satisfies from the page cache (hence no real I/O load, most of the CPU usage is from data copies), so it's repeatedly reading the same disk blocks, which doesn't sound right. I increased osd_memory_target to 2.4G (from 1.2G) live with `ceph config`, and CPU usage immediately dropped to near zero. However, waiting a bit eventually the CPU thrashing returns after memory usage increases. Restarting an OSD has a similar effect. I believe that something is wrong with the OSD bluestore cache allocation/flush policy, and when the cache becomes full it starts thrashing reads instead of evicting colder cached data (or perhaps some cache bucket is starving another cache bucket of space). I would appreciate some hints on how to debug this. Are there any cache stats I should be looking at, or info on how the cache is partitioned? Here is a perf call trace of the thrashing: > - OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) > - 95.04% ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&) > - OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&) > - 95.03% PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&) > - PGBackend::handle_message(boost::intrusive_ptr<OpRequest>) > - ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>) > - 88.86% ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&, ECListener&) > - 88.78% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>) > - BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusiv > - 88.75% BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*) > - 88.71% BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&) > - BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&) > - 88.69% BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std > - 88.68% BlueStore::_wctx_finish(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, BlueStore::Writ > - 88.65% BlueStore::Collection::load_shared_blob(boost::intrusive_ptr<BlueStore::SharedBlob>) > - 88.64% RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::ch > - 88.61% rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*) > - rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::DBImpl::GetImplOptions&) > - 88.43% rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::PinnableWideColumns*, std > - 88.41% rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Sl > - rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::SliceTransform const*, bo > - 88.32% rocksdb::BlockBasedTable::FullFilterKeyMayMatch(rocksdb::FilterBlockReader*, rocksdb::Slice const&, bool, rocksdb::SliceTransfor > - rocksdb::FullFilterBlockReader::MayMatch(rocksdb::Slice const&, bool, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, rocks > - 88.29% rocksdb::FilterBlockReaderCommon<rocksdb::ParsedFullFilterBlock>::GetOrReadFilterBlock(bool, rocksdb::GetContext*, rocksdb > - rocksdb::FilterBlockReaderCommon<rocksdb::ParsedFullFilterBlock>::ReadFilterBlock(rocksdb::BlockBasedTable const*, rocksdb::Fi > - rocksdb::Status rocksdb::BlockBasedTable::RetrieveBlock<rocksdb::ParsedFullFilterBlock>(rocksdb::FilePrefetchBuffer*, rocks > - rocksdb::Status rocksdb::BlockBasedTable::MaybeReadBlockAndLoadToCache<rocksdb::ParsedFullFilterBlock>(rocksdb::FilePref > - 88.25% rocksdb::BlockFetcher::ReadBlockContents() [ note: split, the rest is + 19.42% rocksdb::VerifyBlockChecksum] > - 68.80% rocksdb::RandomAccessFileReader::Read(rocksdb::IOOptions const&, unsigned long, unsigned long, rocksdb::Sli [ note: at this level the call trace splits into 4, but it all leads to the same place ] > - 35.99% 0xaaaac0992e40 > BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const > BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*) > - KernelDevice::read_random(unsigned long, unsigned long, char*, bool) > - 35.98% __libc_pread > - el0_svc > - invoke_syscall > - 35.97% __arm64_sys_pread64 > - 35.96% vfs_read > - blkdev_read_iter > - 35.93% filemap_read > - 35.28% copy_page_to_iter > 35.01% __arch_copy_to_user - Hector

7 17

How to change to RocksDB LZ4 after upgrade to Ceph 19
by Niklas Hambüchen 11 Aug '25

11 Aug '25

Release notes https://ceph.io/en/news/blog/2024/v19-2-0-squid-released/#highlights say: > BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. On one EC pool with many files I have such warnings: osd.6 spilled over 8.0 GiB metadata from 'db' device (36 GiB used of 60 GiB) to slow device and hope that the LZ4 compression might help with that. What do I need to do to "migrate" my OSDs to that? I already upgrade the cluster to Ceph 19. Running grep 'Options.compression: ' /var/log/ceph/ceph-osd.6.log already shows the switch from rocksdb: Options.compression: NoCompression to rocksdb: Options.compression: LZ4 in the logs. But does restarting the OSDs already automtaically compress everything with LZ4, or do I need to run some operation such as ceph daemon osd.6 compact or similar to make sure that existing data is actually compressed? Also, could anybody clarify whether this setting is `compression_algorithm` from https://docs.ceph.com/en/squid/rados/operations/pools/#setting-pool-values https://docs.ceph.com/en/squid/rados/configuration/bluestore-config-ref/#co… or if that's something different (e.g. if that's "actual data" instead of "metadata")? I suspect it's different, because `ceph config show-with-defaults osd.6 | grep compression` reveals: bluestore_compression_algorithm snappy bluestore_compression_mode none bluestore_rocksdb_options compression=kLZ4Compression,... From this it looks like `bluestore_compression*` is the "Inline compression" for data (https://docs.ceph.com/en/reef/rados/configuration/bluestore-config-ref/#inl…) and `bluestore_rocksdb_options` is what the "RocksDB compression" is about. Still the question remains on how to bring all existing data over to be compressed. Thanks!

3 2

Join the Ceph New Users Workshop
by Anthony Middleton 11 Aug '25

11 Aug '25

Registration for the Ceph New Users Workshop is now open! This free workshop will be held on Monday, Oct 27, the day before Cephalocon, in Vancouver. RSVP here: https://events.clyso.tech/event/2/registrations/2/ Registration closes on Oct 24 at 11:59 p.m. PDT. Priority will be given to users who are registered for Cephalocon. If you have any questions, don't hesitate to get in touch with me at community-manager(a)ceph.io.

1 0

Best/Safest way to power off cluster
by gagan tiwari 11 Aug '25

11 Aug '25

Hi Guys, I have recently set-up a production ceph cluster which consists of 3 monitor nodes and 7 Osd nodes. There is power maintenance activity scheduled at the data centre coming weekend and due to that I need to power off all the devices. Can you please advise me on the safest way to power off all servers? Should I power off all 7 OSD servers one by one followed by all 3 monitor nodes or vice versa? Thanks, Gagan

6 14

Squid 19.2.3 rm-cluster does not zap OSDs
by Eugen Block 11 Aug '25

11 Aug '25

Hi *, I have a VM which I use frequently to test cephadm bootstrap operations as well as upgrades, it's a single node with a few devices attached. After successfully testing the upgrade to 19.2.3, I wanted to test the bootstrap again, but removing the cluster with the --zap-osds flag doesn't actually remove the VGs/LVs. This used to work just fine until 19.2.2. This is the command I used: ceph:~ # cephadm --image myregistry/ceph_v19.2.3 rm-cluster --fsid {FSID} --zap-osds --force There's not much to see in the cephadm.log after these sort of lines: 2025-08-01 09:04:02,103 7f3bf5206b80 DEBUG systemctl: Removed "/etc/systemd/system/ceph-6b501d0a-6ea3-11f0-a251-fa163e2ad8c5.target.wants/ceph-6b501d0a-6ea3-11f0-a251-fa163e2ad8c5(a)osd.0.service". But it looks like the inventory output in json format, it's quite lengthy, so I'll spare you the output here. I can add it to a tracker though, if there isn't one yet. Has this already been reported? Thanks, Eugen

1 1

Display ceph version on ceph -s output
by Frédéric Nass 11 Aug '25

11 Aug '25

Dear community, Some time ago, I wondered why the Ceph version didn't show up in the ceph status (ceph -s) output. Perhaps it was because a Ceph cluster could run multiple versions at once (during an upgrade for example) and this information wasn't centralized, before the 'ceph versions' command was introduced. Regardless, I think it would be a great addition to have the Ceph version displayed in the ceph status. It could look something like this below. On a cluster where all daemons are on version 18.2.7: $ ceph -s cluster: version: 18.2.7 id: f4147dec-4c5d-11f0-a310-bc24113b5ce1 health: HEALTH_OK On a cluster that is currently being upgraded from 18.2.4 to 18.2.7: $ ceph -s cluster: version: 18.2.7 (432), 18.2.4 (248) <----- Could be refined. For example (3 MONs, 3 MGRs, 426 OSDs) until all daemons are on the same version. id: f4147dec-4c5d-11f0-a310-bc24113b5ce1 health: HEALTH_WARN Let me know if you like the idea and I'll create a tracker. Best regards, Frédéric. -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass(a)clyso.com

7 6

osd latencies and grafana dashboards, squid 19.2.2
by Christopher Durham 11 Aug '25

11 Aug '25

In my 19.2.2/squid cluster, (Rocky 9 Linux) I am trying to determine if I am having issues with OSD latency. The following URL: https://sysdig.com/blog/monitoring-ceph-prometheus/ states the following about prometheus metrics: * ceph_osd_op_r_latency_count: Returns the number of reading operations running. * ceph_osd_op_r_latency_sum: Returns the time, in milliseconds, taken by the reading operations. This metric includes the queue time. * ceph_osd_op_w_latency_count: Returns the number of writing operations running. * ceph_osd_op_w_latency_sum: Returns the time, in milliseconds, taken by the writing operations. This metric includes the queue time. and * ceph_osd_commit_latency_ms: Returns the time it takes OSD to read or write to the journal. * ceph_osd_apply_latency_ms: Returns the time it takes to write the journal to the physical disk. The first set states 'includes the queue time'. What exactly does this mean? Does this mean that this is the time waiting before writing to the journal while in the memory of the ceph-osd daemon? If so, does the latter two metrics mean that once the writes start, this is the time it takes to write to the journal or the disk? Does the first set of metrics *include* the latter? In other words, are the apply/commit latencies included in the *[r,w]_latency_sum? The URL above suggests that to calculate the write latency for a given OSD, you do the following: (rate(ceph_osd_op_w_latency_sum[5m]) / rate(ceph_osd_op_w_latency_count[5m]) >= 0) However, the grafana dashboard 'OSD Overview' (in ceph-dashboards-19.2.2 rpm), does something very similar for max,avg,quantile: max(rate(ceph_osd_op_w_max_latency_sum(cluster=\$cluster\,}[$__rate_interval])/ on (ceph_daemon) rate(ceph_osd_op_w_max_latency_count(cluster=\$cluster\,}[$__rate_interval]) * 1000)) The extra multiple of 1000 seems extraneous based on the fact that the *latency_count is already in milliseconds, and the graph itself shows 'ms'. and led me to think I have disk latency issues, as these numbers are high. (maybe I do, and maybe I misunderstand something) Other dashboards such as 'Ceph Cluster - Advanced' which, in the panel 'OSD Commit Latency Distribution' use similar promQL expressions but without the extra multiple of 1000, which looks alot better for evaluation of my latencies. So the question is, is the extra multiple of 1000 incorrect in the 'OSD Overview' dashboard? Or am I not understanding things correctly? Also, does 'ceph osd perf' just show the apply/commit sum/count metrics from above? Thanks for any assistance. -Chris

2 2

Squid 19.2.1 dashboard javascript error
by Chris Palmer 09 Aug '25

09 Aug '25

Firstly thank you so much for the 19.2.1 release. Initial testing suggests that the blockers that we had in 19.2.0 have all been resolved, so we are proceeding with further testing. We have noticed one small problem in 19.2.1 that was not present in 19.2.0 though. We use the older-style dashboard (mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false). The problem happens on the Dashboard screen when health changes to WARN. If you click on WARN you get a small empty dropdown instead of the list of warnings. A javascript error is logged, and using browser inspection there is the additional bit of info that it happens in polyfill: 2025-02-07T15:59:00.970+0000 7f1d63877640 0 [dashboard ERROR frontend.error] (https://<redacted>:8443/#/dashboard): NG0901 Error: NG0901 at d.find (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3342365) at le.ngDoCheck (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3173112) at Qe (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225586) at bt (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225341) at cs (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225051) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259043) at jf (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3266563) at S1 (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259790) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259801) at fg (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3267248) Also, after this happens, no dropdowns work again until the page is forcibly refreshed. Environment is RPM install on Centos 9 Stream. I've created issue [0]. Thanks, Chris [0] https://tracker.ceph.com/issues/69867 <https://tracker.ceph.com/issues/69867?next_issue_id=69865&prev_issue_id=90>

4 9

ceph 19.2.2 - adding new hard drives messed up the order of existing ones - OSD down
by Steven Vacaroaia 08 Aug '25

08 Aug '25

Hi, I needed to add more spinning HDD to my nodes ( SuperMicro SSG-641E-E1CR36L) and made the mistake of NOT setting up osd_auto_discovery to "false" so ceph created OSDs on all 5 new spinning HDD This was an issue as I wanted to configure the OSDs the same as existing ones ( i.e. with WAL/DB on NVME) when the other 37 drives arrive No big harm done though because I could zap them and reconfigure ( after running ceph orch apply osd --all-available-devices --unmaged=true) when I will receive the remaining 37 drives ( as I will add 6 drives on each server) The interesting part is that, for whatever reason, one of the existing SSD based OSD is down now because the SSD drive it used changed from /dev/sdp to /dev/sdu therefore there is no "block" entry under /var/lib/ceph/FSID/osd.XX I am not sure why adding spinning disks mess up the order/naming of sdd disks I would appreciate some advice regarding the best course of action to reconfigure the OSD that is down The cluster is healthy , not busy, with all the other OSDs working as expected Has 7 hosts with 12 SSD and 6 HDD drives each ( one of them is the one with issues), 2 EC 4+2 pool , 2 MDS and few metadata pools replicated on NVME There are also 3 NVME disks on each dedicated to pools Many thanks Steven

2 1

60/90 bays + 6 NVME Supermicro
by Manuel Rios - EDH 08 Aug '25

08 Aug '25

Hi Ceph , Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL? Newest models support 144cores in single socket and several TB of ram without issues. But as far as we understood in the technical notes it use SAS Expanders for connect all disk, from 2 to 4 SAS Expanders to connect the whole chasis. We're looking for next configuration : Intel Xeon 96cores @ 2.4 1TB RAM 2 SSD for OS. 4 NVME x 30TB NVME KIOXIA 90 HDD * 22 TB HGST 4x25 Gbps or 2x100 Gbps Main use RGW. Regards [cid:image001.jpg@01DC07CB.6CB5C6C0] MANUEL RIOS FERNANDEZ CEO - EasyDataHost Phone: 677677179 Web: www.easydatahost.com<http://www.easydatahost.com/> Email: mriosfer(a)easydatahost.com<mailto:mriosfer@easydatahost.com> [T?tulo: LinkedIn - Descripci?n: image of LinkedIn icon] <https://es.linkedin.com/in/manuel-rios-fernandez-14880949?original_referer=…> ADVERTENCIA LEGAL: Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado. Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. [Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Flor ecoTECH]No imprimas si no es necesario. Protejamos el Medio Ambiente.

5 6

How set pg number for pools
by Albert Shih 07 Aug '25

07 Aug '25

Hi, Stupid question but how can I set the number of pg for a pool when I got differents (1.7 To , 2.7T, 2.9To and 3.4To) size of ssd ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 07 août 2025 14:06:35 CEST

2 1

Pgs troubleshooting
by GLE, Vivien 05 Aug '25

05 Aug '25

Hi, After replacing 2 OSD (data corruption), this is the stats of my testing ceph cluster ceph pg stat 498 pgs: 37 peering, 1 active+remapped+backfilling, 1 active+clean+remapped, 1 active+recovery_wait+undersized+remapped, 1 backfill_unfound+undersized+degraded+remapped+peered, 1 remapped+peering, 12 active+clean+scrubbing+deep, 1 active+undersized, 442 active+clean, 1 active+recovering+undersized+remapped 34 GiB data, 175 GiB used, 6.2 TiB / 6.4 TiB avail; 1.7 KiB/s rd, 1 op/s; 31/39768 objects degraded (0.078%); 6/39768 objects misplaced (0.015%); 1/13256 objects unfound (0.008%) ceph osd stat 7 osds: 7 up (since 20h), 7 in (since 20h); epoch: e427538; 4 remapped pgs Anyone had an idea of where to start to get a healthy cluster ? Thanks ! Vivien

4 22

Cephadm using node exporter container from previous installation
by gagan tiwari 05 Aug '25

05 Aug '25

HI Guys, Good Morning! I have re-deployed the ceph cluster using cephadm because of some issues with osds. Everything is working fine now with the new cluster except that Cephadm is still trying to deploy node_exporter daemon on one of the monitor hosts using the old container which no longer exists. node_exporter is running fine on that host in the new container. But ceph is still trying to deploy it on that host using an old container and since it couldn't find that podman container it throws the warning. I did try to remove that node_exporter service on that node and added it again but no luck. When I reboot that host after its coming ceph status shows "Health OK" but after sometime it starts showing same waring ceph health detail HEALTH_WARN Failed to place 1 daemon(s) [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s) Failed while placing node-exporter.ceph-mon2 on ceph-mon2: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-1aa23d18-6eb0-11f0-823c-cc28aa448c14-node-exporter-ceph-mon2 /usr/bin/podman: stderr Error: no such container ceph-1aa23d18-6eb0-11f0-823c-cc28aa448c14-node-exporter-ceph-mon2 Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-1aa23d18-6eb0-11f0-823c-cc28aa448c14-node-exporter.ceph-mon2 /usr/bin/podman: stderr Error: no such container ceph-1aa23d18-6eb0-11f0-823c-cc28aa448c14-node-exporter.ceph-mon2 Deploy daemon node-exporter.ceph-mon2 ... Verifying port 0.0.0.0:9100 ... Cannot bind to IP 0.0.0.0 port 9100: [Errno 98] Address already in use ERROR: TCP Port(s) '0.0.0.0:9100' required for node-exporter already in use Please let me know how to fix it. Thanks, Gagan

2 4

v19.2.3 Squid released
by Yuri Weinstein 04 Aug '25

04 Aug '25

We're happy to announce the third backport release in the Squid series. https://ceph.io/en/news/blog/2025/v19-2-3-squid-released/ Notable Changes --------------- * RGW: PutObjectLockConfiguration can now be used to enable S3 Object Lock on an existing versioning-enabled bucket that was not created with Object Lock enabled. * RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired. Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867 * RBD: Moving an image that is a member of a group to trash is no longer allowed. `rbd trash mv` command now behaves the same way as `rbd rm` in this scenario. * MGR: MGR's always-on modules/plugins can now be force-disabled. This can be necessary in cases where MGR(s) needs to be prevented from being flooded by the module commands when corresponding Ceph service is down/degraded. * RGW: An authentication bypass vulnerability in STS [CVE-2023-43040] has been fixed. * RGW: S3 policy now enforces ARN-based conditionals. * RGW: Copying an object to itself no longer causes data loss. Potential corruption on ETIMEDOUT (not enabled by default), was also fixed. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.3.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: c92aebb279828e9c3c1f5d24613efca272649e62

11 22

Changing the failure domain of an EC cluster still shows old profile
by Niklas Hambüchen 04 Aug '25

04 Aug '25

I made an EC 4+2 cluster with `crush-failure-domain=host`. Later after adding more machines, I changed it from `host` to `datacenter`: ceph osd erasure-code-profile set my_ec_profile_datacenter k=4 m=2 crush-failure-domain=datacenter crush-device-class=hdd ceph osd crush rule create-erasure rule_my_data_ec_datacenter my_ec_profile_datacenter ceph osd pool set my_data_ec crush_rule rule_my_data_ec_datacenter This seems to have worked, and `ceph osd pool get my_data_ec crush_rule` outputs: crush_rule: rule_my_data_ec_datacenter But `ceph osd pool ls detail` still shows pool 3 'my_data_ec' erasure profile my_ec_profile ... with `my_ec_profile` instead of `my_ec_profile_datacenter`. Is this a problem? Who wins, the profile or the crush-rule? If it's not a problem, it is at least confusing; can I fix it somehow? Thanks! Niklas

3 3

to the maintainer/owners of the list - dmar dkim.... Yahoo?
by lejeczek 03 Aug '25

03 Aug '25

Hi guys. If my memory serves me well - being a consumer of quite a few, rhel/fedora related, mailing lists - relatively old issue, namely.. Some policies (dkim, dmarc, more?) Yahoo did change (perhaps other popular mail providers) which resulted is some tweaks to a mailing list are necessary, so mailing list would send senders their own emails. Could it be that this list "suffers" from this problem - I've only subscribed now, I seem not to get my own emails. many thanks, L.

1 0

Performance issues
by Ron Gage 02 Aug '25

02 Aug '25

Hello from Detroit MI: I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD Each OSD has 2 x NVME disks for Ceph, each at 370 Gig The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000) Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate. iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc). The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump. Typical command line used for rados bench: rados bench -p s3block 20 write What the heck am I doing wrong here? Ron Gage

4 8

Re: How to improve write latency on R3 pool Images
by Devender Singh 02 Aug '25

02 Aug '25

Hello Anthony I am directly measuring on all rbd images from ceph node using rbd iostat. Any parameter to improve images ? Regards Dev On Fri, 1 Aug 2025 at 8:05 PM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > How are you measuring this? > > VMs do add an emulation cost on devices. > > > On Aug 1, 2025, at 11:03 PM, Devender Singh <devender(a)netskrt.io> wrote: > > No all’s are ssd and none > > Regards > Dev > > > On Fri, 1 Aug 2025 at 7:58 PM, Anthony D'Atri <anthony.datri(a)gmail.com> > wrote: > >> What is this the output of? Is this reading from HDDs? >> >> > On Aug 1, 2025, at 9:01 PM, Devender Singh <devender(a)netskrt.io> wrote: >> > >> > Hello All >> > >> > Using R3 pool with images running ubuntu vms. >> > >> > Having three nodes… >> > >> > I have default settings for ms_asyc threads to 3 and rbd_op_thread to 1 >> > How to improve below latency…mainly write.... >> > >> > # sed 's/^.\{60\}//' a >> > WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT >> > 50/s 0/s 275 KiB/s 0 B/s 17.36 ms 0.00 ns >> > 45/s 4/s 230 KiB/s 56 KiB/s 66.02 ms 6.90 ms >> > 25/s 0/s 146 KiB/s 0 B/s 68.52 ms 0.00 ns >> > 24/s 0/s 548 KiB/s 0 B/s 22.49 ms 0.00 ns >> > 22/s 0/s 114 KiB/s 0 B/s 20.85 ms 0.00 ns >> > 18/s 0/s 204 KiB/s 0 B/s 48.31 ms 0.00 ns >> > 17/s 0/s 182 KiB/s 0 B/s 10.00 ms 0.00 ns >> > 15/s 4/s 25 MiB/s 17 KiB/s 254.61 ms 17.42 ms >> > 12/s 0/s 268 KiB/s 0 B/s 29.67 ms 0.00 ns >> > 12/s 2/s 13 MiB/s 8.8 KiB/s 147.05 ms 2.30 ms >> > 11/s 0/s 77 KiB/s 0 B/s 50.42 ms 0.00 ns >> > 11/s 0/s 74 KiB/s 0 B/s 35.54 ms 0.00 ns >> > 10/s 0/s 101 KiB/s 0 B/s 80.83 ms 0.00 ns >> > 9/s 0/s 65 KiB/s 0 B/s 24.81 ms 0.00 ns >> > 7/s 0/s 37 KiB/s 0 B/s 59.11 ms 0.00 ns >> > 7/s 0/s 33 KiB/s 0 B/s 43.95 ms 0.00 ns >> > 6/s 8/s 31 KiB/s 34 KiB/s 131.78 ms 127.46 us >> > 4/s 0/s 18 KiB/s 0 B/s 56.90 ms 0.00 ns >> > 4/s 0/s 23 KiB/s 0 B/s 74.13 ms 0.00 ns >> > 3/s 0/s 18 KiB/s 0 B/s 38.25 ms 0.00 ns >> > 2/s 0/s 10 KiB/s 0 B/s 26.28 ms 0.00 ns >> > 2/s 0/s 11 KiB/s 0 B/s 49.80 ms 0.00 ns >> > 1/s 0/s 90 KiB/s 0 B/s 43.02 ms 0.00 ns >> > 1/s 0/s 8.8 KiB/s 0 B/s 24.99 ms 0.00 ns >> > 1/s 0/s 10 KiB/s 0 B/s 102.38 ms 0.00 ns >> > 1/s 0/s 15 KiB/s 0 B/s 57.23 ms 0.00 ns >> > 0/s 0/s 7.2 KiB/s 0 B/s 149.90 ms 0.00 ns >> > 0/s 0/s 4 KiB/s 0 B/s 1.58 ms 0.00 ns >> > 0/s 0/s 3.2 KiB/s 0 B/s 1.02 ms 0.00 ns >> > 0/s 0/s 819 B/s 0 B/s 40.45 ms 0.00 ns >> > >> > Regards >> > Dev >> > >> > >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

1 0

How to improve write latency on R3 pool Images
by Devender Singh 02 Aug '25

02 Aug '25

Hello All Using R3 pool with images running ubuntu vms. Having three nodes… I have default settings for ms_asyc threads to 3 and rbd_op_thread to 1 How to improve below latency…mainly write.... # sed 's/^.\{60\}//' a WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT 50/s 0/s 275 KiB/s 0 B/s 17.36 ms 0.00 ns 45/s 4/s 230 KiB/s 56 KiB/s 66.02 ms 6.90 ms 25/s 0/s 146 KiB/s 0 B/s 68.52 ms 0.00 ns 24/s 0/s 548 KiB/s 0 B/s 22.49 ms 0.00 ns 22/s 0/s 114 KiB/s 0 B/s 20.85 ms 0.00 ns 18/s 0/s 204 KiB/s 0 B/s 48.31 ms 0.00 ns 17/s 0/s 182 KiB/s 0 B/s 10.00 ms 0.00 ns 15/s 4/s 25 MiB/s 17 KiB/s 254.61 ms 17.42 ms 12/s 0/s 268 KiB/s 0 B/s 29.67 ms 0.00 ns 12/s 2/s 13 MiB/s 8.8 KiB/s 147.05 ms 2.30 ms 11/s 0/s 77 KiB/s 0 B/s 50.42 ms 0.00 ns 11/s 0/s 74 KiB/s 0 B/s 35.54 ms 0.00 ns 10/s 0/s 101 KiB/s 0 B/s 80.83 ms 0.00 ns 9/s 0/s 65 KiB/s 0 B/s 24.81 ms 0.00 ns 7/s 0/s 37 KiB/s 0 B/s 59.11 ms 0.00 ns 7/s 0/s 33 KiB/s 0 B/s 43.95 ms 0.00 ns 6/s 8/s 31 KiB/s 34 KiB/s 131.78 ms 127.46 us 4/s 0/s 18 KiB/s 0 B/s 56.90 ms 0.00 ns 4/s 0/s 23 KiB/s 0 B/s 74.13 ms 0.00 ns 3/s 0/s 18 KiB/s 0 B/s 38.25 ms 0.00 ns 2/s 0/s 10 KiB/s 0 B/s 26.28 ms 0.00 ns 2/s 0/s 11 KiB/s 0 B/s 49.80 ms 0.00 ns 1/s 0/s 90 KiB/s 0 B/s 43.02 ms 0.00 ns 1/s 0/s 8.8 KiB/s 0 B/s 24.99 ms 0.00 ns 1/s 0/s 10 KiB/s 0 B/s 102.38 ms 0.00 ns 1/s 0/s 15 KiB/s 0 B/s 57.23 ms 0.00 ns 0/s 0/s 7.2 KiB/s 0 B/s 149.90 ms 0.00 ns 0/s 0/s 4 KiB/s 0 B/s 1.58 ms 0.00 ns 0/s 0/s 3.2 KiB/s 0 B/s 1.02 ms 0.00 ns 0/s 0/s 819 B/s 0 B/s 40.45 ms 0.00 ns Regards Dev

1 0

squid 19.2.2 - osd_memory_target_autotune - best practices when host has lots of RAM
by Steven Vacaroaia 01 Aug '25

01 Aug '25

Hi What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ? My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD Should I disable autotune and allocate more RAM? I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD Many thanks Steven

3 5

Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected on all newly added osd disks
by gagan tiwari 01 Aug '25

01 Aug '25

Hi Guys, I am setting up a new ceph cluster. After adding osd devices when I did run *ceph orch device ls* It shows * Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected for all osd devices.* All the OSD devices are new nvme disks. I tried to rm , zap , destroy disks and re-added them but again same messages for all disks. Please let me know how to fix this. Thanks, Gagan

3 5

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

restart dashboard ?
by lejeczek 01 Aug '25

01 Aug '25

Hi guys. A newbie here - go easy on me. I'm following docs on adding certificates to dashboard - from an external CA. I do 'ceph mgr module disable/enable' & 'ceph mgr fail' but those do not seem to get desired effect. How to get ceph take certs changes without disrupting ceph? many thanks, L.

2 1