September 2025 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

Debian Packages for Trixie
by Andrew 09 Feb '26

09 Feb '26

Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew

6 6

How important is the "default" data pool being replicated for CephFS
by Mikael Öhman 07 Nov '25

07 Nov '25

The recommendations for cephfs is to make a replicated default data pool, and adding any EC data pools using layouts: https://docs.ceph.com/en/latest/cephfs/createfs/ > If erasure-coded pools are planned for file system data, it is best to configure the default as a replicated pool to improve small-object write and read performance when updating backtraces. I have an cephfs that unfortunately wasn't set up like this: they just made an EC pool on the slow HDDs as the default, which sounds like the worst case scenario to me. I would like to add a NVME data pool to this ceph fs, but recommended gives me pause on if i should instead go through the hassle of creating a new cephfs and migrating all users. I've tried to run some mdtest with small 1k files to see if i could measure this difference, but speed is about the same in my relatively small tests so far. I'm also not sure what impact I should realistically expect here. I don't even know if creating files counts as "updating backtraces", so my testing might just be pointless. I guess my core question is; just how important is this suggestion to keep the default data pool on replicated NVME? Setup: 14 hosts x 42 HDD + 3 NVMEs for db/wal 2*2x25 GbitE bonds 12 hosts x 10 NVME. 2*2x100 GbitE bonds Old CephFS setup: - metadata: replicated NVME - data-pools: EC 10+2 on HDD (i plan to add a EC NVME pool here via layouts) New CephFS setup as recommended: - metadata: replicated NVME - data-pools: replicated NVME (default), EC 8+2 on HDD via layout, EC 8+2 on NVME via layout. Ceph 18.2.7 Best regards, Mikael

4 5

Help with Jaeger Tracing for Ceph RGW in Kubernetes (Rook-Ceph Deployment)
by Ashok Kumar M 31 Oct '25

31 Oct '25

Hi Ceph Community, I'm currently running a Ceph cluster inside a Kubernetes environment using Rook-Ceph, and I'm working on enabling distributed tracing for Ceph RGW. As per the https://docs.ceph.com/en/latest/jaegertracing/ , I used the following docker run command to deploy a Jaeger all-in-one instance for testing: docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6799:6799/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:latest \ --processor.jaeger-compact.server-host-port=6799 However, when using the latest Jaeger version, I get the following error: unknown flag: --processor.jaeger-compact.server-host-port After researching, I discovered that this flag is only available in older versions of Jaeger (e.g., v1.27) and is no longer supported in Jaeger v2+. I then tried using a specific older Jaeger version (jaegertracing/all-in-one:1.27) that supports the flag. The container runs fine and listens on port 6799 as expected, but I still don’t see any RGW traces appearing in the Jaeger UI. My Setup: Ceph cluster deployed via Rook in Kubernetes Ceph RGW is running as a pod inside the cluster Jaeger all-in-one is running externally (for testing) Jaeger port 6799/udp is exposed and forwarded Tracing is enabled in Ceph config Questions: Could the issue be related to the Ceph RGW pod not being able to send UDP traces to the Jaeger container? Is there a specific configuration needed inside the RGW pod or Rook-Ceph to make sure tracing data is being sent correctly? Should I consider using the default Jaeger port 6831/udp instead of 6799/udp, and configure Ceph accordingly? Are there any known compatibility issues between Ceph RGW tracing and Jaeger v1.x? Could you suggest some troubleshooting steps to help verify whether traces are actually being sent from RGW and received by Jaeger? Any advice or working examples from a similar Kubernetes-based setup would be greatly appreciated. Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 2

DriveGroup Spec question
by Robert Sander 17 Oct '25

17 Oct '25

Hi, we have OSD nodes currently consisting of two 605GB SSDs and six 18TB HDDs. The hosts have room for twelve HDDs. We created a drivegroup spec that looks like this: spec: block_db_size: 100GB data_devices: rotational: true size: '18TB:' db_devices: rotational: false size: '550GB:650GB' db_slots: 6 Initially this creates 6 OSDs with their RocksDB+WAL on the SSDs, 3 each which is nice for load balancing. But when we add another HDD it gets a 17.9TB data volume and a 100GB DB volume, both on the HDD: sdm 8:192 0 18T 0 disk ├─ceph--846e1a59--aff6--4ef8--9b71--de7241531677-osd--block--026e8cef--123d--47d9--9b30--211f94edf96c 252:16 0 17.9T 0 lvm └─ceph--846e1a59--aff6--4ef8--9b71--de7241531677-osd--db--88c47d0b--f5c6--4cec--8909--c5f8036ca459 252:17 0 100G 0 lvm I would have assumed that the remaining 305GB on the SSDs would be used. How do we achieve this? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 9

tls certs per manager - does it work?
by lejeczek 16 Oct '25

16 Oct '25

Hi guys. I thought I had a problem with restarting dashboard - following in docs on ssl certs. But ! I might have other issue - restarting dashboard does what is expected of - but, with these: -> $ ceph dashboard set-ssl-certificate-key podster2.mine.priv -i /root/podster2.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster2.mine.priv -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key podster1.mine.priv -i /root/podster1.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster1.mine.priv -i /root/podster1.mine.priv.crt If I go to podster2 I get podster1's cert, perhaps these do not do anything, If I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster2.mine.priv.key then podster1 has podster2's cert - naturally - but (weirdly?) if go to podster2 (by FQDN) then I get redirected to IP (as in URL) of when I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster1.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster1.mine.priv.key and I go to podster2 then, no redirection, "only" warning about cert being of podster1's, podster1 has a good cert! If I do (again) - having podster1 have "correct" cert, with no per-node/manager cert - per-node/mgr cert, as shown earlier, then.. again, podster1 has its correct cert, podster2 has podster1's cert (no ! redirection to IP though). How much of an "issue" this might be - given the fact that I deployed anew cluster twice (lab) - and this reproduces each time. Or, I'm missing some obvious, big picture - and it's simply bad luck, deploying cluster twice. ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) all thoughts are much appreciated. many thanks, L.

2 9

Logging recommendation on docker/cephadm pacific
by J-P Methot 06 Oct '25

06 Oct '25

Hi, We've noticed that on some of our older setups (Pacific here), we're getting what appears to be duplicate logs for our ceph docker containers deployed with cephadm. These logs are located at "/var/lib/docker/containers/*containerid*-json.log", are generated by Docker and don't rotate by default. We're wondering what would be the best practice here. Would it be best to setup log rotation for these files or is it safe to just stop Docker from logging at all? -- Mes salutations distinguées | Best regards Jean-Philippe MÉTHOT (Il, Lui | He/Him) Administrateur Infra sénior | Senior Infra administrator PlanetHoster AVIS: Ce courriel est confidentiel et peut être l'objet de secret professionnel. Tout usage ou diffusion de celui-ci par une personne autre que son destinataire est illégal. Si vous recevez ce courriel par erreur, veuillez nous en aviser sans délai et le supprimer de votre système. This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system.

2 1

help OSD deploying via yaml
by GLE, Vivien 03 Oct '25

03 Oct '25

Hi, For testing purpose we need to deploy : - 1 pool of 6 SSD OSD - 1 pool of 6 HDD OSD - 1 pool of 6 HDD OSD with 2 SSD for DB+WAL I tried to orch apply this yaml but it doesn't work as expected Osd part of the yaml -> service_type: osd service_id: osd_spec placement: hosts: - host1 - host2 - host3 - host4 - host5 - host6 data_devices: paths: - /dev/sda - /dev/sdc spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: osd_spec_wall placement: hosts: - host1 - host2 - host3 - host4 - host5 - host6 spec: data_devices: paths: - /dev/sdf db_devices: paths: - /dev/sde limit: 2 db_slots: 3 Only 1 db on /dev/sde from host1 has been created and this OSD showed up as half full at his creation: ceph osd df | grep 25 25 hdd 0.63669 1.00000 652 GiB 373 GiB 1.5 MiB 1 KiB 38 MiB 279 GiB 57.15 35.93 0 up ceph-volume lvm list ====== osd.25 ====== [block] /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block device /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block uuid vqJGf8-V5g0-S1cA-BAcN-Qm9D-9VTx-xsc8wk cephx lockbox secret cluster fsid id cluster name ceph crush device class db device /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 db uuid Rtj5KA-Qxjk-IFmY-3ffQ-gSAu-Snte-uku8HC encrypted 0 osd fsid 2f009760-fc2b-46d5-984d-e8200dfd9d9d osd id 25 osdspec affinity osd_spec_wall type block vdo 0 with tpm 0 devices /dev/sdf [db] /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 block device /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block uuid vqJGf8-V5g0-S1cA-BAcN-Qm9D-9VTx-xsc8wk cephx lockbox secret cluster fsid id cluster name ceph crush device class db device /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 db uuid Rtj5KA-Qxjk-IFmY-3ffQ-gSAu-Snte-uku8HC encrypted 0 osd fsid 2f009760-fc2b-46d5-984d-e8200dfd9d9d osd id 25 osdspec affinity osd_spec_wall type db vdo 0 with tpm 0 devices /dev/sde The other /dev/sde device did show up as data_device instead of db_device (example here on host2) : ceph-volume lvm list ====== osd.17 ====== [block] /dev/ceph-629f98b0-5ed4-4e75-81b9-e85ca76afb15/osd-block-5d43d683-1f7f-4dc1-935e-6a79745252f9 block device /dev/ceph-629f98b0-5ed4-4e75-81b9-e85ca76afb15/osd-block-5d43d683-1f7f-4dc1-935e-6a79745252f9 block uuid HQpp1l-x7IB-kA2W-6gWO-BGlM-VN2k-vYf43R cephx lockbox secret cluster fsid id cluster name ceph crush device class encrypted 0 osd fsid 5d43d683-1f7f-4dc1-935e-6a79745252f9 osd id 17 osdspec affinity osd_spec type block vdo 0 with tpm 0 devices /dev/sde Thx for your help Vivien

3 8

tentacle 20.1.1 RC1 QE validation status
by Yuri Weinstein 03 Oct '25

03 Oct '25

Details of this release are summarized here: https://tracker.ceph.com/issues/73123#note-1 Release Notes - TBD LRC upgrade - TBD Dev Leads, please review the list of suites for completeness, as this is a new release. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv upgrade/squid-x: Laura upgrade/reef-x: Laura upgrade/telemetry-upgrade: PASSED upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: PASSED powercycle: PASSED TIA

8 13

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

SQUID ganesha failures
by Patrick Begou 02 Oct '25

02 Oct '25

Hi all, I have a Ceph Cluster using squid 19.2.2. I'm using CephFS storage with kernel mounts on clients and it works fine but for some older computers (still CentOS8 and one CentOS7 because it is running a commercial application) I need to use NFS so I've deployed Ganesha. But a few minutes after the first NFS client mount the volume (Just testing from the first client) and uses it, the Ceph NFS service falls in error on one node (not always the same node) and I have to reboot it. # ceph orch ps --daemon-type nfs NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID nfs.whitaker-nfs.0.2.whitaker04-ceph.itzyqf whitaker04-ceph *:12049 running (49m) 5m ago 3h 26.3M - 5.9 4892a7ef541b 42fd4ab7bba9 nfs.whitaker-nfs.1.0.whitaker05-ceph.vocxmr whitaker05-ceph *:12049 running (63m) 5m ago 23h 85.7M - 5.9 4892a7ef541b ab1c06be2ce7 nfs.whitaker-nfs.2.0.whitaker01-ceph.rhwuth whitaker01-ceph *:12049 running (33m) 5m ago 23h 24.6M - 5.9 4892a7ef541b eb93963f05cd nfs.whitaker-nfs.3.0.whitaker03-ceph.wqfnnq whitaker03-ceph *:12049 error 5m ago 23h - - <unknown> <unknown> <unknown> nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn whitaker02-ceph *:12049 running (49m) 4m ago 3h 28.3M - 5.9 4892a7ef541b 5cb71f9d8b89 and the client freeze on the mount point. Any idea about this or how to track the problem ? Thanks Patrick ---------------------------------------------- Details of my NFS CEPH cluster, building commands and status: ---------------------------------------------- # *ceph nfs cluster create whitaker-nfs \ whitaker01-ceph,whitaker02-ceph,whitaker03-ceph,whitaker04-ceph,whitaker05-ceph \ --ingress --virtual_ip 172.30.1.102/24 --ingress-mode haproxy-protocol* *# ceph nfs cluster info whitaker-nfs* { "whitaker-nfs": { "backend": [ { "hostname": "whitaker01-ceph", "ip": "172.30.1.97", "port": 12049 }, { "hostname": "whitaker02-ceph", "ip": "172.30.1.98", "port": 12049 }, { "hostname": "whitaker03-ceph", "ip": "172.30.1.99", "port": 12049 }, { "hostname": "whitaker04-ceph", "ip": "172.30.1.100", "port": 12049 }, { "hostname": "whitaker05-ceph", "ip": "172.30.1.101", "port": 12049 } ], "monitor_port": 9049, "port": 2049, "virtual_ip": "172.30.1.102" } } *# ceph nfs export create cephfs \ --cluster-id whitaker-nfs \ --pseudo-path /data \ --fsname data \ --client_addr 172.30.1.0/24* *# ceph nfs export ls whitaker-nfs --detailed* [ { "access_type": "none", "clients": [ { "access_type": "rw", "addresses": [ "172.30.1.0/24" ], "squash": "none" } ], "cluster_id": "whitaker-nfs", "export_id": 1, "fsal": { "cmount_path": "/", "fs_name": "data", "name": "CEPH", "user_id": "nfs.whitaker-nfs.data.79da3503" }, "path": "/", "protocols": [ 4 ], "pseudo": "/data", "security_label": true, "squash": "none", "transports": [ "TCP" ] } ]

3 6

Upgrading from Reef to Squid, "init missing pg_pool_t for deleted pool for pg please downgrade to luminous and allow pg deletion to complete before upgrading
by Daniel Williams 30 Sep '25

30 Sep '25

Some background pool 57 is a new rbd pool (12MiB used) that I was just experimenting with (performance of striped hdd rbd devices), I don't think I deleted it but can't say for sure (it appears in ceph df) since it doesn't matter. This pool was created on reef, a full deep scrub has been done several times over since moving to reef (March 2024), likely no deep scrub has been done since moving to squid since I've had lots of troubles... This error however has broken a 150TiB machine and worse I don't know that a restart won't break others.. After a host reboot I've lost half the OSDs on that host, they all refuse to start with: -725> 2025-09-25T18:02:37.157+0000 7f93d0aab8c0 -1 Falling back to public interface -2> 2025-09-25T18:02:40.033+0000 7f93d0aab8c0 -1 osd.21 2098994 init missing pg_pool_t for deleted pool 57 for pg 57.3s7; please downgrade to luminous and allow pg deletion to complete before upgrading -1> 2025-09-25T18:02:40.037+0000 7f93d0aab8c0 -1 ./src/osd/OSD.cc: In function 'int OSD::init()' thread 7f93d0aab8c0 time 2025-09-25T18:02:40.040491+0000 ./src/osd/OSD.cc: 3867: ceph_abort_msg("abort() called") ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb7) [0x560b78a7056a] 2: /usr/bin/ceph-osd(+0x385bcb) [0x560b789f0bcb] 3: main() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f93d165dd90] 5: __libc_start_main() 6: _start() 0> 2025-09-25T18:02:40.037+0000 7f93d0aab8c0 -1 *** Caught signal (Aborted) ** in thread 7f93d0aab8c0 thread_name:ceph-osd ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f93d1676520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x16a) [0x560b78a7061d] 6: /usr/bin/ceph-osd(+0x385bcb) [0x560b789f0bcb] 7: main() 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f93d165dd90] 9: __libc_start_main() 10: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -725> 2025-09-25T18:02:37.157+0000 7f93d0aab8c0 -1 Falling back to public interface -2> 2025-09-25T18:02:40.033+0000 7f93d0aab8c0 -1 osd.21 2098994 init missing pg_pool_t for deleted pool 57 for pg 57.3s7; please downgrade to luminous and allow pg deletion to complete before upgrading -1> 2025-09-25T18:02:40.037+0000 7f93d0aab8c0 -1 ./src/osd/OSD.cc: In function 'int OSD::init()' thread 7f93d0aab8c0 time 2025-09-25T18:02:40.040491+0000 ./src/osd/OSD.cc: 3867: ceph_abort_msg("abort() called") ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xb7) [0x560b78a7056a] 2: /usr/bin/ceph-osd(+0x385bcb) [0x560b789f0bcb] 3: main() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f93d165dd90] 5: __libc_start_main() 6: _start() 0> 2025-09-25T18:02:40.037+0000 7f93d0aab8c0 -1 *** Caught signal (Aborted) ** in thread 7f93d0aab8c0 thread_name:ceph-osd ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f93d1676520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x16a) [0x560b78a7061d] 6: /usr/bin/ceph-osd(+0x385bcb) [0x560b789f0bcb] 7: main() 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f93d165dd90] 9: __libc_start_main() 10: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aborted Will deleting the PG help? Is there any way I can recover these OSDs? Will moving back to reef help? Daniel

2 3

Ceph GWCLI issue
by Kardos László 30 Sep '25

30 Sep '25

Hello, We have encountered the following issue in our production environment: A new RBD Image was created within an existing pool, and its status is reported as "unknown" in GWCLI. Based on our tests, this does not appear to cause operational issues, but we would like to investigate the root cause. No relevant information regarding this issue was found in the logs. GWCLI output: o- / .......................................................................... ............................................... [...] o- cluster .......................................................................... ............................... [Clusters: 1] | o- ceph .......................................................................... .................................. [HEALTH_OK] | o- pools .......................................................................... ............................... [Pools: 11] | | o- .mgr ................................................................ [(x3), Commit: 0.00Y/15591725M (0%), Used: 194124K] | | o- .nfs ................................................................. [(x3), Commit: 0.00Y/15591725M (0%), Used: 16924b] | | o- xxxx-test ............................................................. [(2+1), Commit: 0.00Y/23727198M (0%), Used: 0.00Y] | | o- xxxxx-erasure-0 ............................................ [(2+1), Commit: 0.00Y/23727198M (0%), Used: 61519257668K] | | o- xxxxxx-repl ...................................................... [(x3), Commit: 0.00Y/15591725M (0%), Used: 130084b] | | o- cephfs.cephfs-test.data ............................................ [(x3), Commit: 0.00Y/15591725M (0%), Used: 9090444K] | | o- cephfs.cephfs-test.meta .......................................... [(x3), Commit: 0.00Y/15591725M (0%), Used: 516415713b] | | o- xxxxx-data ..................................................... [(3+1), Commit: 0.00Y/9604386M (0%), Used: 7547753556K] | | o- xxxxx-rpl .......................................................... [(x3), Commit: 12.0T/4268616M (294%), Used: 85265b] | | o- xxxxx-data ................................................... [(3+1), Commit: 0.00Y/5011626M (0%), Used: 10955179612K] | | o- replicated_xxxx ............................................... [(x3), Commit: 25.0T/2280846592K (1176%), Used: 46912b] | o- topology .......................................................................... ..................... [OSDs: 42,MONs: 5] o- disks .......................................................................... ............................. [37.0T, Disks: 3] | o- xxxx-rpl .......................................................................... ................... [xxxx-rpl (12.0T)] | | o- xxxxx_lun0 ........................................................................ [xxxx-rpl/xxxxx_lun0 (Online, 12.0T)] | o- replicated_xxxx .......................................................................... ..... [replicated_xxxx (25.0T)] | o- xxxx_lun0 ............................................................... [replicated_xxxx/xxxx_lun0 (Online, 12.0T)] | o- xxxx_lun_new ........................................................ [replicated_xxxx/xxxx_lun_new (Unknown, 13.0T)] The image (xxxx_lun_new) is provisioned to multiple ESXi hosts, mounted, and formatted with VMFS6. The datastore is writable and readable by the hosts. There is a change in the block size of the RBD Image: the older RBD Images use a 4 MiB block size, while the new RBD Image uses a 512 KiB block size. RBD Image Parameters: For replicated_xxxx / xxxx_lun0 (Online status in GWCLI): rbd image 'xxxx_lun0': size 12 TiB in 3145728 objects order 22 (4 MiB objects) snapshot_count: 0 id: 5c1b5ecfdfa46 data_pool: xxxx0-data block_name_prefix: rbd_data.14.5c1b5ecfdfa46 format: 2 features: exclusive-lock, data-pool op_features: flags: create_timestamp: Tue Jul 8 13:02:11 2025 access_timestamp: Thu Sep 25 13:49:47 2025 modify_timestamp: Thu Sep 25 13:50:05 2025 For replicated_xxxx / xxxx_lun_new (Unknown status in GWCLI): rbd image 'xxxx_lun_new': size 13 TiB in 27262976 objects order 19 (512 KiB objects) snapshot_count: 0 id: 1945d9cf9f41ab data_pool: xxxx0-data block_name_prefix: rbd_data.14.1945d9cf9f41ab format: 2 features: exclusive-lock, data-pool op_features: flags: create_timestamp: Wed Sep 24 11:21:21 2025 access_timestamp: Thu Sep 25 13:50:42 2025 modify_timestamp: Thu Sep 25 13:49:48 2025 Pool Parameters: pool 14 'replicated_xxxx' replicated size 3 min_size 2 crush_rule 7 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 30743 flags hashpspool stripe_width 0 application rbd,rgw Ceph version: ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Question: What could be causing the RBD Image (xxxx_lun_new) to appear in an "unknown" state in GWCLI?

6 7

ceph 19.2.3 - how to identify the OSD disk
by Steven Vacaroaia 29 Sep '25

29 Sep '25

Hi, There must be something basic that I am missing but I have an "osd down" issue and cannot pinpoint which disk need to be replaced Steps I took 1. "Identify the problem" cepg osd stat 185 osds: 184 up (since 3d), 184 in (since 3d); epoch: e47876; 941 remapped pgs 2." identify the OSD" ceph osd tree | grep down 51 ssd_class 6.98630 osd.51 down 0 1.00000 osd.51 ceph-host-3 stopped 3m ago 9w - 26.1G <unknown> <unknown> <unknown> 3. "check the block" ls -al /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/osd.51 block -> /dev/ceph-419569a5-9a68-4688-8cf4-17ee450a8659/osd-block-75fe30c3-8cfa-49c7-80a1-6d79c3e4b4e6 4. "find the device" ls -al /dev/ceph-419569a5-9a68-4688-8cf4-17ee450a8659/osd-block-75fe30c3-8cfa-49c7-80a1-6d79c3e4b4e6 /dev/dm-7 5. "what type " blkid /dev/dm-7 /dev/dm-7: TYPE="ceph_bluestore" This is weird since osd 51 was type "ssd_class" so just a regular OSD deployed on SSD drive 5. which disk has dm-7 lsblk -o NAME,SERIAL | grep ceph-419569a5-9a68-4688-8cf4-17ee450a8659/osd-block-75fe30c3-8cfa-49c7-80a1-6d79c3e4b4e6 no results ?? blkid /dev/dm-7 /dev/dm-7: TYPE="ceph_bluestore"

2 2

Ceph MGR balancer violates EC CRUSH rules
by Denis Polom 29 Sep '25

29 Sep '25

Hi folks, on our Ceph v19.2.1 distributed across 4 failure domains we have a EC pool using following crush rule: { "rule_id": 4, "rule_name": "ec53", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_indep", "num": 4, "type": "tor" }, { "op": "choose_indep", "num": 2, "type": "osd" }, { "op": "emit" } ] } and following EC profile crush-device-class= crush-failure-domain=tor crush-root=default jerasure-per-chunk-alignment=false k=5 m=3 plugin=jerasure technique=reed_sol_van w=8 here is our crush tree: ID CLASS WEIGHT TYPE NAME -40 15.71931 root metadata -41 3.49318 tor s1-bos-e-10-ng.metadata -42 1.74660 host osd1-metadata 98 ssd 1.74660 osd.98 -49 1.74660 host osd2-metadata 99 ssd 1.74660 osd.99 -124 3.49318 tor s1-bos-f-7-ng.metadata -52 1.74660 host osd3-metadata 103 ssd 1.74660 osd.103 -76 1.74660 host osd9-metadata 106 ssd 1.74660 osd.106 -56 5.23978 tor s1-bos-f-8-ng.metadata -55 1.74660 host osd4-metadata 85 ssd 1.74660 osd.85 -61 1.74660 host osd5-metadata 93 ssd 1.74660 osd.93 -64 1.74660 host osd6-metadata 97 ssd 1.74660 osd.97 -67 3.49318 tor s1-bos-f-9-ng.metadata -70 1.74660 host osd7-metadata 104 ssd 1.74660 osd.104 -73 1.74660 host osd8-metadata 105 ssd 1.74660 osd.105 -1 2733.78516 root default -21 745.57776 tor s1-bos-e-10-ng -7 80.40544 host osd1 0 hdd 7.30959 osd.0 10 hdd 7.30959 osd.10 26 hdd 7.30959 osd.26 28 hdd 7.30959 osd.28 44 hdd 7.30959 osd.44 53 hdd 7.30959 osd.53 62 hdd 7.30959 osd.62 70 hdd 7.30959 osd.70 79 hdd 7.30959 osd.79 89 hdd 7.30959 osd.89 107 hdd 7.30959 osd.107 -88 116.95337 host osd12 128 hdd 7.30959 osd.128 139 hdd 7.30959 osd.139 142 hdd 7.30959 osd.142 146 hdd 7.30959 osd.146 150 hdd 7.30959 osd.150 152 hdd 7.30959 osd.152 153 hdd 7.30959 osd.153 154 hdd 7.30959 osd.154 191 hdd 7.30959 osd.191 192 hdd 7.30959 osd.192 193 hdd 7.30959 osd.193 194 hdd 7.30959 osd.194 195 hdd 7.30959 osd.195 196 hdd 7.30959 osd.196 199 hdd 7.30959 osd.199 200 hdd 7.30959 osd.200 -94 116.95337 host osd15 156 hdd 7.30959 osd.156 157 hdd 7.30959 osd.157 158 hdd 7.30959 osd.158 159 hdd 7.30959 osd.159 160 hdd 7.30959 osd.160 161 hdd 7.30959 osd.161 162 hdd 7.30959 osd.162 163 hdd 7.30959 osd.163 164 hdd 7.30959 osd.164 165 hdd 7.30959 osd.165 166 hdd 7.30959 osd.166 167 hdd 7.30959 osd.167 171 hdd 7.30959 osd.171 172 hdd 7.30959 osd.172 177 hdd 7.30959 osd.177 183 hdd 7.30959 osd.183 -17 80.40544 host osd2 4 hdd 7.30959 osd.4 15 hdd 7.30959 osd.15 19 hdd 7.30959 osd.19 33 hdd 7.30959 osd.33 38 hdd 7.30959 osd.38 52 hdd 7.30959 osd.52 58 hdd 7.30959 osd.58 69 hdd 7.30959 osd.69 73 hdd 7.30959 osd.73 81 hdd 7.30959 osd.81 95 hdd 7.30959 osd.95 -115 87.71503 host osd22 275 hdd 7.30959 osd.275 276 hdd 7.30959 osd.276 277 hdd 7.30959 osd.277 278 hdd 7.30959 osd.278 279 hdd 7.30959 osd.279 280 hdd 7.30959 osd.280 281 hdd 7.30959 osd.281 282 hdd 7.30959 osd.282 283 hdd 7.30959 osd.283 284 hdd 7.30959 osd.284 285 hdd 7.30959 osd.285 286 hdd 7.30959 osd.286 -118 87.71503 host osd23 287 hdd 7.30959 osd.287 288 hdd 7.30959 osd.288 289 hdd 7.30959 osd.289 290 hdd 7.30959 osd.290 291 hdd 7.30959 osd.291 292 hdd 7.30959 osd.292 293 hdd 7.30959 osd.293 294 hdd 7.30959 osd.294 295 hdd 7.30959 osd.295 296 hdd 7.30959 osd.296 297 hdd 7.30959 osd.297 298 hdd 7.30959 osd.298 -130 87.71503 host osd25 311 hdd 7.30959 osd.311 312 hdd 7.30959 osd.312 313 hdd 7.30959 osd.313 314 hdd 7.30959 osd.314 315 hdd 7.30959 osd.315 316 hdd 7.30959 osd.316 317 hdd 7.30959 osd.317 318 hdd 7.30959 osd.318 319 hdd 7.30959 osd.319 320 hdd 7.30959 osd.320 321 hdd 7.30959 osd.321 322 hdd 7.30959 osd.322 -134 87.71503 host osd29 359 hdd 7.30959 osd.359 360 hdd 7.30959 osd.360 361 hdd 7.30959 osd.361 362 hdd 7.30959 osd.362 363 hdd 7.30959 osd.363 364 hdd 7.30959 osd.364 365 hdd 7.30959 osd.365 366 hdd 7.30959 osd.366 367 hdd 7.30959 osd.367 368 hdd 7.30959 osd.368 369 hdd 7.30959 osd.369 370 hdd 7.30959 osd.370 -127 774.81604 tor s1-bos-f-7-ng -79 116.95337 host osd10 108 hdd 7.30959 osd.108 109 hdd 7.30959 osd.109 110 hdd 7.30959 osd.110 111 hdd 7.30959 osd.111 112 hdd 7.30959 osd.112 113 hdd 7.30959 osd.113 114 hdd 7.30959 osd.114 115 hdd 7.30959 osd.115 116 hdd 7.30959 osd.116 117 hdd 7.30959 osd.117 118 hdd 7.30959 osd.118 119 hdd 7.30959 osd.119 120 hdd 7.30959 osd.120 121 hdd 7.30959 osd.121 122 hdd 7.30959 osd.122 123 hdd 7.30959 osd.123 -85 116.95337 host osd11 125 hdd 7.30959 osd.125 126 hdd 7.30959 osd.126 127 hdd 7.30959 osd.127 129 hdd 7.30959 osd.129 138 hdd 7.30959 osd.138 144 hdd 7.30959 osd.144 149 hdd 7.30959 osd.149 151 hdd 7.30959 osd.151 188 hdd 7.30959 osd.188 189 hdd 7.30959 osd.189 190 hdd 7.30959 osd.190 197 hdd 7.30959 osd.197 198 hdd 7.30959 osd.198 201 hdd 7.30959 osd.201 202 hdd 7.30959 osd.202 203 hdd 7.30959 osd.203 -91 116.95337 host osd13 130 hdd 7.30959 osd.130 134 hdd 7.30959 osd.134 135 hdd 7.30959 osd.135 140 hdd 7.30959 osd.140 143 hdd 7.30959 osd.143 147 hdd 7.30959 osd.147 148 hdd 7.30959 osd.148 155 hdd 7.30959 osd.155 168 hdd 7.30959 osd.168 169 hdd 7.30959 osd.169 170 hdd 7.30959 osd.170 176 hdd 7.30959 osd.176 178 hdd 7.30959 osd.178 180 hdd 7.30959 osd.180 182 hdd 7.30959 osd.182 184 hdd 7.30959 osd.184 -121 87.71503 host osd24 299 hdd 7.30959 osd.299 300 hdd 7.30959 osd.300 301 hdd 7.30959 osd.301 302 hdd 7.30959 osd.302 303 hdd 7.30959 osd.303 304 hdd 7.30959 osd.304 305 hdd 7.30959 osd.305 306 hdd 7.30959 osd.306 307 hdd 7.30959 osd.307 308 hdd 7.30959 osd.308 309 hdd 7.30959 osd.309 310 hdd 7.30959 osd.310 -131 87.71503 host osd26 323 hdd 7.30959 osd.323 324 hdd 7.30959 osd.324 325 hdd 7.30959 osd.325 326 hdd 7.30959 osd.326 327 hdd 7.30959 osd.327 328 hdd 7.30959 osd.328 329 hdd 7.30959 osd.329 330 hdd 7.30959 osd.330 331 hdd 7.30959 osd.331 332 hdd 7.30959 osd.332 333 hdd 7.30959 osd.333 334 hdd 7.30959 osd.334 -5 80.40544 host osd3 2 hdd 7.30959 osd.2 12 hdd 7.30959 osd.12 21 hdd 7.30959 osd.21 34 hdd 7.30959 osd.34 42 hdd 7.30959 osd.42 50 hdd 7.30959 osd.50 54 hdd 7.30959 osd.54 63 hdd 7.30959 osd.63 100 hdd 7.30959 osd.100 101 hdd 7.30959 osd.101 102 hdd 7.30959 osd.102 -135 87.71503 host osd30 371 hdd 7.30959 osd.371 372 hdd 7.30959 osd.372 373 hdd 7.30959 osd.373 374 hdd 7.30959 osd.374 375 hdd 7.30959 osd.375 376 hdd 7.30959 osd.376 377 hdd 7.30959 osd.377 378 hdd 7.30959 osd.378 379 hdd 7.30959 osd.379 380 hdd 7.30959 osd.380 381 hdd 7.30959 osd.381 382 hdd 7.30959 osd.382 -13 80.40544 host osd9 8 hdd 7.30959 osd.8 13 hdd 7.30959 osd.13 24 hdd 7.30959 osd.24 32 hdd 7.30959 osd.32 39 hdd 7.30959 osd.39 48 hdd 7.30959 osd.48 56 hdd 7.30959 osd.56 64 hdd 7.30959 osd.64 75 hdd 7.30959 osd.75 84 hdd 7.30959 osd.84 94 hdd 7.30959 osd.94 -25 584.76685 tor s1-bos-f-8-ng -97 87.71503 host osd16 216 hdd 7.30959 osd.216 217 hdd 7.30959 osd.217 218 hdd 7.30959 osd.218 219 hdd 7.30959 osd.219 220 hdd 7.30959 osd.220 221 hdd 7.30959 osd.221 222 hdd 7.30959 osd.222 223 hdd 7.30959 osd.223 224 hdd 7.30959 osd.224 225 hdd 7.30959 osd.225 226 hdd 7.30959 osd.226 227 hdd 7.30959 osd.227 -100 87.71503 host osd17 204 hdd 7.30959 osd.204 205 hdd 7.30959 osd.205 206 hdd 7.30959 osd.206 207 hdd 7.30959 osd.207 208 hdd 7.30959 osd.208 209 hdd 7.30959 osd.209 210 hdd 7.30959 osd.210 211 hdd 7.30959 osd.211 212 hdd 7.30959 osd.212 213 hdd 7.30959 osd.213 214 hdd 7.30959 osd.214 215 hdd 7.30959 osd.215 -103 80.40544 host osd18 228 hdd 7.30959 osd.228 229 hdd 7.30959 osd.229 230 hdd 7.30959 osd.230 231 hdd 7.30959 osd.231 232 hdd 7.30959 osd.232 233 hdd 7.30959 osd.233 234 hdd 7.30959 osd.234 235 hdd 7.30959 osd.235 236 hdd 7.30959 osd.236 237 hdd 7.30959 osd.237 238 hdd 7.30959 osd.238 -132 87.71503 host osd27 335 hdd 7.30959 osd.335 336 hdd 7.30959 osd.336 337 hdd 7.30959 osd.337 338 hdd 7.30959 osd.338 339 hdd 7.30959 osd.339 340 hdd 7.30959 osd.340 341 hdd 7.30959 osd.341 342 hdd 7.30959 osd.342 343 hdd 7.30959 osd.343 344 hdd 7.30959 osd.344 345 hdd 7.30959 osd.345 346 hdd 7.30959 osd.346 -11 80.40544 host osd4 3 hdd 7.30959 osd.3 16 hdd 7.30959 osd.16 18 hdd 7.30959 osd.18 31 hdd 7.30959 osd.31 36 hdd 7.30959 osd.36 49 hdd 7.30959 osd.49 59 hdd 7.30959 osd.59 68 hdd 7.30959 osd.68 72 hdd 7.30959 osd.72 77 hdd 7.30959 osd.77 82 hdd 7.30959 osd.82 -19 80.40544 host osd5 6 hdd 7.30959 osd.6 11 hdd 7.30959 osd.11 25 hdd 7.30959 osd.25 27 hdd 7.30959 osd.27 41 hdd 7.30959 osd.41 46 hdd 7.30959 osd.46 60 hdd 7.30959 osd.60 66 hdd 7.30959 osd.66 80 hdd 7.30959 osd.80 88 hdd 7.30959 osd.88 91 hdd 7.30959 osd.91 -3 80.40544 host osd6 7 hdd 7.30959 osd.7 14 hdd 7.30959 osd.14 20 hdd 7.30959 osd.20 30 hdd 7.30959 osd.30 37 hdd 7.30959 osd.37 47 hdd 7.30959 osd.47 57 hdd 7.30959 osd.57 67 hdd 7.30959 osd.67 74 hdd 7.30959 osd.74 83 hdd 7.30959 osd.83 96 hdd 7.30959 osd.96 -23 628.62439 tor s1-bos-f-9-ng -82 116.95337 host osd14 124 hdd 7.30959 osd.124 131 hdd 7.30959 osd.131 132 hdd 7.30959 osd.132 133 hdd 7.30959 osd.133 136 hdd 7.30959 osd.136 137 hdd 7.30959 osd.137 141 hdd 7.30959 osd.141 145 hdd 7.30959 osd.145 173 hdd 7.30959 osd.173 174 hdd 7.30959 osd.174 175 hdd 7.30959 osd.175 179 hdd 7.30959 osd.179 181 hdd 7.30959 osd.181 185 hdd 7.30959 osd.185 186 hdd 7.30959 osd.186 187 hdd 7.30959 osd.187 -106 87.71503 host osd19 239 hdd 7.30959 osd.239 240 hdd 7.30959 osd.240 241 hdd 7.30959 osd.241 242 hdd 7.30959 osd.242 243 hdd 7.30959 osd.243 244 hdd 7.30959 osd.244 245 hdd 7.30959 osd.245 246 hdd 7.30959 osd.246 247 hdd 7.30959 osd.247 248 hdd 7.30959 osd.248 249 hdd 7.30959 osd.249 250 hdd 7.30959 osd.250 -109 87.71503 host osd20 251 hdd 7.30959 osd.251 252 hdd 7.30959 osd.252 253 hdd 7.30959 osd.253 254 hdd 7.30959 osd.254 255 hdd 7.30959 osd.255 256 hdd 7.30959 osd.256 257 hdd 7.30959 osd.257 258 hdd 7.30959 osd.258 259 hdd 7.30959 osd.259 260 hdd 7.30959 osd.260 261 hdd 7.30959 osd.261 262 hdd 7.30959 osd.262 -112 87.71503 host osd21 263 hdd 7.30959 osd.263 264 hdd 7.30959 osd.264 265 hdd 7.30959 osd.265 266 hdd 7.30959 osd.266 267 hdd 7.30959 osd.267 268 hdd 7.30959 osd.268 269 hdd 7.30959 osd.269 270 hdd 7.30959 osd.270 271 hdd 7.30959 osd.271 272 hdd 7.30959 osd.272 273 hdd 7.30959 osd.273 274 hdd 7.30959 osd.274 -133 87.71503 host osd28 347 hdd 7.30959 osd.347 348 hdd 7.30959 osd.348 349 hdd 7.30959 osd.349 350 hdd 7.30959 osd.350 351 hdd 7.30959 osd.351 352 hdd 7.30959 osd.352 353 hdd 7.30959 osd.353 354 hdd 7.30959 osd.354 355 hdd 7.30959 osd.355 356 hdd 7.30959 osd.356 357 hdd 7.30959 osd.357 358 hdd 7.30959 osd.358 -15 80.40544 host osd7 1 hdd 7.30959 osd.1 9 hdd 7.30959 osd.9 22 hdd 7.30959 osd.22 35 hdd 7.30959 osd.35 43 hdd 7.30959 osd.43 51 hdd 7.30959 osd.51 55 hdd 7.30959 osd.55 65 hdd 7.30959 osd.65 76 hdd 7.30959 osd.76 86 hdd 7.30959 osd.86 92 hdd 7.30959 osd.92 -9 80.40544 host osd8 5 hdd 7.30959 osd.5 17 hdd 7.30959 osd.17 23 hdd 7.30959 osd.23 29 hdd 7.30959 osd.29 40 hdd 7.30959 osd.40 45 hdd 7.30959 osd.45 61 hdd 7.30959 osd.61 71 hdd 7.30959 osd.71 78 hdd 7.30959 osd.78 87 hdd 7.30959 osd.87 90 hdd 7.30959 osd.90 Ceph MGR balancer is running in upmap mode, clients are "luminous". During one FD s1-bos-f-9-ng outage there were many PGs in inactive state, for example: pg 11.558 is stuck inactive for 102s, current state undersized+degraded+peered, last acting [176,169,NONE,NONE,199,NONE,227,221] this clearly shows that PG was distributed unevenly through FDs as according crush rule it should be placed on 2 OSDs per FD. # for i in 176 169 199 227 221;do echo -n "$i: "; ceph osd find $i | jq -r .crush_location.tor;done 176: s1-bos-f-7-ng 169: s1-bos-f-7-ng 199: s1-bos-e-10-ng 227: s1-bos-f-8-ng 221: s1-bos-f-8-ng # ceph osd tree | grep tor -41 3.49318 tor s1-bos-e-10-ng.metadata -124 3.49318 tor s1-bos-f-7-ng.metadata -56 5.23978 tor s1-bos-f-8-ng.metadata -67 3.49318 tor s1-bos-f-9-ng.metadata -21 745.57776 tor s1-bos-e-10-ng -127 774.81604 tor s1-bos-f-7-ng -25 584.76685 tor s1-bos-f-8-ng -23 628.62439 tor s1-bos-f-9-ng after our FD came up I did run pg-upmap-items manually to make placement correct. But after enabling MGR balancer there were attempts to move PG placement again to different OSD under different FD that breaks crush rule. Now I know this cluster FD weights are very different in CRUSH map. So I've tried to use osdmaptool to simulate balancer. Our balancer settings were: mgr advanced mgr/balancer/mode upmap mgr advanced mgr/balancer/upmap_max_deviation 1 mgr advanced mgr/balancer/upmap_max_optimizations 100 So I tried to adjust max deviation to level, that balancer would not try to move PGs to OSDs in different FD that breaks crush rule. I ended up on value 50, and honestly don't know if at this value balancer will try to do anything anytime. From my perspective MGR balancer is ignoring CRUSH rule - at least on EC pools. I observed this behaviour on our another Ceph cluster and also only on EC pool even if FD weights there were almost same. I even tried TheJJ Ceph-balancer, and it breaks crush rule also. Any ideas or similar experiences? Thank you

2 4

Seemingly stuck scrubbing and snaptrims
by Gustavo Garcia Rondina 27 Sep '25

27 Sep '25

Hi Ceph community, We have a Ceph cluster with 6 OSD nodes and 168 OSDs (28 per node each with a 18 TB data disk). We are running quincy 17.2.6. The cluster was not properly maintained for a while and it was in not the greatest shape. After some work, we got here: [ceph: root@ceph-admin1 /]# ceph -s cluster: id: 26315dca-383a-11ee-9d49-xxxxxxxxxxxx health: HEALTH_WARN 257 pgs not deep-scrubbed in time 259 pgs not scrubbed in time services: mon: 5 daemons, quorum ceph-admin1,ceph-admin2,ceph-osd1,ceph-osd2,ceph-osd3 (age 6M) mgr: ceph-admin2.sipadf(active, since 1y), standbys: ceph-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 168 osds: 168 up (since 6w), 168 in (since 6w) data: volumes: 2/2 healthy pools: 9 pools, 2273 pgs objects: 699.66M objects, 1.5 PiB usage: 2.2 PiB used, 529 TiB / 2.7 PiB avail pgs: 1986 active+clean 125 active+clean+snaptrim_wait 110 active+clean+scrubbing+deep 47 active+clean+snaptrim 5 active+clean+scrubbing The snaptrim_wait and snaptrim numbers are stuck on these values for weeks, if not months. We also had almost all PGs not (deep-)scrubbed in time, but after tweaking a few parameters the numbers got down to what you see above. However, now they seem to not be lowering, or doing so very slowly (a couple every 2-3 days). Dumping all PGs and filtering for the SCRUB_SCHEDULING column, I can see that some PGs have been scrubbing for a long time: [root@ceph-admin1 ~]# ceph pg dump | grep -e 'scrubbing for' -e SCHED | awk '{print $1,$(NF-2)}' | sort -n -k2 | tail 5.624 4788773s 5.5f2 4790148s 5.4b 4791924s 5.686 4795526s 5.538 4796221s 5.5c8 4796573s 5.722 4796937s 5.483 4797551s 5.81 4799856s 5.233 10052851s While recent ones look reasonable, so new scrubs are starting: [root@ceph-admin1 ~]# ceph pg dump | grep -e 'scrubbing for' -e SCHED | awk '{print $1,$(NF-2)}' | sort -n -k2 | head PG_STAT SCRUB_SCHEDULING 5.14b 53s 5.382 71s 5.ff 187s 6.9 265s 5.1c3 354s 5.70a 364s 5.596 367s 5.1fc 897s 5.3c8 1420s Looking closer to PG 5.233, it has a long snaptrimq length, but it doesn't seem to be blocked: [root@ceph-admin1 ~]# ceph pg 5.233 query | grep -E 'snaptrimq_len|blocked_by|waiting_on' "snaptrimq_len": 2682, "blocked_by": [], However it does seem to be waiting on someone: [root@ceph-admin1 ~]# ceph pg 5.233 query | grep -A5 'waiting_on_whom' "waiting_on_whom": [ "164(4)" ], "schedule": "scrubbing" The primary OSD for PG 5.233 is osd.136 on node ceph-osd4, and its logs do not show anything remarkable. The HDD for osd.136 is in good shape, no SMART errors, no I/O errors anywhere on ceph-osd4. Is it the scrubbing that's blocking the snaptrim, or the other way? Also, looking at historic ops: [root@ceph-admin1 ~]# ceph tell osd.136 dump_historic_ops_by_duration { "size": 20, "duration": 600, ... "ops": [ { "description": "rep_scrubmap(6.9 e14586 from shard 151)", "initiated_at": "2025-09-25T09:57:46.150913-0500", "age": 356.3390488, "duration": 0.12764220300000001, "type_data": { "flag_point": "started", "events": [ ... { "event": "header_read", "time": "2025-09-25T09:57:46.150912-0500", "duration": 4294967295.9999995 }, This last duration above looks odd as 4294967295.9999995 is 2^32. Not sure what it means, but thought it was strange. Any clues on any of this? Thank you, Gustavo

2 1

Ceph Community Round-up: New Member, Blogs, Survey & Cephalocon!
by Anthony Middleton 26 Sep '25

26 Sep '25

Hello Ceph Friends! I hope this week has gone well for you. Before logging off for the weekend, I wanted to share a few updates and announcements. Here's what's happening in our community this week: *Welcome, Western Digital!*The Ceph Foundation is excited to welcome Western Digital as our newest member. More details soon! Meet our members - https://ceph.io/en/foundation/members/ *Why should you attend Cephalocon 2025?*Long-time Ceph Ambassador Frédéric Nass shares why he's excited for Vancouver. Read his blog - https://ceph.io/en/news/blog/2025/cephalocon-blog/. <https://ceph.io/en/news/blog/2025/cephalocon-blog/> *Tech Deep Dive*Check out the 4-part blog series by Daniel Alexander Parkes and Anthony D'Atri on Health-Aware Load Balancing for Ceph RGW with Consul. - Part 1: Building Health-Aware Load Balancing for the Ceph Object Gateway (RGW)with Consul <https://ceph.io/en/news/blog/2025/consul-lb1/> - Part 2: Seamless DNS Routing with CoreDNS and Virtual-Host Style Bucket Access <https://ceph.io/en/news/blog/2025/consul-lb2/> - Part 3. Consul-Powered Global Load Balancing for Ceph Object Gateway (RGW) <https://ceph.io/en/news/blog/2025/consul-lb3/> - Part 4. Consul-Powered Global Load Balancing for Ceph Object Gateway (RGW) <https://ceph.io/en/news/blog/2025/consul-lb4/> *Your Feedback Needed*Help guide Ceph's future by answering our one-question survey on operability. Take the survey - https://forms.gle/RmFeS2CE1BVj4LPz8 *Cephalocon 2025 is Almost Here!*We are nearly one month away from Cephalocon 2025 in Vancouver. Promote the registration link within your social media. Registration link - https://events.linuxfoundation.org/cephalocon/register/. Also, please note the Ceph New Users Workshop on October 27th - https://events.clyso.tech/event/2/registrations/2/. *Join the Cephalocon Slack Channel*Your go-to space for travel tips, schedule updates, and after-events fun. Lastly, if you are interested in becoming more active in our community, please reach out to me at community-manager(a)ceph.io. I am always happy to chat about our community and how we can support one another. Take care! -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Missing directories on download.ceph.com?
by Oliver Freyermuth 25 Sep '25

25 Sep '25

Am 25.09.25 um 17:54 schrieb Matt Vandermeulen: > Interesting, from our mirror it looks like these (and more) are absolute symlinks: > > lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-15.2.16 -> /overflow/debian-15.2.16 > lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-15.2.17 -> /overflow/debian-15.2.17 > lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-octopus -> /overflow/debian-octopus > lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-15.2.16 -> /overflow/rpm-15.2.16 > lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-15.2.17 -> /overflow/rpm-15.2.17 > lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-octopus -> /overflow/rpm-octopus > > Looks like it has been this way for quite some time based on our zfs snapshots: > > lrwxrwxrwx 1 1109 1109 11 Aug 9 2022 .zfs/snapshot/2024-11-01-0620/rpm-octopus -> rpm-15.2.17 > lrwxrwxrwx 1 root root 21 Nov 14 2024 .zfs/snapshot/2024-12-01-0620/rpm-octopus -> /overflow/rpm-octopus > > If you need 15.2.16, a snapshot of it is accessible via: https://mirror.reenigne.net/ceph-snapshots/2024-11-01-0620/rpm-15.2.16/ Many thanks for the helpful mirroring of the snapshots! The way we noticed is since some of our older clients started to report: - Status code: 404 for https://download.ceph.com/rpm-octopus/el8/x86_64/repodata/repomd.xml (IP: 131.220.166.104) today when running DNF, so maybe the absolute symlink target "/overflow/" was only available on the download.ceph.com server and was purged today, or on a separate storage which failed... Cheers and thanks, Oliver > > --- > Thanks, > Matt Vandermeulen > > Software Engineering, and Server Solutions > Reenigne > > Ph: 716.217.0502 > > On 2025-09-25 12:41, Oliver Freyermuth wrote: >> Dear Cephalopodians, >> >> it seems that recently some directories were emptied / became inaccessible on download.ceph.com, for example (some randomly selected ones): >> >> https://download.ceph.com/rpm-octopus >> https://download.ceph.com/rpm-15.2.16 >> >> As older releases such as: >> >> https://download.ceph.com/rpm-mimic/ >> >> are still there, I presume this was not intentional, so I am alerting here in case there might for example be a file system problem on the download mirror. >> >> Cheers, >> Oliver -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

2 2

Poor cephfs kernel client performance on 512-thread AMD EPYC systems
by Vladimir Brik 25 Sep '25

25 Sep '25

Hello We have two computing clusters, a larger one with old machines running Centos 7, and a new test cluster consisting of three 512-thread AMD EPYC system running AlmaLinux 9 (5.14.0-570.42.2 kernel). Both clusters use cephfs 17.2.5 as their shared file system. Everything is fine when metadata-intensive jobs are spread-out among the compute nodes of the older cluster (no more than one or two dozen of those jobs per system). However, if a large number of metadata-intensive jobs run on our big 512-thread systems, ceph performance essentially collapses (at least 60x slowdown compared to the old cluster), with processes spending most of their time in the kernel instead of doing their work. Under load, the EPYC compute nodes have plenty of spare memory; NICs are barely utilized; average CPU utilization 50-80%, almost all of it in sys. Our cephfs metadata pool is on SSDs. It seems like local lock contention is the bottleneck, rather than the MDSes (which are busy-ish, but are not obviously overloaded). I have attached two flamegraphs illustrating the situation. * slowpath.svg shows what happens when a lot (200-500) of processes are frequently calling the stat system call on the same files. * osq_lock.svg show what happens when, I think, a lot of processes are either listing a directory or accessing files in a directory (with about 1000 files). We have also observed MDSes frequently evicting the big EPIC machines, but I haven't figured out what workloads trigger that yet. Does anybody have any advice on how to improve cephfs performance/stability on large systems? Thanks in advance Vlad

1 0

Missing directories on download.ceph.com?
by Oliver Freyermuth 25 Sep '25

25 Sep '25

Dear Cephalopodians, it seems that recently some directories were emptied / became inaccessible on download.ceph.com, for example (some randomly selected ones): https://download.ceph.com/rpm-octopus https://download.ceph.com/rpm-15.2.16 As older releases such as: https://download.ceph.com/rpm-mimic/ are still there, I presume this was not intentional, so I am alerting here in case there might for example be a file system problem on the download mirror. Cheers, Oliver -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

1 0

Object name in Lua postrequest script
by Rok Jaklič 25 Sep '25

25 Sep '25

Hi, Is it possible to get an object name in Lua postrequest script when I am deleting an object? Request.Object.Name does not return anything probably because it's not "there" anymore. Kind regards, Rok

1 0

Re: tentacle 20.1.1 RC1 QE validation status
by Travis Nielsen 24 Sep '25

24 Sep '25

Rook tests look good for tentacle. Thanks, Travis On Wed, Sep 24, 2025 at 5:29 PM Yuri Weinstein <yweinste(a)redhat.com> wrote: > pls reply to all > > ---------- Forwarded message --------- > From: Yuri Weinstein <yweinste(a)redhat.com> > Date: Mon, Sep 22, 2025 at 7:45 AM > Subject: tentacle 20.1.1 RC1 QE validation status > To: dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> > > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/73123#note-1 > > Release Notes - TBD > LRC upgrade - TBD > > Dev Leads, please review the list of suites for completeness, as this is a > new release. > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv > > upgrade/squid-x: Laura > upgrade/reef-x: Laura > > upgrade/telemetry-upgrade: PASSED > > upgrade/tentacle-p2p: N/A > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > crimson-rados: PASSED > powercycle: PASSED > > TIA >

1 0

CephFS shares with multiple access rights: Use subvolumes or multiple filesystems?
by Sophonet 24 Sep '25

24 Sep '25

Hi list, for multiple project-level file shares (with individual access rights) I am planning to use CephFS. Technically this can be implemented both with multiple toplevel cephfs or with a single cephfs in the cluster and subvolumes. What is the preferred choice? I have not found any guidance in docs.ceph.com <http://docs.ceph.com/>. The only location that suggests to use subvolumes is https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=systems-cephfs-volumes…. However, how can I avoid that only one MDS is responsible for serving all subvolumes? Is there some current literature (books or web docs) that contain recommendations and examples? A couple of ceph-related books are available in well-known online book stores, but many of them are rather old (6 years or even more). Thanks a lot, Sophonet

2 5

[ceph-user][ceph-ansible] octopus repository removed from ceph
by Danish Khan 24 Sep '25

24 Sep '25

Dear Team, It seems the octopus repository is either removed or not working. https://download.ceph.com/debian-octopus Do we have any alternative or work around to install octopus packages? Regards, Danish

1 0

Alerts to a mail server?
by lejeczek 24 Sep '25

24 Sep '25

Hi guys. I'm trying to set 'alerts' and with _ssl_ I get: "ceph SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1147)" and without ssl: "ceph SMTP AUTH extension not supported by server." I'm browsing the net but failing to find a clear "fix" for it. Anybody care to share some ideas? Mail server/system - postfix - is otherwise serving a ok. many thanks, L.

1 0

Re: SQUID ganesha failures
by Patrick Bégou 24 Sep '25

24 Sep '25

Hi, nobody reach with this ganesha failure yet ? I've ran some additional configuration tests and discovered that the crashes of ganesha.nfsd only occurs when setting "--ingress-mode haproxy-protocol". Ganesha.nfsd is stable with "--ingress-mode keepalive-only" and "--ingress-mode haproxy-standard". The crashes with "--ingress-mode haproxy-protocol" occurs randomly on the servers, even if no client mount the NFS partition. Patrick Le 21/09/2025 à 17:27, Patrick Bégou a écrit : > Hi, > > just to provide some additional information, it is the ganesha.nfsd > process which core-dumps : > > sept. 19 17:06:12 whitaker02-ceph > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn[85697]: > 19/09/2025 15:06:12 : epoch 68cd5fb6 : whitaker02-ceph : > ganesha.nfsd-2[svc_8] rpc :TIRPC :EVENT :svc_vc_recv: 0x7f99c400a6b0 > fd 50 proxy header rest len failed header rlen = % (will set dead) > sept. 19 17:06:12 whitaker02-ceph > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn[85697]: > 19/09/2025 15:06:12 : epoch 68cd5fb6 : whitaker02-ceph : > ganesha.nfsd-2[reaper] nfs_try_lift_grace :STATE :EVENT :check > grace:reclaim complete(0) clid count(0) > sept. 19 17:06:12 whitaker02-ceph > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn[85697]: > 19/09/2025 15:06:12 : epoch 68cd5fb6 : whitaker02-ceph : > ganesha.nfsd-2[svc_10] rpc :TIRPC :EVENT :svc_vc_recv: 0x7f99b0004060 > fd 50 proxy ignored for local > sept. 19 17:06:13 whitaker02-ceph systemd-coredump[120951]: Process > 85701 (ganesha.nfsd) of user 0 dumped core. > > Stack > trace of thread 77: > #0 > 0x00007f9a939ad32e n/a (/usr/lib64/libntirpc.so.5.8 + 0x2232e) > ELF object > binary architecture: AMD x86-64 > sept. 19 17:06:14 whitaker02-ceph podman[120958]: 2025-09-19 > 17:06:14.123651716 +0200 CEST m=+0.029310490 container died > 5cb71f9d8b8986292dd8916ad0f610b9c12f5c08a1de505cd6f7a11137446263 > (image=quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6, > name=ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn, > org.opencontainers.image.authors=Ceph Release Team > <ceph-maintainers(a)ceph.io>, > CEPH_SHA1=0eceb0defba60152a8182f7bd87d164b639885b8, > org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, > org.label-schema.build-date=20250303, io.buildah.version=1.39.3, > CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=squid, > FROM_IMAGE=quay.io/centos/centos:stream9, > org.opencontainers.image.documentation=https://docs.ceph.com/, > org.label-schema.license=GPLv2, > GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, > org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) > sept. 19 17:06:14 whitaker02-ceph podman[120958]: 2025-09-19 > 17:06:14.139752858 +0200 CEST m=+0.045411632 container remove > 5cb71f9d8b8986292dd8916ad0f610b9c12f5c08a1de505cd6f7a11137446263 > (image=quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6, > name=ceph-aa64f278-3ba8-11f0-b327-303ea701bc10-nfs-whitaker-nfs-4-0-whitaker02-ceph-wjynvn, > FROM_IMAGE=quay.io/centos/centos:stream9, > org.label-schema.vendor=CentOS, > CEPH_SHA1=0eceb0defba60152a8182f7bd87d164b639885b8, CEPH_REF=squid, > org.opencontainers.image.documentation=https://docs.ceph.com/, > org.label-schema.name=CentOS Stream 9 Base Image, > org.label-schema.license=GPLv2, > CEPH_GIT_REPO=https://github.com/ceph/ceph.git, > GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream…, > OSD_FLAVOR=default, org.label-schema.build-date=20250303, > org.opencontainers.image.authors=Ceph Release Team > <ceph-maintainers(a)ceph.io>, io.buildah.version=1.39.3, > org.label-schema.schema-version=1.0) > sept. 19 17:06:14 whitaker02-ceph systemd[1]: > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10(a)nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn.service: > Main process exited, code=exited, status=139/n/a > sept. 19 17:06:14 whitaker02-ceph systemd[1]: > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10(a)nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn.service: > Failed with result 'exit-code'. > sept. 19 17:06:14 whitaker02-ceph systemd[1]: > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10(a)nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn.service: > Consumed 3.485s CPU time. > sept. 19 17:06:24 whitaker02-ceph systemd[1]: > ceph-aa64f278-3ba8-11f0-b327-303ea701bc10(a)nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn.service: > Scheduled restart job, restart counter is at 8. > > Patrick >

1 0

Why does recovering objects take much longer than the outage that caused them?
by Niklas Hambüchen 23 Sep '25

23 Sep '25

I noticed that for my clusters, even a short 5-minute network outage or single-host reboot can cause pgs: 5586988/366684639 objects misplaced (1.524%) which at the speed of recovery: 2.2 GiB/s, 676 objects/s can take hours to recover. I don't understand how this can be. If it's down for so short, how can rebalancing can take this long? I'm using Ceph 19.2.2 on HDDs with SSDs as BlueStore "db" device. Is this perhaps that writes of new files are written linearly to HDD (fast) but recovery seeks around on my HDDs in random order (slow)? In any case, this asymmetry is quite annoying. Could anything be done against it? Thanks! Niklas

6 9

radosgw-admin usage show reports doubled totals for some uids. [reef]
by 游峻權 23 Sep '25

23 Sep '25

Hi, We are running Ceph Reef 18.2.7, and we noticed an inconsistency when checking RGW usage statistics with radosgw-admin usage show. Cluster environment: - 3 MONs, 36 OSDs (HDD + SSD mixed) - RGW daemons: 3 - Deployment method: cephadm When querying a specific UID, the totals look correct: # radosgw-admin usage show --uid 0249bd92a6804133bf8e81f41da928cb | grep total -A5 "total": { "bytes_sent": 468098, "bytes_received": 35998026, "ops": 80, "successful_ops": 77 } But when dumping all usage and filtering for the same UID, the totals are exactly doubled: # radosgw-admin usage show | grep 0249bd92a6804133bf8e81f41da928cb -A100 | grep total -A5 "total": { "bytes_sent": 936196, "bytes_received": 71996052, "ops": 160, "successful_ops": 154 } Interestingly, if we specify a date filter, the numbers return to normal again: # radosgw-admin usage show --date=2025-09-24 | grep 0249bd92a6804133bf8e81f41da928cb -A100 | grep total -A5 "total": { "bytes_sent": 468098, "bytes_received": 35998026, "ops": 80, "successful_ops": 77 } This issue does not affect all UIDs, only some of them. Why does radosgw-admin usage show report doubled totals for certain UIDs when dumping all usage, while --uid or --date shows correct values? Is this expected behavior or a bug? Thanks, Jack

1 0

Recheck for "large omap objects" warning
by Jan Kasprzak 23 Sep '25

23 Sep '25

Hello, Ceph users, on my Ceph cluster with my custom rgw-like front-end client storing metadata to omap I got "1 large omap objects" warning. According to cluster log, the large object is osd.39 (osd.39) 2292 : cluster [WRN] Large omap object found. Object: 76:0bfe63c7:::idx.00000000004:head PG: 76.e3c67fd0 (76.10) Key count: 259173 Size (bytes): 12440304 I have increased the limit using ceph config set osd osd_deep_scrub_large_omap_object_key_threshold 1000000 but the warning still remains. How can I get rid of the warning after increasing the size limit? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

radosgw multisite unable to initialize site config
by Kevin Hrpcek 22 Sep '25

22 Sep '25

Hey all, I've been having some problems with a relatively basic rgw multisite set up. I initially had it working a few weeks ago with a simple config but rebuilt it with ssl enabled for the endpoints and then started having some problems to work through. I eventually had that working but then tore it all down so I could verify the recreation steps but now I'm having problems with the multisite. There are a few different errors.... I set up the master zone without problems, but if I use the rgw module with a token and spec it fails to sync the sync user and the primary zone can't sync back from the secondary zone. A permission denied error for data syncing primary->secondary but secondary-> primary works because it has the credentials. So i used to be able to manually create the user on the secondary zone and get data syncing but the metadata wouldn't sync with the master showing an error message but when I check the mdlog on the primary it looks okay. 2025-09-11T15:15:18.771+0000 7fc29e96c640 1 req 14061042141473454829 0.001000005s failed to read mdlog history: (2) No such file or directory I ended up trying a full clean reset of rgw on both clusters so I deleted the daemons and all related pools. The primary starts up but I still have the problem of the sync user not being created on the secondary and now when I try to create a user on the secondary for syncing or just for the dashboard to work I get errors that I've tracked down to https://github.com/ceph/ceph/blob/62bcf65e8c0995783bb3e368909716346874ad62/… but am not quite following whats all going on in the source to determine why. # radosgw-admin user create --uid=dashboard --display-name=dashboard --system couldn't init storage provider # radosgw-admin user create --uid=sysuser-asdf --display-name=sysuser-asdf --access-key=xxxxxxxxxxxxx --secret-key=yyyyyyyyyyyyyyy --system --rgw-realm=asdf Unable to initialize site config. Any thoughts? It seems like I may have messed something up in the ceph config database from multiple tear downs and restarts of the multisite setup but I'm not seeing what it could be. ceph 19.2.1 using cephadm on alma 9 Thanks! Kevin

4 8

Experience with mixed-arch clusters?
by Alexander Schreiber 22 Sep '25

22 Sep '25

Hi, does anyone have experience with mixed-arch Ceph clusters? Specifically, mixing amd64 and aarch64 machines in the same cluster? I've got my hands one some NVidia Bluefield-2 DPUs: 2x 25G ports, 8x 64bit ARM cores, 16/32G ECC RAM. I'm pondering if sticking those into amd64 OSD machines and using them as non-storage (MDS, MON) nodes would make sense. Or would that end up mightily confusing the tooling, e.g. cephadm? Kind regards, Alex. -- "Opportunity is missed by most people because it is dressed in overalls and looks like work." -- Thomas A. Edison

2 2

SSL cert expired for download.ceph.com
by Boris 21 Sep '25

21 Sep '25

Hi, it looks like the SSL cert just expired.

1 0

advice needed for sizing of wal+db device for hdd
by Adrian Sevcenco 19 Sep '25

19 Sep '25

Hi! i'm trying to estimate the size requiremenent for a wal+db volume for 26 TB HDD while trying to minimize the administrative hassle for configuration (meaning that i would prefer a wal+db volume not separate db from wal) (RGW usage, 3 node cluster) so, for a jbod that can go up to 44 drives (not all drives present, at most up to 12) i have 2x 6.4TB NVME AFAIU for metadata (db) device, i need raid as losing metadata, means losing the associated OSD .. did i get this right? So, what would the the best practice to map db+wal to hdd OSD? should i do a mdraid from the 2 nvme and that to split in 12 partitions? should i split the nvme ssd in 12 namespaces and make individual mdraid for each and that map to osds? For a 6.4 TB NVME (divided to 12) and 26 TB OSD, the db/wal would be ~533 GB, so around 2.05% .. how terrible is this number for RGW usage (i get that the recommended is 4%) At what size the data is in danger if db is too small and when it become safe but only with performance degradation? Thanks a lot!!! Adrian

4 8

Upgrade CEPH 14.x -> 16.X + switch from Filestore to Bluestore = strange behavior
by Olivier Delcourt 19 Sep '25

19 Sep '25

Hi, After reading your posts for years, I feel compelled to ask for your help/advice. First, I need to explain the context of our CEPH cluster, the problems we have encountered, and finally, my questions. Thanks taking the time for reading me. Cheers, Olivier *** Background *** Our CEPH cluster was created in 2015 with version 0.94.x (Hammer), which has been upgraded over time to version 10.2.x (Jewel), then 12.x (Luminous) and then 14.x (Nautilus). The MONitors and CEPHstores have always run on Linux Debian, with versions updated according to the requirements for supporting the underlying hardware and/or CEPH releases. In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient. Each of the storage servers comes with HDDs whose size varies depending on the server generation, as well as SSDs whose size is more consistent but still varies (depending on price). The idea has always been to add HDD and SSD storage to the CEPH cluster when we add storage servers to expand it or replace old ones. At the OSD level, the basic rule has always been followed: one device = one OSD with metadatas (FileStore) on dedicated partitioned SSDs (up to 6 for 32 OSDs) and, for the past few years, on a partitioned NVMe RAID1 (MD). In total, we have: 100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB 84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.6 PiB 1.7 PiB 1.9 PiB 1.9 PiB 53.45 ssd 480 TiB 321 TiB 158 TiB 158 TiB 33.04 TOTAL 4.0 PiB 2.0 PiB 2.1 PiB 2.1 PiB 51.08 Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -2 3660.19751 root main_storage -11 1222.29883 datacenter DC1 -68 163.79984 host cephstore16 -280 109.09988 host cephstore28 -20 109.09988 host cephstore34 -289 109.09988 host cephstore31 -31 116.39990 host cephstore40 -205 116.39990 host cephstore37 -81 109.09988 host cephstore22 -71 163.79984 host cephstore19 -84 109.09988 host cephstore25 -179 116.39990 host cephstore43 -12 1222.29883 datacenter DC2 -69 163.79984 host cephstore17 -82 109.09988 host cephstore23 -295 109.09988 host cephstore32 -72 163.79984 host cephstore20 -283 109.09988 host cephstore29 -87 109.09988 host cephstore35 -85 109.09988 host cephstore26 -222 116.39990 host cephstore44 -36 116.39990 host cephstore41 -242 116.39990 host cephstore38 -25 1215.59998 datacenter DC3 -70 163.80000 host cephstore18 -74 163.80000 host cephstore21 -83 99.00000 host cephstore24 -86 110.00000 host cephstore27 -286 110.00000 host cephstore30 -298 99.00000 host cephstore33 -102 110.00000 host cephstore36 -304 120.00000 host cephstore39 -136 120.00000 host cephstore42 -307 120.00000 host cephstore45 -1 516.06305 root high-speed_storage -21 171.91544 datacenter xDC1 -62 16.84781 host xcephstore16 -259 14.00000 host xcephstore28 -3 14.00000 host xcephstore34 -268 14.00000 host xcephstore31 -310 14.30786 host xcephstore40 -105 14.30786 host xcephstore37 -46 30.68784 host xcephstore10 -75 11.67993 host xcephstore22 -61 16.09634 host xcephstore19 -78 11.67993 host xcephstore25 -322 14.30786 host xcephstore43 -15 171.16397 datacenter xDC2 -63 16.09634 host xcephstore17 -76 11.67993 host xcephstore23 -274 14.00000 host xcephstore32 -65 16.09634 host xcephstore20 -262 14.00000 host xcephstore29 -13 14.00000 host xcephstore35 -79 11.67993 host xcephstore26 -51 30.68784 host xcephstore11 -325 14.30786 host xcephstore44 -313 14.30786 host xcephstore41 -175 14.30786 host xcephstore38 -28 172.98364 datacenter xDC3 -56 30.68784 host xcephstore12 -64 16.09200 host xcephstore18 -67 16.09200 host xcephstore21 -77 12.00000 host xcephstore24 -80 12.00000 host xcephstore27 -265 14.39999 host xcephstore30 -277 14.39990 host xcephstore33 -17 14.39999 host xcephstore36 -204 14.30399 host xcephstore39 -319 14.30399 host xcephstore42 -328 14.30396 host xcephstore45 Our allocation rules are: # rules rule main_storage_ruleset { id 0 type replicated min_size 1 max_size 10 step take main_storage step chooseleaf firstn 0 type datacenter step emit } rule high-speed_storage_ruleset { id 1 type replicated min_size 1 max_size 10 step take high-speed_storage step chooseleaf firstn 0 type datacenter step emit } All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1). This CEPH cluster is currently only used for RBD. The volumes are used by our ~ 1,200 KVM VMs. *** Problems *** Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem). * First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start. We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore. * Second problem: Since our failure domain is a DC, we had to upgrade a DC and then wait for it to recover (~500 TB net). SSD storage recovery takes a few hours, while HDD storage recovery takes approximately three days. Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us. I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help). Once our VMs were shut down (crashed), the cluster completed its recovery (HDD-type OSDs) and, curiously, the SSD-type OSDs began to ‘empty’. The day after that, we began updating the storage servers in our second DC, and the phenomenon started again. We did not wait until we reached full_ratio to shut down our virtualisation environment and this time, the ‘SSD’ OSDs began to ‘empty’ after the following commands: ceph osd unset noscrub && ceph osd unset nodeep-scrub. In fact, we used to block scrubs and deep scrubs during massive upgrades and recoveries to save I/O. This never caused any problems in FileStore. It should be added that since we started using the CEPH Cluster (2015), scrubs have only been enabled at night so as not to impact production I/O, via the following options: osd_recovery_delay_start = 5, osd_scrub_begin_hour = 19, osd_scrub_end_hour = 7, osd_scrub_sleep = 0.1 (the latter may be removed since classes are now available) . After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer). *** Questions / Assumptions / Opinions *** Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution and is not desirable in the medium term, but we are dependent on recovery time (and, in addition, on the issue I am presenting to you here). Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ? I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case: root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’, This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up? (It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).) Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.

5 9

Scrub errors after OSD_TOO_MANY_REPAIRS: how to recover?
by Michel Jouvin 18 Sep '25

18 Sep '25

Hi, We experienced one week ago problems with a dying OSD resulting in OSD_TOO_MANY_REPAIRS, unfortunately unnoticed (it seems our monitoring system is not notifying these errors properly). When we realized the error, we removed the problematic OSD (ceph orch osd rm --replace), despite the scrubbing errors: the resulting backfills succeeded but did not fix the scrub errors. The collegue who took care of this problem decided to lauch a `ceph pg repair` on the 3 PGs with reported inconsistencies but it doesn't seem to converge. 'ceph -s' still reports : 3 active+clean+scrubbing+deep+inconsistent+repair after a few hours and for at least one of the PG, there is the following message every 3s: Sep 18 12:30:55 ceph-76212 ceph-mon[2506]: osd.72 pg 11.e2d Deep scrub errors, upgrading scrub to deep-scrub Not sure if it is the sign of a problem or just because the operation is ongoing. I'm looking for advices on what to do to move forward. There was not yet a report from users of an impact of this but it doesn't mean there is none... The affected pool is storing RBD volumes (from OpenStack Cinder). A side issue is that we have 'osd_scrub_auto_repair=true' so I'd expect the repair to start automatically but it doesn't seem it was the case... Thanks in advance for any advice. Best regards, Michel

3 3

Failing upgrade 18.2.7 to 19.2.3 - failing to activate via raw takes 5 minutes (before proceeding to lvm)
by Mikael Öhman 18 Sep '25

18 Sep '25

I'm fighting with a ceph upgrade, going 18.2.7 to 19.2.3. This time again the ceph-volume activate step is taking to long, triggering failures due to systemd service timing out so the orch daemon fails (though the osd does eventually come up, the daemon is still dead, and upgrade halts). I can also reproduce the slowdown of startup with cephadm ceph-volume raw list (I don't use raw devices, but the ceph-volume activation method hardcodes checking raw first https://github.com/ceph/ceph/blob/4d5ad8c1ef04f38d14402f0d89f2df2b7d254c2c/… ) That's takes 6s on 18.2.7, but 4m32s minutes on 19.2.3 ! I have 42 spinning drives per host (with multipath). It's spending all of it's time in the new method: self.exclude_lvm_osd_devices() and the list of items to scan, given all the duplication from multipath + and mapper names, it ends up with 308 items to scan in my setup. With good old print debugging, i found that while the threadpool speeds things up a bit, it simply takes to long to construct all those Device() objects. In fact, just creating a single Device() object, since it needs to call disk.get_devices() at least once, since this list does not include all devices, it filters out things like "/dev/mapper/mpathxx" from the list, but the code always regenerates (the same) device list if the path isn't found: if not sys_info.devices.get(self.path): sys_info.devices = disk.get_devices() will now force it to re-generate this list >400 times (initial 32 times in parallel, followed by about 400 more which will never match the device name). In the end, it's again O(n^2) computational time to list these raw devices with ceph-volume. So with 32 threads in the pool, it's also now requires running heavy load for 5 minutes before completing this trivial task every time the deamon needs to start.

3 4

September Ceph Science User Group Virtual Meeting
by Enrico Bocchi 18 Sep '25

18 Sep '25

Dear All, We will be having a Ceph science/research/big cluster call on Tuesday, September 23rd, as per ceph.io calendar: https://ceph.io/en/community/meetups/ Kevin Hrpcek from UCAR will report on his recent experience migrating from the Ceph CSI to the new Ceph CSI Operator. This is an informal open call for community members primarily from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc. There is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific, this can be added to the Pad linked below. If you are interested in giving a short talk during one of the next meetings, please reach out to me. Virtual event details: Sept 23rd, 2025 7am PDT 9am Central US (CDT) 14:00 UTC 16:00 Central European Summer Time -- please double-check the meeting time for your time zone -- Join the meeting at https://meet.jit.si/ceph-science-wg The meeting will be recorded and posted to the Ceph YouTube channel. Event Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20250923 Main Pad for the Ceph Science User Group: https://pad.ceph.com/p/Ceph_Science_User_Group_Index See you next Tuesday! Enrico

1 0

Per-RBD-image stats
by William David Edwards 18 Sep '25

18 Sep '25

Hi, What do people use to store per-RBD-image I/O stats? To retrospectively see which VM was keeping a Ceph cluster busy in terms of I/O, I'd like to track `rbd perf image iostat` and `rbd perf image iotop`-like metrics. I've looked at the LibreNMS plugin [1], Prometheus metrics in the manager [2], and Ceph ingestion in Elasticsearch [3]. None of them seem to export per-RBD metrics. I can't find much information about this use case on the mailing list either, though it's hard for me to imagine I'm the only one with this want. How do other users handle this? William David Edwards [1]: https://github.com/librenms/librenms-agent/blob/master/agent-local/ceph [2]: https://docs.ceph.com/en/latest/mgr/prometheus/ [3]: https://www.elastic.co/docs/reference/integrations/ceph

5 4

Reef CEPHADM_CHECK_PUBLIC_MEMBERSHIP false positive alerts on some hosts (IPv6 only cluster)
by Boris 18 Sep '25

18 Sep '25

Hi, I am currently debugging an issue with the ceph config checks. We have some random hosts that alert "HOSTNAME does not have an interface on any public network" but they have. It is IPv6, static configured and, because we don't have a cluster_network, OSDs are bound to that IP in the specific network. I went through the netplan config and they are basically the same on all hosts. And after rebooting the hosts some of them resolved and some didn't. How can I dig deeper to figure out what is going on. All services are ceph-orch podman containers All hosts are ubuntu 22.04 with latest HWE kernel (6.8.0-79-generic) Best wishes Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 3

Where/how to set PG auto repair/heal?
by lejeczek 18 Sep '25

18 Sep '25

Hi guys. I get PG inconsistencies constantly. I have a small, 3-node lab cluster, eg. -> $ ceph health detail HEALTH_ERR 4 scrub errors; Possible data damage: 4 pgs inconsistent [ERR] OSD_SCRUB_ERRORS: 4 scrub errors [ERR] PG_DAMAGED: Possible data damage: 4 pgs inconsistent pg 5.15 is active+clean+inconsistent, acting [5,3,4] pg 5.4f is active+clean+inconsistent, acting [5,4,3] pg 5.64 is active+clean+inconsistent, acting [5,3,4] pg 5.78 is active+clean+inconsistent, acting [3,4,5] I read but I failed to find a doc/howto on relevant tweaking/setting - could you point me to such a doc, if one exists, where it is covered? Also, are there perhaps different best-practices on small VS large cluster & manual VS auto PG heal/repair? Say - should PG repair be always set/delegated to self/auto VS to never? many thanks, L.

2 1

CephFS maturity on Squid?
by William David Edwards 18 Sep '25

18 Sep '25

Hi, For those running CephFS on Squid: how mature is the release in your experience? I'm especially interested in those using CephFS in a shared hosting scenario, with many directories containing many small files. I'm extra wary of new releases nowadays, as we've ran into several production-impacting Ceph bugs in the past months post-upgrade, across clusters. (Most notably, OSDs failing causing the MDS to OOM (17.2.7 -> 17.2.8) [1], and major performance issues after upgrading to Reef caused by the aligned 64k blocks constraint [2] (RBD though, not CephFS).) With kind regards, William David Edwards [1]: https://tracker.ceph.com/issues/69764 [2]: https://github.com/ceph/ceph/pull/54772

4 4

Is there a faster way to merge PGs?
by Justin Mammarella 18 Sep '25

18 Sep '25

Hi folks, We are in the process of shrinking a CEPH cluster from around 2000 osds to 300 osds. We need to reduce our pg count from 16384 to 8192. After setting pg_num to 8192, it’s currently it’s taking 3 hours to reduce the pool’s PG count by one. Which given the number of PGs, will take over 2 years to complete. Are there any tunables to increase the speed of pg merging? Our alternative is to create a new pool and transfer the data Regards, Justin

2 3

cephfs volume mount error using both ceph-csi and mount.ceph (using -o mon_addr)
by John Jasen 17 Sep '25

17 Sep '25

So, when I try to get a container to mount via cephfs, it uses these options: "-t ceph cephfs@60eb7a9e-8cc6-11f0-a7d0-e43d1a153e91.k8cephfs=/volumes/k8s/csi-vol-c76f1825-df8e-477e-981a-bf5fccb66fe3/fbed6ce0-4446-449c-9a14-8c7b297499a2 /var/lib/kubelet/plugins/ kubernetes.io/csi/cephfs.csi.ceph.com/153624a7bdfffe5dfdec7dfd5c866889b7a54… -o mon_addr=my.ip.addr.1:3300/my.ip.addr.5:3300/my.ip.addr.6:3300/my.ip.addr.7:3300/my.ipaddr.11:3300,secretfile=/tmp/csi/keys/keyfile-4057439945,_netdev" And it fails: stderr: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized When I feed these options into mount.ceph, it fails, with the same errors. When I feed each of my.ip.addr into mount.ceph as mon_addr, I get similar failures. When I remove mon_addr options totally from the mount.ceph command, forcing it to work through /etc/ceph/ceph.conf for finding monitors -- it succeeds. ceph.conf is a minimal one built from ceph config generate-minimal-conf, which reads: "# minimal ceph.conf for 60eb7a9e-8cc6-11f0-a7d0-e43d1a153e91 [global] fsid = 60eb7a9e-8cc6-11f0-a7d0-e43d1a153e91 mon_host = [v2:my.ip.addr.11:3300/0,v1:my.ip.addr.11:6789/0] [v2:my.ip.addr.1:3300/0,v1:my.ip.addr.1:6789/0] v2:my.ip.addr.5:3300/0 [v2:my.ip.addr.6:3300/0,v1:my.ip.addr.6:6789/0] [v2:my.ip.addr.7:3300/0,v1:172.17. 10.7:6789/0]" Any idea what's going on?

1 1

Async dirops maturity
by William David Edwards 17 Sep '25

17 Sep '25

Hi, Async dirops were enabled in the kernel by default in 2021 [1]. In 2023, a patch was made to revert that default because of MDS crashes [2]: > We have identified an issue in the MDS affecting CephFS users using > the kernel driver. The issue was first introduced in the octopus > release that added support for clients to perform asynchronous > directory operations using the `nowsync` mount option. The issue > presents itself as an MDS crash ... but the conclusion was then drawn that `mds_client_delegate_inos_pct` can be used instead to disable async dirops, so the patch didn't go through [3]: > After some digging and talking to Jeff, I figured that it's possible > to disable async dirops from the mds side by setting > `mds_client_delegate_inos_pct` config to 0: [...] > So, I guess this patch is really not required. We can suggest this > config update to users and document it for now. We lack tests with > this config disabled, so I'll be adding the same before recommending > it out. Will keep you posted. ... and in 19.2.0 (2024), `mds_client_delegate_inos_pct` defaults to 0, which disables async dirops [4]: > CephFS: Disallow delegating preallocated inode ranges to clients. > Config mds_client_delegate_inos_pct defaults to 0 which disables async > dirops in the kclient. Aforementioned 'MDS crash' is said to be caused by bugs 61009[5] and 58489[6]. Neither have been fixed, and it's not even clear to me how the latter relates to async dirops / preallocated inodes. All in all: what is the state of async dirops? Is it anticipated to be mature in the near future? Is it actively being worked on? If not, what do other users (especially in a shared hosting scenario, with many directories with many small files) use to speed up `rm -rf`s without having every unlink() take a round-trip to the MDS? With kind regards, William David Edwards [1]: https://github.com/torvalds/linux/commit/f7a67b463fb83a4b9b11ceaa8ec4950b8f… [2]: https://www.spinics.net/lists/ceph-devel/msg58389.html [3]: https://www.spinics.net/lists/ceph-devel/msg58394.html [4]: https://ceph.io/en/news/blog/2024/v19-2-0-squid-released/ [5]: https://tracker.ceph.com/issues/61009 [6]: https://tracker.ceph.com/issues/58489

1 0

Subject : Account-Level API Support in Ceph RGW for Production Use
by Dhivya G 17 Sep '25

17 Sep '25

Dear Ceph Community, I hope this message finds you well. We are currently working on integrating Ceph RGW into our system and have encountered a challenge regarding account-level management via API. At present, we are using radosgw-admin commands programmatically within our code to manage and retrieve account-level information. While this approach works, we understand that relying on radosgw-admin is not recommended for production environments due to its direct interaction with the backend and potential impact on performance and stability. We have explored the available documentation and interfaces but could not find a dedicated Account-level API (e.g., for managing users, retrieving account metadata, etc.) that can be safely used in production over RESTful protocols (such as S3 or Swift-compatible APIs). We are seeking clarification on the following points: Is there an officially supported Account API in Ceph RGW that allows account-level operations via REST or other production-safe interfaces? If not, is there a roadmap or recommended alternative for performing these actions in a production-ready manner? Are there any best practices or tools from the community that can bridge this gap? We would greatly appreciate any guidance, documentation, or recommendations you can share . Best regards Dhivya G https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

3 2

[ceph-user]ceph-ansible || Ceph upgrade from octopus to quincy
by Danish Khan 15 Sep '25

15 Sep '25

Dear Team, We are upgrading ceph-ansible cluster from octopus to quincy, we have upgraded few of our cluster successfully using below document: https://docs.ceph.com/projects/ceph-ansible/en/latest/day-2/upgrade.html But Ceph clusters with multisite enabled are not getting upgraded. due to limitation and complexity of ceph-ansible. Does anyone one any idea how we can upgrade ceph-ansible clusters with multi-site. I was planning to break the multi-site for a few days and upgrade the cluster and enable multisite again. but I don't have any smooth steps to do it. Can someone please help me on this. Regards, Danish

1 0

Which Ceph RPMs on AlmaLinux 10?
by Jan Kasprzak 15 Sep '25

15 Sep '25

Hi, Ceph users, does anybody use Ceph on AlmaLinux10 (RHEL 10)? If so, which RPMs? I tried to rebuild 19.2.3 el9 src.rpm from download.ceph.com, and got the following problems: - there is no pathfix.py in python3-devel in AlmaLinux 10 - src/common/openssl_opts_handler.cc needs #include <openssl/err.h> - the same file (openssl_opts_handler.cc) calls ENGINE_load_builtin_engines() which does not exist in Alma10 openssl anymore Which Ceph RPMs do you use on AlmaLinux 10? I am about to deploy a new Ceph cluster, and I would rather use the newest enterprise distro (Alma10). Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

capabilities needed for subvolume management?
by Robert Sander 12 Sep '25

12 Sep '25

Hi, what capabilities does a cephx key need to manage subvolumes in a filesystem? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

v20.1.0 Tentacle RC0 released
by Yuri Weinstein 12 Sep '25

12 Sep '25

This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0 Feature highlights: * RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4

5 10

Re: snap-schedule not running
by Sophonet 12 Sep '25

12 Sep '25

Thanks Eugen, after enabling debugging temporarily, I have seen an error in the logs when creating the snapshot with mkdir snapshot in .snap including a suspicious line mds.0.server is a descendent of a subvolume dir Indeed, when doing a snapshot on the root folder of the filesystem (which works), the log files tell me the inode of the root directory and ceph daemon mds.MDSID dump inode 0x1 (for the root folder) outputs many information in json, including „is_subvolume“: true Indeed, for a different filesystem, where snapshots work as expected in root folder and subfolders, the same corresponding information from the root inode says „is_subvolume“: false. How can the root folder in a filesystem be a subvolume and what can I do to fix this? Also I am puzzled what has caused this problem… Thanks, Sophonet

2 1

[18.2.4 Reef] Using RAID1 as Metadata Device in Ceph – Risks and Recommendations
by Alex from North 11 Sep '25

11 Sep '25

Hi all! I’ve modified Ceph to allow using a RAID1 array as the metadata device. Specifically, I've updated ceph_volume/util/device.py and ceph_volume/util/disk.py to recognize the "raid1" device type as valid device for OSDs and metadata storage. The changes are listed below. My questions: - What are the risks or issues which may arise from using RAID1 for metadata? (e.g., performance, reliability, data integrity) - Why this could be a bad idea, or in what scenarios could it be beneficial? My test setup: - Ceph version: 18.2.4 Reef (stable) - OS: Ubuntu 22.04.5 LTS - Hardware: 1 host, RAID1 array on two NVMe SSDs (/dev/nvme0n1 and /dev/nvme0n2, 19.98 GiB) via mdadm for block.db, 4 HDDs for OSD data - Additional details: Tested on a small cluster on VMs; no noticeable performance changes so far, but I’m concerned about long-term implications. Code changes: diff --git a/src/ceph-volume/ceph_volume/util/device.py b/src/ceph-volume/ceph_volume/util/device.py index 1b52774d1a1..148a8e326f8 100644 --- a/src/ceph-volume/ceph_volume/util/device.py +++ b/src/ceph-volume/ceph_volume/util/device.py @@ -237,7 +237,7 @@ class Device(object): self.disk_api = dev device_type = dev.get('TYPE', '') # always check is this is an lvm member - valid_types = ['part', 'disk', 'mpath'] + valid_types = ['part', 'disk', 'mpath', 'raid1'] if allow_loop_devices(): valid_types.append('loop') if device_type in valid_types: @@ -489,7 +489,7 @@ class Device(object): elif self.blkid_api: api = self.blkid_api if api: - valid_types = ['disk', 'device', 'mpath'] + valid_types = ['disk', 'device', 'mpath', 'raid1'] if allow_loop_devices(): valid_types.append('loop') return self.device_type in valid_types diff --git a/src/ceph-volume/ceph_volume/util/disk.py b/src/ceph-volume/ceph_volume/util/disk.py index 2984c391d06..5dafd6dc62a 100644 --- a/src/ceph-volume/ceph_volume/util/disk.py +++ b/src/ceph-volume/ceph_volume/util/disk.py @@ -362,7 +362,7 @@ def is_device(dev): TYPE = lsblk(dev).get('TYPE') if TYPE: - return TYPE in ['disk', 'mpath'] + return TYPE in ['disk', 'mpath', 'raid1'] # fallback to stat return _stat_is_device(os.lstat(dev).st_mode) and not is_partition(dev) Any feedback is really appreciated as all of us care about not to lost our precious bytes :)

3 6

Mirroring images per snapshot to a different pool on remote site
by Emmanuel Lacour 11 Sep '25

11 Sep '25

Dear ceph users, I would like to enable mirroring of a site A to a site B, using per images mirroring and snapshots, but I would like to point to a different pool name on site B (there is already a pool with same name used on site B and I want to target another one with less replication). It seems this is not supported (by reading the doc), but I ask here in case of ...

1 0

OSD socket closed
by Samuel Moya Tinoco 10 Sep '25

10 Sep '25

Hi everyone, Recently i had a problem with my pve cluster and while looking through logs we find this kind of logs in every node Aug 29 10:44:26 pve01-poz kernel: [697358.187283] libceph: osd17 (1)10.0.0.3:6810 socket closed (con state OPEN) Aug 29 10:45:43 pve01-poz kernel: [697434.733343] libceph: osd1 (1)10.0.0.1:6837 socket closed (con state OPEN) Aug 29 10:46:54 pve01-poz kernel: [697506.414196] libceph: osd15 (1)10.0.0.1:6836 socket closed (con state OPEN) Aug 29 10:47:56 pve01-poz kernel: [697567.855454] libceph: osd4 (1)10.0.0.1:6802 socket closed (con state OPEN) Aug 29 10:49:08 pve01-poz kernel: [697639.497740] libceph: osd4 (1)10.0.0.1:6802 socket closed (con state OPEN) Aug 29 10:51:50 pve01-poz kernel: [697802.060504] libceph: osd12 (1)10.0.0.3:6811 socket closed (con state OPEN) Aug 29 10:55:37 pve01-poz kernel: [698028.664214] libceph: osd1 (1)10.0.0.1:6837 socket closed (con state OPEN) The problem we had was with corosync that is using another network and other ethernet ports. But we'd like to know if this logs are normal and it's just part of how Ceph work or if we should dig deep to check what is causing this logs Thank you in advance for your time and help Samuel Moya Tinoco Departamento de Sistemas y Redes Móvil: (+34) 606985997 smoya(a)vivelibre.es<mailto:smoya@vivelibre.es> ViveLibre C/ La Orotava 4 28660 Boadilla del Monte Madrid www.vivelibre.es<http://www.vivelibre.es/> [cid:image001.png@01DC219E.CAC50090] Soluciones inteligentes para la autonomía personal "Tanto este mensaje como todos los posibles documentos adjuntos al mismo son confidenciales y están dirigidos exclusivamente a los destinatarios de los mismos. Por favor, si Usted no es uno de dichos destinatarios, notifíquenos este hecho y elimine el mensaje de su sistema. Queda prohibida la copia, difusión o revelación de su contenido a terceros sin el previo consentimiento por escrito de VIVELIBRE AUTONOMÍA PERSONAL S.L.U. (VIVELIBRE). En caso contrario, vulnerará la legislación vigente. De conformidad con lo establecido en el Reglamento (UE) 2016/679, General de Protección de Datos, le informamos de que sus datos son objeto de tratamiento por VIVELIBRE, en calidad de Responsable del Tratamiento. VIVELIBRE tratará sus datos con la finalidad de mantener la relación contractual, gestionar su solicitud, así como remitirle comunicaciones de carácter comercial relacionadas con su ámbito de actividad y los servicios prestados. Si desea ejercitar sus derechos de acceso, rectificación, supresión, limitación, oposición o portabilidad, puede dirigirse a la dirección postal Calle de la Orotava 4, 28660, Boadilla del Monte, (Madrid) o a la dirección de correo electrónico protecciondedatos(a)vivelibre.es. Para obtener más información sobre cómo tratamos sus datos, consulta nuestra Política de Privacidad en Política de Privacidad - Vivelibre."

4 3

test
by wissem mimouna 10 Sep '25

10 Sep '25

test

1 0

peer... is using msgr V1 protocol ?
by lejeczek 09 Sep '25

09 Sep '25

Hi guys. I'm getting following in logs: ... 2025-09-09T11:49:00.866+0000 7f3976084640 -1 --2- 10.1.1.61:0/3730235752 >> [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] conn(0x55ee3d59c400 0x55ee3ec6a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] is using msgr V1 protocol --2- 10.1.1.61:0/3730235752 >> [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] conn(0x55ee3d59c400 0x55ee3ec6a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] is using msgr V1 protocol 2025-09-09T11:49:00.876+0000 7f3976885640 -1 --2- 10.1.1.61:0/1308618359 >> [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] conn(0x55ee3e85a000 0x55ee3e860000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] is using msgr V1 protocol --2- 10.1.1.61:0/1308618359 >> [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] conn(0x55ee3e85a000 0x55ee3e860000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] is using msgr V1 protocol ... This three-node containerized cluster. Only one peer-node logs these error - its IP is 10.1.1.61 - xx.62 & xx.63 are other two peer-nodes. Where/how would you start troubleshooting this? many thanks, L.

2 2

ceph restful key permissions
by BASSAGET Cédric 09 Sep '25

09 Sep '25

Hello, I'm trying to create a restful api key with read-only permissions, but I didn't find the way to do that. Is that doable ? Regards Cédric

2 1

Ceph OSD with migrated db device fails after upgrade to 19.2.3 from 18.2.7
by Soenke Schippmann 09 Sep '25

09 Sep '25

Hi, after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume. OSD log shows: 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help. ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated. Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error: --> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f ceph-volume lvm list shows outdated db uuid and db device: ===== osd.256 ====== [block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc [db] /dev/sdai1 PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though). Is there any way to fix this? Best, Sönke -- Sönke Schippmann Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen E-Mail: schippmann(a)uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327 http://www.uni-bremen.de/zfn/

5 6

Reducing the OSD Heartbeat Grace & Interval
by Alexander Hussein-Kershaw 09 Sep '25

09 Sep '25

Hi Folks, I'm running Ceph on VMs in Azure. Occasionally, a maintenance event will take a VM down (typically it will freeze the VM for 30s). There are some limited controls over this on Azure, but they are pretty lackluster. I'm nervous about the 15s heartbeat grace period (default) and the heartbeat interval, which I think will take down an OSD after 15s if no response to heartbeats. But before that point I expect the unresponsive OSD to block all writes to the PGs involved, for the duration of the grace period. I'm considering reducing the 15s heartbeat to a lower value to reduce the impact of this, with the aim of removing an unresponsive OSD from the cluster faster. Hoping to get a feel on this before I consider it further. * Is this a totally stupid idea? * Has anyone had any experience tweaking these parameters? * How low is reasonable to go? I set the interval and grace to 1 sec and the cluster seems stable, but I've not ran a load test yet. * Does the heartbeat have a dedicated thread or is it prone to be being blocked behind other traffic? Many thanks, Alex

1 0

After upgrading Ceph cluster from Octopus (v15.2.13) to Quincy (v17.2.8), read latency on HDD OSDs increased from 15ms to around 100ms during peak I/O periods.
by Best Regards 09 Sep '25

09 Sep '25

Hi,  We used Ceph's RBD feature in triple-replica mode. Every day at midnight, the virtual machine guest_os has a logrotate task to cut and compress various logs. Before the upgrade, at midnight, IO latency increased from 7ms to around 15ms, which had a minor impact. However, after upgrading to version Q, latency jumped from 7ms to 100ms, with some OSDs even experiencing latency reaching 200ms, significantly impacting guest_os operations. We upgraded from O to P, and then from P to Q the next day. Version P ran for a day with normal latency, but after upgrading to version Q, latency increased dramatically (this refers to the midnight period; there was no noticeable difference at other times). I checked the monitoring data before and after the upgrade, from the pool level to the OSD level and then to the driver level, and found that the ceph_pool_rd indicator and the ceph_osd_op_r (all OSDs associated with the pool) indicator did not show any significant increase before and after the upgrade, but the node_disk_reads_completed_total indicator (associated with the OSD) increased significantly after the upgrade. Then I randomly selected several related OSDs and checked the monitoring data before and after the upgrade, and found that the number of read-type IOs did increase, almost doubling from the original 150 ops to 300+ ops, which seems to exceed the capacity of the HDD and the latency is very large. I searched a lot of blogs, and they said that it might be related to the bluestore_min_alloc_size_hdd parameter being reduced from 65536 to 4096, but many articles do not recommend modifying this value multiple times and should be handled with caution, so I ask for help from the experts to confirm this problem for me, and whether there is a safer solution. Thanks. Best Regards wu_chulin(a)qq.com Best Regards

2 2

CSC Meeting Notes September 8th 2025
by Adam King 08 Sep '25

08 Sep '25

- Tentacle RC0 finishing touches, open items - Radek and Laura will update release announcement for core today - Requiring teuthology runs for PRs that modify src/ dir - RFE: https://tracker.ceph.com/issues/72743 - would also be nice to have some process to make sure PRs don't wait around too long without a run - especially those spanning multiple components or areas without a clear owner - Our organization on Github needs some attention, we have banners mentioning we need to accept the ToS, one about sales tax, and it looks like our number of members is greater than our number of seats - banners have been around for some time - appears to only be visible for those with the admin role in the ceph org - US sales tax and exemption: Please update your billing information for verification and add a sales tax exemption certificate, if applicable. Learn more about this change. - Terms of Service review required. Organizations on the enterprise plan must now accept the GitHub Customer Agreement. Not doing so may put your organization at risk of being downgraded. - Tentacle updates - Will Tentacle release GA be based on RC1 (current RC), or will there be an RC2 before GA? - dependent on results of ongoing testing - topic to be moved to next week - If RC2 is planned, when will it be built? - What is the expected timeline for RC2 and Tentacle GA? - no responses on https://github.com/ceph/ceph/pull/63912 to add ubuntu 24 to qa suites - Adam King to add to orch run

1 0

Moving to Zoom for today's CDS session
by Anthony Middleton 08 Sep '25

08 Sep '25

Jitsi seems to be down. Here's the link for the CDS session - https://us06web.zoom.us/j/84591547745?pwd=nc9T96xfWKbeO5J5GxEBgxUKsCIdxc.1

1 0

RadosGW read/list-all user across all tenants, users, buckets
by Jacques Hoffmann 08 Sep '25

08 Sep '25

Hello Ceph:) We are wondering if it is possible to create some RadosGW credentials/user with readonly (and allowed to list everything) on all buckets. We are working on a backup strategy, and it would be nice if there was a user that can list all buckets and read all object from all users, with just one access-key/secret-key pair. Is that possible? Ceph Squid does support the IAM API, but that just raises more questions than it answers. Can we get this readonly access across all tenants, for all users and buckets, globally? I would assume that tenants are now somewhat isolated, so maybe this is not possible. The rules of this are fairly unclear to me currently, especially because so far I've assumed that bucket names are globally unique, which i am no longer sure of with multitenancy. Thank you, Jacques Hoffmann -- Jacques Hoffmann <jacques.hoffmann(a)vis.ethz.ch> VIS - Verein der Informatikstudierenden an der ETH Zürich CAB E 31, Universitätstr. 6, ETH Zentrum, CH-8092 Zürich https://www.vis.ethz.ch

2 1

Deploy haproxy.nfs
by Thomas 07 Sep '25

07 Sep '25

Hi, I'm experimenting with Ceph+NFS and I would like to set up a NFS-HA solution. I tried to deploy ingress.nfs service via Dashboard. It worked well on amd64, but it failed on arm64: Failed while placing haproxy.nfs.tp-ceph.wbCeph-2.yvjyvl on wbCeph-2.example.com: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-145d1d9a-698a-11f0-ac29-2ccf67085bc4-haproxy-nfs-tp-ceph-wbCeph-2-yvjyvl /usr/bin/podman: stderr Error: no such container ceph-145d1d9a-698a-11f0-ac29-2ccf67085bc4-haproxy-nfs-tp-ceph-wbCeph-2-yvjyvl Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-145d1d9a-698a-11f0-ac29-2ccf67085bc4-haproxy.nfs.tp-ceph.wbCeph-2.yvjyvl /usr/bin/podman: stderr Error: no such container ceph-145d1d9a-698a-11f0-ac29-2ccf67085bc4-haproxy.nfs.tp-ceph.wbCeph-2.yvjyvl Deploy daemon haproxy.nfs.tp-ceph.wbCeph-2.yvjyvl ... Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/haproxy:2.3 -e NODE_NAME=wbCeph-2.example.com quay.io/ceph/haproxy:2.3 -c %u %g /var/lib stat: stderr WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64) stat: stderr ERROR (catatonit:2): failed to exec pid1: Exec format error ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/haproxy:2.3 -e NODE_NAME=wbCeph-2.example.com quay.io/ceph/haproxy:2.3 -c %u %g /var/lib: WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64) ERROR (catatonit:2): failed to exec pid1: Exec format error How can I tell ceph to pull the image for the arm64 platform? Thank's in advance. Thomas

1 0

nfs-ganesha, subvolumes and subtree checking?
by Davíð Steinn Geirsson 06 Sep '25

06 Sep '25

Hey all, When I moved my file server shares to CephFS, I set each share on its own CephFS. The reason was this comment in the nfs-ganesha example config: # Note that FSAL_CEPH does not support subtree checking, so there is # no way to validate that a filehandle presented by a client is # reachable via an exported subtree. # # For that reason, we just export "/" here. Now, this is fine for low numbers of shares, but as they have grown it feels a bit overkill to be creating two or more new pools (metadata + data + sometimes another EC data) for each share. Tuning the PG numbers for those pools is also kind of a pain. I'm wondering, would using a subvolume for the share provide the needed security isolation? Best, Davíð

2 3

Question about shard placement in erasure code pools
by Soeren Malchow 06 Sep '25

06 Sep '25

Dears, maybe someon can explain/answer my question. We are working on setting up a new ceph cluster (underneath a proxmox cluster), the setup is as follows 3 datacenters 3 hosts per datacenter (all in one rack) 8 physical disks per host This is the profile configutation ceph osd erasure-code-profile set ec_profile plugin=isa technique=reed_sol_van k=4 m=2 crush-root=default crush-failure-domain=datacenter crush-device-class=nvme crush-osds-per-failure-domain=2 crush-num-failure-domains=3 We have a hierarchy - created with a custom_location_hook that contains * datacenter 01 * rack 01 * host 01 * host 02 * host 03 * datacenter 02 * rack 01 * host 01 * host 02 * host 03 * datacenter 03 * rack 01 * host 01 * host 02 * host 03 So this means (my understanding) that we are placing 2 shards of data in each datacenter. We want to make sure that we only have max one share of data per host, and in future if possible also distributed across racks once we should ahve more per DC. Is the bucket hierarchy automatically taken into account when choosing the OSD ? Or if not, where do i start, with CRUSH rules ? (I am hesitant to manually modify CRUSH rules. Thanks in advance Soeren

2 1

Is there a preferred EC plugin?
by Boris 05 Sep '25

05 Sep '25

Hi, we currently use the jerasure plugin, but maybe there is a new better default? We use EC mostly for large rgw data with files >1GB. Cheers Boris

3 3

Append to EC pool object (docs vs. reality)
by Jan Kasprzak 05 Sep '25

05 Sep '25

Hello, Ceph users, my erasure-coded pool is inconsistent with the documentation, apparently existing objects cannot be appended to. The documentation states the following: ================ https://docs.ceph.com/en/latest/rados/operations/erasure-code/ By default, erasure-coded pools work only with operations that perform full object writes and appends (for example, RGW). ================ Appends do not work for me: $ ceph osd pool create test-ec erasure k5m2 pool 'test-ec' created $ echo -n 12345678 | rados --pool test-ec put testobj - $ rados --pool test-ec stat testobj test-ec/testobj mtime 2025-09-05T12:51:20.000000+0200, size 8 $ echo -n 12345678 | rados --pool test-ec put testobj - --offset 8 error putting test-ec/testobj: (95) Operation not supported $ echo -n 12345678 | rados --pool test-ec append testobj - error appending test-ec/testobj: (95) Operation not supported When I enable the allow_ec_overwrites flag, everything works as expected: $ ceph osd pool set test-ec allow_ec_overwrites true set pool 73 allow_ec_overwrites to true $ echo -n 12345678 | rados --pool test-ec put testobj - --offset 8 $ echo -n 12345678 | rados --pool test-ec append testobj - $ rados --pool test-ec stat testobj test-ec/testobj mtime 2025-09-05T12:54:30.000000+0200, size 24 Should the documentation be fixed somehow (maybe appending works only at certain block sizes or whatever), or am I doing something incorrect? My cluster is Ceph 19.2.2. Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

Re: snap-schedule not running
by Sophonet 04 Sep '25

04 Sep '25

Quick update / follow-up question: In the mgr logs I have seen that the schedule is indeed running, but snapshots cannot be created due to an error when cephfs.LibCephFS.mkdir ist called (Operation not permitted). Apparently, the scheduler is not using client.admin as ID (for the keyring when connecting to the target filesystem), since "ceph auth get client.admin" shows "caps mds = allow *". I have not found any documentation about setting the client.ID for a schedule. Any hints how the permission error can be avoided? Thanks, sophonet (I am posting this update for the third time now, hoping that the two earlier attempts that did not show up in the list are not just delayed...)

2 3

Ceph Scrubs
by Alex 03 Sep '25

03 Sep '25

Hi. We're being told to move the osd_scrub_*_hour from nightly to just the weekends. How horrible of an idea is it? Does Ceph keep statistics of how often it finds scrub errors? Can we tweak the osd_scrub_load_threshold ? Thanks.

1 0

squid 19.2.2 - drive in use but ceph is seeing it as "available"
by Steven Vacaroaia 03 Sep '25

03 Sep '25

Hi, I would appreciate some guidance here I am trying to reconfigure OSD to put DB/WAL on NVME so I "out" and "ceph osd rm" them ( because they were automatically configured and I did not "set-unamage" them properly ) then "wipefs -af " them Now CEPH is seeing drives available for deploying OSD on but it fails doing it due to "device or resource busy" Attempting to destroy and zap it fails due to "device or resource busy" even after I "wipefs -af /dev/sdaa" it this is what it is on it ( output of lsblk) sdaa 18.2T disk └─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm lvs,vgs does not find any logical disks /volume cephadm ceph-volume inventory is seeing the disks as "available = True" cephadm ceph-volume lvm list is NOT finding any LVM on /dev/sdaa docker ps -a does not show any "left over" OSD docker There is a reference ( block link) to the /dev/ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f016 in /var/lib/cephFSID/removed/osd.NUMER_DATE How can I 'clean up' the disk ? Many thanks Steven cephadm ceph-volume lvm zap --destroy /dev/sdaa Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 Not using image 'sha256:4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmp419ql4q7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm zap --destroy /dev/sdaa /usr/bin/docker: stderr --> Zapping: /dev/sdaa /usr/bin/docker: stderr --> Removing all BlueStore signature on /dev/sdaa if any... /usr/bin/docker: stderr Running command: /usr/bin/ceph-bluestore-tool zap-device --dev /dev/sdaa --yes-i-really-really-mean-it /usr/bin/docker: stderr stderr: error from zap: (16) Device or resource busy /usr/bin/docker: stderr 2025-09-02T17:41:02.385+0000 72def157f9c0 -1 bdev(0x57b37fe52000 /dev/sdaa) open open got: (16) Device or resource busy Some extra info cephadm ceph-volume inventory output Device Path Size Device nodes rotates available Model name /dev/nvme3n1 1.46 TB nvme3n1 False True MTFDKCC1T6TGQ-1BK1DABYY /dev/sdaa 18.19 TB sdaa True True ST20000NM007D-3D /dev/sdv 18.19 TB sdv True True ST20000NM007D-3D /dev/sdw 18.19 TB sdw True True ST20000NM007D-3D /dev/sdx 18.19 TB sdx True True ST20000NM007D-3D /dev/sdy 18.19 TB sdy True True ST20000NM007D-3D /dev/sdz 18.19 TB sdz True True ST20000NM007D-3D lsblk output NAME SIZE TYPE MOUNTPOINT sdaa 18.2T disk └─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169 18.2T lvm lvs output ( greped for lsblk /dev/sdaa and couldn't find it ) osd-block-29a5b1f5-d442-4756-a61d-e230832a3577 ceph-013dc6ae-90c7-4271-b308-895bbafe8840 -wi-ao---- <6.99t osd-block-5f7471f4-22ee-4cf9-9a16-2854a0c26496 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 -wi-ao---- <6.99t osd-block-ff6ca83d-80c5-4fff-a50d-399d94ba1b1b ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea -wi-ao---- 18.19t osd-block-fc022b7e-3d85-4435-998e-6513020bbdf3 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 -wi-ao---- 13.97t osd-block-347afb43-812c-4330-a795-060e76f33f89 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 -wi-ao---- <6.99t osd-block-c96901b7-8b7e-4a94-be0a-ab1316fd114e ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 -wi-ao---- <6.99t osd-block-5e069648-e0ec-4490-a3a9-e758d6e3e1d4 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 -wi-ao---- 13.97t osd-block-5d7dc85e-dcc8-4d87-b7ba-d61201dccd5e ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e -wi-ao---- <6.99t osd-block-d18a4abc-f723-4ac8-9d05-d02edd7e6018 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 -wi-ao---- <6.99t osd-block-33404537-6f45-4f10-af8a-8de491e8bcc9 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 -wi-ao---- 18.19t osd-block-153d1ed5-007c-439e-ad9e-c20d1880ff4d ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac -wi-ao---- 18.19t osd-block-15f14591-43c9-42ed-8410-d7f216019b55 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 -wi-ao---- 18.19t osd-block-44c310eb-2002-4817-8b2e-beacb0a70806 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 -wi-ao---- <6.99t osd-block-b55d7d9f-8e2a-4da5-b9c5-ace96975286d ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a -wi-ao---- 13.97t osd-block-a97d2620-d07a-4bd7-8f19-66801f5d613b ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 -wi-ao---- 18.19t osd-block-5c134f31-3a17-4fe9-989f-fb3a1dc2c0e9 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 -wi-ao---- <6.99t osd-block-89390a0d-6bc3-41a5-a79b-ebac922d540f ceph-d8a23938-be11-4cee-ba13-951996ca9328 -wi-ao---- <6.99t osd-block-cfabfe08-5e42-46ba-8412-1d264e9312d3 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 -wi-ao---- <6.99t osd-block-2e86866b-b537-4b29-86e7-490e627f82be ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e -wi-ao---- 18.19t osd-block-ab0b170e-91e0-41d4-a5ea-b5c4fef6c3d8 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 -wi-ao---- <6.99t osd-db-012f9187-1596-4db8-96ba-809a96bce8fd ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-124f39e2-f28d-47a8-acce-fb1364f7a638 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-17edf308-b36b-42be-8855-55833d35f983 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-55eb88b3-89a0-4931-a1be-d73e2aaf527f ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-5e03ba4b-5641-49c6-a477-1b37dffcd398 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-db-b22a1142-8dec-4035-96b1-7717c4587bcb ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g osd-block-f9c81b89-a903-4a11-9c2a-9171b4a5760a ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 -wi-ao---- <6.99t VGS output ( ( greped for lsblk /dev/sdaa result and couldn't find it ) VG #PV #LV #SN Attr VSize VFree ceph-013dc6ae-90c7-4271-b308-895bbafe8840 1 1 0 wz--n- <6.99t 0 ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 1 1 0 wz--n- <6.99t 0 ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea 1 1 0 wz--n- 18.19t 0 ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 1 1 0 wz--n- 13.97t 0 ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 1 1 0 wz--n- <6.99t 0 ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 1 1 0 wz--n- <6.99t 0 ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 1 1 0 wz--n- 13.97t 0 ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e 1 1 0 wz--n- <6.99t 0 ceph-6f7e544b-9ead-40c4-82ce-646272461b36 1 1 0 wz--n- <6.99t 0 ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 1 1 0 wz--n- 18.19t 0 ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac 1 1 0 wz--n- 18.19t 0 ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 1 1 0 wz--n- 18.19t 0 ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 1 1 0 wz--n- <6.99t 0 ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a 1 1 0 wz--n- 13.97t 0 ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 1 1 0 wz--n- 18.19t 0 ceph-c9e00bac-501b-4449-b23c-d914c61a6838 1 1 0 wz--n- <6.99t 0 ceph-d8a23938-be11-4cee-ba13-951996ca9328 1 1 0 wz--n- <6.99t 0 ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 1 1 0 wz--n- <6.99t 0 ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e 1 1 0 wz--n- 18.19t 0 ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 1 1 0 wz--n- <6.99t 0 ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 1 6 0 wz--n- <1.46t 8.00m ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 1 1 0 wz--n- <6.99t 0

2 2

squid 19.2.2 - added disk does not reflect in available capacity on pools
by Steven Vacaroaia 03 Sep '25

03 Sep '25

Hi, I have added 42 x 18TB HDD disks ( 6 on each of the 7 servers ) My expectation was that the pools configured to use "hdd_class" will have their capacity increased ( e.g. default.rgw.buckets.data which is uses an EC 4+2 pool for data ) It seems it is not happening ...yet ?! Is it because the peering is still going ? Many thanks Steven Below are outputs from ceph -s ceph df ceph osd df tree cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore 1 failed cephadm daemon(s) 569 pgs not deep-scrubbed in time 2609 pgs not scrubbed in time 39 daemons have recently crashed services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 2m) mgr: ceph-host-1.lqlece(active, since 18h), standbys: ceph-host-2.suiuxi mds: 19/19 daemons up, 7 standby osd: 181 osds: 181 up (since 4d), 181 in (since 14h); 2770 remapped pgs flags noautoscale data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 256.82M objects, 484 TiB usage: 742 TiB used, 1.5 PiB / 2.2 PiB avail pgs: 575889786/1468742421 objects misplaced (39.210%) 4247 active+clean 2763 active+remapped+backfill_wait 77 active+clean+scrubbing 43 active+clean+scrubbing+deep 7 active+remapped+backfilling io: client: 84 B/s rd, 2.8 KiB/s wr, 0 op/s rd, 1 op/s wr recovery: 384 MiB/s, 128 objects/s --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd_class 1.4 PiB 814 TiB 579 TiB 579 TiB 41.54 nvme_class 293 TiB 293 TiB 216 GiB 216 GiB 0.07 ssd_class 587 TiB 424 TiB 163 TiB 163 TiB 27.80 TOTAL 2.2 PiB 1.5 PiB 742 TiB 742 TiB 32.64 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 277 MiB 71 831 MiB 0 93 TiB .rgw.root 2 32 1.6 KiB 6 72 KiB 0 93 TiB default.rgw.log 3 32 63 KiB 210 972 KiB 0 93 TiB default.rgw.control 4 32 0 B 8 0 B 0 93 TiB default.rgw.meta 5 32 1.4 KiB 8 72 KiB 0 93 TiB default.rgw.buckets.data 6 2048 289 TiB 100.35M 434 TiB 68.04 136 TiB default.rgw.buckets.index 7 1024 5.4 GiB 521 16 GiB 0 93 TiB default.rgw.buckets.non-ec 8 32 551 B 1 13 KiB 0 93 TiB metadata_fs_ssd 9 128 6.1 GiB 15.69M 18 GiB 0 93 TiB ssd_ec_project 10 1024 108 TiB 44.46M 162 TiB 29.39 260 TiB metadata_fs_hdd 11 128 9.9 GiB 8.38M 30 GiB 0.01 93 TiB hdd_ec_archive 12 1024 90 TiB 87.94M 135 TiB 39.79 136 TiB metadata_fs_nvme 13 32 260 MiB 177 780 MiB 0 93 TiB metadata_fs_ssd_rep 14 32 17 MiB 103 51 MiB 0 93 TiB ssd_rep_projects 15 1024 132 B 1 12 KiB 0 130 TiB nvme_rep_projects 16 512 3.5 KiB 30 336 KiB 0 93 TiB ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2272.93311 - 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 1.00 - root default -7 254.54175 - 255 TiB 104 TiB 102 TiB 9.6 GiB 455 GiB 151 TiB 40.78 1.25 - host ceph-host-1 107 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 4 KiB 48 GiB 5.2 TiB 71.71 2.20 422 up osd.107 110 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 4 KiB 45 GiB 5.3 TiB 71.30 2.18 421 up osd.110 116 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 4 KiB 43 GiB 5.3 TiB 71.50 2.19 424 up osd.116 123 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 14 KiB 44 GiB 5.5 TiB 70.20 2.15 414 up osd.123 130 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 56 GiB 4.2 TiB 76.99 2.36 451 up osd.130 137 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 50 GiB 4.3 TiB 76.47 2.34 449 up osd.137 152 hdd_class 18.19040 1.00000 18 TiB 241 GiB 239 GiB 10 KiB 1.8 GiB 18 TiB 1.29 0.04 7 up osd.152 2 nvme_class 13.97260 1.00000 14 TiB 14 GiB 906 MiB 3.9 GiB 8.9 GiB 14 TiB 0.10 0.00 300 up osd.2 8 nvme_class 13.97260 1.00000 14 TiB 7.2 GiB 846 MiB 2.5 GiB 3.9 GiB 14 TiB 0.05 0.00 293 up osd.8 15 nvme_class 13.97260 1.00000 14 TiB 11 GiB 887 MiB 3.3 GiB 6.9 GiB 14 TiB 0.08 0.00 288 up osd.15 23 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 6 KiB 14 GiB 5.2 TiB 26.28 0.81 106 up osd.23 30 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.4 GiB 5.0 TiB 27.73 0.85 114 up osd.30 37 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 6 KiB 15 GiB 5.2 TiB 26.28 0.80 104 up osd.37 44 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.3 GiB 5.1 TiB 27.34 0.84 111 up osd.44 52 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 4 KiB 15 GiB 5.1 TiB 26.35 0.81 104 up osd.52 59 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.79 0.85 107 up osd.59 66 ssd_class 6.98630 1.00000 7.0 TiB 1.8 TiB 1.8 TiB 4 KiB 14 GiB 5.1 TiB 26.29 0.81 104 up osd.66 72 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.8 TiB 4 KiB 15 GiB 5.1 TiB 26.68 0.82 107 up osd.72 79 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.1 TiB 27.05 0.83 111 up osd.79 86 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 4 KiB 10 GiB 5.0 TiB 28.49 0.87 116 up osd.86 93 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 5.5 GiB 5.1 TiB 27.29 0.84 112 up osd.93 100 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.19 0.86 114 up osd.100 -15 345.49374 - 345 TiB 106 TiB 104 TiB 8.7 GiB 523 GiB 239 TiB 30.69 0.94 - host ceph-host-2 115 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 7 KiB 43 GiB 5.9 TiB 68.17 2.09 402 up osd.115 122 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 2 KiB 45 GiB 5.4 TiB 70.72 2.17 420 up osd.122 134 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 12 KiB 52 GiB 4.7 TiB 74.54 2.28 440 up osd.134 141 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 54 GiB 4.3 TiB 76.58 2.35 447 up osd.141 144 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 4 KiB 56 GiB 4.5 TiB 75.77 2.32 443 up osd.144 146 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 7 KiB 52 GiB 4.4 TiB 76.13 2.33 449 up osd.146 159 hdd_class 18.19040 1.00000 18 TiB 203 GiB 200 GiB 10 KiB 2.4 GiB 18 TiB 1.09 0.03 6 up osd.159 176 hdd_class 18.19040 1.00000 18 TiB 135 GiB 133 GiB 6 KiB 1.4 GiB 18 TiB 0.72 0.02 4 up osd.176 177 hdd_class 18.19040 1.00000 18 TiB 74 GiB 73 GiB 6 KiB 1.7 GiB 18 TiB 0.40 0.01 2 up osd.177 178 hdd_class 18.19040 1.00000 18 TiB 238 GiB 236 GiB 7 KiB 2.3 GiB 18 TiB 1.28 0.04 5 up osd.178 179 hdd_class 18.19040 1.00000 18 TiB 76 GiB 75 GiB 6 KiB 828 MiB 18 TiB 0.41 0.01 2 up osd.179 180 hdd_class 18.19040 1.00000 18 TiB 198 GiB 197 GiB 10 KiB 1.7 GiB 18 TiB 1.07 0.03 5 up osd.180 6 nvme_class 13.97260 1.00000 14 TiB 11 GiB 903 MiB 2.9 GiB 7.0 GiB 14 TiB 0.08 0.00 282 up osd.6 13 nvme_class 13.97260 1.00000 14 TiB 9.4 GiB 1.1 GiB 2.7 GiB 5.6 GiB 14 TiB 0.07 0.00 300 up osd.13 20 nvme_class 13.97260 1.00000 14 TiB 11 GiB 862 MiB 3.1 GiB 7.2 GiB 14 TiB 0.08 0.00 288 up osd.20 26 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.58 0.88 108 up osd.26 32 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.21 0.86 107 up osd.32 41 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.23 0.86 113 up osd.41 48 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 11 KiB 16 GiB 5.0 TiB 28.19 0.86 114 up osd.48 53 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 15 GiB 5.0 TiB 28.21 0.86 107 up osd.53 61 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.22 0.86 112 up osd.61 68 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.19 0.86 107 up osd.68 74 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 9 KiB 16 GiB 5.0 TiB 28.56 0.88 114 up osd.74 82 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.79 0.85 106 up osd.82 89 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.85 0.85 108 up osd.89 95 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 11 KiB 16 GiB 5.0 TiB 28.58 0.88 112 up osd.95 104 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 9 KiB 16 GiB 5.0 TiB 27.83 0.85 108 up osd.104 -13 345.49374 - 345 TiB 108 TiB 106 TiB 9.5 GiB 423 GiB 238 TiB 31.16 0.95 - host ceph-host-3 114 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 4 KiB 42 GiB 6.3 TiB 66.07 2.02 389 up osd.114 121 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 9 KiB 44 GiB 5.9 TiB 67.77 2.08 400 up osd.121 127 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 14 KiB 48 GiB 4.7 TiB 74.32 2.28 437 up osd.127 133 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 54 GiB 4.0 TiB 78.33 2.40 461 up osd.133 140 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 2 KiB 50 GiB 4.4 TiB 76.20 2.33 447 up osd.140 143 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 50 GiB 4.4 TiB 76.39 2.34 446 up osd.143 154 hdd_class 18.19040 1.00000 18 TiB 281 GiB 279 GiB 9 KiB 2.3 GiB 18 TiB 1.51 0.05 8 up osd.154 158 hdd_class 18.19040 1.00000 18 TiB 725 GiB 722 GiB 20 KiB 3.0 GiB 17 TiB 3.89 0.12 22 up osd.158 163 hdd_class 18.19040 1.00000 18 TiB 696 GiB 692 GiB 22 KiB 3.2 GiB 18 TiB 3.73 0.11 22 up osd.163 167 hdd_class 18.19040 1.00000 18 TiB 458 GiB 455 GiB 15 KiB 2.4 GiB 18 TiB 2.46 0.08 11 up osd.167 171 hdd_class 18.19040 1.00000 18 TiB 788 GiB 785 GiB 25 KiB 3.1 GiB 17 TiB 4.23 0.13 23 up osd.171 175 hdd_class 18.19040 1.00000 18 TiB 518 GiB 515 GiB 22 KiB 3.0 GiB 18 TiB 2.78 0.09 15 up osd.175 4 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 1.1 GiB 2.8 GiB 5.2 GiB 14 TiB 0.06 0.00 283 up osd.4 11 nvme_class 13.97260 1.00000 14 TiB 10 GiB 848 MiB 3.0 GiB 6.4 GiB 14 TiB 0.07 0.00 289 up osd.11 19 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 922 MiB 3.6 GiB 4.6 GiB 14 TiB 0.06 0.00 289 up osd.19 25 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.8 GiB 5.0 TiB 28.42 0.87 113 up osd.25 31 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.05 0.86 110 up osd.31 38 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.1 GiB 5.1 TiB 27.34 0.84 110 up osd.38 45 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.1 GiB 5.0 TiB 27.74 0.85 108 up osd.45 51 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 14 GiB 5.0 TiB 28.16 0.86 111 up osd.51 58 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 12 GiB 5.1 TiB 27.03 0.83 105 up osd.58 65 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.04 0.83 109 up osd.65 73 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.1 TiB 27.03 0.83 109 up osd.73 80 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.9 GiB 5.0 TiB 28.41 0.87 110 up osd.80 87 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.8 GiB 5.0 TiB 28.42 0.87 114 up osd.87 94 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.10 0.86 114 up osd.94 101 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.2 GiB 5.0 TiB 28.07 0.86 113 up osd.101 -11 345.49374 - 345 TiB 107 TiB 105 TiB 8.2 GiB 439 GiB 239 TiB 30.95 0.95 - host ceph-host-4 108 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 4 KiB 43 GiB 6.1 TiB 67.05 2.05 394 up osd.108 113 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 7 KiB 47 GiB 6.3 TiB 65.92 2.02 387 up osd.113 120 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 2 KiB 40 GiB 6.2 TiB 66.60 2.04 389 up osd.120 126 hdd_class 18.43300 0.95001 18 TiB 15 TiB 15 TiB 7 KiB 58 GiB 3.6 TiB 80.37 2.46 473 up osd.126 132 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 6 KiB 52 GiB 4.1 TiB 78.00 2.39 457 up osd.132 139 hdd_class 18.43300 1.00000 18 TiB 15 TiB 14 TiB 14 KiB 53 GiB 3.9 TiB 78.87 2.42 463 up osd.139 151 hdd_class 18.19040 1.00000 18 TiB 38 GiB 37 GiB 6 KiB 1.1 GiB 18 TiB 0.20 0.01 1 up osd.151 156 hdd_class 18.19040 1.00000 18 TiB 205 GiB 203 GiB 12 KiB 1.9 GiB 18 TiB 1.10 0.03 6 up osd.156 161 hdd_class 18.19040 1.00000 18 TiB 853 GiB 849 GiB 4 KiB 3.9 GiB 17 TiB 4.58 0.14 26 up osd.161 165 hdd_class 18.19040 1.00000 18 TiB 569 GiB 566 GiB 22 KiB 2.7 GiB 18 TiB 3.05 0.09 15 up osd.165 169 hdd_class 18.19040 1.00000 18 TiB 891 GiB 888 GiB 24 KiB 3.3 GiB 17 TiB 4.78 0.15 25 up osd.169 173 hdd_class 18.19040 1.00000 18 TiB 591 GiB 589 GiB 24 KiB 2.6 GiB 18 TiB 3.17 0.10 19 up osd.173 3 nvme_class 13.97260 1.00000 14 TiB 10 GiB 929 MiB 2.8 GiB 6.3 GiB 14 TiB 0.07 0.00 283 up osd.3 10 nvme_class 13.97260 1.00000 14 TiB 8.7 GiB 872 MiB 2.6 GiB 5.3 GiB 14 TiB 0.06 0.00 271 up osd.10 17 nvme_class 13.97260 1.00000 14 TiB 9.5 GiB 904 MiB 2.8 GiB 5.8 GiB 14 TiB 0.07 0.00 298 up osd.17 24 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.3 GiB 5.0 TiB 28.11 0.86 112 up osd.24 33 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.41 0.87 113 up osd.33 40 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.8 GiB 5.0 TiB 28.44 0.87 115 up osd.40 47 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 14 GiB 5.1 TiB 27.03 0.83 106 up osd.47 55 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.22 0.86 108 up osd.55 60 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.06 0.83 111 up osd.60 67 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.3 GiB 5.1 TiB 27.28 0.84 110 up osd.67 75 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 109 up osd.75 83 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 16 GiB 5.1 TiB 27.05 0.83 105 up osd.83 88 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.0 GiB 5.0 TiB 28.42 0.87 109 up osd.88 98 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 4.7 GiB 5.0 TiB 28.41 0.87 114 up osd.98 103 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.7 GiB 5.0 TiB 28.07 0.86 107 up osd.103 -3 290.92255 - 291 TiB 104 TiB 103 TiB 8.7 GiB 452 GiB 187 TiB 35.81 1.10 - host ceph-host-5 105 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 53 GiB 5.3 TiB 71.10 2.18 417 up osd.105 109 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 7 KiB 47 GiB 5.1 TiB 72.56 2.22 428 up osd.109 117 hdd_class 18.43300 0.95001 18 TiB 13 TiB 13 TiB 6 KiB 47 GiB 5.3 TiB 71.49 2.19 422 up osd.117 124 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 52 GiB 5.2 TiB 71.87 2.20 424 up osd.124 129 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 2 KiB 50 GiB 4.4 TiB 75.91 2.33 449 up osd.129 136 hdd_class 18.43300 1.00000 18 TiB 14 TiB 13 TiB 11 KiB 49 GiB 4.9 TiB 73.30 2.25 432 up osd.136 181 hdd_class 18.19040 1.00000 18 TiB 109 GiB 108 GiB 7 KiB 846 MiB 18 TiB 0.59 0.02 3 up osd.181 182 hdd_class 18.19040 1.00000 18 TiB 356 GiB 354 GiB 12 KiB 2.6 GiB 18 TiB 1.91 0.06 11 up osd.182 183 hdd_class 18.19040 1.00000 18 TiB 155 GiB 154 GiB 10 KiB 1.8 GiB 18 TiB 0.83 0.03 5 up osd.183 1 nvme_class 13.97260 1.00000 14 TiB 12 GiB 875 MiB 3.2 GiB 7.7 GiB 14 TiB 0.08 0.00 290 up osd.1 7 nvme_class 13.97260 1.00000 14 TiB 7.4 GiB 1.2 GiB 2.6 GiB 3.7 GiB 14 TiB 0.05 0.00 285 up osd.7 14 nvme_class 13.97260 1.00000 14 TiB 11 GiB 837 MiB 3.0 GiB 7.6 GiB 14 TiB 0.08 0.00 290 up osd.14 21 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.2 GiB 5.1 TiB 27.24 0.83 106 up osd.21 28 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.1 TiB 27.08 0.83 105 up osd.28 35 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.05 0.83 104 up osd.35 42 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.9 GiB 5.0 TiB 27.72 0.85 107 up osd.42 49 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 15 GiB 5.0 TiB 28.19 0.86 112 up osd.49 56 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.2 GiB 5.0 TiB 28.10 0.86 107 up osd.56 63 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.10 0.86 111 up osd.63 70 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.4 GiB 5.0 TiB 28.01 0.86 112 up osd.70 77 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 109 up osd.77 84 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 104 up osd.84 90 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.42 0.87 114 up osd.90 96 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 15 GiB 5.1 TiB 27.06 0.83 104 up osd.96 -9 345.49374 - 345 TiB 107 TiB 106 TiB 9.7 GiB 446 GiB 238 TiB 31.02 0.95 - host ceph-host-6 111 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 6 KiB 45 GiB 6.4 TiB 65.30 2.00 384 up osd.111 118 hdd_class 18.43300 1.00000 18 TiB 12 TiB 12 TiB 11 KiB 52 GiB 6.0 TiB 67.51 2.07 397 up osd.118 128 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 12 KiB 50 GiB 4.4 TiB 75.96 2.33 445 up osd.128 135 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 48 GiB 4.7 TiB 74.68 2.29 440 up osd.135 142 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 56 GiB 4.1 TiB 77.54 2.38 454 up osd.142 145 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 55 GiB 4.4 TiB 75.98 2.33 445 up osd.145 153 hdd_class 18.19040 1.00000 18 TiB 169 GiB 167 GiB 7 KiB 1.7 GiB 18 TiB 0.90 0.03 5 up osd.153 157 hdd_class 18.19040 1.00000 18 TiB 488 GiB 486 GiB 14 KiB 2.7 GiB 18 TiB 2.62 0.08 15 up osd.157 162 hdd_class 18.19040 1.00000 18 TiB 440 GiB 437 GiB 17 KiB 2.6 GiB 18 TiB 2.36 0.07 11 up osd.162 166 hdd_class 18.19040 1.00000 18 TiB 852 GiB 849 GiB 24 KiB 3.3 GiB 17 TiB 4.57 0.14 25 up osd.166 170 hdd_class 18.19040 1.00000 18 TiB 473 GiB 470 GiB 22 KiB 2.5 GiB 18 TiB 2.54 0.08 14 up osd.170 174 hdd_class 18.19040 1.00000 18 TiB 673 GiB 670 GiB 20 KiB 3.0 GiB 18 TiB 3.61 0.11 21 up osd.174 5 nvme_class 13.97260 1.00000 14 TiB 9.1 GiB 846 MiB 2.9 GiB 5.3 GiB 14 TiB 0.06 0.00 276 up osd.5 12 nvme_class 13.97260 1.00000 14 TiB 12 GiB 873 MiB 3.5 GiB 7.8 GiB 14 TiB 0.08 0.00 296 up osd.12 18 nvme_class 13.97260 1.00000 14 TiB 15 GiB 857 MiB 3.3 GiB 11 GiB 14 TiB 0.10 0.00 288 up osd.18 27 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.5 GiB 5.0 TiB 28.44 0.87 114 up osd.27 34 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.11 0.86 113 up osd.34 39 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.09 0.86 108 up osd.39 46 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.5 GiB 5.1 TiB 27.65 0.85 107 up osd.46 54 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 8.0 GiB 5.1 TiB 27.70 0.85 107 up osd.54 62 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.5 GiB 5.0 TiB 28.49 0.87 109 up osd.62 69 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 8.2 GiB 5.0 TiB 28.08 0.86 113 up osd.69 76 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.78 0.85 111 up osd.76 81 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.1 GiB 5.0 TiB 28.46 0.87 115 up osd.81 91 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 15 GiB 5.0 TiB 27.82 0.85 108 up osd.91 97 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 7 KiB 5.0 GiB 5.0 TiB 28.47 0.87 109 up osd.97 102 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.11 0.86 108 up osd.102 -5 345.49374 - 345 TiB 106 TiB 105 TiB 8.8 GiB 435 GiB 239 TiB 30.73 0.94 - host ceph-host-7 106 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 2 KiB 47 GiB 5.8 TiB 68.70 2.10 406 up osd.106 112 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 6 KiB 51 GiB 5.8 TiB 68.47 2.10 403 up osd.112 119 hdd_class 18.43300 1.00000 18 TiB 13 TiB 13 TiB 11 KiB 54 GiB 5.7 TiB 69.31 2.12 407 up osd.119 125 hdd_class 18.43300 1.00000 18 TiB 13 TiB 12 TiB 4 KiB 45 GiB 5.7 TiB 68.98 2.11 410 up osd.125 131 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 11 KiB 57 GiB 4.2 TiB 77.12 2.36 454 up osd.131 138 hdd_class 18.43300 1.00000 18 TiB 14 TiB 14 TiB 9 KiB 58 GiB 4.1 TiB 77.85 2.38 456 up osd.138 150 hdd_class 18.19040 1.00000 18 TiB 133 GiB 131 GiB 7 KiB 1.5 GiB 18 TiB 0.71 0.02 4 up osd.150 155 hdd_class 18.19040 1.00000 18 TiB 48 GiB 46 GiB 9 KiB 2.0 GiB 18 TiB 0.26 0.01 2 up osd.155 160 hdd_class 18.19040 1.00000 18 TiB 873 GiB 869 GiB 4 KiB 3.6 GiB 17 TiB 4.69 0.14 27 up osd.160 164 hdd_class 18.19040 1.00000 18 TiB 850 GiB 847 GiB 4 KiB 3.2 GiB 17 TiB 4.57 0.14 25 up osd.164 168 hdd_class 18.19040 1.00000 18 TiB 696 GiB 693 GiB 19 KiB 3.2 GiB 18 TiB 3.74 0.11 17 up osd.168 172 hdd_class 18.19040 1.00000 18 TiB 852 GiB 848 GiB 2 KiB 3.7 GiB 17 TiB 4.57 0.14 25 up osd.172 0 nvme_class 13.97260 1.00000 14 TiB 10 GiB 858 MiB 2.8 GiB 6.3 GiB 14 TiB 0.07 0.00 293 up osd.0 9 nvme_class 13.97260 1.00000 14 TiB 9.3 GiB 847 MiB 2.9 GiB 5.6 GiB 14 TiB 0.07 0.00 281 up osd.9 16 nvme_class 13.97260 1.00000 14 TiB 10 GiB 897 MiB 3.1 GiB 6.3 GiB 14 TiB 0.07 0.00 288 up osd.16 22 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.2 GiB 5.1 TiB 27.28 0.84 107 up osd.22 29 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.42 0.87 112 up osd.29 36 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.3 GiB 5.0 TiB 28.08 0.86 109 up osd.36 43 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.42 0.87 115 up osd.43 50 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 5.6 GiB 5.1 TiB 27.28 0.84 111 up osd.50 57 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 5.3 GiB 5.1 TiB 27.32 0.84 112 up osd.57 64 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 4.9 GiB 5.0 TiB 28.45 0.87 113 up osd.64 71 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.4 GiB 5.0 TiB 28.46 0.87 109 up osd.71 78 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 4 KiB 14 GiB 5.1 TiB 27.08 0.83 105 up osd.78 85 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 8.9 GiB 5.0 TiB 28.50 0.87 114 up osd.85 92 ssd_class 6.98630 1.00000 7.0 TiB 2.0 TiB 2.0 TiB 6 KiB 5.1 GiB 5.0 TiB 28.41 0.87 109 up osd.92 99 ssd_class 6.98630 1.00000 7.0 TiB 1.9 TiB 1.9 TiB 6 KiB 12 GiB 5.0 TiB 27.77 0.85 112 up osd.99 TOTAL 2.2 PiB 742 TiB 731 TiB 63 GiB 3.1 TiB 1.5 PiB 32.64 MIN/MAX VAR: 0.00/2.46 STDDEV: 26.17

2 6

squid 19.2.2 cephadm - adding more placement hosts using osd spec yml file
by Steven Vacaroaia 02 Sep '25

02 Sep '25

Hi, I am using cephadm and deployed all my OSDs using a spec file I just noticed that osd_ssds has only one placement host instead of all 7 How do I add the other placement hosts ? Re running spec file with "hosts = *" and --dry-run does not indicated it will do anything Many thanks Steven This is the spec file # SSD OSDs service_type: osd service_id: ssd_osds placement: host_pattern: "*" crush_device_class: ssd_class spec: data_devices: rotational: 0 size: '6T:7T' The relevant part of ceph orch ls osd.all-available-devices 4 8m ago 16h * osd.hdd_osds 74 8m ago 16h * osd.nvme_osds 25 8m ago 5w * osd.ssd_osds 84 8m ago 3w ceph-host-1 The exported part of osd.ssd_osds --- service_type: osd service_id: ssd_osds service_name: osd.ssd_osds placement: hosts: - ceph-host-1 spec: crush_device_class: ssd_class data_devices: rotational: 0 size: 6T:7T filter_logic: AND objectstore: bluestore The result of applying the osd-ssd-config.yml ( with --dry-run) WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any of these conditions change, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+-------------+ +---------+------+--------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+

2 3

Librados async operations in C
by Jan Kasprzak 02 Sep '25

02 Sep '25

Hello, Ceph users, TL;DR: how to integrate librados async operations with an event-loop library? Details: I need to store lots of objects, but I don't need a full-featured S3 API, and my objects are mostly small (majority is under 10 KB, see below), so a per-object overhead is significant for me. If I understand it correctly (do I?) radosgw stores each S3 object in its own RADOS object, so it is not suitable for my purpose -- packing multiple source objects to a larger RADOS object is a necessity. I am considering writing my own frontend with RADOS pool[s] as a storage backend, using librados (or maybe libradosstriper for large objects), EC pool for data, and a replicated pool for OMAP object metadata. Librados supports async I/O with callbacks, but I am not sure how does it fit into a wider environment of other asynchronous operations in the same process: - is librados thread-safe, or should I use, for example, a per-thread rados_ioctx_t or even rados_t instance? Should I do some kind of locking when calling librados functions from a multi-threaded program? - is it possible to incorporate librados to an event-driven program, using I/O event loop library (libevent, glib, ...)? If so, how can I make librados to wait for socket I/O using the global event loop? - is it still recommended to have at most 100K OMAP keys per RADOS object? I have created an empty object and set about 120K OMAP keys on it using rados(1) command, and so far the cluster is HEALTH_OK. - what is the preferred RADOS object size (for packing many small source objects into a single RADOS object)? =============================== FWIW, one of the instances of my data has the following size and distribution of object sizes: Total: 210M objs, 114 TB total size 0B-1KB: 21M objs, 18 GB total size 1KB-10KB: 137M objs, 344 GB total size 10KB-100KB: 31M objs, 1 TB total size 100KB-1MB: 15M objs, 5 TB total size 1MB-10MB: 4M objs, 15 TB total size 10MB-100MB: 966K objs, 25 TB total size 100MB-1GB: 137K objs, 38 TB total size 1GB-10GB: 13K objs, 26 TB total size 10GB-100GB: 162 objs, 3 TB total size I have a not-so-big Ceph cluster (3 mons, 15-ish OSD nodes, each with two HDD-based OSDs with metadata/OMAP on NVMe). Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

1 0

Grafana: Ceph-Cluster Advanced "in" OSDs not counted correctly
by Eugen Block 02 Sep '25

02 Sep '25

Hi *, on a Squid (19.2.2) cluster with Grafana version 9.4.12 the "OSDs" panel on the "Ceph Cluster - Advanced" page shows some inconsistency, not sure if the attached screenshot will make it to the list. We had to set an OSD out, waiting for the customer to make a decision about it. There are 126 OSDs in total, with one "out" the number of "in" OSDs should be 125, but it's still 126 although one OSD is shown as "out". I'm not going to comment on the colours (in are red, out is green). Prometheus provides the correct status for the out OSD (0.0), but the expression is: "alias": "In", ... "expr": "count(ceph_osd_in{cluster=~\"$cluster\", })", which is just counting the entries, so it will always be the total number of OSDs. I guess one could just adapt the expression for the "out" OSDs: "alias": "Out", ... "expr": "sum(ceph_osd_in{cluster=~\"$cluster\", } == bool 1)", I haven't checked tracker yet, is this already known? And when this is targeted, maybe fix the colours as well. To me it looks like it's still wrong in the latest version of ceph-mixin [0]. Regards, Eugen [0] https://github.com/ceph/ceph/blob/346846543c6bfc93a360476c739580cd2344fec0/…

1 1

How to remove failed OSD & reuse it?
by lejeczek 02 Sep '25

02 Sep '25

Hi guys. I've browsing through the net in a search of a relatively clear "howto" but I failed to find one. It's rather many, sometimes different notes/thoughts on how to deal with such/similar situation. Having a 3-node containerized cluster which lost osd - it crushed, there is nothing wrong with the node, nothing wrong with the disk, but never mind that. Is there a howto which covers containerized environment? One example I followed is: https://docs.redhat.com/en/documentation/red_hat_ceph_storage/1.2.3/html/re… but it is - to me - clear, what to do with "broken" containers. I'm got to: -> $ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.68359 root default -3 0 host podster1 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 up 1.00000 1.00000 4 hdd 0.29300 osd.4 up 1.00000 1.00000 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 up 1.00000 1.00000 5 hdd 0.29300 osd.5 up 1.00000 1.00000 yet: -> $ ceph orch ps --daemon-type=osd NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.0 podster1.mine.priv error 7m ago 3w - 4096M <unknown> <unknown> <unknown> osd.1 podster3.mine.priv running (25h) 7m ago 3w 942M 4096M 19.2.3 aade1b12b8e6 d71051ea79dc osd.2 podster2.mine.priv running (6d) 7m ago 3w 1192M 4096M 19.2.3 aade1b12b8e6 e8d05142a73a osd.3 podster1.mine.priv error 7m ago 2w - 4096M <unknown> <unknown> <unknown> osd.4 podster2.mine.priv running (6d) 7m ago 2w 3293M 4096M 19.2.3 aade1b12b8e6 6116277f69d1 osd.5 podster3.mine.priv running (25h) 7m ago 2w 2963M 4096M 19.2.3 aade1b12b8e6 d671bf73cc01 what would be next bits needed to complete such removal&reuse/re-create of osd(s)? p.s. This a 'lab' setup so I'm not worried, but it'd be great to complete this process in a healthy manner. many thanks, L.

2 7