- ceph-users - lists.ceph.io

CephFS maturity on Squid?
by William David Edwards 18 Sep '25

18 Sep '25

Hi, For those running CephFS on Squid: how mature is the release in your experience? I'm especially interested in those using CephFS in a shared hosting scenario, with many directories containing many small files. I'm extra wary of new releases nowadays, as we've ran into several production-impacting Ceph bugs in the past months post-upgrade, across clusters. (Most notably, OSDs failing causing the MDS to OOM (17.2.7 -> 17.2.8) [1], and major performance issues after upgrading to Reef caused by the aligned 64k blocks constraint [2] (RBD though, not CephFS).) With kind regards, William David Edwards [1]: https://tracker.ceph.com/issues/69764 [2]: https://github.com/ceph/ceph/pull/54772

4 4

Is there a faster way to merge PGs?
by Justin Mammarella 18 Sep '25

18 Sep '25

Hi folks, We are in the process of shrinking a CEPH cluster from around 2000 osds to 300 osds. We need to reduce our pg count from 16384 to 8192. After setting pg_num to 8192, it’s currently it’s taking 3 hours to reduce the pool’s PG count by one. Which given the number of PGs, will take over 2 years to complete. Are there any tunables to increase the speed of pg merging? Our alternative is to create a new pool and transfer the data Regards, Justin

2 3

cephfs volume mount error using both ceph-csi and mount.ceph (using -o mon_addr)
by John Jasen 17 Sep '25

17 Sep '25

So, when I try to get a container to mount via cephfs, it uses these options: "-t ceph cephfs@60eb7a9e-8cc6-11f0-a7d0-e43d1a153e91.k8cephfs=/volumes/k8s/csi-vol-c76f1825-df8e-477e-981a-bf5fccb66fe3/fbed6ce0-4446-449c-9a14-8c7b297499a2 /var/lib/kubelet/plugins/ kubernetes.io/csi/cephfs.csi.ceph.com/153624a7bdfffe5dfdec7dfd5c866889b7a54… -o mon_addr=my.ip.addr.1:3300/my.ip.addr.5:3300/my.ip.addr.6:3300/my.ip.addr.7:3300/my.ipaddr.11:3300,secretfile=/tmp/csi/keys/keyfile-4057439945,_netdev" And it fails: stderr: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized When I feed these options into mount.ceph, it fails, with the same errors. When I feed each of my.ip.addr into mount.ceph as mon_addr, I get similar failures. When I remove mon_addr options totally from the mount.ceph command, forcing it to work through /etc/ceph/ceph.conf for finding monitors -- it succeeds. ceph.conf is a minimal one built from ceph config generate-minimal-conf, which reads: "# minimal ceph.conf for 60eb7a9e-8cc6-11f0-a7d0-e43d1a153e91 [global] fsid = 60eb7a9e-8cc6-11f0-a7d0-e43d1a153e91 mon_host = [v2:my.ip.addr.11:3300/0,v1:my.ip.addr.11:6789/0] [v2:my.ip.addr.1:3300/0,v1:my.ip.addr.1:6789/0] v2:my.ip.addr.5:3300/0 [v2:my.ip.addr.6:3300/0,v1:my.ip.addr.6:6789/0] [v2:my.ip.addr.7:3300/0,v1:172.17. 10.7:6789/0]" Any idea what's going on?

1 1

Async dirops maturity
by William David Edwards 17 Sep '25

17 Sep '25

Hi, Async dirops were enabled in the kernel by default in 2021 [1]. In 2023, a patch was made to revert that default because of MDS crashes [2]: > We have identified an issue in the MDS affecting CephFS users using > the kernel driver. The issue was first introduced in the octopus > release that added support for clients to perform asynchronous > directory operations using the `nowsync` mount option. The issue > presents itself as an MDS crash ... but the conclusion was then drawn that `mds_client_delegate_inos_pct` can be used instead to disable async dirops, so the patch didn't go through [3]: > After some digging and talking to Jeff, I figured that it's possible > to disable async dirops from the mds side by setting > `mds_client_delegate_inos_pct` config to 0: [...] > So, I guess this patch is really not required. We can suggest this > config update to users and document it for now. We lack tests with > this config disabled, so I'll be adding the same before recommending > it out. Will keep you posted. ... and in 19.2.0 (2024), `mds_client_delegate_inos_pct` defaults to 0, which disables async dirops [4]: > CephFS: Disallow delegating preallocated inode ranges to clients. > Config mds_client_delegate_inos_pct defaults to 0 which disables async > dirops in the kclient. Aforementioned 'MDS crash' is said to be caused by bugs 61009[5] and 58489[6]. Neither have been fixed, and it's not even clear to me how the latter relates to async dirops / preallocated inodes. All in all: what is the state of async dirops? Is it anticipated to be mature in the near future? Is it actively being worked on? If not, what do other users (especially in a shared hosting scenario, with many directories with many small files) use to speed up `rm -rf`s without having every unlink() take a round-trip to the MDS? With kind regards, William David Edwards [1]: https://github.com/torvalds/linux/commit/f7a67b463fb83a4b9b11ceaa8ec4950b8f… [2]: https://www.spinics.net/lists/ceph-devel/msg58389.html [3]: https://www.spinics.net/lists/ceph-devel/msg58394.html [4]: https://ceph.io/en/news/blog/2024/v19-2-0-squid-released/ [5]: https://tracker.ceph.com/issues/61009 [6]: https://tracker.ceph.com/issues/58489

1 0

Subject : Account-Level API Support in Ceph RGW for Production Use
by Dhivya G 17 Sep '25

17 Sep '25

Dear Ceph Community, I hope this message finds you well. We are currently working on integrating Ceph RGW into our system and have encountered a challenge regarding account-level management via API. At present, we are using radosgw-admin commands programmatically within our code to manage and retrieve account-level information. While this approach works, we understand that relying on radosgw-admin is not recommended for production environments due to its direct interaction with the backend and potential impact on performance and stability. We have explored the available documentation and interfaces but could not find a dedicated Account-level API (e.g., for managing users, retrieving account metadata, etc.) that can be safely used in production over RESTful protocols (such as S3 or Swift-compatible APIs). We are seeking clarification on the following points: Is there an officially supported Account API in Ceph RGW that allows account-level operations via REST or other production-safe interfaces? If not, is there a roadmap or recommended alternative for performing these actions in a production-ready manner? Are there any best practices or tools from the community that can bridge this gap? We would greatly appreciate any guidance, documentation, or recommendations you can share . Best regards Dhivya G https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

3 2

[ceph-user]ceph-ansible || Ceph upgrade from octopus to quincy
by Danish Khan 15 Sep '25

15 Sep '25

Dear Team, We are upgrading ceph-ansible cluster from octopus to quincy, we have upgraded few of our cluster successfully using below document: https://docs.ceph.com/projects/ceph-ansible/en/latest/day-2/upgrade.html But Ceph clusters with multisite enabled are not getting upgraded. due to limitation and complexity of ceph-ansible. Does anyone one any idea how we can upgrade ceph-ansible clusters with multi-site. I was planning to break the multi-site for a few days and upgrade the cluster and enable multisite again. but I don't have any smooth steps to do it. Can someone please help me on this. Regards, Danish

1 0

Which Ceph RPMs on AlmaLinux 10?
by Jan Kasprzak 15 Sep '25

15 Sep '25

Hi, Ceph users, does anybody use Ceph on AlmaLinux10 (RHEL 10)? If so, which RPMs? I tried to rebuild 19.2.3 el9 src.rpm from download.ceph.com, and got the following problems: - there is no pathfix.py in python3-devel in AlmaLinux 10 - src/common/openssl_opts_handler.cc needs #include <openssl/err.h> - the same file (openssl_opts_handler.cc) calls ENGINE_load_builtin_engines() which does not exist in Alma10 openssl anymore Which Ceph RPMs do you use on AlmaLinux 10? I am about to deploy a new Ceph cluster, and I would rather use the newest enterprise distro (Alma10). Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

capabilities needed for subvolume management?
by Robert Sander 12 Sep '25

12 Sep '25

Hi, what capabilities does a cephx key need to manage subvolumes in a filesystem? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

v20.1.0 Tentacle RC0 released
by Yuri Weinstein 12 Sep '25

12 Sep '25

This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0 Feature highlights: * RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4

5 10