- ceph-users - lists.ceph.io

Override nfs parameter in NFS_CORE_PARAM
by Salatiel Filho 04 Jan '26

04 Jan '26

Hello , everyone. Apparently there is something wrong in the default config for ganesha nfs in tentacle. The nfs server will never start. ( https://tracker.ceph.com/issues/74191 / https://tracker.ceph.com/issues/74307 ) I've been able to successfully make the ganesha-server start by editing the ganesha.conf.j2 inside the ceph base container and replacing the line Protocols = 3,4 with Protocols = 4 inside the NFS_CORE_PARAM block. Unfortunately this temporary workaround is hard to maintain, so i have decided to look for a way to override the value in the ceph way of doing overrides. I used ceph nfs cluster config set main -i nfs-override.yml to override that parameter. NFS_CORE_PARAM { Protocols = 4; } I can see that the config is set by # ceph nfs cluster config get main NFS_CORE_PARAM { Protocols = 4; } But even after redeploy, the defaults of Protocols = 3,4 is never overridden. Any suggestions on how to override this? Atenciosamente/Kind regards, Salatiel

1 0

standby MDSes ?
by lejeczek 04 Jan '26

04 Jan '26

Hi guys. Is this ok & expected that standby MDS for a filesystem looks as such: -> $ ceph fs status APKI APKI - 4 clients ==== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 5516 143 2757 POOL TYPE USED AVAIL cephfs.APKI.meta metadata 671M 74.0G cephfs.APKI.data data 16.1G 74.0G STANDBY MDS APKI.podster3.utwwpp MONERO.podster3.gwtegj MDS version: ceph version 20.1.1 (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - RelWithDebInfo) What puzzles me from that above is: MONERO.podster3.gwtegj I have only two files systems - this is a small lab with ceph - and both were created with "default" ceph cmd, meaning that pools where created automatically by cmd. Is there no special and exclusive relationship between file system & MDS - MONERO.podster3 was created when file system/volume MONERO was created, yet it is available to a different file system? (what also boggles my mind is "size" the two file systems report, like they share something, but not relevant yet in case I have something seriously dis-configured relating what stat "status" showed) many thanks, L.

1 0

Re: OSDs not starting at boot after upgrade to Tentacle
by Chris Palmer 02 Jan '26

02 Jan '26

OK, found it. The ExecStart line in /usr/lib/systemd/system/ceph-volume@.service is wrong in Tentacle. squid: ExecStart=/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT /usr/sbin/ceph-volume-systemd %i' tentacle: ExecStart=/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT @CMAKE_INSTALL_PREFIX@/sbin/ceph-volume-systemd %i' Obviously the @CMAKE_INSTALL_PREFIX@ isn't going to work in there... Reinstating the line as in squid brings it back to life. On 02/01/2026 16:35, Chris Palmer via ceph-users wrote: > I've just upgraded a Centos 9 Stream test cluster from squid 19.2.3 to > tentacle 20.2.0. It is a non-cephadm package install. It all looked > pretty good, except that now no OSD processes start automatically > after a reboot: > > * There have been no other changes whatever to the system other than > ceph. > * OSD processes would start automatically on squid. > * After a reboot, all of the /var/lib/ceph/osd/ceph-n directories > still exist but are empty. > * A manual activation brings them all back to life (sudo ceph-volume > lvm activate --all). > * Nothing at all relevant in the ceph logs. > * After a reboot, and before manually activating, /run/systemd/system > is empty. After activation all of > the /run/systemd/system/ceph-osd.target.wants symlinks are back in > place, and the /var/lib/ceph/osd/ceph-n directories have been > repopulated. > * It is happening on every osd node. > * As expected, after 10 minutes the osds are marked out (unless noout > is set). > > I really can't spot what is causing this. Can anyone give me any > pointers? > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

OSDs not starting at boot after upgrade to Tentacle
by Chris Palmer 02 Jan '26

02 Jan '26

I've just upgraded a Centos 9 Stream test cluster from squid 19.2.3 to tentacle 20.2.0. It is a non-cephadm package install. It all looked pretty good, except that now no OSD processes start automatically after a reboot: * There have been no other changes whatever to the system other than ceph. * OSD processes would start automatically on squid. * After a reboot, all of the /var/lib/ceph/osd/ceph-n directories still exist but are empty. * A manual activation brings them all back to life (sudo ceph-volume lvm activate --all). * Nothing at all relevant in the ceph logs. * After a reboot, and before manually activating, /run/systemd/system is empty. After activation all of the /run/systemd/system/ceph-osd.target.wants symlinks are back in place, and the /var/lib/ceph/osd/ceph-n directories have been repopulated. * It is happening on every osd node. * As expected, after 10 minutes the osds are marked out (unless noout is set). I really can't spot what is causing this. Can anyone give me any pointers? Thanks, Chris

1 0

Re: CephFS performance issue with a certain application
by Martin Gerhard Loschwitz 01 Jan '26

01 Jan '26

Anthony, > Am 01.01.2026 um 03:01 schrieb Anthony D'Atri via ceph-users <ceph-users(a)ceph.io>: > > Forgive me if already asked, but: > > * Is the CephFS metadata pool on enterprise-class NVMe SSDs? Which model, and how many? Firmware up to date? all pools are within the same NVMes within the same RADOS cluster. There are also no separate devices for BlueStore WAL or any other metadata. > * Are those shared with other uses? Boot, OSD, etc? No. Ceph-OSDs are in exclusive use by RADOS: > * How many PGs in the metadata pool, across how many OSDs? Cluster has roughly 8200 PGs out of which the large majority is for the data and metadata pools. I will go find the exact numbers. Cluster is 12x16 15.56TB NVMes. > * Is the first CephFS data pool on SSDs, with setfattr layouts used to attach subdirectories to additional data pools? > Everything is on NVMes, but ttbomk, no setfattr was used in any of these. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

2 1

Re: CephFS performance issue with a certain application
by Anthony D'Atri 31 Dec '25

31 Dec '25

Forgive me if already asked, but: * Is the CephFS metadata pool on enterprise-class NVMe SSDs? Which model, and how many? Firmware up to date? * Are those shared with other uses? Boot, OSD, etc? * How many PGs in the metadata pool, across how many OSDs? * Is the first CephFS data pool on SSDs, with setfattr layouts used to attach subdirectories to additional data pools?

1 0

Re: CephFS performance issue with a certain application
by Georgi Steffenhagen 31 Dec '25

31 Dec '25

Hi Martin, Mark’s suggestions around multi-active MDSes and round-robin pinning of the subdirectories make a lot of sense for your layout and seem like a very promising next step, especially given the large, mostly static directory fan-out. One thing that may be worth looking at in parallel, just to broaden the picture a bit, are two additional areas that sometimes show up in similar CephFS workloads: First, even when MDS CPU usage looks perfectly fine, some of the latency patterns you describe (especially gaps between metadata steps) can come from the MDS waiting on journal commits or metadata-pool RADOS operations. It might be useful to correlate those stalls with OSD-side commit/apply latency, RocksDB/WAL behavior, or any background activity (scrubs, recovery) affecting the metadata pool specifically. Second, on the client side, highly parallel write/replace workloads can occasionally amplify implicit synchronization points. Even if the application logic is straightforward, it can be worth double-checking whether the Java/NIO path introduces fsync/force calls, atomic moves, or similar semantics that could turn metadata operations into synchronous commit points under load. These don’t contradict the multi-MDS approach at all, but together with the pinning experiments they might help narrow down whether the dominant bottleneck is namespace contention, metadata-pool latency, or an interaction between the two. Hope this is useful, and good luck with the next round of testing. Best :) Georgi Steffenhagen Storage Architecture On 31 December 2025 16:48:59 CET, Mark Nelson via ceph-users <ceph-users(a)ceph.io> wrote: >Hi Martin, > >How many active MDS processes are you running? Given you have so many sub-directories and they appear to be static, you may want to set up a new parallel test directory where you have all 65535 sub-directories round-robin pinned across multiple active MDSes and see how that compares to the existing configuration. You can achieve real gains this way, though multi-active MDSes may hit some bugs that aren't present in single-active MDS setups. > >Since you are not seeing obvious CPU load on any of the existing MDS processes, I would also be curious what a wallclock profile of the authoritative MDS for the parent directory would show. You can try mine if you'd like, it's available here: > >https://github.com/markhpc/uwpmp > >You'll need debug symbols installed. The libunwind backend is the most reliable so I would stick with that even though it is slower than libdw. > > >Thanks, >Mark > >On 12/31/25 2:50 AM, Martin Gerhard Loschwitz via ceph-users wrote: >> Folks, >> >> i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: >> >> * Read a file from the CephFS >> * Modify it >> * Write it back to the MDS in a new path with changed content >> >> Average file size is somewhere in the ballpark of 200-300kb. >> >> The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. >> >> We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. >> >> We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. >> >> We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. >> >> I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! >> >> Best regards >> Martin >> >-- >Best Regards, >Mark Nelson >Head of R&D (USA) > >Clyso GmbH >p: +49 89 21552391 12 >a: Loristraße 8 | 80335 München | Germany >w: https://clyso.com | e: mark.nelson(a)clyso.com > >We are hiring: https://www.clyso.com/jobs/ >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS performance issue with a certain application
by Mark Nelson 31 Dec '25

31 Dec '25

Hi Martin, How many active MDS processes are you running? Given you have so many sub-directories and they appear to be static, you may want to set up a new parallel test directory where you have all 65535 sub-directories round-robin pinned across multiple active MDSes and see how that compares to the existing configuration. You can achieve real gains this way, though multi-active MDSes may hit some bugs that aren't present in single-active MDS setups. Since you are not seeing obvious CPU load on any of the existing MDS processes, I would also be curious what a wallclock profile of the authoritative MDS for the parent directory would show. You can try mine if you'd like, it's available here: https://github.com/markhpc/uwpmp You'll need debug symbols installed. The libunwind backend is the most reliable so I would stick with that even though it is slower than libdw. Thanks, Mark On 12/31/25 2:50 AM, Martin Gerhard Loschwitz via ceph-users wrote: > Folks, > > i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: > > * Read a file from the CephFS > * Modify it > * Write it back to the MDS in a new path with changed content > > Average file size is somewhere in the ballpark of 200-300kb. > > The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. > > We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. > > We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. > > We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. > > I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! > > Best regards > Martin > -- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

1 0

Re: Best upgrade path?
by Robert Sander 31 Dec '25

31 Dec '25

Hi Jan, On 12/29/25 09:28, Jan Marek via ceph-users wrote: > I have a ceph cluster in version 16.1.10 on Ubuntu not managed by > cephadm. > > I would like to upgrade it to 19.1.3 and to cephadm mode. > > Please, can you advise me the best upgrade path? IMHO the path would be to adapt the cluster to cephadm first and then let the orchestrator do the upgrades. https://docs.ceph.com/en/pacific/cephadm/adoption/ https://docs.ceph.com/en/pacific/cephadm/upgrade/ Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

CephFS performance issue with a certain application
by Martin Gerhard Loschwitz 31 Dec '25

31 Dec '25

Folks, i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: * Read a file from the CephFS * Modify it * Write it back to the MDS in a new path with changed content Average file size is somewhere in the ballpark of 200-300kb. The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0