- ceph-users - lists.ceph.io

Re: OSDs not starting at boot after upgrade to Tentacle
by Chris Palmer 02 Jan '26

02 Jan '26

OK, found it. The ExecStart line in /usr/lib/systemd/system/ceph-volume@.service is wrong in Tentacle. squid: ExecStart=/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT /usr/sbin/ceph-volume-systemd %i' tentacle: ExecStart=/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT @CMAKE_INSTALL_PREFIX@/sbin/ceph-volume-systemd %i' Obviously the @CMAKE_INSTALL_PREFIX@ isn't going to work in there... Reinstating the line as in squid brings it back to life. On 02/01/2026 16:35, Chris Palmer via ceph-users wrote: > I've just upgraded a Centos 9 Stream test cluster from squid 19.2.3 to > tentacle 20.2.0. It is a non-cephadm package install. It all looked > pretty good, except that now no OSD processes start automatically > after a reboot: > > * There have been no other changes whatever to the system other than > ceph. > * OSD processes would start automatically on squid. > * After a reboot, all of the /var/lib/ceph/osd/ceph-n directories > still exist but are empty. > * A manual activation brings them all back to life (sudo ceph-volume > lvm activate --all). > * Nothing at all relevant in the ceph logs. > * After a reboot, and before manually activating, /run/systemd/system > is empty. After activation all of > the /run/systemd/system/ceph-osd.target.wants symlinks are back in > place, and the /var/lib/ceph/osd/ceph-n directories have been > repopulated. > * It is happening on every osd node. > * As expected, after 10 minutes the osds are marked out (unless noout > is set). > > I really can't spot what is causing this. Can anyone give me any > pointers? > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

OSDs not starting at boot after upgrade to Tentacle
by Chris Palmer 02 Jan '26

02 Jan '26

I've just upgraded a Centos 9 Stream test cluster from squid 19.2.3 to tentacle 20.2.0. It is a non-cephadm package install. It all looked pretty good, except that now no OSD processes start automatically after a reboot: * There have been no other changes whatever to the system other than ceph. * OSD processes would start automatically on squid. * After a reboot, all of the /var/lib/ceph/osd/ceph-n directories still exist but are empty. * A manual activation brings them all back to life (sudo ceph-volume lvm activate --all). * Nothing at all relevant in the ceph logs. * After a reboot, and before manually activating, /run/systemd/system is empty. After activation all of the /run/systemd/system/ceph-osd.target.wants symlinks are back in place, and the /var/lib/ceph/osd/ceph-n directories have been repopulated. * It is happening on every osd node. * As expected, after 10 minutes the osds are marked out (unless noout is set). I really can't spot what is causing this. Can anyone give me any pointers? Thanks, Chris

1 0

Re: CephFS performance issue with a certain application
by Martin Gerhard Loschwitz 01 Jan '26

01 Jan '26

Anthony, > Am 01.01.2026 um 03:01 schrieb Anthony D'Atri via ceph-users <ceph-users(a)ceph.io>: > > Forgive me if already asked, but: > > * Is the CephFS metadata pool on enterprise-class NVMe SSDs? Which model, and how many? Firmware up to date? all pools are within the same NVMes within the same RADOS cluster. There are also no separate devices for BlueStore WAL or any other metadata. > * Are those shared with other uses? Boot, OSD, etc? No. Ceph-OSDs are in exclusive use by RADOS: > * How many PGs in the metadata pool, across how many OSDs? Cluster has roughly 8200 PGs out of which the large majority is for the data and metadata pools. I will go find the exact numbers. Cluster is 12x16 15.56TB NVMes. > * Is the first CephFS data pool on SSDs, with setfattr layouts used to attach subdirectories to additional data pools? > Everything is on NVMes, but ttbomk, no setfattr was used in any of these. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

2 1

Re: CephFS performance issue with a certain application
by Anthony D'Atri 01 Jan '26

01 Jan '26

Forgive me if already asked, but: * Is the CephFS metadata pool on enterprise-class NVMe SSDs? Which model, and how many? Firmware up to date? * Are those shared with other uses? Boot, OSD, etc? * How many PGs in the metadata pool, across how many OSDs? * Is the first CephFS data pool on SSDs, with setfattr layouts used to attach subdirectories to additional data pools?

1 0

Re: CephFS performance issue with a certain application
by Georgi Steffenhagen 01 Jan '26

01 Jan '26

Hi Martin, Mark’s suggestions around multi-active MDSes and round-robin pinning of the subdirectories make a lot of sense for your layout and seem like a very promising next step, especially given the large, mostly static directory fan-out. One thing that may be worth looking at in parallel, just to broaden the picture a bit, are two additional areas that sometimes show up in similar CephFS workloads: First, even when MDS CPU usage looks perfectly fine, some of the latency patterns you describe (especially gaps between metadata steps) can come from the MDS waiting on journal commits or metadata-pool RADOS operations. It might be useful to correlate those stalls with OSD-side commit/apply latency, RocksDB/WAL behavior, or any background activity (scrubs, recovery) affecting the metadata pool specifically. Second, on the client side, highly parallel write/replace workloads can occasionally amplify implicit synchronization points. Even if the application logic is straightforward, it can be worth double-checking whether the Java/NIO path introduces fsync/force calls, atomic moves, or similar semantics that could turn metadata operations into synchronous commit points under load. These don’t contradict the multi-MDS approach at all, but together with the pinning experiments they might help narrow down whether the dominant bottleneck is namespace contention, metadata-pool latency, or an interaction between the two. Hope this is useful, and good luck with the next round of testing. Best :) Georgi Steffenhagen Storage Architecture On 31 December 2025 16:48:59 CET, Mark Nelson via ceph-users <ceph-users(a)ceph.io> wrote: >Hi Martin, > >How many active MDS processes are you running? Given you have so many sub-directories and they appear to be static, you may want to set up a new parallel test directory where you have all 65535 sub-directories round-robin pinned across multiple active MDSes and see how that compares to the existing configuration. You can achieve real gains this way, though multi-active MDSes may hit some bugs that aren't present in single-active MDS setups. > >Since you are not seeing obvious CPU load on any of the existing MDS processes, I would also be curious what a wallclock profile of the authoritative MDS for the parent directory would show. You can try mine if you'd like, it's available here: > >https://github.com/markhpc/uwpmp > >You'll need debug symbols installed. The libunwind backend is the most reliable so I would stick with that even though it is slower than libdw. > > >Thanks, >Mark > >On 12/31/25 2:50 AM, Martin Gerhard Loschwitz via ceph-users wrote: >> Folks, >> >> i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: >> >> * Read a file from the CephFS >> * Modify it >> * Write it back to the MDS in a new path with changed content >> >> Average file size is somewhere in the ballpark of 200-300kb. >> >> The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. >> >> We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. >> >> We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. >> >> We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. >> >> I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! >> >> Best regards >> Martin >> >-- >Best Regards, >Mark Nelson >Head of R&D (USA) > >Clyso GmbH >p: +49 89 21552391 12 >a: Loristraße 8 | 80335 München | Germany >w: https://clyso.com | e: mark.nelson(a)clyso.com > >We are hiring: https://www.clyso.com/jobs/ >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS performance issue with a certain application
by Mark Nelson 31 Dec '25

31 Dec '25

Hi Martin, How many active MDS processes are you running? Given you have so many sub-directories and they appear to be static, you may want to set up a new parallel test directory where you have all 65535 sub-directories round-robin pinned across multiple active MDSes and see how that compares to the existing configuration. You can achieve real gains this way, though multi-active MDSes may hit some bugs that aren't present in single-active MDS setups. Since you are not seeing obvious CPU load on any of the existing MDS processes, I would also be curious what a wallclock profile of the authoritative MDS for the parent directory would show. You can try mine if you'd like, it's available here: https://github.com/markhpc/uwpmp You'll need debug symbols installed. The libunwind backend is the most reliable so I would stick with that even though it is slower than libdw. Thanks, Mark On 12/31/25 2:50 AM, Martin Gerhard Loschwitz via ceph-users wrote: > Folks, > > i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: > > * Read a file from the CephFS > * Modify it > * Write it back to the MDS in a new path with changed content > > Average file size is somewhere in the ballpark of 200-300kb. > > The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. > > We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. > > We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. > > We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. > > I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! > > Best regards > Martin > -- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

1 0

Re: Best upgrade path?
by Robert Sander 31 Dec '25

31 Dec '25

Hi Jan, On 12/29/25 09:28, Jan Marek via ceph-users wrote: > I have a ceph cluster in version 16.1.10 on Ubuntu not managed by > cephadm. > > I would like to upgrade it to 19.1.3 and to cephadm mode. > > Please, can you advise me the best upgrade path? IMHO the path would be to adapt the cluster to cephadm first and then let the orchestrator do the upgrades. https://docs.ceph.com/en/pacific/cephadm/adoption/ https://docs.ceph.com/en/pacific/cephadm/upgrade/ Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

CephFS performance issue with a certain application
by Martin Gerhard Loschwitz 31 Dec '25

31 Dec '25

Folks, i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: * Read a file from the CephFS * Modify it * Write it back to the MDS in a new path with changed content Average file size is somewhere in the ballpark of 200-300kb. The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Thank You, Ceph Community and Looking Ahead to 2026
by Anthony Middleton 30 Dec '25

30 Dec '25

Hello, Ceph friends! I hope the holiday season is treating you well and that you' re finding time to relax and recharge after a great year. I want to sincerely thank you for the warm welcome as I step into the role of Ceph Community Manager. It has been a privilege to engage with such a collaborative, thoughtful, and technically driven community, and I am grateful for the openness and support you have shown from day one. The Ceph Community celebrated many big wins this year, including the release of V 20. 2. 0 "Tentacle." Congratulations to everyone involved, from CSC members to other Ceph contributors. The related blog post has already reached over 3, 3,000 views on ceph. io, reflecting both the quality of the work and the community' s interest in sharing it. The Ceph Foundation appreciates all Ceph Community members who continue to invest time to improving documentation, publishing blogs, fixing bugs, and answering questions on Slack, mailing lists, and GitHub. These efforts lower barriers for new users and contributors and enhance usability for operators day to day. Thanks also to everyone who helped make Cephalocon 2025 in Vancouver a success, along with five outstanding Ceph Days events around the world in 2025: Ceph Days India, Silicon Valley, Seattle, London, and Berlin. These events do not happen without significant volunteer efforts, and your commitment continues to showcase the strength of the Ceph community worldwide. Looking ahead to 2026, I encourage everyone to stay involved and help us accomplish even more. Let' s work to expand Ceph Days into new regions, boost engagement on LinkedIn, Slack, our mailing lists, and Twitter, and find new ways to connect users and contributors. If you' re interested in becoming a Ceph Ambassador or want more information about contributing to Ceph code, I invite you to fill out the Ceph Contributors Intake Form: https://t. ly/contributors <https://t.ly/contributors>. Thank you again for everything you do to support Ceph and one another. I am excited about what we can achieve together in the year ahead. Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 30 Dec '25

30 Dec '25

> * The filesystem (BlueStore or whatever) or DBMS (RocksDB or > whatever) is required to issue "write buffer commit" commands > to the device whenever necessary to ensure metadata or data is > not lost by cache power loss and remains consistent. .. which is why it’s advisable to disable HDD volatile write cache. Yet, I’ve seen data lost when power is lost with my own eyes. When an SSD is too cheap to include a supercap, I’m sure not going to trust that it implements the above rigorously. PLP != DRAMless. > * _Some_ loss is inevitable: there can be data still in the > middle of being written if the loss happen during a "write to > device" operation and efficient filesystems can do pretty > large ones. I did a PDU test and 2/3 of the OSDs were corrupted. YMMV, not going to debate you, Jerry. >> Just don’t waste money on 3 DWPD SKUs or RAID HBAs. > > Usually but I have seen some cases where the DB/WAL SSDs really > should have been 3 DWPD, especially if the SSD is shared among > data-only OSDs, which is almost inevitable in case of servers > with many HDDs. 3DWPD and 1DWPD SSDs are usually the exact same hardware, with different overprovisioning. You can turn one into the other, or pick something in between, with software. That said, see above where you wrote about operational false economies, hassling with hybrid OSDs is right up there. >In those cases better to be safe than sorry. Offload SSDs require paying extra for universal drive bays or rear cages. The former are prone to fussy tri-mode HBAs, and the latter to too large an HDD to SSD ratio. Unless you go down the dead-end of SATA SSDs. IMHO many deployments are better off with monolithic HDD OSDs, using more available slots for more spindles, with separate index/metadata pools. ymmv.

1 0