January 2026 - ceph-users

Re: Question concerning selection of secondary OSDs
by Wesley Dillingham 09 Jan '26

09 Jan '26

"leading to the write acknowledgement being sent to the client, because min_size=2 A misunderstanding of min_size Ceph requires all replicas (or shards in EC) to ack the primary before the client is acknowledged. The primary OSD and therefore client are subjected to the latency of the hop between the primary and all replicas. min_size is how many up to date replicas are required for the PG to be active (accepts IO) Respectfully, *Wes Dillingham* On Fri, Jan 9, 2026 at 4:36 PM Martin Gerhard Loschwitz via ceph-users < ceph-users(a)ceph.io> wrote: > Folks, > > I am having a knowledge question concerning the selection of secondary > OSDs in Ceph. > > I have a cluster here that consists of three nodes. For the sake of the > argument, I have simulated latency between the third node and the other two > using tc and netem. I have set the priority-affinity of the OSDs on the > third node to 0, and indeed, RADOS is not using any of these OSDs as > primary OSD, so this part works as expected. > > Furthermore, my expectation for a pool that has size=3 and min_size=2 is > that for any given write, the primary OSD on nodes 1 or 2 will select a > secondary OSD in node 1/2 respectively and one in node 3. Which would then > lead me to believe that any client writing into the cluster from node 1 > will only ever have the latency between node 1 and node 2 as an actual > performance penalty because > > * client selects primary OSD on node 1 or node 2 > * primary OSD selects secondary OSDs and starts transfer in parallel > * Write to OSD with lower latency will finish much sooner than the one to > the other OSD, leading to the write acknowledgement being sent to the > client, because min_size=2 > > But that appears not to be the case. priority-affinity has a very slight > impact, but the overall performance when writing into the cluster with > queue depth 1 and request size of 4k still very much resembles a scenario > in which every single write appears to be latency-penalized with the > latency between node1/2 and node 3. > > Where is my understanding incorrect? Or are there any configuration > settings for this? I tried to search for this, but the only results I can > find refer to priority-affinity. I am looking into something like > „secondary affinity“ I guess, but I do not think that such a thing exists > in Ceph. Which leads me to believe that my understanding of this is > seriously wrong somehow. > > Any hint will be greatly appreciated. Thank you very much in advance. > > Best regards > Martin > > -- > Martin Gerhard Loschwitz > Geschäftsführer / CEO, True West IT Services GmbH > Phone: +49 2433 5253130 > Mobile: +49 176 61832178 > Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany > Legal: HRB 21985, Amtsgericht Mönchengladbach > VAT: DE363893844 > > True West IT Services GmbH is compliant with the GDPR regulation on data > protection and privacy in the European Union and the European Economic > Area. You can request the information on how we collect and process your > private data according to the law by contacting the email sender. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Question concerning selection of secondary OSDs
by Martin Gerhard Loschwitz 09 Jan '26

09 Jan '26

Folks, I am having a knowledge question concerning the selection of secondary OSDs in Ceph. I have a cluster here that consists of three nodes. For the sake of the argument, I have simulated latency between the third node and the other two using tc and netem. I have set the priority-affinity of the OSDs on the third node to 0, and indeed, RADOS is not using any of these OSDs as primary OSD, so this part works as expected. Furthermore, my expectation for a pool that has size=3 and min_size=2 is that for any given write, the primary OSD on nodes 1 or 2 will select a secondary OSD in node 1/2 respectively and one in node 3. Which would then lead me to believe that any client writing into the cluster from node 1 will only ever have the latency between node 1 and node 2 as an actual performance penalty because * client selects primary OSD on node 1 or node 2 * primary OSD selects secondary OSDs and starts transfer in parallel * Write to OSD with lower latency will finish much sooner than the one to the other OSD, leading to the write acknowledgement being sent to the client, because min_size=2 But that appears not to be the case. priority-affinity has a very slight impact, but the overall performance when writing into the cluster with queue depth 1 and request size of 4k still very much resembles a scenario in which every single write appears to be latency-penalized with the latency between node1/2 and node 3. Where is my understanding incorrect? Or are there any configuration settings for this? I tried to search for this, but the only results I can find refer to priority-affinity. I am looking into something like „secondary affinity“ I guess, but I do not think that such a thing exists in Ceph. Which leads me to believe that my understanding of this is seriously wrong somehow. Any hint will be greatly appreciated. Thank you very much in advance. Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Re: Squid Cephadm deploys Quincy MDS
by Loïc Tortay 09 Jan '26

09 Jan '26

On 09/01/2026 15:55, Chris Palmer via ceph-users wrote: > I'm working my way through adopting a test Squid cluster. MONs, MGRs, > OSDs are now adopted ok. The cluster was installed from scratch on Squid. > > I just tried adopting the MDS services. Adding the new containerized > ones appeared to work, except... > > ceph versions: > > "mds": { > "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > squid (stable)": 2 > }, > > ceph orch apply mds cephfs "c1,c2" > > ceph versions: > > "mds": { > "ceph version 17.0.0-7183-g54142666 > (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, > "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > squid (stable)": 2 > }, > > > How have I ended up with quincy ones??? > > At this stage they were only standby so have stopped them... > Hello, If you did not specify a specific image version, the global configuration is used and, if that is not defined, it will use a hardcoded image name: ceph config set global container_image quay.io/ceph/ceph:v19.2.3 See for instance: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/THAH2JFQNB7… (we have also had this issue while adopting a Quincy cluster) Once the image configuration is set, I think we just used "ceph orch daemon redeploy mds.$MDS_NAME" to get the correct version up and running. Loïc. -- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Re: Squid Cephadm deploys Quincy MDS
by Eugen Block 09 Jan '26

09 Jan '26

Hi, ensure you have configured the right base image. Check with: ceph config ls | grep container_image container_image And set it to a specific version with: ceph config set global container_image <version> Zitat von Chris Palmer via ceph-users <ceph-users(a)ceph.io>: > I'm working my way through adopting a test Squid cluster. MONs, > MGRs, OSDs are now adopted ok. The cluster was installed from > scratch on Squid. > > I just tried adopting the MDS services. Adding the new containerized > ones appeared to work, except... > > ceph versions: > > "mds": { > "ceph version 19.2.3 > (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 > }, > > ceph orch apply mds cephfs "c1,c2" > > ceph versions: > > "mds": { > "ceph version 17.0.0-7183-g54142666 > (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, > "ceph version 19.2.3 > (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 > }, > > > How have I ended up with quincy ones??? > > At this stage they were only standby so have stopped them... > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Squid Cephadm deploys Quincy MDS
by Chris Palmer 09 Jan '26

09 Jan '26

I'm working my way through adopting a test Squid cluster. MONs, MGRs, OSDs are now adopted ok. The cluster was installed from scratch on Squid. I just tried adopting the MDS services. Adding the new containerized ones appeared to work, except... ceph versions: "mds": { "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 }, ceph orch apply mds cephfs "c1,c2" ceph versions: "mds": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 }, How have I ended up with quincy ones??? At this stage they were only standby so have stopped them... Thanks, Chris

1 0

Re: Cannot adopt osd (on squid)
by Chris Palmer 09 Jan '26

09 Jan '26

On 09/01/2026 13:22, Loïc Tortay via ceph-users wrote: > On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: >> I have a squid package-installed test cluster that I am trying to >> adopt into cephadm. >> >> Monitors & managers have been adopted, and the orchestrator is >> functioning. >> >> When I try to adopt an OSD though (e.g. osd.0), cephadm creates the >> new directory (<fsid>/osd.0) and populates the files from the legacy >> directory. It then crashes complaining that the directory osd/ceph-0 >> is not empty so cannot be removed. In some cases none of the files >> have been removed, and in others the block symlink and >> require_osd_release file remain. I can remove the new <fsid>/osd.0 >> and use ceph-volume to activate the OSD again. >> >> cephadm.log reports that every file has been moved and the ownership >> set - even for files that still remain in the legacy directory. It >> doesn't report any other errors (apart from the directory not empty >> error. when it crashes). >> >> I really can't find where to go from here. Any pointers appreciated... >> > > Hello, > We've had the same issue, with clusters initially installed/managed > with Ceph-Ansible. > > The issue (w/ Ceph-Ansible) seems to be related to some failure during > the initial configuration which left multiple mounts on the same > "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active > mountpoint therefore it can't be removed). > "mount|grep /var/lib/ceph|sort|uniq -c|sort -n" shows which OSDs > have multiple mounts (first field). > > After the failed adoption, our workaround/recovery procedure was the > following ($NOK is the OSD number): > # Move the configuration out of the way (do not remove it) > mv -v /var/lib/ceph/$CLUSTERID/osd.$NOK /var/lib/ceph/osd.$NOK.nok > # If there are more than 2 mounts on the same directory, umount all > but # one to allow the adoption process to complete (the configuration > will # not be valid, the OSD won't start) > cephadm adopt --style legacy --name osd.$NOK > # Copy the correct configuration files to the adopted OSD directory > cd /var/lib/ceph/$CLUSTERID/osd.$NOK > cp -v ../../osd.$NOK.nok/{keyring,fsid} . > # $OKOSD is the number of a successfully adopted OSD. > cp -v ../../osd.$OKOSD/config . > # > # Check the "block*" links point to the same devices as in the first > # directory ("/var/lib/ceph/osd.$NOK.nok"), correct the links if > # they're wrong > # > # Make sure the files & links belong the correct UID for the containers > chown -hv 167:167 block* > chown -v 167:167 * > > systemctl status ceph-$CLUSTERID(a)osd.$NOK.service > systemctl start ceph-$CLUSTERID(a)osd.$NOK.service > systemctl status ceph-$CLUSTERID(a)osd.$NOK.service > > Other things we have noted after adoptions are: > - do *not* remove the packages used before the adoption, they (at > least the RPMs) have pre-remove scripts which stop the OSDs/Ceph on > the node > - cleanup "logrotate" conflicting configurations from the packages & > containers (i.e. remove the configuration files provided by the packages) > > > Loïc. Thanks for the info, and the last two hints! Will have to watch out for those... This test cluster was freshly built on squid 19.2.3 for testing this migration, purely using the manual install method (not ceph-ansible). I've already triple-checked the mounts, symlinks, ownerships, etc, and we don't have any of those problems. Some progress though. The problem is not entirely repeatable. The files are always replicated into the new directory correctly. Sometimes no files are removed from the legacy directory (and it fails as it is not empty), sometimes some are left (and it fails), and sometimes it works. Even on the same OSD. If it fails, I can manually remove the files from the legacy directory, and repeat the adopt. It then completes, removing the legacy directory, although the OSD is still stopped. Manually starting it using "ceph orch daemon start osd.n" works. I've also noticed that sometimes it takes a while to appear in "ceph orch ps", and I have to wait until it does before starting it. I'm going to do some reboot and daemon restart tests, then continue on the rest of the OSDs.... Thanks, Chris

1 0

Re: Cannot adopt osd (on squid)
by Loïc Tortay 09 Jan '26

09 Jan '26

On 09/01/2026 14:22, Loïc Tortay via ceph-users wrote: > On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: >> I have a squid package-installed test cluster that I am trying to >> adopt into cephadm. >> >> Monitors & managers have been adopted, and the orchestrator is >> functioning. >> >> When I try to adopt an OSD though (e.g. osd.0), cephadm creates the >> new directory (<fsid>/osd.0) and populates the files from the legacy >> directory. It then crashes complaining that the directory osd/ceph-0 >> is not empty so cannot be removed. In some cases none of the files >> have been removed, and in others the block symlink and >> require_osd_release file remain. I can remove the new <fsid>/osd.0 and >> use ceph-volume to activate the OSD again. >> >> cephadm.log reports that every file has been moved and the ownership >> set - even for files that still remain in the legacy directory. It >> doesn't report any other errors (apart from the directory not empty >> error. when it crashes). >> >> I really can't find where to go from here. Any pointers appreciated... >> > > Hello, > We've had the same issue, with clusters initially installed/managed with > Ceph-Ansible. > > The issue (w/ Ceph-Ansible) seems to be related to some failure during > the initial configuration which left multiple mounts on the same > "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active mountpoint > therefore it can't be removed). > Of course this is the "/var/lib/ceph/osd/ceph-$OSD" directory which has multiple mounts, not the one used by the container. Loïc.-- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Re: Cannot adopt osd (on squid)
by Loïc Tortay 09 Jan '26

09 Jan '26

On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: > I have a squid package-installed test cluster that I am trying to adopt > into cephadm. > > Monitors & managers have been adopted, and the orchestrator is functioning. > > When I try to adopt an OSD though (e.g. osd.0), cephadm creates the new > directory (<fsid>/osd.0) and populates the files from the legacy > directory. It then crashes complaining that the directory osd/ceph-0 is > not empty so cannot be removed. In some cases none of the files have > been removed, and in others the block symlink and require_osd_release > file remain. I can remove the new <fsid>/osd.0 and use ceph-volume to > activate the OSD again. > > cephadm.log reports that every file has been moved and the ownership set > - even for files that still remain in the legacy directory. It doesn't > report any other errors (apart from the directory not empty error. when > it crashes). > > I really can't find where to go from here. Any pointers appreciated... > Hello, We've had the same issue, with clusters initially installed/managed with Ceph-Ansible. The issue (w/ Ceph-Ansible) seems to be related to some failure during the initial configuration which left multiple mounts on the same "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active mountpoint therefore it can't be removed). "mount|grep /var/lib/ceph|sort|uniq -c|sort -n" shows which OSDs have multiple mounts (first field). After the failed adoption, our workaround/recovery procedure was the following ($NOK is the OSD number): # Move the configuration out of the way (do not remove it) mv -v /var/lib/ceph/$CLUSTERID/osd.$NOK /var/lib/ceph/osd.$NOK.nok # If there are more than 2 mounts on the same directory, umount all but # one to allow the adoption process to complete (the configuration will # not be valid, the OSD won't start) cephadm adopt --style legacy --name osd.$NOK # Copy the correct configuration files to the adopted OSD directory cd /var/lib/ceph/$CLUSTERID/osd.$NOK cp -v ../../osd.$NOK.nok/{keyring,fsid} . # $OKOSD is the number of a successfully adopted OSD. cp -v ../../osd.$OKOSD/config . # # Check the "block*" links point to the same devices as in the first # directory ("/var/lib/ceph/osd.$NOK.nok"), correct the links if # they're wrong # # Make sure the files & links belong the correct UID for the containers chown -hv 167:167 block* chown -v 167:167 * systemctl status ceph-$CLUSTERID(a)osd.$NOK.service systemctl start ceph-$CLUSTERID(a)osd.$NOK.service systemctl status ceph-$CLUSTERID(a)osd.$NOK.service Other things we have noted after adoptions are: - do *not* remove the packages used before the adoption, they (at least the RPMs) have pre-remove scripts which stop the OSDs/Ceph on the node - cleanup "logrotate" conflicting configurations from the packages & containers (i.e. remove the configuration files provided by the packages) Loïc. -- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Cannot adopt osd (on squid)
by Chris Palmer 09 Jan '26

09 Jan '26

I have a squid package-installed test cluster that I am trying to adopt into cephadm. Monitors & managers have been adopted, and the orchestrator is functioning. When I try to adopt an OSD though (e.g. osd.0), cephadm creates the new directory (<fsid>/osd.0) and populates the files from the legacy directory. It then crashes complaining that the directory osd/ceph-0 is not empty so cannot be removed. In some cases none of the files have been removed, and in others the block symlink and require_osd_release file remain. I can remove the new <fsid>/osd.0 and use ceph-volume to activate the OSD again. cephadm.log reports that every file has been moved and the ownership set - even for files that still remain in the legacy directory. It doesn't report any other errors (apart from the directory not empty error. when it crashes). I really can't find where to go from here. Any pointers appreciated... Thanks, Chris

1 0

RGW crashes: caught signal abort, failed to read body: partial message
by Christopher James 08 Jan '26

08 Jan '26

Hi, In our ceph cluster deployment, reef 18.2.7, rgw daemons keep crashing with the following trace. A abort signal is caught by the rgws. in thread 7f2b3eb71640 thread_name:notif-worker0 ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f2bb2fe0bf0] 2: /lib64/libc.so.6(+0x8bf5c) [0x7f2bb302df5c] 3: raise() 4: abort() 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2bb3342b21] 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2bb334e53c] 7: /lib64/libstdc++.so.6(+0xac509) [0x7f2bb334d509] 8: __gxx_personality_v0() 9: /lib64/libgcc_s.so.1(+0x112d4) [0x7f2bb31bd2d4] 10: _Unwind_Resume() 11: /usr/bin/radosgw(+0x2ddb93) [0x55677058db93] 12: /usr/bin/radosgw(+0x3cd8cd) [0x55677067d8cd] 13: /lib64/libstdc++.so.6(+0xdbae4) [0x7f2bb337cae4] 14: /lib64/libc.so.6(+0x8a21a) [0x7f2bb302c21a] 15: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. When I grep the thread ID which is getting abort signal, these are the last logs: 2026-01-08T13:28:56.614+0000 7f2b3eb71640 -1 *** Caught signal (Aborted) ** in thread 7f2b3eb71640 thread_name:notif-worker0 -78> 2026-01-08T13:28:56.610+0000 7f2b3eb71640 4 failed to read body: partial message 0> 2026-01-08T13:28:56.614+0000 7f2b3eb71640 -1 *** Caught signal (Aborted) ** in thread 7f2b3eb71640 thread_name:notif-worker0 7f2b3eb71640 / notif-worker0 the failed to read body: partial mesage in this log is related to a GET request. I found a bugzilla in the redhat forum which has the same trace but occurs when a CopyObject is done on an encrypted object. In our cluster SSE is not enabled. So I don't think that is related. Just for ref the link to the bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2362258 Has anybody faced this issue before? Could anybody help with this? Regards Chris

1 0

Re: Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Janne Johansson 08 Jan '26

08 Jan '26

> Let’s pretend my original mail had never existed … > and for those running into a similar issue: Ensure the PG autoscaler is actually working properly or else your pools will have … limited numbers of PGs. Thanks for letting us know. -- May the most significant bit of your life be positive.

1 0

Re: Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Martin Gerhard Loschwitz 08 Jan '26

08 Jan '26

Let’s pretend my original mail had never existed … and for those running into a similar issue: Ensure the PG autoscaler is actually working properly or else your pools will have … limited numbers of PGs. M. -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Martin Gerhard Loschwitz 08 Jan '26

08 Jan '26

Folks, on two completely unrelated NVMe-only clusters, I am seeing detrimental performance for benchmarks with QD=256 and BS=4K: Where a comparable SAS-SSD cluster has 95k IOPS, with NVMes, I see only 7000 (one cluster) and 14000 (for the other one). I must be overseeing something very obvious here. What I have found out that „ceph tell osd bench“ for all OSDs is fine and within the expectations (50k+ IOPS) and network is fine (24,7Gbit/s). Benchmark is performed using „rados bench“ with „-t 256“ and -b 4K“. Total number of OSDs is 12. Dell Enterprise NVMes, 15.6TB per piece. During benchmarks, it looks as if only one OSD is even active on each of the nodes according to htop and iostat. The other ones appear to be inactive. I’m going nuts over this … What is going on here? :( Thank you very much in advance. M. -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Re: Object replication issue between zones for RGW Account owned buckets
by Ashok Kumar M 08 Jan '26

08 Jan '26

Hi, I’m currently using Ceph version 19.2.3 and have a setup with two zones (zone-a and zone-b) under a single zonegroup (zonegroup-a) and realm (realm-a). When I replicate objects from bucket-a (in zone-a) to bucket-b (in zone-b) for buckets owned by a regular user (ex: my-user), the replication works successfully. However, replication does not work for buckets owned by the RGW Account (ex: RGW84754594545284760). In this case: There are no errors shown in the sync status on either zone. The replication status of the objects in the source bucket remains in a “PENDING” state indefinitely. Has anyone else encountered a similar issue, or is there any known workaround for this? I would really appreciate any kind of advice or guidance regarding this. Thanks, Ashok M Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 1

Has anyone been able to use NFS in tentacle or it is really broken?
by Salatiel Filho 07 Jan '26

07 Jan '26

Hi everyone, has anyone been able to use NFS in tentacle? The few people I know using it in squid said it has broken after tentacle upgrade. I have tried to create a NFS from scratch in a newly deployed instance of tentacle and it just does not work: The ganesha nfs server will keep dying. I've been able to workaround that by fully disabling the protocol 3 in the ganesha template directly in the tentacle image and deploying this new image, but I was expecting more people talking about this if it is a general problem. Maybe NFS is not being used anymore ? I can see a couple tickets in the issue tracker, but with no reply at all. Atenciosamente/Kind regards, Salatiel

1 0

Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 07 Jan '26

07 Jan '26

Hi, this is kind of a follow-up on a two years old thread [0] and I wanted to raise some awareness for the corresponding tracker [1]. Back then we managed to limit the impact on the mon store performance with some paxos configs, but now the OSDs are impacted as well when there are new ones created: Each newly created OSD process grows in RAM usage to around 140 GB within a few minutes, easily causing oom killers on hosts if multiple OSDs are created at once. The residual RAM usage drops to the memory target after it successfully booted. The reason are the purged_snaps that are loaded during OSD boot (snap_mapper.record_purged_snaps purged_snaps), two years ago the customer had more than 42 million purged_snap entries. I don't know how many there are today, I don't have access myself, but I'll try to get a current number. Anyway, the only way to safely create OSDs is one by one, maybe two at once depending on the host's RAM capacity. An automated (unattended) OSD deployment is currently not possible. Unfortunately, the new tool [2] doesn't seem to work as expected, at least in my test cluster it didn't have any impact on the number of purged_snaps in the mon store. That's why we haven't tried it on the customer cluster(s) yet. How do other operators/admins/users deal with this kind of scenario? Having many snapshots can't be a corner case, but I can't remember having read anything like this on the list(s). I'd appreciate any comments, although I'm aware that everybody is probably busy travelling to Cephalocon. ;-) Thanks! Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZEMGKBLMERE… [1] https://tracker.ceph.com/issues/64519 [2] https://github.com/ceph/ceph/pull/57548

1 1

Re: Network Card
by Anthony D'Atri 06 Jan '26

06 Jan '26

Others may have more specific experience, but FWIW having 100GE of any make puts you ahead of a lot of deployments. That said, I suspect that the Mellanox CX6 may be overkill unless you can leverage substantial offload. OCP lets you conserve scarce standard PCIe slots for other, future purposes, so I like to provision such mezzanine cards when possible. > On Jan 6, 2026, at 10:59 AM, ceph_elk via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > is there any specific proposal/measurement about good/bad 100G network cards ? > > We plan Dell Servers with Broadcom 57508 Dual Port 100GbE QSFP56, OCP NIC 3.0 card but Nviadia is available too > NVIDIA ConnectX-6-Adapter, 2 Anschlüsse, 100 GbE QSFP56, OCP 3.0, mit Kabel > > Which one should be the ceph prefered one? > > Thanks in advance > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Network Card
by ceph_elk 06 Jan '26

06 Jan '26

Hi, is there any specific proposal/measurement about good/bad 100G network cards ? We plan Dell Servers with Broadcom 57508 Dual Port 100GbE QSFP56, OCP NIC 3.0 card but Nviadia is available too NVIDIA ConnectX-6-Adapter, 2 Anschlüsse, 100 GbE QSFP56, OCP 3.0, mit Kabel Which one should be the ceph prefered one? Thanks in advance

1 0

Re: standby MDSes ?
by Gregory Farnum 05 Jan '26

05 Jan '26

On Sun, Jan 4, 2026 at 1:48 AM lejeczek via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi guys. > > Is this ok & expected that standby MDS for a filesystem > looks as such: > > -> $ ceph fs status APKI > APKI - 4 clients > ==== > RANK STATE MDS ACTIVITY DNS > INOS DIRS CAPS > 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 > 5516 143 2757 > POOL TYPE USED AVAIL > cephfs.APKI.meta metadata 671M 74.0G > cephfs.APKI.data data 16.1G 74.0G > STANDBY MDS > APKI.podster3.utwwpp > MONERO.podster3.gwtegj > MDS version: ceph version 20.1.1 > (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - > RelWithDebInfo) > > What puzzles me from that above is: MONERO.podster3.gwtegj > I have only two files systems - this is a small lab with > ceph - and both were created with "default" ceph cmd, > meaning that pools where created automatically by cmd. > Is there no special and exclusive relationship between file > system & MDS - MONERO.podster3 was created when file > system/volume MONERO was created, yet it is available to a > different file system? Yes, this is fine. The monitors can generally assign a standby MDS to any filesystem, and if you look at the overall "ceph fs status" view you will see the standby daemons are listed out separately from the active FSes for that reason. > (what also boggles my mind is "size" the two file systems > report, like they share something, but not relevant yet in > case I have something seriously dis-configured relating what > stat "status" showed) The size statistics you are seeing here are drawn from RADOS reporting, so unless you have FSes configured to use different OSDs (via different CRUSH rules for the relevant pools) or have set pool quotas, they are going to report the same available capacity. -Greg > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceohfs+S3
by ceph_elk 05 Jan '26

05 Jan '26

Hi folks, Please be so kind and tell me how I can provide a bunch of files via cephfs AND S3? I'll have a hunderts of folders with a couple of files which needs to be shared via S3 to different users one per folder Thanks in advance Elkman

1 0

Re: multi site divergence?
by Casey Bodley 05 Jan '26

05 Jan '26

hi Chris, if you create a new placement target for this fast storage, you'd have to provide pool mappings for it on both zones even if you only want one zone to use it. and you're right that sync policy can disable replication, but that just means both zones will have separate views of the bucket. nothing would prevent clients from uploading objects to the 'wrong' zone, or deleting the empty bucket on one zone even though it's populated on the other you might prefer to create a separate zonegroup/zone for these non-replicated buckets. buckets created in that zonegroup would still be visible to the original two zones, but http requests would redirect to the correct zonegroup/zone so there'd be no confusion about what zones contain its object data. the downside is that it means deploying additional radosgws for this new zonegroup/zone at one of the sites and teaching clients/applications how to create buckets there in https://github.com/ceph/ceph/pull/62772 i've been exploring a "zone-local bucket" feature that uses zone redirects without requiring a separate zonegroup/zone, but the boto3 sdk doesn't properly handle these redirects - and that apparently applies to existing cross-zonegroup redirects as well On Sun, Dec 21, 2025 at 9:11 PM Christopher Durham via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > I have a multi site configuration with two sites that works just find for S3 bucket replication in both directions for the sites. - Rocky 9 and Ceph 19.2.2. > The customer now wishes to add some additional higher speed storage on one side that does not get replicated for the buckets on that storage. > My thought is that this should be a separate pool. and use bucket sync policy to prevent buckets created on that pool from replicating. But as I understand it, the bucket itself must exist on both sides. What do I do to properly create that bucket on the side that will not have the data? Do I do something clever with placement groups to have the bucket created on existing storage on the one side, but have it created on the new storage pool on the side where the new storage will be? > Thanks for any information you all may have > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Problems with orchestrator
by Carsten Götze 05 Jan '26

05 Jan '26

Hello! Here is the snippet from cephadm.log: cephadm ['--image', 'quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86', '--timeout', '895', '_orch', 'deploy', '--fsid', '9a7485f3-9530-11f0-8bb9-6c92cff2f5d0'] 2025-12-10 09:45:26,706 7e80250e7080 DEBUG Loaded deploy configuration: {'fsid': '9a7485f3-9530-11f0-8bb9-6c92cff2f5d0', 'name': 'nfs.rootfs.0.0.atlas.luxjmr', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_n ame': 'nfs.rootfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86'], 'rank': 0, 'rank_generation': 0, 'extra_container_args': None, 'extra_entry point_args': None}, 'config_blobs': {'pool': '.nfs', 'namespace': 'rootfs', 'userid': 'nfs.rootfs.0.0.atlas.luxjmr', 'extra_args': ['-N', 'NIV_EVENT'], 'files': {'ganesha.conf': '# This file is generated by cephadm.\nNFS_CORE_PARAM {\n Enable_NLM = false;\n Enable_RQUOTA = false;\n Protocols = 3, 4;\n mount_path_pseudo = true;\n Enable_UDP = false;\n NFS_Port = 2049;\n allow_set_io_flusher_fail = true;\n Monitoring_Port = 9587;\n}\n\nNFSv4 {\n Delegations = false;\n RecoveryBackend = "rados_cluster";\n Minor_Versions = 1, 2;\n Server_Scope = "9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-rootfs";\n IdmapConf = "/etc/ganesha/idmap.conf";\n}\n\nRADOS_KV {\n UserId = "nfs.rootfs.0.0.atlas.luxjmr";\n nodeid = 0;\n pool = ".nfs";\n namespace = "rootfs";\n}\n\nRADOS_URLS {\n UserId = "nfs.rootfs.0.0.atlas.luxjmr";\n watch_url = "rados://.nfs/rootfs/conf-nfs.rootfs";\n}\n\nRGW {\n cluster = "ceph";\n name = "client.nfs.rootfs.0.0.atlas.luxjmr-rgw";\n}\n\n%url rados://.nfs/rootfs/conf-nfs.rootfs', 'idmap.conf': ''}, 'config': '# minimal ceph.conf for 9a7485f3-9530-11f0-8bb9-6c92cff2f5d0\n[global]\n\tfsid = 9a7485f3-9530-11f0-8bb9-6c92cff2f5d0\n\tmon_host = [v2:134.169.37.21:3300/0,v1:134.169.37.21:6789/0] [v2:134.169.37.17:3300/0,v1:134.169.37.17:6789/0] [v2:134.169.37.12:3300/0,v1:134.169.37.12:6789/0] [v2:134.169.37.15:3300/0,v1:134.169.37.15:6789/0] [v2:134.169.37.14:3300/0,v1:134.169.37.14:6789/0]\n', 'keyring': '[client.nfs.rootfs.0.0.atlas.luxjmr]\nkey = AQAlMzlpAZ3EMBAAXYArqFVcLYC0ND5DjamvWw==\n', 'rgw': {'cluster': 'ceph', 'user': 'nfs.rootfs.0.0.atlas.luxjmr-rgw', 'keyring': '[client.nfs.rootfs.0.0.atlas.luxjmr-rgw]\nkey = AQAmMzlpyOP1BxAARjK7MCMV/u1CHCuCnCeqXg==\n'}}} 2025-12-10 09:45:26,706 7e80250e7080 DEBUG Determined image: 'quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86' 2025-12-10 09:45:26,735 7e80250e7080 INFO Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs-rootfs-0-0-atlas-luxjmr 2025-12-10 09:45:26,735 7e80250e7080 INFO /usr/bin/docker: stdout 2025-12-10 09:45:26,735 7e80250e7080 INFO /usr/bin/docker: stderr Error response from daemon: No such container: ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs-rootfs-0-0-atlas-luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO /usr/bin/docker: stdout 2025-12-10 09:45:26,748 7e80250e7080 INFO /usr/bin/docker: stderr Error response from daemon: No such container: ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO Deploy daemon nfs.rootfs.0.0.atlas.luxjmr ... 2025-12-10 09:45:26,925 7e80250e7080 INFO Verifying port 0.0.0.0:2049 ... 2025-12-10 09:45:26,927 7e80250e7080 INFO Creating ganesha config... 2025-12-10 09:45:26,927 7e80250e7080 INFO Write file: /var/lib/ceph/9a7485f3-9530-11f0-8bb9-6c92cff2f5d0/nfs.rootfs.0.0.atlas.luxjmr/etc/ganesha/ganesha.conf 2025-12-10 09:45:26,927 7e80250e7080 INFO Write file: /var/lib/ceph/9a7485f3-9530-11f0-8bb9-6c92cff2f5d0/nfs.rootfs.0.0.atlas.luxjmr/etc/ganesha/idmap.conf 2025-12-10 09:45:27,242 7e80250e7080 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:27,242 7e80250e7080 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service: Unit ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service not loaded. 2025-12-10 09:45:27,541 7e80250e7080 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0.target.wants/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service → /etc/systemd/system/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0@.service. 2025-12-10 09:45:27,572 7e80250e7080 DEBUG firewalld does not appear to be present 2025-12-10 09:45:27,573 7e80250e7080 DEBUG Not possible to enable service <nfs>. firewalld.service is not available 2025-12-10 09:45:27,573 7e80250e7080 DEBUG firewalld does not appear to be present 2025-12-10 09:45:27,573 7e80250e7080 DEBUG Not possible to open ports <[2049]>. firewalld.service is not available With best regards Carsten Götze ------------------------------------------------------------------ Carsten Goetze Computer Graphics tel: +49 531 391-2109 TU Braunschweig fax: +49 531 391-2103 Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > Am 03.01.2026 um 18:52 schrieb Salatiel Filho <salatiel.filho(a)gmail.com>: > > Does nfs deploy in ceph 20.2.0 requires firewalld? > Would you mind sharing some logs that complain about that ? > > Atenciosamente/Kind regards, > Salatiel > > On Tue, Dec 16, 2025 at 9:54 AM Carsten Götze via ceph-users > <ceph-users(a)ceph.io> wrote: >> >> Hi! >> >> I'm running a ceph cluster using cephadm and recently upgraded from squid to tentacle 20.2.0. >> Until recently everything worked fine until I started the nfs module. The nfs daemons were reported to be running but after some 10 minutes or so all of them were reported to be dead except one. nfs service on port 2049 was never provided on any of the nodes even while the daemons were supposed to be running. >> As i found out later, the nfs daemons were never started at all, because the setup process required a systemd-firewalld to be installed on the system which of course wasn't. >> After some headaches with the newly installed firewalld I decided to roll back, delete the firewalld and postpone the nfs deployment. >> I then tried to stop the nfs daemons with 'ceph orch daemon stop', which did nothing, even after waiting some 10 minutes. I had to reissue the command several times to make the reportedly dead nfs daemons vanish from the 'ceph orch ps' list. The one daemon that was reported to be still running however would only die after 'ceph orch daemon stop --force' and was in an 'error' state thereafter and could not be removed from the 'ceph orch ps' list by no means. So I decided to delete the managing nfs service from the 'ceph orch ls' list, in hope that it would also tear down the remaining nfs daemon. >> This obviously was a bad idea since, the service is now in the state of deleting. However it cannot be deleted, because there is still the one daemon in error state, which cannot be deleted because it was never running at all. >> As a last measure I forcefully removed the docker container on the node with the cephadm command, but even though there are no traces left of that nfs daemon, it is still listed when running 'ceph orch ps'. >> I also noticed that the 'ceph orch device ls' is out of sync with reality and 'ceph orch ps' is still listing osds that I've already shutdown and deleted. I therefore suspect, that the orchestrator has stopped collecting state information from the nodes. >> Is there a way to force the orchestrator to sync its state information with the nodes? >> Where do I find meaningful logs for the orchestrator? >> >> With best regards, >> Carsten Götze >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: standby MDSes ?
by Eugen Block 05 Jan '26

05 Jan '26

Hi, if you don't specifically assign an MDS to a CephFS, the standby daemons will be standby in general, so for all Filesystems. You can configure a FS affinity though [0]: > You may set mds_join_fs on the standby to enforce your preference: > $ ceph config set mds.b mds_join_fs cephfs So in your case, you set it like this: $ ceph config set mds.MONERO.podster3.gwtegj mds_join_fs MONERO Regarding the size you might need to provide more details, not sure what exactly you're referring to from your output. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephfs/standby/#configuring-mds-file-system… Zitat von lejeczek via ceph-users <ceph-users(a)ceph.io>: > Hi guys. > > Is this ok & expected that standby MDS for a filesystem looks as such: > > -> $ ceph fs status APKI > APKI - 4 clients > ==== > RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS > 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 5516 143 2757 > POOL TYPE USED AVAIL > cephfs.APKI.meta metadata 671M 74.0G > cephfs.APKI.data data 16.1G 74.0G > STANDBY MDS > APKI.podster3.utwwpp > MONERO.podster3.gwtegj > MDS version: ceph version 20.1.1 > (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - > RelWithDebInfo) > > What puzzles me from that above is: MONERO.podster3.gwtegj > I have only two files systems - this is a small lab with ceph - and > both were created with "default" ceph cmd, meaning that pools where > created automatically by cmd. > Is there no special and exclusive relationship between file system & > MDS - MONERO.podster3 was created when file system/volume MONERO was > created, yet it is available to a different file system? > > (what also boggles my mind is "size" the two file systems report, > like they share something, but not relevant yet in case I have > something seriously dis-configured relating what stat "status" showed) > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Override nfs parameter in NFS_CORE_PARAM
by Salatiel Filho 04 Jan '26

04 Jan '26

Hello , everyone. Apparently there is something wrong in the default config for ganesha nfs in tentacle. The nfs server will never start. ( https://tracker.ceph.com/issues/74191 / https://tracker.ceph.com/issues/74307 ) I've been able to successfully make the ganesha-server start by editing the ganesha.conf.j2 inside the ceph base container and replacing the line Protocols = 3,4 with Protocols = 4 inside the NFS_CORE_PARAM block. Unfortunately this temporary workaround is hard to maintain, so i have decided to look for a way to override the value in the ceph way of doing overrides. I used ceph nfs cluster config set main -i nfs-override.yml to override that parameter. NFS_CORE_PARAM { Protocols = 4; } I can see that the config is set by # ceph nfs cluster config get main NFS_CORE_PARAM { Protocols = 4; } But even after redeploy, the defaults of Protocols = 3,4 is never overridden. Any suggestions on how to override this? Atenciosamente/Kind regards, Salatiel

1 0

standby MDSes ?
by lejeczek 04 Jan '26

04 Jan '26

Hi guys. Is this ok & expected that standby MDS for a filesystem looks as such: -> $ ceph fs status APKI APKI - 4 clients ==== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 5516 143 2757 POOL TYPE USED AVAIL cephfs.APKI.meta metadata 671M 74.0G cephfs.APKI.data data 16.1G 74.0G STANDBY MDS APKI.podster3.utwwpp MONERO.podster3.gwtegj MDS version: ceph version 20.1.1 (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - RelWithDebInfo) What puzzles me from that above is: MONERO.podster3.gwtegj I have only two files systems - this is a small lab with ceph - and both were created with "default" ceph cmd, meaning that pools where created automatically by cmd. Is there no special and exclusive relationship between file system & MDS - MONERO.podster3 was created when file system/volume MONERO was created, yet it is available to a different file system? (what also boggles my mind is "size" the two file systems report, like they share something, but not relevant yet in case I have something seriously dis-configured relating what stat "status" showed) many thanks, L.

1 0

Re: OSDs not starting at boot after upgrade to Tentacle
by Chris Palmer 02 Jan '26

02 Jan '26

OK, found it. The ExecStart line in /usr/lib/systemd/system/ceph-volume@.service is wrong in Tentacle. squid: ExecStart=/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT /usr/sbin/ceph-volume-systemd %i' tentacle: ExecStart=/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT @CMAKE_INSTALL_PREFIX@/sbin/ceph-volume-systemd %i' Obviously the @CMAKE_INSTALL_PREFIX@ isn't going to work in there... Reinstating the line as in squid brings it back to life. On 02/01/2026 16:35, Chris Palmer via ceph-users wrote: > I've just upgraded a Centos 9 Stream test cluster from squid 19.2.3 to > tentacle 20.2.0. It is a non-cephadm package install. It all looked > pretty good, except that now no OSD processes start automatically > after a reboot: > > * There have been no other changes whatever to the system other than > ceph. > * OSD processes would start automatically on squid. > * After a reboot, all of the /var/lib/ceph/osd/ceph-n directories > still exist but are empty. > * A manual activation brings them all back to life (sudo ceph-volume > lvm activate --all). > * Nothing at all relevant in the ceph logs. > * After a reboot, and before manually activating, /run/systemd/system > is empty. After activation all of > the /run/systemd/system/ceph-osd.target.wants symlinks are back in > place, and the /var/lib/ceph/osd/ceph-n directories have been > repopulated. > * It is happening on every osd node. > * As expected, after 10 minutes the osds are marked out (unless noout > is set). > > I really can't spot what is causing this. Can anyone give me any > pointers? > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

OSDs not starting at boot after upgrade to Tentacle
by Chris Palmer 02 Jan '26

02 Jan '26

I've just upgraded a Centos 9 Stream test cluster from squid 19.2.3 to tentacle 20.2.0. It is a non-cephadm package install. It all looked pretty good, except that now no OSD processes start automatically after a reboot: * There have been no other changes whatever to the system other than ceph. * OSD processes would start automatically on squid. * After a reboot, all of the /var/lib/ceph/osd/ceph-n directories still exist but are empty. * A manual activation brings them all back to life (sudo ceph-volume lvm activate --all). * Nothing at all relevant in the ceph logs. * After a reboot, and before manually activating, /run/systemd/system is empty. After activation all of the /run/systemd/system/ceph-osd.target.wants symlinks are back in place, and the /var/lib/ceph/osd/ceph-n directories have been repopulated. * It is happening on every osd node. * As expected, after 10 minutes the osds are marked out (unless noout is set). I really can't spot what is causing this. Can anyone give me any pointers? Thanks, Chris

1 0

Re: CephFS performance issue with a certain application
by Martin Gerhard Loschwitz 01 Jan '26

01 Jan '26

Anthony, > Am 01.01.2026 um 03:01 schrieb Anthony D'Atri via ceph-users <ceph-users(a)ceph.io>: > > Forgive me if already asked, but: > > * Is the CephFS metadata pool on enterprise-class NVMe SSDs? Which model, and how many? Firmware up to date? all pools are within the same NVMes within the same RADOS cluster. There are also no separate devices for BlueStore WAL or any other metadata. > * Are those shared with other uses? Boot, OSD, etc? No. Ceph-OSDs are in exclusive use by RADOS: > * How many PGs in the metadata pool, across how many OSDs? Cluster has roughly 8200 PGs out of which the large majority is for the data and metadata pools. I will go find the exact numbers. Cluster is 12x16 15.56TB NVMes. > * Is the first CephFS data pool on SSDs, with setfattr layouts used to attach subdirectories to additional data pools? > Everything is on NVMes, but ttbomk, no setfattr was used in any of these. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

2 1

Re: CephFS performance issue with a certain application
by Anthony D'Atri 01 Jan '26

01 Jan '26

Forgive me if already asked, but: * Is the CephFS metadata pool on enterprise-class NVMe SSDs? Which model, and how many? Firmware up to date? * Are those shared with other uses? Boot, OSD, etc? * How many PGs in the metadata pool, across how many OSDs? * Is the first CephFS data pool on SSDs, with setfattr layouts used to attach subdirectories to additional data pools?

1 0

Re: CephFS performance issue with a certain application
by Georgi Steffenhagen 01 Jan '26

01 Jan '26

Hi Martin, Mark’s suggestions around multi-active MDSes and round-robin pinning of the subdirectories make a lot of sense for your layout and seem like a very promising next step, especially given the large, mostly static directory fan-out. One thing that may be worth looking at in parallel, just to broaden the picture a bit, are two additional areas that sometimes show up in similar CephFS workloads: First, even when MDS CPU usage looks perfectly fine, some of the latency patterns you describe (especially gaps between metadata steps) can come from the MDS waiting on journal commits or metadata-pool RADOS operations. It might be useful to correlate those stalls with OSD-side commit/apply latency, RocksDB/WAL behavior, or any background activity (scrubs, recovery) affecting the metadata pool specifically. Second, on the client side, highly parallel write/replace workloads can occasionally amplify implicit synchronization points. Even if the application logic is straightforward, it can be worth double-checking whether the Java/NIO path introduces fsync/force calls, atomic moves, or similar semantics that could turn metadata operations into synchronous commit points under load. These don’t contradict the multi-MDS approach at all, but together with the pinning experiments they might help narrow down whether the dominant bottleneck is namespace contention, metadata-pool latency, or an interaction between the two. Hope this is useful, and good luck with the next round of testing. Best :) Georgi Steffenhagen Storage Architecture On 31 December 2025 16:48:59 CET, Mark Nelson via ceph-users <ceph-users(a)ceph.io> wrote: >Hi Martin, > >How many active MDS processes are you running? Given you have so many sub-directories and they appear to be static, you may want to set up a new parallel test directory where you have all 65535 sub-directories round-robin pinned across multiple active MDSes and see how that compares to the existing configuration. You can achieve real gains this way, though multi-active MDSes may hit some bugs that aren't present in single-active MDS setups. > >Since you are not seeing obvious CPU load on any of the existing MDS processes, I would also be curious what a wallclock profile of the authoritative MDS for the parent directory would show. You can try mine if you'd like, it's available here: > >https://github.com/markhpc/uwpmp > >You'll need debug symbols installed. The libunwind backend is the most reliable so I would stick with that even though it is slower than libdw. > > >Thanks, >Mark > >On 12/31/25 2:50 AM, Martin Gerhard Loschwitz via ceph-users wrote: >> Folks, >> >> i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: >> >> * Read a file from the CephFS >> * Modify it >> * Write it back to the MDS in a new path with changed content >> >> Average file size is somewhere in the ballpark of 200-300kb. >> >> The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. >> >> We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. >> >> We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. >> >> We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. >> >> I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! >> >> Best regards >> Martin >> >-- >Best Regards, >Mark Nelson >Head of R&D (USA) > >Clyso GmbH >p: +49 89 21552391 12 >a: Loristraße 8 | 80335 München | Germany >w: https://clyso.com | e: mark.nelson(a)clyso.com > >We are hiring: https://www.clyso.com/jobs/ >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0