- ceph-users - lists.ceph.io

Re: Cannot adopt osd (on squid)
by Chris Palmer 09 Jan '26

09 Jan '26

On 09/01/2026 13:22, Loïc Tortay via ceph-users wrote: > On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: >> I have a squid package-installed test cluster that I am trying to >> adopt into cephadm. >> >> Monitors & managers have been adopted, and the orchestrator is >> functioning. >> >> When I try to adopt an OSD though (e.g. osd.0), cephadm creates the >> new directory (<fsid>/osd.0) and populates the files from the legacy >> directory. It then crashes complaining that the directory osd/ceph-0 >> is not empty so cannot be removed. In some cases none of the files >> have been removed, and in others the block symlink and >> require_osd_release file remain. I can remove the new <fsid>/osd.0 >> and use ceph-volume to activate the OSD again. >> >> cephadm.log reports that every file has been moved and the ownership >> set - even for files that still remain in the legacy directory. It >> doesn't report any other errors (apart from the directory not empty >> error. when it crashes). >> >> I really can't find where to go from here. Any pointers appreciated... >> > > Hello, > We've had the same issue, with clusters initially installed/managed > with Ceph-Ansible. > > The issue (w/ Ceph-Ansible) seems to be related to some failure during > the initial configuration which left multiple mounts on the same > "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active > mountpoint therefore it can't be removed). > "mount|grep /var/lib/ceph|sort|uniq -c|sort -n" shows which OSDs > have multiple mounts (first field). > > After the failed adoption, our workaround/recovery procedure was the > following ($NOK is the OSD number): > # Move the configuration out of the way (do not remove it) > mv -v /var/lib/ceph/$CLUSTERID/osd.$NOK /var/lib/ceph/osd.$NOK.nok > # If there are more than 2 mounts on the same directory, umount all > but # one to allow the adoption process to complete (the configuration > will # not be valid, the OSD won't start) > cephadm adopt --style legacy --name osd.$NOK > # Copy the correct configuration files to the adopted OSD directory > cd /var/lib/ceph/$CLUSTERID/osd.$NOK > cp -v ../../osd.$NOK.nok/{keyring,fsid} . > # $OKOSD is the number of a successfully adopted OSD. > cp -v ../../osd.$OKOSD/config . > # > # Check the "block*" links point to the same devices as in the first > # directory ("/var/lib/ceph/osd.$NOK.nok"), correct the links if > # they're wrong > # > # Make sure the files & links belong the correct UID for the containers > chown -hv 167:167 block* > chown -v 167:167 * > > systemctl status ceph-$CLUSTERID(a)osd.$NOK.service > systemctl start ceph-$CLUSTERID(a)osd.$NOK.service > systemctl status ceph-$CLUSTERID(a)osd.$NOK.service > > Other things we have noted after adoptions are: > - do *not* remove the packages used before the adoption, they (at > least the RPMs) have pre-remove scripts which stop the OSDs/Ceph on > the node > - cleanup "logrotate" conflicting configurations from the packages & > containers (i.e. remove the configuration files provided by the packages) > > > Loïc. Thanks for the info, and the last two hints! Will have to watch out for those... This test cluster was freshly built on squid 19.2.3 for testing this migration, purely using the manual install method (not ceph-ansible). I've already triple-checked the mounts, symlinks, ownerships, etc, and we don't have any of those problems. Some progress though. The problem is not entirely repeatable. The files are always replicated into the new directory correctly. Sometimes no files are removed from the legacy directory (and it fails as it is not empty), sometimes some are left (and it fails), and sometimes it works. Even on the same OSD. If it fails, I can manually remove the files from the legacy directory, and repeat the adopt. It then completes, removing the legacy directory, although the OSD is still stopped. Manually starting it using "ceph orch daemon start osd.n" works. I've also noticed that sometimes it takes a while to appear in "ceph orch ps", and I have to wait until it does before starting it. I'm going to do some reboot and daemon restart tests, then continue on the rest of the OSDs.... Thanks, Chris

1 0

Re: Cannot adopt osd (on squid)
by Loïc Tortay 09 Jan '26

09 Jan '26

On 09/01/2026 14:22, Loïc Tortay via ceph-users wrote: > On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: >> I have a squid package-installed test cluster that I am trying to >> adopt into cephadm. >> >> Monitors & managers have been adopted, and the orchestrator is >> functioning. >> >> When I try to adopt an OSD though (e.g. osd.0), cephadm creates the >> new directory (<fsid>/osd.0) and populates the files from the legacy >> directory. It then crashes complaining that the directory osd/ceph-0 >> is not empty so cannot be removed. In some cases none of the files >> have been removed, and in others the block symlink and >> require_osd_release file remain. I can remove the new <fsid>/osd.0 and >> use ceph-volume to activate the OSD again. >> >> cephadm.log reports that every file has been moved and the ownership >> set - even for files that still remain in the legacy directory. It >> doesn't report any other errors (apart from the directory not empty >> error. when it crashes). >> >> I really can't find where to go from here. Any pointers appreciated... >> > > Hello, > We've had the same issue, with clusters initially installed/managed with > Ceph-Ansible. > > The issue (w/ Ceph-Ansible) seems to be related to some failure during > the initial configuration which left multiple mounts on the same > "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active mountpoint > therefore it can't be removed). > Of course this is the "/var/lib/ceph/osd/ceph-$OSD" directory which has multiple mounts, not the one used by the container. Loïc.-- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Re: Cannot adopt osd (on squid)
by Loïc Tortay 09 Jan '26

09 Jan '26

On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: > I have a squid package-installed test cluster that I am trying to adopt > into cephadm. > > Monitors & managers have been adopted, and the orchestrator is functioning. > > When I try to adopt an OSD though (e.g. osd.0), cephadm creates the new > directory (<fsid>/osd.0) and populates the files from the legacy > directory. It then crashes complaining that the directory osd/ceph-0 is > not empty so cannot be removed. In some cases none of the files have > been removed, and in others the block symlink and require_osd_release > file remain. I can remove the new <fsid>/osd.0 and use ceph-volume to > activate the OSD again. > > cephadm.log reports that every file has been moved and the ownership set > - even for files that still remain in the legacy directory. It doesn't > report any other errors (apart from the directory not empty error. when > it crashes). > > I really can't find where to go from here. Any pointers appreciated... > Hello, We've had the same issue, with clusters initially installed/managed with Ceph-Ansible. The issue (w/ Ceph-Ansible) seems to be related to some failure during the initial configuration which left multiple mounts on the same "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active mountpoint therefore it can't be removed). "mount|grep /var/lib/ceph|sort|uniq -c|sort -n" shows which OSDs have multiple mounts (first field). After the failed adoption, our workaround/recovery procedure was the following ($NOK is the OSD number): # Move the configuration out of the way (do not remove it) mv -v /var/lib/ceph/$CLUSTERID/osd.$NOK /var/lib/ceph/osd.$NOK.nok # If there are more than 2 mounts on the same directory, umount all but # one to allow the adoption process to complete (the configuration will # not be valid, the OSD won't start) cephadm adopt --style legacy --name osd.$NOK # Copy the correct configuration files to the adopted OSD directory cd /var/lib/ceph/$CLUSTERID/osd.$NOK cp -v ../../osd.$NOK.nok/{keyring,fsid} . # $OKOSD is the number of a successfully adopted OSD. cp -v ../../osd.$OKOSD/config . # # Check the "block*" links point to the same devices as in the first # directory ("/var/lib/ceph/osd.$NOK.nok"), correct the links if # they're wrong # # Make sure the files & links belong the correct UID for the containers chown -hv 167:167 block* chown -v 167:167 * systemctl status ceph-$CLUSTERID(a)osd.$NOK.service systemctl start ceph-$CLUSTERID(a)osd.$NOK.service systemctl status ceph-$CLUSTERID(a)osd.$NOK.service Other things we have noted after adoptions are: - do *not* remove the packages used before the adoption, they (at least the RPMs) have pre-remove scripts which stop the OSDs/Ceph on the node - cleanup "logrotate" conflicting configurations from the packages & containers (i.e. remove the configuration files provided by the packages) Loïc. -- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Cannot adopt osd (on squid)
by Chris Palmer 09 Jan '26

09 Jan '26

I have a squid package-installed test cluster that I am trying to adopt into cephadm. Monitors & managers have been adopted, and the orchestrator is functioning. When I try to adopt an OSD though (e.g. osd.0), cephadm creates the new directory (<fsid>/osd.0) and populates the files from the legacy directory. It then crashes complaining that the directory osd/ceph-0 is not empty so cannot be removed. In some cases none of the files have been removed, and in others the block symlink and require_osd_release file remain. I can remove the new <fsid>/osd.0 and use ceph-volume to activate the OSD again. cephadm.log reports that every file has been moved and the ownership set - even for files that still remain in the legacy directory. It doesn't report any other errors (apart from the directory not empty error. when it crashes). I really can't find where to go from here. Any pointers appreciated... Thanks, Chris

1 0

RGW crashes: caught signal abort, failed to read body: partial message
by Christopher James 08 Jan '26

08 Jan '26

Hi, In our ceph cluster deployment, reef 18.2.7, rgw daemons keep crashing with the following trace. A abort signal is caught by the rgws. in thread 7f2b3eb71640 thread_name:notif-worker0 ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f2bb2fe0bf0] 2: /lib64/libc.so.6(+0x8bf5c) [0x7f2bb302df5c] 3: raise() 4: abort() 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2bb3342b21] 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2bb334e53c] 7: /lib64/libstdc++.so.6(+0xac509) [0x7f2bb334d509] 8: __gxx_personality_v0() 9: /lib64/libgcc_s.so.1(+0x112d4) [0x7f2bb31bd2d4] 10: _Unwind_Resume() 11: /usr/bin/radosgw(+0x2ddb93) [0x55677058db93] 12: /usr/bin/radosgw(+0x3cd8cd) [0x55677067d8cd] 13: /lib64/libstdc++.so.6(+0xdbae4) [0x7f2bb337cae4] 14: /lib64/libc.so.6(+0x8a21a) [0x7f2bb302c21a] 15: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. When I grep the thread ID which is getting abort signal, these are the last logs: 2026-01-08T13:28:56.614+0000 7f2b3eb71640 -1 *** Caught signal (Aborted) ** in thread 7f2b3eb71640 thread_name:notif-worker0 -78> 2026-01-08T13:28:56.610+0000 7f2b3eb71640 4 failed to read body: partial message 0> 2026-01-08T13:28:56.614+0000 7f2b3eb71640 -1 *** Caught signal (Aborted) ** in thread 7f2b3eb71640 thread_name:notif-worker0 7f2b3eb71640 / notif-worker0 the failed to read body: partial mesage in this log is related to a GET request. I found a bugzilla in the redhat forum which has the same trace but occurs when a CopyObject is done on an encrypted object. In our cluster SSE is not enabled. So I don't think that is related. Just for ref the link to the bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2362258 Has anybody faced this issue before? Could anybody help with this? Regards Chris

1 0

Re: Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Janne Johansson 08 Jan '26

08 Jan '26

> Let’s pretend my original mail had never existed … > and for those running into a similar issue: Ensure the PG autoscaler is actually working properly or else your pools will have … limited numbers of PGs. Thanks for letting us know. -- May the most significant bit of your life be positive.

1 0

Re: Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Martin Gerhard Loschwitz 08 Jan '26

08 Jan '26

Let’s pretend my original mail had never existed … and for those running into a similar issue: Ensure the PG autoscaler is actually working properly or else your pools will have … limited numbers of PGs. M. -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Martin Gerhard Loschwitz 08 Jan '26

08 Jan '26

Folks, on two completely unrelated NVMe-only clusters, I am seeing detrimental performance for benchmarks with QD=256 and BS=4K: Where a comparable SAS-SSD cluster has 95k IOPS, with NVMes, I see only 7000 (one cluster) and 14000 (for the other one). I must be overseeing something very obvious here. What I have found out that „ceph tell osd bench“ for all OSDs is fine and within the expectations (50k+ IOPS) and network is fine (24,7Gbit/s). Benchmark is performed using „rados bench“ with „-t 256“ and -b 4K“. Total number of OSDs is 12. Dell Enterprise NVMes, 15.6TB per piece. During benchmarks, it looks as if only one OSD is even active on each of the nodes according to htop and iostat. The other ones appear to be inactive. I’m going nuts over this … What is going on here? :( Thank you very much in advance. M. -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Re: Object replication issue between zones for RGW Account owned buckets
by Ashok Kumar M 08 Jan '26

08 Jan '26

Hi, I’m currently using Ceph version 19.2.3 and have a setup with two zones (zone-a and zone-b) under a single zonegroup (zonegroup-a) and realm (realm-a). When I replicate objects from bucket-a (in zone-a) to bucket-b (in zone-b) for buckets owned by a regular user (ex: my-user), the replication works successfully. However, replication does not work for buckets owned by the RGW Account (ex: RGW84754594545284760). In this case: There are no errors shown in the sync status on either zone. The replication status of the objects in the source bucket remains in a “PENDING” state indefinitely. Has anyone else encountered a similar issue, or is there any known workaround for this? I would really appreciate any kind of advice or guidance regarding this. Thanks, Ashok M Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 1

Has anyone been able to use NFS in tentacle or it is really broken?
by Salatiel Filho 07 Jan '26

07 Jan '26

Hi everyone, has anyone been able to use NFS in tentacle? The few people I know using it in squid said it has broken after tentacle upgrade. I have tried to create a NFS from scratch in a newly deployed instance of tentacle and it just does not work: The ganesha nfs server will keep dying. I've been able to workaround that by fully disabling the protocol 3 in the ganesha template directly in the tentacle image and deploying this new image, but I was expecting more people talking about this if it is a general problem. Maybe NFS is not being used anymore ? I can see a couple tickets in the issue tracker, but with no reply at all. Atenciosamente/Kind regards, Salatiel

1 0