- ceph-users - lists.ceph.io

Re: Squid Cephadm deploys Quincy MDS
by Eugen Block 09 Jan '26

09 Jan '26

Hi, ensure you have configured the right base image. Check with: ceph config ls | grep container_image container_image And set it to a specific version with: ceph config set global container_image <version> Zitat von Chris Palmer via ceph-users <ceph-users(a)ceph.io>: > I'm working my way through adopting a test Squid cluster. MONs, > MGRs, OSDs are now adopted ok. The cluster was installed from > scratch on Squid. > > I just tried adopting the MDS services. Adding the new containerized > ones appeared to work, except... > > ceph versions: > > "mds": { > "ceph version 19.2.3 > (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 > }, > > ceph orch apply mds cephfs "c1,c2" > > ceph versions: > > "mds": { > "ceph version 17.0.0-7183-g54142666 > (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, > "ceph version 19.2.3 > (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 > }, > > > How have I ended up with quincy ones??? > > At this stage they were only standby so have stopped them... > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Squid Cephadm deploys Quincy MDS
by Chris Palmer 09 Jan '26

09 Jan '26

I'm working my way through adopting a test Squid cluster. MONs, MGRs, OSDs are now adopted ok. The cluster was installed from scratch on Squid. I just tried adopting the MDS services. Adding the new containerized ones appeared to work, except... ceph versions: "mds": { "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 }, ceph orch apply mds cephfs "c1,c2" ceph versions: "mds": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 }, How have I ended up with quincy ones??? At this stage they were only standby so have stopped them... Thanks, Chris

1 0

Re: Cannot adopt osd (on squid)
by Chris Palmer 09 Jan '26

09 Jan '26

On 09/01/2026 13:22, Loïc Tortay via ceph-users wrote: > On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: >> I have a squid package-installed test cluster that I am trying to >> adopt into cephadm. >> >> Monitors & managers have been adopted, and the orchestrator is >> functioning. >> >> When I try to adopt an OSD though (e.g. osd.0), cephadm creates the >> new directory (<fsid>/osd.0) and populates the files from the legacy >> directory. It then crashes complaining that the directory osd/ceph-0 >> is not empty so cannot be removed. In some cases none of the files >> have been removed, and in others the block symlink and >> require_osd_release file remain. I can remove the new <fsid>/osd.0 >> and use ceph-volume to activate the OSD again. >> >> cephadm.log reports that every file has been moved and the ownership >> set - even for files that still remain in the legacy directory. It >> doesn't report any other errors (apart from the directory not empty >> error. when it crashes). >> >> I really can't find where to go from here. Any pointers appreciated... >> > > Hello, > We've had the same issue, with clusters initially installed/managed > with Ceph-Ansible. > > The issue (w/ Ceph-Ansible) seems to be related to some failure during > the initial configuration which left multiple mounts on the same > "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active > mountpoint therefore it can't be removed). > "mount|grep /var/lib/ceph|sort|uniq -c|sort -n" shows which OSDs > have multiple mounts (first field). > > After the failed adoption, our workaround/recovery procedure was the > following ($NOK is the OSD number): > # Move the configuration out of the way (do not remove it) > mv -v /var/lib/ceph/$CLUSTERID/osd.$NOK /var/lib/ceph/osd.$NOK.nok > # If there are more than 2 mounts on the same directory, umount all > but # one to allow the adoption process to complete (the configuration > will # not be valid, the OSD won't start) > cephadm adopt --style legacy --name osd.$NOK > # Copy the correct configuration files to the adopted OSD directory > cd /var/lib/ceph/$CLUSTERID/osd.$NOK > cp -v ../../osd.$NOK.nok/{keyring,fsid} . > # $OKOSD is the number of a successfully adopted OSD. > cp -v ../../osd.$OKOSD/config . > # > # Check the "block*" links point to the same devices as in the first > # directory ("/var/lib/ceph/osd.$NOK.nok"), correct the links if > # they're wrong > # > # Make sure the files & links belong the correct UID for the containers > chown -hv 167:167 block* > chown -v 167:167 * > > systemctl status ceph-$CLUSTERID(a)osd.$NOK.service > systemctl start ceph-$CLUSTERID(a)osd.$NOK.service > systemctl status ceph-$CLUSTERID(a)osd.$NOK.service > > Other things we have noted after adoptions are: > - do *not* remove the packages used before the adoption, they (at > least the RPMs) have pre-remove scripts which stop the OSDs/Ceph on > the node > - cleanup "logrotate" conflicting configurations from the packages & > containers (i.e. remove the configuration files provided by the packages) > > > Loïc. Thanks for the info, and the last two hints! Will have to watch out for those... This test cluster was freshly built on squid 19.2.3 for testing this migration, purely using the manual install method (not ceph-ansible). I've already triple-checked the mounts, symlinks, ownerships, etc, and we don't have any of those problems. Some progress though. The problem is not entirely repeatable. The files are always replicated into the new directory correctly. Sometimes no files are removed from the legacy directory (and it fails as it is not empty), sometimes some are left (and it fails), and sometimes it works. Even on the same OSD. If it fails, I can manually remove the files from the legacy directory, and repeat the adopt. It then completes, removing the legacy directory, although the OSD is still stopped. Manually starting it using "ceph orch daemon start osd.n" works. I've also noticed that sometimes it takes a while to appear in "ceph orch ps", and I have to wait until it does before starting it. I'm going to do some reboot and daemon restart tests, then continue on the rest of the OSDs.... Thanks, Chris

1 0

Re: Cannot adopt osd (on squid)
by Loïc Tortay 09 Jan '26

09 Jan '26

On 09/01/2026 14:22, Loïc Tortay via ceph-users wrote: > On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: >> I have a squid package-installed test cluster that I am trying to >> adopt into cephadm. >> >> Monitors & managers have been adopted, and the orchestrator is >> functioning. >> >> When I try to adopt an OSD though (e.g. osd.0), cephadm creates the >> new directory (<fsid>/osd.0) and populates the files from the legacy >> directory. It then crashes complaining that the directory osd/ceph-0 >> is not empty so cannot be removed. In some cases none of the files >> have been removed, and in others the block symlink and >> require_osd_release file remain. I can remove the new <fsid>/osd.0 and >> use ceph-volume to activate the OSD again. >> >> cephadm.log reports that every file has been moved and the ownership >> set - even for files that still remain in the legacy directory. It >> doesn't report any other errors (apart from the directory not empty >> error. when it crashes). >> >> I really can't find where to go from here. Any pointers appreciated... >> > > Hello, > We've had the same issue, with clusters initially installed/managed with > Ceph-Ansible. > > The issue (w/ Ceph-Ansible) seems to be related to some failure during > the initial configuration which left multiple mounts on the same > "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active mountpoint > therefore it can't be removed). > Of course this is the "/var/lib/ceph/osd/ceph-$OSD" directory which has multiple mounts, not the one used by the container. Loïc.-- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Re: Cannot adopt osd (on squid)
by Loïc Tortay 09 Jan '26

09 Jan '26

On 09/01/2026 13:15, Chris Palmer via ceph-users wrote: > I have a squid package-installed test cluster that I am trying to adopt > into cephadm. > > Monitors & managers have been adopted, and the orchestrator is functioning. > > When I try to adopt an OSD though (e.g. osd.0), cephadm creates the new > directory (<fsid>/osd.0) and populates the files from the legacy > directory. It then crashes complaining that the directory osd/ceph-0 is > not empty so cannot be removed. In some cases none of the files have > been removed, and in others the block symlink and require_osd_release > file remain. I can remove the new <fsid>/osd.0 and use ceph-volume to > activate the OSD again. > > cephadm.log reports that every file has been moved and the ownership set > - even for files that still remain in the legacy directory. It doesn't > report any other errors (apart from the directory not empty error. when > it crashes). > > I really can't find where to go from here. Any pointers appreciated... > Hello, We've had the same issue, with clusters initially installed/managed with Ceph-Ansible. The issue (w/ Ceph-Ansible) seems to be related to some failure during the initial configuration which left multiple mounts on the same "/var/lib/ceph/$CLUSTERID/osd.$OSD" directory (it's an active mountpoint therefore it can't be removed). "mount|grep /var/lib/ceph|sort|uniq -c|sort -n" shows which OSDs have multiple mounts (first field). After the failed adoption, our workaround/recovery procedure was the following ($NOK is the OSD number): # Move the configuration out of the way (do not remove it) mv -v /var/lib/ceph/$CLUSTERID/osd.$NOK /var/lib/ceph/osd.$NOK.nok # If there are more than 2 mounts on the same directory, umount all but # one to allow the adoption process to complete (the configuration will # not be valid, the OSD won't start) cephadm adopt --style legacy --name osd.$NOK # Copy the correct configuration files to the adopted OSD directory cd /var/lib/ceph/$CLUSTERID/osd.$NOK cp -v ../../osd.$NOK.nok/{keyring,fsid} . # $OKOSD is the number of a successfully adopted OSD. cp -v ../../osd.$OKOSD/config . # # Check the "block*" links point to the same devices as in the first # directory ("/var/lib/ceph/osd.$NOK.nok"), correct the links if # they're wrong # # Make sure the files & links belong the correct UID for the containers chown -hv 167:167 block* chown -v 167:167 * systemctl status ceph-$CLUSTERID(a)osd.$NOK.service systemctl start ceph-$CLUSTERID(a)osd.$NOK.service systemctl status ceph-$CLUSTERID(a)osd.$NOK.service Other things we have noted after adoptions are: - do *not* remove the packages used before the adoption, they (at least the RPMs) have pre-remove scripts which stop the OSDs/Ceph on the node - cleanup "logrotate" conflicting configurations from the packages & containers (i.e. remove the configuration files provided by the packages) Loïc. -- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Cannot adopt osd (on squid)
by Chris Palmer 09 Jan '26

09 Jan '26

I have a squid package-installed test cluster that I am trying to adopt into cephadm. Monitors & managers have been adopted, and the orchestrator is functioning. When I try to adopt an OSD though (e.g. osd.0), cephadm creates the new directory (<fsid>/osd.0) and populates the files from the legacy directory. It then crashes complaining that the directory osd/ceph-0 is not empty so cannot be removed. In some cases none of the files have been removed, and in others the block symlink and require_osd_release file remain. I can remove the new <fsid>/osd.0 and use ceph-volume to activate the OSD again. cephadm.log reports that every file has been moved and the ownership set - even for files that still remain in the legacy directory. It doesn't report any other errors (apart from the directory not empty error. when it crashes). I really can't find where to go from here. Any pointers appreciated... Thanks, Chris

1 0

RGW crashes: caught signal abort, failed to read body: partial message
by Christopher James 08 Jan '26

08 Jan '26

Hi, In our ceph cluster deployment, reef 18.2.7, rgw daemons keep crashing with the following trace. A abort signal is caught by the rgws. in thread 7f2b3eb71640 thread_name:notif-worker0 ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f2bb2fe0bf0] 2: /lib64/libc.so.6(+0x8bf5c) [0x7f2bb302df5c] 3: raise() 4: abort() 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2bb3342b21] 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2bb334e53c] 7: /lib64/libstdc++.so.6(+0xac509) [0x7f2bb334d509] 8: __gxx_personality_v0() 9: /lib64/libgcc_s.so.1(+0x112d4) [0x7f2bb31bd2d4] 10: _Unwind_Resume() 11: /usr/bin/radosgw(+0x2ddb93) [0x55677058db93] 12: /usr/bin/radosgw(+0x3cd8cd) [0x55677067d8cd] 13: /lib64/libstdc++.so.6(+0xdbae4) [0x7f2bb337cae4] 14: /lib64/libc.so.6(+0x8a21a) [0x7f2bb302c21a] 15: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. When I grep the thread ID which is getting abort signal, these are the last logs: 2026-01-08T13:28:56.614+0000 7f2b3eb71640 -1 *** Caught signal (Aborted) ** in thread 7f2b3eb71640 thread_name:notif-worker0 -78> 2026-01-08T13:28:56.610+0000 7f2b3eb71640 4 failed to read body: partial message 0> 2026-01-08T13:28:56.614+0000 7f2b3eb71640 -1 *** Caught signal (Aborted) ** in thread 7f2b3eb71640 thread_name:notif-worker0 7f2b3eb71640 / notif-worker0 the failed to read body: partial mesage in this log is related to a GET request. I found a bugzilla in the redhat forum which has the same trace but occurs when a CopyObject is done on an encrypted object. In our cluster SSE is not enabled. So I don't think that is related. Just for ref the link to the bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2362258 Has anybody faced this issue before? Could anybody help with this? Regards Chris

1 0

Re: Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Janne Johansson 08 Jan '26

08 Jan '26

> Let’s pretend my original mail had never existed … > and for those running into a similar issue: Ensure the PG autoscaler is actually working properly or else your pools will have … limited numbers of PGs. Thanks for letting us know. -- May the most significant bit of your life be positive.

1 0

Re: Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Martin Gerhard Loschwitz 08 Jan '26

08 Jan '26

Let’s pretend my original mail had never existed … and for those running into a similar issue: Ensure the PG autoscaler is actually working properly or else your pools will have … limited numbers of PGs. M. -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Performance issue: NVMe-only clusters with BS=4K/QD=256 benchmark
by Martin Gerhard Loschwitz 08 Jan '26

08 Jan '26

Folks, on two completely unrelated NVMe-only clusters, I am seeing detrimental performance for benchmarks with QD=256 and BS=4K: Where a comparable SAS-SSD cluster has 95k IOPS, with NVMes, I see only 7000 (one cluster) and 14000 (for the other one). I must be overseeing something very obvious here. What I have found out that „ceph tell osd bench“ for all OSDs is fine and within the expectations (50k+ IOPS) and network is fine (24,7Gbit/s). Benchmark is performed using „rados bench“ with „-t 256“ and -b 4K“. Total number of OSDs is 12. Dell Enterprise NVMes, 15.6TB per piece. During benchmarks, it looks as if only one OSD is even active on each of the nodes according to htop and iostat. The other ones appear to be inactive. I’m going nuts over this … What is going on here? :( Thank you very much in advance. M. -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0