- ceph-users - lists.ceph.io

CSC minutes 2026-01-12
by Patrick Donnelly 12 Jan '26

12 Jan '26

Hello, Today's CSC meeting minutes: * Status of Reef? Branch reports being locked, confounding docs backports. From Ville "I was looking at a problem where docs for all the EOL releases are missing the supposed warning admonition at the top "This document is for an unsupported version of Ceph". Would have wanted to get this fixed before Reef actual EOL." [aad] + Patrick D. (me) sent an email about this separately to the dev list. * (v18.2.8 QA): David still working on getting the lab on its feet to run QA tests + Postgres issue right now investigating + Hopefully midweek it'll be operational * Other Lab updates + Folio + jenkins builders are being reracked + Smithis next to be reracked/cabled + http/https access to lab-extras repo is not accessible from the lab due to security blocks by IBM - install-deps and ansible have been updated * Ceph Community Manager Updates + Teams webpage updated on Ceph.io - https://ceph.io/en/community/team/ + CSC Election process document in progress. - Seeking feedback on document from the Chair of Foundation * CVE testing and testing releases with each Ceph distribution + (present practice is to) use security list or maybe ceph-maintainers to sync -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

NVMe-first crush rule
by Jan Kasprzak 12 Jan '26

12 Jan '26

Hello, Ceph users, how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, with all replicas on different hosts? One of my clusters has been upgraded to two brand new ~6TB NVMe disks together with previously used two 12TB HDDs per host. Right now I have two HDD-based OSDs per host, each with ~200GB partition on NVMe for metadata. The rest of NVMe is free, so I can now create two additional ~6TB NVMe OSDs per host. My question is how to create a CRUSH rule which would have three replicas on three different hosts, with one of these replicas on NVMe, and the remaining two on HDD. In the docs here: https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… there is an example of a SSD-first rule, which is close to what I would like to do, but it seems the SSD replica in that rule can end up on the same host as one of the HDD replicas. How can I avoid that? And a noob question -- how can I create a crush rule from the "rule something { ... }" text in that example? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: Change placement omf mds-daemons
by Eugen Block 11 Jan '26

11 Jan '26

Yes, I can reproduce this on 19.2.3 and 19.2.1. On Tentacle it seems to work as expected though, the MDS service is changed to use label. The last Reef version (18.2.7) also just changes the service instead of creating a new one. So that seems to be a regression in Squid. I would recommend to create a tracker issue if there isn't one already. Zitat von Johan via ceph-users <ceph-users(a)ceph.io>: > Ceph 19.2.3 using cephadm/docker > > I have tried to change placement of a mds to label using the > dashboard. When I do, ceph creates a new mds-service, mds.label, > instead of changing the original one. > > Im certain I have done this, using the dashboard, before on earlier > ceph-versions. > > I can without problem change the placement using the command-line > ceph orch apply mds filesystem --placement="label:mds" > > Anyone else seen this? Is this behaviour by the dashboard > intentional or a bug? > > /Johan > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Change placement omf mds-daemons
by Johan 10 Jan '26

10 Jan '26

Ceph 19.2.3 using cephadm/docker I have tried to change placement of a mds to label using the dashboard. When I do, ceph creates a new mds-service, mds.label, instead of changing the original one. Im certain I have done this, using the dashboard, before on earlier ceph-versions. I can without problem change the placement using the command-line ceph orch apply mds filesystem --placement="label:mds" Anyone else seen this? Is this behaviour by the dashboard intentional or a bug? /Johan

1 0

Re: Question concerning selection of secondary OSDs
by Janne Johansson 10 Jan '26

10 Jan '26

> Furthermore, my expectation for a pool that has size=3 and min_size=2 is that for any given write, the primary OSD on nodes 1 or 2 will select a secondary OSD in node 1/2 respectively and one in node 3. Which would then lead me to believe that any client writing into the cluster from node 1 will only ever have the latency between node 1 and node 2 as an actual performance penalty because > > * client selects primary OSD on node 1 or node 2 > * primary OSD selects secondary OSDs and starts transfer in parallel > * Write to OSD with lower latency will finish much sooner than the one to the other OSD, leading to the write acknowledgement being sent to the client, because min_size=2 > > But that appears not to be the case. priority-affinity has a very slight impact, but the overall performance That is indeed not the case, the write will only be accepted as completed when all three replicas have ACKed the write. With your situation, it would be rather quick for a client to fill up any buffers on the fast OSD hosts so that the slow OSD host can't keep up, and you would be in a situation where the cluster "promises" to keep 3 copies but in reality you only have two, and if the client keeps on writing or changing data on the fast OSDs, you would be sure to have a really bad situation in case one of the two fast nodes restarts, since you now have a lot of PGs for which the slow OSD host has bad/old/stale data and the one remaining fast OSD host would give you repl=1 "safety" for those PGs. So ceph will wait until it has 3 ACKs before client gets the ACK. -- May the most significant bit of your life be positive.

1 0

Re: Question concerning selection of secondary OSDs
by Wesley Dillingham 09 Jan '26

09 Jan '26

"leading to the write acknowledgement being sent to the client, because min_size=2 A misunderstanding of min_size Ceph requires all replicas (or shards in EC) to ack the primary before the client is acknowledged. The primary OSD and therefore client are subjected to the latency of the hop between the primary and all replicas. min_size is how many up to date replicas are required for the PG to be active (accepts IO) Respectfully, *Wes Dillingham* On Fri, Jan 9, 2026 at 4:36 PM Martin Gerhard Loschwitz via ceph-users < ceph-users(a)ceph.io> wrote: > Folks, > > I am having a knowledge question concerning the selection of secondary > OSDs in Ceph. > > I have a cluster here that consists of three nodes. For the sake of the > argument, I have simulated latency between the third node and the other two > using tc and netem. I have set the priority-affinity of the OSDs on the > third node to 0, and indeed, RADOS is not using any of these OSDs as > primary OSD, so this part works as expected. > > Furthermore, my expectation for a pool that has size=3 and min_size=2 is > that for any given write, the primary OSD on nodes 1 or 2 will select a > secondary OSD in node 1/2 respectively and one in node 3. Which would then > lead me to believe that any client writing into the cluster from node 1 > will only ever have the latency between node 1 and node 2 as an actual > performance penalty because > > * client selects primary OSD on node 1 or node 2 > * primary OSD selects secondary OSDs and starts transfer in parallel > * Write to OSD with lower latency will finish much sooner than the one to > the other OSD, leading to the write acknowledgement being sent to the > client, because min_size=2 > > But that appears not to be the case. priority-affinity has a very slight > impact, but the overall performance when writing into the cluster with > queue depth 1 and request size of 4k still very much resembles a scenario > in which every single write appears to be latency-penalized with the > latency between node1/2 and node 3. > > Where is my understanding incorrect? Or are there any configuration > settings for this? I tried to search for this, but the only results I can > find refer to priority-affinity. I am looking into something like > „secondary affinity“ I guess, but I do not think that such a thing exists > in Ceph. Which leads me to believe that my understanding of this is > seriously wrong somehow. > > Any hint will be greatly appreciated. Thank you very much in advance. > > Best regards > Martin > > -- > Martin Gerhard Loschwitz > Geschäftsführer / CEO, True West IT Services GmbH > Phone: +49 2433 5253130 > Mobile: +49 176 61832178 > Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany > Legal: HRB 21985, Amtsgericht Mönchengladbach > VAT: DE363893844 > > True West IT Services GmbH is compliant with the GDPR regulation on data > protection and privacy in the European Union and the European Economic > Area. You can request the information on how we collect and process your > private data according to the law by contacting the email sender. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Question concerning selection of secondary OSDs
by Martin Gerhard Loschwitz 09 Jan '26

09 Jan '26

Folks, I am having a knowledge question concerning the selection of secondary OSDs in Ceph. I have a cluster here that consists of three nodes. For the sake of the argument, I have simulated latency between the third node and the other two using tc and netem. I have set the priority-affinity of the OSDs on the third node to 0, and indeed, RADOS is not using any of these OSDs as primary OSD, so this part works as expected. Furthermore, my expectation for a pool that has size=3 and min_size=2 is that for any given write, the primary OSD on nodes 1 or 2 will select a secondary OSD in node 1/2 respectively and one in node 3. Which would then lead me to believe that any client writing into the cluster from node 1 will only ever have the latency between node 1 and node 2 as an actual performance penalty because * client selects primary OSD on node 1 or node 2 * primary OSD selects secondary OSDs and starts transfer in parallel * Write to OSD with lower latency will finish much sooner than the one to the other OSD, leading to the write acknowledgement being sent to the client, because min_size=2 But that appears not to be the case. priority-affinity has a very slight impact, but the overall performance when writing into the cluster with queue depth 1 and request size of 4k still very much resembles a scenario in which every single write appears to be latency-penalized with the latency between node1/2 and node 3. Where is my understanding incorrect? Or are there any configuration settings for this? I tried to search for this, but the only results I can find refer to priority-affinity. I am looking into something like „secondary affinity“ I guess, but I do not think that such a thing exists in Ceph. Which leads me to believe that my understanding of this is seriously wrong somehow. Any hint will be greatly appreciated. Thank you very much in advance. Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Re: Squid Cephadm deploys Quincy MDS
by Loïc Tortay 09 Jan '26

09 Jan '26

On 09/01/2026 15:55, Chris Palmer via ceph-users wrote: > I'm working my way through adopting a test Squid cluster. MONs, MGRs, > OSDs are now adopted ok. The cluster was installed from scratch on Squid. > > I just tried adopting the MDS services. Adding the new containerized > ones appeared to work, except... > > ceph versions: > > "mds": { > "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > squid (stable)": 2 > }, > > ceph orch apply mds cephfs "c1,c2" > > ceph versions: > > "mds": { > "ceph version 17.0.0-7183-g54142666 > (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, > "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > squid (stable)": 2 > }, > > > How have I ended up with quincy ones??? > > At this stage they were only standby so have stopped them... > Hello, If you did not specify a specific image version, the global configuration is used and, if that is not defined, it will use a hardcoded image name: ceph config set global container_image quay.io/ceph/ceph:v19.2.3 See for instance: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/THAH2JFQNB7… (we have also had this issue while adopting a Quincy cluster) Once the image configuration is set, I think we just used "ceph orch daemon redeploy mds.$MDS_NAME" to get the correct version up and running. Loïc. -- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Re: Squid Cephadm deploys Quincy MDS
by Eugen Block 09 Jan '26

09 Jan '26

Hi, ensure you have configured the right base image. Check with: ceph config ls | grep container_image container_image And set it to a specific version with: ceph config set global container_image <version> Zitat von Chris Palmer via ceph-users <ceph-users(a)ceph.io>: > I'm working my way through adopting a test Squid cluster. MONs, > MGRs, OSDs are now adopted ok. The cluster was installed from > scratch on Squid. > > I just tried adopting the MDS services. Adding the new containerized > ones appeared to work, except... > > ceph versions: > > "mds": { > "ceph version 19.2.3 > (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 > }, > > ceph orch apply mds cephfs "c1,c2" > > ceph versions: > > "mds": { > "ceph version 17.0.0-7183-g54142666 > (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, > "ceph version 19.2.3 > (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 > }, > > > How have I ended up with quincy ones??? > > At this stage they were only standby so have stopped them... > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Squid Cephadm deploys Quincy MDS
by Chris Palmer 09 Jan '26

09 Jan '26

I'm working my way through adopting a test Squid cluster. MONs, MGRs, OSDs are now adopted ok. The cluster was installed from scratch on Squid. I just tried adopting the MDS services. Adding the new containerized ones appeared to work, except... ceph versions: "mds": { "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 }, ceph orch apply mds cephfs "c1,c2" ceph versions: "mds": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 2 }, How have I ended up with quincy ones??? At this stage they were only standby so have stopped them... Thanks, Chris

1 0