- ceph-users - lists.ceph.io

Re: NVMe-first crush rule
by Eugen Block 13 Jan '26

13 Jan '26

Hi, first one note: mixing device classes as you plan will only be beneficial for reads, those will be faster. But writes still need to be acked from all replicas before acked to the client. For the rest see responses inline. Zitat von Jan Kasprzak via ceph-users <ceph-users(a)ceph.io>: > Hello, Ceph users, > > how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, > with all replicas on different hosts? > > One of my clusters has been upgraded to two brand new ~6TB NVMe disks > together with previously used two 12TB HDDs per host. > Right now I have two HDD-based OSDs per host, each with ~200GB partition > on NVMe for metadata. The rest of NVMe is free, so I can now create > two additional ~6TB NVMe OSDs per host. > > My question is how to create a CRUSH rule which would have three replicas > on three different hosts, with one of these replicas on NVMe, and the > remaining two on HDD. > > In the docs here: > https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… > there is an example of a SSD-first rule, which is close to what I would like > to do, but it seems the SSD replica in that rule can end up on the same host > as one of the HDD replicas. How can I avoid that? I assume you're referring to this rule: step take default class ssd step chooseleaf firstn 1 type host step emit step take default class hdd step chooseleaf firstn 0 type host step emit The first 'step chooseleaf firstn 1 type host' advises CRUSH to select an OSD on one (exactly 1) host within the SSD class. Then it has to select the remaining hosts from the HDD class ('firstn 0' means the remaining number of replicas to distribute). Since you specify your failure domain to "host" (by using "type host" within the crush rule), all replicas of the same PG will be distributed across different hosts. > And a noob question -- how can I create a crush rule from the > "rule something { ... }" text in that example? If you need some advanced rule, you're going to have to use crushtool on the extracted crushmap. I usually start by creating a rule via CLI, then extract the crushmap, decompile it, make changes to it, compile and test it before injecting it back into the cluster. For example: # create simple rule ceph osd crush rule create-replicated test-rack-rule default rack hdd # dump crushmap ceph osd getcrushmap -o crushmap.bin # decompile crushtool -d crushmap.bin -o crushmap.txt # edit rule with any text editor # compile crushtool -c crushmap.bin -o crushmap.new.bin # test crushtool -i crushmap.new.bin --test --num-rep 3 --rule <RULE ID> --show-bad-mappings crushtool -i crushmap.new.bin --test --num-rep 3 --rule <RULE ID> --show-mappings | less If --show-bad-mappings does show any output, you made a mistake somewhere. With --show-mappings you can check the distribution of the PGs, the result is an acting set of OSDs, e.g.: CRUSH rule 5 x 0 [6,3,7] Then you can check the OSD's device class: for i in {6,3,7}; do ceph osd metadata $i | jq -r '.bluestore_bdev_type'; done hdd hdd hdd > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

Downgrading from Tentacle to Squid
by Michael Worsham 13 Jan '26

13 Jan '26

Currently in a test-lab / sandbox environment, I am running the latest version of Ceph Tentacle release (v20.2.0) in a virtualized environment. If I wanted to downgrade back to Squid (v19.2.3), what steps would I go about doing that? I am not worried about any of the data in the environment as its strictly for testing purposes, patch assessment for OS and Ceph, etc. ubuntu@ceph-mon1:~$ sudo ceph versions { "mon": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "mgr": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "osd": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 3 }, "mds": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 1 }, "rgw": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "overall": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 10 } } Environment is running Ubuntu v22.04 and Ceph was deployed via cephadm. -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Re: NVMe-first crush rule
by Reed Dier 13 Jan '26

13 Jan '26

So I have a bit of an ugly setup because of history... That said, this is what I/we call a hybrid pool with the first/primary PG on SSD, and secondary/tertiary replicas of the PG on HDD. My SSD and HDD OSD's are in physically separate hosts, no intermingling of media, which may make things more tricky. And because this was all stood up ages ago before device classes were in ceph, the SSD hosts are in a totally separate root in crush. So in my scenario, I have it take from a host in the ssd root. And for the second take, it chooses two hosts from the hdd root. > { > "rule_id": {{ HYBRID RULE ID }}, > "rule_name": "{{ HYBRID RULE NAME }}", > "type": 1, > "steps": [ > { > "op": "take", > "item": {{ SSD ROOT NUM }}, > "item_name": "{{ SSD ROOT }}" > }, > { > "op": "chooseleaf_firstn", > "num": 1, > "type": "{{ SSD FAILURE DOMAIN }}" > }, > { > "op": "emit" > }, > { > "op": "take", > "item": -1, > "item_name": "{{ HDD ROOT }}" > }, > { > "op": "chooseleaf_firstn", > "num": -1, > "type": "{{ HDD FAILURE DOMAIN }}" > }, > { > "op": "emit" > } > ] > }, Since these are intermingled device classes intra-host. It might just be easiest to create a small pool with a few PGs, and then look to see what the up PG set is > ceph pg dump | grep ^${newPoolID}.0 | awk '{print $17}' And compare the array of OSDs to see if the OSDs are actually distributed across different hosts or if it places two on the same host. Reed > On Jan 12, 2026, at 1:53 PM, Jan Kasprzak via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello, Ceph users, > > how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, > with all replicas on different hosts? > > One of my clusters has been upgraded to two brand new ~6TB NVMe disks > together with previously used two 12TB HDDs per host. > Right now I have two HDD-based OSDs per host, each with ~200GB partition > on NVMe for metadata. The rest of NVMe is free, so I can now create > two additional ~6TB NVMe OSDs per host. > > My question is how to create a CRUSH rule which would have three replicas > on three different hosts, with one of these replicas on NVMe, and the > remaining two on HDD. > > In the docs here: > https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… > there is an example of a SSD-first rule, which is close to what I would like > to do, but it seems the SSD replica in that rule can end up on the same host > as one of the HDD replicas. How can I avoid that? > > And a noob question -- how can I create a crush rule from the > "rule something { ... }" text in that example? > > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [Nautilus][Upgrade] OS migration and Ceph upgrade best practices?
by Janne Johansson 13 Jan '26

13 Jan '26

> We are planning to migrate and upgrade an existing production Ceph cluster and > would like to confirm the recommended approach. > > Current setup: > - OS: CentOS 7.9 > - Ceph: Nautilus 14.2.22 > - Deployment: bare metal > - Services: OSD, MON and RGW > > Goals: > - Move away from CentOS 7 (EOL) > - Upgrade Ceph to a supported release > - Use Ubuntu LTS as the target OS > > Our current plan is: > 1. Introduce new nodes running Ubuntu 22.04 > 2. Gradually replace CentOS 7 nodes while keeping Ceph at Nautilus > 3. After OS migration is complete, upgrade Ceph step by step > (Nautilus -> Octopus -> Pacific -> Quincy / Reef) > > We would like to confirm: > 1. Is node replacement generally preferred over in-place OS upgrades for > production Ceph clusters? > 2. Is Ubuntu 22.04 currently the recommended OS for production deployments? > 3. Is Ubuntu 24.04 still considered early for Ceph production use? > > Any feedback or experience would be appreciated. We have done the same, but in our case we did: Centos7 -> upgrade ceph to Octopus (latest one can find for C7) Switch OS one host at a time to Ubuntu 20.x (can install octopus) Upgrade ubuntu 20.x hosts to Quincy or Reef Upgrade ubuntu OS to 22.x (or 24.x) Upgrade to Squid,Tentacle or wherever you plan to end up. Takes a long time on the calendar but works. -- May the most significant bit of your life be positive.

1 0

Re: Advice on managing adopted OSDs
by Eugen Block 13 Jan '26

13 Jan '26

Hi, that is expected since the orchestrator didn't create those OSDs, just adopted them. It won't be different in any other release. You can create a new spec file that reflects your current setup, the docs contain a couple of examples, here's a simple one: ---snip--- # cat osd-specs.yaml service_type: osd service_id: osd-hdd-ssd service_name: osd.osd-hdd-ssd placement: hosts: - host1 - host2 - host3 spec: data_devices: rotational: 1 size: 15G db_devices: rotational: 0 size: 13G:16G filter_logic: AND objectstore: bluestore ---snip--- Note that the above spec is a different one than in the following example. If you're sure that your new spec will match, you can apply it, but nothing will happen to the existing OSDs. If there are unused block devices that would apply to the specs, the orchestrator would try to deploy new OSDs. Once you have a different osd spec in place, you can set the appropriate string in the unit.meta file of the OSD: # grep service /var/lib/ceph/FSID/osd.0/unit.meta "service_name": "osd.osd-hdd-ssd", After a couple of minutes, the orchestrator will refresh its view on OSDs (transition from the unmanaged service to the managed one): # cat osd-standalone.yaml service_type: osd service_name: osd.standalone service_id: osd.standalone placement: label: osd spec: data_devices: rotational: 1 filter_logic: AND objectstore: bluestore After chaning the first OSD's unit.meta file NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 5 2s ago - <unmanaged> osd.osd.standalone 1 2s ago 5m label:osd As soon as all OSDs have been transitioned, the unmanaged service will vanish: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd.standalone 6 0s ago 8m label:osd I'm not sure if there's a smoother way, but that's how I've done it in the past because it's relatively quick (although it's been a while since I last adopted a prod cluster to cephadm). Regards, Eugen Zitat von Chris Palmer via ceph-users <ceph-users(a)ceph.io>: > I would appreciate advice on the "best" way to manage adopted OSDs. > > This is a test squid 19.2.3 cluster, originally installed using > centos 9 packages. I am testing the procedure for adopting, then > will use cephadm to update to tentacle. Once I am happy with the > whole process I will repeat on a small production cluster. > > I have successfully adopted mon, mgr, osd, rgw & mds as described in > the docs. That left rgw & mds managed, but mon, mgr & osds > unmanaged. Setting the mon & mgr services managed was easy by > tweaking the yml and applying. But I am unclear how to best handle > osds. That leaves me in this state: > > [root@c4 ~]# ceph orch ls > NAME PORTS RUNNING REFRESHED AGE PLACEMENT > mds.cephfs 2/2 51s ago 21h c1;c2 > mgr 3/3 52s ago 21h c1;c2;c4 > mon 3/3 51s ago 21h c1;c2;c3 > osd 12 51s ago - <unmanaged> > rgw.rgw ?:80 2/2 51s ago 21h c1;c2 > > > The exported yml had the following for osds, which had to be > commented out as it would not parse (missing placement, missing > data_devices etc): > > service_type: osd > service_name: osd > unmanaged: true > spec: > filter_logic: AND > objectstore: bluestore > > > My questions are: > > * With the OSDs unmanaged, what functionality am I missing? In > particular, things like using cephadm to replace a failed disk... > * These are only small clusters, so I am happy to have the OSDs > enumerated in the yml, and not have ceph try to find and > automatically use available disks. Is it possible to have the full > functionality except from automatically grabbing available disks? > * How can I generate yml that reflects the current state of the OSDs? > We currently have pools (per application) for NVMe, SSD, and HDD > with shared NVMe DB/WAL. > * I can't find documentation for transitioning OSDs from unmanaged to > managed. Have I just overlooked it? > * Is any part of this easier on tentacle than squid? If so I can > upgrade the package squid to tentacle before adopting. (But unless > there is a good reason, I would rather have familiarised myself with > the cephadm upgrade process before starting any of this on the > production cluster...) > > Many thanks for any insights. > > Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Advice on managing adopted OSDs
by Chris Palmer 13 Jan '26

13 Jan '26

I would appreciate advice on the "best" way to manage adopted OSDs. This is a test squid 19.2.3 cluster, originally installed using centos 9 packages. I am testing the procedure for adopting, then will use cephadm to update to tentacle. Once I am happy with the whole process I will repeat on a small production cluster. I have successfully adopted mon, mgr, osd, rgw & mds as described in the docs. That left rgw & mds managed, but mon, mgr & osds unmanaged. Setting the mon & mgr services managed was easy by tweaking the yml and applying. But I am unclear how to best handle osds. That leaves me in this state: [root@c4 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mds.cephfs 2/2 51s ago 21h c1;c2 mgr 3/3 52s ago 21h c1;c2;c4 mon 3/3 51s ago 21h c1;c2;c3 osd 12 51s ago - <unmanaged> rgw.rgw ?:80 2/2 51s ago 21h c1;c2 The exported yml had the following for osds, which had to be commented out as it would not parse (missing placement, missing data_devices etc): service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore My questions are: * With the OSDs unmanaged, what functionality am I missing? In particular, things like using cephadm to replace a failed disk... * These are only small clusters, so I am happy to have the OSDs enumerated in the yml, and not have ceph try to find and automatically use available disks. Is it possible to have the full functionality except from automatically grabbing available disks? * How can I generate yml that reflects the current state of the OSDs? We currently have pools (per application) for NVMe, SSD, and HDD with shared NVMe DB/WAL. * I can't find documentation for transitioning OSDs from unmanaged to managed. Have I just overlooked it? * Is any part of this easier on tentacle than squid? If so I can upgrade the package squid to tentacle before adopting. (But unless there is a good reason, I would rather have familiarised myself with the cephadm upgrade process before starting any of this on the production cluster...) Many thanks for any insights. Chris

1 0

Re: Change placement omf mds-daemons
by Nizamudeen A 13 Jan '26

13 Jan '26

Hi Johan, Thanks for raising this issue. We tested it yesterday and figured out that its an issue in 19.2.3 but fixed in later versions and it was a regression in that release alone. We opened a tracker: https://tracker.ceph.com/issues/74376 and a fix as well: https://github.com/ceph/ceph/pull/66884. We will soon get it merged in that branch. Will be fixed in the next release. I saw you also opened a separate tracker later, I'll probably close it as the duplicate of the one we created. Sorry to inform you late on this. but thanks for reporting. Regards, Nizam On Sat, Jan 10, 2026 at 10:57 PM Johan via ceph-users <ceph-users(a)ceph.io> wrote: > Ceph 19.2.3 using cephadm/docker > > I have tried to change placement of a mds to label using the dashboard. > When I do, ceph creates a new mds-service, mds.label, instead of > changing the original one. > > Im certain I have done this, using the dashboard, before on earlier > ceph-versions. > > I can without problem change the placement using the command-line > ceph orch apply mds filesystem --placement="label:mds" > > Anyone else seen this? Is this behaviour by the dashboard intentional or > a bug? > > /Johan > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Nizamudeen A Sr. Software Engineer - IBM Partner Engineer IBM and Red Hat Ceph Storage Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 0

Re: [Nautilus][Upgrade] OS migration and Ceph upgrade best practices?
by Anthony D'Atri 13 Jan '26

13 Jan '26

1 0

[Nautilus][Upgrade] OS migration and Ceph upgrade best practices?
by jie wang 12 Jan '26

12 Jan '26

Hi Ceph users, We are planning to migrate and upgrade an existing production Ceph cluster and would like to confirm the recommended approach. Current setup: - OS: CentOS 7.9 - Ceph: Nautilus 14.2.22 - Deployment: bare metal - Services: OSD, MON and RGW Goals: - Move away from CentOS 7 (EOL) - Upgrade Ceph to a supported release - Use Ubuntu LTS as the target OS Our current plan is: 1. Introduce new nodes running Ubuntu 22.04 2. Gradually replace CentOS 7 nodes while keeping Ceph at Nautilus 3. After OS migration is complete, upgrade Ceph step by step (Nautilus -> Octopus -> Pacific -> Quincy / Reef) We would like to confirm: 1. Is node replacement generally preferred over in-place OS upgrades for production Ceph clusters? 2. Is Ubuntu 22.04 currently the recommended OS for production deployments? 3. Is Ubuntu 24.04 still considered early for Ceph production use? Any feedback or experience would be appreciated. Thanks.

1 0

Re: CSC minutes 2026-01-12
by Gregory Farnum 12 Jan '26

12 Jan '26

On Mon, Jan 12, 2026 at 12:02 PM Patrick Donnelly via Dev <dev(a)ceph.io> wrote: > Hello, > > Today's CSC meeting minutes: > > * Status of Reef? Branch reports being locked, confounding docs > backports. From Ville "I was looking at a problem where docs for all > the EOL releases are missing the supposed warning admonition at the > top "This document is for an unsupported version of Ceph". Would have > wanted to get this fixed before Reef actual EOL." [aad] > + Patrick D. (me) sent an email about this separately to the dev list. > * (v18.2.8 QA): David still working on getting the lab on its feet to > run QA tests > + Postgres issue right now investigating > + Hopefully midweek it'll be operational > * Other Lab updates > + Folio + jenkins builders are being reracked > + Smithis next to be reracked/cabled > + http/https access to lab-extras repo is not accessible from the > lab due to security blocks by IBM This isn’t quite right and it’s important — http*s* acces is working fine; it’s just http port 80 we aren’t going to get. :) > - install-deps and ansible have been updated > * Ceph Community Manager Updates > + Teams webpage updated on Ceph.io - https://ceph.io/en/community/team/ > + CSC Election process document in progress. > - Seeking feedback on document from the Chair of Foundation > * CVE testing and testing releases with each Ceph distribution > + (present practice is to) use security list or maybe ceph-maintainers > to sync > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io > >

1 0