So I have a bit of an ugly setup because of history...
That said, this is what I/we call a hybrid pool with the first/primary PG on SSD, and secondary/tertiary replicas of the PG on HDD.
My SSD and HDD OSD's are in physically separate hosts, no intermingling of media, which may make things more tricky.
And because this was all stood up ages ago before device classes were in ceph, the SSD hosts are in a totally separate root in crush.

So in my scenario, I have it take from a host in the ssd root.
And for the second take, it chooses two hosts from the hdd root.

        {
            "rule_id": {{ HYBRID RULE ID }},
            "rule_name": "{{ HYBRID RULE NAME }}",
            "type": 1,
            "steps": [
                {
                    "op": "take",
                    "item": {{ SSD ROOT NUM }},
                    "item_name": "{{ SSD ROOT }}"
                },
                {
                    "op": "chooseleaf_firstn",
                    "num": 1,
                    "type": "{{ SSD FAILURE DOMAIN }}"
                },
                {
                    "op": "emit"
                },
                {
                    "op": "take",
                    "item": -1,
                    "item_name": "{{ HDD ROOT }}"
                },
                {
                    "op": "chooseleaf_firstn",
                    "num": -1,
                    "type": "{{ HDD FAILURE DOMAIN }}"
                },
                {
                    "op": "emit"
                }
            ]
        },

Since these are intermingled device classes intra-host.
It might just be easiest to create a small pool with a few PGs, and then look to see what the up PG set is
ceph pg dump | grep ^${newPoolID}.0 | awk '{print $17}' 
And compare the array of OSDs to see if the OSDs are actually distributed across different hosts or if it places two on the same host.

Reed

On Jan 12, 2026, at 1:53 PM, Jan Kasprzak via ceph-users <ceph-users@ceph.io> wrote:

Hello, Ceph users,

how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs,
with all replicas on different hosts?

One of my clusters has been upgraded to two brand new ~6TB NVMe disks
together with previously used two 12TB HDDs per host.
Right now I have two HDD-based OSDs per host, each with ~200GB partition
on NVMe for metadata. The rest of NVMe is free, so I can now create
two additional ~6TB NVMe OSDs per host.

My question is how to create a CRUSH rule which would have three replicas
on three different hosts, with one of these replicas on NVMe, and the
remaining two on HDD.

In the docs here:
https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-rules
there is an example of a SSD-first rule, which is close to what I would like
to do, but it seems the SSD replica in that rule can end up on the same host
as one of the HDD replicas. How can I avoid that?

And a noob question -- how can I create a crush rule from the
"rule something { ... }" text in that example?

Thanks!

-Yenya

--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| https://www.fi.muni.cz/~kas/                        GPG: 4096R/A45477D5 |
   I don't like Python; its lack of inline, anonymous, multi-statement
   functions makes me sad.                                --Eric Wastl
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io