Just to give it a bump, I created this tracker a while ago as a feature request. Creating crush rules in the dashboard could be a nice feature, I believe. https://tracker.ceph.com/issues/68459 Zitat von Eugen Block <eblock@nde.ag>:

Hi,

first one note: mixing device classes as you plan will only be beneficial for reads, those will be faster. But writes still need to be acked from all replicas before acked to the client.

For the rest see responses inline.

Zitat von Jan Kasprzak via ceph-users <ceph-users@ceph.io>:

...

Hello, Ceph users,

how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, with all replicas on different hosts?

One of my clusters has been upgraded to two brand new ~6TB NVMe disks together with previously used two 12TB HDDs per host. Right now I have two HDD-based OSDs per host, each with ~200GB partition on NVMe for metadata. The rest of NVMe is free, so I can now create two additional ~6TB NVMe OSDs per host.

My question is how to create a CRUSH rule which would have three replicas on three different hosts, with one of these replicas on NVMe, and the remaining two on HDD.

In the docs here: https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-rul... there is an example of a SSD-first rule, which is close to what I would like to do, but it seems the SSD replica in that rule can end up on the same host as one of the HDD replicas. How can I avoid that?

I assume you're referring to this rule:

step take default class ssd step chooseleaf firstn 1 type host step emit step take default class hdd step chooseleaf firstn 0 type host step emit

The first 'step chooseleaf firstn 1 type host' advises CRUSH to select an OSD on one (exactly 1) host within the SSD class. Then it has to select the remaining hosts from the HDD class ('firstn 0' means the remaining number of replicas to distribute). Since you specify your failure domain to "host" (by using "type host" within the crush rule), all replicas of the same PG will be distributed across different hosts.

...

And a noob question -- how can I create a crush rule from the "rule something { ... }" text in that example?

If you need some advanced rule, you're going to have to use crushtool on the extracted crushmap. I usually start by creating a rule via CLI, then extract the crushmap, decompile it, make changes to it, compile and test it before injecting it back into the cluster.

For example:

# create simple rule ceph osd crush rule create-replicated test-rack-rule default rack hdd

# dump crushmap ceph osd getcrushmap -o crushmap.bin

# decompile crushtool -d crushmap.bin -o crushmap.txt

# edit rule with any text editor

# compile crushtool -c crushmap.bin -o crushmap.new.bin

# test crushtool -i crushmap.new.bin --test --num-rep 3 --rule <RULE ID> --show-bad-mappings crushtool -i crushmap.new.bin --test --num-rep 3 --rule <RULE ID> --show-mappings | less

If --show-bad-mappings does show any output, you made a mistake somewhere. With --show-mappings you can check the distribution of the PGs, the result is an acting set of OSDs, e.g.:

CRUSH rule 5 x 0 [6,3,7]

Then you can check the OSD's device class:

for i in {6,3,7}; do ceph osd metadata $i | jq -r '.bluestore_bdev_type'; done hdd hdd hdd

...

Thanks!

-Yenya

-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io