Hello, Ceph users, In my Ceph cluster I have only OSDs of class "hdd" now. In preparation of adding also NVMe-based OSDs I wanted to prevent cluster rebalancing as soon as I start adding the new OSDs. So I thought adding "class hdd" to all my CRUSH rules would do what I want - i.e. pin all the PGs where they are, on HDD-based OSDs. I did the following: ceph osd getcrushmap -o /var/tmp/crushmap.bin crushtool -d /var/tmp/crushmap.bin > /var/tmp/crushmap.txt vi /var/tmp/crushmap.txt edit: - step take default + step take default class hdd crushtool -c /var/tmp/crushmap.txt -o /var/tmp/crushmap-new.bin ceph osd setcrushmap -i /var/tmp/crushmap-new.bin However, after doing this, my cluster reports 90 % PGs as remapped, and wants to completely rebalance itself. After reverting the crush map with ceph osd setcrushmap -i /var/tmp/crushmap.bin it quickly went back to a steady state. How can I pin the PGs to HDD-based OSDs where they are, without the cluster rebalancing itself? The whole CRUSH map can be seen here: https://www.fi.muni.cz/~kas/tmp/crushmap.txt Thanks! -Yenya Jan Kasprzak via ceph-users wrote:

Hello, Ceph users,

how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, with all replicas on different hosts?

One of my clusters has been upgraded to two brand new ~6TB NVMe disks together with previously used two 12TB HDDs per host. Right now I have two HDD-based OSDs per host, each with ~200GB partition on NVMe for metadata. The rest of NVMe is free, so I can now create two additional ~6TB NVMe OSDs per host.

My question is how to create a CRUSH rule which would have three replicas on three different hosts, with one of these replicas on NVMe, and the remaining two on HDD.

In the docs here: https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-rul... there is an example of a SSD-first rule, which is close to what I would like to do, but it seems the SSD replica in that rule can end up on the same host as one of the HDD replicas. How can I avoid that?

And a noob question -- how can I create a crush rule from the "rule something { ... }" text in that example?

Thanks!

-Yenya

-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl