Hi, check out the --reclassify parameter for crushtool, works like a charm. https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/L3XPGQMJ2Y32... Zitat von Jan Kasprzak via ceph-users <ceph-users@ceph.io>:

Hello, Ceph users,

In my Ceph cluster I have only OSDs of class "hdd" now. In preparation of adding also NVMe-based OSDs I wanted to prevent cluster rebalancing as soon as I start adding the new OSDs. So I thought adding "class hdd" to all my CRUSH rules would do what I want - i.e. pin all the PGs where they are, on HDD-based OSDs. I did the following:

ceph osd getcrushmap -o /var/tmp/crushmap.bin crushtool -d /var/tmp/crushmap.bin > /var/tmp/crushmap.txt vi /var/tmp/crushmap.txt

edit: - step take default + step take default class hdd

crushtool -c /var/tmp/crushmap.txt -o /var/tmp/crushmap-new.bin ceph osd setcrushmap -i /var/tmp/crushmap-new.bin

However, after doing this, my cluster reports 90 % PGs as remapped, and wants to completely rebalance itself. After reverting the crush map with

ceph osd setcrushmap -i /var/tmp/crushmap.bin

it quickly went back to a steady state.

How can I pin the PGs to HDD-based OSDs where they are, without the cluster rebalancing itself?

The whole CRUSH map can be seen here:

https://www.fi.muni.cz/~kas/tmp/crushmap.txt

Thanks!

-Yenya

Jan Kasprzak via ceph-users wrote:

...

Hello, Ceph users,

how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, with all replicas on different hosts?

One of my clusters has been upgraded to two brand new ~6TB NVMe disks together with previously used two 12TB HDDs per host. Right now I have two HDD-based OSDs per host, each with ~200GB partition on NVMe for metadata. The rest of NVMe is free, so I can now create two additional ~6TB NVMe OSDs per host.

My question is how to create a CRUSH rule which would have three replicas on three different hosts, with one of these replicas on NVMe, and the remaining two on HDD.

In the docs here: https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-rul... there is an example of a SSD-first rule, which is close to what I would like to do, but it seems the SSD replica in that rule can end up on the same host as one of the HDD replicas. How can I avoid that?

And a noob question -- how can I create a crush rule from the "rule something { ... }" text in that example?

Thanks!

-Yenya

-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io