Andreas, Anthony, Andreas Eriksson via ceph-users wrote:

I've done exactly this recently.

Changing crush rule to "step take default class hdd". And with 'nobackfill' set, I ran "pgremapper cancel-backfill --yes" two or three times to make the misplaced objects reduce from 95% -> 18%.

OK, this helped somewhat. Both pgremapper and upmap-remapped.py failed for me, I had to do this: ceph osd set-require-min-compat-client octopus Then I edited the crush rule for my biggest replicated pool adding "class hdd" to the line "step take default", compiled and set a new crush map, and ran pgremapper. Then there were no remapped pgs. So far good. Then I added "class hdd" to the crush rules of all the other pools as well and set a new crush map. After that, pgremapper printed the following type errors: WARNING: pg 77.1b: conflicting mapping 34->9 found when trying to map 39->34 WARNING: pg 71.12: conflicting mapping 8->21 found when trying to map 6->8 WARNING: pg 68.11: conflicting mapping 25->23 found when trying to map 37->25 WARNING: pg 68.11: conflicting mapping 4->0 found when trying to map 11->4 (many more probably for all PGs of these three pools, with pools 68, 71, and 77 being my erasure-coded pools). ./upmap-remapped.py | sh -x apparently does not print this type of error. Now I have 56 out of ~650 pgs remapped, and about 6 % of objects misplaced. Interestingly enough, most of the remapped pgs are from my main replicated pool, which were not remapped after the first run of pgremapper. But 6 % this is small enough, so I am letting it remap itself now. Thanks! -Yenya

This was on a Quincy (17.2.x) cluster.

On 1/19/26 5:48 PM, Anthony D'Atri via ceph-users wrote:

...

I’m surprised that this would cause data to move.

...

How can I pin the PGs to HDD-based OSDs where they are, without the cluster rebalancing itself? You can use upmap-remapped.py, though the balancer will progressively still move them if that’s what CRUSH really wants.

I have an RFE in to at the very least raise a warning if any rules/pools do not specify a device class, ideally setting one by default.

...

The whole CRUSH map can be seen here:

https://www.fi.muni.cz/~kas/tmp/crushmap.txt

Thanks!

-Yenya

Jan Kasprzak via ceph-users wrote:

...

Hello, Ceph users,

how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, with all replicas on different hosts?

One of my clusters has been upgraded to two brand new ~6TB NVMe disks together with previously used two 12TB HDDs per host. Right now I have two HDD-based OSDs per host, each with ~200GB partition on NVMe for metadata. The rest of NVMe is free, so I can now create two additional ~6TB NVMe OSDs per host.

My question is how to create a CRUSH rule which would have three replicas on three different hosts, with one of these replicas on NVMe, and the remaining two on HDD.

In the docs here: https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-rul... there is an example of a SSD-first rule, which is close to what I would like to do, but it seems the SSD replica in that rule can end up on the same host as one of the HDD replicas. How can I avoid that?

And a noob question -- how can I create a crush rule from the "rule something { ... }" text in that example?

Thanks!

-Yenya

-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl