I've done exactly this recently. Changing crush rule to "step take default class hdd". And with 'nobackfill' set, I ran "pgremapper cancel-backfill --yes" two or three times to make the misplaced objects reduce from 95% -> 18%. This was on a Quincy (17.2.x) cluster. On 1/19/26 5:48 PM, Anthony D'Atri via ceph-users wrote:
I’m surprised that this would cause data to move.
How can I pin the PGs to HDD-based OSDs where they are, without the cluster rebalancing itself? You can use upmap-remapped.py, though the balancer will progressively still move them if that’s what CRUSH really wants.
I have an RFE in to at the very least raise a warning if any rules/pools do not specify a device class, ideally setting one by default.
The whole CRUSH map can be seen here:
https://www.fi.muni.cz/~kas/tmp/crushmap.txt
Thanks!
-Yenya
Jan Kasprzak via ceph-users wrote:
Hello, Ceph users,
how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, with all replicas on different hosts?
One of my clusters has been upgraded to two brand new ~6TB NVMe disks together with previously used two 12TB HDDs per host. Right now I have two HDD-based OSDs per host, each with ~200GB partition on NVMe for metadata. The rest of NVMe is free, so I can now create two additional ~6TB NVMe OSDs per host.
My question is how to create a CRUSH rule which would have three replicas on three different hosts, with one of these replicas on NVMe, and the remaining two on HDD.
In the docs here: https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-rul... there is an example of a SSD-first rule, which is close to what I would like to do, but it seems the SSD replica in that rule can end up on the same host as one of the HDD replicas. How can I avoid that?
And a noob question -- how can I create a crush rule from the "rule something { ... }" text in that example?
Thanks!
-Yenya
-- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io