On Aug 20, 2025, at 1:57 PM, jbuburuzlist <jbuburuzlist@sce.carleton.ca> wrote:
Hello
I just added more nodes to quincy cluster(existing drives are SAS). The new nodes are sata drives.
The problem is after creating a new replicated crush rule and applying to a new pool. The ceph cluster is still using osd with "hdd" class and not using sata drives.
What command are you using to create an image? Does it specify the new pool
I tried a number of crush rules using replicated and EC and I cannot get the pools to use sata osds.
Please send: `ceph osd df` `ceph osd crush rule dump` `ceph osd dump | grep pool` `ceph osd df tree` Let's make sure that your new pool uses CRUSH rule 9.
Here is crush rule for sata: { "rule_id": 9, "rule_name": "rbdsatareplicated", "type": 1, "steps": [ { "op": "take", "item": -33, "item_name": "default~sata" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }
I see in the crush rule "default~sata". I am wondering if this "default~" is causing me problems?
`default` is the CRUSH root. It is highly likely this is what you want.
The ceph cluster is still using osd with "hdd" class and not using sata drives.
How are you determining this? Remember that by default RBD volumes are thin-provisioned, and won't take up significant underlying capacity (a la `ceph df) until data is written to them.
Just another note. After I added the new sata nodes, my first crush rule used ec instead of replicated. I was getting errors when I tried to create a image on the ec pool "qemu-img error rbd create: Operation not supported on new pool".
When using EC for RBD you need a separate, replicated metadata pool. Did you have one? Did you configure the client(s) to specify EC and both data and metadata pools? Did you explicitly specify the EC pool? rbd create --size IMAGE_SIZE --data-pool EC_POOL REPLICATED_POOL/IMAGE_NAME ? Did you set ceph osd pool set <ERASURE_CODED_POOL_NAME> allow_ec_overwrites true ceph osd pool set POOL_NAME application rbd ? Note that with current Ceph releases the write latency to an EC RBD pool usually precludes the use of EC, especially when backed by spinners. That said, do you really need to segregate the SATA and SAS drives into different pools?
So I tried a test pool using replicated instead.
Currently on the cluster I can create any number of pools and images if I use replicated/ec and the SAS drives which have a class of "hdd".
Hope this makes sense?
Here is some additional details:
My existing nodes used SAS and are labelled:
ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1122.64075 root default -3 34.03793 host node-01 0 hdd 2.26920 osd.0 up 1.00000 1.00000 1 hdd 2.26920 osd.1 up 1.00000 1.00000 2 hdd 2.26920 osd.2 up 1.00000 1.00000 ..etc
After adding SATA nodes I have:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1122.64075 root default -3 34.03793 host node-01 0 hdd 2.26920 osd.0 up 1.00000 1.00000 1 hdd 2.26920 osd.1 up 1.00000 1.00000 2 hdd 2.26920 osd.2 up 1.00000 1.00000 .... ... -34 133.78070 host node-20 141 sata 11.14839 osd.141 up 1.00000 1.00000 142 sata 11.14839 osd.142 up 1.00000 1.00000 143 sata 11.14839 osd.143 up 1.00000 1.00000 144 sata 11.14839 osd.144 up 1.00000 1.00000 145 sata 11.14839 osd.145 up 1.00000 1.00000 146 sata 11.14839 osd.146 up 1.00000 1.00000
...
Thanks jerry _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io