classes crush rules new cluster
It looks like it is not possible to create crush rules when you don't have harddrives active in this class. I am testing with new squid and did not add ssd's yet, eventhough I added class like this. ceph osd crush class create ssd I can't execute this ceph osd crush rule create-replicated replicated_ssd default host ssd Is there any way around this?
You could decompile the crushmap, add a dummy OSD (with a non-existing ID) with your new device class and add a rule, then compile it and inject. Here's an excerpt from a lab cluster with 4 OSDs (0..3), adding a fifth non-existing: device 4 osd.4 class test rule testrule { id 6 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class test step chooseleaf indep 0 type host step emit } Note that testing this rule with crushtool won't work here since the fake OSD isn't assigned to a hosts. But what's the point in having a rule without the corresponding devices? You won't be able to create a pool with that rule anyway until the OSDs are present. Zitat von Marc <Marc@f1-outsourcing.eu>:
It looks like it is not possible to create crush rules when you don't have harddrives active in this class.
I am testing with new squid and did not add ssd's yet, eventhough I added class like this.
ceph osd crush class create ssd
I can't execute this ceph osd crush rule create-replicated replicated_ssd default host ssd
Is there any way around this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I know a bit the work-arounds for manually editing the crush map. I just think this is not the best way to get acquainted to with a new ceph cluster. I would make these hdd,nvme,ssd classes available directly.
You could decompile the crushmap, add a dummy OSD (with a non-existing ID) with your new device class and add a rule, then compile it and inject. Here's an excerpt from a lab cluster with 4 OSDs (0..3), adding a fifth non-existing:
device 4 osd.4 class test
rule testrule { id 6 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class test step chooseleaf indep 0 type host step emit }
Note that testing this rule with crushtool won't work here since the fake OSD isn't assigned to a hosts.
But what's the point in having a rule without the corresponding devices? You won't be able to create a pool with that rule anyway until the OSDs are present.
Zitat von Marc <Marc@f1-outsourcing.eu>:
It looks like it is not possible to create crush rules when you don't have harddrives active in this class.
I am testing with new squid and did not add ssd's yet, eventhough I added class like this.
ceph osd crush class create ssd
I can't execute this ceph osd crush rule create-replicated replicated_ssd default host ssd
Is there any way around this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Or, just reassign one existing OSD to the new class.
Note that testing this rule with crushtool won't work here since the fake OSD isn't assigned to a hosts.
But what's the point in having a rule without the corresponding devices? You won't be able to create a pool with that rule anyway until the OSDs are present.
There is that.
Or, just reassign one existing OSD to the new class.
Note that testing this rule with crushtool won't work here since the fake OSD isn't assigned to a hosts.
But what's the point in having a rule without the corresponding devices? You won't be able to create a pool with that rule anyway until the OSDs are present.
There is that.
Yes indeed so you can prepare creating all rules and pools before you have added the osds. Especially handy if you have a few shell commands you paste.
Hi yall, Am 29.11.24 um 08:51 schrieb Eugen Block:
rule testrule { id 6 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class test step chooseleaf indep 0 type host step emit }
Does anyone know a good and comprehensive discussion about all the options for a crush rule, and what they do. Of course I know the original documentation, but I find that too short, and leaves me with many questions. Thanks for any hints. -- Andre Tann
Which questions do you have? When I first started to deal with crush rules I was overwhelmed, but with a bit of practice and trial & error you're going to figure it out. Maybe this helps a bit (inline comments): id 6 -> self explanatory type erasure -> self explanatory step set_chooseleaf_tries 5 -> stick to defaults, usually works (number of max attempts to find suitable OSDs) step set_choose_tries 100 -> stick to defaults, usually works (number of max attempts to find suitable buckets, e.g. hosts) step take default class test -> "default" is the usual default crush root (check 'ceph osd tree'), you can specify other roots if you have them step chooseleaf indep 0 type host -> within bucket "root" (from "step take default") choose {pool-num-replicas} hosts step emit -> execute There are more details in [0]. @Marc: I guess one could argue about that. On the one hand, those three device classes you mention are discovered automatically when OSDs are added (depending on controllers, etc.), so an operator doesn't have to deal with it. Just create rules using those classes. On the other hand, some users like to have full control over everything and don't need that kind of automatic discovery. [0] https://docs.ceph.com/en/latest/rados/operations/crush-map-edits/#crush-map-... Zitat von Andre Tann <atann@alphasrv.net>:
Hi yall,
Am 29.11.24 um 08:51 schrieb Eugen Block:
rule testrule { id 6 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class test step chooseleaf indep 0 type host step emit }
Does anyone know a good and comprehensive discussion about all the options for a crush rule, and what they do.
Of course I know the original documentation, but I find that too short, and leaves me with many questions.
Thanks for any hints.
-- Andre Tann _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ahoi Eugen, Am 29.11.24 um 11:31 schrieb Eugen Block:
step set_chooseleaf_tries 5 -> stick to defaults, usually works (number of max attempts to find suitable OSDs)
Why do we need more than one attempt to find an OSD? Why is the result different if we walk through a rule more than once?
step take default class test -> "default" is the usual default crush root (check 'ceph osd tree'), you can specify other roots if you have them
where are these classes defined? Or is "default class test" the name of a root? Most probably not. Could I also say step take default type host? What are the keywords that are allowed after the root's name?
step chooseleaf indep 0 type host -> within bucket "root" (from "step take default") choose {pool-num-replicas} hosts
What if I did exactly this, but have nested fault domains (e.g. racks > hosts)? Would the rule then pick {pool-num-replicas} hosts out of different racks, even though this rule doesn't mention racks anywhere? But what if I have size=4, but only two racks, would the picked hosts spread evenly across the two racks, or randomly, like 1 host in one rack, 3 in the other, or all 4 in one rack? Assume a pool with size=4, could I say step take default choose firstn 1 type row choose firstn 3 type racks chooseleaf firstn 0 type host Meaning: - force all chunks of a pg in one row - force all chunks in exactly three racks inside this row - out of these three racks, pick 4 hosts I don't want to say that the latter makes much sense, I just wonder if it would work that way. -- Andre Tann
Andre, see responses inline. Zitat von Andre Tann <atann@alphasrv.net>:
Ahoi Eugen,
Am 29.11.24 um 11:31 schrieb Eugen Block:
step set_chooseleaf_tries 5 -> stick to defaults, usually works (number of max attempts to find suitable OSDs)
Why do we need more than one attempt to find an OSD? Why is the result different if we walk through a rule more than once?
There have been cases with a large number of OSDs where crush "gave up too soon". Although I haven't read about that in quite a while, it may or may not still be an issue.
step take default class test -> "default" is the usual default crush root (check 'ceph osd tree'), you can specify other roots if you have them
where are these classes defined? Or is "default class test" the name of a root? Most probably not.
You define those classes. By default, Ceph creates a "default" entry point into the crush tree of type "root": ceph osd tree | head -2 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.14648 root default You can create multiple roots with arbitrary names. Those roots can be addressed in crush rules. Before there were device classes, users split their trees into multiple roots, for example one for HDD, one for SSD devices.
Could I also say step take default type host?
I haven't tried that, I would assume that the entry point still has to be a bucket of type "root". I encourage you to play around in a lab cluster to get familiar with crushmaps and especially the crushtool, you'll benefit from it.
What are the keywords that are allowed after the root's name?
Fair question, I'm only aware of "class XYZ", so the device classes. I haven't checked in detail though.
step chooseleaf indep 0 type host -> within bucket "root" (from "step take default") choose {pool-num-replicas} hosts
What if I did exactly this, but have nested fault domains (e.g. racks > hosts)? Would the rule then pick {pool-num-replicas} hosts out of different racks, even though this rule doesn't mention racks anywhere?
Since I don't have racks in my lab cluster, I don't specify them. You need to modify your rule(s) according to your infrastructure, my example was just a simple one from one of my lab clusters.
But what if I have size=4, but only two racks, would the picked hosts spread evenly across the two racks, or randomly, like 1 host in one rack, 3 in the other, or all 4 in one rack?
You can (and most likely will) end up with the random result if you don't specifically tell crush what to do.
Assume a pool with size=4, could I say
step take default choose firstn 1 type row choose firstn 3 type racks chooseleaf firstn 0 type host
Meaning: - force all chunks of a pg in one row - force all chunks in exactly three racks inside this row - out of these three racks, pick 4 hosts
I don't want to say that the latter makes much sense, I just wonder if it would work that way.
I think it would, but again, give it a try. You can create "virtual" rows and racks, just add the respective buckets to the crushmap (of your test cluster): ceph osd crush add-bucket row1 row root=default added bucket row1 type row to location {root=default} ceph osd crush add-bucket rack1 rack row=row1 added bucket rack1 type rack to location {row=row1} ceph osd crush add-bucket rack2 rack row=row1 added bucket rack2 type rack to location {row=row1} ceph osd crush add-bucket rack3 rack row=row1 added bucket rack3 type rack to location {row=row1} Then move some of your hosts into the racks with ceph 'osd crush move...' and test your crush rules.
-- Andre Tann _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Andre Tann
-
Anthony D'Atri
-
Eugen Block
-
Marc