Hi all, I'm trying to understand how crush rules need to be set up, and much to my surprise I cannot find examples and/or good explanations (or I'm too stupid to understand them ;) ) My setup looks like this: DC1 node01 node02 node03 node04 node05 DC2 node01 node01 node01 node01 -- Andre Tann
Sorry, sent too early. So here we go again: My setup looks like this: DC1 node01 node02 node03 node04 node05 DC2 node06 node07 node08 node09 node10 I want a replicated pool with size=4. Two copies should go in each DC, and then no two copies on a single node. How can I describe this in a crush rule? If someone has a link where a similar strategy is explained, I'm more than happy to figure out myself how to do it. However, while the Ceph documentation is very good in general, the CRUSH rule explanation is difficult to understand for me. Thanks. -- Andre Tann
How can I describe this in a crush rule?
Let me please add the point that causes me the most difficulties: I consider DC and host both to be failure domains. But still I accept that two copies go into one DC, but I don't want to accept that two copies go to one host. And also, how can I say "2 copies in DC1 and 2 copies in DC2", and not for example 3 copies in one DC, 1 copy in the other? -- Andre Tann
Sorry, sent too early. So here we go again: My setup looks like this:
DC1 node01 node02 node03 node04 node05 DC2 node06 node07 node08 node09 node10
I want a replicated pool with size=4. Two copies should go in each DC, and then no two copies on a single node. How can I describe this in a crush rule?
This post seem to show that, except they have their root named "nvme" and they split on rack and not dc, but that is not important. https://unix.stackexchange.com/questions/781250/ceph-crush-rules-explanation... with the answer at the bottom: for example this should work as well, to have 4 replicas in total, distributed across two racks: step take default class nvme step choose firstn 2 type rack step chooseleaf firstn 2 type host -- May the most significant bit of your life be positive.
I have worked with crush and crush rules a lot over the last 12 years. I would always recommend testing the rules with a crushtool, for example. https://docs.ceph.com/en/reef/man/8/crushtool/ joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mi., 20. Nov. 2024 um 11:31 Uhr schrieb Janne Johansson < icepic.dz@gmail.com>:
Sorry, sent too early. So here we go again: My setup looks like this:
DC1 node01 node02 node03 node04 node05 DC2 node06 node07 node08 node09 node10
I want a replicated pool with size=4. Two copies should go in each DC, and then no two copies on a single node. How can I describe this in a crush rule?
This post seem to show that, except they have their root named "nvme" and they split on rack and not dc, but that is not important.
https://unix.stackexchange.com/questions/781250/ceph-crush-rules-explanation...
with the answer at the bottom:
for example this should work as well, to have 4 replicas in total, distributed across two racks: step take default class nvme step choose firstn 2 type rack step chooseleaf firstn 2 type host
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Andre, I think what you really want to look at is stretch mode. There have been long discussions on this list why a crush rule with rep 4 and 2 copies per DC will not handle a DC failure as expected. Stretch mode will make sure writes happen in a way that prevents split brain scenarios. Hand-crafted crush rules for this purpose require 3 or more DCs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Janne Johansson <icepic.dz@gmail.com> Sent: Wednesday, November 20, 2024 11:30 AM To: Andre Tann Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Crush rule examples
Sorry, sent too early. So here we go again: My setup looks like this:
DC1 node01 node02 node03 node04 node05 DC2 node06 node07 node08 node09 node10
I want a replicated pool with size=4. Two copies should go in each DC, and then no two copies on a single node. How can I describe this in a crush rule?
This post seem to show that, except they have their root named "nvme" and they split on rack and not dc, but that is not important. https://unix.stackexchange.com/questions/781250/ceph-crush-rules-explanation... with the answer at the bottom: for example this should work as well, to have 4 replicas in total, distributed across two racks: step take default class nvme step choose firstn 2 type rack step chooseleaf firstn 2 type host -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, thanks a lot for the hint, and I have read the documentation about this. What is not clear to me is this: == snip The first category of these failures that we will discuss involves inconsistent networks -- if there is a netsplit (a disconnection between two servers that splits the network into two pieces), Ceph might be unable to mark OSDs down and remove them from the acting PG sets. == snip Why is Ceph not able to mark OSDs down, and why is it unclear whether or not it is able to do so ("might")? Cheers Andre Am 20.11.24 um 12:23 schrieb Frank Schilder:
Hi Andre,
I think what you really want to look at is stretch mode. There have been long discussions on this list why a crush rule with rep 4 and 2 copies per DC will not handle a DC failure as expected. Stretch mode will make sure writes happen in a way that prevents split brain scenarios.
Hand-crafted crush rules for this purpose require 3 or more DCs.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Janne Johansson <icepic.dz@gmail.com> Sent: Wednesday, November 20, 2024 11:30 AM To: Andre Tann Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Crush rule examples
Sorry, sent too early. So here we go again: My setup looks like this:
DC1 node01 node02 node03 node04 node05 DC2 node06 node07 node08 node09 node10
I want a replicated pool with size=4. Two copies should go in each DC, and then no two copies on a single node. How can I describe this in a crush rule?
This post seem to show that, except they have their root named "nvme" and they split on rack and not dc, but that is not important.
https://unix.stackexchange.com/questions/781250/ceph-crush-rules-explanation...
with the answer at the bottom:
for example this should work as well, to have 4 replicas in total, distributed across two racks: step take default class nvme step choose firstn 2 type rack step chooseleaf firstn 2 type host
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Andre Tann
Den tors 21 nov. 2024 kl 09:45 skrev Andre Tann <atann@alphasrv.net>:
Hi Frank, thanks a lot for the hint, and I have read the documentation about this. What is not clear to me is this:
== snip The first category of these failures that we will discuss involves inconsistent networks -- if there is a netsplit (a disconnection between two servers that splits the network into two pieces), Ceph might be unable to mark OSDs down and remove them from the acting PG sets. == snip
Why is Ceph not able to mark OSDs down, and why is it unclear whether or not it is able to do so ("might")?
I think designs with 2 DCs usually have one or two mons per DC, and then a third/fifth mon in a (small) third site so it can arbitrate which side is up and which isn't. OSDs report to each other but also to mons about their existence. -- May the most significant bit of your life be positive.
Am 21.11.24 um 10:56 schrieb Janne Johansson:
== snip The first category of these failures that we will discuss involves inconsistent networks -- if there is a netsplit (a disconnection between two servers that splits the network into two pieces), Ceph might be unable to mark OSDs down and remove them from the acting PG sets. == snip
Why is Ceph not able to mark OSDs down, and why is it unclear whether or not it is able to do so ("might")?
I think designs with 2 DCs usually have one or two mons per DC, and then a third/fifth mon in a (small) third site so it can arbitrate which side is up and which isn't. OSDs report to each other but also to mons about their existence.
Yes absolutely, you need another qdevice/witness/mon... in a third location for the quorum, and my setup will have that. But still I don't see why Ceph should not be able to mark an OSD down if one site went down. -- Andre Tann
Hi Janne Am 20.11.24 um 11:30 schrieb Janne Johansson:
This post seem to show that, except they have their root named "nvme" and they split on rack and not dc, but that is not important.
https://unix.stackexchange.com/questions/781250/ceph-crush-rules-explanation...
This is indeed a good example, thanks. Let me put some thoughts/questions here:
step choose firstn 2 type rack
This choses 2 racks out of all available racks. As there are 2 racks available, all are chosen.
step chooseleaf firstn 2 type host
For each selected rack from the previous step, 2 hosts are chosen. But as the action is "chooseleaf", in fact not the hosts are picked, but one random (?) OSD in each of the 2 selected hosts. In the end we have 4 OSDs in 4 different hosts, 2 in each rack. Is this understanding correct? Shouldn't we note this one additionally: min_size 4 max_size 4 Reason: If we wanted to place more ore less than 4 replicas, the rule won't work. Or what would happen if we don't specify min/max_size? Should lead to an error in case the pool is e.g. size=5, shouldn't it? One last question: if we edit a crush map after a pool was created on it, what happens? In my understanding, this lead to massive data shifting so that the placements comply with the new rules. That right? Thanks again -- Andre Tann
Den tors 21 nov. 2024 kl 19:18 skrev Andre Tann <atann@alphasrv.net>:
This post seem to show that, except they have their root named "nvme" and they split on rack and not dc, but that is not important.
https://unix.stackexchange.com/questions/781250/ceph-crush-rules-explanation...
This is indeed a good example, thanks. Let me put some thoughts/questions here:
step choose firstn 2 type rack
This choses 2 racks out of all available racks. As there are 2 racks available, all are chosen.
Yes, and you would name it DC instead of course.
step chooseleaf firstn 2 type host
For each selected rack from the previous step, 2 hosts are chosen. But as the action is "chooseleaf", in fact not the hosts are picked, but one random (?) OSD in each of the 2 selected hosts.
Well, it picks a leaf out of the host, which is a branch in the tree. I see it as after picking the host, don't do anything special but just grab an OSD from there.
In the end we have 4 OSDs in 4 different hosts, 2 in each rack. Is this understanding correct?
I believe so, yes.
Shouldn't we note this one additionally:
min_size 4
Not necessary, you could allow for min_size 3 so that single-drive problems doesn't cause the PG to stop.
max_size 4
Reason: If we wanted to place more ore less than 4 replicas, the rule won't work. Or what would happen if we don't specify min/max_size? Should lead to an error in case the pool is e.g. size=5, shouldn't it?
Yes, but when you figure you need a repl=5 pool you would have to make a rule that picks 3 from one DC. I'm sure there is a way to say "..and then you pick as many hosts as needed", but I don't know it offhand. Might be that the above rule would allow 5 copies, but the fifth ends up on the same host as the one of the others.
One last question: if we edit a crush map after a pool was created on it, what happens? In my understanding, this lead to massive data shifting so that the placements comply with the new rules. That right?
Yes, but it can be mitigated somewhat using the remappers and let the balancer slowly do the changes. 1. set norebalance 2. stop the balancer 3. apply the new crush rule on pool 4. let the mons figure out all new places for the PGs 5. run one of the remapper tools, jj-balancer, upmap-remapper.py or the golang pgremapper, which makes most (sometimes all) PGs think they are in the correct place after all 6. unset norebalance 7. start the ceph balancer with a setting of max misplaced % that suits the load you want to have during the moves. -- May the most significant bit of your life be positive.
participants (4)
-
Andre Tann
-
Frank Schilder
-
Janne Johansson
-
Joachim Kraftmayer