Editing Crush Map to fix osd_crush_chooseleaf_type = 0
Hi all, We have a 12 OSD node cluster in which I just recently found out that 'osd_crush_chooseleaf_type = 0' made it's way into our ceph.conf file, probably from previous testing. I believe this is the reason a recent maintenance on an OSD node caused data to stop flowing. In researching how to fix this, I just wanted to confirm a few things and see if anybody who has done this before has any perspective or things to look out for. 1) I believe the correct way to fix this is by following the 5 step method in the documentation; Get, Decompile, Edit, Recompile, Set. Is that correct and is the line I should change 'choose_firstn' to 'chooseleaf_firstn'? Do I only make this change on 1 mon and it will propagate it to all other mons and osds? 2) Does the process start immediately following the setcrushmap command? 3) Any files to backup prior to this operation? This is production data so we can not have any data loss. 4) Any other notes/things to be aware of? Thank you
Hi,
1) I believe the correct way to fix this is by following the 5 step method in the documentation; Get, Decompile, Edit, Recompile, Set. Is that correct and is the line I should change 'choose_firstn' to 'chooseleaf_firstn'? Do I only make this change on 1 mon and it will propagate it to all other mons and osds?
first remove the entry from ceph.conf. ;-) You can paste your current ruleset here if you like others to take a look first, it would be easier to confirm knowing the rest. But those are the required steps, yes. And don't forget to test the modified rule before setting it (change rule id and num-rep to your needs): crushtool -i crushmap.bin --test --rule 2 --num-rep 6 --show-mappings crushtool -i crushmap.bin --test --rule 2 --num-rep 6 --show-bad-mappings and modify until you get the expected distribution of PGs.
2) Does the process start immediately following the setcrushmap command?
Yes.
3) Any files to backup prior to this operation? This is production data so we can not have any data loss.
To be able to rollback you can or should always backup the crushmap you're about to change.
4) Any other notes/things to be aware of?
Test the new rule before applying it. Regards, Eugen Zitat von Matt Dunavant <mdunavant@convokesystems.com>:
Hi all,
We have a 12 OSD node cluster in which I just recently found out that 'osd_crush_chooseleaf_type = 0' made it's way into our ceph.conf file, probably from previous testing. I believe this is the reason a recent maintenance on an OSD node caused data to stop flowing. In researching how to fix this, I just wanted to confirm a few things and see if anybody who has done this before has any perspective or things to look out for.
1) I believe the correct way to fix this is by following the 5 step method in the documentation; Get, Decompile, Edit, Recompile, Set. Is that correct and is the line I should change 'choose_firstn' to 'chooseleaf_firstn'? Do I only make this change on 1 mon and it will propagate it to all other mons and osds?
2) Does the process start immediately following the setcrushmap command?
3) Any files to backup prior to this operation? This is production data so we can not have any data loss.
4) Any other notes/things to be aware of?
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the reply! I've pasted what I believe are the applicable parts of the crush map below. I see that the rule id is 0, but what is num-rep? # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step choose firstn 0 type osd step emit } # end crush map
Thanks for the reply! I've pasted what I believe are the applicable parts of the crush map below. I see that the rule id is 0, but what is num-rep?
num-rep is the number of replicas you want to test, so basically the size paramater of the pool this rule applies to. Do you have any hierachy in your osd tree? If there's no other type like datacenter, rack, chassis etc. then changing 'choose firstn' to 'chooseleaf firstn' should be enough, I think. Zitat von Matt Dunavant <mdunavant@convokesystems.com>:
Thanks for the reply! I've pasted what I believe are the applicable parts of the crush map below. I see that the rule id is 0, but what is num-rep?
# begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54
# rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step choose firstn 0 type osd step emit }
# end crush map _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
My replica size on the pool is 3, so I'll use that to test. There is no other type in my map like dc, rack, etc; just servers. Do you know what a successful run of the test command looks like? I just ran it myself and it spits out a number of crush rules (in this case 1024) and then ends with: double free or corruption (out) *** Caught signal (Aborted) ** in thread 7f4915c58dc0 thread_name:crushtool The show-bad-mappings version shows nothing and then spits out the same output as above. I'm going to assume that means it's working properly?
Although I’m confused about the error from crushtool it seems the results are fine. The show-mappings displays a long list of possible mappings to OSDs. If you provide num-rep 3 you should see three different OSDs in each line, and if the rule works correctly those OSDs never map to the same host. You can make sure by running ‚ceph osd find ID‘ for some of those sets. If bad-mappings doesn’t show anything it means crush finds a valid set of OSDs for each attempt. Zitat von Matt Dunavant <mdunavant@convokesystems.com>:
My replica size on the pool is 3, so I'll use that to test. There is no other type in my map like dc, rack, etc; just servers. Do you know what a successful run of the test command looks like? I just ran it myself and it spits out a number of crush rules (in this case 1024) and then ends with:
double free or corruption (out) *** Caught signal (Aborted) ** in thread 7f4915c58dc0 thread_name:crushtool
The show-bad-mappings version shows nothing and then spits out the same output as above. I'm going to assume that means it's working properly? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Gotcha, 1 more question: During the process, data will be available right? Just performance will be impacted by the rebalancing correct?
Yes, I don’t see any reason why it wouldn’t be. The client performance can be controlled by max-backfills and recovery-max-active settings, there was a thread this week about those. Zitat von Matt Dunavant <mdunavant@convokesystems.com>:
Gotcha, 1 more question: During the process, data will be available right? Just performance will be impacted by the rebalancing correct? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Eugen Block voice : +49 40 5595175 NDE Netzdesign und -entwicklung AG fax : +49 40 5595177 Postfach 61 03 15 e-mail : username@nde.ag D-22423 Hamburg Vertretungsberechtigter Vorstand: Jens-U. Mozdzen Vorsitzende des Aufsichtsrats: Angelika Torlée-Mozdzen Sitz und Registergericht: Hamburg, HRB 90934 USt-IdNr: DE 814 013 983
participants (2)
-
Eugen Block
-
Matt Dunavant