21 Aug
2020
21 Aug
'20
5:48 p.m.
Hi all,
We have a 12 OSD node cluster in which I just recently found out that
'osd_crush_chooseleaf_type = 0' made it's way into our ceph.conf file,
probably from previous testing. I believe this is the reason a recent maintenance on an
OSD node caused data to stop flowing. In researching how to fix this, I just wanted to
confirm a few things and see if anybody who has done this before has any perspective or
things to look out for.
1) I believe the correct way to fix this is by following the 5 step method in the
documentation; Get, Decompile, Edit, Recompile, Set. Is that correct and is the line I
should change 'choose_firstn' to 'chooseleaf_firstn'? Do I only make this
change on 1 mon and it will propagate it to all other mons and osds?
2) Does the process start immediately following the setcrushmap command?
3) Any files to backup prior to this operation? This is production data so we can not have
any data loss.
4) Any other notes/things to be aware of?
Thank you
21 Aug
21 Aug
6:04 p.m.
Hi,
1) I believe the correct way to fix this is by
following the 5 step
method in the documentation; Get, Decompile, Edit, Recompile, Set.
Is that correct and is the line I should change 'choose_firstn' to
'chooseleaf_firstn'? Do I only make this change on 1 mon and it will
propagate it to all other mons and osds?
first remove the entry from ceph.conf. ;-) You can paste your current
ruleset here if you like others to take a look first, it would be
easier to confirm knowing the rest.
But those are the required steps, yes. And don't forget to test the
modified rule before setting it (change rule id and num-rep to your
needs):
crushtool -i crushmap.bin --test --rule 2 --num-rep 6 --show-mappings
crushtool -i crushmap.bin --test --rule 2 --num-rep 6 --show-bad-mappings
and modify until you get the expected distribution of PGs.
2) Does the process start immediately following the
setcrushmap command?
Yes.
3) Any files to backup prior to this operation? This
is production
data so we can not have any data loss.
To be able to rollback you can or should always backup the crushmap
you're about to change.
4) Any other notes/things to be aware of?
Test the new rule before applying it.
Regards,
Eugen
Zitat von Matt Dunavant <mdunavant(a)convokesystems.com>om>:
> Hi all,
>
> We have a 12 OSD node cluster in which I just recently found out
> that 'osd_crush_chooseleaf_type = 0' made it's way into our
> ceph.conf file, probably from previous testing. I believe this is
> the reason a recent maintenance on an OSD node caused data to stop
> flowing. In researching how to fix this, I just wanted to confirm a
> few things and see if anybody who has done this before has any
> perspective or things to look out for.
>
1) I believe the correct way to fix this is by
following the 5 step
method in the documentation; Get, Decompile, Edit, Recompile, Set.
Is that correct and is the line I should change 'choose_firstn' to
'chooseleaf_firstn'? Do I only make this change on 1 mon and it will
propagate it to all other mons and osds?
>
2) Does the process start immediately following the
setcrushmap command?
>
3) Any files to backup prior to this operation? This
is production
data so we can not have any data loss.
>
4) Any other notes/things to be aware of?
>
> Thank you
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:36 p.m.
Thanks for the reply! I've pasted what I believe are the applicable parts of the crush
map below. I see that the rule id is 0, but what is num-rep?
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
# end crush map
7:47 p.m.
Thanks for the reply! I've pasted what I believe
are the applicable
parts of the crush map below. I see that the rule id is 0, but what
is num-rep?
num-rep is the number of replicas you want to test, so basically the
size paramater of the pool this rule applies to. Do you have any
hierachy in your osd tree? If there's no other type like datacenter,
rack, chassis etc. then changing 'choose firstn' to 'chooseleaf
firstn' should be enough, I think.
Zitat von Matt Dunavant <mdunavant(a)convokesystems.com>om>:
Thanks for the reply! I've pasted what I believe
are the applicable
parts of the crush map below. I see that the rule id is 0, but what
is num-rep?
>
> # begin crush map
> tunable choose_local_tries 0
> tunable choose_local_fallback_tries 0
> tunable choose_total_tries 50
> tunable chooseleaf_descend_once 1
> tunable chooseleaf_vary_r 1
> tunable chooseleaf_stable 1
> tunable straw_calc_version 1
> tunable allowed_bucket_algs 54
>
> # rules
> rule replicated_rule {
> id 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step choose firstn 0 type osd
> step emit
> }
>
> # end crush map
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:53 p.m.
My replica size on the pool is 3, so I'll use that to test. There is no other type in
my map like dc, rack, etc; just servers. Do you know what a successful run of the test
command looks like? I just ran it myself and it spits out a number of crush rules (in this
case 1024) and then ends with:
double free or corruption (out)
*** Caught signal (Aborted) **
in thread 7f4915c58dc0 thread_name:crushtool
The show-bad-mappings version shows nothing and then spits out the same output as above.
I'm going to assume that means it's working properly?
9:19 p.m.
Although I’m confused about the error from crushtool it seems the
results are fine. The show-mappings displays a long list of possible
mappings to OSDs. If you provide num-rep 3 you should see three
different OSDs in each line, and if the rule works correctly those
OSDs never map to the same host. You can make sure by running ‚ceph
osd find ID‘ for some of those sets. If bad-mappings doesn’t show
anything it means crush finds a valid set of OSDs for each attempt.
Zitat von Matt Dunavant <mdunavant(a)convokesystems.com>om>:
My replica size on the pool is 3, so I'll use that
to test. There is
no other type in my map like dc, rack, etc; just servers. Do you
know what a successful run of the test command looks like? I just
ran it myself and it spits out a number of crush rules (in this case
1024) and then ends with:
double free or corruption (out)
*** Caught signal (Aborted) **
in thread 7f4915c58dc0 thread_name:crushtool
The show-bad-mappings version shows nothing and then spits out the
same output as above. I'm going to assume that means it's working
properly?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:33 p.m.
Gotcha, 1 more question: During the process, data will be available right? Just
performance will be impacted by the rebalancing correct?
10:12 p.m.
Yes, I don’t see any reason why it wouldn’t be. The client performance
can be controlled by max-backfills and recovery-max-active settings,
there was a thread this week about those.
Zitat von Matt Dunavant <mdunavant(a)convokesystems.com>om>:
Gotcha, 1 more question: During the process, data will
be available
right? Just performance will be impacted by the rebalancing correct?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Eugen Block voice : +49 40 5595175
NDE Netzdesign und -entwicklung AG fax : +49 40 5595177
Postfach 61 03 15 e-mail : username(a)nde.ag
D-22423 Hamburg
Vertretungsberechtigter Vorstand: Jens-U. Mozdzen
Vorsitzende des Aufsichtsrats: Angelika Torlée-Mozdzen
Sitz und Registergericht: Hamburg, HRB 90934
USt-IdNr: DE 814 013 983
1344
days inactive
1344
days old
7 comments
2 participants
participants (2)
-
Eugen Block -
Matt Dunavant