Hi Torkil,
Num is 0 but it's not replicated so how does this translate to picking 3 of 3 datacenters?
it doesn't really make a difference if replicated or not, it just defines how many crush buckets to choose, so it applies in the same way as for your replicated pool.
I am thinking we should just change 3 to 2 for the chooseleaf line for the 4+2 rule since for 4+5 each DC needs 3 shards and for 4+2 each DC needs 2 shards. Comments?
Unless your output from the 4+2 rule is incomplete, it doesn't currently contain a line how many datacenters to choose, so don't forget that. ;-) But yeah, you could either have
step choose indep 0 type datacenter step chooseleaf indep 2 type host
or
step choose indep 3 type datacenter step chooseleaf indep 2 type host
The result should be the same. But I recommend to verify with the crushtool: # get current crushmap ceph osd getcrushmap -o crushmap.bin # decompile crushmap crushtool -d crushmap.bin -o crushmap.txt # change your crush rule # test it crushtool -i crushmap.test --test --rule <rule_id> --show-mappings --num-rep 6 crushtool -i crushmap.test --test --rule <rule_id> --show-bad-mappings --num-rep 6 You'll see the osd mappings which will tell you if the PGs would be distributed as required. Regards, Eugen Zitat von Torkil Svensgaard <torkil@drcmr.dk>:
I was just looking at our crush rules as we need to change them from failure domain host to failure domain datacenter. The replicated ones seem trivial but what about this one for EC 4+2?
rule rbd_ec_data { id 0 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit }
We already have this crush rule for EC 4+5:
" rule cephfs.hdd.data { id 7 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type datacenter step chooseleaf indep 3 type host step emit } "
I don't understand the "num" argument for the choose step. The documentation[1] says:
" If {num} == 0, choose pool-num-replicas buckets (as many buckets as are available).
If pool-num-replicas > {num} > 0, choose that many buckets.
If {num} < 0, choose pool-num-replicas - {num} buckets. "
Num is 0 but it's not replicated so how does this translate to picking 3 of 3 datacenters?
I am thinking we should just change 3 to 2 for the chooseleaf line for the 4+2 rule since for 4+5 each DC needs 3 shards and for 4+2 each DC needs 2 shards. Comments?
Mvh.
Torkil
[1] https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/
-- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io