CRUSH rule for EC 6+2 on 6-node cluster
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host. If I use the default rule with an osd failure domain like this: step take default step choose indep 0 type osd step emit I get clustering of 3-4 chunks on some of the hosts: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 8.0 1 harrahs 3 paris 4 aladdin 8.1 1 aladdin 1 excalibur 2 mandalaybay 4 paris 8.2 1 harrahs 2 aladdin 2 mirage 3 paris ...
However, if I change the rule to use: step take default step choose indep 0 type host step chooseleaf indep 2 type osd step emit I get the data spread across 4 hosts with 2 chunks per host: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 done 8.0 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.1 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.2 2 harrahs 2 mandalaybay 2 mirage 2 paris ...
Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2. Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab). Thanks, Bryan
I was able to figure out the solution with this rule: step take default step choose indep 0 type host step chooseleaf indep 1 type osd step emit step take default step choose indep 0 type host step chooseleaf indep 1 type osd step emit Now the data is spread how I want it to be: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 done 8.0 1 excalibur 1 harrahs 1 mandalaybay 1 mirage 2 aladdin 2 paris 8.1 1 aladdin 1 excalibur 1 harrahs 1 mirage 2 mandalaybay 2 paris 8.2 1 aladdin 1 excalibur 1 harrahs 1 mirage 2 mandalaybay 2 paris ...
Hopefully someone else will find this useful. Bryan
On May 12, 2021, at 9:58 AM, Bryan Stillwell <bstillwell@godaddy.com> wrote:
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host.
If I use the default rule with an osd failure domain like this:
step take default step choose indep 0 type osd step emit
I get clustering of 3-4 chunks on some of the hosts:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 8.0 1 harrahs 3 paris 4 aladdin 8.1 1 aladdin 1 excalibur 2 mandalaybay 4 paris 8.2 1 harrahs 2 aladdin 2 mirage 3 paris ...
However, if I change the rule to use:
step take default step choose indep 0 type host step chooseleaf indep 2 type osd step emit
I get the data spread across 4 hosts with 2 chunks per host:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 done 8.0 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.1 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.2 2 harrahs 2 mandalaybay 2 mirage 2 paris ...
Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab).
Thanks, Bryan
Would something like this work? step take default step choose indep 4 type host step chooseleaf indep 1 type osd step emit step take default step choose indep 0 type host step chooseleaf indep 1 type osd step emit J. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell <bstillwell@godaddy.com> wrote:
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host.
If I use the default rule with an osd failure domain like this:
step take default
step choose indep 0 type osd
step emit
I get clustering of 3-4 chunks on some of the hosts:
for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do =======================================================================================
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
8.0
1 harrahs
3 paris
4 aladdin
8.1
1 aladdin
1 excalibur
2 mandalaybay
4 paris
8.2
1 harrahs
2 aladdin
2 mirage
3 paris
...
However, if I change the rule to use:
step take default
step choose indep 0 type host
step chooseleaf indep 2 type osd
step emit
I get the data spread across 4 hosts with 2 chunks per host:
for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do =======================================================================================
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
done
8.0
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.1
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.2
2 harrahs
2 mandalaybay
2 mirage
2 paris
...
Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab).
Thanks,
Bryan
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
This works better than my solution. It allows the cluster to put more PGs on the systems with more space on them: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 done 8.0 1 excalibur 1 mandalaybay 2 aladdin 2 harrahs 2 paris 8.1 1 aladdin 1 excalibur 1 harrahs 1 mirage 2 mandalaybay 2 paris 8.2 1 aladdin 1 mandalaybay 2 harrahs 2 mirage 2 paris ...
Thanks! Bryan
On May 13, 2021, at 2:58 AM, Ján Senko <janos@protonmail.ch> wrote:
Caution: This email is from an external sender. Please do not click links or open attachments unless you recognize the sender and know the content is safe. Forward suspicious emails to isitbad@.
Would something like this work?
step take default step choose indep 4 type host step chooseleaf indep 1 type osd step emit step take default step choose indep 0 type host step chooseleaf indep 1 type osd step emit
J.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell <bstillwell@godaddy.com> wrote:
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host.
If I use the default rule with an osd failure domain like this:
step take default
step choose indep 0 type osd
step emit
I get clustering of 3-4 chunks on some of the hosts:
for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do =======================================================================================
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
8.0
1 harrahs
3 paris
4 aladdin
8.1
1 aladdin
1 excalibur
2 mandalaybay
4 paris
8.2
1 harrahs
2 aladdin
2 mirage
3 paris
...
However, if I change the rule to use:
step take default
step choose indep 0 type host
step chooseleaf indep 2 type osd
step emit
I get the data spread across 4 hosts with 2 chunks per host:
for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do =======================================================================================
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
done
8.0
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.1
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.2
2 harrahs
2 mandalaybay
2 mirage
2 paris
...
Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab).
Thanks,
Bryan
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
Actually both our solutions don't work very well. Frequently the same OSD was chosen for multiple chunks: 8.72 9751 0 0 0 40895512576 0 0 1302 active+clean 2h 224790'12801 225410:49810 [13,1,14,11,18,2,19,13]p13 [13,1,14,11,18,2,19,13]p13 2021-05-11T22:41:11.332885+0000 2021-05-11T22:41:11.332885+0000 8.7f 9695 0 0 0 40661680128 0 0 2184 active+clean 5h 224790'12850 225409:57529 [8,17,4,1,14,0,19,8]p8 [8,17,4,1,14,0,19,8]p8 2021-05-11T22:41:11.332885+0000 2021-05-11T22:41:11.332885+0000 I'm now considering using device classes and assigning the OSDs to either hdd1 or hdd2... Unless someone has another idea? Thanks, Bryan
On May 14, 2021, at 12:35 PM, Bryan Stillwell <bstillwell@godaddy.com> wrote:
This works better than my solution. It allows the cluster to put more PGs on the systems with more space on them:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 done 8.0 1 excalibur 1 mandalaybay 2 aladdin 2 harrahs 2 paris 8.1 1 aladdin 1 excalibur 1 harrahs 1 mirage 2 mandalaybay 2 paris 8.2 1 aladdin 1 mandalaybay 2 harrahs 2 mirage 2 paris ...
Thanks! Bryan
On May 13, 2021, at 2:58 AM, Ján Senko <janos@protonmail.ch> wrote:
Caution: This email is from an external sender. Please do not click links or open attachments unless you recognize the sender and know the content is safe. Forward suspicious emails to isitbad@.
Would something like this work?
step take default step choose indep 4 type host step chooseleaf indep 1 type osd step emit step take default step choose indep 0 type host step chooseleaf indep 1 type osd step emit
J.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell <bstillwell@godaddy.com> wrote:
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host.
If I use the default rule with an osd failure domain like this:
step take default
step choose indep 0 type osd
step emit
I get clustering of 3-4 chunks on some of the hosts:
for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do =======================================================================================
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
8.0
1 harrahs
3 paris
4 aladdin
8.1
1 aladdin
1 excalibur
2 mandalaybay
4 paris
8.2
1 harrahs
2 aladdin
2 mirage
3 paris
...
However, if I change the rule to use:
step take default
step choose indep 0 type host
step chooseleaf indep 2 type osd
step emit
I get the data spread across 4 hosts with 2 chunks per host:
for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do =======================================================================================
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
done
8.0
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.1
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.2
2 harrahs
2 mandalaybay
2 mirage
2 paris
...
Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab).
Thanks,
Bryan
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Bryan, I had to do something similar, and never found a rule to place "up to" 2 chunks per host, so I stayed with the placement of *exactly* 2 chunks per host. But I did this slightly differently to what you wrote earlier: my rule chooses exactly 4 hosts, then chooses exactly 2 osds on each: type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 4 type host step choose indep 2 type osd step emit If you really need the "up to 2" approach then maybe you can split each host into two "host" crush buckets, with half the OSDs in each. Then a normal host-wise rule should work. Cheers, Dan On Sun, May 16, 2021 at 2:34 AM Bryan Stillwell <bstillwell@godaddy.com> wrote:
Actually both our solutions don't work very well. Frequently the same OSD was chosen for multiple chunks:
8.72 9751 0 0 0 40895512576 0 0 1302 active+clean 2h 224790'12801 225410:49810 [13,1,14,11,18,2,19,13]p13 [13,1,14,11,18,2,19,13]p13 2021-05-11T22:41:11.332885+0000 2021-05-11T22:41:11.332885+0000 8.7f 9695 0 0 0 40661680128 0 0 2184 active+clean 5h 224790'12850 225409:57529 [8,17,4,1,14,0,19,8]p8 [8,17,4,1,14,0,19,8]p8 2021-05-11T22:41:11.332885+0000 2021-05-11T22:41:11.332885+0000
I'm now considering using device classes and assigning the OSDs to either hdd1 or hdd2... Unless someone has another idea?
Thanks, Bryan
On May 14, 2021, at 12:35 PM, Bryan Stillwell <bstillwell@godaddy.com> wrote:
This works better than my solution. It allows the cluster to put more PGs on the systems with more space on them:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 done 8.0 1 excalibur 1 mandalaybay 2 aladdin 2 harrahs 2 paris 8.1 1 aladdin 1 excalibur 1 harrahs 1 mirage 2 mandalaybay 2 paris 8.2 1 aladdin 1 mandalaybay 2 harrahs 2 mirage 2 paris ...
Thanks! Bryan
On May 13, 2021, at 2:58 AM, Ján Senko <janos@protonmail.ch> wrote:
Caution: This email is from an external sender. Please do not click links or open attachments unless you recognize the sender and know the content is safe. Forward suspicious emails to isitbad@.
Would something like this work?
step take default step choose indep 4 type host step chooseleaf indep 1 type osd step emit step take default step choose indep 0 type host step chooseleaf indep 1 type osd step emit
J.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell <bstillwell@godaddy.com> wrote:
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host.
If I use the default rule with an osd failure domain like this:
step take default
step choose indep 0 type osd
step emit
I get clustering of 3-4 chunks on some of the hosts:
for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do =======================================================================================
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
8.0
1 harrahs
3 paris
4 aladdin
8.1
1 aladdin
1 excalibur
2 mandalaybay
4 paris
8.2
1 harrahs
2 aladdin
2 mirage
3 paris
...
However, if I change the rule to use:
step take default
step choose indep 0 type host
step chooseleaf indep 2 type osd
step emit
I get the data spread across 4 hosts with 2 chunks per host:
for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do =======================================================================================
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
done
8.0
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.1
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.2
2 harrahs
2 mandalaybay
2 mirage
2 paris
...
Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab).
Thanks,
Bryan
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hallo Dan, Bryan, I have a rule similar to yours, for an 8+4 pool, with only difference that I replaced the second "choose" with "chooseleaf", which I understand should make no difference: rule default.rgw.buckets.data { id 6 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class big step choose indep 5 type host step chooseleaf indep 2 type osd step emit } I am on Nautilus 14.2.16 and while performing a maintenance the other day, I noticed 2 PGs were incomplete and caused troubles to some users. I then verified that (thanks Bryan for the command): [cephmgr@cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map 116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' ; done | sort | uniq -c | sort -n -k1 2 r2srv07.ct1.box.garr 2 r2srv10.ct1.box.garr 2 r3srv07.ct1.box.garr 4 r1srv02.ct1.box.garr You see that 4 PGs were put on r1srv02. May be this happened due to some temporary unavailability of the host at some point? As all my servers are now up and running, is there a way to force the placement rule to rerun? Thanks! Fulvio Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
Hi Bryan,
I had to do something similar, and never found a rule to place "up to" 2 chunks per host, so I stayed with the placement of *exactly* 2 chunks per host.
But I did this slightly differently to what you wrote earlier: my rule chooses exactly 4 hosts, then chooses exactly 2 osds on each:
type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 4 type host step choose indep 2 type osd step emit
If you really need the "up to 2" approach then maybe you can split each host into two "host" crush buckets, with half the OSDs in each. Then a normal host-wise rule should work.
Cheers, Dan
-- Fulvio Galeazzi GARR-CSD Department skype: fgaleazzi70 tel.: +39-334-6533-250
Hi Fulvio, That's strange... It doesn't seem right to me. Are there any upmaps for that PG? ceph osd dump | grep upmap | grep 116.453 Cheers, Dan On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi@garr.it> wrote:
Hallo Dan, Bryan, I have a rule similar to yours, for an 8+4 pool, with only difference that I replaced the second "choose" with "chooseleaf", which I understand should make no difference:
rule default.rgw.buckets.data { id 6 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class big step choose indep 5 type host step chooseleaf indep 2 type osd step emit }
I am on Nautilus 14.2.16 and while performing a maintenance the other day, I noticed 2 PGs were incomplete and caused troubles to some users. I then verified that (thanks Bryan for the command):
[cephmgr@cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map 116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' ; done | sort | uniq -c | sort -n -k1 2 r2srv07.ct1.box.garr 2 r2srv10.ct1.box.garr 2 r3srv07.ct1.box.garr 4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02. May be this happened due to some temporary unavailability of the host at some point? As all my servers are now up and running, is there a way to force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
Hi Bryan,
I had to do something similar, and never found a rule to place "up to" 2 chunks per host, so I stayed with the placement of *exactly* 2 chunks per host.
But I did this slightly differently to what you wrote earlier: my rule chooses exactly 4 hosts, then chooses exactly 2 osds on each:
type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 4 type host step choose indep 2 type osd step emit
If you really need the "up to 2" approach then maybe you can split each host into two "host" crush buckets, with half the OSDs in each. Then a normal host-wise rule should work.
Cheers, Dan
-- Fulvio Galeazzi GARR-CSD Department skype: fgaleazzi70 tel.: +39-334-6533-250
Hold on: 8+4 needs 12 osds but you only show 10 there. Shouldn't you choose 6 type host and then chooseleaf 2 type osd? .. Dan On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi@garr.it> wrote:
Hallo Dan, Bryan, I have a rule similar to yours, for an 8+4 pool, with only difference that I replaced the second "choose" with "chooseleaf", which I understand should make no difference:
rule default.rgw.buckets.data { id 6 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class big step choose indep 5 type host step chooseleaf indep 2 type osd step emit }
I am on Nautilus 14.2.16 and while performing a maintenance the other day, I noticed 2 PGs were incomplete and caused troubles to some users. I then verified that (thanks Bryan for the command):
[cephmgr@cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map 116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' ; done | sort | uniq -c | sort -n -k1 2 r2srv07.ct1.box.garr 2 r2srv10.ct1.box.garr 2 r3srv07.ct1.box.garr 4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02. May be this happened due to some temporary unavailability of the host at some point? As all my servers are now up and running, is there a way to force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
Hi Bryan,
I had to do something similar, and never found a rule to place "up to" 2 chunks per host, so I stayed with the placement of *exactly* 2 chunks per host.
But I did this slightly differently to what you wrote earlier: my rule chooses exactly 4 hosts, then chooses exactly 2 osds on each:
type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 4 type host step choose indep 2 type osd step emit
If you really need the "up to 2" approach then maybe you can split each host into two "host" crush buckets, with half the OSDs in each. Then a normal host-wise rule should work.
Cheers, Dan
-- Fulvio Galeazzi GARR-CSD Department skype: fgaleazzi70 tel.: +39-334-6533-250
Hallo Dan, Nathan, thanks for your replies and apologies for my silence. Sorry I had made a typo... the rule is really 6+4. And to reply to Nathan's message, the rule was built like this in anticipation of getting additional servers, at which point in time I will relax the "2 chunks per OSD" part. [cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd pool get default.rgw.buckets.data erasure_code_profile erasure_code_profile: ec_6and4_big [cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd erasure-code-profile get ec_6and4_big crush-device-class=big crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=6 m=4 plugin=jerasure technique=reed_sol_van w=8 Indeed, Dan: [cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd dump | grep upmap | grep 116.453 pg_upmap_items 116.453 [76,49,129,108] Don't think I ever set such an upmap myself. Do you think it would be good to try and remove all upmaps, let the upmap balancer do its magic, and check again? Thanks! Fulvio On 20/05/2021 18:59, Dan van der Ster wrote:
Hold on: 8+4 needs 12 osds but you only show 10 there. Shouldn't you choose 6 type host and then chooseleaf 2 type osd?
.. Dan
On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi@garr.it <mailto:fulvio.galeazzi@garr.it>> wrote:
Hallo Dan, Bryan, I have a rule similar to yours, for an 8+4 pool, with only difference that I replaced the second "choose" with "chooseleaf", which I understand should make no difference:
rule default.rgw.buckets.data { id 6 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class big step choose indep 5 type host step chooseleaf indep 2 type osd step emit }
I am on Nautilus 14.2.16 and while performing a maintenance the other day, I noticed 2 PGs were incomplete and caused troubles to some users. I then verified that (thanks Bryan for the command):
[cephmgr@cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map 116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' ; done | sort | uniq -c | sort -n -k1 2 r2srv07.ct1.box.garr 2 r2srv10.ct1.box.garr 2 r3srv07.ct1.box.garr 4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02. May be this happened due to some temporary unavailability of the host at some point? As all my servers are now up and running, is there a way to force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto: > Hi Bryan, > > I had to do something similar, and never found a rule to place "up to" > 2 chunks per host, so I stayed with the placement of *exactly* 2 > chunks per host. > > But I did this slightly differently to what you wrote earlier: my rule > chooses exactly 4 hosts, then chooses exactly 2 osds on each: > > type erasure > min_size 3 > max_size 10 > step set_chooseleaf_tries 5 > step set_choose_tries 100 > step take default class hdd > step choose indep 4 type host > step choose indep 2 type osd > step emit > > If you really need the "up to 2" approach then maybe you can split > each host into two "host" crush buckets, with half the OSDs in each. > Then a normal host-wise rule should work. > > Cheers, Dan >
Hi Fulvio, I suggest removing only the upmaps which are clearly incorrect, and then see if the upmap balancer re-creates them. Perhaps they were created when they were not incorrect, when you had a different crush rule? Or perhaps you're running an old version of ceph which had buggy balancer implementation? Cheers, Dan On Thu, May 27, 2021 at 5:16 PM Fulvio Galeazzi <fulvio.galeazzi@garr.it> wrote:
Hallo Dan, Nathan, thanks for your replies and apologies for my silence.
Sorry I had made a typo... the rule is really 6+4. And to reply to Nathan's message, the rule was built like this in anticipation of getting additional servers, at which point in time I will relax the "2 chunks per OSD" part.
[cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd pool get default.rgw.buckets.data erasure_code_profile erasure_code_profile: ec_6and4_big [cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd erasure-code-profile get ec_6and4_big crush-device-class=big crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=6 m=4 plugin=jerasure technique=reed_sol_van w=8
Indeed, Dan:
[cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd dump | grep upmap | grep 116.453 pg_upmap_items 116.453 [76,49,129,108]
Don't think I ever set such an upmap myself. Do you think it would be good to try and remove all upmaps, let the upmap balancer do its magic, and check again?
Thanks!
Fulvio
On 20/05/2021 18:59, Dan van der Ster wrote:
Hold on: 8+4 needs 12 osds but you only show 10 there. Shouldn't you choose 6 type host and then chooseleaf 2 type osd?
.. Dan
On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi@garr.it <mailto:fulvio.galeazzi@garr.it>> wrote:
Hallo Dan, Bryan, I have a rule similar to yours, for an 8+4 pool, with only difference that I replaced the second "choose" with "chooseleaf", which I understand should make no difference:
rule default.rgw.buckets.data { id 6 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class big step choose indep 5 type host step chooseleaf indep 2 type osd step emit }
I am on Nautilus 14.2.16 and while performing a maintenance the other day, I noticed 2 PGs were incomplete and caused troubles to some users. I then verified that (thanks Bryan for the command):
[cephmgr@cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map 116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' ; done | sort | uniq -c | sort -n -k1 2 r2srv07.ct1.box.garr 2 r2srv10.ct1.box.garr 2 r3srv07.ct1.box.garr 4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02. May be this happened due to some temporary unavailability of the host at some point? As all my servers are now up and running, is there a way to force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto: > Hi Bryan, > > I had to do something similar, and never found a rule to place "up to" > 2 chunks per host, so I stayed with the placement of *exactly* 2 > chunks per host. > > But I did this slightly differently to what you wrote earlier: my rule > chooses exactly 4 hosts, then chooses exactly 2 osds on each: > > type erasure > min_size 3 > max_size 10 > step set_chooseleaf_tries 5 > step set_choose_tries 100 > step take default class hdd > step choose indep 4 type host > step choose indep 2 type osd > step emit > > If you really need the "up to 2" approach then maybe you can split > each host into two "host" crush buckets, with half the OSDs in each. > Then a normal host-wise rule should work. > > Cheers, Dan >
Hallo Dan, I am using Nautilus with a slightly outdated version 14.2.16, and I don't remember me playing with upmaps in the past. Following your suggestion, I removed a bunch of upmaps (the "longer" lines) and after a while I verified that all PGs are properly mapped. Thanks! Fulvio Il 5/27/2021 5:33 PM, Dan van der Ster ha scritto:
Hi Fulvio,
I suggest removing only the upmaps which are clearly incorrect, and then see if the upmap balancer re-creates them. Perhaps they were created when they were not incorrect, when you had a different crush rule? Or perhaps you're running an old version of ceph which had buggy balancer implementation?
Cheers, Dan
On Thu, May 27, 2021 at 5:16 PM Fulvio Galeazzi <fulvio.galeazzi@garr.it> wrote:
Hallo Dan, Nathan, thanks for your replies and apologies for my silence.
Sorry I had made a typo... the rule is really 6+4. And to reply to Nathan's message, the rule was built like this in anticipation of getting additional servers, at which point in time I will relax the "2 chunks per OSD" part.
[cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd pool get default.rgw.buckets.data erasure_code_profile erasure_code_profile: ec_6and4_big [cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd erasure-code-profile get ec_6and4_big crush-device-class=big crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=6 m=4 plugin=jerasure technique=reed_sol_van w=8
Indeed, Dan:
[cephmgr@cephAdmPA1.cephAdmPA1 ~]$ ceph osd dump | grep upmap | grep 116.453 pg_upmap_items 116.453 [76,49,129,108]
Don't think I ever set such an upmap myself. Do you think it would be good to try and remove all upmaps, let the upmap balancer do its magic, and check again?
Thanks!
Fulvio
On 20/05/2021 18:59, Dan van der Ster wrote:
Hold on: 8+4 needs 12 osds but you only show 10 there. Shouldn't you choose 6 type host and then chooseleaf 2 type osd?
.. Dan
On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi@garr.it <mailto:fulvio.galeazzi@garr.it>> wrote:
Hallo Dan, Bryan, I have a rule similar to yours, for an 8+4 pool, with only difference that I replaced the second "choose" with "chooseleaf", which I understand should make no difference:
rule default.rgw.buckets.data { id 6 type erasure min_size 3 max_size 10 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class big step choose indep 5 type host step chooseleaf indep 2 type osd step emit }
I am on Nautilus 14.2.16 and while performing a maintenance the other day, I noticed 2 PGs were incomplete and caused troubles to some users. I then verified that (thanks Bryan for the command):
[cephmgr@cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map 116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' ; done | sort | uniq -c | sort -n -k1 2 r2srv07.ct1.box.garr 2 r2srv10.ct1.box.garr 2 r3srv07.ct1.box.garr 4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02. May be this happened due to some temporary unavailability of the host at some point? As all my servers are now up and running, is there a way to force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto: > Hi Bryan, > > I had to do something similar, and never found a rule to place "up to" > 2 chunks per host, so I stayed with the placement of *exactly* 2 > chunks per host. > > But I did this slightly differently to what you wrote earlier: my rule > chooses exactly 4 hosts, then chooses exactly 2 osds on each: > > type erasure > min_size 3 > max_size 10 > step set_chooseleaf_tries 5 > step set_choose_tries 100 > step take default class hdd > step choose indep 4 type host > step choose indep 2 type osd > step emit > > If you really need the "up to 2" approach then maybe you can split > each host into two "host" crush buckets, with half the OSDs in each. > Then a normal host-wise rule should work. > > Cheers, Dan >
-- Fulvio Galeazzi GARR-CSD Department skype: fgaleazzi70 tel.: +39-334-6533-250
The obvious thing to do is to set 4+2 instead - is that not an option? On Wed, May 12, 2021 at 11:58 AM Bryan Stillwell <bstillwell@godaddy.com> wrote:
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host.
If I use the default rule with an osd failure domain like this:
step take default step choose indep 0 type osd step emit
I get clustering of 3-4 chunks on some of the hosts:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 8.0 1 harrahs 3 paris 4 aladdin 8.1 1 aladdin 1 excalibur 2 mandalaybay 4 paris 8.2 1 harrahs 2 aladdin 2 mirage 3 paris ...
However, if I change the rule to use:
step take default step choose indep 0 type host step chooseleaf indep 2 type osd step emit
I get the data spread across 4 hosts with 2 chunks per host:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do
echo $pg for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host' done | sort | uniq -c | sort -n -k1 done 8.0 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.1 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.2 2 harrahs 2 mandalaybay 2 mirage 2 paris ...
Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab).
Thanks, Bryan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Bryan Stillwell
-
Dan van der Ster
-
Fulvio Galeazzi
-
Ján Senko
-
Nathan Fish