12 May
2021
12 May
'21
5:58 p.m.
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as
much as possible, but not more than 2 chunks per host.
If I use the default rule with an osd failure domain like this:
step take default
step choose indep 0 type osd
step emit
I get clustering of 3-4 chunks on some of the hosts:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
8.0
1 harrahs
3 paris
4 aladdin
8.1
1 aladdin
1 excalibur
2 mandalaybay
4 paris
8.2
1 harrahs
2 aladdin
2 mirage
3 paris
...
However, if I change the rule to use:
step take default
step choose indep 0 type host
step chooseleaf indep 2 type osd
step emit
I get the data spread across 4 hosts with 2 chunks per host:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
> done
8.0
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.1
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.2
2 harrahs
2 mandalaybay
2 mirage
2 paris
...
Is it possible to get the data to spread out over more hosts? I plan on expanding the
cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
Also, before you recommend adding two more hosts and switching to a host-based failure
domain, the cluster is on a variety of hardware with between 2-6 drives per host and
drives that are 4TB-12TB in size (it's part of my home lab).
Thanks,
Bryan
12 May
12 May
6:28 p.m.
I was able to figure out the solution with this rule:
step take default
step choose indep 0 type host
step chooseleaf indep 1 type osd
step emit
step take default
step choose indep 0 type host
step chooseleaf indep 1 type osd
step emit
Now the data is spread how I want it to be:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
done
8.0
1 excalibur
1 harrahs
1 mandalaybay
1 mirage
2 aladdin
2 paris
8.1
1 aladdin
1 excalibur
1 harrahs
1 mirage
2 mandalaybay
2 paris
8.2
1 aladdin
1 excalibur
1 harrahs
1 mirage
2 mandalaybay
2 paris
...
Hopefully someone else will find this useful.
Bryan
On May 12, 2021, at 9:58 AM, Bryan Stillwell
<bstillwell(a)godaddy.com> wrote:
I'm trying to figure out a CRUSH rule that will spread data out across my cluster as
much as possible, but not more than 2 chunks per host.
If I use the default rule with an osd failure domain like this:
step take default
step choose indep 0 type osd
step emit
I get clustering of 3-4 chunks on some of the hosts:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
8.0
1 harrahs
3 paris
4 aladdin
8.1
1 aladdin
1 excalibur
2 mandalaybay
4 paris
8.2
1 harrahs
2 aladdin
2 mirage
3 paris
...
However, if I change the rule to use:
step take default
step choose indep 0 type host
step chooseleaf indep 2 type osd
step emit
I get the data spread across 4 hosts with 2 chunks per host:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
> done
8.0
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.1
2 aladdin
2 harrahs
2 mandalaybay
2 paris
8.2
2 harrahs
2 mandalaybay
2 mirage
2 paris
...
Is it possible to get the data to spread out over more hosts? I plan on expanding the
cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
Also, before you recommend adding two more hosts and switching to a host-based failure
domain, the cluster is on a variety of hardware with between 2-6 drives per host and
drives that are 4TB-12TB in size (it's part of my home lab).
Thanks,
Bryan
13 May
13 May
10:58 a.m.
Would something like this work?
step take default
step choose indep 4 type host
step chooseleaf indep 1 type osd
step emit
step take default
step choose indep 0 type host
step chooseleaf indep 1 type osd
step emit
J.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
> I'm trying to figure out a CRUSH rule that will spread data out across my cluster
as much as possible, but not more than 2 chunks per host.
>
> If I use the default rule with an osd failure domain like this:
>
> step take default
>
> step choose indep 0 type osd
>
> step emit
>
> I get clustering of 3-4 chunks on some of the hosts:
>
> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
>
=======================================================================================
>
> > echo $pg
> >
> > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
> >
> > ceph osd find $osd | jq -r '.host'
> >
> > done | sort | uniq -c | sort -n -k1
>
> 8.0
>
> 1 harrahs
>
> 3 paris
>
> 4 aladdin
>
> 8.1
>
> 1 aladdin
>
> 1 excalibur
>
> 2 mandalaybay
>
> 4 paris
>
> 8.2
>
> 1 harrahs
>
> 2 aladdin
>
> 2 mirage
>
> 3 paris
>
> ...
>
> However, if I change the rule to use:
>
> step take default
>
> step choose indep 0 type host
>
> step chooseleaf indep 2 type osd
>
> step emit
>
> I get the data spread across 4 hosts with 2 chunks per host:
>
> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
>
=======================================================================================
>
> > echo $pg
> >
> > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
> >
> > ceph osd find $osd | jq -r '.host'
> >
> > done | sort | uniq -c | sort -n -k1
> >
> > done
>
> 8.0
>
> 2 aladdin
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 paris
>
> 8.1
>
> 2 aladdin
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 paris
>
> 8.2
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 mirage
>
> 2 paris
>
> ...
>
> Is it possible to get the data to spread out over more hosts? I plan on expanding the
cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
>
> Also, before you recommend adding two more hosts and switching to a host-based
failure domain, the cluster is on a variety of hardware with between 2-6 drives per host
and drives that are 4TB-12TB in size (it's part of my home lab).
>
> Thanks,
>
> Bryan
>
> ceph-users mailing list -- ceph-users(a)ceph.io
>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
14 May
14 May
8:35 p.m.
This works better than my solution. It allows the cluster to put more PGs on the systems
with more space on them:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
done
8.0
1 excalibur
1 mandalaybay
2 aladdin
2 harrahs
2 paris
8.1
1 aladdin
1 excalibur
1 harrahs
1 mirage
2 mandalaybay
2 paris
8.2
1 aladdin
1 mandalaybay
2 harrahs
2 mirage
2 paris
...
Thanks!
Bryan
On May 13, 2021, at 2:58 AM, Ján Senko
<janos(a)protonmail.ch> wrote:
Caution: This email is from an external sender. Please do not click links or open
attachments unless you recognize the sender and know the content is safe. Forward
suspicious emails to isitbad@.
Would something like this work?
step take default
step choose indep 4 type host
step chooseleaf indep 1 type osd
step emit
step take default
step choose indep 0 type host
step chooseleaf indep 1 type osd
step emit
J.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
> I'm trying to figure out a CRUSH rule that will spread data out across my cluster
as much as possible, but not more than 2 chunks per host.
>
> If I use the default rule with an osd failure domain like this:
>
> step take default
>
> step choose indep 0 type osd
>
> step emit
>
> I get clustering of 3-4 chunks on some of the hosts:
>
> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
>
=======================================================================================
>
>> echo $pg
>>
>> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
>>
>> ceph osd find $osd | jq -r '.host'
>>
>> done | sort | uniq -c | sort -n -k1
>
> 8.0
>
> 1 harrahs
>
> 3 paris
>
> 4 aladdin
>
> 8.1
>
> 1 aladdin
>
> 1 excalibur
>
> 2 mandalaybay
>
> 4 paris
>
> 8.2
>
> 1 harrahs
>
> 2 aladdin
>
> 2 mirage
>
> 3 paris
>
> ...
>
> However, if I change the rule to use:
>
> step take default
>
> step choose indep 0 type host
>
> step chooseleaf indep 2 type osd
>
> step emit
>
> I get the data spread across 4 hosts with 2 chunks per host:
>
> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
>
=======================================================================================
>
>> echo $pg
>>
>> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
>>
>> ceph osd find $osd | jq -r '.host'
>>
>> done | sort | uniq -c | sort -n -k1
>>
>> done
>
> 8.0
>
> 2 aladdin
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 paris
>
> 8.1
>
> 2 aladdin
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 paris
>
> 8.2
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 mirage
>
> 2 paris
>
> ...
>
> Is it possible to get the data to spread out over more hosts? I plan on expanding the
cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
>
> Also, before you recommend adding two more hosts and switching to a host-based
failure domain, the cluster is on a variety of hardware with between 2-6 drives per host
and drives that are 4TB-12TB in size (it's part of my home lab).
>
> Thanks,
>
> Bryan
>
> ceph-users mailing list -- ceph-users(a)ceph.io
>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
16 May
16 May
2:34 a.m.
Actually both our solutions don't work very well. Frequently the same OSD was chosen
for multiple chunks:
8.72 9751 0 0 0 40895512576 0 0 1302
active+clean 2h 224790'12801 225410:49810
[13,1,14,11,18,2,19,13]p13 [13,1,14,11,18,2,19,13]p13 2021-05-11T22:41:11.332885+0000
2021-05-11T22:41:11.332885+0000
8.7f 9695 0 0 0 40661680128 0 0 2184
active+clean 5h 224790'12850 225409:57529
[8,17,4,1,14,0,19,8]p8 [8,17,4,1,14,0,19,8]p8 2021-05-11T22:41:11.332885+0000
2021-05-11T22:41:11.332885+0000
I'm now considering using device classes and assigning the OSDs to either hdd1 or
hdd2... Unless someone has another idea?
Thanks,
Bryan
On May 14, 2021, at 12:35 PM, Bryan Stillwell
<bstillwell(a)godaddy.com> wrote:
This works better than my solution. It allows the cluster to put more PGs on the systems
with more space on them:
# for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
echo $pg
for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
ceph osd find $osd | jq -r '.host'
done | sort | uniq -c | sort -n -k1
done
8.0
1 excalibur
1 mandalaybay
2 aladdin
2 harrahs
2 paris
8.1
1 aladdin
1 excalibur
1 harrahs
1 mirage
2 mandalaybay
2 paris
8.2
1 aladdin
1 mandalaybay
2 harrahs
2 mirage
2 paris
...
Thanks!
Bryan
On May 13, 2021, at 2:58 AM, Ján Senko
<janos(a)protonmail.ch> wrote:
Caution: This email is from an external sender. Please do not click links or open
attachments unless you recognize the sender and know the content is safe. Forward
suspicious emails to isitbad@.
Would something like this work?
step take default
step choose indep 4 type host
step chooseleaf indep 1 type osd
step emit
step take default
step choose indep 0 type host
step chooseleaf indep 1 type osd
step emit
J.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
> I'm trying to figure out a CRUSH rule that will spread data out across my cluster
as much as possible, but not more than 2 chunks per host.
>
> If I use the default rule with an osd failure domain like this:
>
> step take default
>
> step choose indep 0 type osd
>
> step emit
>
> I get clustering of 3-4 chunks on some of the hosts:
>
> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
>
=======================================================================================
>
>> echo $pg
>>
>> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
>>
>> ceph osd find $osd | jq -r '.host'
>>
>> done | sort | uniq -c | sort -n -k1
>
> 8.0
>
> 1 harrahs
>
> 3 paris
>
> 4 aladdin
>
> 8.1
>
> 1 aladdin
>
> 1 excalibur
>
> 2 mandalaybay
>
> 4 paris
>
> 8.2
>
> 1 harrahs
>
> 2 aladdin
>
> 2 mirage
>
> 3 paris
>
> ...
>
> However, if I change the rule to use:
>
> step take default
>
> step choose indep 0 type host
>
> step chooseleaf indep 2 type osd
>
> step emit
>
> I get the data spread across 4 hosts with 2 chunks per host:
>
> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
>
=======================================================================================
>
>> echo $pg
>>
>> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
>>
>> ceph osd find $osd | jq -r '.host'
>>
>> done | sort | uniq -c | sort -n -k1
>>
>> done
>
> 8.0
>
> 2 aladdin
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 paris
>
> 8.1
>
> 2 aladdin
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 paris
>
> 8.2
>
> 2 harrahs
>
> 2 mandalaybay
>
> 2 mirage
>
> 2 paris
>
> ...
>
> Is it possible to get the data to spread out over more hosts? I plan on expanding the
cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
>
> Also, before you recommend adding two more hosts and switching to a host-based
failure domain, the cluster is on a variety of hardware with between 2-6 drives per host
and drives that are 4TB-12TB in size (it's part of my home lab).
>
> Thanks,
>
> Bryan
>
> ceph-users mailing list -- ceph-users(a)ceph.io
>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:40 p.m.
Hi Bryan,
I had to do something similar, and never found a rule to place "up to"
2 chunks per host, so I stayed with the placement of *exactly* 2
chunks per host.
But I did this slightly differently to what you wrote earlier: my rule
chooses exactly 4 hosts, then chooses exactly 2 osds on each:
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 4 type host
step choose indep 2 type osd
step emit
If you really need the "up to 2" approach then maybe you can split
each host into two "host" crush buckets, with half the OSDs in each.
Then a normal host-wise rule should work.
Cheers, Dan
On Sun, May 16, 2021 at 2:34 AM Bryan Stillwell <bstillwell(a)godaddy.com> wrote:
>
> Actually both our solutions don't work very well. Frequently the same OSD was
chosen for multiple chunks:
>
>
> 8.72 9751 0 0 0 40895512576 0 0
1302 active+clean 2h 224790'12801 225410:49810
[13,1,14,11,18,2,19,13]p13 [13,1,14,11,18,2,19,13]p13 2021-05-11T22:41:11.332885+0000
2021-05-11T22:41:11.332885+0000
> 8.7f 9695 0 0 0 40661680128 0 0
2184 active+clean 5h 224790'12850 225409:57529
[8,17,4,1,14,0,19,8]p8 [8,17,4,1,14,0,19,8]p8 2021-05-11T22:41:11.332885+0000
2021-05-11T22:41:11.332885+0000
>
> I'm now considering using device classes and assigning the OSDs to either hdd1 or
hdd2... Unless someone has another idea?
>
> Thanks,
> Bryan
>
> > On May 14, 2021, at 12:35 PM, Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
> >
> > This works better than my solution. It allows the cluster to put more PGs on
the systems with more space on them:
> >
> > # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
> >> echo $pg
> >> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
> >> ceph osd find $osd | jq -r '.host'
> >> done | sort | uniq -c | sort -n -k1
> >> done
> > 8.0
> > 1 excalibur
> > 1 mandalaybay
> > 2 aladdin
> > 2 harrahs
> > 2 paris
> > 8.1
> > 1 aladdin
> > 1 excalibur
> > 1 harrahs
> > 1 mirage
> > 2 mandalaybay
> > 2 paris
> > 8.2
> > 1 aladdin
> > 1 mandalaybay
> > 2 harrahs
> > 2 mirage
> > 2 paris
> > ...
> >
> > Thanks!
> > Bryan
> >
> >> On May 13, 2021, at 2:58 AM, Ján Senko <janos(a)protonmail.ch> wrote:
> >>
> >> Caution: This email is from an external sender. Please do not click links or
open attachments unless you recognize the sender and know the content is safe. Forward
suspicious emails to isitbad@.
> >>
> >>
> >>
> >> Would something like this work?
> >>
> >> step take default
> >> step choose indep 4 type host
> >> step chooseleaf indep 1 type osd
> >> step emit
> >> step take default
> >> step choose indep 0 type host
> >> step chooseleaf indep 1 type osd
> >> step emit
> >>
> >> J.
> >>
> >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >>
> >> On Wednesday, May 12th, 2021 at 17:58, Bryan Stillwell
<bstillwell(a)godaddy.com> wrote:
> >>
> >>> I'm trying to figure out a CRUSH rule that will spread data out
across my cluster as much as possible, but not more than 2 chunks per host.
> >>>
> >>> If I use the default rule with an osd failure domain like this:
> >>>
> >>> step take default
> >>>
> >>> step choose indep 0 type osd
> >>>
> >>> step emit
> >>>
> >>> I get clustering of 3-4 chunks on some of the hosts:
> >>>
> >>> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
> >>>
=======================================================================================
> >>>
> >>>> echo $pg
> >>>>
> >>>> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
> >>>>
> >>>> ceph osd find $osd | jq -r '.host'
> >>>>
> >>>> done | sort | uniq -c | sort -n -k1
> >>>
> >>> 8.0
> >>>
> >>> 1 harrahs
> >>>
> >>> 3 paris
> >>>
> >>> 4 aladdin
> >>>
> >>> 8.1
> >>>
> >>> 1 aladdin
> >>>
> >>> 1 excalibur
> >>>
> >>> 2 mandalaybay
> >>>
> >>> 4 paris
> >>>
> >>> 8.2
> >>>
> >>> 1 harrahs
> >>>
> >>> 2 aladdin
> >>>
> >>> 2 mirage
> >>>
> >>> 3 paris
> >>>
> >>> ...
> >>>
> >>> However, if I change the rule to use:
> >>>
> >>> step take default
> >>>
> >>> step choose indep 0 type host
> >>>
> >>> step chooseleaf indep 2 type osd
> >>>
> >>> step emit
> >>>
> >>> I get the data spread across 4 hosts with 2 chunks per host:
> >>>
> >>> for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
> >>>
=======================================================================================
> >>>
> >>>> echo $pg
> >>>>
> >>>> for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
> >>>>
> >>>> ceph osd find $osd | jq -r '.host'
> >>>>
> >>>> done | sort | uniq -c | sort -n -k1
> >>>>
> >>>> done
> >>>
> >>> 8.0
> >>>
> >>> 2 aladdin
> >>>
> >>> 2 harrahs
> >>>
> >>> 2 mandalaybay
> >>>
> >>> 2 paris
> >>>
> >>> 8.1
> >>>
> >>> 2 aladdin
> >>>
> >>> 2 harrahs
> >>>
> >>> 2 mandalaybay
> >>>
> >>> 2 paris
> >>>
> >>> 8.2
> >>>
> >>> 2 harrahs
> >>>
> >>> 2 mandalaybay
> >>>
> >>> 2 mirage
> >>>
> >>> 2 paris
> >>>
> >>> ...
> >>>
> >>> Is it possible to get the data to spread out over more hosts? I plan on
expanding the cluster in the near future and would like to see more hosts get 1 chunk
instead of 2.
> >>>
> >>> Also, before you recommend adding two more hosts and switching to a
host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives
per host and drives that are 4TB-12TB in size (it's part of my home lab).
> >>>
> >>> Thanks,
> >>>
> >>> Bryan
> >>>
> >>> ceph-users mailing list -- ceph-users(a)ceph.io
> >>>
> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
20 May
20 May
1:30 p.m.
Hallo Dan, Bryan,
I have a rule similar to yours, for an 8+4 pool, with only
difference that I replaced the second "choose" with "chooseleaf",
which
I understand should make no difference:
rule default.rgw.buckets.data {
id 6
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class big
step choose indep 5 type host
step chooseleaf indep 2 type osd
step emit
}
I am on Nautilus 14.2.16 and while performing a maintenance the other
day, I noticed 2 PGs were incomplete and caused troubles to some users.
I then verified that (thanks Bryan for the command):
[cephmgr(a)cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map
116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host'
; done | sort | uniq -c | sort -n -k1
2 r2srv07.ct1.box.garr
2 r2srv10.ct1.box.garr
2 r3srv07.ct1.box.garr
4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02.
May be this happened due to some temporary unavailability of the host at
some point? As all my servers are now up and running, is there a way to
force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
Hi Bryan,
I had to do something similar, and never found a rule to place "up to"
2 chunks per host, so I stayed with the placement of *exactly* 2
chunks per host.
But I did this slightly differently to what you wrote earlier: my rule
chooses exactly 4 hosts, then chooses exactly 2 osds on each:
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 4 type host
step choose indep 2 type osd
step emit
If you really need the "up to 2" approach then maybe you can split
each host into two "host" crush buckets, with half the OSDs in each.
Then a normal host-wise rule should work.
Cheers, Dan
--
Fulvio Galeazzi
GARR-CSD Department
skype: fgaleazzi70
tel.: +39-334-6533-250
6:36 p.m.
Hi Fulvio,
That's strange... It doesn't seem right to me.
Are there any upmaps for that PG?
ceph osd dump | grep upmap | grep 116.453
Cheers, Dan
On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi(a)garr.it>
wrote:
Hallo Dan, Bryan,
I have a rule similar to yours, for an 8+4 pool, with only
difference that I replaced the second "choose" with "chooseleaf",
which
I understand should make no difference:
rule default.rgw.buckets.data {
id 6
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class big
step choose indep 5 type host
step chooseleaf indep 2 type osd
step emit
}
I am on Nautilus 14.2.16 and while performing a maintenance the other
day, I noticed 2 PGs were incomplete and caused troubles to some users.
I then verified that (thanks Bryan for the command):
[cephmgr(a)cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map
116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host'
; done | sort | uniq -c | sort -n -k1
2 r2srv07.ct1.box.garr
2 r2srv10.ct1.box.garr
2 r3srv07.ct1.box.garr
4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02.
May be this happened due to some temporary unavailability of the host at
some point? As all my servers are now up and running, is there a way to
force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
Hi Bryan,
I had to do something similar, and never found a rule to place "up to"
2 chunks per host, so I stayed with the placement of *exactly* 2
chunks per host.
But I did this slightly differently to what you wrote earlier: my rule
chooses exactly 4 hosts, then chooses exactly 2 osds on each:
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 4 type host
step choose indep 2 type osd
step emit
If you really need the "up to 2" approach then maybe you can split
each host into two "host" crush buckets, with half the OSDs in each.
Then a normal host-wise rule should work.
Cheers, Dan
--
Fulvio Galeazzi
GARR-CSD Department
skype: fgaleazzi70
tel.: +39-334-6533-250
6:59 p.m.
Hold on: 8+4 needs 12 osds but you only show 10 there. Shouldn't you choose
6 type host and then chooseleaf 2 type osd?
.. Dan
On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi(a)garr.it>
wrote:
Hallo Dan, Bryan,
I have a rule similar to yours, for an 8+4 pool, with only
difference that I replaced the second "choose" with "chooseleaf",
which
I understand should make no difference:
rule default.rgw.buckets.data {
id 6
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class big
step choose indep 5 type host
step chooseleaf indep 2 type osd
step emit
}
I am on Nautilus 14.2.16 and while performing a maintenance the other
day, I noticed 2 PGs were incomplete and caused troubles to some users.
I then verified that (thanks Bryan for the command):
[cephmgr(a)cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map
116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r '.host'
; done | sort | uniq -c | sort -n -k1
2 r2srv07.ct1.box.garr
2 r2srv10.ct1.box.garr
2 r3srv07.ct1.box.garr
4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02.
May be this happened due to some temporary unavailability of the host at
some point? As all my servers are now up and running, is there a way to
force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
Hi Bryan,
I had to do something similar, and never found a rule to place "up to"
2 chunks per host, so I stayed with the placement of *exactly* 2
chunks per host.
But I did this slightly differently to what you wrote earlier: my rule
chooses exactly 4 hosts, then chooses exactly 2 osds on each:
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 4 type host
step choose indep 2 type osd
step emit
If you really need the "up to 2" approach then maybe you can split
each host into two "host" crush buckets, with half the OSDs in each.
Then a normal host-wise rule should work.
Cheers, Dan
--
Fulvio Galeazzi
GARR-CSD Department
skype: fgaleazzi70
tel.: +39-334-6533-250
27 May
27 May
5:16 p.m.
Hallo Dan, Nathan, thanks for your replies and apologies for my silence.
Sorry I had made a typo... the rule is really 6+4. And to reply to
Nathan's message, the rule was built like this in anticipation of
getting additional servers, at which point in time I will relax the "2
chunks per OSD" part.
[cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd pool get
default.rgw.buckets.data erasure_code_profile
erasure_code_profile: ec_6and4_big
[cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd erasure-code-profile get
ec_6and4_big
crush-device-class=big
crush-failure-domain=osd
crush-root=default
jerasure-per-chunk-alignment=false
k=6
m=4
plugin=jerasure
technique=reed_sol_van
w=8
Indeed, Dan:
[cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd dump | grep upmap | grep 116.453
pg_upmap_items 116.453 [76,49,129,108]
Don't think I ever set such an upmap myself. Do you think it would be
good to try and remove all upmaps, let the upmap balancer do its magic,
and check again?
Thanks!
Fulvio
On 20/05/2021 18:59, Dan van der Ster wrote:
Hold on: 8+4 needs 12 osds but you only show 10 there.
Shouldn't you
choose 6 type host and then chooseleaf 2 type osd?
.. Dan
On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi(a)garr.it
<mailto:fulvio.galeazzi@garr.it>> wrote:
Hallo Dan, Bryan,
I have a rule similar to yours, for an 8+4 pool, with only
difference that I replaced the second "choose" with "chooseleaf",
which
I understand should make no difference:
rule default.rgw.buckets.data {
id 6
type erasure
min_size 3
max_size 10
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class big
step choose indep 5 type host
step chooseleaf indep 2 type osd
step emit
}
I am on Nautilus 14.2.16 and while performing a maintenance the
other
day, I noticed 2 PGs were incomplete and caused troubles to some users.
I then verified that (thanks Bryan for the command):
[cephmgr(a)cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map
116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r
'.host'
; done | sort | uniq -c | sort -n -k1
2 r2srv07.ct1.box.garr
2 r2srv10.ct1.box.garr
2 r3srv07.ct1.box.garr
4 r1srv02.ct1.box.garr
You see that 4 PGs were put on r1srv02.
May be this happened due to some temporary unavailability of the
host at
some point? As all my servers are now up and running, is there a way to
force the placement rule to rerun?
Thanks!
Fulvio
Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
Hi Bryan,
I had to do something similar, and never found a rule to place
"up
to"
2 chunks per host, so I stayed with the placement
of *exactly* 2
chunks per host.
But I did this slightly differently to what you wrote earlier: my
rule
> chooses exactly 4 hosts, then chooses exactly 2 osds on each:
>
> type erasure
> min_size 3
> max_size 10
> step set_chooseleaf_tries 5
> step set_choose_tries 100
> step take default class hdd
> step choose indep 4 type host
> step choose indep 2 type osd
> step emit
>
> If you really need the "up to 2" approach then maybe you can split
> each host into two "host" crush buckets, with half the OSDs in each.
> Then a normal host-wise rule should work.
>
> Cheers, Dan
>
5:33 p.m.
Hi Fulvio,
I suggest removing only the upmaps which are clearly incorrect, and
then see if the upmap balancer re-creates them.
Perhaps they were created when they were not incorrect, when you had a
different crush rule?
Or perhaps you're running an old version of ceph which had buggy
balancer implementation?
Cheers, Dan
On Thu, May 27, 2021 at 5:16 PM Fulvio Galeazzi <fulvio.galeazzi(a)garr.it> wrote:
>
> Hallo Dan, Nathan, thanks for your replies and apologies for my silence.
>
> Sorry I had made a typo... the rule is really 6+4. And to reply to
> Nathan's message, the rule was built like this in anticipation of
> getting additional servers, at which point in time I will relax the "2
> chunks per OSD" part.
>
> [cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd pool get
> default.rgw.buckets.data erasure_code_profile
> erasure_code_profile: ec_6and4_big
> [cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd erasure-code-profile get
> ec_6and4_big
> crush-device-class=big
> crush-failure-domain=osd
> crush-root=default
> jerasure-per-chunk-alignment=false
> k=6
> m=4
> plugin=jerasure
> technique=reed_sol_van
> w=8
>
> Indeed, Dan:
>
> [cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd dump | grep upmap | grep 116.453
> pg_upmap_items 116.453 [76,49,129,108]
>
> Don't think I ever set such an upmap myself. Do you think it would be
> good to try and remove all upmaps, let the upmap balancer do its magic,
> and check again?
>
> Thanks!
>
> Fulvio
>
>
> On 20/05/2021 18:59, Dan van der Ster wrote:
> > Hold on: 8+4 needs 12 osds but you only show 10 there. Shouldn't you
> > choose 6 type host and then chooseleaf 2 type osd?
> >
> > .. Dan
> >
> >
> > On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi(a)garr.it
> > <mailto:fulvio.galeazzi@garr.it>> wrote:
> >
> > Hallo Dan, Bryan,
> > I have a rule similar to yours, for an 8+4 pool, with only
> > difference that I replaced the second "choose" with
"chooseleaf", which
> > I understand should make no difference:
> >
> > rule default.rgw.buckets.data {
> > id 6
> > type erasure
> > min_size 3
> > max_size 10
> > step set_chooseleaf_tries 5
> > step set_choose_tries 100
> > step take default class big
> > step choose indep 5 type host
> > step chooseleaf indep 2 type osd
> > step emit
> > }
> >
> > I am on Nautilus 14.2.16 and while performing a maintenance the
> > other
> > day, I noticed 2 PGs were incomplete and caused troubles to some users.
> > I then verified that (thanks Bryan for the command):
> >
> > [cephmgr(a)cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map
> > 116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r
'.host'
> > ; done | sort | uniq -c | sort -n -k1
> > 2 r2srv07.ct1.box.garr
> > 2 r2srv10.ct1.box.garr
> > 2 r3srv07.ct1.box.garr
> > 4 r1srv02.ct1.box.garr
> >
> > You see that 4 PGs were put on r1srv02.
> > May be this happened due to some temporary unavailability of the
> > host at
> > some point? As all my servers are now up and running, is there a way to
> > force the placement rule to rerun?
> >
> > Thanks!
> >
> > Fulvio
> >
> >
> > Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
> > > Hi Bryan,
> > >
> > > I had to do something similar, and never found a rule to place
> > "up to"
> > > 2 chunks per host, so I stayed with the placement of *exactly* 2
> > > chunks per host.
> > >
> > > But I did this slightly differently to what you wrote earlier: my
> > rule
> > > chooses exactly 4 hosts, then chooses exactly 2 osds on each:
> > >
> > > type erasure
> > > min_size 3
> > > max_size 10
> > > step set_chooseleaf_tries 5
> > > step set_choose_tries 100
> > > step take default class hdd
> > > step choose indep 4 type host
> > > step choose indep 2 type osd
> > > step emit
> > >
> > > If you really need the "up to 2" approach then maybe you can
split
> > > each host into two "host" crush buckets, with half the OSDs
in each.
> > > Then a normal host-wise rule should work.
> > >
> > > Cheers, Dan
> > >
>
4 Jun
4 Jun
10:50 a.m.
Hallo Dan,
I am using Nautilus with a slightly outdated version 14.2.16, and I
don't remember me playing with upmaps in the past.
Following your suggestion, I removed a bunch of upmaps (the "longer"
lines) and after a while I verified that all PGs are properly mapped.
Thanks!
Fulvio
Il 5/27/2021 5:33 PM, Dan van der Ster ha scritto:
Hi Fulvio,
I suggest removing only the upmaps which are clearly incorrect, and
then see if the upmap balancer re-creates them.
Perhaps they were created when they were not incorrect, when you had a
different crush rule?
Or perhaps you're running an old version of ceph which had buggy
balancer implementation?
Cheers, Dan
On Thu, May 27, 2021 at 5:16 PM Fulvio Galeazzi <fulvio.galeazzi(a)garr.it> wrote:
>
> Hallo Dan, Nathan, thanks for your replies and apologies for my silence.
>
> Sorry I had made a typo... the rule is really 6+4. And to reply to
> Nathan's message, the rule was built like this in anticipation of
> getting additional servers, at which point in time I will relax the "2
> chunks per OSD" part.
>
> [cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd pool get
> default.rgw.buckets.data erasure_code_profile
> erasure_code_profile: ec_6and4_big
> [cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd erasure-code-profile get
> ec_6and4_big
> crush-device-class=big
> crush-failure-domain=osd
> crush-root=default
> jerasure-per-chunk-alignment=false
> k=6
> m=4
> plugin=jerasure
> technique=reed_sol_van
> w=8
>
> Indeed, Dan:
>
> [cephmgr(a)cephAdmPA1.cephAdmPA1 ~]$ ceph osd dump | grep upmap | grep 116.453
> pg_upmap_items 116.453 [76,49,129,108]
>
> Don't think I ever set such an upmap myself. Do you think it would be
> good to try and remove all upmaps, let the upmap balancer do its magic,
> and check again?
>
> Thanks!
>
> Fulvio
>
>
> On 20/05/2021 18:59, Dan van der Ster wrote:
>> Hold on: 8+4 needs 12 osds but you only show 10 there. Shouldn't you
>> choose 6 type host and then chooseleaf 2 type osd?
>>
>> .. Dan
>>
>>
>> On Thu, May 20, 2021, 1:30 PM Fulvio Galeazzi <fulvio.galeazzi(a)garr.it
>> <mailto:fulvio.galeazzi@garr.it>> wrote:
>>
>> Hallo Dan, Bryan,
>> I have a rule similar to yours, for an 8+4 pool, with only
>> difference that I replaced the second "choose" with
"chooseleaf", which
>> I understand should make no difference:
>>
>> rule default.rgw.buckets.data {
>> id 6
>> type erasure
>> min_size 3
>> max_size 10
>> step set_chooseleaf_tries 5
>> step set_choose_tries 100
>> step take default class big
>> step choose indep 5 type host
>> step chooseleaf indep 2 type osd
>> step emit
>> }
>>
>> I am on Nautilus 14.2.16 and while performing a maintenance the
>> other
>> day, I noticed 2 PGs were incomplete and caused troubles to some users.
>> I then verified that (thanks Bryan for the command):
>>
>> [cephmgr(a)cephAdmCT1.cephAdmCT1 clusterCT]$ for osd in $(ceph pg map
>> 116.453 -f json | jq -r '.up[]'); do ceph osd find $osd | jq -r
'.host'
>> ; done | sort | uniq -c | sort -n -k1
>> 2 r2srv07.ct1.box.garr
>> 2 r2srv10.ct1.box.garr
>> 2 r3srv07.ct1.box.garr
>> 4 r1srv02.ct1.box.garr
>>
>> You see that 4 PGs were put on r1srv02.
>> May be this happened due to some temporary unavailability of the
>> host at
>> some point? As all my servers are now up and running, is there a way to
>> force the placement rule to rerun?
>>
>> Thanks!
>>
>> Fulvio
>>
>>
>> Il 5/16/2021 11:40 PM, Dan van der Ster ha scritto:
>> > Hi Bryan,
>> >
>> > I had to do something similar, and never found a rule to place
>> "up to"
>> > 2 chunks per host, so I stayed with the placement of *exactly* 2
>> > chunks per host.
>> >
>> > But I did this slightly differently to what you wrote earlier: my
>> rule
>> > chooses exactly 4 hosts, then chooses exactly 2 osds on each:
>> >
>> > type erasure
>> > min_size 3
>> > max_size 10
>> > step set_chooseleaf_tries 5
>> > step set_choose_tries 100
>> > step take default class hdd
>> > step choose indep 4 type host
>> > step choose indep 2 type osd
>> > step emit
>> >
>> > If you really need the "up to 2" approach then maybe you can
split
>> > each host into two "host" crush buckets, with half the OSDs
in each.
>> > Then a normal host-wise rule should work.
>> >
>> > Cheers, Dan
>> >
>
--
Fulvio Galeazzi
GARR-CSD Department
skype: fgaleazzi70
tel.: +39-334-6533-250
20 May
20 May
6:47 p.m.
The obvious thing to do is to set 4+2 instead - is that not an option?
On Wed, May 12, 2021 at 11:58 AM Bryan Stillwell <bstillwell(a)godaddy.com> wrote:
>
> I'm trying to figure out a CRUSH rule that will spread data out across my cluster
as much as possible, but not more than 2 chunks per host.
>
> If I use the default rule with an osd failure domain like this:
>
> step take default
> step choose indep 0 type osd
> step emit
>
> I get clustering of 3-4 chunks on some of the hosts:
>
> # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
> > echo $pg
> > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
> > ceph osd find $osd | jq -r '.host'
> > done | sort | uniq -c | sort -n -k1
> 8.0
> 1 harrahs
> 3 paris
> 4 aladdin
> 8.1
> 1 aladdin
> 1 excalibur
> 2 mandalaybay
> 4 paris
> 8.2
> 1 harrahs
> 2 aladdin
> 2 mirage
> 3 paris
> ...
>
> However, if I change the rule to use:
>
> step take default
> step choose indep 0 type host
> step chooseleaf indep 2 type osd
> step emit
>
> I get the data spread across 4 hosts with 2 chunks per host:
>
> # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r
'.pg_stats[].pgid'); do
> > echo $pg
> > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do
> > ceph osd find $osd | jq -r '.host'
> > done | sort | uniq -c | sort -n -k1
> > done
> 8.0
> 2 aladdin
> 2 harrahs
> 2 mandalaybay
> 2 paris
> 8.1
> 2 aladdin
> 2 harrahs
> 2 mandalaybay
> 2 paris
> 8.2
> 2 harrahs
> 2 mandalaybay
> 2 mirage
> 2 paris
> ...
>
> Is it possible to get the data to spread out over more hosts? I plan on expanding
the cluster in the near future and would like to see more hosts get 1 chunk instead of 2.
>
> Also, before you recommend adding two more hosts and switching to a host-based
failure domain, the cluster is on a variety of hardware with between 2-6 drives per host
and drives that are 4TB-12TB in size (it's part of my home lab).
>
> Thanks,
> Bryan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1050
days inactive
1073
days old
12 comments
5 participants
participants (5)
-
Bryan Stillwell -
Dan van der Ster -
Fulvio Galeazzi -
Ján Senko -
Nathan Fish