18 Feb
2020
18 Feb
'20
11:56 a.m.
Hello
I have a cluster, (Nautilus 14.2.4) where one pool I'd like to keep on a
dedicated OSDs. So I setup a rule that covers *3* dedicated OSDs (using
device classes) and assigned it to pool with replication factor *3*. Only
10% PGs were assigned and rebalanced, where rest of them stuck in
*undersized* state.
What mechanism prevents CRUSH algorithm to assign the same set of OSDs to
all PGs in a pool? How can I control it?
Jacek
--
Jacek Suchenia
jacek.suchenia(a)gmail.com
19 Feb
19 Feb
12:27 a.m.
On 2/18/20 6:56 PM, Jacek Suchenia wrote:
Hello
I have a cluster, (Nautilus 14.2.4) where one pool I'd like to keep on a
dedicated OSDs. So I setup a rule that covers *3* dedicated OSDs (using
device classes) and assigned it to pool with replication factor *3*. Only
10% PGs were assigned and rebalanced, where rest of them stuck in
*undersized* state.
Can you share the rule and some snippets of the CRUSHMap?
Wido
> What mechanism prevents CRUSH algorithm to assign the same set of OSDs to
> all PGs in a pool? How can I control it?
>
> Jacek
>
2:41 a.m.
Hello Wido
Sure, here is a rule:
ceph osd crush rule dump s3_rule
{
"rule_id": 1,
"rule_name": "s3_rule",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -21,
"item_name": "default~s3"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
and here is a shadow crush map:
-21 s3 7.09189 root default~s3
-20 s3 7.09189 region lab1~s3
-19 s3 7.09189 room cr1.lab1~s3
-18 s3 7.09189 rack sr1.cr1.lab1~s3
-15 s3 3.53830 host kw01sv09.sr1.cr1.lab1~s3
11 s3 3.53830 osd.11
-17 s3 3.53830 host kw01sv10.sr1.cr1.lab1~s3
10 s3 3.53830 osd.10
-16 s3 0.01529 host kw01sv11.sr1.cr1.lab1~s3
0 s3 0.01529 osd.0
Now status is:
25 pgs degraded, 25 pgs undersized
All of them are from the same pool, while this pool is using 32 PGs - so 7
are correctly assigned on [0, 10, 11] while the rest is only on [10, 11]
Jacek
śr., 19 lut 2020 o 07:27 Wido den Hollander <wido(a)42on.com> napisał(a):
On 2/18/20 6:56 PM, Jacek Suchenia wrote:
--
Jacek Suchenia
jacek.suchenia(a)gmail.com
Hello
I have a cluster, (Nautilus 14.2.4) where one pool I'd like to keep on a
dedicated OSDs. So I setup a rule that covers *3* dedicated OSDs (using
device classes) and assigned it to pool with replication factor *3*. Only
10% PGs were assigned and rebalanced, where rest of them stuck in
*undersized* state.
Can you share the rule and some snippets of the CRUSHMap?
Wido
What mechanism prevents CRUSH algorithm to assign
the same set of OSDs to
all PGs in a pool? How can I control it?
Jacek
2:47 a.m.
Den ons 19 feb. 2020 kl 09:42 skrev Jacek Suchenia <jacek.suchenia(a)gmail.com
:
Hello Wido
Sure, here is a rule:
-15 s3 3.53830 host kw01sv09.sr1.cr1.lab1~s3
11 s3 3.53830 osd.11
-17 s3 3.53830 host kw01sv10.sr1.cr1.lab1~s3
10 s3 3.53830 osd.10
-16 s3 0.01529 host kw01sv11.sr1.cr1.lab1~s3
0 s3 0.01529 osd.0
The sizes seem _very_ uneven? Perhaps it figures it can't place another PG
on osd.0 due to its tiny size, and hence can't form a decent replica=3
using it, and it can't form one without it either, since you have only
those OSDs.
--
May the most significant bit of your life be positive.
3 a.m.
Janne
Thanks for good spot however all of them are 3.53830, that change was left
after some tests to kick CRUSH algorithm
Jacek
śr., 19 lut 2020 o 09:47 Janne Johansson <icepic.dz(a)gmail.com> napisał(a):
Den ons 19 feb. 2020 kl 09:42 skrev Jacek Suchenia
<
jacek.suchenia(a)gmail.com>gt;:
--
Jacek Suchenia
jacek.suchenia(a)gmail.com
Hello Wido
Sure, here is a rule:
-15 s3 3.53830 host kw01sv09.sr1.cr1.lab1~s3
11 s3 3.53830 osd.11
-17 s3 3.53830 host kw01sv10.sr1.cr1.lab1~s3
10 s3 3.53830 osd.10
-16 s3 0.01529 host kw01sv11.sr1.cr1.lab1~s3
0 s3 0.01529 osd.0
The sizes seem _very_ uneven? Perhaps it figures it can't place another PG
on osd.0 due to its tiny size, and hence can't form a decent replica=3
using it, and it can't form one without it either, since you have only
those OSDs.
--
May the most significant bit of your life be positive.
3:38 a.m.
Ok, I found an issue
I changed a class when OSD was reweighted then weight for this OSD in this
class was different that default (current) one
And yes - significantly different size caused this problem with degraded
and undersized
Thanks Janne and Wido for help
Jacek
śr., 19 lut 2020 o 10:00 Jacek Suchenia <jacek.suchenia(a)gmail.com>
napisał(a):
Janne
Thanks for good spot however all of them are 3.53830, that change was left
after some tests to kick CRUSH algorithm
Jacek
śr., 19 lut 2020 o 09:47 Janne Johansson <icepic.dz(a)gmail.com> napisał(a):
--
Jacek Suchenia
jacek.suchenia(a)gmail.com
Den ons 19 feb. 2020 kl 09:42 skrev Jacek
Suchenia <
jacek.suchenia(a)gmail.com>gt;:
--
Jacek Suchenia
jacek.suchenia(a)gmail.com
Hello Wido
Sure, here is a rule:
-15 s3 3.53830 host kw01sv09.sr1.cr1.lab1~s3
11 s3 3.53830 osd.11
-17 s3 3.53830 host kw01sv10.sr1.cr1.lab1~s3
10 s3 3.53830 osd.10
-16 s3 0.01529 host kw01sv11.sr1.cr1.lab1~s3
0 s3 0.01529 osd.0
The sizes seem _very_ uneven? Perhaps it figures it can't place another
PG on osd.0 due to its tiny size, and hence can't form a decent replica=3
using it, and it can't form one without it either, since you have only
those OSDs.
--
May the most significant bit of your life be positive.
1541
days inactive
1542
days old
5 comments
3 participants
participants (3)
-
Jacek Suchenia -
Janne Johansson -
Wido den Hollander