Hello I have a cluster, (Nautilus 14.2.4) where one pool I'd like to keep on a dedicated OSDs. So I setup a rule that covers *3* dedicated OSDs (using device classes) and assigned it to pool with replication factor *3*. Only 10% PGs were assigned and rebalanced, where rest of them stuck in *undersized* state. What mechanism prevents CRUSH algorithm to assign the same set of OSDs to all PGs in a pool? How can I control it? Jacek -- Jacek Suchenia jacek.suchenia@gmail.com
On 2/18/20 6:56 PM, Jacek Suchenia wrote:
Hello
I have a cluster, (Nautilus 14.2.4) where one pool I'd like to keep on a dedicated OSDs. So I setup a rule that covers *3* dedicated OSDs (using device classes) and assigned it to pool with replication factor *3*. Only 10% PGs were assigned and rebalanced, where rest of them stuck in *undersized* state.
Can you share the rule and some snippets of the CRUSHMap? Wido
What mechanism prevents CRUSH algorithm to assign the same set of OSDs to all PGs in a pool? How can I control it?
Jacek
Hello Wido Sure, here is a rule: ceph osd crush rule dump s3_rule { "rule_id": 1, "rule_name": "s3_rule", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~s3" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } and here is a shadow crush map: -21 s3 7.09189 root default~s3 -20 s3 7.09189 region lab1~s3 -19 s3 7.09189 room cr1.lab1~s3 -18 s3 7.09189 rack sr1.cr1.lab1~s3 -15 s3 3.53830 host kw01sv09.sr1.cr1.lab1~s3 11 s3 3.53830 osd.11 -17 s3 3.53830 host kw01sv10.sr1.cr1.lab1~s3 10 s3 3.53830 osd.10 -16 s3 0.01529 host kw01sv11.sr1.cr1.lab1~s3 0 s3 0.01529 osd.0 Now status is: 25 pgs degraded, 25 pgs undersized All of them are from the same pool, while this pool is using 32 PGs - so 7 are correctly assigned on [0, 10, 11] while the rest is only on [10, 11] Jacek śr., 19 lut 2020 o 07:27 Wido den Hollander <wido@42on.com> napisał(a):
On 2/18/20 6:56 PM, Jacek Suchenia wrote:
Hello
I have a cluster, (Nautilus 14.2.4) where one pool I'd like to keep on a dedicated OSDs. So I setup a rule that covers *3* dedicated OSDs (using device classes) and assigned it to pool with replication factor *3*. Only 10% PGs were assigned and rebalanced, where rest of them stuck in *undersized* state.
Can you share the rule and some snippets of the CRUSHMap?
Wido
What mechanism prevents CRUSH algorithm to assign the same set of OSDs to all PGs in a pool? How can I control it?
Jacek
-- Jacek Suchenia jacek.suchenia@gmail.com
Den ons 19 feb. 2020 kl 09:42 skrev Jacek Suchenia <jacek.suchenia@gmail.com
:
Hello Wido
Sure, here is a rule: -15 s3 3.53830 host kw01sv09.sr1.cr1.lab1~s3 11 s3 3.53830 osd.11 -17 s3 3.53830 host kw01sv10.sr1.cr1.lab1~s3 10 s3 3.53830 osd.10 -16 s3 0.01529 host kw01sv11.sr1.cr1.lab1~s3 0 s3 0.01529 osd.0
The sizes seem _very_ uneven? Perhaps it figures it can't place another PG on osd.0 due to its tiny size, and hence can't form a decent replica=3 using it, and it can't form one without it either, since you have only those OSDs. -- May the most significant bit of your life be positive.
Janne Thanks for good spot however all of them are 3.53830, that change was left after some tests to kick CRUSH algorithm Jacek śr., 19 lut 2020 o 09:47 Janne Johansson <icepic.dz@gmail.com> napisał(a):
Den ons 19 feb. 2020 kl 09:42 skrev Jacek Suchenia < jacek.suchenia@gmail.com>:
Hello Wido
Sure, here is a rule: -15 s3 3.53830 host kw01sv09.sr1.cr1.lab1~s3 11 s3 3.53830 osd.11 -17 s3 3.53830 host kw01sv10.sr1.cr1.lab1~s3 10 s3 3.53830 osd.10 -16 s3 0.01529 host kw01sv11.sr1.cr1.lab1~s3 0 s3 0.01529 osd.0
The sizes seem _very_ uneven? Perhaps it figures it can't place another PG on osd.0 due to its tiny size, and hence can't form a decent replica=3 using it, and it can't form one without it either, since you have only those OSDs.
-- May the most significant bit of your life be positive.
-- Jacek Suchenia jacek.suchenia@gmail.com
Ok, I found an issue I changed a class when OSD was reweighted then weight for this OSD in this class was different that default (current) one And yes - significantly different size caused this problem with degraded and undersized Thanks Janne and Wido for help Jacek śr., 19 lut 2020 o 10:00 Jacek Suchenia <jacek.suchenia@gmail.com> napisał(a):
Janne
Thanks for good spot however all of them are 3.53830, that change was left after some tests to kick CRUSH algorithm
Jacek
śr., 19 lut 2020 o 09:47 Janne Johansson <icepic.dz@gmail.com> napisał(a):
Den ons 19 feb. 2020 kl 09:42 skrev Jacek Suchenia < jacek.suchenia@gmail.com>:
Hello Wido
Sure, here is a rule: -15 s3 3.53830 host kw01sv09.sr1.cr1.lab1~s3 11 s3 3.53830 osd.11 -17 s3 3.53830 host kw01sv10.sr1.cr1.lab1~s3 10 s3 3.53830 osd.10 -16 s3 0.01529 host kw01sv11.sr1.cr1.lab1~s3 0 s3 0.01529 osd.0
The sizes seem _very_ uneven? Perhaps it figures it can't place another PG on osd.0 due to its tiny size, and hence can't form a decent replica=3 using it, and it can't form one without it either, since you have only those OSDs.
-- May the most significant bit of your life be positive.
-- Jacek Suchenia jacek.suchenia@gmail.com
-- Jacek Suchenia jacek.suchenia@gmail.com
participants (3)
-
Jacek Suchenia
-
Janne Johansson
-
Wido den Hollander