Good evening Frank,
Frank Schilder <frans(a)dtu.dk> writes:
Hi Nico and Mark,
your crush trees look indeed like they have been converted properly to
using device classes already. Changing something within one device
class should not influence placement in another. Maybe I'm overlooking
something?
That's the same question we were asking ourselves last week (and still
do).
The only other place I know of where such a mix-up
could occur are the crush rules. Do your rules look like this:
Ours are slightly simpler due to less osds/hierarchy, but besides that I
think they should be fine:
{
"rule_id": 2,
"rule_name": "ssd",
"ruleset": 2,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -12,
"item_name": "default~ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
Differences I see is chooseleaf_indep vs. chooseleaf_firstn, from
replicated vs. ec pool. But either way we are below the correct root
already.
I'm really surprised that with your crush tree you
observe changes in
SSD implying changes in HDD placements. I was really rough on our
mimic cluster with moving disks in and out and between servers and I
have never seen this problem. Could it be a regression in nautilus? Is
the auto-balancer interfering?
The auto balancer was off during our last big rebalance, however I am
also wondering if this is a nautilus regression, as we have never seen
it in Luminous.
We migrated luminous -> nautilus with a baby-step (1 day or so) of mimic
in between, so I am not able to say whether this behaviour already
changed somehow in mimic or just in Nautilus.
Our case last week was:
- Move 4 SSDs from one host to another
- The whole cluster - all pools - became unresponsive, slow ops everywhere.
- Network bandwidth was never exhausted, enough CPU cores idle, RAM available
While we do co-host SSD & HDD on the same hosts, we were not able to
detect any resource exhaustion that would have prevented the other pools
to function properly.
Best regards,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit
www.datacenterlight.ch