Good evening Frank, Frank Schilder <frans@dtu.dk> writes:
Hi Nico and Mark,
your crush trees look indeed like they have been converted properly to using device classes already. Changing something within one device class should not influence placement in another. Maybe I'm overlooking something?
That's the same question we were asking ourselves last week (and still do).
The only other place I know of where such a mix-up could occur are the crush rules. Do your rules look like this:
Ours are slightly simpler due to less osds/hierarchy, but besides that I think they should be fine: { "rule_id": 2, "rule_name": "ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -12, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, Differences I see is chooseleaf_indep vs. chooseleaf_firstn, from replicated vs. ec pool. But either way we are below the correct root already.
I'm really surprised that with your crush tree you observe changes in SSD implying changes in HDD placements. I was really rough on our mimic cluster with moving disks in and out and between servers and I have never seen this problem. Could it be a regression in nautilus? Is the auto-balancer interfering?
The auto balancer was off during our last big rebalance, however I am also wondering if this is a nautilus regression, as we have never seen it in Luminous. We migrated luminous -> nautilus with a baby-step (1 day or so) of mimic in between, so I am not able to say whether this behaviour already changed somehow in mimic or just in Nautilus. Our case last week was: - Move 4 SSDs from one host to another - The whole cluster - all pools - became unresponsive, slow ops everywhere. - Network bandwidth was never exhausted, enough CPU cores idle, RAM available While we do co-host SSD & HDD on the same hosts, we were not able to detect any resource exhaustion that would have prevented the other pools to function properly. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch