On 07-03-2024 18:16, Gregory Farnum wrote:
On Thu, Mar 7, 2024 at 9:09 AM Stefan Kooman <stefan@bit.nl> wrote:
Hi,
TL;DR
Failure domain considered is data center. Cluster in stretch mode [1].
- What is the minimum amount of monitor nodes (apart from tie breaker) needed per failure domain?
You need at least two monitors per site. This is because in stretch mode, OSDs can *only* connect to monitors in their data center. You don't want a restarting monitor to lead to the entire cluster switching to degraded stretch mode.
It does not seem to work like that. I mean, it's true that OSDs within that data center cannot peer or be restarted when the monitor is unavailable, but it can be offline without leading to unavailability. When I stop a monitor (does not matter which one) the cluster stays online. If one of the monitors in a data center is offline, OSDs in that datacenter that get restarted won't be able to come online anymore (as expected). Only when the whole data center is offline will stretch mode be triggered. The docs are pretty clear about this, and it's exactly what I see in practice: "If all OSDs and monitors in one of the data centers become inaccessible at once, the surviving data center enters a “degraded stretch mode”." It does not have to be "all at once" though. When the last remaining daemon in a failure domain goes offline degraded stretch mode gets triggered. Offline all OSDs and keep the monitor(s) alive? 100% inactive PGs as the "peer from two buckets" enforcement rule is still active (as designed).
- What is the minimum amount of storage nodes needed per failure domain?
I don't think there's anything specific here beyond supporting 2 replicas per site (so, at least two OSD hosts unless you configure things weirdly).
lol. I personally don't think its weird: it's the counterpart of the minimal replicated 3-node cluster.
- Are device classes supported with stretch mode?
Stretch mode doesn't have any specific logic around device classes, so if you provide CRUSH rules that work with them it should be okay? But definitely not tested.
It does not seem to work. But I'll repeat the process once more and double check.
- is min_size = 1 in "degraded stretch mode" a hard coded requirement or can this be changed to it at leave min_size = 2 (yes, I'm aware that no other OSD may go down in the surviving data center or PGs will become unavailable).
Hmm, I'm pretty sure this is hard coded but haven't checked in a while. Hopefully somebody else can chime in here.
I would like this to be configurable to be honest. As it's a policy decision and not a technical requirement. Non stretch mode clusters also have the freedom of choice. If the operator values data consistency over data availability it should be honored.
I've converted a (test) 3 node replicated cluster (2 storage nodes, 1 node with monitor only, min_size=2, size=4) setup to a "stretch mode" setup [1]. That works as expected.
CRUSH rule (adjusted to work with 1 host and 2 OSDs per device class per data center only)
rule stretch_rule { id 5 type replicated step take dc1 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take dc2 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit }
The documentation seems to suggest that 2 storage nodes and 2 monitor nodes are needed at a minimum. Is that correct? I wonder why? For a minimal (as possible) cluster I don't see the need for one additional monitor per datacenter. Does the tiebreaker monitor function as a normal monitor (apart from it not allowed to become leader)?
When stretch rules with device classes are used things don't work as expected anymore. Example crush rule:
rule stretch_rule_ssd { id 4 type replicated step take dc1 class ssd step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take dc2 class ssd step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit }
A similar crush rule for hdd exists. When I change the crush_rule for one of the pools to use stretch_rule_ssd the PGs on OSDs with device class ssd become inactive as soon as one of the data centers goes offline (and "degraded stretched mode" has been activated, and only 1 bucket, data center, is needed for peering). I don't understand why. Another issue with this is that as soon as the datacenter is online again, the recovery will never finish by itself and a "ceph osd force_healthy_stretch_mode --yes-i-really-mean-it" is needed to get HEALTH_OK
Hmm. My first guess is there's something going on with the shadow CRUSH tree from device classes, but somebody would need to dig into that. I think you should file a ticket.
I will do some more testing and file a ticket if I'm confident it should work but doesn't. Thanks! Gr. Stefan