Hi, TL;DR Failure domain considered is data center. Cluster in stretch mode [1]. - What is the minimum amount of monitor nodes (apart from tie breaker) needed per failure domain? - What is the minimum amount of storage nodes needed per failure domain? - Are device classes supported with stretch mode? - is min_size = 1 in "degraded stretch mode" a hard coded requirement or can this be changed to it at leave min_size = 2 (yes, I'm aware that no other OSD may go down in the surviving data center or PGs will become unavailable). I've converted a (test) 3 node replicated cluster (2 storage nodes, 1 node with monitor only, min_size=2, size=4) setup to a "stretch mode" setup [1]. That works as expected. CRUSH rule (adjusted to work with 1 host and 2 OSDs per device class per data center only) rule stretch_rule { id 5 type replicated step take dc1 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take dc2 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } The documentation seems to suggest that 2 storage nodes and 2 monitor nodes are needed at a minimum. Is that correct? I wonder why? For a minimal (as possible) cluster I don't see the need for one additional monitor per datacenter. Does the tiebreaker monitor function as a normal monitor (apart from it not allowed to become leader)? When stretch rules with device classes are used things don't work as expected anymore. Example crush rule: rule stretch_rule_ssd { id 4 type replicated step take dc1 class ssd step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take dc2 class ssd step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } A similar crush rule for hdd exists. When I change the crush_rule for one of the pools to use stretch_rule_ssd the PGs on OSDs with device class ssd become inactive as soon as one of the data centers goes offline (and "degraded stretched mode" has been activated, and only 1 bucket, data center, is needed for peering). I don't understand why. Another issue with this is that as soon as the datacenter is online again, the recovery will never finish by itself and a "ceph osd force_healthy_stretch_mode --yes-i-really-mean-it" is needed to get HEALTH_OK Can anyone explain to me why this is? Gr. Stefan [1]: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/