Hello, I am trying to get a better understanding of how Ceph handles changes to CRUSH rules, such as changing the failure domain. I performed this (maybe somewhat academic, sorry) excercise, and would love to verify my conclusions (or get a good explanation of my observations): Starting point: - 2 Nodes - 6 OSDs each - osd_crush_chooseleaf_type = 0 (OSD) - pool size = 3 - pool min size = 2 As expected, the default CRUSH rule would distribute each PG across three random OSDs, without regard for the host they are on. I then added the following CRUSH rule (initially unused): rule replicated_three_over_two { id 1 type replicated step take default step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } Tests with `crushtool` gave me confidence that it worked as I expected: unlike the default rule, this one always made sure at least one copy was on a different host. When I applied this to rule to an existing pool, for a moment I got a HEALTH_WARN "Reduced data availability: 6 pgs peering" and the PGs being shown as inactive (cluster then proceeded to heal itself). This sort of surprised me. I have read threads like [1], where it seems even larger remappings go down without a hitch. I changed back to the default rule, this time with `osd set norebalance` set. But the outcome was the same (brief reduced availability, then healed). Having read more about peering, I now assume that a very brief interruption due to peering is likely unavoidable? But, more importantly, the length of the interruption probably does not scale with the size of the pool (I don't have much data in there right now). So changing the rule for a big pool would still mean only a brief interruption, is that correct? Trying to wrap my head around what exactly the lower-level implications of changing the CRUSH rule could be, I remembered some forum thread I had read, but initially dismissed as somewhat incoherent. Someone claimed that one should set the pool's min_size to 1 when changing rules. For science, I gave this a try. Interestingly enough, the warning about reduced data availability didn't happen (even though there were again inactive/peering PGs). I arrived at the following mental model of what happens on a CRUSH rule update: the lead OSD remains steady, but the placement of the two replicas is re-calculated. In case they need to be moved, peering needs to happen, briefly blocking. But, in case of min_size = 1, the lead OSD, I don't know, says YOLO and accepts writes without having finished peering? If so, why would the PG be considered inactive, though? And, out of curiosity: besides the obvious possibility of corrupting your only copy of an object, are there any implications to setting min_size to 1 before changing a rule? Like, could a write in the right moment cause the peering process to permanently fail or something? [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/UUXT4EBPZVO5... Thanks in advance for any enlightenment :) Conrad