On 30/04/2025 00:14, Conrad Hoffmann wrote:
Hello,
I am trying to get a better understanding of how Ceph handles changes to CRUSH rules, such as changing the failure domain. I performed this (maybe somewhat academic, sorry) excercise, and would love to verify my conclusions (or get a good explanation of my observations):
Starting point: - 2 Nodes - 6 OSDs each - osd_crush_chooseleaf_type = 0 (OSD) - pool size = 3 - pool min size = 2
As expected, the default CRUSH rule would distribute each PG across three random OSDs, without regard for the host they are on.
I then added the following CRUSH rule (initially unused):
rule replicated_three_over_two { id 1 type replicated step take default step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit }
Tests with `crushtool` gave me confidence that it worked as I expected: unlike the default rule, this one always made sure at least one copy was on a different host.
When I applied this to rule to an existing pool, for a moment I got a HEALTH_WARN "Reduced data availability: 6 pgs peering" and the PGs being shown as inactive (cluster then proceeded to heal itself). This sort of surprised me. I have read threads like [1], where it seems even larger remappings go down without a hitch. I changed back to the default rule, this time with `osd set norebalance` set. But the outcome was the same (brief reduced availability, then healed).
Having read more about peering, I now assume that a very brief interruption due to peering is likely unavoidable?
But, more importantly, the length of the interruption probably does not scale with the size of the pool (I don't have much data in there right now). So changing the rule for a big pool would still mean only a brief interruption, is that correct?
Trying to wrap my head around what exactly the lower-level implications of changing the CRUSH rule could be, I remembered some forum thread I had read, but initially dismissed as somewhat incoherent. Someone claimed that one should set the pool's min_size to 1 when changing rules. For science, I gave this a try. Interestingly enough, the warning about reduced data availability didn't happen (even though there were again inactive/peering PGs).
I arrived at the following mental model of what happens on a CRUSH rule update: the lead OSD remains steady, but the placement of the two replicas is re-calculated. In case they need to be moved, peering needs to happen, briefly blocking. But, in case of min_size = 1, the lead OSD, I don't know, says YOLO and accepts writes without having finished peering? If so, why would the PG be considered inactive, though?
And, out of curiosity: besides the obvious possibility of corrupting your only copy of an object, are there any implications to setting min_size to 1 before changing a rule? Like, could a write in the right moment cause the peering process to permanently fail or something?
[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/UUXT4EBPZVO5...
Thanks in advance for any enlightenment :) Conrad _______________________________________________
I would not comment on specific corner case where you first set osd_crush_chooseleaf_type = 0 (osd) and have only 2 nodes, as there could be exceptions i am not aware of. But if i understand you, you are interested in general idea of how ceph handles changes to crush rules and understand how peering is involved..etc. So i will comment on general case (no osd_crush_chooseleaf_type changes, cluster with 3+ nodes). Assume you have a pg in pool with failure domain host, mapped to OSDs A, B, C and is active-clean. Then you change rule to failure domain rack...the monitors will issue a new epoch (version) map which maps pg to OSD D, E, F. To my understanding, once this mapping is computed, it really does not matter how it came to be: whether it came from Crush failure domain change (our case), or from OSDs or hosts being added or removed or by upmap override/bypass to Crush. OSD D will then get this mapping and realizes it is primary for this pg, it will lookup from past epoch and understand the prev epoch had A, B, C in active clean state, so it will initiate peering with B, C, D, E, F (had the prev state not been active-clean, more history and more OSDs may be involved). D realizes that temporarily, it should make A a temporary primary (this may involve another epoch change, not sure). The agreement (peering result) among these OSDs will result in D, E, F being in the Up set and A, B, C being in the Acting set with A as the temporary acting primary. After the peering step, the pg will be active serving i/o using A, B, C. A will then initiate backfill operation to D, E, F. Once backfiling completes, a new epoch is reached where D becomes the primary and D, E, F are both the Up set and Active set. A, B, C will not be involved with the pg anymore and D will instruct them to delete since the pg is now active+clean. If general case you should not be observing peering stuck for a long time or inactive pgs. If however there are incorrect custom backfill settings that make backfill load/traffic too high for hardware being used, if this is the case and given that such crush changes can require a large portion of your data be moved, the stress load can cause OSD heartbeats and other traffic to timeout and you could see inactive and stuck pgs...but again this would be due to incorrect settings. /maged