We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and another 1000 OSDs to be added. We are concerned with issues related to a large number of peering connections and remappings when a node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and can leave tens of minutes of PGs inactive
Could you please confirm that the periods of PG inactivity are short enough to prevent client visible glitches, or even better, could you recommend an optimal protocol for adding new OSDs to a cluster of this size to ensure a smooth convergence? Do you have any tested or recommended timeout adjustments for the stretch configuration?
You could ask the new OSDs to join with initial size of 0.01 (or 0.0001 but that has certain issues too) so that few/no PGs decides to move over, then increase size later on slowly. We used to set initial OSD crush weight to 0.0001 which works, but clients that check "max free space" think the cluster is almost full, so now we do 0.01 on existing clusters, which for some reason makes free-testing-clients not think the cluster is about to be full any second. Then stop the auto-balancer, have norebalance set, bump sizes one OSD at a time, sleep some 20-30 secs (in our case), bump next OSD and so on, and lastly run pgremapper to cancel backfills using upmap, then unset norebalance and start the balancer so it can un-upmap each such PG in an orderly fashion. -- May the most significant bit of your life be positive.