Can you please talk more about how you’re running the cluster and what data is already on it? There’s nothing about growing a stretch cluster that naively would lead to this behavior, but it does have a lot more constraints than a regular Ceph cluster does about which OSDs it can use to go active with (because it needs a current copy in both sites, and it’s trying to keep 4 copies up to date rather than 3). -Greg On Mon, Apr 6, 2026 at 7:26 AM Anthony D'Atri via ceph-users < ceph-users@ceph.io> wrote:
Are you using upmap-remapped to mitigate the thundering herd? And, if you're using the PG autoscaler, to mitigate the thundering herd of backfill that it adds?
On Apr 6, 2026, at 7:33 AM, Michal Strnad via ceph-users < ceph-users@ceph.io> wrote:
Hi.
We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and another 1000 OSDs to be added. We are concerned with issues related to a large number of peering connections and remappings when a node with 38 OSDs is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and can leave tens of minutes of PGs inactive, which we see reflected in the ceph status output before the cluster finally stabilizes.
In a classic (non‑stretch) Ceph cluster the convergence is noticeably quicker (inactive PGs appear only for a very short interval). In our stretch deployment, however, the window when PGs are inactive is considerably longer, consistent with our observations. The network latency between the two sites is consistently below 10 ms with no packet loss.
We are currently trying to determine whether this impacts client‑side read/write simulations via S3, but this has not yet been verified. Our next step is to test with RBD, which should be more resilient and demonstrable. Naturally, we would like to control the process in a way that minimizes the impact on clients. Could you please confirm that the periods of PG inactivity are short enough to prevent client visible glitches, or even better, could you recommend an optimal protocol for adding new OSDs to a cluster of this size to ensure a smooth convergence? Do you have any tested or recommended timeout adjustments for the stretch configuration?
Thank you for any feedback -- Michal _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io