stretched cluster or not, with mon in 3 DC and osds on 2 DC
Hi, I'm currently reading the documentation about stretched cluster, I would like to known if it's needed or not with this kind of 3 dc setup: 3km (0.2ms) DC1--------------DC2 30km(3ms) | | 30km (2-3ms) |--------DC3------ DC1 && DC2 are near each other, small latency. (0.2ms) DC3 is at 30km with bigger latency. (2ms) separated links between dc with different physical path 1 monitor on each dc osd on DC1/DC2, with size=4 Cluster is full nvme or ssd, lowest latency is required for osd replication. Now, I really don't known if latency monitor at DC3 could have an impact on osd read/write latency is this monitor elected ? vs stretched cluster with osd only use local dc monitors ? What is the advantage of stretch cluster here ? (with good redudant links between sites)
Hi, since I just read that documentation page [1] on Friday, I can't tell you anything that isn't on that page. But that particular problem of which monitor gets elected should be solvable simply by using connectivity election mode [2], shouldn't it? Apart from the latency to the mon, the stretch cluster is mainly about the failover characteristics of the OSDs: When DC1 or DC2 fails, without a stretch cluster, the other DC will try to replicate all the data again to reach size=4 again. With a stretch cluster, it will happily live with size=2 until the other DC comes back online. So when it's right to assume that if - god forbid - one of the DCs goes offline, it will come back online not too long after again, so that the cluster can live with size=2 during that phase, then a stretch cluster probably is the better choice. Also, as the documentation states, there are edge cases where even given an appropriate CRUSH rule, size=4 min_size=2 don't necessarily mean you have a live copy of every PG in each of the two DCs. Best regards, Jan-Philipp [1]: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/ [2]: https://docs.ceph.com/en/latest/rados/operations/change-mon-elections/
participants (2)
-
aderumier＠odiso.com
-
Jan-Philipp Litza