----- Le 2 Oct 24, à 16:21, Victor Rodriguez <vrodriguez@soltecsis.com> a écrit :
Hi,
What makes this cluster a non-local cluster?
It's hosted in OVH's 3AZ, with each host in a different DC, each at around 30-60km's away from each other, hence the relatively high latency. Yeah. 0.6 to 1ms are expect latencies for that long distances.
0'6 and 1 millisecond RTT latency seems too high for all-flash clusters and intense 4K write workloads.
I'm fully aware of the limitations imposed by the latency. I was wondering if there is something that could be done to improve performance under this conditions. Beyond the limits of physics, no.
Measured performance is more than enough for the workloads that the cluster will host, as 4k QD=1 sync writes/reads are not the main I/O pattern. So in the end, the 4K IO benchmark isn't that important. The ability to use this cluster will depend solely on the workload, then.
The upmap-read or read balancer modes may help with reads but not writes where 1.2ms+ latency will still be observed.
AFAIK upmap-read isn't available in Reef, at least does not show up in the docs [1].
Thanks! You're right. I forgot you're using Reef. Still you could try setting pg-upmap-primary manually, if I'm not mistaken.
Regards, Frédéric.
[1] [ https://docs.ceph.com/en/reef/rados/operations/balancer/ | https://docs.ceph.com/en/reef/rados/operations/balancer/ ]
Regards, Frédéric.
----- Le 1 Oct 24, à 18:23, Victor Rodriguez [ mailto:vrodriguez@soltecsis.com | vrodriguez@soltecsis.com ] a écrit :
Hello,
I'm trying to get the most from a Ceph Reef 3 node clusters, with 6 NVMe OSD each. Each node is between 0'6 and 1 millisecond RTT. Obviously performance isn't as good as with local clusters, usually around ~0'2ms. 4k in Q1D1 I/O write request are the most affected by this, as expected, with a performance loss of around 70% from an all local cluster with similar hardware. In general, measurements match the increased latency.
Are there any guidelines about what to tune or adjust to get the most of this kind of setups with "high" latencies?
Thanks in advance!
Victor
_______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]