Hi Vitali, Sorry for a long response, I was on vacation. On 2020-07-20 00:44, vitalif@yourcmc.ru wrote:
Hi Roman,
It's always really interesting to read your messages :) maybe you'll join our telegram chat @ceph_ru? One of your colleagues is there :)
Promise a lot of fun? ;)
Client-based replication is of course the fastest,
With this comparison I answer the question: if it is fastest, then how much. Because obvious "of course is the fastest", well, not so reasoned :)
but the problem is that it's unclear how to provide consistency with it.
Here I tried to highlight client-based replication problems: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/N46NR7NBHWBQL4B2ASU... And yes, comes with restrictions, but there are scenarios which do not require strong sequential consistency of log-based replication, e.g. if you run an 1 rbd client per 1 image and a filesystem on top with journaling and strong requests order why not to rely on the filesystem recovery mechanisms? That is the question which bothers me for quite a while and that is exactly the reason why I started pech osd: to find some answers.
Maybe it's possible with some restrictions, but... I don't think it's possible in Ceph :)
No, for sure not. Not RADOS strong consistency semantics. But why? Take what you need and cut what is useless to fit your requirements.
By the way, have you tested their Crimson OSD?
Yes, I did. But with Crimson OSD everything went wrong: the first problem I came across is that I was not able to reach desired number of 120 OSDs running: the average number after various restarts I got from monitor is ~50. I did not try to debug and simply reduced number of OSDs to 35 (5 hosts, 7 ODSs on each) and reran all tests, so here is the results for all types of osds (for fair reference): 35OSDS/crimson/primary-copy write/iops write/bw write/clat_ns/mean 4k 3.50 K 14.30 MB/s 888.00 ms 8k 4.97 K 40.37 MB/s 687.27 ms 16k 4.90 K 79.63 MB/s 709.63 ms 32k 4.48 K 145.50 MB/s 703.80 ms 64k 4.46 K 290.72 MB/s 731.06 ms 128k 4.38 K 570.44 MB/s 720.32 ms 256k 4.13 K 1.05 GB/s 755.77 ms 512k 2.56 K 1.32 GB/s 1.15 s 1m 1.16 K 1.24 GB/s 2.22 s 35OSDS/ceph/primary-copy write/iops write/bw write/clat_ns/mean 4k 90.74 K 355.92 MB/s 44.51 ms 8k 75.03 K 588.98 MB/s 53.45 ms 16k 92.58 K 1.42 GB/s 42.10 ms 32k 122.95 K 3.76 GB/s 33.27 ms 64k 101.28 K 6.20 GB/s 39.45 ms 128k 45.28 K 5.57 GB/s 83.26 ms 256k 26.08 K 6.44 GB/s 138.16 ms 512k 14.49 K 7.23 GB/s 254.74 ms 1m 5.91 K 6.05 GB/s 588.28 ms 35OSDS/pech/primary-copy write/iops write/bw write/clat_ns/mean 4k 289.22 K 1.10 GB/s 14.94 ms 8k 231.93 K 1.77 GB/s 15.94 ms 16k 228.60 K 3.49 GB/s 17.28 ms 32k 208.95 K 6.39 GB/s 19.08 ms 64k 106.66 K 6.53 GB/s 37.69 ms 128k 53.48 K 6.57 GB/s 73.03 ms 256k 25.03 K 6.19 GB/s 139.59 ms 512k 12.63 K 6.32 GB/s 302.50 ms 1m 5.91 K 6.05 GB/s 650.03 ms I did not notice anything strange in crimsons logs and did try to debug, so do not know why the results are so bad for the crimson case.
Is it any faster than current implementation? (regarding iodepth=1 fsync=1 latency)
My original goal was to test real distributed load: many osd hosts, many clients hosts (I was keen to see how Pech was behaving). Your "latency" load does not require a cluster setup and can be executed on a localhost with 3 osds (x3 replication), so here are the results: "-o ms_crc_data=false -o debug_osd=0 -o debug_ms=0" rbd.fio rw=randwrite iodepth=1 numjobs=1 runtume=10 size=256m /// crimson-osd 4k IOPS=101, BW=406KiB/s, Lat=9846.09usec 8k IOPS=100, BW=802KiB/s, Lat=9973.39usec 16k IOPS=99, BW=1599KiB/s, Lat=10000.17usec 32k IOPS=96, BW=3088KiB/s, Lat=10355.56usec 64k IOPS=591, BW=36.0MiB/s, Lat=1687.63usec 128k IOPS=508, BW=63.6MiB/s, Lat=1963.95usec 256k IOPS=379, BW=94.9MiB/s, Lat=2632.29usec 512k IOPS=338, BW=169MiB/s, Lat=2952.39usec 1m IOPS=166, BW=166MiB/s, Lat=6011.29usec /// ceph-osd 4k IOPS=1908, BW=7634KiB/s, Lat=522.07usec 8k IOPS=1838, BW=14.4MiB/s, Lat=542.10usec 16k IOPS=1751, BW=27.4MiB/s, Lat=568.98usec 32k IOPS=2048, BW=64.0MiB/s, Lat=486.48usec 64k IOPS=1985, BW=124MiB/s, Lat=501.80usec 128k IOPS=1869, BW=234MiB/s, Lat=532.96usec 256k IOPS=1645, BW=411MiB/s, Lat=605.66usec 512k IOPS=1195, BW=598MiB/s, Lat=833.64usec 1m IOPS=704, BW=705MiB/s, Lat=1414.01usec /// pech-osd OSD=X; CEPH=~/devel/ceph-upstream; ./pech-osd --mon_addrs 192.168.0.97:50001 --server_ip 0.0.0.0 --name $OSD --fsid `cat $CEPH/build/dev/osd$OSD/fsid` --class_dir $CEPH/build/lib --log_level 5 --replication primary-copy --nocrc 4k IOPS=5618, BW=21.9MiB/s, Lat=176.48usec 8k IOPS=5654, BW=44.2MiB/s, Lat=175.26usec 16k IOPS=5504, BW=86.0MiB/s, Lat=180.20usec 32k IOPS=4976, BW=156MiB/s, Lat=199.37usec 64k IOPS=4334, BW=271MiB/s, Lat=229.09usec 128k IOPS=3397, BW=425MiB/s, Lat=292.52usec 256k IOPS=2392, BW=598MiB/s, Lat=416.12usec 512k IOPS=1505, BW=753MiB/s, Lat=661.25usec 1m IOPS=687, BW=688MiB/s, Lat=1446.60usec Results should be treated carefully, since practically these numbers are almost certainly unreachable, but here you are right: numbers give a clear upper bound. -- Roman