Hi, we are currently testing out ways to increase Ceph performance because what we experience so far is very close to unusable. For the test cluster we utilizing 4 nodes with the following hardware data: Dual 200GBe Mellanox Ethernet 2x EPYC Rome 7302 16x 32GB 3200MHz ECC 9x 15.36TB Micron 9300 Pro For production this will be extended to all 8 nodes if it shows promising results. - Ceph was installed with cephadm. - MDS and ODS are located on the same nodes. - Mostly using stock config - Network performance tested with iperf3 seems fine, 26Gbits/s with -P4 on single port (details below). Close to 200Gbits with 10 parallel instances and servers. When testing a mounted CephFS on the working nodes in various configurations I only got <50MB/s for fuse mount and <270MB/s for kernel mounts. (dd command and output attached below) In addition ceph dashboard and our graphana monitoring reports packet loss on all relevant interfaces during load. Which does not occur during the normal iperf load tests or rsync/scp file transfer. Rados Bench shows performance around 2000MB/s which is not max performance of the SSDs but fine for us (details below). Why is the filesystem so slow compared to the individual components? Cheers Dominik Test details: ------------------------------------------------------------------------------------------------------ Some tests done on working nodes: Ceph mounted with ceph-fuse root@ml2ran10:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4,3 GB, 4,0 GiB) copied, 88,2933 s, 48,6 MB/s Ceph mounted with kernel driver: root@ml2ran06:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.0989 s, 267 MB/s Storage Node With fuse root@ml2rsn05:/mnt/ml2r_storage/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 53.9977 s, 79.5 MB/s Kernel mount: dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.6726 s, 243 MB/s _______________________________________________________ Iperf3 iperf3 --zerocopy -n 10240M -P4 -c ml2ran08s0 -p 4701 -i 15 -b 200000000000 Connecting to host ml2ran08s0, port 4701 [ 5] local 129.217.31.180 port 43958 connected to 129.217.31.218 port 4701 [ 7] local 129.217.31.180 port 43960 connected to 129.217.31.218 port 4701 [ 9] local 129.217.31.180 port 43962 connected to 129.217.31.218 port 4701 [ 11] local 129.217.31.180 port 43964 connected to 129.217.31.218 port 4701 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 632 KBytes [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 522 KBytes [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 612 KBytes [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 430 KBytes [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 5] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 sender [ 7] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 9] 0.00-3.21 sec 2.49 GBytes 6.67 Gbits/sec receiver [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 11] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 sender [SUM] 0.00-3.21 sec 9.98 GBytes 26.7 Gbits/sec receiver _________________________________________________________________ Rados Bench on storage node: # rados bench -p testbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ml2rsn05_2829244 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 747 731 2923.84 2924 0.0178757 0.0216262 2 16 1506 1490 2979.71 3036 0.0308664 0.0213685 3 16 2267 2251 3000.99 3044 0.0259053 0.0212556 4 16 3058 3042 3041.62 3164 0.0227621 0.0209792 5 16 3850 3834 3066.8 3168 0.0130519 0.0208148 6 16 4625 4609 3072.26 3100 0.151371 0.0207904 7 16 5381 5365 3065.28 3024 0.0300368 0.0208345 8 16 6172 6156 3077.57 3164 0.0197728 0.0207714 9 16 6971 6955 3090.67 3196 0.0142751 0.0206786 10 14 7772 7758 3102.76 3212 0.0181034 0.020605 Total time run: 10.0179 Total writes made: 7772 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 3103.23 Stddev Bandwidth: 93.3676 Max bandwidth (MB/sec): 3212 Min bandwidth (MB/sec): 2924 Average IOPS: 775 Stddev IOPS: 23.3419 Max IOPS: 803 Min IOPS: 731 Average Latency(s): 0.020598 Stddev Latency(s): 0.00731743 Max latency(s): 0.151371 Min latency(s): 0.00966991 # rados bench -p testbench 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 15 657 642 2567.32 2568 0.011104 0.022631 2 15 1244 1229 2456.9 2348 0.0115248 0.019485 3 16 1499 1483 1976.64 1016 0.00722983 0.0177887 4 15 1922 1907 1906.16 1696 0.0142242 0.0327382 5 15 2593 2578 2061.6 2684 0.011758 0.0301774 6 16 3142 3126 2083.23 2192 0.00915926 0.027478 7 16 3276 3260 1862.23 536 0.00824714 0.0267449 8 16 3606 3590 1794.43 1320 0.0118938 0.0350541 9 16 4293 4277 1900.32 2748 0.0301886 0.0330604 10 14 5003 4989 1995.04 2848 0.0389717 0.0314977 Total time run: 10.0227 Total reads made: 5003 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1996.67 Average IOPS: 499 Stddev IOPS: 202.3 Max IOPS: 712 Min IOPS: 134 Average Latency(s): 0.0314843 Max latency(s): 3.04463 Min latency(s): 0.00551523 # rados bench -p testbench 10 rand hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 15 15 0 0 0 - 0 1 15 680 665 2657.61 2660 0.00919807 0.0224833 2 15 1273 1258 2514.26 2372 0.00839656 0.0247125 3 16 1863 1847 2461.4 2356 0.00994467 0.0236565 4 16 2064 2048 2047.14 804 0.00809139 0.0223506 5 16 2064 2048 1637.79 0 - 0.0223506 6 16 2477 2461 1640.12 826 0.0286315 0.0383254 7 16 3102 3086 1762.89 2500 0.0267464 0.0349189 8 16 3513 3497 1748 1644 0.00890952 0.032269 9 16 3617 3601 1600 416 0.00626917 0.0316019 10 15 4014 3999 1599.18 1592 0.0461076 0.0393606 Total time run: 10.0481 Total reads made: 4014 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1597.91 Average IOPS: 399 Stddev IOPS: 239.089 Max IOPS: 665 Min IOPS: 0 Average Latency(s): 0.0394035 Max latency(s): 3.00962 Min latency(s): 0.00449537