OK, where to start. I have been debugging intensively the last two days, but can't seem to wrap my head around the performance issues we see in one of our two hyperconverged (ceph) proxmox clusters. Let me introduce our two clusters and some of the debugging results. *1. Cluster for internal purposes (performs as expected)* 3 x Supermicro servers with identical specs: CPU: 1 x 7-7700K CPU @ 4.20GHz (1 Socket) 4 cores / 8 threads RAM: 64 GB RAM OSDs: 4 per node. 1 per SSD (Intel S4610) (12 OSDs in all) 1 x 10GbE RJ45 nic. MTU 9000 No bonding _A total of 3 servers with a total of 12 OSDs_ Network: 1 x Unifi Switch 16 XG *2. Cluster for VPS's for customers (performs much worse than internal)* 3 x Dell R630 with the following specs: CPU: 2 x E5-2697 v3 @ 2.60GHz (2 Sockets) 28 cores / 56 threads RAM 256GB OSDs: 10 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 2 x Supermicro X11SRM-VF with the following specs: CPU 1 x 1 W-2145 CPU @ 3.70GHz (1 Socket) 8 cores / 16 threads RAM: 256 GB OSDs 8 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 1 x Dell R630 with the following specs: CPU 2 x CPU E5-2696 v4 @ 2.20GHz (2 Sockets) 44 cores / 88 threads RAM: 256 GB OSDs 8 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 _A total of 6 servers with a total of 54 OSDs_ Network: 2 x Dell N4032F 10GbE SFP+ Switch connected with MLAG. Each node is connected to each switch. To get a fair comparison i made the following fio tests one one host in each cluster on a rbd block device that i created: *1. cluster:* |fio --randrepeat=1 --ioengine=libaio --sync=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process test: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=1): [w(1)][100.0%][w=25.0MiB/s][w=6409 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=869177: Wed Nov 13 11:15:59 2019 write: IOPS=4158, BW=16.2MiB/s (17.0MB/s)(4096MiB/252126msec); 0 zone resets bw ( KiB/s): min= 2075, max=32968, per=99.96%, avg=16627.60, stdev=9635.42, samples=504 iops : min= 518, max= 8242, avg=4156.88, stdev=2408.86, samples=504 cpu : usr=0.53%, sys=3.81%, ctx=109599, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=16.2MiB/s (17.0MB/s), 16.2MiB/s-16.2MiB/s (17.0MB/s-17.0MB/s), io=4096MiB (4295MB), run=252126-252126msec Disk stats (read/write): rbd0: ios=46/1221898, merge=0/1870438, ticks=25/4654920, in_queue=1980016, util=84.70%| *2. cluster* |fio --randrepeat=1 --ioengine=libaio --sync=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process Jobs: 1 (f=1): [w(1)][99.9%][w=7024KiB/s][w=1756 IOPS][eta 00m:01s] test: (groupid=0, jobs=1): err= 0: pid=794096: Wed Nov 13 11:25:56 2019 write: IOPS=1353, BW=5415KiB/s (5545kB/s)(4096MiB/774601msec); 0 zone resets bw ( KiB/s): min= 40, max=30600, per=100.00%, avg=5420.24, stdev=3710.17, samples=1547 iops : min= 10, max= 7650, avg=1355.06, stdev=927.54, samples=1547 cpu : usr=0.16%, sys=1.19%, ctx=100028, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=5415KiB/s (5545kB/s), 5415KiB/s-5415KiB/s (5545kB/s-5545kB/s), io=4096MiB (4295MB), run=774601-774601msec Disk stats (read/write): rbd0: ios=0/1222639, merge=0/1784089, ticks=0/12124812, in_queue=9514280, util=45.14%| And identical rados bench tests: 1. cluster:https://i.imgur.com/AdARCA6.png 2. clusterhttps://i.imgur.com/Di7mYQh.png I have fio tested all disks. I have tested the network. I can't seem to find the reason why performance on my 2. cluster is relatively poor compared to 1. cluster. -- Dennis Højgaard Powerhosting Support *t:* +45 7222 4457 |*e:* dh@powerhosting.dk |*w:* https://powerhosting.dk