Re: ceph-osd performance on ram disk
Thank you! I know that article, but they promise 6 core use per OSD, and I got barely over three, and all this in totally synthetic environment with no SDD to blame (brd is more than fast and have a very consistent latency under any kind of load). On Thu, Sep 10, 2020, 19:39 Marc Roos <M.Roos@f1-outsourcing.eu> wrote:
Hi George,
Very interesting and also a bit expecting result. Some messages posted here are already indicating that getting expensive top of the line hardware does not really result in any performance increase above some level. Vitaliy has documented something similar[1]
[1] https://yourcmc.ru/wiki/Ceph_performance
-----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] ceph-osd performance on ram disk
I'm creating a benchmark suite for Сeph.
During benchmarking of benchmark, I've checked how fast ceph-osd works. I decided to skip all 'SSD mess' and use brd (block ram disk, modprobe brd) as underlying storage. Brd itself can yield up to 2.7Mpps in fio. In single thread mode (iodepth=1) it can yield up to 750k IOPS. LVM over brd gives about 600kIOPS in single-threaded mode with iodepth=1 (16us latency).
But, as soon as I put ceph-osd (bluestore) on it, I see something very odd. No matter how much parallel load I push onto this OSD, it never gives more than 30 kIOPS, and I can't understand where bottleneck is.
CPU utilization: ~300%. There are 8 cores on my setup, so, CPU is not a bottleneck.
Network: I've moved benchmark on the same host as OSD, so it's a localhost. Even counting network, it's still far away from saturation. 30kIOPS (4k) is about 1Gb/s, but I have 10G links. Anyway, tests are run on localhost, so network is irrelevant (I've checked it, traffic is on localhost). Test itself consumes about 70% CPU of one core, so there are plenty left.
Replication: I've killed it (size=1, single osd in the pool).
single-threaded latency: 200us, 4.8kIOPS. iopdeth=32: 2ms (15kIOPS). iodepth=16,numjobs=8: 5ms (24k IOPS)
I'm running fio with 'rados' ioengine, and it looks like putting more workers doesn't change much, so it's not rados ioengine.
As there is plenty CPU and IO left, there is only one possible place for bottleneck: some time-consuming single-threaded code in ceph-osd.
Are there any knobs to tweak to see higher performance for ceph-osd? I'm pretty sure it's not any kind of leveling, GC or other 'iops-related' issues (brd has performance of two order of magnitude higher).
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi George Author of Ceph_performance here! :) I suspect you're running tests with 1 PG. Every PG's requests are always serialized, that's why OSD doesn't utilize all threads with 1 PG. You need something like 8 PGs per OSD. More than 8 usually doesn't improve results. Also note that read tests are meaningless after full overwrite on small OSDs because everything fits in cache. Restart the OSD to clear it. You can drop the cache via the admin socket too, but restarting is the simplest way. I've repeated your test with brd. My results with 8 PGs after filling the RBD image, turning CPU powersave off and restarting the OSD are: # fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randread -pool=ramdisk -rbdname=testimg read: IOPS=3586, BW=14.0MiB/s (14.7MB/s)(411MiB/29315msec) lat (usec): min=182, max=5710, avg=277.41, stdev=90.16 # fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randwrite -pool=ramdisk -rbdname=testimg write: IOPS=1247, BW=4991KiB/s (5111kB/s)(67.0MiB/13746msec); 0 zone resets lat (usec): min=555, max=4015, avg=799.45, stdev=142.92 # fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randwrite -pool=ramdisk -rbdname=testimg write: IOPS=4138, BW=16.2MiB/s (16.9MB/s)(282MiB/17451msec); 0 zone resets 658% CPU # fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randread -pool=ramdisk -rbdname=testimg read: IOPS=15.7k, BW=61.4MiB/s (64.4MB/s)(979MiB/15933msec) 540% CPU Basically the same shit as on an NVMe. So even an "in-memory Ceph" is slow, haha.
Thank you!
I know that article, but they promise 6 core use per OSD, and I got barely over three, and all this in totally synthetic environment with no SDD to blame (brd is more than fast and have a very consistent latency under any kind of load).
By the way, DON'T USE rados bench. It's an incorrect benchmark. ONLY use fio 10 сентября 2020 г. 22:35:53 GMT+03:00, vitalif@yourcmc.ru пишет:
Hi George
Author of Ceph_performance here! :)
I suspect you're running tests with 1 PG. Every PG's requests are always serialized, that's why OSD doesn't utilize all threads with 1 PG. You need something like 8 PGs per OSD. More than 8 usually doesn't improve results.
Also note that read tests are meaningless after full overwrite on small OSDs because everything fits in cache. Restart the OSD to clear it. You can drop the cache via the admin socket too, but restarting is the simplest way.
I've repeated your test with brd. My results with 8 PGs after filling the RBD image, turning CPU powersave off and restarting the OSD are:
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randread -pool=ramdisk -rbdname=testimg read: IOPS=3586, BW=14.0MiB/s (14.7MB/s)(411MiB/29315msec) lat (usec): min=182, max=5710, avg=277.41, stdev=90.16
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randwrite -pool=ramdisk -rbdname=testimg write: IOPS=1247, BW=4991KiB/s (5111kB/s)(67.0MiB/13746msec); 0 zone resets lat (usec): min=555, max=4015, avg=799.45, stdev=142.92
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randwrite -pool=ramdisk -rbdname=testimg write: IOPS=4138, BW=16.2MiB/s (16.9MB/s)(282MiB/17451msec); 0 zone resets 658% CPU
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randread -pool=ramdisk -rbdname=testimg read: IOPS=15.7k, BW=61.4MiB/s (64.4MB/s)(979MiB/15933msec) 540% CPU
Basically the same shit as on an NVMe. So even an "in-memory Ceph" is slow, haha.
Thank you!
I know that article, but they promise 6 core use per OSD, and I got barely over three, and all this in totally synthetic environment with no SDD to blame (brd is more than fast and have a very consistent latency under any kind of load).
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- With best regards, Vitaliy Filippov
I know. I tested fio before testing ceph with fio. On null ioengine fio can handle up to 14M IOPS (on my dusty lab's R220). On blk_null to gets down to 2.4-2.8M IOPS. On brd it drops to sad 700k IOPS. BTW, never run synthetic high-performance benchmarks on kvm. My old server with 'makelinuxfastagain' fixes make one io request in 3.4us, and on KVM VM it become 24us. Some guy said it got about 8.5us on vmware. It's all on purely software stack without any hypervisor IO. 24us sounds like a small number, but if your synthetics makes 200k iops, it's 4us. You can't make 200k on VM with 24us syscall time. On Thu, Sep 10, 2020, 22:49 Виталий Филиппов <vitalif@yourcmc.ru> wrote:
By the way, DON'T USE rados bench. It's an incorrect benchmark. ONLY use fio
10 сентября 2020 г. 22:35:53 GMT+03:00, vitalif@yourcmc.ru пишет:
Hi George
Author of Ceph_performance here! :)
I suspect you're running tests with 1 PG. Every PG's requests are always serialized, that's why OSD doesn't utilize all threads with 1 PG. You need something like 8 PGs per OSD. More than 8 usually doesn't improve results.
Also note that read tests are meaningless after full overwrite on small OSDs because everything fits in cache. Restart the OSD to clear it. You can drop the cache via the admin socket too, but restarting is the simplest way.
I've repeated your test with brd. My results with 8 PGs after filling the RBD image, turning CPU powersave off and restarting the OSD are:
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randread -pool=ramdisk -rbdname=testimg read: IOPS=3586, BW=14.0MiB/s (14.7MB/s)(411MiB/29315msec) lat (usec): min=182, max=5710, avg=277.41, stdev=90.16
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randwrite -pool=ramdisk -rbdname=testimg write: IOPS=1247, BW=4991KiB/s (5111kB/s)(67.0MiB/13746msec); 0 zone resets lat (usec): min=555, max=4015, avg=799.45, stdev=142.92
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randwrite -pool=ramdisk -rbdname=testimg write: IOPS=4138, BW=16.2MiB/s (16.9MB/s)(282MiB/17451msec); 0 zone resets 658% CPU
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randread -pool=ramdisk -rbdname=testimg read: IOPS=15.7k, BW=61.4MiB/s (64.4MB/s)(979MiB/15933msec) 540% CPU
Basically the same shit as on an NVMe. So even an "in-memory Ceph" is slow, haha.
Thank you!
I know that article, but they promise 6 core use per OSD, and I got barely over three, and all this in totally synthetic environment with no SDD to blame (brd is more than fast and have a very consistent latency under any kind of load).
------------------------------ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- With best regards, Vitaliy Filippov
Yeah, of course... but RBD is primarily used for KVM VMs, so the results from a VM are the thing that real clients see. So they do mean something... :) I know. I tested fio before testing cephwith fio. On null ioengine fio can handle up to 14M IOPS (on my dusty lab's R220). On blk_null to gets down to 2.4-2.8M IOPS. On brd it drops to sad 700k IOPS. BTW, never run synthetic high-performance benchmarks on kvm. My old server with 'makelinuxfastagain' fixes make one io request in 3.4us, and on KVM VM it become 24us. Some guy said it got about 8.5us on vmware. It's all on purely software stack without any hypervisor IO. 24us sounds like a small number, but if your synthetics makes 200k iops, it's 4us. You can't make 200k on VM with 24us syscall time.
Latency from a client side is not an issue. It just combines with other latencies in the stack. The more client lags, the easier it's for the cluster. Here, the thing I talk, is slightly different. When you want to establish baseline performance for osd daemon (disregarding block device and network latencies), sudden order-of-magnitude delay on syscalls cause disproportionate skew to results. It does not relate to the production in any way, only to ceph-osd benchmarks. On Thu, Sep 10, 2020, 23:21 <vitalif@yourcmc.ru> wrote:
Yeah, of course... but RBD is primarily used for KVM VMs, so the results from a VM are the thing that real clients see. So they do mean something... :)
I know. I tested fio before testing ceph with fio. On null ioengine fio can handle up to 14M IOPS (on my dusty lab's R220). On blk_null to gets down to 2.4-2.8M IOPS. On brd it drops to sad 700k IOPS. BTW, never run synthetic high-performance benchmarks on kvm. My old server with 'makelinuxfastagain' fixes make one io request in 3.4us, and on KVM VM it become 24us. Some guy said it got about 8.5us on vmware. It's all on purely software stack without any hypervisor IO. 24us sounds like a small number, but if your synthetics makes 200k iops, it's 4us. You can't make 200k on VM with 24us syscall time.
On 10/09/2020 22:35, vitalif@yourcmc.ru wrote:
Hi George
Author of Ceph_performance here! :)
I suspect you're running tests with 1 PG. Every PG's requests are always serialized, that's why OSD doesn't utilize all threads with 1 PG. You need something like 8 PGs per OSD. More than 8 usually doesn't improve results.
Also note that read tests are meaningless after full overwrite on small OSDs because everything fits in cache. Restart the OSD to clear it. You can drop the cache via the admin socket too, but restarting is the simplest way.
I've repeated your test with brd. My results with 8 PGs after filling the RBD image, turning CPU powersave off and restarting the OSD are:
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randread -pool=ramdisk -rbdname=testimg read: IOPS=3586, BW=14.0MiB/s (14.7MB/s)(411MiB/29315msec) lat (usec): min=182, max=5710, avg=277.41, stdev=90.16
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randwrite -pool=ramdisk -rbdname=testimg write: IOPS=1247, BW=4991KiB/s (5111kB/s)(67.0MiB/13746msec); 0 zone resets lat (usec): min=555, max=4015, avg=799.45, stdev=142.92
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randwrite -pool=ramdisk -rbdname=testimg write: IOPS=4138, BW=16.2MiB/s (16.9MB/s)(282MiB/17451msec); 0 zone resets 658% CPU
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randread -pool=ramdisk -rbdname=testimg read: IOPS=15.7k, BW=61.4MiB/s (64.4MB/s)(979MiB/15933msec) 540% CPU
Basically the same shit as on an NVMe. So even an "in-memory Ceph" is slow, haha.
Hello! Thank you for feedback! PG idea is really good. Unfortunately, autoscale made it to 32, and I have 30 kIOPS of 32-pg 1-size pool on ramdisk. :-/ I've checked read speed (I hadn't done this before, I have no idea why), and I got amazing 160kIOPS, but I suspect it's caching. Anyway, thank you for data, I assume 600% CPU in exchange for ~16-17kIOPS for OSD.
participants (3)
-
George Shuklin
-
vitalif＠yourcmc.ru
-
Виталий Филиппов