10 Sep
2020
10 Sep
'20
5:22 p.m.
Thank you!
I know that article, but they promise 6 core use per OSD, and I got barely
over three, and all this in totally synthetic environment with no SDD to
blame (brd is more than fast and have a very consistent latency under any
kind of load).
On Thu, Sep 10, 2020, 19:39 Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote:
Hi George,
Very interesting and also a bit expecting result. Some messages posted
here are already indicating that getting expensive top of the line
hardware does not really result in any performance increase above some
level. Vitaliy has documented something similar[1]
[1]
https://yourcmc.ru/wiki/Ceph_performance
-----Original Message-----
To: ceph-users(a)ceph.io
Subject: [ceph-users] ceph-osd performance on ram disk
I'm creating a benchmark suite for Сeph.
During benchmarking of benchmark, I've checked how fast ceph-osd works.
I decided to skip all 'SSD mess' and use brd (block ram disk, modprobe
brd) as underlying storage. Brd itself can yield up to 2.7Mpps in fio.
In single thread mode (iodepth=1) it can yield up to 750k IOPS. LVM over
brd gives about 600kIOPS in single-threaded mode with iodepth=1 (16us
latency).
But, as soon as I put ceph-osd (bluestore) on it, I see something very
odd. No matter how much parallel load I push onto this OSD, it never
gives more than 30 kIOPS, and I can't understand where bottleneck is.
CPU utilization: ~300%. There are 8 cores on my setup, so, CPU is not a
bottleneck.
Network: I've moved benchmark on the same host as OSD, so it's a
localhost. Even counting network, it's still far away from saturation.
30kIOPS (4k) is about 1Gb/s, but I have 10G links. Anyway, tests are run
on localhost, so network is irrelevant (I've checked it, traffic is on
localhost). Test itself consumes about 70% CPU of one core, so there are
plenty left.
Replication: I've killed it (size=1, single osd in the pool).
single-threaded latency: 200us, 4.8kIOPS.
iopdeth=32: 2ms (15kIOPS).
iodepth=16,numjobs=8: 5ms (24k IOPS)
I'm running fio with 'rados' ioengine, and it looks like putting more
workers doesn't change much, so it's not rados ioengine.
As there is plenty CPU and IO left, there is only one possible place for
bottleneck: some time-consuming single-threaded code in ceph-osd.
Are there any knobs to tweak to see higher performance for ceph-osd? I'm
pretty sure it's not any kind of leveling, GC or other 'iops-related'
issues (brd has performance of two order of magnitude higher).
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10 Sep
10 Sep
7:35 p.m.
New subject: ceph-osd performance on ram disk
Hi George
Author of Ceph_performance here! :)
I suspect you're running tests with 1 PG. Every PG's requests are always
serialized, that's why OSD doesn't utilize all threads with 1 PG. You need
something like 8 PGs per OSD. More than 8 usually doesn't improve results.
Also note that read tests are meaningless after full overwrite on small OSDs because
everything fits in cache. Restart the OSD to clear it. You can drop the cache via the
admin socket too, but restarting is the simplest way.
I've repeated your test with brd. My results with 8 PGs after filling the RBD image,
turning CPU powersave off and restarting the OSD are:
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randread -pool=ramdisk
-rbdname=testimg
read: IOPS=3586, BW=14.0MiB/s (14.7MB/s)(411MiB/29315msec)
lat (usec): min=182, max=5710, avg=277.41, stdev=90.16
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randwrite -pool=ramdisk
-rbdname=testimg
write: IOPS=1247, BW=4991KiB/s (5111kB/s)(67.0MiB/13746msec); 0 zone resets
lat (usec): min=555, max=4015, avg=799.45, stdev=142.92
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randwrite -pool=ramdisk
-rbdname=testimg
write: IOPS=4138, BW=16.2MiB/s (16.9MB/s)(282MiB/17451msec); 0 zone resets
658% CPU
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randread -pool=ramdisk
-rbdname=testimg
read: IOPS=15.7k, BW=61.4MiB/s (64.4MB/s)(979MiB/15933msec)
540% CPU
Basically the same shit as on an NVMe. So even an "in-memory Ceph" is slow,
haha.
> Thank you!
>
> I know that article, but they promise 6 core use per OSD, and I got barely
> over three, and all this in totally synthetic environment with no SDD to
> blame (brd is more than fast and have a very consistent latency under any
> kind of load).
7:49 p.m.
New subject: ceph-osd performance on ram disk
By the way, DON'T USE rados bench. It's an incorrect benchmark. ONLY use fio
10 сентября 2020 г. 22:35:53 GMT+03:00, vitalif(a)yourcmc.ru пишет:
Hi George
Author of Ceph_performance here! :)
I suspect you're running tests with 1 PG. Every PG's requests are
always serialized, that's why OSD doesn't utilize all threads with 1
PG. You need something like 8 PGs per OSD. More than 8 usually doesn't
improve results.
Also note that read tests are meaningless after full overwrite on small
OSDs because everything fits in cache. Restart the OSD to clear it. You
can drop the cache via the admin socket too, but restarting is the
simplest way.
I've repeated your test with brd. My results with 8 PGs after filling
the RBD image, turning CPU powersave off and restarting the OSD are:
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randread
-pool=ramdisk -rbdname=testimg
read: IOPS=3586, BW=14.0MiB/s (14.7MB/s)(411MiB/29315msec)
lat (usec): min=182, max=5710, avg=277.41, stdev=90.16
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randwrite
-pool=ramdisk -rbdname=testimg
write: IOPS=1247, BW=4991KiB/s (5111kB/s)(67.0MiB/13746msec); 0 zone
resets
lat (usec): min=555, max=4015, avg=799.45, stdev=142.92
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randwrite
-pool=ramdisk -rbdname=testimg
write: IOPS=4138, BW=16.2MiB/s (16.9MB/s)(282MiB/17451msec); 0 zone
resets
658% CPU
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randread
-pool=ramdisk -rbdname=testimg
read: IOPS=15.7k, BW=61.4MiB/s (64.4MB/s)(979MiB/15933msec)
540% CPU
Basically the same shit as on an NVMe. So even an "in-memory Ceph" is
slow, haha.
--
With best regards,
Vitaliy Filippov
Thank you!
I know that article, but they promise 6 core use per OSD, and I got
barely
over three, and all this in totally synthetic
environment with no SDD
to
blame (brd is more than fast and have a very
consistent latency under
any
kind of load).
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:06 p.m.
New subject: ceph-osd performance on ram disk
I know. I tested fio before testing ceph
with fio. On null ioengine fio can handle up to 14M IOPS (on my dusty
lab's R220). On blk_null to gets down to 2.4-2.8M IOPS.
On brd it drops to sad 700k IOPS.
BTW, never run synthetic high-performance benchmarks on kvm. My old server
with 'makelinuxfastagain' fixes make one io request in 3.4us, and on KVM VM
it become 24us. Some guy said it got about 8.5us on vmware. It's all on
purely software stack without any hypervisor IO.
24us sounds like a small number, but if your synthetics makes 200k iops,
it's 4us. You can't make 200k on VM with 24us syscall time.
On Thu, Sep 10, 2020, 22:49 Виталий Филиппов <vitalif(a)yourcmc.ru> wrote:
By the way, DON'T USE rados bench. It's an
incorrect benchmark. ONLY use
fio
10 сентября 2020 г. 22:35:53 GMT+03:00, vitalif(a)yourcmc.ru пишет:
Hi George
Author of Ceph_performance here! :)
I suspect you're running tests with 1 PG. Every PG's requests are always
serialized, that's why OSD doesn't utilize all threads with 1 PG. You need
something like 8 PGs per OSD. More than 8 usually doesn't improve results.
Also note that read tests are meaningless after full overwrite on small OSDs because
everything fits in cache. Restart the OSD to clear it. You can drop the cache via the
admin socket too, but restarting is the simplest way.
I've repeated your test with brd. My results with 8 PGs after filling the RBD image,
turning CPU powersave off and restarting the OSD are:
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randread -pool=ramdisk
-rbdname=testimg
read: IOPS=3586, BW=14.0MiB/s (14.7MB/s)(411MiB/29315msec)
lat (usec): min=182, max=5710, avg=277.41, stdev=90.16
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randwrite -pool=ramdisk
-rbdname=testimg
write: IOPS=1247, BW=4991KiB/s (5111kB/s)(67.0MiB/13746msec); 0 zone resets
lat (usec): min=555, max=4015, avg=799.45, stdev=142.92
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randwrite -pool=ramdisk
-rbdname=testimg
write: IOPS=4138, BW=16.2MiB/s (16.9MB/s)(282MiB/17451msec); 0 zone resets
658% CPU
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randread -pool=ramdisk
-rbdname=testimg
read: IOPS=15.7k, BW=61.4MiB/s (64.4MB/s)(979MiB/15933msec)
540% CPU
Basically the same shit as on an NVMe. So even an "in-memory Ceph" is slow,
haha.
Thank you!
--
With best regards,
Vitaliy Filippov
I know that article, but they promise 6 core use per OSD, and I got barely
over three, and all this in totally synthetic environment with no SDD to
blame (brd is more than fast and have a very consistent latency under any
kind of load).
------------------------------
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:21 p.m.
New subject: ceph-osd performance on ram disk
Yeah, of course... but RBD is primarily used for KVM VMs, so the results from a VM are the
thing that real clients see. So they do mean something... :)
I know. I tested fio before testing cephwith fio. On null ioengine fio can handle up to
14M IOPS (on my dusty lab's R220). On blk_null to gets down to 2.4-2.8M IOPS. On brd
it drops to sad 700k IOPS. BTW, never run synthetic high-performance benchmarks on kvm.
My old server with 'makelinuxfastagain' fixes make one io request in 3.4us, and on
KVM VM it become 24us. Some guy said it got about 8.5us on vmware. It's all on purely
software stack without any hypervisor IO.
24us sounds like a small number, but if your synthetics makes 200k iops, it's 4us. You
can't make 200k on VM with 24us syscall time.
8:26 p.m.
New subject: ceph-osd performance on ram disk
Latency from a client side is not an issue. It just combines with other
latencies in the stack. The more client lags, the easier it's for the
cluster.
Here, the thing I talk, is slightly different. When you want to establish
baseline performance for osd daemon (disregarding block device and network
latencies), sudden order-of-magnitude delay on syscalls cause
disproportionate skew to results.
It does not relate to the production in any way, only to ceph-osd
benchmarks.
On Thu, Sep 10, 2020, 23:21 <vitalif(a)yourcmc.ru> wrote:
Yeah, of course... but RBD is primarily used for KVM
VMs, so the results
from a VM are the thing that real clients see. So they do mean something...
:)
I know. I tested fio before testing ceph
with fio. On null ioengine fio can handle up to 14M IOPS (on my dusty
lab's R220). On blk_null to gets down to 2.4-2.8M IOPS.
On brd it drops to sad 700k IOPS.
BTW, never run synthetic high-performance benchmarks on kvm. My old server
with 'makelinuxfastagain' fixes make one io request in 3.4us, and on KVM VM
it become 24us. Some guy said it got about 8.5us on vmware. It's all on
purely software stack without any hypervisor IO.
24us sounds like a small number, but if your synthetics makes 200k iops,
it's 4us. You can't make 200k on VM with 24us syscall time.
11 Sep
11 Sep
6:49 a.m.
New subject: ceph-osd performance on ram disk
On 10/09/2020 22:35, vitalif(a)yourcmc.ru wrote:
Hi George
Author of Ceph_performance here! :)
I suspect you're running tests with 1 PG. Every PG's requests are always
serialized, that's why OSD doesn't utilize all threads with 1 PG. You need
something like 8 PGs per OSD. More than 8 usually doesn't improve results.
Also note that read tests are meaningless after full overwrite on small OSDs because
everything fits in cache. Restart the OSD to clear it. You can drop the cache via the
admin socket too, but restarting is the simplest way.
I've repeated your test with brd. My results with 8 PGs after filling the RBD image,
turning CPU powersave off and restarting the OSD are:
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randread -pool=ramdisk
-rbdname=testimg
read: IOPS=3586, BW=14.0MiB/s (14.7MB/s)(411MiB/29315msec)
lat (usec): min=182, max=5710, avg=277.41, stdev=90.16
# fio -name=test -ioengine=rbd -bs=4k -iodepth=1 -rw=randwrite -pool=ramdisk
-rbdname=testimg
write: IOPS=1247, BW=4991KiB/s (5111kB/s)(67.0MiB/13746msec); 0 zone resets
lat (usec): min=555, max=4015, avg=799.45, stdev=142.92
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randwrite -pool=ramdisk
-rbdname=testimg
write: IOPS=4138, BW=16.2MiB/s (16.9MB/s)(282MiB/17451msec); 0 zone resets
658% CPU
# fio -name=test -ioengine=rbd -bs=4k -iodepth=128 -rw=randread -pool=ramdisk
-rbdname=testimg
read: IOPS=15.7k, BW=61.4MiB/s (64.4MB/s)(979MiB/15933msec)
540% CPU
Basically the same shit as on an NVMe. So even an "in-memory Ceph" is slow,
haha.
Hello!
Thank you for feedback! PG idea is really good. Unfortunately, autoscale
made it to 32, and I have 30 kIOPS of 32-pg 1-size pool on ramdisk. :-/
I've checked read speed (I hadn't done this before, I have no idea why),
and I got amazing 160kIOPS, but I suspect it's caching.
Anyway, thank you for data, I assume 600% CPU in exchange for
~16-17kIOPS for OSD.
1330
days inactive
1331
days old
6 comments
3 participants
participants (3)
-
George Shuklin -
vitalif＠yourcmc.ru -
Виталий Филиппов