RBD persistent writeback cache performance
Hi, With Pacific (16.2.0) out of the door we have the RBD persistent WriteBack cache for RBD: https://docs.ceph.com/en/latest/rbd/rbd-persistent-write-back-cache/ Has anybody performed some benchmarks with the RBD cache? Interested in the QD=1 bs=4k performance mainly. I don't have any proper hardware available to run benchmarks on yet. Wido
Hi, So I've been doing some tests with v16.2.4 with a 2TB Samsung PM983 SSD mounted under /mnt/rbd-cache rbd_persistent_cache_mode = ssd rbd_persistent_cache_size = 2G rbd_persistent_cache_path = /mnt/rbd-cache rbd_plugins = pwl_cache I tried both XFS and EXT4 as the filesystem. This however leads to fio or 'rbd bench' to crash: root@infra-138-b16-27:~# fio fio/rbd_rw_1.fio rbd_w_iodepth_1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.1 Starting 1 process Segmentation fault1)][13.3%][r=0KiB/s,w=14.7MiB/s][r=0,w=3768 IOPS][eta 00m:52s] root@infra-138-b16-27:~# (The IOps seem great!) My fio test is fairly simple: [global] ioengine=rbd clientname=admin pool=rbd rbdname=fio1 invalidate=0 bs=4k runtime=60 direct=1 [rbd_w_iodepth_1] rw=randwrite iodepth=1 I have tried to trace it with gdb, but I didn't get further with my backtrace then: (gdb) bt #0 ContextWQ::process (ctx=0x7fffb8081480, this=0x7fffb8012470) at ./src/common/WorkQueue.h:556 #1 ThreadPool::PointerWQ<Context>::_void_process (this=0x7fffb8012470, item=0x7fffb8081480, handle=...) at ./src/common/WorkQueue.h:341 #2 0x00007fffec600912 in ThreadPool::worker (this=0x7fffb8012018, wt=<optimized out>) at ./src/common/WorkQueue.cc:117 #3 0x00007fffec601801 in ThreadPool::WorkThread::entry (this=<optimized out>) at ./src/common/WorkQueue.h:395 #4 0x00007ffff5c796db in start_thread (arg=0x7fffb17fa700) at pthread_create.c:463 #5 0x00007ffff579e71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Has anybody been able to use pwl_cache successfully? Wido On 01/04/2021 21:01, Wido den Hollander wrote:
Hi,
With Pacific (16.2.0) out of the door we have the RBD persistent WriteBack cache for RBD: https://docs.ceph.com/en/latest/rbd/rbd-persistent-write-back-cache/
Has anybody performed some benchmarks with the RBD cache?
Interested in the QD=1 bs=4k performance mainly.
I don't have any proper hardware available to run benchmarks on yet.
Wido _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Tue, Jun 8, 2021 at 7:11 PM Wido den Hollander <wido@42on.com> wrote:
Hi,
So I've been doing some tests with v16.2.4 with a 2TB Samsung PM983 SSD mounted under /mnt/rbd-cache
rbd_persistent_cache_mode = ssd rbd_persistent_cache_size = 2G rbd_persistent_cache_path = /mnt/rbd-cache rbd_plugins = pwl_cache
I tried both XFS and EXT4 as the filesystem.
This however leads to fio or 'rbd bench' to crash:
root@infra-138-b16-27:~# fio fio/rbd_rw_1.fio rbd_w_iodepth_1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.1 Starting 1 process Segmentation fault1)][13.3%][r=0KiB/s,w=14.7MiB/s][r=0,w=3768 IOPS][eta 00m:52s] root@infra-138-b16-27:~#
(The IOps seem great!)
My fio test is fairly simple:
[global] ioengine=rbd clientname=admin pool=rbd rbdname=fio1 invalidate=0 bs=4k runtime=60 direct=1
[rbd_w_iodepth_1] rw=randwrite iodepth=1
I have tried to trace it with gdb, but I didn't get further with my backtrace then:
(gdb) bt #0 ContextWQ::process (ctx=0x7fffb8081480, this=0x7fffb8012470) at ./src/common/WorkQueue.h:556 #1 ThreadPool::PointerWQ<Context>::_void_process (this=0x7fffb8012470, item=0x7fffb8081480, handle=...) at ./src/common/WorkQueue.h:341 #2 0x00007fffec600912 in ThreadPool::worker (this=0x7fffb8012018, wt=<optimized out>) at ./src/common/WorkQueue.cc:117 #3 0x00007fffec601801 in ThreadPool::WorkThread::entry (this=<optimized out>) at ./src/common/WorkQueue.h:395 #4 0x00007ffff5c796db in start_thread (arg=0x7fffb17fa700) at pthread_create.c:463 #5 0x00007ffff579e71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Has anybody been able to use pwl_cache successfully?
Hi Wido, Unfortunately "rbd_persistent_cache_mode = ssd" cache has shipped rather broken. This particular crash is most likely already fixed in master, but there are a few more outstanding. There is a dozen of "[pwl ssd] ..." tickets in the rbd project, the fixes would be backported to pacific once the ssd mode is stable enough. Until then, I would to stick to "rbd_persistent_cache_mode = rwl" or avoid the pwl_cache plugin entirely. Thanks, Ilya
On 08/06/2021 21:04, Ilya Dryomov wrote:
On Tue, Jun 8, 2021 at 7:11 PM Wido den Hollander <wido@42on.com> wrote:
Hi,
So I've been doing some tests with v16.2.4 with a 2TB Samsung PM983 SSD mounted under /mnt/rbd-cache
rbd_persistent_cache_mode = ssd rbd_persistent_cache_size = 2G rbd_persistent_cache_path = /mnt/rbd-cache rbd_plugins = pwl_cache
I tried both XFS and EXT4 as the filesystem.
This however leads to fio or 'rbd bench' to crash:
root@infra-138-b16-27:~# fio fio/rbd_rw_1.fio rbd_w_iodepth_1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.1 Starting 1 process Segmentation fault1)][13.3%][r=0KiB/s,w=14.7MiB/s][r=0,w=3768 IOPS][eta 00m:52s] root@infra-138-b16-27:~#
(The IOps seem great!)
My fio test is fairly simple:
[global] ioengine=rbd clientname=admin pool=rbd rbdname=fio1 invalidate=0 bs=4k runtime=60 direct=1
[rbd_w_iodepth_1] rw=randwrite iodepth=1
I have tried to trace it with gdb, but I didn't get further with my backtrace then:
(gdb) bt #0 ContextWQ::process (ctx=0x7fffb8081480, this=0x7fffb8012470) at ./src/common/WorkQueue.h:556 #1 ThreadPool::PointerWQ<Context>::_void_process (this=0x7fffb8012470, item=0x7fffb8081480, handle=...) at ./src/common/WorkQueue.h:341 #2 0x00007fffec600912 in ThreadPool::worker (this=0x7fffb8012018, wt=<optimized out>) at ./src/common/WorkQueue.cc:117 #3 0x00007fffec601801 in ThreadPool::WorkThread::entry (this=<optimized out>) at ./src/common/WorkQueue.h:395 #4 0x00007ffff5c796db in start_thread (arg=0x7fffb17fa700) at pthread_create.c:463 #5 0x00007ffff579e71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Has anybody been able to use pwl_cache successfully?
Hi Wido,
Unfortunately "rbd_persistent_cache_mode = ssd" cache has shipped rather broken. This particular crash is most likely already fixed in master, but there are a few more outstanding. There is a dozen of "[pwl ssd] ..." tickets in the rbd project, the fixes would be backported to pacific once the ssd mode is stable enough.
Until then, I would to stick to "rbd_persistent_cache_mode = rwl" or avoid the pwl_cache plugin entirely.
I tried with rwl and am now using XFS with DAX enabled and it works. Performance-wise I see an improvement of 2x in terms of IOps with qd=1 bs=4k My kernel is reporting that my PM983 4TB NVMe I am using as a backing device is 100% util, but that seems off. Can we expect any fixes for this cache in .5 or .6? Wido
Thanks,
Ilya
On Wed, Jun 9, 2021 at 1:45 PM Wido den Hollander <wido@42on.com> wrote:
On 08/06/2021 21:04, Ilya Dryomov wrote:
On Tue, Jun 8, 2021 at 7:11 PM Wido den Hollander <wido@42on.com> wrote:
Hi,
So I've been doing some tests with v16.2.4 with a 2TB Samsung PM983 SSD mounted under /mnt/rbd-cache
rbd_persistent_cache_mode = ssd rbd_persistent_cache_size = 2G rbd_persistent_cache_path = /mnt/rbd-cache rbd_plugins = pwl_cache
I tried both XFS and EXT4 as the filesystem.
This however leads to fio or 'rbd bench' to crash:
root@infra-138-b16-27:~# fio fio/rbd_rw_1.fio rbd_w_iodepth_1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.1 Starting 1 process Segmentation fault1)][13.3%][r=0KiB/s,w=14.7MiB/s][r=0,w=3768 IOPS][eta 00m:52s] root@infra-138-b16-27:~#
(The IOps seem great!)
My fio test is fairly simple:
[global] ioengine=rbd clientname=admin pool=rbd rbdname=fio1 invalidate=0 bs=4k runtime=60 direct=1
[rbd_w_iodepth_1] rw=randwrite iodepth=1
I have tried to trace it with gdb, but I didn't get further with my backtrace then:
(gdb) bt #0 ContextWQ::process (ctx=0x7fffb8081480, this=0x7fffb8012470) at ./src/common/WorkQueue.h:556 #1 ThreadPool::PointerWQ<Context>::_void_process (this=0x7fffb8012470, item=0x7fffb8081480, handle=...) at ./src/common/WorkQueue.h:341 #2 0x00007fffec600912 in ThreadPool::worker (this=0x7fffb8012018, wt=<optimized out>) at ./src/common/WorkQueue.cc:117 #3 0x00007fffec601801 in ThreadPool::WorkThread::entry (this=<optimized out>) at ./src/common/WorkQueue.h:395 #4 0x00007ffff5c796db in start_thread (arg=0x7fffb17fa700) at pthread_create.c:463 #5 0x00007ffff579e71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Has anybody been able to use pwl_cache successfully?
Hi Wido,
Unfortunately "rbd_persistent_cache_mode = ssd" cache has shipped rather broken. This particular crash is most likely already fixed in master, but there are a few more outstanding. There is a dozen of "[pwl ssd] ..." tickets in the rbd project, the fixes would be backported to pacific once the ssd mode is stable enough.
Until then, I would to stick to "rbd_persistent_cache_mode = rwl" or avoid the pwl_cache plugin entirely.
I tried with rwl and am now using XFS with DAX enabled and it works.
Performance-wise I see an improvement of 2x in terms of IOps with qd=1 bs=4k
My kernel is reporting that my PM983 4TB NVMe I am using as a backing device is 100% util, but that seems off.
Can we expect any fixes for this cache in .5 or .6?
Some may land in .5 but .6 would probably be the earliest for the ssd mode to actually be usable. Thanks, Ilya
participants (2)
-
Ilya Dryomov
-
Wido den Hollander