On Wed, Jun 9, 2021 at 1:45 PM Wido den Hollander <wido@42on.com> wrote:
On 08/06/2021 21:04, Ilya Dryomov wrote:
On Tue, Jun 8, 2021 at 7:11 PM Wido den Hollander <wido@42on.com> wrote:
Hi,
So I've been doing some tests with v16.2.4 with a 2TB Samsung PM983 SSD mounted under /mnt/rbd-cache
rbd_persistent_cache_mode = ssd rbd_persistent_cache_size = 2G rbd_persistent_cache_path = /mnt/rbd-cache rbd_plugins = pwl_cache
I tried both XFS and EXT4 as the filesystem.
This however leads to fio or 'rbd bench' to crash:
root@infra-138-b16-27:~# fio fio/rbd_rw_1.fio rbd_w_iodepth_1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.1 Starting 1 process Segmentation fault1)][13.3%][r=0KiB/s,w=14.7MiB/s][r=0,w=3768 IOPS][eta 00m:52s] root@infra-138-b16-27:~#
(The IOps seem great!)
My fio test is fairly simple:
[global] ioengine=rbd clientname=admin pool=rbd rbdname=fio1 invalidate=0 bs=4k runtime=60 direct=1
[rbd_w_iodepth_1] rw=randwrite iodepth=1
I have tried to trace it with gdb, but I didn't get further with my backtrace then:
(gdb) bt #0 ContextWQ::process (ctx=0x7fffb8081480, this=0x7fffb8012470) at ./src/common/WorkQueue.h:556 #1 ThreadPool::PointerWQ<Context>::_void_process (this=0x7fffb8012470, item=0x7fffb8081480, handle=...) at ./src/common/WorkQueue.h:341 #2 0x00007fffec600912 in ThreadPool::worker (this=0x7fffb8012018, wt=<optimized out>) at ./src/common/WorkQueue.cc:117 #3 0x00007fffec601801 in ThreadPool::WorkThread::entry (this=<optimized out>) at ./src/common/WorkQueue.h:395 #4 0x00007ffff5c796db in start_thread (arg=0x7fffb17fa700) at pthread_create.c:463 #5 0x00007ffff579e71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Has anybody been able to use pwl_cache successfully?
Hi Wido,
Unfortunately "rbd_persistent_cache_mode = ssd" cache has shipped rather broken. This particular crash is most likely already fixed in master, but there are a few more outstanding. There is a dozen of "[pwl ssd] ..." tickets in the rbd project, the fixes would be backported to pacific once the ssd mode is stable enough.
Until then, I would to stick to "rbd_persistent_cache_mode = rwl" or avoid the pwl_cache plugin entirely.
I tried with rwl and am now using XFS with DAX enabled and it works.
Performance-wise I see an improvement of 2x in terms of IOps with qd=1 bs=4k
My kernel is reporting that my PM983 4TB NVMe I am using as a backing device is 100% util, but that seems off.
Can we expect any fixes for this cache in .5 or .6?
Some may land in .5 but .6 would probably be the earliest for the ssd mode to actually be usable. Thanks, Ilya