On Thu, Aug 10, 2023, 17:36 Murilo Morais <murilo@evocorp.com.br> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second,
This is 3000iops. I would call that bad for 60 drives and a replication of 3. Which amount of iops did you expect? which is quite annoying. I achieve the same rate if rw=read.
If I use librbd's cache I get a considerable improvement in writing, but reading remains the same.
I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
This I concur is a weird result compared to 60 disks. Are you using the same disks and all other parameters the same, like the replication factor? Is the performance really the same? Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
I don't know exactly what to look for or configure to have any improvement. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io