Em qui., 10 de ago. de 2023 às 12:47, Hans van den Bogert < hansbogert@gmail.com> escreveu:
On Thu, Aug 10, 2023, 17:36 Murilo Morais <murilo@evocorp.com.br> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second,
This is 3000iops. I would call that bad for 60 drives and a replication of 3. Which amount of iops did you expect?
which is quite annoying. I achieve the same rate if rw=read.
If I use librbd's cache I get a considerable improvement in writing, but reading remains the same.
I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either.
Note: I tested it on another smaller cluster, with 36 SAS disks and got
the
same result.
This I concur is a weird result compared to 60 disks. Are you using the same disks and all other parameters the same, like the replication factor? Is the performance really the same? Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
Yes is yes. I will try with others in parallel and compare the results.
I don't know exactly what to look for or configure to have any
improvement.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io