Good afternoon everybody! I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img If I increase the blocksize I can easily reach 1.5 GBps or more. But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read. If I use librbd's cache I get a considerable improvement in writing, but reading remains the same. I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either. Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result. I don't know exactly what to look for or configure to have any improvement.
On Thu, Aug 10, 2023, 17:36 Murilo Morais <murilo@evocorp.com.br> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second,
This is 3000iops. I would call that bad for 60 drives and a replication of 3. Which amount of iops did you expect? which is quite annoying. I achieve the same rate if rw=read.
If I use librbd's cache I get a considerable improvement in writing, but reading remains the same.
I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
This I concur is a weird result compared to 60 disks. Are you using the same disks and all other parameters the same, like the replication factor? Is the performance really the same? Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
I don't know exactly what to look for or configure to have any improvement. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second,
This is 3000iops. I would call that bad for 60 drives and a replication of 3. Which amount of iops did you expect?
How is this related to 60 drives? His test is only on 3 drives at a time not?
Em qui., 10 de ago. de 2023 às 12:47, Hans van den Bogert < hansbogert@gmail.com> escreveu:
On Thu, Aug 10, 2023, 17:36 Murilo Morais <murilo@evocorp.com.br> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second,
This is 3000iops. I would call that bad for 60 drives and a replication of 3. Which amount of iops did you expect?
which is quite annoying. I achieve the same rate if rw=read.
If I use librbd's cache I get a considerable improvement in writing, but reading remains the same.
I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either.
Note: I tested it on another smaller cluster, with 36 SAS disks and got
the
same result.
This I concur is a weird result compared to 60 disks. Are you using the same disks and all other parameters the same, like the replication factor? Is the performance really the same? Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
Yes is yes. I will try with others in parallel and compare the results.
I don't know exactly what to look for or configure to have any
improvement.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read.
If I use librbd's cache I get a considerable improvement in writing, but reading remains the same.
I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
I don't know exactly what to look for or configure to have any improvement.
What are you expecting? This is what I have on a vm with an rbd from a hdd pool <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> [@~]# fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -size=1G -iodepth=16 -rw=write -filename=./test.img test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=57.5MiB/s][r=0,w=14.7k IOPS][eta 00m:00s] [@~]# fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -size=1G -iodepth=1 -rw=write -filename=./test.img test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=19.9MiB/s][r=0,w=5090 IOPS][eta 00m:00s]
Em qui., 10 de ago. de 2023 às 13:01, Marc <Marc@f1-outsourcing.eu> escreveu:
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read.
If I use librbd's cache I get a considerable improvement in writing, but reading remains the same.
I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
I don't know exactly what to look for or configure to have any improvement.
What are you expecting?
I expected something a little better (at least in reading), since the other one, with less disks, is showing the same rates. :(
This is what I have on a vm with an rbd from a hdd pool
<driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
I'm using exactly this in libvirt.
[@~]# fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -size=1G -iodepth=16 -rw=write -filename=./test.img test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=57.5MiB/s][r=0,w=14.7k IOPS][eta 00m:00s]
With writeback I get constant 100 Megs, which is pretty good. I can live with writeback.
[@~]# fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -size=1G -iodepth=1 -rw=write -filename=./test.img test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=19.9MiB/s][r=0,w=5090 IOPS][eta 00m:00s]
Thanks for showing your results, it's something I can compare to.
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
Old hardware? SAS is mostly dead.
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
On what kind of client?
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read.
If your client is VM especially, check if you have IOPS throttling. With small block sizes you'll throttle IOPS long before bandwidth.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
SAS has a price premium over SATA, and still requires an HBA. Many chassis vendors really want you to buy an anachronistic RoC HBA. Eschewing SAS and the HBA helps close the gap to justify SSDs, the TCO just doesn't favor spinners.
Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
Agreed that Ceph is a scale out solution, not DAS, but note the difference reported with a larger block size.
How is this related to 60 drives? His test is only on 3 drives at a time not?
RBD volumes by and large will live on most or all OSDs in the pool.
I don't know exactly what to look for or configure to have any improvement. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, You can use the following formula to roughly calculate the IOPS you can get from a cluster: (Drive_IOPS * Number_of_Drives * 0.75) / Cluster_Size. For example, for 60 10K rpm SAS drives each capable of 200 4K IOPS and a replicated pool with size 3: (~200 * 60 * 0.75) / 3 = ~3000 IOPS with block size = 4K. That's what the OP is getting, give or take. /Z On Thu, 10 Aug 2023 at 20:20, Anthony D'Atri <aad@dreamsnake.net> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
Old hardware? SAS is mostly dead.
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
On what kind of client?
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read.
If your client is VM especially, check if you have IOPS throttling. With small block sizes you'll throttle IOPS long before bandwidth.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
SAS has a price premium over SATA, and still requires an HBA. Many chassis vendors really want you to buy an anachronistic RoC HBA.
Eschewing SAS and the HBA helps close the gap to justify SSDs, the TCO just doesn't favor spinners.
Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
Agreed that Ceph is a scale out solution, not DAS, but note the difference reported with a larger block size.
How is this related to 60 drives? His test is only on 3 drives at a time not?
RBD volumes by and large will live on most or all OSDs in the pool.
I don't know exactly what to look for or configure to have any
improvement.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It makes sense. Em qui., 10 de ago. de 2023 às 16:04, Zakhar Kirpichenko <zakhar@gmail.com> escreveu:
Hi,
You can use the following formula to roughly calculate the IOPS you can get from a cluster: (Drive_IOPS * Number_of_Drives * 0.75) / Cluster_Size.
For example, for 60 10K rpm SAS drives each capable of 200 4K IOPS and a replicated pool with size 3: (~200 * 60 * 0.75) / 3 = ~3000 IOPS with block size = 4K.
That's what the OP is getting, give or take.
/Z
On Thu, 10 Aug 2023 at 20:20, Anthony D'Atri <aad@dreamsnake.net> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
Old hardware? SAS is mostly dead.
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
On what kind of client?
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read.
If your client is VM especially, check if you have IOPS throttling. With small block sizes you'll throttle IOPS long before bandwidth.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
SAS has a price premium over SATA, and still requires an HBA. Many chassis vendors really want you to buy an anachronistic RoC HBA.
Eschewing SAS and the HBA helps close the gap to justify SSDs, the TCO just doesn't favor spinners.
Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
Agreed that Ceph is a scale out solution, not DAS, but note the difference reported with a larger block size.
How is this related to 60 drives? His test is only on 3 drives at a time not?
RBD volumes by and large will live on most or all OSDs in the pool.
I don't know exactly what to look for or configure to have any
improvement.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, In case of pool a cluster where most pools are with erasure code 4+2, what would you consider as value for cluster_size? Giuseppe On 10.08.23, 21:06, "Zakhar Kirpichenko" <zakhar@gmail.com <mailto:zakhar@gmail.com>> wrote: Hi, You can use the following formula to roughly calculate the IOPS you can get from a cluster: (Drive_IOPS * Number_of_Drives * 0.75) / Cluster_Size. For example, for 60 10K rpm SAS drives each capable of 200 4K IOPS and a replicated pool with size 3: (~200 * 60 * 0.75) / 3 = ~3000 IOPS with block size = 4K. That's what the OP is getting, give or take. /Z On Thu, 10 Aug 2023 at 20:20, Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
Old hardware? SAS is mostly dead.
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
On what kind of client?
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read.
If your client is VM especially, check if you have IOPS throttling. With small block sizes you'll throttle IOPS long before bandwidth.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
SAS has a price premium over SATA, and still requires an HBA. Many chassis vendors really want you to buy an anachronistic RoC HBA.
Eschewing SAS and the HBA helps close the gap to justify SSDs, the TCO just doesn't favor spinners.
Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
Agreed that Ceph is a scale out solution, not DAS, but note the difference reported with a larger block size.
How is this related to 60 drives? His test is only on 3 drives at a time not?
RBD volumes by and large will live on most or all OSDs in the pool.
I don't know exactly what to look for or configure to have any
improvement.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
On 10/08/2023 22:04, Zakhar Kirpichenko wrote:
Hi,
You can use the following formula to roughly calculate the IOPS you can get from a cluster: (Drive_IOPS * Number_of_Drives * 0.75) / Cluster_Size.
For example, for 60 10K rpm SAS drives each capable of 200 4K IOPS and a replicated pool with size 3: (~200 * 60 * 0.75) / 3 = ~3000 IOPS with block size = 4K.
Good approximation but some comments: -This could apply to hdds, not to ssd/nvme. -If you do not use external wal/db ssd device you would get lower results -You would need to test with an io depth several times higher than your hdd count. In your case with 60 hdds you would use 256 or so, using 16 as you did will not stress the hdds to their max. -Test with deep scrubbing off as it could impact performance specially with hdds. /maged
That's what the OP is getting, give or take.
/Z
On Thu, 10 Aug 2023 at 20:20, Anthony D'Atri <aad@dreamsnake.net> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each Old hardware? SAS is mostly dead.
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img On what kind of client?
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read. If your client is VM especially, check if you have IOPS throttling. With small block sizes you'll throttle IOPS long before bandwidth.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result. SAS has a price premium over SATA, and still requires an HBA. Many chassis vendors really want you to buy an anachronistic RoC HBA.
Eschewing SAS and the HBA helps close the gap to justify SSDs, the TCO just doesn't favor spinners.
Maybe the 5 host cluster is not saturated by your current fio test. Try running 2 or 4 in parallel.
Agreed that Ceph is a scale out solution, not DAS, but note the difference reported with a larger block size.
How is this related to 60 drives? His test is only on 3 drives at a time not?
RBD volumes by and large will live on most or all OSDs in the pool.
I don't know exactly what to look for or configure to have any improvement. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Murilo, This is an expected result, and it is not specific to Ceph. Any storage that consists of multiple disks will produce a performance gain over a single disk only if the workload allows for concurrent use of these disks - which is not the case with your 4K benchmark due to the de-facto missing readahead. The default readahead in Linux is just 128 kilobytes, and it means that even in a linear read scenario the benchmark has no way to hit multiple RADOS objects at once. Reminder: they are 4 megabytes in size by default with RBD. To allow for faster linear reads and writes, please create a file, /etc/udev/rules.d/80-rbd.rules, with the following contents (assuming that the VM sees the RBD as /dev/sda): KERNEL=="sda", ENV{DEVTYPE}=="disk", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="32768" Or test it without any udev rule like this: bloskdev --setra 65536 /dev/sda The difference in numbers is because one is in kilobytes and one is in 512-byte sectors. Mandatory warning: this setting can hurt other workloads. On Thu, Aug 10, 2023 at 11:37 PM Murilo Morais <murilo@evocorp.com.br> wrote:
Good afternoon everybody!
I have the following scenario: Pool RBD replication x3 5 hosts with 12 SAS spinning disks each
I'm using exactly the following line with FIO to test: fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G -iodepth=16 -rw=write -filename=./test.img
If I increase the blocksize I can easily reach 1.5 GBps or more.
But when I use blocksize in 4K I get a measly 12 Megabytes per second, which is quite annoying. I achieve the same rate if rw=read.
If I use librbd's cache I get a considerable improvement in writing, but reading remains the same.
I already tested with rbd_read_from_replica_policy=balance but I didn't notice any difference. I tried to leave readahead enabled by setting rbd_readahead_disable_after_bytes=0 but I didn't see any difference in sequential reading either.
Note: I tested it on another smaller cluster, with 36 SAS disks and got the same result.
I don't know exactly what to look for or configure to have any improvement. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov
This is an expected result, and it is not specific to Ceph. Any storage that consists of multiple disks will produce a performance gain over a single disk only if the workload allows for concurrent use of these disks - which is not the case with your 4K benchmark due to the de-facto missing readahead.
I don't think that affects writes.
To allow for faster linear reads and writes, please create a file, /etc/udev/rules.d/80-rbd.rules, with the following contents (assuming that the VM sees the RBD as /dev/sda):
I think the OP didn't say that the client is a VM. It might be, it might be a KRBD mount, it might be through CSI, we don't know. I asked the OP privately if it's a VM because the first thing I check in this situation is libvirt / librbd throttling. It sounds a *lot* like saturating an IOPS client throttle, something I've seen time and again in virtualization scenarios.
To allow for faster linear reads and writes, please create a file, /etc/udev/rules.d/80-rbd.rules, with the following contents (assuming that the VM sees the RBD as /dev/sda):
KERNEL=="sda", ENV{DEVTYPE}=="disk", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="32768"
Or test it without any udev rule like this:
bloskdev --setra 65536 /dev/sda
The difference in numbers is because one is in kilobytes and one is in 512-byte sectors.
Mandatory warning: this setting can hurt other workloads.
Such as what workloads? Would the results show in the average latency rbd overview? Or is it better to monitor somewhere else?
On 12/08/2023 13:04, Marc wrote:
To allow for faster linear reads and writes, please create a file, /etc/udev/rules.d/80-rbd.rules, with the following contents (assuming that the VM sees the RBD as /dev/sda):
KERNEL=="sda", ENV{DEVTYPE}=="disk", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="32768"
Or test it without any udev rule like this:
bloskdev --setra 65536 /dev/sda
The difference in numbers is because one is in kilobytes and one is in 512-byte sectors.
Mandatory warning: this setting can hurt other workloads.
Such as what workloads? Would the results show in the average latency rbd overview? Or is it better to monitor somewhere else?
32 MB read ahead is too excessive for the vast majority of workloads. The hdd would only do about 10 iops, so any random reads will perform very bad. I would say maybe unless you write your own rados app with full control on large sequential writes and reads then it is very risky. If you store small objects, like files on cephfs, the scrub load could kill your hdd. Even large objects, depending on how/when they were written/updated their physical extents on the disk may not be continuous even for logical sequential extents and could require several ops to read. i would not go above 1MB read ahead for hdd as it will not impact your random iops too much. Generally if you are looking for 4k iops performance like the original post, you really should consider ssd/nvme. /maged
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yep. Remember that most Ceph clusters serve a number of simultaneous clients, so the “IO blender” effect more or less presents a random workload to drives. Dedicated single-client node-local drives might benefit from such strategies. But really gymnastics like this for uncertain gain serve to reinforce the fact that HDDs are a false economy. Have we yet established for sure that the OP’s client is or is not a VM? Still smells a lot like iops throttling.
On Aug 13, 2023, at 7:51 AM, Maged Mokhtar <mmokhtar@petasan.org> wrote:
On 12/08/2023 13:04, Marc wrote:
To allow for faster linear reads and writes, please create a file, /etc/udev/rules.d/80-rbd.rules, with the following contents (assuming that the VM sees the RBD as /dev/sda):
KERNEL=="sda", ENV{DEVTYPE}=="disk", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="32768"
Or test it without any udev rule like this:
bloskdev --setra 65536 /dev/sda
The difference in numbers is because one is in kilobytes and one is in 512-byte sectors.
Mandatory warning: this setting can hurt other workloads.
Such as what workloads? Would the results show in the average latency rbd overview? Or is it better to monitor somewhere else?
32 MB read ahead is too excessive for the vast majority of workloads. The hdd would only do about 10 iops, so any random reads will perform very bad. I would say maybe unless you write your own rados app with full control on large sequential writes and reads then it is very risky. If you store small objects, like files on cephfs, the scrub load could kill your hdd. Even large objects, depending on how/when they were written/updated their physical extents on the disk may not be continuous even for logical sequential extents and could require several ops to read. i would not go above 1MB read ahead for hdd as it will not impact your random iops too much.
Generally if you are looking for 4k iops performance like the original post, you really should consider ssd/nvme.
/maged
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (9)
-
Alexander E. Patrakov
-
Anthony D'Atri
-
Anthony D'Atri
-
Hans van den Bogert
-
Lo Re Giuseppe
-
Maged Mokhtar
-
Marc
-
Murilo Morais
-
Zakhar Kirpichenko