Slow write speed on 3-node cluster with 6* SATA Harddisks (~ 3.5 MB/s)
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed. The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s). I wonder if I can do anything to speed things up? My Hardware is as the following: 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40 The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine. I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like the following: read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing. If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s. Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster? Thank you in advance for any help, Best Regards, Hermann -- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, harddisks are awful slow. That's normal and expected and result of random io as it would be in a Ceph cluster. You can speed up raw bandwidth performance using EC but not on such small clusters and not when having high io load. As you mentioned Proxmox, when it comes to VM workloads spinning media is never an option, use Flash! -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Mo., 4. Nov. 2019 um 23:44 Uhr schrieb Hermann Himmelbauer < hermann@qwer.tk>:
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like the following:
read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s
write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster?
Thank you in advance for any help,
Best Regards, Hermann
-- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Use `cache=writeback` QEMU option for HDD clusters, that should solve your issue
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like the following:
read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s
write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster?
Thank you in advance for any help,
Best Regards, Hermann
Hi, Thank you for your quick reply, Proxmox offers me "writeback" (cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my "dd" test, this makes no difference at all. I still have write speeds of ~ 4,5 MB/s. Perhaps "dd" disables the write cache? Would it perhaps help to put the journal or something else on a SSD? Best Regards, Hermann Am 05.11.19 um 11:49 schrieb vitalif@yourcmc.ru:
Use `cache=writeback` QEMU option for HDD clusters, that should solve your issue
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like the following:
read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s
write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster?
Thank you in advance for any help,
Best Regards, Hermann
-- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers)
Yes, cache=unsafe has no effect with RBD. Hm, that's strange, you should get ~40*6 MB/s linear write with 6 HDDs and Bluestore. Try to create a test image and test it with 'fio -ioengine=rbd -name=test -direct=1 -rw=write -bs=4M -iodepth=16 -pool=<pool> -rbdname=<rbd>' from outside a VM. If you still get 4 MB/s, something's wrong with your ceph. If you get adequate performance, something's wrong with your VM settings. 5 ноября 2019 г. 14:31:38 GMT+03:00, Hermann Himmelbauer <hermann@qwer.tk> пишет:
Hi, Thank you for your quick reply, Proxmox offers me "writeback" (cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my "dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards, Hermann
Use `cache=writeback` QEMU option for HDD clusters, that should solve your issue
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like
Am 05.11.19 um 11:49 schrieb vitalif@yourcmc.ru: the
following:
read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s
write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster?
Thank you in advance for any help,
Best Regards, Hermann
-- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers)
-- With best regards, Vitaliy Filippov
Dear Vitaliy, dear Paul, Changing the block size for "dd" makes a huge difference. However, still some things are not fully clear to me: As recommended, I tried writing / reading directly to the rbd and this is blazingly fast: fio -ioengine=rbd -name=test -direct=1 -rw=read -bs=4M -iodepth=16 -pool=SATA -rbdname=vm-100-disk-0 write: IOPS=40, BW=160MiB/s (168MB/s)(4096MiB/25529msec) read: IOPS=135, BW=542MiB/s (568MB/s)(4096MiB/7556msec) When I do the same within the virtual machine, I get the following results: fio --filename=/dev/vdb -name=test -direct=1 -rw=write -bs=4M -iodepth=16 -------- cache = writeback ---------- Blocksize: 4M: read : io=4096.0MB, bw=97640KB/s, iops=23, runt= 42957msec write: io=4096.0MB, bw=85250KB/s, iops=20, runt= 49200msec Blocksize: 4k: read : io=4096.0MB, bw=3988.6KB/s, iops=997, runt=1051599msec write: io=4096.0MB, bw=14529KB/s, iops=3632, runt=288686msec ------------------------------------- The speeds are much slower whereas I don't really know why. Nevertheless, this seams also really reasonable, although it's strange that there's no difference between "cache = unsafe" and "cache = writeback". Moreover, I find it strange that reading with 4k-blocks is that slow, while writing is still o.k. My virtual machine is Debian 8, with a paravirtualized block device (/dev/vdb), the process (and qemu parameters) look like the following: root 1854681 18.3 8.0 5813032 1980804 ? Sl 16:00 20:55 /usr/bin/kvm -id 100 -name bya-backend -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=e53d6e2d-708e-4511-bb26-b0f1aefd81c6 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/100.vnc,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:7c6dc4c7e9f -drive if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/mnt/pve/pontos-images/images/100/vm-100-disk-2.raw,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -drive file=rbd:SATA/vm-100-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/SATA.keyring,if=none,id=drive-virtio1,cache=writeback,format=raw,aio=threads,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device e1000,mac=46:22:36:C3:37:7E,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -machine type=pc In case you have some further speedup-hints for me, I'm glad. Nevertheless thank you a lot for help! Best Regards, Hermann Am 05.11.19 um 12:49 schrieb Виталий Филиппов:
Yes, cache=unsafe has no effect with RBD. Hm, that's strange, you should get ~40*6 MB/s linear write with 6 HDDs and Bluestore.
Try to create a test image and test it with 'fio -ioengine=rbd -name=test -direct=1 -rw=write -bs=4M -iodepth=16 -pool=<pool> -rbdname=<rbd>' from outside a VM.
If you still get 4 MB/s, something's wrong with your ceph. If you get adequate performance, something's wrong with your VM settings.
5 ноября 2019 г. 14:31:38 GMT+03:00, Hermann Himmelbauer <hermann@qwer.tk> пишет:
Hi, Thank you for your quick reply, Proxmox offers me "writeback" (cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my "dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards, Hermann
Am 05.11.19 um 11:49 schrieb vitalif@yourcmc.ru:
Use `cache=writeback` QEMU option for HDD clusters, that should solve your issue
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like the following:
read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s
write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster?
Thank you in advance for any help,
Best Regards, Hermann
-- With best regards, Vitaliy Filippov
-- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers)
On Wed, Nov 6, 2019 at 5:57 PM Hermann Himmelbauer <hermann@qwer.tk> wrote:
Dear Vitaliy, dear Paul,
Changing the block size for "dd" makes a huge difference.
However, still some things are not fully clear to me:
As recommended, I tried writing / reading directly to the rbd and this is blazingly fast:
fio -ioengine=rbd -name=test -direct=1 -rw=read -bs=4M -iodepth=16 -pool=SATA -rbdname=vm-100-disk-0
write: IOPS=40, BW=160MiB/s (168MB/s)(4096MiB/25529msec) read: IOPS=135, BW=542MiB/s (568MB/s)(4096MiB/7556msec)
When I do the same within the virtual machine, I get the following results:
fio --filename=/dev/vdb -name=test -direct=1 -rw=write -bs=4M -iodepth=16
--ioengine=aio Also, VirtIO block devices are usually slower than virtual SCSI disks on VirtIO SCSI controllers. Paul
-------- cache = writeback ---------- Blocksize: 4M: read : io=4096.0MB, bw=97640KB/s, iops=23, runt= 42957msec write: io=4096.0MB, bw=85250KB/s, iops=20, runt= 49200msec
Blocksize: 4k: read : io=4096.0MB, bw=3988.6KB/s, iops=997, runt=1051599msec write: io=4096.0MB, bw=14529KB/s, iops=3632, runt=288686msec -------------------------------------
The speeds are much slower whereas I don't really know why. Nevertheless, this seams also really reasonable, although it's strange that there's no difference between "cache = unsafe" and "cache = writeback". Moreover, I find it strange that reading with 4k-blocks is that slow, while writing is still o.k.
My virtual machine is Debian 8, with a paravirtualized block device (/dev/vdb), the process (and qemu parameters) look like the following:
root 1854681 18.3 8.0 5813032 1980804 ? Sl 16:00 20:55 /usr/bin/kvm -id 100 -name bya-backend -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=e53d6e2d-708e-4511-bb26-b0f1aefd81c6 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/100.vnc,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:7c6dc4c7e9f -drive if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/mnt/pve/pontos-images/images/100/vm-100-disk-2.raw,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -drive file=rbd:SATA/vm-100-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/SATA.keyring,if=none,id=drive-virtio1,cache=writeback,format=raw,aio=threads,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device e1000,mac=46:22:36:C3:37:7E,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -machine type=pc
In case you have some further speedup-hints for me, I'm glad.
Nevertheless thank you a lot for help!
Best Regards, Hermann
Am 05.11.19 um 12:49 schrieb Виталий Филиппов:
Yes, cache=unsafe has no effect with RBD. Hm, that's strange, you should get ~40*6 MB/s linear write with 6 HDDs and Bluestore.
Try to create a test image and test it with 'fio -ioengine=rbd -name=test -direct=1 -rw=write -bs=4M -iodepth=16 -pool=<pool> -rbdname=<rbd>' from outside a VM.
If you still get 4 MB/s, something's wrong with your ceph. If you get adequate performance, something's wrong with your VM settings.
5 ноября 2019 г. 14:31:38 GMT+03:00, Hermann Himmelbauer <hermann@qwer.tk> пишет:
Hi, Thank you for your quick reply, Proxmox offers me "writeback" (cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my "dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards, Hermann
Am 05.11.19 um 11:49 schrieb vitalif@yourcmc.ru:
Use `cache=writeback` QEMU option for HDD clusters, that should solve your issue
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like the following:
read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s
write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster?
Thank you in advance for any help,
Best Regards, Hermann
-- With best regards, Vitaliy Filippov
-- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, My three-node lab cluster is similar to yours but with 3x bluestore OSDs per node (4TB SATA spinning disks) and 1x shared DB/WAL (240GB SATA SSD) device per node. I'm only using gigabit networking (one interface public, one interface cluster) also ceph 14.2.4 with 3x replicas. I would have expected your dd commands to use the cache, try these instead inside your VM: # Write test dd if=/dev/zero of=/zero.file bs=32M oflag=direct status=progress # Read test dd if=/zero.file of=/dev/null bs=32M iflag=direct status=progress You can obviously delete /zero.file when you're finished. - bs=32M tells dd to read/write 32MB at a time, I think the default is something like 512 bytes which slows things up significantly without a cache. - oflag/iflag=direct will use direct I/O bypassing the cache. - status=progress is just instead of where you're using pv to show the transfer rate. On my cluster I get 124MB/sec read (maxing out the network) and 74MB/sec write. Without bs=32M I get more like 1MB/sec read and write. The VM I'm using for this test is cache=writeback and virtio-scsi (i.e. sda rather than vda). Simon On 05/11/2019 11:31, Hermann Himmelbauer wrote:
Hi, Thank you for your quick reply, Proxmox offers me "writeback" (cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my "dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards, Hermann
Good you are getting 124MB/s via Gbit, I have only been able to get 110MB/s. If you are interested, I am also having 4TB sata hdd without db/wal on ssd, 4 nodes, but 10Gbit [@]# dd if=/dev/zero of=zero.file bs=32M oflag=direct status=progress 3758096384 bytes (3.8 GB) copied, 36.364817 s, 103 MB/s^C 113+0 records in 113+0 records out 3791650816 bytes (3.8 GB) copied, 36.6824 s, 103 MB/s [@]# dd if=zero.file of=/dev/null bs=32M iflag=direct status=progress 3657433088 bytes (3.7 GB) copied, 15.346591 s, 238 MB/s 113+0 records in 113+0 records out 3791650816 bytes (3.8 GB) copied, 15.8171 s, 240 MB/s -----Original Message----- From: Simon Ironside [mailto:sironside@caffetine.org] Sent: dinsdag 5 november 2019 13:08 To: ceph-users@ceph.io Subject: [ceph-users] Re: Slow write speed on 3-node cluster with 6* SATA Harddisks (~ 3.5 MB/s) Hi, My three-node lab cluster is similar to yours but with 3x bluestore OSDs per node (4TB SATA spinning disks) and 1x shared DB/WAL (240GB SATA SSD) device per node. I'm only using gigabit networking (one interface public, one interface cluster) also ceph 14.2.4 with 3x replicas. I would have expected your dd commands to use the cache, try these instead inside your VM: # Write test dd if=/dev/zero of=/zero.file bs=32M oflag=direct status=progress # Read test dd if=/zero.file of=/dev/null bs=32M iflag=direct status=progress You can obviously delete /zero.file when you're finished. - bs=32M tells dd to read/write 32MB at a time, I think the default is something like 512 bytes which slows things up significantly without a cache. - oflag/iflag=direct will use direct I/O bypassing the cache. - status=progress is just instead of where you're using pv to show the transfer rate. On my cluster I get 124MB/sec read (maxing out the network) and 74MB/sec write. Without bs=32M I get more like 1MB/sec read and write. The VM I'm using for this test is cache=writeback and virtio-scsi (i.e. sda rather than vda). Simon On 05/11/2019 11:31, Hermann Himmelbauer wrote:
Hi, Thank you for your quick reply, Proxmox offers me "writeback" (cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my "dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards, Hermann
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, Nov 4, 2019 at 11:44 PM Hermann Himmelbauer <hermann@qwer.tk> wrote:
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like the following:
read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s
write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s
you are mainly measuring latency, not bandwidth here. Use a larger block size (bs=4M) to measure bandwidth. Using /dev/zero as source can also be a bad benchmark if you are running with detect-zeroes=unmap (which is the proxmox default IIRC) Paul
When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster?
Thank you in advance for any help,
Best Regards, Hermann
-- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Hermann Himmelbauer
-
Marc Roos
-
Martin Verges
-
Paul Emmerich
-
Simon Ironside
-
vitalif＠yourcmc.ru
-
Виталий Филиппов