5 Nov
2019
5 Nov
'19
1:43 a.m.
Hi,
I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and
recreated my ceph cluster with a new release (14.2.4 bluestore) -
basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80
MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf,
so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple
"dd" read and write test within a virtual machine (Debian 8) like the
following:
read:
dd if=/dev/vdb | pv | dd of=/dev/null
-> 80 MB/s
write:
dd if=/dev/zero | pv | dd of=/dev/vdb
-> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS
storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0
/dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph
cluster which is SSD based (Samsung PM1725b) which should be much
faster, however, I still wonder if there is any way to speed up my
harddisk based cluster?
Thank you in advance for any help,
Best Regards,
Hermann
--
hermann(a)qwer.tk
PGP/GPG: 299893C7 (on keyservers)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5 Nov
5 Nov
10:36 a.m.
Hello,
harddisks are awful slow. That's normal and expected and result of random
io as it would be in a Ceph cluster.
You can speed up raw bandwidth performance using EC but not on such small
clusters and not when having high io load.
As you mentioned Proxmox, when it comes to VM workloads spinning media is
never an option, use Flash!
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Mo., 4. Nov. 2019 um 23:44 Uhr schrieb Hermann Himmelbauer <
hermann(a)qwer.tk>gt;:
Hi,
I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and
recreated my ceph cluster with a new release (14.2.4 bluestore) -
basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80
MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf,
so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple
"dd" read and write test within a virtual machine (Debian 8) like the
following:
read:
dd if=/dev/vdb | pv | dd of=/dev/null
-> 80 MB/s
write:
dd if=/dev/zero | pv | dd of=/dev/vdb
-> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS
storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0
/dev/sdX", I gain a little bit of performance, write speed is then 4.3
MB/s.
Thanks to your help from the list I plan to install a second ceph
cluster which is SSD based (Samsung PM1725b) which should be much
faster, however, I still wonder if there is any way to speed up my
harddisk based cluster?
Thank you in advance for any help,
Best Regards,
Hermann
--
hermann(a)qwer.tk
PGP/GPG: 299893C7 (on keyservers)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:49 p.m.
Use `cache=writeback` QEMU option for HDD clusters, that should solve
your issue
> Hi,
> I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and
> recreated my ceph cluster with a new release (14.2.4 bluestore) -
> basically hoping to gain some I/O speed.
>
> The installation went flawlessly, reading is faster than before (~ 80
> MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
>
> I wonder if I can do anything to speed things up?
>
> My Hardware is as the following:
>
> 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
> 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
> interconnected via Infiniband 40
>
> The network should be reasonably fast, I measure ~ 16 GBit/s with
> iperf,
> so this seems fine.
>
> I use ceph for RBD only, so my measurement is simply doing a very
> simple
> "dd" read and write test within a virtual machine (Debian 8) like the
> following:
>
> read:
> dd if=/dev/vdb | pv | dd of=/dev/null
> -> 80 MB/s
>
>
> write:
> dd if=/dev/zero | pv | dd of=/dev/vdb
> -> 3.5 MB/s
>
> When I do the same on the virtual machine on a disk that is on a NFS
> storage, I get something about 30 MB/s for reading and writing.
>
> If I disable the write cache on all OSD disks via "hdparm -W 0
> /dev/sdX", I gain a little bit of performance, write speed is then 4.3
> MB/s.
>
> Thanks to your help from the list I plan to install a second ceph
> cluster which is SSD based (Samsung PM1725b) which should be much
> faster, however, I still wonder if there is any way to speed up my
> harddisk based cluster?
>
> Thank you in advance for any help,
>
> Best Regards,
> Hermann
2:31 p.m.
Hi,
Thank you for your quick reply, Proxmox offers me "writeback"
(cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my
"dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards,
Hermann
Am 05.11.19 um 11:49 schrieb vitalif(a)yourcmc.ru:
Use `cache=writeback` QEMU option for HDD clusters,
that should solve
your issue
> Hi,
> I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and
> recreated my ceph cluster with a new release (14.2.4 bluestore) -
> basically hoping to gain some I/O speed.
>
> The installation went flawlessly, reading is faster than before (~ 80
> MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
>
> I wonder if I can do anything to speed things up?
>
> My Hardware is as the following:
>
> 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
> 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
> interconnected via Infiniband 40
>
> The network should be reasonably fast, I measure ~ 16 GBit/s with iperf,
> so this seems fine.
>
> I use ceph for RBD only, so my measurement is simply doing a very simple
> "dd" read and write test within a virtual machine (Debian 8) like the
> following:
>
> read:
> dd if=/dev/vdb | pv | dd of=/dev/null
> -> 80 MB/s
>
>
> write:
> dd if=/dev/zero | pv | dd of=/dev/vdb
> -> 3.5 MB/s
>
> When I do the same on the virtual machine on a disk that is on a NFS
> storage, I get something about 30 MB/s for reading and writing.
>
> If I disable the write cache on all OSD disks via "hdparm -W 0
> /dev/sdX", I gain a little bit of performance, write speed is then 4.3
> MB/s.
>
> Thanks to your help from the list I plan to install a second ceph
> cluster which is SSD based (Samsung PM1725b) which should be much
> faster, however, I still wonder if there is any way to speed up my
> harddisk based cluster?
>
> Thank you in advance for any help,
>
> Best Regards,
> Hermann
--
hermann(a)qwer.tk
PGP/GPG: 299893C7 (on keyservers)
2:49 p.m.
Yes, cache=unsafe has no effect with RBD. Hm, that's strange, you should get ~40*6
MB/s linear write with 6 HDDs and Bluestore.
Try to create a test image and test it with 'fio -ioengine=rbd -name=test -direct=1
-rw=write -bs=4M -iodepth=16 -pool=<pool> -rbdname=<rbd>' from outside a
VM.
If you still get 4 MB/s, something's wrong with your ceph. If you get adequate
performance, something's wrong with your VM settings.
5 ноября 2019 г. 14:31:38 GMT+03:00, Hermann Himmelbauer <hermann(a)qwer.tk> пишет:
Hi,
Thank you for your quick reply, Proxmox offers me "writeback"
(cache=writeback) and "writeback unsafe" (cache=unsafe), however, for
my
"dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards,
Hermann
Am 05.11.19 um 11:49 schrieb vitalif(a)yourcmc.ru:
--
With best regards,
Vitaliy Filippov
Use `cache=writeback` QEMU option for HDD
clusters, that should solve
your issue
> Hi,
> I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and
> recreated my ceph cluster with a new release (14.2.4 bluestore) -
> basically hoping to gain some I/O speed.
>
> The installation went flawlessly, reading is faster than before (~
80
> MB/s), however, the write speed is still
really slow (~ 3,5 MB/s).
>
> I wonder if I can do anything to speed things up?
>
> My Hardware is as the following:
>
> 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
> 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
> interconnected via Infiniband 40
>
> The network should be reasonably fast, I measure ~ 16 GBit/s with
iperf,
> so this seems fine.
>
> I use ceph for RBD only, so my measurement is simply doing a very
simple
> "dd" read and write test within a
virtual machine (Debian 8) like
the
> following:
>
> read:
> dd if=/dev/vdb | pv | dd of=/dev/null
> -> 80 MB/s
>
>
> write:
> dd if=/dev/zero | pv | dd of=/dev/vdb
> -> 3.5 MB/s
>
> When I do the same on the virtual machine on a disk that is on a NFS
> storage, I get something about 30 MB/s for reading and writing.
>
> If I disable the write cache on all OSD disks via "hdparm -W 0
> /dev/sdX", I gain a little bit of performance, write speed is then
4.3
> MB/s.
>
> Thanks to your help from the list I plan to install a second ceph
> cluster which is SSD based (Samsung PM1725b) which should be much
> faster, however, I still wonder if there is any way to speed up my
> harddisk based cluster?
>
> Thank you in advance for any help,
>
> Best Regards,
> Hermann
--
hermann(a)qwer.tk
PGP/GPG: 299893C7 (on keyservers)
6 Nov
6 Nov
7:56 p.m.
Dear Vitaliy, dear Paul,
Changing the block size for "dd" makes a huge difference.
However, still some things are not fully clear to me:
As recommended, I tried writing / reading directly to the rbd and this
is blazingly fast:
fio -ioengine=rbd -name=test -direct=1 -rw=read -bs=4M -iodepth=16
-pool=SATA -rbdname=vm-100-disk-0
write: IOPS=40, BW=160MiB/s (168MB/s)(4096MiB/25529msec)
read: IOPS=135, BW=542MiB/s (568MB/s)(4096MiB/7556msec)
When I do the same within the virtual machine, I get the following results:
fio --filename=/dev/vdb -name=test -direct=1 -rw=write -bs=4M -iodepth=16
-------- cache = writeback ----------
Blocksize: 4M:
read : io=4096.0MB, bw=97640KB/s, iops=23, runt= 42957msec
write: io=4096.0MB, bw=85250KB/s, iops=20, runt= 49200msec
Blocksize: 4k:
read : io=4096.0MB, bw=3988.6KB/s, iops=997, runt=1051599msec
write: io=4096.0MB, bw=14529KB/s, iops=3632, runt=288686msec
-------------------------------------
The speeds are much slower whereas I don't really know why.
Nevertheless, this seams also really reasonable, although it's strange
that there's no difference between "cache = unsafe" and "cache =
writeback". Moreover, I find it strange that reading with 4k-blocks is
that slow, while writing is still o.k.
My virtual machine is Debian 8, with a paravirtualized block device
(/dev/vdb), the process (and qemu parameters) look like the following:
root 1854681 18.3 8.0 5813032 1980804 ? Sl 16:00 20:55
/usr/bin/kvm -id 100 -name bya-backend -chardev
socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon
chardev=qmp,mode=control -chardev
socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon
chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/100.pid
-daemonize -smbios type=1,uuid=e53d6e2d-708e-4511-bb26-b0f1aefd81c6 -smp
4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot
menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg
-vnc unix:/var/run/qemu-server/100.vnc,password -cpu
kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -device
pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device
pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device
piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device
usb-tablet,id=tablet,bus=uhci.0,port=1 -device
VGA,id=vga,bus=pci.0,addr=0x2 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi
initiator-name=iqn.1993-08.org.debian:01:7c6dc4c7e9f -drive
if=none,id=drive-ide2,media=cdrom,aio=threads -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive
file=/mnt/pve/pontos-images/images/100/vm-100-disk-2.raw,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on
-device
virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100
-drive
file=rbd:SATA/vm-100-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/SATA.keyring,if=none,id=drive-virtio1,cache=writeback,format=raw,aio=threads,detect-zeroes=on
-device virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb
-netdev
type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown
-device
e1000,mac=46:22:36:C3:37:7E,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
-machine type=pc
In case you have some further speedup-hints for me, I'm glad.
Nevertheless thank you a lot for help!
Best Regards,
Hermann
Am 05.11.19 um 12:49 schrieb Виталий Филиппов:
Yes, cache=unsafe has no effect with RBD. Hm,
that's strange, you should
get ~40*6 MB/s linear write with 6 HDDs and Bluestore.
Try to create a test image and test it with 'fio -ioengine=rbd
-name=test -direct=1 -rw=write -bs=4M -iodepth=16 -pool=<pool>
-rbdname=<rbd>' from outside a VM.
If you still get 4 MB/s, something's wrong with your ceph. If you get
adequate performance, something's wrong with your VM settings.
5 ноября 2019 г. 14:31:38 GMT+03:00, Hermann Himmelbauer
<hermann(a)qwer.tk> пишет:
Hi,
Thank you for your quick reply, Proxmox offers me "writeback"
(cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my
"dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards,
Hermann
Am 05.11.19 um 11:49 schrieb vitalif(a)yourcmc.ru:
Use `cache=writeback` QEMU option for HDD clusters, that should
solve
your issue
Hi,
I recently upgraded my 3-node cluster to proxmox 6 /
debian-10 and
recreated my ceph cluster with a new release (14.2.4
bluestore) -
basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than
before (~ 80
MB/s), however, the write speed is still really slow (~ 3,5
MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s
with iperf,
so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a
very simple
"dd" read and write test within a virtual machine (Debian 8)
like the
following:
read:
dd if=/dev/vdb | pv | dd of=/dev/null
-> 80 MB/s
write:
dd if=/dev/zero | pv | dd of=/dev/vdb
-> 3.5 MB/s
When I do the same on the virtual machine on a disk that is
on a NFS
storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0
/dev/sdX", I gain a little bit of performance, write speed
is then 4.3
MB/s.
Thanks to your help from the list I plan to install a second
ceph
cluster which is SSD based (Samsung PM1725b) which should be
much
faster, however, I still wonder if there is any way to speed
up my
harddisk based cluster?
Thank you in advance for any help,
Best Regards,
Hermann
--
With best regards,
Vitaliy Filippov
--
hermann(a)qwer.tk
PGP/GPG: 299893C7 (on keyservers)
8:31 p.m.
On Wed, Nov 6, 2019 at 5:57 PM Hermann Himmelbauer <hermann(a)qwer.tk> wrote:
Dear Vitaliy, dear Paul,
Changing the block size for "dd" makes a huge difference.
However, still some things are not fully clear to me:
As recommended, I tried writing / reading directly to the rbd and this
is blazingly fast:
fio -ioengine=rbd -name=test -direct=1 -rw=read -bs=4M -iodepth=16
-pool=SATA -rbdname=vm-100-disk-0
write: IOPS=40, BW=160MiB/s (168MB/s)(4096MiB/25529msec)
read: IOPS=135, BW=542MiB/s (568MB/s)(4096MiB/7556msec)
When I do the same within the virtual machine, I get the following results:
fio --filename=/dev/vdb -name=test -direct=1 -rw=write -bs=4M -iodepth=16
--ioengine=aio
Also, VirtIO block devices are usually slower than virtual SCSI disks
on VirtIO SCSI controllers.
Paul
>
> -------- cache = writeback ----------
> Blocksize: 4M:
> read : io=4096.0MB, bw=97640KB/s, iops=23, runt= 42957msec
> write: io=4096.0MB, bw=85250KB/s, iops=20, runt= 49200msec
>
> Blocksize: 4k:
> read : io=4096.0MB, bw=3988.6KB/s, iops=997, runt=1051599msec
> write: io=4096.0MB, bw=14529KB/s, iops=3632, runt=288686msec
> -------------------------------------
>
> The speeds are much slower whereas I don't really know why.
> Nevertheless, this seams also really reasonable, although it's strange
> that there's no difference between "cache = unsafe" and "cache =
> writeback". Moreover, I find it strange that reading with 4k-blocks is
> that slow, while writing is still o.k.
>
> My virtual machine is Debian 8, with a paravirtualized block device
> (/dev/vdb), the process (and qemu parameters) look like the following:
>
> root 1854681 18.3 8.0 5813032 1980804 ? Sl 16:00 20:55
> /usr/bin/kvm -id 100 -name bya-backend -chardev
> socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon
> chardev=qmp,mode=control -chardev
> socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon
> chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/100.pid
> -daemonize -smbios type=1,uuid=e53d6e2d-708e-4511-bb26-b0f1aefd81c6 -smp
> 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot
> menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg
> -vnc unix:/var/run/qemu-server/100.vnc,password -cpu
> kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -device
> pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device
> pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device
> piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device
> usb-tablet,id=tablet,bus=uhci.0,port=1 -device
> VGA,id=vga,bus=pci.0,addr=0x2 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi
> initiator-name=iqn.1993-08.org.debian:01:7c6dc4c7e9f -drive
> if=none,id=drive-ide2,media=cdrom,aio=threads -device
> ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive
>
file=/mnt/pve/pontos-images/images/100/vm-100-disk-2.raw,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on
> -device
> virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100
> -drive
>
file=rbd:SATA/vm-100-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/SATA.keyring,if=none,id=drive-virtio1,cache=writeback,format=raw,aio=threads,detect-zeroes=on
> -device virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb
> -netdev
>
type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown
> -device
> e1000,mac=46:22:36:C3:37:7E,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
> -machine type=pc
>
> In case you have some further speedup-hints for me, I'm glad.
>
> Nevertheless thank you a lot for help!
>
> Best Regards,
> Hermann
>
> Am 05.11.19 um 12:49 schrieb Виталий Филиппов:
> > Yes, cache=unsafe has no effect with RBD. Hm, that's strange, you should
> > get ~40*6 MB/s linear write with 6 HDDs and Bluestore.
> >
> > Try to create a test image and test it with 'fio -ioengine=rbd
> > -name=test -direct=1 -rw=write -bs=4M -iodepth=16 -pool=<pool>
> > -rbdname=<rbd>' from outside a VM.
> >
> > If you still get 4 MB/s, something's wrong with your ceph. If you get
> > adequate performance, something's wrong with your VM settings.
> >
> > 5 ноября 2019 г. 14:31:38 GMT+03:00, Hermann Himmelbauer
> > <hermann(a)qwer.tk> пишет:
> >
> > Hi,
> > Thank you for your quick reply, Proxmox offers me "writeback"
> > (cache=writeback) and "writeback unsafe" (cache=unsafe), however,
for my
> > "dd" test, this makes no difference at all.
> >
> > I still have write speeds of ~ 4,5 MB/s.
> >
> > Perhaps "dd" disables the write cache?
> >
> > Would it perhaps help to put the journal or something else on a SSD?
> >
> > Best Regards,
> > Hermann
> >
> > Am 05.11.19 um 11:49 schrieb vitalif(a)yourcmc.ru:
> >
> > Use `cache=writeback` QEMU option for HDD clusters, that should
> > solve
> > your issue
> >
> > Hi,
> > I recently upgraded my 3-node cluster to proxmox 6 /
> > debian-10 and
> > recreated my ceph cluster with a new release (14.2.4
> > bluestore) -
> > basically hoping to gain some I/O speed.
> >
> > The installation went flawlessly, reading is faster than
> > before (~ 80
> > MB/s), however, the write speed is still really slow (~ 3,5
> > MB/s).
> >
> > I wonder if I can do anything to speed things up?
> >
> > My Hardware is as the following:
> >
> > 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
> > 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
> > interconnected via Infiniband 40
> >
> > The network should be reasonably fast, I measure ~ 16 GBit/s
> > with iperf,
> > so this seems fine.
> >
> > I use ceph for RBD only, so my measurement is simply doing a
> > very simple
> > "dd" read and write test within a virtual machine (Debian
8)
> > like the
> > following:
> >
> > read:
> > dd if=/dev/vdb | pv | dd of=/dev/null
> > -> 80 MB/s
> >
> >
> > write:
> > dd if=/dev/zero | pv | dd of=/dev/vdb
> > -> 3.5 MB/s
> >
> > When I do the same on the virtual machine on a disk that is
> > on a NFS
> > storage, I get something about 30 MB/s for reading and writing.
> >
> > If I disable the write cache on all OSD disks via "hdparm -W 0
> > /dev/sdX", I gain a little bit of performance, write speed
> > is then 4.3
> > MB/s.
> >
> > Thanks to your help from the list I plan to install a second
> > ceph
> > cluster which is SSD based (Samsung PM1725b) which should be
> > much
> > faster, however, I still wonder if there is any way to speed
> > up my
> > harddisk based cluster?
> >
> > Thank you in advance for any help,
> >
> > Best Regards,
> > Hermann
> >
> >
> > --
> > With best regards,
> > Vitaliy Filippov
>
> --
> hermann(a)qwer.tk
> PGP/GPG: 299893C7 (on keyservers)
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5 Nov
5 Nov
3:08 p.m.
Hi,
My three-node lab cluster is similar to yours but with 3x bluestore OSDs
per node (4TB SATA spinning disks) and 1x shared DB/WAL (240GB SATA SSD)
device per node. I'm only using gigabit networking (one interface
public, one interface cluster) also ceph 14.2.4 with 3x replicas.
I would have expected your dd commands to use the cache, try these
instead inside your VM:
# Write test
dd if=/dev/zero of=/zero.file bs=32M oflag=direct status=progress
# Read test
dd if=/zero.file of=/dev/null bs=32M iflag=direct status=progress
You can obviously delete /zero.file when you're finished.
- bs=32M tells dd to read/write 32MB at a time, I think the default is
something like 512 bytes which slows things up significantly without a
cache.
- oflag/iflag=direct will use direct I/O bypassing the cache.
- status=progress is just instead of where you're using pv to show the
transfer rate.
On my cluster I get 124MB/sec read (maxing out the network) and 74MB/sec
write. Without bs=32M I get more like 1MB/sec read and write. The VM I'm
using for this test is cache=writeback and virtio-scsi (i.e. sda rather
than vda).
Simon
On 05/11/2019 11:31, Hermann Himmelbauer wrote:
> Hi,
> Thank you for your quick reply, Proxmox offers me "writeback"
> (cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my
> "dd" test, this makes no difference at all.
>
> I still have write speeds of ~ 4,5 MB/s.
>
> Perhaps "dd" disables the write cache?
>
> Would it perhaps help to put the journal or something else on a SSD?
>
> Best Regards,
> Hermann
3:47 p.m.
Good you are getting 124MB/s via Gbit, I have only been able to get
110MB/s.
If you are interested, I am also having 4TB sata hdd without db/wal on
ssd, 4 nodes, but 10Gbit
[@]# dd if=/dev/zero of=zero.file bs=32M oflag=direct status=progress
3758096384 bytes (3.8 GB) copied, 36.364817 s, 103 MB/s^C
113+0 records in
113+0 records out
3791650816 bytes (3.8 GB) copied, 36.6824 s, 103 MB/s
[@]# dd if=zero.file of=/dev/null bs=32M iflag=direct status=progress
3657433088 bytes (3.7 GB) copied, 15.346591 s, 238 MB/s
113+0 records in
113+0 records out
3791650816 bytes (3.8 GB) copied, 15.8171 s, 240 MB/s
-----Original Message-----
From: Simon Ironside [mailto:sironside@caffetine.org]
Sent: dinsdag 5 november 2019 13:08
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Slow write speed on 3-node cluster with 6*
SATA Harddisks (~ 3.5 MB/s)
Hi,
My three-node lab cluster is similar to yours but with 3x bluestore OSDs
per node (4TB SATA spinning disks) and 1x shared DB/WAL (240GB SATA SSD)
device per node. I'm only using gigabit networking (one interface
public, one interface cluster) also ceph 14.2.4 with 3x replicas.
I would have expected your dd commands to use the cache, try these
instead inside your VM:
# Write test
dd if=/dev/zero of=/zero.file bs=32M oflag=direct status=progress
# Read test
dd if=/zero.file of=/dev/null bs=32M iflag=direct status=progress
You can obviously delete /zero.file when you're finished.
- bs=32M tells dd to read/write 32MB at a time, I think the default is
something like 512 bytes which slows things up significantly without a
cache.
- oflag/iflag=direct will use direct I/O bypassing the cache.
- status=progress is just instead of where you're using pv to show the
transfer rate.
On my cluster I get 124MB/sec read (maxing out the network) and 74MB/sec
write. Without bs=32M I get more like 1MB/sec read and write. The VM I'm
using for this test is cache=writeback and virtio-scsi (i.e. sda rather
than vda).
Simon
On 05/11/2019 11:31, Hermann Himmelbauer wrote:
Hi,
Thank you for your quick reply, Proxmox offers me "writeback"
(cache=writeback) and "writeback unsafe" (cache=unsafe), however, for
my "dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards,
Hermann
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
4:39 p.m.
On Mon, Nov 4, 2019 at 11:44 PM Hermann Himmelbauer <hermann(a)qwer.tk> wrote:
Hi,
I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and
recreated my ceph cluster with a new release (14.2.4 bluestore) -
basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80
MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf,
so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple
"dd" read and write test within a virtual machine (Debian 8) like the
following:
read:
dd if=/dev/vdb | pv | dd of=/dev/null
-> 80 MB/s
write:
dd if=/dev/zero | pv | dd of=/dev/vdb
-> 3.5 MB/s
you are mainly measuring latency, not bandwidth here. Use a larger
block size (bs=4M) to measure bandwidth.
Using /dev/zero as source can also be a bad benchmark if you are
running with detect-zeroes=unmap (which is the proxmox default IIRC)
Paul
>
> When I do the same on the virtual machine on a disk that is on a NFS
> storage, I get something about 30 MB/s for reading and writing.
>
> If I disable the write cache on all OSD disks via "hdparm -W 0
> /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
>
> Thanks to your help from the list I plan to install a second ceph
> cluster which is SSD based (Samsung PM1725b) which should be much
> faster, however, I still wonder if there is any way to speed up my
> harddisk based cluster?
>
> Thank you in advance for any help,
>
> Best Regards,
> Hermann
>
>
> --
> hermann(a)qwer.tk
> PGP/GPG: 299893C7 (on keyservers)
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1657
days inactive
1659
days old
9 comments
7 participants
participants (7)
-
Hermann Himmelbauer -
Marc Roos -
Martin Verges -
Paul Emmerich -
Simon Ironside -
vitalif＠yourcmc.ru -
Виталий Филиппов