Yes, cache=unsafe has no effect with RBD. Hm, that's strange, you should get ~40*6 MB/s linear write with 6 HDDs and Bluestore. Try to create a test image and test it with 'fio -ioengine=rbd -name=test -direct=1 -rw=write -bs=4M -iodepth=16 -pool=<pool> -rbdname=<rbd>' from outside a VM. If you still get 4 MB/s, something's wrong with your ceph. If you get adequate performance, something's wrong with your VM settings. 5 ноября 2019 г. 14:31:38 GMT+03:00, Hermann Himmelbauer <hermann@qwer.tk> пишет:
Hi, Thank you for your quick reply, Proxmox offers me "writeback" (cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my "dd" test, this makes no difference at all.
I still have write speeds of ~ 4,5 MB/s.
Perhaps "dd" disables the write cache?
Would it perhaps help to put the journal or something else on a SSD?
Best Regards, Hermann
Use `cache=writeback` QEMU option for HDD clusters, that should solve your issue
Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed.
The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).
I wonder if I can do anything to speed things up?
My Hardware is as the following:
3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40
The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine.
I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like
Am 05.11.19 um 11:49 schrieb vitalif@yourcmc.ru: the
following:
read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s
write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s
When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing.
If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.
Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster?
Thank you in advance for any help,
Best Regards, Hermann
-- hermann@qwer.tk PGP/GPG: 299893C7 (on keyservers)
-- With best regards, Vitaliy Filippov