Hi,
Thank you for your quick reply, Proxmox offers me "writeback"
(cache=writeback) and "writeback unsafe" (cache=unsafe), however, for my
"dd" test, this makes no difference at all.

I still have write speeds of ~ 4,5 MB/s.

Perhaps "dd" disables the write cache?

Would it perhaps help to put the journal or something else on a SSD?

Best Regards,
Hermann

Am 05.11.19 um 11:49 schrieb vitalif@yourcmc.ru:
Use `cache=writeback` QEMU option for HDD clusters, that should solve
your issue

 Hi,
 I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and
 recreated my ceph cluster with a new release (14.2.4 bluestore) -
 basically hoping to gain some I/O speed.

 The installation went flawlessly, reading is faster than before (~ 80
 MB/s), however, the write speed is still really slow (~ 3,5 MB/s).

 I wonder if I can do anything to speed things up?

 My Hardware is as the following:

 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
 interconnected via Infiniband 40

 The network should be reasonably fast, I measure ~ 16 GBit/s with iperf,
 so this seems fine.

 I use ceph for RBD only, so my measurement is simply doing a very simple
 "dd" read and write test within a virtual machine (Debian 8) like the
 following:

 read:
 dd if=/dev/vdb | pv | dd of=/dev/null
 -> 80 MB/s

 write:
 dd if=/dev/zero | pv | dd of=/dev/vdb
 -> 3.5 MB/s

 When I do the same on the virtual machine on a disk that is on a NFS
 storage, I get something about 30 MB/s for reading and writing.

 If I disable the write cache on all OSD disks via "hdparm -W 0
 /dev/sdX", I gain a little bit of performance, write speed is then 4.3
 MB/s.

 Thanks to your help from the list I plan to install a second ceph
 cluster which is SSD based (Samsung PM1725b) which should be much
 faster, however, I still wonder if there is any way to speed up my
 harddisk based cluster?

 Thank you in advance for any help,

 Best Regards,
 Hermann