Poor Windows performance on ceph RBD.
Dear all, maybe someone can give me a pointer here. We are running OpenNebula with ceph RBD as a back-end store. We have a pool of spinning disks to create large low-demand data disks, mainly for backups and other cold storage. Everything is fine when using linux VMs. However, Windows VMs perform poorly, they are like a factor 20 slower than a similarly created linux VM. If anyone has pointers what to look for, we would be very grateful. The OpenNebula installation is more or less default. The current OS and libvirt versions we use are: Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64 libvirt-client.x86_64 4.5.0-23.el7_7.1 @updates qemu-kvm-ev.x86_64 10:2.12.0-33.1.el7 @centos-qemu-ev Some benchmark results from good to worse workloads: rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 16 : 450MB/s rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 230MB/s rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 190MB/s rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 150MB/s rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type write --io-threads 1 : 26MB/s dd with conv=fdatasync gives awesome 500MB/s inside linux VM for sequential write of 4GB. We copied a couple of large ISO files inside the Windows VM and for the first ca. 1 to 1.5G it performs as expected. Thereafter, however, write speed drops rapidly to ca. 25MB/s and does not recover. It is almost as if Windows translates large sequential writes to small random writes. If anyone has seen and solved this before, please let us know. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
hello I did not do windows vm on kvm since years but back in time for good io performance on windows vm on kvm virtio driver has to be installed. https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virt... oau Le jeudi 02 avril 2020 à 15:28 +0000, Frank Schilder a écrit :
Dear all,
maybe someone can give me a pointer here. We are running OpenNebula with ceph RBD as a back-end store. We have a pool of spinning disks to create large low-demand data disks, mainly for backups and other cold storage. Everything is fine when using linux VMs. However, Windows VMs perform poorly, they are like a factor 20 slower than a similarly created linux VM.
If anyone has pointers what to look for, we would be very grateful.
The OpenNebula installation is more or less default. The current OS and libvirt versions we use are:
Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64 libvirt-client.x86_64 4.5.0- 23.el7_7.1 @updates qemu-kvm-ev.x86_64 10:2.12.0- 33.1.el7 @centos-qemu-ev
Some benchmark results from good to worse workloads:
rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 16 : 450MB/s rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 230MB/s rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 190MB/s rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 150MB/s rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type write --io-threads 1 : 26MB/s
dd with conv=fdatasync gives awesome 500MB/s inside linux VM for sequential write of 4GB.
We copied a couple of large ISO files inside the Windows VM and for the first ca. 1 to 1.5G it performs as expected. Thereafter, however, write speed drops rapidly to ca. 25MB/s and does not recover. It is almost as if Windows translates large sequential writes to small random writes.
If anyone has seen and solved this before, please let us know.
Thanks and best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Olivier, thanks for your answer. We are using the virtio driver already. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Olivier AUDRY <olivier@nmlq.fr> Sent: 02 April 2020 18:20:49 To: Frank Schilder; ceph-users Subject: Re: [ceph-users] Poor Windows performance on ceph RBD. hello I did not do windows vm on kvm since years but back in time for good io performance on windows vm on kvm virtio driver has to be installed. https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virt... oau Le jeudi 02 avril 2020 à 15:28 +0000, Frank Schilder a écrit :
Dear all,
maybe someone can give me a pointer here. We are running OpenNebula with ceph RBD as a back-end store. We have a pool of spinning disks to create large low-demand data disks, mainly for backups and other cold storage. Everything is fine when using linux VMs. However, Windows VMs perform poorly, they are like a factor 20 slower than a similarly created linux VM.
If anyone has pointers what to look for, we would be very grateful.
The OpenNebula installation is more or less default. The current OS and libvirt versions we use are:
Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64 libvirt-client.x86_64 4.5.0- 23.el7_7.1 @updates qemu-kvm-ev.x86_64 10:2.12.0- 33.1.el7 @centos-qemu-ev
Some benchmark results from good to worse workloads:
rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 16 : 450MB/s rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 230MB/s rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 190MB/s rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 150MB/s rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type write --io-threads 1 : 26MB/s
dd with conv=fdatasync gives awesome 500MB/s inside linux VM for sequential write of 4GB.
We copied a couple of large ISO files inside the Windows VM and for the first ca. 1 to 1.5G it performs as expected. Thereafter, however, write speed drops rapidly to ca. 25MB/s and does not recover. It is almost as if Windows translates large sequential writes to small random writes.
If anyone has seen and solved this before, please let us know.
Thanks and best regards,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, we are using same environment, Opennebula + Ceph. Our ceph cluster is composed by 5 ceph OSD Hosts with SSD, spinning 10ktrs and 7.2ktrs, with 10Gb/s fiber network Each spinning OSD are associated with a db and wall devices on SSD Nearly all our Windows VM RBD images are in a 10k/trs pool with erasure coding. For the moment we are house about 15 VM (RDS and exchange) What we are noting : - VM are far from respondig as well as on our old 10k SAN ( less than 30%) - RBD average Latency is oscillating between 50ms to 250ms with some peaks that can reach the second - some tests (crystal test drive) from inside the VM can show performance up to 700MB/s on read and 170 MB/s on write, but a single file copy barely reach 150 MB/s and stay at a poor 25 MB/s most of the time - test on 4K rnd, show some iops performance up to 4K iops read and 2kiops write, but view from RDB point of view, it's like the image iops cant barely go over 500 iops(read+write) Since we have to migrate our VM from the old SAN to Ceph, I am really worried, there is mode than 150 VMs on it, and our Ceph seems to have hard time to cope with 15 VMs. I can't find accurate date and relevant calculus templates that should permit me to evualate what I can expect All the documents I've read (and I read a lot ;) ) only reports empirical ascertainment with "it's better", or "it's worst". There is a lot of parameters we can tweaks like block size, striping, stripe size, strip count, ... but those are poorly documented, especially the relation between them. I will be more than happy to work with some peoples who are in the same situation to try to find some solutions, methods which can help us to be sure of our design. And break the "make the cluster, tweak it, and maybe it will be fine for you". I feel that each of us ( as I read in forums and mailing list) are a bit lonesome. Google is a real friend, but if feel he reached its limits ;) Maybe my call will reach some volontee. Best regards JC Passard CTO Provectio France
Hi, I think you are hit by two different problems at the same time. The second problem might be the same that we also experience, namely that Windows VMs have very strange performance characteristics with libvirt, vd driver and RBD. With copy operations on very large files (>2GB) we see a sharp drop of bandwidth after ca. 1 to 1.5GB to a measly 25MB/s for as yet unknown reasons. We cannot reproduce this behaviour with Linux VMs, so chances that this is a Windows and not a ceph problem are rather high. The first problem, however, has to do with how ceph uses disks. Bare spinning disks have very poor performance characteristics and a lot of development since their invention has been on smart controllers (internal and external) with volatile and persistent caches and OS file buffers that attempt to translate usual user's workloads into something that works reasonable well with spinning drives. The main ideas being to re-order and merge I/O, cache hot data and absorb I/O bursts for constant write back. The SANs you are used to are almost certainly high-end products with all the magic money can currently afford. Ceph forcefully bypasses all of such logic and a rule of thumb I'm following is that with ceph and current hardware, using current generation drives will provide previous generation's drive performance. With NVMes you can achieve SSD performance, with SSDs you get good spinning SAS drive performance and with SAS drives you get, well, floppy or zip drive performance. I'm afraid that's what you are seeing with 15VMs saturating the available aggregated performance of the spindles. If you want to stick with spindles as a data store, what you need is fast, reliable persistent cache. Reliable here means that the firmware is free of bugs with respect to power outages, which is quite a requirement in itself. Some expensive disk controllers claim to have that, they offer persistent NVMe cache. How much you want to trust the firmware is a different story. Alternatively, you could consider a few TB NVMe drives for a ceph cache pool. People report that they are happy with that. As long as the cache pool can hold all hot data plus write bursts, I would also expect this to work fine. Instead of caching we decided to go for a split. We use datacenter grade low-cost SSDs for a small all-flash pool for OS RBD disks and a large HDD-only pool for data storage. This works quite well since the major annoying simultaneous I/O workload of Windows VMs happens on the OS disks. For ordinary data access, an EC HDD pool is perfectly fine and we provision machines with a second large data disk on HDD. Our users are quite happy with that model. In any case, we are still stuck with the strange performance drop with Windows machines that you also seem to observe and are still looking for help with that. If you manage to figure out what is going on, I would like to hear about that. So far, we haven't found a clue. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: jcharles@provectio.fr <jcharles@provectio.fr> Sent: 11 June 2020 12:38:32 To: ceph-users@ceph.io Subject: [ceph-users] Re: Poor Windows performance on ceph RBD. Hello, we are using same environment, Opennebula + Ceph. Our ceph cluster is composed by 5 ceph OSD Hosts with SSD, spinning 10ktrs and 7.2ktrs, with 10Gb/s fiber network Each spinning OSD are associated with a db and wall devices on SSD Nearly all our Windows VM RBD images are in a 10k/trs pool with erasure coding. For the moment we are house about 15 VM (RDS and exchange) What we are noting : - VM are far from respondig as well as on our old 10k SAN ( less than 30%) - RBD average Latency is oscillating between 50ms to 250ms with some peaks that can reach the second - some tests (crystal test drive) from inside the VM can show performance up to 700MB/s on read and 170 MB/s on write, but a single file copy barely reach 150 MB/s and stay at a poor 25 MB/s most of the time - test on 4K rnd, show some iops performance up to 4K iops read and 2kiops write, but view from RDB point of view, it's like the image iops cant barely go over 500 iops(read+write) Since we have to migrate our VM from the old SAN to Ceph, I am really worried, there is mode than 150 VMs on it, and our Ceph seems to have hard time to cope with 15 VMs. I can't find accurate date and relevant calculus templates that should permit me to evualate what I can expect All the documents I've read (and I read a lot ;) ) only reports empirical ascertainment with "it's better", or "it's worst". There is a lot of parameters we can tweaks like block size, striping, stripe size, strip count, ... but those are poorly documented, especially the relation between them. I will be more than happy to work with some peoples who are in the same situation to try to find some solutions, methods which can help us to be sure of our design. And break the "make the cluster, tweak it, and maybe it will be fine for you". I feel that each of us ( as I read in forums and mailing list) are a bit lonesome. Google is a real friend, but if feel he reached its limits ;) Maybe my call will reach some volontee. Best regards JC Passard CTO Provectio France _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, really thanks for your answer. I will try to investigate to the 2nd problems, and I will keep you informed if I find something. For the first one, I am experimenting cache pool as you pointed me to, I never tried it because the ceph documentation doesn't really encourage this with rbd. For the moment the results seem to be quite ok. I am facing now with the 1$ question : "how to monitor the cache and rbd image inside it ?". I can't quite find some documentation on metrics associated with it, and now my metric with those image are zero flat ;) . I will keep searching. Bye the way, thanks for your preceding answer. Best Regards
To anyone who is following this thread, we found a possible explanation for (some of) our observations. We are running Windows servers version 2016 and 2019 as storage servers exporting data on an rbd image/disk. We recently found that Windows server 2016 runs fine. It is still not as fast as Linux + SAMBA share on an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth. With Windows server 2019, however, we observe near-complete stall of file transfers and time-outs using standard copy tools (robocopy). We don't have an explanation yet and are downgrading Windows servers where possible. If anyone has a hint what we can do, please let us know. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
To anyone who is following this thread, we found a possible explanation for (some of) our observations.
If someone is following this, they probably want the possible explanation and not the knowledge of you having the possible explanation. So you are saying if you do eg. a core installation (without gui) of 2016/2019 disable all services. The fio test results are signficantly different to eg. a centos 7 vm doing the same fio test? Are you sure this is not related to other processes writing to disk? -----Original Message----- From: Frank Schilder [mailto:frans@dtu.dk] Sent: maandag 13 juli 2020 9:28 To: ceph-users@ceph.io Subject: [ceph-users] Re: Poor Windows performance on ceph RBD. To anyone who is following this thread, we found a possible explanation for (some of) our observations. We are running Windows servers version 2016 and 2019 as storage servers exporting data on an rbd image/disk. We recently found that Windows server 2016 runs fine. It is still not as fast as Linux + SAMBA share on an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth. With Windows server 2019, however, we observe near-complete stall of file transfers and time-outs using standard copy tools (robocopy). We don't have an explanation yet and are downgrading Windows servers where possible. If anyone has a hint what we can do, please let us know. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
To anyone who is following this thread, we found a possible explanation for (some of) our observations.
If someone is following this, they probably want the possible explanation and not the knowledge of you having the possible explanation.
So you are saying if you do eg. a core installation (without gui) of 2016/2019 disable all services. The fio test results are signficantly different to eg. a centos 7 vm doing the same fio test? Are you sure this is not related to other processes writing to disk?
Right, its not an explanation but rather a further observation. We don't really have an explanation yet. Its an identical installation of both server versions, same services configured. Our operators are not really into debugging Windows, that's why we were asking here. Their hypothesis is, that the VD driver for accessing RBD images has problems with Windows servers newer than 2016. I'm not a Windows guy, so can't really comment on this. The test we do is a simple copy-test of a single 10g file and we monitor the transfer speed. This info was cut out of this e-mail, the original report for reference is: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ANHJQZLJT47... . We are very sure that it is not related to other processes writing to disk, we monitor that too. There is also no competition on the RBD pool at the time of testing. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc Roos <M.Roos@f1-outsourcing.eu> Sent: 13 July 2020 10:24 To: ceph-users; Frank Schilder Subject: RE: [ceph-users] Re: Poor Windows performance on ceph RBD.
To anyone who is following this thread, we found a possible explanation for (some of) our observations.
If someone is following this, they probably want the possible explanation and not the knowledge of you having the possible explanation. So you are saying if you do eg. a core installation (without gui) of 2016/2019 disable all services. The fio test results are signficantly different to eg. a centos 7 vm doing the same fio test? Are you sure this is not related to other processes writing to disk? -----Original Message----- From: Frank Schilder [mailto:frans@dtu.dk] Sent: maandag 13 juli 2020 9:28 To: ceph-users@ceph.io Subject: [ceph-users] Re: Poor Windows performance on ceph RBD. To anyone who is following this thread, we found a possible explanation for (some of) our observations. We are running Windows servers version 2016 and 2019 as storage servers exporting data on an rbd image/disk. We recently found that Windows server 2016 runs fine. It is still not as fast as Linux + SAMBA share on an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth. With Windows server 2019, however, we observe near-complete stall of file transfers and time-outs using standard copy tools (robocopy). We don't have an explanation yet and are downgrading Windows servers where possible. If anyone has a hint what we can do, please let us know. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 13/07/2020 10:43, Frank Schilder wrote:
To anyone who is following this thread, we found a possible explanation for (some of) our observations. If someone is following this, they probably want the possible explanation and not the knowledge of you having the possible explanation.
So you are saying if you do eg. a core installation (without gui) of 2016/2019 disable all services. The fio test results are signficantly different to eg. a centos 7 vm doing the same fio test? Are you sure this is not related to other processes writing to disk? Right, its not an explanation but rather a further observation. We don't really have an explanation yet.
Its an identical installation of both server versions, same services configured. Our operators are not really into debugging Windows, that's why we were asking here. Their hypothesis is, that the VD driver for accessing RBD images has problems with Windows servers newer than 2016. I'm not a Windows guy, so can't really comment on this.
The test we do is a simple copy-test of a single 10g file and we monitor the transfer speed. This info was cut out of this e-mail, the original report for reference is: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ANHJQZLJT47... .
We are very sure that it is not related to other processes writing to disk, we monitor that too. There is also no competition on the RBD pool at the time of testing.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Marc Roos <M.Roos@f1-outsourcing.eu> Sent: 13 July 2020 10:24 To: ceph-users; Frank Schilder Subject: RE: [ceph-users] Re: Poor Windows performance on ceph RBD.
To anyone who is following this thread, we found a possible explanation for (some of) our observations. If someone is following this, they probably want the possible explanation and not the knowledge of you having the possible explanation.
So you are saying if you do eg. a core installation (without gui) of 2016/2019 disable all services. The fio test results are signficantly different to eg. a centos 7 vm doing the same fio test? Are you sure this is not related to other processes writing to disk?
-----Original Message----- From: Frank Schilder [mailto:frans@dtu.dk] Sent: maandag 13 juli 2020 9:28 To: ceph-users@ceph.io Subject: [ceph-users] Re: Poor Windows performance on ceph RBD.
To anyone who is following this thread, we found a possible explanation for (some of) our observations.
We are running Windows servers version 2016 and 2019 as storage servers exporting data on an rbd image/disk. We recently found that Windows server 2016 runs fine. It is still not as fast as Linux + SAMBA share on an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth. With Windows server 2019, however, we observe near-complete stall of file transfers and time-outs using standard copy tools (robocopy). We don't have an explanation yet and are downgrading Windows servers where possible.
If anyone has a hint what we can do, please let us know.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I am not sure exactly how you are testing the speed on Windows, but 2 possible factors are block size and caching. Block size depends on the client application, so a Windows file copy from ui will have a 512k block size which is different than xcopy or robocopy, the later can change block size depending on flags / restart mode..etc. Similarly the dd command on Linux will give different speed depending on block size. Caching: Caching will make a big difference for sequential writes as it merges smaller blocks, but in some cases it is not obvious if caching is being used or not since it could be at different layers, for example in your Linux Samba export test, there could be caching done at the gateway, a clustered setup with high availability may explicitly turn caching off. What you report as initial high speed then decrease could be indicative of initially writing to a cache buffer then slowing when it fills. It will help to quantify/compare latency (iops qd=1) via: on Linux: rbd bench --io-type write POOL_NAME/IMAGE_NAME --io-threads=1 --io-size 4K --io-pattern rand --rbd_cache=false fio --name=xx --filename=FILE_NAME --iodepth=1 --rw=randwrite --bs=4k --direct=1 --runtime=30 --time_based On Windows vm: diskspd -b4k -d30 -o1 -t1 -r -Su -w100 -c1G FILE_NAME Measure/compare sequential writes with 512k block size on Linux: rbd bench --io-type write POOL_NAME/IMAGE_NAME --io-threads=1 --io-size 512K --io-pattern seq --rbd_cache=false fio --name=xx --filename=FILE_NAME --iodepth=1 --rw=write --bs=512k --direct=1 --runtime=30 --time_based On Windows vm: diskspd -b512k -d30 -o1 -t1 -Su -w100 -c1G FILE_NAME /Maged
participants (5)
-
Frank Schilder
-
jcharles＠provectio.fr
-
Maged Mokhtar
-
Marc Roos
-
Olivier AUDRY