2 Apr
2020
2 Apr
'20
8:28 a.m.
Dear all,
maybe someone can give me a pointer here. We are running OpenNebula with ceph RBD as a
back-end store. We have a pool of spinning disks to create large low-demand data disks,
mainly for backups and other cold storage. Everything is fine when using linux VMs.
However, Windows VMs perform poorly, they are like a factor 20 slower than a similarly
created linux VM.
If anyone has pointers what to look for, we would be very grateful.
The OpenNebula installation is more or less default. The current OS and libvirt versions
we use are:
Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64
libvirt-client.x86_64 4.5.0-23.el7_7.1 @updates
qemu-kvm-ev.x86_64 10:2.12.0-33.1.el7 @centos-qemu-ev
Some benchmark results from good to worse workloads:
rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 16 :
450MB/s
rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 1 :
230MB/s
rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write --io-threads 1 :
190MB/s
rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type write --io-threads 1 :
150MB/s
rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type write --io-threads 1 :
26MB/s
dd with conv=fdatasync gives awesome 500MB/s inside linux VM for sequential write of 4GB.
We copied a couple of large ISO files inside the Windows VM and for the first ca. 1 to
1.5G it performs as expected. Thereafter, however, write speed drops rapidly to ca. 25MB/s
and does not recover. It is almost as if Windows translates large sequential writes to
small random writes.
If anyone has seen and solved this before, please let us know.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
2 Apr
2 Apr
9:20 a.m.
hello
I did not do windows vm on kvm since years but back in time for good io
performance on windows vm on kvm virtio driver has to be installed.
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-vir…
oau
Le jeudi 02 avril 2020 à 15:28 +0000, Frank Schilder a écrit :
Dear all,
maybe someone can give me a pointer here. We are running OpenNebula
with ceph RBD as a back-end store. We have a pool of spinning disks
to create large low-demand data disks, mainly for backups and other
cold storage. Everything is fine when using linux VMs. However,
Windows VMs perform poorly, they are like a factor 20 slower than a
similarly created linux VM.
If anyone has pointers what to look for, we would be very grateful.
The OpenNebula installation is more or less default. The current OS
and libvirt versions we use are:
Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64
libvirt-client.x86_64 4.5.0-
23.el7_7.1 @updates
qemu-kvm-ev.x86_64 10:2.12.0-
33.1.el7 @centos-qemu-ev
Some benchmark results from good to worse workloads:
rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write
--io-threads 16 : 450MB/s
rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write
--io-threads 1 : 230MB/s
rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write
--io-threads 1 : 190MB/s
rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type
write --io-threads 1 : 150MB/s
rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type
write --io-threads 1 : 26MB/s
dd with conv=fdatasync gives awesome 500MB/s inside linux VM for
sequential write of 4GB.
We copied a couple of large ISO files inside the Windows VM and for
the first ca. 1 to 1.5G it performs as expected. Thereafter, however,
write speed drops rapidly to ca. 25MB/s and does not recover. It is
almost as if Windows translates large sequential writes to small
random writes.
If anyone has seen and solved this before, please let us know.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3 Apr
3 Apr
midnight
Dear Olivier,
thanks for your answer. We are using the virtio driver already.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Olivier AUDRY <olivier(a)nmlq.fr>
Sent: 02 April 2020 18:20:49
To: Frank Schilder; ceph-users
Subject: Re: [ceph-users] Poor Windows performance on ceph RBD.
hello
I did not do windows vm on kvm since years but back in time for good io
performance on windows vm on kvm virtio driver has to be installed.
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-vir…
oau
Le jeudi 02 avril 2020 à 15:28 +0000, Frank Schilder a écrit :
Dear all,
maybe someone can give me a pointer here. We are running OpenNebula
with ceph RBD as a back-end store. We have a pool of spinning disks
to create large low-demand data disks, mainly for backups and other
cold storage. Everything is fine when using linux VMs. However,
Windows VMs perform poorly, they are like a factor 20 slower than a
similarly created linux VM.
If anyone has pointers what to look for, we would be very grateful.
The OpenNebula installation is more or less default. The current OS
and libvirt versions we use are:
Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64
libvirt-client.x86_64 4.5.0-
23.el7_7.1 @updates
qemu-kvm-ev.x86_64 10:2.12.0-
33.1.el7 @centos-qemu-ev
Some benchmark results from good to worse workloads:
rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write
--io-threads 16 : 450MB/s
rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write
--io-threads 1 : 230MB/s
rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write
--io-threads 1 : 190MB/s
rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type
write --io-threads 1 : 150MB/s
rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type
write --io-threads 1 : 26MB/s
dd with conv=fdatasync gives awesome 500MB/s inside linux VM for
sequential write of 4GB.
We copied a couple of large ISO files inside the Windows VM and for
the first ca. 1 to 1.5G it performs as expected. Thereafter, however,
write speed drops rapidly to ca. 25MB/s and does not recover. It is
almost as if Windows translates large sequential writes to small
random writes.
If anyone has seen and solved this before, please let us know.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11 Jun
11 Jun
3:38 a.m.
Hello,
we are using same environment, Opennebula + Ceph.
Our ceph cluster is composed by 5 ceph OSD Hosts with SSD, spinning 10ktrs and 7.2ktrs,
with 10Gb/s fiber network
Each spinning OSD are associated with a db and wall devices on SSD
Nearly all our Windows VM RBD images are in a 10k/trs pool with erasure coding.
For the moment we are house about 15 VM (RDS and exchange)
What we are noting :
- VM are far from respondig as well as on our old 10k SAN ( less than 30%)
- RBD average Latency is oscillating between 50ms to 250ms with some peaks that can
reach the second
- some tests (crystal test drive) from inside the VM can show performance up to 700MB/s
on read and 170 MB/s on write, but a single file copy barely reach 150 MB/s and stay at a
poor 25 MB/s most of the time
- test on 4K rnd, show some iops performance up to 4K iops read and 2kiops write, but
view from RDB point of view, it's like the image iops cant barely go over 500
iops(read+write)
Since we have to migrate our VM from the old SAN to Ceph, I am really worried, there is
mode than 150 VMs on it, and our Ceph seems to have hard time to cope with 15 VMs.
I can't find accurate date and relevant calculus templates that should permit me to
evualate what I can expect
All the documents I've read (and I read a lot ;) ) only reports empirical
ascertainment with "it's better", or "it's worst".
There is a lot of parameters we can tweaks like block size, striping, stripe size, strip
count, ... but those are poorly documented, especially the relation between them.
I will be more than happy to work with some peoples who are in the same situation to try
to find some solutions, methods which can help us to be sure of our design. And break the
"make the cluster, tweak it, and maybe it will be fine for you". I feel that
each of us ( as I read in forums and mailing list) are a bit lonesome. Google is a real
friend, but if feel he reached its limits ;)
Maybe my call will reach some volontee.
Best regards
JC Passard
CTO Provectio
France
12 Jun
12 Jun
1:44 a.m.
Hi,
I think you are hit by two different problems at the same time. The second problem might
be the same that we also experience, namely that Windows VMs have very strange performance
characteristics with libvirt, vd driver and RBD. With copy operations on very large files
(>2GB) we see a sharp drop of bandwidth after ca. 1 to 1.5GB to a measly 25MB/s for as
yet unknown reasons. We cannot reproduce this behaviour with Linux VMs, so chances that
this is a Windows and not a ceph problem are rather high.
The first problem, however, has to do with how ceph uses disks. Bare spinning disks have
very poor performance characteristics and a lot of development since their invention has
been on smart controllers (internal and external) with volatile and persistent caches and
OS file buffers that attempt to translate usual user's workloads into something that
works reasonable well with spinning drives. The main ideas being to re-order and merge
I/O, cache hot data and absorb I/O bursts for constant write back. The SANs you are used
to are almost certainly high-end products with all the magic money can currently afford.
Ceph forcefully bypasses all of such logic and a rule of thumb I'm following is that
with ceph and current hardware, using current generation drives will provide previous
generation's drive performance. With NVMes you can achieve SSD performance, with SSDs
you get good spinning SAS drive performance and with SAS drives you get, well, floppy or
zip drive performance. I'm afraid that's what you are seeing with 15VMs saturating
the available aggregated performance of the spindles.
If you want to stick with spindles as a data store, what you need is fast, reliable
persistent cache. Reliable here means that the firmware is free of bugs with respect to
power outages, which is quite a requirement in itself. Some expensive disk controllers
claim to have that, they offer persistent NVMe cache. How much you want to trust the
firmware is a different story. Alternatively, you could consider a few TB NVMe drives for
a ceph cache pool. People report that they are happy with that. As long as the cache pool
can hold all hot data plus write bursts, I would also expect this to work fine.
Instead of caching we decided to go for a split. We use datacenter grade low-cost SSDs for
a small all-flash pool for OS RBD disks and a large HDD-only pool for data storage. This
works quite well since the major annoying simultaneous I/O workload of Windows VMs happens
on the OS disks. For ordinary data access, an EC HDD pool is perfectly fine and we
provision machines with a second large data disk on HDD. Our users are quite happy with
that model.
In any case, we are still stuck with the strange performance drop with Windows machines
that you also seem to observe and are still looking for help with that. If you manage to
figure out what is going on, I would like to hear about that. So far, we haven't found
a clue.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: jcharles(a)provectio.fr <jcharles(a)provectio.fr>
Sent: 11 June 2020 12:38:32
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Poor Windows performance on ceph RBD.
Hello,
we are using same environment, Opennebula + Ceph.
Our ceph cluster is composed by 5 ceph OSD Hosts with SSD, spinning 10ktrs and 7.2ktrs,
with 10Gb/s fiber network
Each spinning OSD are associated with a db and wall devices on SSD
Nearly all our Windows VM RBD images are in a 10k/trs pool with erasure coding.
For the moment we are house about 15 VM (RDS and exchange)
What we are noting :
- VM are far from respondig as well as on our old 10k SAN ( less than 30%)
- RBD average Latency is oscillating between 50ms to 250ms with some peaks that can
reach the second
- some tests (crystal test drive) from inside the VM can show performance up to 700MB/s
on read and 170 MB/s on write, but a single file copy barely reach 150 MB/s and stay at a
poor 25 MB/s most of the time
- test on 4K rnd, show some iops performance up to 4K iops read and 2kiops write, but
view from RDB point of view, it's like the image iops cant barely go over 500
iops(read+write)
Since we have to migrate our VM from the old SAN to Ceph, I am really worried, there is
mode than 150 VMs on it, and our Ceph seems to have hard time to cope with 15 VMs.
I can't find accurate date and relevant calculus templates that should permit me to
evualate what I can expect
All the documents I've read (and I read a lot ;) ) only reports empirical
ascertainment with "it's better", or "it's worst".
There is a lot of parameters we can tweaks like block size, striping, stripe size, strip
count, ... but those are poorly documented, especially the relation between them.
I will be more than happy to work with some peoples who are in the same situation to try
to find some solutions, methods which can help us to be sure of our design. And break the
"make the cluster, tweak it, and maybe it will be fine for you". I feel that
each of us ( as I read in forums and mailing list) are a bit lonesome. Google is a real
friend, but if feel he reached its limits ;)
Maybe my call will reach some volontee.
Best regards
JC Passard
CTO Provectio
France
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
16 Jun
16 Jun
6:07 a.m.
Hello,
really thanks for your answer. I will try to investigate to the 2nd problems, and I will
keep you informed if I find something.
For the first one, I am experimenting cache pool as you pointed me to, I never tried it
because the ceph documentation doesn't really encourage this with rbd.
For the moment the results seem to be quite ok. I am facing now with the 1$ question :
"how to monitor the cache and rbd image inside it ?".
I can't quite find some documentation on metrics associated with it, and now my metric
with those image are zero flat ;) . I will keep searching.
Bye the way, thanks for your preceding answer.
Best Regards
13 Jul
13 Jul
12:27 a.m.
To anyone who is following this thread, we found a possible explanation for (some of) our
observations.
We are running Windows servers version 2016 and 2019 as storage servers exporting data on
an rbd image/disk. We recently found that Windows server 2016 runs fine. It is still not
as fast as Linux + SAMBA share on an rbd image (ca. 50%), but runs with a reasonable
sustained bandwidth. With Windows server 2019, however, we observe near-complete stall of
file transfers and time-outs using standard copy tools (robocopy). We don't have an
explanation yet and are downgrading Windows servers where possible.
If anyone has a hint what we can do, please let us know.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
1:24 a.m.
> To anyone who is following this thread, we found a
possible
explanation for
> (some of) our observations.
If someone is following this, they probably want the possible
explanation and not the knowledge of you having the possible
explanation.
So you are saying if you do eg. a core installation (without gui) of
2016/2019 disable all services. The fio test results are signficantly
different to eg. a centos 7 vm doing the same fio test? Are you sure
this is not related to other processes writing to disk?
-----Original Message-----
From: Frank Schilder [mailto:frans@dtu.dk]
Sent: maandag 13 juli 2020 9:28
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Poor Windows performance on ceph RBD.
To anyone who is following this thread, we found a possible explanation
for (some of) our observations.
We are running Windows servers version 2016 and 2019 as storage servers
exporting data on an rbd image/disk. We recently found that Windows
server 2016 runs fine. It is still not as fast as Linux + SAMBA share on
an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth.
With Windows server 2019, however, we observe near-complete stall of
file transfers and time-outs using standard copy tools (robocopy). We
don't have an explanation yet and are downgrading Windows servers where
possible.
If anyone has a hint what we can do, please let us know.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
1:43 a.m.
> To anyone who is following this thread, we found
a possible explanation for
> (some of) our observations.
If someone is following this, they probably want the
possible
explanation and not the knowledge of you having the possible
explanation.
So you are saying if you do eg. a core installation
(without gui) of
2016/2019 disable all services. The fio test results are signficantly
different to eg. a centos 7 vm doing the same fio test? Are you sure
this is not related to other processes writing to disk?
Right, its not an explanation but rather a further observation. We don't really have
an explanation yet.
Its an identical installation of both server versions, same services configured. Our
operators are not really into debugging Windows, that's why we were asking here. Their
hypothesis is, that the VD driver for accessing RBD images has problems with Windows
servers newer than 2016. I'm not a Windows guy, so can't really comment on this.
The test we do is a simple copy-test of a single 10g file and we monitor the transfer
speed. This info was cut out of this e-mail, the original report for reference is:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ANHJQZLJT4…
.
We are very sure that it is not related to other processes writing to disk, we monitor
that too. There is also no competition on the RBD pool at the time of testing.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Marc Roos <M.Roos(a)f1-outsourcing.eu>
Sent: 13 July 2020 10:24
To: ceph-users; Frank Schilder
Subject: RE: [ceph-users] Re: Poor Windows performance on ceph RBD.
> To anyone who is following this thread, we found a
possible
explanation for
> (some of) our observations.
If someone is following this, they probably want the possible
explanation and not the knowledge of you having the possible
explanation.
So you are saying if you do eg. a core installation (without gui) of
2016/2019 disable all services. The fio test results are signficantly
different to eg. a centos 7 vm doing the same fio test? Are you sure
this is not related to other processes writing to disk?
-----Original Message-----
From: Frank Schilder [mailto:frans@dtu.dk]
Sent: maandag 13 juli 2020 9:28
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Poor Windows performance on ceph RBD.
To anyone who is following this thread, we found a possible explanation
for (some of) our observations.
We are running Windows servers version 2016 and 2019 as storage servers
exporting data on an rbd image/disk. We recently found that Windows
server 2016 runs fine. It is still not as fast as Linux + SAMBA share on
an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth.
With Windows server 2019, however, we observe near-complete stall of
file transfers and time-outs using standard copy tools (robocopy). We
don't have an explanation yet and are downgrading Windows servers where
possible.
If anyone has a hint what we can do, please let us know.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
3:34 a.m.
On 13/07/2020 10:43, Frank Schilder wrote:
I am not sure exactly how you are testing the speed on Windows, but 2
possible factors are block size and caching.
Block size depends on the client application, so a Windows file copy
from ui will have a 512k block size which is different than xcopy or
robocopy, the later can change block size depending on flags / restart
mode..etc. Similarly the dd command on Linux will give different speed
depending on block size.
Caching: Caching will make a big difference for sequential writes as it
merges smaller blocks, but in some cases it is not obvious if caching is
being used or not since it could be at different layers, for example in
your Linux Samba export test, there could be caching done at the
gateway, a clustered setup with high availability may explicitly turn
caching off. What you report as initial high speed then decrease could
be indicative of initially writing to a cache buffer then slowing when
it fills.
It will help to quantify/compare latency (iops qd=1) via:
on Linux:
rbd bench --io-type write POOL_NAME/IMAGE_NAME --io-threads=1 --io-size
4K --io-pattern rand --rbd_cache=false
fio --name=xx --filename=FILE_NAME --iodepth=1 --rw=randwrite --bs=4k
--direct=1 --runtime=30 --time_based
On Windows vm:
diskspd -b4k -d30 -o1 -t1 -r -Su -w100 -c1G FILE_NAME
Measure/compare sequential writes with 512k block size
on Linux:
rbd bench --io-type write POOL_NAME/IMAGE_NAME --io-threads=1 --io-size
512K --io-pattern seq --rbd_cache=false
fio --name=xx --filename=FILE_NAME --iodepth=1 --rw=write --bs=512k
--direct=1 --runtime=30 --time_based
On Windows vm:
diskspd -b512k -d30 -o1 -t1 -Su -w100 -c1G FILE_NAME
/Maged
To anyone who is following this thread, we found a
possible explanation for
(some of) our observations.
If someone is following this, they probably want the
possible
explanation and not the knowledge of you having the possible
explanation. So you are saying if you do eg. a core
installation (without gui) of
2016/2019 disable all services. The fio test results are signficantly
different to eg. a centos 7 vm doing the same fio test? Are you sure
this is not related to other processes writing to disk?
Right, its not an
explanation but rather a further observation. We don't really have an explanation
yet.
Its an identical installation of both server versions, same services configured. Our
operators are not really into debugging Windows, that's why we were asking here. Their
hypothesis is, that the VD driver for accessing RBD images has problems with Windows
servers newer than 2016. I'm not a Windows guy, so can't really comment on this.
The test we do is a simple copy-test of a single 10g file and we monitor the transfer
speed. This info was cut out of this e-mail, the original report for reference is:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ANHJQZLJT4…
.
We are very sure that it is not related to other processes writing to disk, we monitor
that too. There is also no competition on the RBD pool at the time of testing.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Marc Roos <M.Roos(a)f1-outsourcing.eu>
Sent: 13 July 2020 10:24
To: ceph-users; Frank Schilder
Subject: RE: [ceph-users] Re: Poor Windows performance on ceph RBD.
> To anyone who is following this thread, we
found a possible
explanation for
> (some of) our observations.
If someone
is following this, they probably want the possible
explanation and not the knowledge of you having the possible
explanation.
So you are saying if you do eg. a core installation (without gui) of
2016/2019 disable all services. The fio test results are signficantly
different to eg. a centos 7 vm doing the same fio test? Are you sure
this is not related to other processes writing to disk?
-----Original Message-----
From: Frank Schilder [mailto:frans@dtu.dk]
Sent: maandag 13 juli 2020 9:28
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Poor Windows performance on ceph RBD.
To anyone who is following this thread, we found a possible explanation
for (some of) our observations.
We are running Windows servers version 2016 and 2019 as storage servers
exporting data on an rbd image/disk. We recently found that Windows
server 2016 runs fine. It is still not as fast as Linux + SAMBA share on
an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth.
With Windows server 2019, however, we observe near-complete stall of
file transfers and time-outs using standard copy tools (robocopy). We
don't have an explanation yet and are downgrading Windows servers where
possible.
If anyone has a hint what we can do, please let us know.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1381
days inactive
1483
days old
9 comments
5 participants
participants (5)
-
Frank Schilder -
jcharles＠provectio.fr -
Maged Mokhtar -
Marc Roos -
Olivier AUDRY