Strange performance drop and low oss performance
Hello, I'm a beginner on ceph. I set up three ceph clusters on google cloud. Cluster1 has three nodes and each node has three disks. Cluster2 has three nodes and each node has two disks. Cluster3 has five nodes and each node has five disks. All disks are HDD. Disk speed shown by `dd if=/dev/zero of=here bs=1G count=1 oflag=direct` is 117MB/s. The network is 10Gbps. Ceph version is 12.2.12. I found something strange: 1. When running `rados bench`, the write performance of all clusters drops dramatically after a few minutes. I created a pool named "scbench" with replicated size 1 (I know it is not safe but I want the highest write speed). The write performance (shown by rados bench -p scbench 1000 write) before and after the drop are: cluster1: 297MB/s 94.5MB/s cluster2: 304MB/s 67.4MB/s cluster3: 494MB/s 267.6MB/s It looks like the performance before the drop is nodes_num * 100MB/s, and the performance after the drop is about osds_num * 10MB/s. I have no idea why there is such a drop and why the performances before the drop are linear with nodes_num. 2. The write performance of object storage (shown by swift-bench -c 64 -s 4096000 -n 100000 -g 0 swift.conf) is much lower than that of storage cluster(shown by rados bench -p scbench 1000 write). I have set the replicated size of "default.rgw.buckets.data" and "default.rgw.buckets.index" to 1 The speed of cluster1 oss is 117MB/s (before the drop) and 26MB/s (after the drop), and the speed of cluster3 oss is 118MB/s (the drop does not happen). Is it normal that the oss write performance is worse than rados write performance? If not, how can I solve the problem? Thanks!
Den ons 5 feb. 2020 kl 11:14 skrev quexian da <daquexian566@gmail.com>:
Hello,
I'm a beginner on ceph. I set up three ceph clusters on google cloud. Cluster1 has three nodes and each node has three disks. Cluster2 has three nodes and each node has two disks. Cluster3 has five nodes and each node has five disks. All disks are HDD. Disk speed shown by `dd if=/dev/zero of=here bs=1G count=1 oflag=direct` is 117MB/s.
The write performance (shown by rados bench -p scbench 1000 write) before and after the drop are:
cluster1: 297MB/s 94.5MB/s cluster2: 304MB/s 67.4MB/s cluster3: 494MB/s 267.6MB/s
It looks like the performance before the drop is nodes_num * 100MB/s, and the performance after the drop is about osds_num * 10MB/s. I have no idea why there is such a drop and why the performances before the drop are linear with nodes_num.
You are probably seeing write caching up to some point when buffer ram is expired, then get down to more "real" disk speeds, minus overhead from ceph. This is why more hosts seem to give more performance in the warmup. (and of course, this applies to physical boxes too, whatever write caches are in use will be multiplied by the amount of hosts you have which uses such caches) Do mind that ceph is not a cluster aimed at winning single-threaded write competitions, but rather having many hosts serve many clients and being able to scale that to large numbers. -- May the most significant bit of your life be positive.
Thanks for your valuable answer! Is the write cache specific to ceph? Could you please provide some links to the documentation about the write cache? Thanks! Do you have any idea about the slow oss speed? Is it normal that the write performance of object gateway is slower than that of rados cluster? Thanks in advance! On Wed, Feb 5, 2020, 10:10 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den ons 5 feb. 2020 kl 11:14 skrev quexian da <daquexian566@gmail.com>:
Hello,
I'm a beginner on ceph. I set up three ceph clusters on google cloud. Cluster1 has three nodes and each node has three disks. Cluster2 has three nodes and each node has two disks. Cluster3 has five nodes and each node has five disks. All disks are HDD. Disk speed shown by `dd if=/dev/zero of=here bs=1G count=1 oflag=direct` is 117MB/s.
The write performance (shown by rados bench -p scbench 1000 write) before and after the drop are:
cluster1: 297MB/s 94.5MB/s cluster2: 304MB/s 67.4MB/s cluster3: 494MB/s 267.6MB/s
It looks like the performance before the drop is nodes_num * 100MB/s, and the performance after the drop is about osds_num * 10MB/s. I have no idea why there is such a drop and why the performances before the drop are linear with nodes_num.
You are probably seeing write caching up to some point when buffer ram is expired, then get down to more "real" disk speeds, minus overhead from ceph. This is why more hosts seem to give more performance in the warmup. (and of course, this applies to physical boxes too, whatever write caches are in use will be multiplied by the amount of hosts you have which uses such caches)
Do mind that ceph is not a cluster aimed at winning single-threaded write competitions, but rather having many hosts serve many clients and being able to scale that to large numbers.
-- May the most significant bit of your life be positive.
Den ons 5 feb. 2020 kl 16:19 skrev quexian da <daquexian566@gmail.com>:
Thanks for your valuable answer! Is the write cache specific to ceph? Could you please provide some links to the documentation about the write cache? Thanks!
It is all the possible caches used by ceph, by the device driver, the filesystem (in filestore+xfs), the controllers (emulated or real) and the harddisk electronics, ie anything between the benchmark software and the spinning disk write head (or not so spinning on ssds).
Do you have any idea about the slow oss speed? Is it normal that the write performance of object gateway is slower than that of rados cluster? Thanks in advance!
Object gateway (be it swift or S3) goes over something that looks like http, so it will most certainly have longer turn around times and hence slower speed for single streams. You may possibly get over parts of that overhead by having many multiple streams and counting the sum of the transfers, but there is no big surprise that individual writes get slower if you have to pass via an external box (the radosgw) using https instead of writing directly to the storage. -- May the most significant bit of your life be positive.
Thanks for your valuable answer about write cache! For object gateway, the performance is got by `swift-bench -t 64` which uses 64 threads concurrently. Will the radosgw and http overhead be so significant (94.5MB/s to 26MB/s for cluster1) when multiple threads are used? Thanks in advance! On Wed, Feb 5, 2020, 11:33 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den ons 5 feb. 2020 kl 16:19 skrev quexian da <daquexian566@gmail.com>:
Thanks for your valuable answer! Is the write cache specific to ceph? Could you please provide some links to the documentation about the write cache? Thanks!
It is all the possible caches used by ceph, by the device driver, the filesystem (in filestore+xfs), the controllers (emulated or real) and the harddisk electronics, ie anything between the benchmark software and the spinning disk write head (or not so spinning on ssds).
Do you have any idea about the slow oss speed? Is it normal that the write performance of object gateway is slower than that of rados cluster? Thanks in advance!
Object gateway (be it swift or S3) goes over something that looks like http, so it will most certainly have longer turn around times and hence slower speed for single streams. You may possibly get over parts of that overhead by having many multiple streams and counting the sum of the transfers, but there is no big surprise that individual writes get slower if you have to pass via an external box (the radosgw) using https instead of writing directly to the storage.
-- May the most significant bit of your life be positive.
For object gateway, the performance is got by `swift-bench -t 64` which uses 64 threads concurrently. Will the radosgw and http overhead be so significant (94.5MB/s to 26MB/s for cluster1) when multiple threads are used? Thanks in advance!
Can't say what it "must" be, but if I log in to one of my rgw's (we have several, loadbalanced) and run ceph benchmarks against spindrive pools (ie, talking ceph directly), I get something like 200MB/s, if I run a write test on the same host, but talking s3-over-http against itself, I get something like 100MB/s, so the overhead in my case seems to be 100% (or 50% however you calculate it). You know there has to be some kind of penalty for doing protocol translations, if for nothing else, because of object store client does checksums, asks rgw to store it, rgw checksums the part(s), asks ceph to store, ceph sends ack, rgw sends ack to client with checksum and client compares before moving to next part. This will be far slower than just plain writes to ceph (the two innermost ops), and can in part be offset by using large IO, parallel streams, multiple rgw backends and so on. -- May the most significant bit of your life be positive.
I didn't have such drop in performance testing 'rados bench 360 write -p rbd' on 3x replicated (slow)hdd pool. Sort of near the average, sometimes drops to 90. But I guess the test hits than an osd that is scrubbing and being used by other processes. -----Original Message----- Sent: 05 February 2020 16:34 To: quexian da Cc: ceph-users Subject: [ceph-users] Re: Strange performance drop and low oss performance Den ons 5 feb. 2020 kl 16:19 skrev quexian da <daquexian566@gmail.com>:
Thanks for your valuable answer! Is the write cache specific to ceph? Could you please provide some links to the documentation about the write cache? Thanks!
It is all the possible caches used by ceph, by the device driver, the filesystem (in filestore+xfs), the controllers (emulated or real) and the harddisk electronics, ie anything between the benchmark software and the spinning disk write head (or not so spinning on ssds).
Do you have any idea about the slow oss speed? Is it normal that the write performance of object gateway is slower than that of rados cluster? Thanks in advance!
Object gateway (be it swift or S3) goes over something that looks like http, so it will most certainly have longer turn around times and hence slower speed for single streams. You may possibly get over parts of that overhead by having many multiple streams and counting the sum of the transfers, but there is no big surprise that individual writes get slower if you have to pass via an external box (the radosgw) using https instead of writing directly to the storage. -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Janne Johansson
-
Marc Roos
-
quexian da