5 Feb
2020
5 Feb
'20
1:14 p.m.
Hello,
I'm a beginner on ceph. I set up three ceph clusters on google cloud.
Cluster1 has three nodes and each node has three disks. Cluster2 has three
nodes and each node has two disks. Cluster3 has five nodes and each node
has five disks.
All disks are HDD. Disk speed shown by `dd if=/dev/zero of=here bs=1G
count=1 oflag=direct` is 117MB/s.
The network is 10Gbps.
Ceph version is 12.2.12.
I found something strange:
1. When running `rados bench`, the write performance of all clusters drops
dramatically after a few minutes. I created a pool named "scbench" with
replicated size 1 (I know it is not safe but I want the highest write
speed). The write performance (shown by rados bench -p scbench 1000 write)
before and after the drop are:
cluster1: 297MB/s 94.5MB/s
cluster2: 304MB/s 67.4MB/s
cluster3: 494MB/s 267.6MB/s
It looks like the performance before the drop is nodes_num * 100MB/s, and
the performance after the drop is about osds_num * 10MB/s. I have no idea
why there is such a drop and why the performances before the drop are
linear with nodes_num.
2. The write performance of object storage (shown by swift-bench -c 64 -s
4096000 -n 100000 -g 0 swift.conf) is much lower than that of storage
cluster(shown by rados bench -p scbench 1000 write). I have set the
replicated size of "default.rgw.buckets.data" and
"default.rgw.buckets.index" to 1
The speed of cluster1 oss is 117MB/s (before the drop) and 26MB/s (after
the drop), and the speed of cluster3 oss is 118MB/s (the drop does not
happen).
Is it normal that the oss write performance is worse than rados write
performance? If not, how can I solve the problem?
Thanks!
5 Feb
5 Feb
5:09 p.m.
Den ons 5 feb. 2020 kl 11:14 skrev quexian da <daquexian566(a)gmail.com>om>:
Hello,
I'm a beginner on ceph. I set up three ceph clusters on google cloud.
Cluster1 has three nodes and each node has three disks. Cluster2 has three
nodes and each node has two disks. Cluster3 has five nodes and each node
has five disks.
All disks are HDD. Disk speed shown by `dd if=/dev/zero of=here bs=1G
count=1 oflag=direct` is 117MB/s.
The write performance (shown by rados bench -p scbench
1000 write)
before and after the drop are:
cluster1: 297MB/s 94.5MB/s
cluster2: 304MB/s 67.4MB/s
cluster3: 494MB/s 267.6MB/s
It looks like the performance before the drop is nodes_num * 100MB/s, and
the performance after the drop is about osds_num * 10MB/s. I have no idea
why there is such a drop and why the performances before the drop are
linear with nodes_num.
You are probably seeing write caching up to some point when buffer ram is
expired, then get down to more "real" disk speeds, minus overhead from ceph.
This is why more hosts seem to give more performance in the warmup. (and of
course, this applies to physical boxes too, whatever write caches are in
use will be multiplied by the amount of hosts you have which uses such
caches)
Do mind that ceph is not a cluster aimed at winning single-threaded write
competitions, but rather having many hosts serve many clients and being
able to scale that to large numbers.
--
May the most significant bit of your life be positive.
6:19 p.m.
Thanks for your valuable answer!
Is the write cache specific to ceph? Could you please provide some links to
the documentation about the write cache? Thanks!
Do you have any idea about the slow oss speed? Is it normal that the write
performance of object gateway is slower than that of rados cluster? Thanks
in advance!
On Wed, Feb 5, 2020, 10:10 PM Janne Johansson <icepic.dz(a)gmail.com> wrote:
Den ons 5 feb. 2020 kl 11:14 skrev quexian da
<daquexian566(a)gmail.com>om>:
Hello,
I'm a beginner on ceph. I set up three ceph clusters on google cloud.
Cluster1 has three nodes and each node has three disks. Cluster2 has three
nodes and each node has two disks. Cluster3 has five nodes and each node
has five disks.
All disks are HDD. Disk speed shown by `dd if=/dev/zero of=here bs=1G
count=1 oflag=direct` is 117MB/s.
The write performance (shown by rados bench -p
scbench 1000 write)
before and after the drop are:
cluster1: 297MB/s 94.5MB/s
cluster2: 304MB/s 67.4MB/s
cluster3: 494MB/s 267.6MB/s
It looks like the performance before the drop is nodes_num * 100MB/s, and
the performance after the drop is about osds_num * 10MB/s. I have no idea
why there is such a drop and why the performances before the drop are
linear with nodes_num.
You are probably seeing write caching up to some point when buffer ram is
expired, then get down to more "real" disk speeds, minus overhead from ceph.
This is why more hosts seem to give more performance in the warmup. (and
of course, this applies to physical boxes too, whatever write caches are in
use will be multiplied by the amount of hosts you have which uses such
caches)
Do mind that ceph is not a cluster aimed at winning single-threaded write
competitions, but rather having many hosts serve many clients and being
able to scale that to large numbers.
--
May the most significant bit of your life be positive.
6:33 p.m.
Den ons 5 feb. 2020 kl 16:19 skrev quexian da <daquexian566(a)gmail.com>om>:
Thanks for your valuable answer!
Is the write cache specific to ceph? Could you please provide some links
to the documentation about the write cache? Thanks!
It is all the possible caches used by ceph, by the device driver, the
filesystem (in filestore+xfs), the controllers (emulated or real) and the
harddisk electronics, ie anything between the benchmark software and the
spinning disk write head (or not so spinning on ssds).
Do you have any idea about the slow oss speed? Is it
normal that the write
performance of object gateway is slower than that of rados cluster? Thanks
in advance!
Object gateway (be it swift or S3) goes over something that looks like
http, so it will most certainly have longer turn around times and hence
slower speed for single streams.
You may possibly get over parts of that overhead by having many multiple
streams and counting the sum of the transfers, but there is no big surprise
that individual writes get slower if you have to pass via an external box
(the radosgw) using https instead of writing directly to the storage.
--
May the most significant bit of your life be positive.
6:59 p.m.
Thanks for your valuable answer about write cache!
For object gateway, the performance is got by `swift-bench -t 64` which
uses 64 threads concurrently. Will the radosgw and http overhead be so
significant (94.5MB/s to 26MB/s for cluster1) when multiple threads are
used? Thanks in advance!
On Wed, Feb 5, 2020, 11:33 PM Janne Johansson <icepic.dz(a)gmail.com> wrote:
Den ons 5 feb. 2020 kl 16:19 skrev quexian da
<daquexian566(a)gmail.com>om>:
Thanks for your valuable answer!
Is the write cache specific to ceph? Could you please provide some links
to the documentation about the write cache? Thanks!
It is all the possible caches used by ceph, by the device driver, the
filesystem (in filestore+xfs), the controllers (emulated or real) and the
harddisk electronics, ie anything between the benchmark software and the
spinning disk write head (or not so spinning on ssds).
Do you have any idea about the slow oss speed? Is
it normal that the
write performance of object gateway is slower than that of rados cluster?
Thanks in advance!
Object gateway (be it swift or S3) goes over something that looks like
http, so it will most certainly have longer turn around times and hence
slower speed for single streams.
You may possibly get over parts of that overhead by having many multiple
streams and counting the sum of the transfers, but there is no big surprise
that individual writes get slower if you have to pass via an external box
(the radosgw) using https instead of writing directly to the storage.
--
May the most significant bit of your life be positive.
6 Feb
6 Feb
11:12 a.m.
For object gateway, the performance is got by `swift-bench -t 64` which
uses 64 threads concurrently. Will the radosgw and http overhead be so
significant (94.5MB/s to 26MB/s for cluster1) when multiple threads are
used? Thanks in advance!
Can't say what it "must" be, but if I log in to one of my rgw's
(we have
several, loadbalanced) and run ceph benchmarks against spindrive pools (ie,
talking ceph directly), I get something like 200MB/s, if I run a write test
on the same host, but talking s3-over-http against itself, I get something
like 100MB/s, so the overhead in my case seems to be 100% (or 50% however
you calculate it).
You know there has to be some kind of penalty for doing protocol
translations, if for nothing else, because of object store client does
checksums, asks rgw to store it, rgw checksums the part(s), asks ceph to
store, ceph sends ack, rgw sends ack to client with checksum and client
compares before moving to next part.
This will be far slower than just plain writes to ceph (the two innermost
ops), and can in part be offset by using large IO, parallel streams,
multiple rgw backends and so on.
--
May the most significant bit of your life be positive.
5 Feb
5 Feb
7:47 p.m.
I didn't have such drop in performance testing 'rados bench 360 write -p
rbd' on 3x replicated (slow)hdd pool. Sort of near the average,
sometimes drops to 90. But I guess the test hits than an osd that is
scrubbing and being used by other processes.
-----Original Message-----
Sent: 05 February 2020 16:34
To: quexian da
Cc: ceph-users
Subject: [ceph-users] Re: Strange performance drop and low oss
performance
Den ons 5 feb. 2020 kl 16:19 skrev quexian da <daquexian566(a)gmail.com>om>:
Thanks for your valuable answer!
Is the write cache specific to ceph? Could you please provide some
links to the documentation about the write cache? Thanks!
It is all the possible caches used by ceph, by the device driver, the
filesystem (in filestore+xfs), the controllers (emulated or real) and
the harddisk electronics, ie anything between the benchmark software and
the spinning disk write head (or not so spinning on ssds).
Do you have any idea about the slow oss speed? Is it
normal that the
write performance of object gateway is slower than that of rados
cluster? Thanks in advance!
Object gateway (be it swift or S3) goes over something that looks like
http, so it will most certainly have longer turn around times and hence
slower speed for single streams.
You may possibly get over parts of that overhead by having many multiple
streams and counting the sum of the transfers, but there is no big
surprise that individual writes get slower if you have to pass via an
external box (the radosgw) using https instead of writing directly to
the storage.
--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
1549
days inactive
1550
days old
6 comments
3 participants
participants (3)
-
Janne Johansson -
Marc Roos -
quexian da