25 Sep
2020
25 Sep
'20
7:39 p.m.
Hey folks!
Just shooting this out there in case someone has some advice. We're
just setting up RGW object storage for one of our new Ceph clusters (3
mons, 1072 OSDs, 34 nodes) and doing some benchmarking before letting
users on it.
We have 10Gb network to our two RGW nodes behind a single ip on
haproxy, and some iperf testing shows I can push that much; latencies
look okay. However, when using a small cosbench cluster I am unable to
get more than ~250Mb of read speed total.
If I add more nodes to the cosbench cluster it just spreads out the
load evenly with the same cap Same results when running two cosbench
clusters from different locations. I don't see any obvious bottlenecks
in terms of the RGW server hardware limitations, but here I am asking
for assistance so I don't put it past me missing something. I have
attached one of my cosbench load files with keys removed, but I get
similar results with different numbers of workers, objects, buckets,
object sizes, and cosbench drivers.
Does anyone have any pointers on what I could find to nail this
bottleneck down? Am I wrong in expecting more throughput? Let me know
if I can get any other info for you.
Cheers,
Dylan
--
Dylan Griff
Senior System Administrator
CLE D063
RCS - Systems - University of Victoria
26 Sep
26 Sep
4:23 a.m.
Can you share the object size details. Try to increase gradually to say 1gb
and measure.
Thanks
On Sat, 26 Sep, 2020, 1:10 am Dylan Griff, <dcgriff(a)uvic.ca> wrote:
Hey folks!
Just shooting this out there in case someone has some advice. We're
just setting up RGW object storage for one of our new Ceph clusters (3
mons, 1072 OSDs, 34 nodes) and doing some benchmarking before letting
users on it.
We have 10Gb network to our two RGW nodes behind a single ip on
haproxy, and some iperf testing shows I can push that much; latencies
look okay. However, when using a small cosbench cluster I am unable to
get more than ~250Mb of read speed total.
If I add more nodes to the cosbench cluster it just spreads out the
load evenly with the same cap Same results when running two cosbench
clusters from different locations. I don't see any obvious bottlenecks
in terms of the RGW server hardware limitations, but here I am asking
for assistance so I don't put it past me missing something. I have
attached one of my cosbench load files with keys removed, but I get
similar results with different numbers of workers, objects, buckets,
object sizes, and cosbench drivers.
Does anyone have any pointers on what I could find to nail this
bottleneck down? Am I wrong in expecting more throughput? Let me know
if I can get any other info for you.
Cheers,
Dylan
--
Dylan Griff
Senior System Administrator
CLE D063
RCS - Systems - University of Victoria
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
28 Sep
28 Sep
1:13 p.m.
New subject: Ceph RGW Performance [EXT]
Hi,
On 25/09/2020 20:39, Dylan Griff wrote:
We have 10Gb network to our two RGW nodes behind a
single ip on
haproxy, and some iperf testing shows I can push that much; latencies
look okay. However, when using a small cosbench cluster I am unable to
get more than ~250Mb of read speed total.
A few thoughts:
i) have you benchmarked your ceph itself (i.e. with rados bench - you'll
want to parameter-sweep with how many clients you run (and/or
threads/client))? That gives you a more useful baseline
ii) what does your RGW node load look like? On our RGW nodes we can
eventually use up all the available CPU, but that required tuning both
cosbench (which you look to have tried) and civetweb - rgw num rados
handles and num_threads in rgw frontends
iii) is haproxy your rate-limiter? We had to significantly increase
nbthread on our haproxy
HTH,
Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
9:50 p.m.
New subject: Ceph RGW Performance [EXT]
Hi,
what is the nbproc setting on the haproxy ?
Hi,
On 25/09/2020 20:39, Dylan Griff wrote:
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
We have 10Gb network to our two RGW nodes behind
a single ip on
haproxy, and some iperf testing shows I can push that much; latencies
look okay. However, when using a small cosbench cluster I am unable to
get more than ~250Mb of read speed total.
A few thoughts:
i) have you benchmarked your ceph itself (i.e. with rados bench -
you'll want to parameter-sweep with how many clients you run (and/or
threads/client))? That gives you a more useful baseline
ii) what does your RGW node load look like? On our RGW nodes we can
eventually use up all the available CPU, but that required tuning
both cosbench (which you look to have tried) and civetweb - rgw num
rados handles and num_threads in rgw frontends
iii) is haproxy your rate-limiter? We had to significantly increase
nbthread on our haproxy
HTH,
Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.ioTo unsubscribe send an
email to ceph-users-leave(a)ceph.io
10:45 p.m.
Thanks everyone for all of your guidance. To answer all the questions!
Are the OSD nodes connected with 10Gb as well?
Yes
Are you using SSDs for your index pool? How many?
Yes, for a node with 39 HDD OSDs we are using 6 Index SSDs
How big are your objects?
Most test run at 64K, but I have gone up to 4M with similar results
Try to increase gradually to say 1gb and measure.
Tried with 1gb size and result was the same, just around 250Mb/s
rados bench gives me ~1Gb/s, so 4 times what I am getting through our haproxy balanced end
point.
Our haproxy is version 1.5.18 and it looks like multithreading was introduced in 1.8 so
that could be culprit.
These cpu stats during the benchmark seem to agree as well
03:34:15 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
03:34:16 PM all 3.23 0.00 4.12 0.00 0.00 0.76 0.00 0.00 0.00 91.88
03:34:16 PM 0 4.00 0.00 2.00 0.00 0.00 1.00 0.00 0.00 0.00 93.00
03:34:16 PM 1 2.06 0.00 2.06 0.00 0.00 0.00 0.00 0.00 0.00 95.88
03:34:16 PM 2 2.04 0.00 3.06 0.00 0.00 0.00 0.00 0.00 0.00 94.90
03:34:16 PM 3 2.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 96.00
03:34:16 PM 4 39.00 0.00 58.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00
03:34:16 PM 5 0.99 0.00 1.98 0.00 0.00 0.00 0.00 0.00 0.00 97.03
03:34:16 PM 6 2.04 0.00 3.06 0.00 0.00 1.02 0.00 0.00 0.00 93.88
03:34:16 PM 7 1.01 0.00 2.02 0.00 0.00 0.00 0.00 0.00 0.00 96.97
03:34:16 PM 8 2.25 0.00 1.12 0.00 0.00 5.62 0.00 0.00 0.00 91.01
03:34:16 PM 9 2.02 0.00 2.02 0.00 0.00 0.00 0.00 0.00 0.00 95.96
03:34:16 PM 10 1.02 0.00 2.04 0.00 0.00 1.02 0.00 0.00 0.00 95.92
03:34:16 PM 11 1.98 0.00 1.98 0.00 0.00 0.99 0.00 0.00 0.00 95.05
03:34:16 PM 12 1.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.97
03:34:16 PM 13 1.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.98
03:34:16 PM 14 1.08 0.00 2.15 0.00 0.00 1.08 0.00 0.00 0.00 95.70
03:34:16 PM 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
03:34:16 PM 16 1.04 0.00 3.12 0.00 0.00 1.04 0.00 0.00 0.00 94.79
03:34:16 PM 17 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
03:34:16 PM 18 1.04 0.00 1.04 0.00 0.00 1.04 0.00 0.00 0.00 96.88
03:34:16 PM 19 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.01
03:34:16 PM 20 2.04 0.00 3.06 0.00 0.00 1.02 0.00 0.00 0.00 93.88
03:34:16 PM 21 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
03:34:16 PM 22 1.04 0.00 2.08 0.00 0.00 2.08 0.00 0.00 0.00 94.79
03:34:16 PM 23 3.00 0.00 7.00 0.00 0.00 0.00 0.00 0.00 0.00 90.00
I'll start by upgrading that, ramping up multithreading and let you know how it
goes.
Cheers,
Dylan
On Fri, 2020-09-25 at 19:39 +0000, Dylan Griff wrote:
Notice: This message was sent from outside the
University of Victoria email system but is claiming to be from UVic. Please be cautious
with links, attachments, and sensitive information.
Notice: This message was sent from outside the University of Victoria email system.
Please be cautious with links and sensitive information.
Hey folks!
Just shooting this out there in case someone has some advice. We're
just setting up RGW object storage for one of our new Ceph clusters (3
mons, 1072 OSDs, 34 nodes) and doing some benchmarking before letting
users on it.
We have 10Gb network to our two RGW nodes behind a single ip on
haproxy, and some iperf testing shows I can push that much; latencies
look okay. However, when using a small cosbench cluster I am unable to
get more than ~250Mb of read speed total.
If I add more nodes to the cosbench cluster it just spreads out the
load evenly with the same cap Same results when running two cosbench
clusters from different locations. I don't see any obvious bottlenecks
in terms of the RGW server hardware limitations, but here I am asking
for assistance so I don't put it past me missing something. I have
attached one of my cosbench load files with keys removed, but I get
similar results with different numbers of workers, objects, buckets,
object sizes, and cosbench drivers.
Does anyone have any pointers on what I could find to nail this
bottleneck down? Am I wrong in expecting more throughput? Let me know
if I can get any other info for you.
Cheers,
Dylan
--
Dylan Griff
Senior System Administrator
CLE D063
RCS - Systems - University of Victoria
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1318
days inactive
1321
days old
4 comments
4 participants
participants (4)
-
Daniel Mezentsev -
Dylan Griff -
martin joy -
Matthew Vernon