10 Feb
2023
10 Feb
'23
6:07 p.m.
Good morning everyone, been running a small Ceph cluster with Proxmox for a while now and
I’ve finally run across an issue I can’t find any information on. I have a 3 node cluster
with 9 Samsung PM983 960GB NVME drives running on a dedicated 10gb network. RBD and CephFS
performance have been great, most of the time I see over 500MBs writes and a rados
benchmark shows 951 MB/s write and 1140 MB/s read bandwidth.
The problem I’m seeing is after setting up RadosGW I can only upload to “S3” at around
25MBs with the official AWS CLI. Using s3cmd is slightly better at around 45MB/s. I’m
going directly to the RadosGW instance with no load balancers in between and no ssl
enabled. Just trying to figure out if this is normal. I’m not expecting it to be as fast
as writing directly to a RBD but I was kinda hoping for more than this.
So what should I expect in performance from the RadosGW?
Here are some rados bench results and my ceph report
https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5
https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438
Thanks
Shawn
10 Feb
10 Feb
6:15 p.m.
Hi Shawn,
To get another S3 upload baseline, I'd recommend doing some upload testing
with s5cmd [1].
1. https://github.com/peak/s5cmd
Matt
On Fri, Feb 10, 2023 at 9:38 AM Shawn Weeks <sweeks(a)weeksconsulting.us>
wrote:
Good morning everyone, been running a small Ceph
cluster with Proxmox for
a while now and I’ve finally run across an issue I can’t find any
information on. I have a 3 node cluster with 9 Samsung PM983 960GB NVME
drives running on a dedicated 10gb network. RBD and CephFS performance have
been great, most of the time I see over 500MBs writes and a rados benchmark
shows 951 MB/s write and 1140 MB/s read bandwidth.
The problem I’m seeing is after setting up RadosGW I can only upload to
“S3” at around 25MBs with the official AWS CLI. Using s3cmd is slightly
better at around 45MB/s. I’m going directly to the RadosGW instance with no
load balancers in between and no ssl enabled. Just trying to figure out if
this is normal. I’m not expecting it to be as fast as writing directly to a
RBD but I was kinda hoping for more than this.
So what should I expect in performance from the RadosGW?
Here are some rados bench results and my ceph report
https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5
https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438
Thanks
Shawn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
7:23 p.m.
With s5cmd and its defaults I got around 127MB/s for a single 16gb test file. Is there any
way to make s5cmd give feedback while it’s running. At first I didn’t think it was working
because it just sat there for a while.
Thanks
Shawn
On Feb 10, 2023, at 8:45 AM, Matt Benjamin <mbenjami(a)redhat.com> wrote:
Hi Shawn,
To get another S3 upload baseline, I'd recommend doing some upload testing with s5cmd
[1].
1. https://github.com/peak/s5cmd
Matt
On Fri, Feb 10, 2023 at 9:38 AM Shawn Weeks
<sweeks@weeksconsulting.us<mailto:sweeks@weeksconsulting.us>> wrote:
Good morning everyone, been running a small Ceph cluster with Proxmox for a while now and
I’ve finally run across an issue I can’t find any information on. I have a 3 node cluster
with 9 Samsung PM983 960GB NVME drives running on a dedicated 10gb network. RBD and CephFS
performance have been great, most of the time I see over 500MBs writes and a rados
benchmark shows 951 MB/s write and 1140 MB/s read bandwidth.
The problem I’m seeing is after setting up RadosGW I can only upload to “S3” at around
25MBs with the official AWS CLI. Using s3cmd is slightly better at around 45MB/s. I’m
going directly to the RadosGW instance with no load balancers in between and no ssl
enabled. Just trying to figure out if this is normal. I’m not expecting it to be as fast
as writing directly to a RBD but I was kinda hoping for more than this.
So what should I expect in performance from the RadosGW?
Here are some rados bench results and my ceph report
https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5
https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438
Thanks
Shawn
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
6:16 p.m.
The problem I’m seeing is after setting up RadosGW I
can only upload to “S3” at around 25MBs with the official AWS CLI. Using s3cmd is slightly
better at around 45MB/s. I’m going directly to the RadosGW instance with no load balancers
in between and no ssl enabled. Just trying to figure out if this is normal. I’m not
expecting it to be as fast as writing directly to a RBD but I was kinda hoping for more
than this.
So what should I expect in performance from the RadosGW?
For s3cmd, I have some perf options I use,
multipart_chunk_size_mb = 256
send_chunk = 262144
recv_chunk = 262144
and frequently see 100-150MB/s for well connected client runs,
especially if you repeat uploads and use s3cmd's --cache-file=FILE
option so that you don't benchmark your local computers ability to
checksum the object(s).
But I would also consider using rclone and/or something that actually
makes sure to split up large files/objects and uploads them in
parallel. We have hdd+nvme clusters on 25GE networks that ingest some
1.5-2 GB/s using lots of threads and many clients, but the totals are
in that vicinity. Several load balancers and some 6-9 rgws to share
the load helps there.
--
May the most significant bit of your life be positive.
7:29 p.m.
With this options I still see around 38-40MB/s for my 16gb test file. So far my testing is
mostly synthetic, I’m going to be using some programs like GitLab and Sonatype Nexus that
store their data in object storage. At work I deal with real S3 and regular see upload
speeds in the 100s of MB/s so I was kinda surprised that the aws cli was only doing 25 or
so.
Thanks
Shawn
On Feb 10, 2023, at 8:46 AM, Janne Johansson
<icepic.dz(a)gmail.com> wrote:
The problem I’m seeing is after setting up
RadosGW I can only upload to “S3” at around 25MBs with the official AWS CLI. Using s3cmd
is slightly better at around 45MB/s. I’m going directly to the RadosGW instance with no
load balancers in between and no ssl enabled. Just trying to figure out if this is normal.
I’m not expecting it to be as fast as writing directly to a RBD but I was kinda hoping for
more than this.
So what should I expect in performance from the RadosGW?
For s3cmd, I have some perf options I use,
multipart_chunk_size_mb = 256
send_chunk = 262144
recv_chunk = 262144
and frequently see 100-150MB/s for well connected client runs,
especially if you repeat uploads and use s3cmd's --cache-file=FILE
option so that you don't benchmark your local computers ability to
checksum the object(s).
But I would also consider using rclone and/or something that actually
makes sure to split up large files/objects and uploads them in
parallel. We have hdd+nvme clusters on 25GE networks that ingest some
1.5-2 GB/s using lots of threads and many clients, but the totals are
in that vicinity. Several load balancers and some 6-9 rgws to share
the load helps there.
--
May the most significant bit of your life be positive.
8:43 p.m.
For reference, with parallel writes using the S3 Go API (via hsbench:
https://github.com/markhpc/hsbench), I was recently doing about 600ish
MB/s to a single RGW instance from one client. RadosGW used around 3ish
HW threads from a 2016 era Xeon to do that. Didn't try single-file
tests in that case though which likely would have been slower.
Mark
On 2/10/23 09:59, Shawn Weeks wrote:
> With this options I still see around 38-40MB/s for my 16gb test file. So far my
testing is mostly synthetic, I’m going to be using some programs like GitLab and Sonatype
Nexus that store their data in object storage. At work I deal with real S3 and regular see
upload speeds in the 100s of MB/s so I was kinda surprised that the aws cli was only doing
25 or so.
>
> Thanks
> Shawn
>
>> On Feb 10, 2023, at 8:46 AM, Janne Johansson <icepic.dz(a)gmail.com> wrote:
>>
>>> The problem I’m seeing is after setting up RadosGW I can only upload to “S3”
at around 25MBs with the official AWS CLI. Using s3cmd is slightly better at around
45MB/s. I’m going directly to the RadosGW instance with no load balancers in between and
no ssl enabled. Just trying to figure out if this is normal. I’m not expecting it to be as
fast as writing directly to a RBD but I was kinda hoping for more than this.
>>>
>>> So what should I expect in performance from the RadosGW?
>>
>> For s3cmd, I have some perf options I use,
>>
>> multipart_chunk_size_mb = 256
>> send_chunk = 262144
>> recv_chunk = 262144
>> and frequently see 100-150MB/s for well connected client runs,
>> especially if you repeat uploads and use s3cmd's --cache-file=FILE
>> option so that you don't benchmark your local computers ability to
>> checksum the object(s).
>>
>> But I would also consider using rclone and/or something that actually
>> makes sure to split up large files/objects and uploads them in
>> parallel. We have hdd+nvme clusters on 25GE networks that ingest some
>> 1.5-2 GB/s using lots of threads and many clients, but the totals are
>> in that vicinity. Several load balancers and some 6-9 rgws to share
>> the load helps there.
>>
>> --
>> May the most significant bit of your life be positive.
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
462
days inactive
462
days old
5 comments
4 participants
participants (4)
-
Janne Johansson -
Mark Nelson -
Matt Benjamin -
Shawn Weeks