18 Jul
2020
18 Jul
'20
1:05 a.m.
Hi All,
I started a small project related to metrics collection and
processing, Ceph was chosen as a storage backend. Decided to use rados
directly, to avoid any additional layers. I got a very simple client -
it works fine, but performance is very low. Can't get more than
30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be
mentioned about the client - I'm using sbcl (yep lisp). Call to rados
API is just cffi call.
Did try in async mode. Wow ! Saturated network bandwidth for large
objects (4Mb and bigger), for small objects - saturated OSD IOPS
~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty
reasonable. Bottom line - issue not with the lisp client - im getting
close to C performance, difference is sync vs async IO.
Why it's so big - sync operations are approx 2-3 times slower then async .
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
18 Jul
18 Jul
2:03 a.m.
Daniel;
How is your pool configured? Replica, or Erasure-Coded?
I don't actually know any of this, but...
I would expect that a synchronous call to a replica pool (R=3) would look something like
this:
Client --> PG Master Host (data)
PG Master Host --> Local Disk (data)
PG Master Host --> PG Replica Host 1 (data)
PG Master Host --> PG Replica Host 2 (data)
PG Replica Host 1 --> Disk (data)
PG Replica Host 2 --> Disk (data)
PG Replica Host 1 --> PG Master Host (ack)
PG Replica Host 2 --> PG Master Host (ack)
PG Master Host --> Client (ack)
Some of that can happen in parallel, for instance the master could be transferring the
file to both replica hosts, while also writing it to disk.
You can imagine why a synchronous call could be significantly slower though, yes?
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Daniel Mezentsev [mailto:dan@soleks.com]
Sent: Friday, July 17, 2020 3:05 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] ceph/rados performace sync vs async
Hi All,
I started a small project related to metrics collection and
processing, Ceph was chosen as a storage backend. Decided to use rados
directly, to avoid any additional layers. I got a very simple client -
it works fine, but performance is very low. Can't get more than
30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be
mentioned about the client - I'm using sbcl (yep lisp). Call to rados
API is just cffi call.
Did try in async mode. Wow ! Saturated network bandwidth for large
objects (4Mb and bigger), for small objects - saturated OSD IOPS
~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty
reasonable. Bottom line - issue not with the lisp client - im getting
close to C performance, difference is sync vs async IO.
Why it's so big - sync operations are approx 2-3 times slower then async .
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:14 a.m.
Hi Dominic,
Pool configured as replica, replica size is 2 (i know it's not
recommended, but it's test environment, and i ca't afford more nodes
running at home). What you are saying - makes sense for me.
Yoo mentioned that even for sync IO some jobs can be done in parallel.
That is above my current level of ceph knowledge, but sounds like very
interesting part to take a look at. Can you give me initial direction
where to start, and is that options are configurable ?
Daniel;
How is your pool configured? Replica, or Erasure-Coded?
I don't actually know any of this, but...
I would expect that a synchronous call to a replica pool (R=3) would
look something like this:
Client --> PG Master Host (data)
PG Master Host --> Local Disk (data)
PG Master Host --> PG Replica Host 1 (data)
PG Master Host --> PG Replica Host 2 (data)
PG Replica Host 1 --> Disk (data)
PG Replica Host 2 --> Disk (data)
PG Replica Host 1 --> PG Master Host (ack)
PG Replica Host 2 --> PG Master Host (ack)
PG Master Host --> Client (ack)
Some of that can happen in parallel, for instance the master could
be transferring the file to both replica hosts, while also writing
it to disk.
You can imagine why a synchronous call could be significantly slower
though, yes?
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com[1]
-----Original Message-----
From: Daniel Mezentsev [mailto:dan@soleks.com]
Sent: Friday, July 17, 2020 3:05 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] ceph/rados performace sync vs async
Hi All,
I started a small project related to metrics collection and
processing, Ceph was chosen as a storage backend. Decided to use rados
directly, to avoid any additional layers. I got a very simple client -
it works fine, but performance is very low. Can't get more than
30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be
mentioned about the client - I'm using sbcl (yep lisp). Call to rados
API is just cffi call.
Did try in async mode. Wow ! Saturated network bandwidth for large
objects (4Mb and bigger), for small objects - saturated OSD IOPS
~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty
reasonable. Bottom line - issue not with the lisp client - im getting
close to C performance, difference is sync vs async IO.
Why it's so big - sync operations are approx 2-3 times slower then async .
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.ioTo unsubscribe send an
email to ceph-users-leave(a)ceph.io
Links:
------
[1] http://www.PerformAir.com
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
3:18 a.m.
Daniel;
As I said, I don't actually KNOW most of this.
As such, what I laid out was conceptual.
Ceph would need to be implemented to perform these operations in parallel, or not.
Conceptually, those areas where operations can be parallelized, making them parallel would
improve wall clock performance in 80% - 90% of cases, thus making this configurable
wouldn't make sense.
That said, I don't know which route the developers went.
All I know is that the client transfers each chunk to the master for its PG, and the
master sends it on to the replicas.
I suspect that replicas must acknowledge the chunk before the master finishes the
synchronous operation.
I suspect that all replicas are transferred (from the master) in parallel.
Given the maturity of Ceph, I suspect this has already been done, unless the developers
ran into a significant issue, but I don't know.
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Daniel Mezentsev [mailto:dan@soleks.com]
Sent: Friday, July 17, 2020 4:14 PM
To: Dominic Hilsbos
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: ceph/rados performace sync vs async
Hi Dominic,
Pool configured as replica, replica size is 2 (i know it's not
recommended, but it's test environment, and i ca't afford more nodes
running at home). What you are saying - makes sense for me.
Yoo mentioned that even for sync IO some jobs can be done in parallel.
That is above my current level of ceph knowledge, but sounds like very
interesting part to take a look at. Can you give me initial direction
where to start, and is that options are configurable ?
Daniel;
How is your pool configured? Replica, or Erasure-Coded?
I don't actually know any of this, but...
I would expect that a synchronous call to a replica pool (R=3) would
look something like this:
Client --> PG Master Host (data)
PG Master Host --> Local Disk (data)
PG Master Host --> PG Replica Host 1 (data)
PG Master Host --> PG Replica Host 2 (data)
PG Replica Host 1 --> Disk (data)
PG Replica Host 2 --> Disk (data)
PG Replica Host 1 --> PG Master Host (ack)
PG Replica Host 2 --> PG Master Host (ack)
PG Master Host --> Client (ack)
Some of that can happen in parallel, for instance the master could
be transferring the file to both replica hosts, while also writing
it to disk.
You can imagine why a synchronous call could be significantly slower
though, yes?
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com[1]
-----Original Message-----
From: Daniel Mezentsev [mailto:dan@soleks.com]
Sent: Friday, July 17, 2020 3:05 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] ceph/rados performace sync vs async
Hi All,
I started a small project related to metrics collection and
processing, Ceph was chosen as a storage backend. Decided to use rados
directly, to avoid any additional layers. I got a very simple client -
it works fine, but performance is very low. Can't get more than
30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be
mentioned about the client - I'm using sbcl (yep lisp). Call to rados
API is just cffi call.
Did try in async mode. Wow ! Saturated network bandwidth for large
objects (4Mb and bigger), for small objects - saturated OSD IOPS
~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty
reasonable. Bottom line - issue not with the lisp client - im getting
close to C performance, difference is sync vs async IO.
Why it's so big - sync operations are approx 2-3 times slower then async .
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.ioTo unsubscribe send an
email to ceph-users-leave(a)ceph.io
Links:
------
[1] http://www.PerformAir.com
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:43 a.m.
Den lör 18 juli 2020 kl 02:18 skrev <DHilsbos(a)performair.com>om>:
Daniel;
As I said, I don't actually KNOW most of this.
Seems correct in my view though.
As such, what I laid out was conceptual.
Ceph would need to be implemented to perform these operations in parallel,
or not. Conceptually, those areas where operations can be parallelized,
making them parallel would improve wall clock performance in 80% - 90% of
cases, thus making this configurable wouldn't make sense.
That said, I don't know which route the developers went.
All I know is that the client transfers each chunk to the master for its
PG, and the master sends it on to the replicas.
I suspect that replicas must acknowledge the chunk before the master
finishes the synchronous operation.
I suspect that all replicas are transferred (from the master) in parallel.
It probably is parallel, but if you can max out your network with the
traffic, then there will still be waiting from the master before all
replicas get their data. In this case it seems it was repl=2, so that is
not an issue, but if you had 1GE and repl > 2 I'm sure you'd notice how
network would make those transfers feel very non-parallel. ;)
Given the maturity of Ceph, I suspect this has already
been done, unless
the developers ran into a significant issue, but I don't know.
One of the things to consider is that if you do one single stream, you are
basically not testing what a cluster can do, but only what the absolute
smallest setup does. If you have 100s of consumers talking to your cluster,
they can't all just fire off a single copy and then immediately proceed and
send the next while all the background traffic still needs to happen, times
100.
Well, you can, but you will still see the lower "real" bandwidth per-client
at that point.
Going async is the same as writing to RAM buffers or the small WAL/journal
on a faster drive and so on. It helps with a short temporary spike if you
didn't have anything running at the same time, but it doesn't really
reflect the true capacity in the cluster for a single node, nor will a
single test show the total capacity of a ceph cluster either, since that
will be the sum of all (lower) single-client speeds.
It will only show what perf you can get until your RAM buffer or
WAL/journal runs out of capacity so you aren't really benching the correct
thing.
--
May the most significant bit of your life be positive.
1:40 p.m.
On 18/07/2020 00:05, Daniel Mezentsev wrote:
Hi All,
I started a small project related to metrics collection and
processing, Ceph was chosen as a storage backend. Decided to use rados
directly, to avoid any additional layers. I got a very simple client -
it works fine, but performance is very low. Can't get more than
30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be
mentioned about the client - I'm using sbcl (yep lisp). Call to rados
API is just cffi call.
Did try in async mode. Wow ! Saturated network bandwidth for large
objects (4Mb and bigger), for small objects - saturated OSD IOPS
~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty
reasonable. Bottom line - issue not with the lisp client - im getting
close to C performance, difference is sync vs async IO.
Why it's so big - sync operations are approx 2-3 times slower then
async .
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
What do you get in your rados benchmark result if you add -t 1 to
indicate 1 thread/queue depth to make it similar to your simple client,
the default is 16. You should also either use 4M block size in your
client or adjust the rados bench block size with -b
/Maged
20 Jul
20 Jul
7:55 p.m.
Hi All,
Did more tests. Just one client, big object / small object, several
clients with big and small objects - and seems like im getting
absolutely reasonable numbers. Big objects are satturating network,
small objects - IOPs on discs. Overall i have better understanding and
im happy about results.
Thanks to everybody for the help.
On 18/07/2020 00:05, Daniel Mezentsev wrote:
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
Hi All,
I started a small project related to metrics collection and
processing, Ceph was chosen as a storage backend. Decided to use
rados directly, to avoid any additional layers. I got a very simple
client - it works fine, but performance is very low. Can't get more
than 30-35MBsec. Rados bench shows 200MBsec for my test pool.
Should be mentioned about the client - I'm using sbcl (yep lisp).
Call to rados API is just cffi call.
Did try in async mode. Wow ! Saturated network bandwidth for large
objects (4Mb and bigger), for small objects - saturated OSD IOPS
~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds
pretty reasonable. Bottom line - issue not with the lisp client -
im getting close to C performance, difference is sync vs async IO.
Why it's so big - sync operations are approx 2-3 times slower then async .
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
What do you get in your rados benchmark result if you add -t 1 to
indicate 1 thread/queue depth to make it similar to your simple
client, the default is 16. You should also either use 4M block size
in your client or adjust the rados bench block size with -b
/Maged
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.ioTo unsubscribe send an
email to ceph-users-leave(a)ceph.io
1383
days inactive
1386
days old
6 comments
4 participants
participants (4)
-
Daniel Mezentsev -
DHilsbos＠performair.com -
Janne Johansson -
Maged Mokhtar