Hi Dominic, Pool configured as replica, replica size is 2 (i know it's not recommended, but it's test environment, and i ca't afford more nodes running at home). What you are saying - makes sense for me. Yoo mentioned that even for sync IO some jobs can be done in parallel. That is above my current level of ceph knowledge, but sounds like very interesting part to take a look at. Can you give me initial direction where to start, and is that options are configurable ?

Daniel;

How is your pool configured? Replica, or Erasure-Coded?

I don't actually know any of this, but...

I would expect that a synchronous call to a replica pool (R=3) would look something like this: Client --> PG Master Host (data) PG Master Host --> Local Disk (data) PG Master Host --> PG Replica Host 1 (data) PG Master Host --> PG Replica Host 2 (data) PG Replica Host 1 --> Disk (data) PG Replica Host 2 --> Disk (data) PG Replica Host 1 --> PG Master Host (ack) PG Replica Host 2 --> PG Master Host (ack) PG Master Host --> Client (ack)

Some of that can happen in parallel, for instance the master could be transferring the file to both replica hosts, while also writing it to disk.

You can imagine why a synchronous call could be significantly slower though, yes?

Thank you,

Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com[1]

-----Original Message----- From: Daniel Mezentsev [mailto:dan@soleks.com] Sent: Friday, July 17, 2020 3:05 PM To: ceph-users@ceph.io Subject: [ceph-users] ceph/rados performace sync vs async

Hi All,

I started a small project related to metrics collection and processing, Ceph was chosen as a storage backend. Decided to use rados directly, to avoid any additional layers. I got a very simple client - it works fine, but performance is very low. Can't get more than 30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be mentioned about the client - I'm using sbcl (yep lisp). Call to rados API is just cffi call.    Did try in async mode. Wow ! Saturated network bandwidth for large objects (4Mb and bigger), for small objects - saturated OSD IOPS ~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty reasonable. Bottom line - issue not with the lisp client - im getting close to C performance, difference is sync vs async IO.   Why it's so big - sync operations are approx 2-3 times slower then async . Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-leave@ceph.io

Links: ------ [1] http://www.PerformAir.com Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

Den lör 18 juli 2020 kl 02:18 skrev <DHilsbos@performair.com>:

Daniel; As I said, I don't actually KNOW most of this.

Seems correct in my view though.

As such, what I laid out was conceptual. Ceph would need to be implemented to perform these operations in parallel, or not. Conceptually, those areas where operations can be parallelized, making them parallel would improve wall clock performance in 80% - 90% of cases, thus making this configurable wouldn't make sense.

That said, I don't know which route the developers went. All I know is that the client transfers each chunk to the master for its PG, and the master sends it on to the replicas. I suspect that replicas must acknowledge the chunk before the master finishes the synchronous operation. I suspect that all replicas are transferred (from the master) in parallel.

It probably is parallel, but if you can max out your network with the traffic, then there will still be waiting from the master before all replicas get their data. In this case it seems it was repl=2, so that is not an issue, but if you had 1GE and repl > 2 I'm sure you'd notice how network would make those transfers feel very non-parallel. ;)

Given the maturity of Ceph, I suspect this has already been done, unless the developers ran into a significant issue, but I don't know.

One of the things to consider is that if you do one single stream, you are basically not testing what a cluster can do, but only what the absolute smallest setup does. If you have 100s of consumers talking to your cluster, they can't all just fire off a single copy and then immediately proceed and send the next while all the background traffic still needs to happen, times 100. Well, you can, but you will still see the lower "real" bandwidth per-client at that point. Going async is the same as writing to RAM buffers or the small WAL/journal on a faster drive and so on. It helps with a short temporary spike if you didn't have anything running at the same time, but it doesn't really reflect the true capacity in the cluster for a single node, nor will a single test show the total capacity of a ceph cluster either, since that will be the sum of all (lower) single-client speeds. It will only show what perf you can get until your RAM buffer or WAL/journal runs out of capacity so you aren't really benching the correct thing. -- May the most significant bit of your life be positive.

Hi All, Did more tests. Just one client, big object / small object, several clients with big and small objects - and seems like im getting absolutely reasonable numbers. Big objects are satturating network, small objects - IOPs on discs. Overall i have better understanding and im happy about results. Thanks to everybody for the help.

On 18/07/2020 00:05, Daniel Mezentsev wrote:

...

Hi All,

 I started a small project related to metrics collection and processing, Ceph was chosen as a storage backend. Decided to use rados directly, to avoid any additional layers. I got a very simple client - it works fine, but performance is very low. Can't get more than 30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be mentioned about the client - I'm using sbcl (yep lisp). Call to rados API is just cffi call.

 Did try in async mode. Wow ! Saturated network bandwidth for large objects (4Mb and bigger), for small objects - saturated OSD IOPS ~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty reasonable. Bottom line - issue not with the lisp client - im getting close to C performance, difference is sync vs async IO.

 Why it's so big - sync operations are approx 2-3 times slower then async .  Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

What do you get in your rados benchmark result if you add  -t 1 to indicate 1 thread/queue depth to make it similar to your simple client, the default is 16. You should also either use 4M block size in your client or adjust the rados bench block size with -b

/Maged

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-leave@ceph.io Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.