Ha, I hadn't thought to check if that existed which was probably pretty short-sighted on my part. :) I suppose the good news is that it might be a good candidate for comparison, and once I've implemented an S3 client endpoint in CBT I should be able to target both pretty easily. Mark On 8/15/19 6:41 PM, David Byte wrote:

Mark, did the S3 engine for fio not work?

Sent from my iPhone. Typos are Apple's fault.

...

On Aug 15, 2019, at 6:37 PM, Mark Nelson <mark.a.nelson@gmail.com> wrote:

Hi Guys,

Earlier this week I was working on investigating the impact of OMAP performance on RGW and wanted to see if putting rocksdb on ramdisk would help speed up bucket index updates. While running tests I found out that the benchmark tool I was using consumed roughly 15 cores of CPU to push 4K puts/second to RGW from 128 threads. That wasn't really viable, so I started looking for alternate S3 benchmarking tools. COSBench is sort of the most well known choice out there, but it's a bit cumbersome if you just want to run some quick tests from the command-line.

I managed to find a simple yet very nice benchmark written in go developed by Wasabi Inc called s3-benchmark. While it works well, it really only targets single buckets and is designed more for AWS testing than for Ceph. I forked the project and pretty much refactored the whole thing to be more useful for the kind of testing I want to do. It's now at the point where I think it might be ready to experiment with. S3 benchmarking has been a semi-recurring topic on the list so I figured other folks might be interested in trying it too and hopefully providing feedback.

The new benchmark is called hsbench and it's available here:

https://github.com/markhpc/hsbench

See the README.md for some of the advantages over the original s3-benchmark program it was forked from. I consider this release to be alpha level quality and disclaim all responsibility if it breaks and deletes every object in all of your buckets. Consider yourself warned. :)

Mark _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

On 8/20/19 7:58 AM, Lars Marowsky-Bree wrote:

On 2019-08-15T19:04:11, Mark Nelson <mnelson@redhat.com> wrote:

...

Ha, I hadn't thought to check if that existed which was probably pretty short-sighted on my part. :) It's somewhat limited (only does GET/PUT/DELETE), but the crucial benefit is that it allows direct comparison between the various protocols.

Indeed!  Now that we have client endpoints and the universal fio wrapper in cbt it should be fairly easy to add that as another supported option there too.   I'll be very curious to see how hsbench and fio compare in terms of client side CPU usage and overall throughput.  I'm pushing about 15K 4K gets with hsbench right now at sufficiently high thread counts, but it's still using a lot of CPU to do it (~700%!) that's far better than getput was doing at least, but I still need to profile to see what's driving the usage up so high. Mark

Thanks, Lars

Hi Lars! Replies inline below On 9/9/19 5:59 AM, Lars Marowsky-Bree wrote:

On 2019-08-20T08:53:50, Mark Nelson <mnelson@redhat.com> wrote:

Hi Mark,

sorry for the slow response. Got sidetracked into business travel ;-)

No worries, I've got too many plates in the air right now so the delay is perfectly fine. :)

I looked over https://github.com/markhpc/hsbench features to consider how it compares to the fio S3/Swift/DAV backend.

fio's so far can only target one bucket (it can only be pointed at one http_host, but in theory, that could be a variable that could be filled via variables or different jobs.

That also limits the number of endpoints - it doesn't RR across multiple endpoints for the same job, but fio would easily support kicking off the same job for multiple endpoints concurrently via one control file, so that'd likely achieve something similar?

I think you could definitely craft something for fio that would give you a similar effect if not work in exactly the same way.

The whole latency reporting and test parametrization is, I think, an advantage of fio.

On the hsbench side, I did try to craft the latency reporting piece so that you can ask for arbitrary percentiles.  Right now we just report min/50%/99%/max, but that could be fleshed out like fio's reporting.  One thing that I'm not super fond of in fio is the way interval logging is done, but that's probably somewhat minor in the grand scheme of things.

With one exception - it doesn't really currently support mixing object sizes (block sizes) within one job easily given how I hacked that into fio. I think it's probably safe, since I assume fio only ever reads blocks/objects back with the same size it wrote them, but it's kinda awkward ;-)

hsbench is quite simple in this regard too.  It doesn't do any kind of special mapping of object sizes, or even do anything like fio's random map so that if you are randomly reading objects back it doesn't hit the same one.  In hsbench we certainly could create some kind of object map (and even store and retrieve that map in S3 for future read tests), but currently hsbench is really targeted at put/get/list/delete for homogeneous data sets as well.

What fio also doesn't do is manage the buckets itself; it doesn't support deletion/creation, it only does PUT/GET/DELETE for objects within existing buckets. (Given that bucket creation could potentially require quite a number of options to be passed, that was sort-of intentional - I expected buckets to be provisioned before benchmarking the cluster.)

It turns out after talking to our RGW guys that a really important test here is bucket list throughput.  I added that in hsbench 0.2 and Matt Benjamin has a PR in the works for testing unordered bucket listings as well.  This is especially important right now because with the way sharding works in RGW, you get much higher parallelism in one bucket if you shard it, but you hurt bucket list performance with higher read-amplification.

It can however delete all objects (just "trim" everything afterwards - https://github.com/axboe/fio/blob/master/examples/http-s3.fio

And it doesn't support multipart uploads. But I think that's the same for hsbench so far.

Yeah, no special provisions for multipart yet.

Of course, hsbench gets to benefit from the aws Go bindings. There weren't any lightweight S3 libraries for C, so fio ended up with it's tiny little rewrite of one. (I think swearing might have been involved around getting the authentication right :-D)

I believe it!  hsbench is mostly just a glorified wrapper around the aws go bindings with some timing and statistics aggregation thrown on top.  The whole thing is like 1K lines of code.  You have more control if you own the S3 library too, but I think for hsbench the aws bindings appear to be good enough so far.

I wonder if you've been able to compare both of them yet; I'd be curious which one is more lightweight and can get higher per client performance?

Honestly I haven't had time to try yet.  I'm trying to be a little clever about concurrency and am using atomics in go to keep away from the mutex/locking overhead that I suspect channels would bring, but I wouldn't be terribly surprised if the fio/s3 implementation could be faster if it's not doing anything silly. I've noticed that hsbench can still use a ton of CPU at high throughput rates.  16K gets/s can consume 6+ xeon E5-2650V3 cores (pretty old these days though).  I suspect that a big part of this is on the networking side and how connections are recycled. Probably need to take a bit more time to look at this in detail. In any event, I haven't been totally idle.  Once I got hsbench working well enough to run tests, we started collecting information from our lab that lead to a bunch of work improving RGW.  You can see some of the test results here: https://docs.google.com/spreadsheets/d/1q8MZJo9rp_3Kvs8ARaIxXf_TZbnnB0dRmrq8... https://docs.google.com/spreadsheets/d/17SiwtVWy3jJdeXocFXrcPFvn8F5x1m8525Bw... And some of the PRs to make RGW faster here: https://github.com/ceph/ceph/pull/29980(cls/rgw: [WIP] Make rgw_bucket_list encode bls rather than dir entries, markhpc) https://github.com/ceph/ceph/pull/29943(rgw/rgw_op: Remove get_val from hotpath via legacy options, markhpc) https://github.com/ceph/ceph/pull/29894(rgw/rgw_reshard: Don't dump RGWBucketReshard JSON in process_single_logshard, markhpc) https://github.com/ceph/ceph/pull/29852(rgw: move bucket reshard checks out of write path, cbodley) Casey also recently posted a proposal to dev@ceph.io for allowing writes during resharding which would fix some of those stalls seen in the graphs.  Matt also has a good idea regarding changing the shard hashing behavior to preserve ordering so that bucket listing times should remain fast even with large numbers of shards. Also, if you look at those graphs you'll see a lot of fluctuation on the put side even after those PRs are applied.  Putting the DB/WAL on ramdisk (without bluefs) stabilizes throughput pretty significantly, so it looks like at least on the write path side hsbench is fast enough to showcase that putting rocksdb on optane or a pmem devices might stabilize write throughput rates and potentially improve overall small object write performance.  Once the new community performance nodes are setup that's one of the first tests I intend to run on them.

Now, my desire for having this all be in the same tool across all benchmarks if feasible, but seeing the benefits of using a language for which an Amazon SDK is available ... https://medium.com/learning-the-go-programming-language/calling-go-functions... A fio plugin wrapper around hsbench's S3 module ought to be possible?

I was actually excited to hear about the fio effort because it's a second option to compare results against!  I would be a little sad to lose that if you adopted some of the hsbench code for the backend, but on the other hand I understand why you might want to do it (certainly the bindings are very convenient!). Mark