Ha, I hadn't thought to check if that existed which was probably pretty short-sighted on my part. :) I suppose the good news is that it might be a good candidate for comparison, and once I've implemented an S3 client endpoint in CBT I should be able to target both pretty easily. Mark On 8/15/19 6:41 PM, David Byte wrote: > Mark, did the S3 engine for fio not work? > > Sent from my iPhone. Typos are Apple's fault. > >> On Aug 15, 2019, at 6:37 PM, Mark Nelson <mark.a.nelson(a)gmail.com> wrote: >> >> Hi Guys, >> >> Earlier this week I was working on investigating the impact of OMAP performance on RGW and wanted to see if putting rocksdb on ramdisk would help speed up bucket index updates. While running tests I found out that the benchmark tool I was using consumed roughly 15 cores of CPU to push 4K puts/second to RGW from 128 threads. That wasn't really viable, so I started looking for alternate S3 benchmarking tools. COSBench is sort of the most well known choice out there, but it's a bit cumbersome if you just want to run some quick tests from the command-line. >> >> I managed to find a simple yet very nice benchmark written in go developed by Wasabi Inc called s3-benchmark. While it works well, it really only targets single buckets and is designed more for AWS testing than for Ceph. I forked the project and pretty much refactored the whole thing to be more useful for the kind of testing I want to do. It's now at the point where I think it might be ready to experiment with. S3 benchmarking has been a semi-recurring topic on the list so I figured other folks might be interested in trying it too and hopefully providing feedback. >> >> The new benchmark is called hsbench and it's available here: >> >> https://github.com/markhpc/hsbench >> >> See the README.md for some of the advantages over the original s3-benchmark program it was forked from. I consider this release to be alpha level quality and disclaim all responsibility if it breaks and deletes every object in all of your buckets. Consider yourself warned. :) >> >> Mark >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >> > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

On 8/20/19 7:58 AM, Lars Marowsky-Bree wrote: Indeed!  Now that we have client endpoints and the universal fio wrapper in cbt it should be fairly easy to add that as another supported option there too.   I'll be very curious to see how hsbench and fio compare in terms of client side CPU usage and overall throughput.  I'm pushing about 15K 4K gets with hsbench right now at sufficiently high thread counts, but it's still using a lot of CPU to do it (~700%!) that's far better than getput was doing at least, but I still need to profile to see what's driving the usage up so high. Mark > > > Thanks, > Lars >

Hi Lars! Replies inline below On 9/9/19 5:59 AM, Lars Marowsky-Bree wrote: No worries, I've got too many plates in the air right now so the delay is perfectly fine. :) I think you could definitely craft something for fio that would give you a similar effect if not work in exactly the same way. On the hsbench side, I did try to craft the latency reporting piece so that you can ask for arbitrary percentiles.  Right now we just report min/50%/99%/max, but that could be fleshed out like fio's reporting.  One thing that I'm not super fond of in fio is the way interval logging is done, but that's probably somewhat minor in the grand scheme of things. hsbench is quite simple in this regard too.  It doesn't do any kind of special mapping of object sizes, or even do anything like fio's random map so that if you are randomly reading objects back it doesn't hit the same one.  In hsbench we certainly could create some kind of object map (and even store and retrieve that map in S3 for future read tests), but currently hsbench is really targeted at put/get/list/delete for homogeneous data sets as well. It turns out after talking to our RGW guys that a really important test here is bucket list throughput.  I added that in hsbench 0.2 and Matt Benjamin has a PR in the works for testing unordered bucket listings as well.  This is especially important right now because with the way sharding works in RGW, you get much higher parallelism in one bucket if you shard it, but you hurt bucket list performance with higher read-amplification. Yeah, no special provisions for multipart yet. I believe it!  hsbench is mostly just a glorified wrapper around the aws go bindings with some timing and statistics aggregation thrown on top.  The whole thing is like 1K lines of code.  You have more control if you own the S3 library too, but I think for hsbench the aws bindings appear to be good enough so far. Honestly I haven't had time to try yet.  I'm trying to be a little clever about concurrency and am using atomics in go to keep away from the mutex/locking overhead that I suspect channels would bring, but I wouldn't be terribly surprised if the fio/s3 implementation could be faster if it's not doing anything silly. I've noticed that hsbench can still use a ton of CPU at high throughput rates.  16K gets/s can consume 6+ xeon E5-2650V3 cores (pretty old these days though).  I suspect that a big part of this is on the networking side and how connections are recycled. Probably need to take a bit more time to look at this in detail. In any event, I haven't been totally idle.  Once I got hsbench working well enough to run tests, we started collecting information from our lab that lead to a bunch of work improving RGW.  You can see some of the test results here: https://docs.google.com/spreadsheets/d/1q8MZJo9rp_3Kvs8ARaIxXf_TZbnnB0dRmrq… https://docs.google.com/spreadsheets/d/17SiwtVWy3jJdeXocFXrcPFvn8F5x1m8525B… And some of the PRs to make RGW faster here: https://github.com/ceph/ceph/pull/29980(cls/rgw: [WIP] Make rgw_bucket_list encode bls rather than dir entries, markhpc) https://github.com/ceph/ceph/pull/29943(rgw/rgw_op: Remove get_val from hotpath via legacy options, markhpc) https://github.com/ceph/ceph/pull/29894(rgw/rgw_reshard: Don't dump RGWBucketReshard JSON in process_single_logshard, markhpc) https://github.com/ceph/ceph/pull/29852(rgw: move bucket reshard checks out of write path, cbodley) Casey also recently posted a proposal to dev(a)ceph.io for allowing writes during resharding which would fix some of those stalls seen in the graphs.  Matt also has a good idea regarding changing the shard hashing behavior to preserve ordering so that bucket listing times should remain fast even with large numbers of shards. Also, if you look at those graphs you'll see a lot of fluctuation on the put side even after those PRs are applied.  Putting the DB/WAL on ramdisk (without bluefs) stabilizes throughput pretty significantly, so it looks like at least on the write path side hsbench is fast enough to showcase that putting rocksdb on optane or a pmem devices might stabilize write throughput rates and potentially improve overall small object write performance.  Once the new community performance nodes are setup that's one of the first tests I intend to run on them. I was actually excited to hear about the fio effort because it's a second option to compare results against!  I would be a little sad to lose that if you adopted some of the hsbench code for the backend, but on the other hand I understand why you might want to do it (certainly the bindings are very convenient!). Mark