Interesting, thanks for the details. Adding the dev@ceph.io list. On Thu, Mar 26, 2020 at 12:07 PM <akmd@metaswitch.com> wrote:

Hello,

I am observing non-intuitive results for a performance test using the S3 API to RGW. I am wondering if others have similar experiences or knowledge here.

Our application is using the “if-none-match” header on S3-API requests. This header is set by the application if it already has a copy of the object in question but wishes to check if there is a newer version. If the etag of the current object matches then RGW sends a 304 response, and if it doesn’t it sends the updated content of the object.

We’re observing that the response time of requests resulting in “304 Not Modified” is typically slower than those for normal object retrieval. This wasn’t intuitive to me – in the 304 case there is no content to transfer over the network and I would expect the request can be satisfied just by looking at the RGW index (I was under the impression that metadata including etag is in the index). Anecdotally, HEAD requests see similar results but I haven't yet analysed in full.

The bucket index is generally only consulted when listing objects. For GET/HEAD requests on objects, we read the etag and other object attributes directly from the associated 'head object' in the buckets.data pool. For small objects (ie. non-multipart uploads), this head object also stores the first 4MB of object data. For GET requests, radosgw will prefetch this data in the same osd request that reads the object's attributes, even in the 304 case where no data is returned to the client. The 'head object' concept works differently for versioned buckets, because each object version has its own head object, and the 'current' version's head object (internally called the 'object logical head', or OLH) acts like a symlink to one of those. So a GET/HEAD request that doesn't specify a ?versionId will require (at least*) two rados reads - one on the OLH object to determine the current version, and another on that version's head object. * OLH performance isn't necessarily deterministic, because it may involve replaying a journal of racing writes of new versions.

Does anyone else have data or experience about expected performance of this scenario? Are there any potential avenues for optimization of configuration ? What kind of commands can I use to debug this further ?

Given the above, I would expect the performance to be very similar with and without If-None-Match. I'd suggest running radosgw with '--debug-ms=1' and comparing the number of osd_op requests it sends between the 206 and 304 cases.

Some details of the current setup: => ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) => Objects are typically 80-100KB. => Versioning is enabled on the bucket. => Our requests specify a Range header (hence will generate 206 not 200). => Multisite features are enabled. => Bucket has 20 shards – I’ve put a dump of "bucket limits" below.

Performance results

Response, Request Count, Median, 75th percentile, 90th percentile, 95th percentile, 206 Partial, 20473, 3, 3, 16, 129, 1200 304 Not Modified, 15644, 9, 16, 46, 212, 1192

Bucket details { "bucket": "albansstack-scsdata", "tenant": "", "num_objects": 465780, "num_shards": 20, "objects_per_shard": 23289, "fill_status": "OK" },

Many thanks,

Alistair. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io