-----Original Message----- From: Casey Bodley <cbodley(a)redhat.com> Sent: Wednesday, October 12, 2022 11:11 PM To: Feng, Hualong <hualong.feng(a)intel.com> Cc: Mark Kogan <mkogan(a)redhat.com>om>; Tang, Guifeng <guifeng.tang(a)intel.com>om>; Ma, Jianpeng <jianpeng.ma(a)intel.com>om>; dev(a)ceph.io Subject: Re: RGW encrypt is implemented by qat batch and queue mode On Thu, Sep 22, 2022 at 9:31 PM Feng, Hualong <hualong.feng(a)intel.com> wrote: reached, a batch submission is performed. synchronous interface. If we don't want the plugin to contain condition variables, we need to implement the plugin in an asynchronous way and provide an asynchronous interface. This requires the RGW to call the interface to make changes. number of threads. The number of QAT instance(hardware resources) is limited. When the number of instances is less than the number of threads, we still need to wait for the free instance. If the asynchronous interface is used, we can use the queue as a buffer to avoid blocking the current thread while waiting for a free instance. condition variable. Do you have a better suggestion here? below you suggest that we could fall back to CPU processing for small object uploads. could we use that same fallback in the cases where we'd otherwise have to block waiting for a QAT instance? CHUNK_SIZE, from the current 4096B, to a bigger one? sorry not the CHUNK_SIZE, but the total amount of data we can feed into QAT at a time. i see in https://github.com/ceph/ceph/pull/47040 that you've found the loop in AES_256_CBC::cbc_transform() which breaks the input into CHUNK_SIZEd pieces, and you converted that loop into a single batch() call - that part looks great if each call to cbc_transform() is getting large enough buffers, it could acquire exclusive access to one QAT instance, feed all of its data through it, then release the instance back to a pool. for a large object workload it seems like this strategy would best utilize the hardware resources, because you never have to coordinate a single batch across several threads. you just need to acquire/release access to a QAT instance every 4MB, which you can use for 32 batches (assuming batch size is 32*4k=128k?) at a time calling the encryption function is not necessarily 4MB. Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request ,Sig nedHeaders=content-encoding;content-length;host;x-amz-content-sha256;x > -amz-date;x-amz-decoded-content-length;x-amz-storage-class,Signature=4 f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9 > --------------- 10000;chunk-signature=ad80c730a21e5b8d04586a2213dd63b9a0e99e0e230 7b0ad > --------------- 10000;chunk-signature=ad80c730a21e5b8d04586a2213dd63b9a0e99e0e230 7b0ad are you sure that this http-level chunking has an effect on the buffer sizes that encryption sees? it may cause the buffers to be segmented at 64k, but encryption and decryption both call bufferlist::c_str() to reallocate a single contiguous buffer: https://github.com/ceph/ceph/blob/9aa8bed/src/rgw/rgw_crypt.cc#L490 so i'd still expect this loop in RGWPutObj::execute() to read up to rgw_max_chunk_size at a time: https://github.com/ceph/ceph/blob/fc01eeb7/src/rgw/rgw_op.cc#L4111-L41 41 if there are cases where the RGWPutObj_BlockEncrypt filter isn't getting large enough buffers, we can use the same strategy as https://github.com/ceph/ceph/pull/21479, where we improved compression ratios by adding a buffering filter in front > > that could avoid the need for cross-thread queues and > > synchronization. compared to your approach in > > https://github.com/ceph/ceph/pull/47845, i imagine this would show > > less of a benefit for small object uploads, but more of a benefit > > for the big ones. do you think this could work? > If in order to avoid the need for cross-thread queues and show less of a benefit for small object uploads, we can turn small objects to CPU processing. Only for big object, we use QAT batch api. > Hi Casey > Thanks for your reply. The detail message and some question are above. > Thanks > -Hualong all of my feedback here relates to large objects, though you've really focused on small objects in https://github.com/ceph/ceph/pull/47845. for small object workloads, i do agree that the queuing and thread synchronization is necessary to take advantage of this batching it's just hard for me to tell whether that extra complexity is worth it. we've tried to minimize any synchronization between rgw's requests so that we're able to scale to thousands of concurrent requests/connections. at scale, i'd worry that lock contention here would negate some of the gains from QAT in workloads with a mix of small and large objects, i think we'd make the best use of QAT if we applied it to the larger objects (>= 128k?) where we can use it most efficiently