Hi all, I recently opened draft PR #66969 [1] which introduces AEAD encryption to librbd to meet stricter security requirements. Unlike the existing AES-XTS implementation (which preserves length), AEAD ciphers introduce a ciphertext expansion (e.g., a 4KB block becomes 4128 bytes: 4KB data + 32 bytes tag/IV). The Challenge I need to store this auxiliary data (e.g. authentication tag) alongside the encrypted payload. Crucially, the data and its auth tag must be committed atomically; if they desynchronize, decryption becomes impossible. Since RADOS guarantees atomicity per object, I am evaluating the best intra-object layout to handle this expansion. The Question I am looking for feedback on which data layout is preferable for the OSD/BlueStore backend. I am considering three options and have benchmarked them on a small development cluster. I would like some architectural validation regarding write amplification and compaction risks, as well as more benchmarking data. Here are the candidate layouts: Option 1: Inline Layout (Current PR Approach) We append the 32-byte tag directly after every 4KB block. - Structure: [ 4KB Data | 32B Tag ][ 4KB Data | 32B Tag ] ... - Pros: Simplicity. Single write operation. - Cons: Breaks 4KB alignment. Every subsequent block writes to a non-4K aligned offset. - My Concern: Does the penalty of unaligned writes on BlueStore outweigh the simplicity? Option 2: Split Layout (Data + Tail Metadata) We write the 4KB encrypted blocks at their original aligned offsets and store the auth. tags in a reserved area at the end of the RADOS object. Similar to the dm-integrity approach. - Structure: [ 4KB Data ][ 4KB Data ] ...[ 32B tag][ 32B tag]... - Pros: Main payload remains 4KB aligned. - Cons: Unaligned IO for managing the tail metadata region + Discontinuous write: Requires updating two non-contiguous offsets within a single atomic Rados transaction (one for data, one for the tag). Option 3: Hybrid Layout (Data + OMAP) We write the 4KB aligned data to the object body and the 32-byte tag to the object's OMAP (RocksDB). - Structure: - Object Body: [ 4KB Data ] [ 4KB Data ] ... - OMAP: Key: Offset -> Value: 32B Tag - Pros: Perfect alignment for data. - Cons: High write amplification and CPU overhead. Every block storage IO requires IO to RocksDB, which is significantly more expensive than a simple appended write to the block device. - My Concern: I suspect this will shift the bottleneck from disk bandwidth to the CPU and RocksDB compaction threads. A high-throughput RBD workload could generate millions of tiny keys, causing aggressive compaction and decreasing performance significantly. E.g. For one 10TB we need to manage 2.5 billion individual key-value pairs. Benchmarking & Preliminary Verdict I created a small librados benchmark script [2] to benchmark these strategies. - Preliminary Results: I have yet to identify a clearly “best” performing layout. My sample size is still too small. - Personal Preference: I lean towards Inline Layout for its robustness and simplicity, despite the alignment loss. Data loss occurs if the encrypted data and its auxiliary are out of sync. With this layout I am the most confident that this does not occur. I would appreciate it if anyone with deep knowledge of OSD/BlueStore write paths could weigh in or rbd or rados developers. Is the alignment loss in Option 1 acceptable, or should we rather pursue Option 2 or 3? It would be crucial if others could run the benchmark script [2] and share the results with me to come to a data driven result. (The script creates RADOS objects, performs I/O, outputs 4 JSON result files, and cleans up. Default size is 16GB, but larger runs would be greatly appreciated). Links: [1] PR: https://github.com/ceph/ceph/pull/66969 [2] Benchmark Script: https://gist.github.com/Greenpepper15/6a0a0eb1ce33645cfcb55b0be9a05a52 Thanks, David Mohren - Clyso