Hi all,
I recently opened draft PR #66969 [1] which introduces AEAD encryption to librbd to meet stricter security requirements.
Unlike the existing AES-XTS implementation (which preserves length), AEAD ciphers introduce a ciphertext expansion (e.g., a 4KB block becomes 4128 bytes: 4KB data + 32 bytes tag/IV).
The Challenge I need to store this auxiliary data (e.g. authentication tag) alongside the encrypted payload. Crucially, the data and its auth tag must be committed atomically; if they desynchronize, decryption becomes impossible. Since RADOS guarantees atomicity per object, I am evaluating the best intra-object layout to handle this expansion.
The Question I am looking for feedback on which data layout is preferable for the OSD/BlueStore backend. I am considering three options and have benchmarked them on a small development cluster. I would like some architectural validation regarding write amplification and compaction risks, as well as more benchmarking data.
Here are the candidate layouts:
We append the 32-byte tag directly after every 4KB block.
Structure: [ 4KB Data | 32B Tag ][ 4KB Data | 32B Tag ] ...
Pros: Simplicity. Single write operation.
Cons: Breaks 4KB alignment. Every subsequent block writes to a non-4K aligned offset.
My Concern: Does the penalty of unaligned writes on BlueStore outweigh the simplicity?
We write the 4KB encrypted blocks at their original aligned offsets and store the auth. tags in a reserved area at the end of the RADOS object. Similar to the dm-integrity approach.
Structure: [ 4KB Data ][ 4KB Data ] ...[ 32B tag][ 32B tag]...
Pros: Main payload remains 4KB aligned.
Cons: Unaligned IO for managing the tail metadata region + Discontinuous write: Requires updating two non-contiguous offsets within a single atomic Rados transaction (one for data, one for the tag).
We write the 4KB aligned data to the object body and the 32-byte tag to the object's OMAP (RocksDB).
Structure:
Object Body: [ 4KB Data ] [ 4KB Data ] ...
OMAP: Key: Offset -> Value: 32B Tag
Pros: Perfect alignment for data.
Cons: High write amplification and CPU overhead. Every block storage IO requires IO to RocksDB, which is significantly more expensive than a simple appended write to the block device.
My Concern: I suspect this will shift the bottleneck from disk bandwidth to the CPU and RocksDB compaction threads. A high-throughput RBD workload could generate millions of tiny keys, causing aggressive compaction and decreasing performance significantly. E.g. For one 10TB we need to manage 2.5 billion individual key-value pairs.
Benchmarking & Preliminary Verdict I created a small librados benchmark script [2] to benchmark these strategies.
Preliminary Results: I have yet to identify a clearly “best” performing layout. My sample size is still too small.
Personal Preference: I lean towards Inline Layout for its robustness and simplicity, despite the alignment loss. Data loss occurs if the encrypted data and its auxiliary are out of sync. With this layout I am the most confident that this does not occur.
I would appreciate it if anyone with deep knowledge of OSD/BlueStore write paths could weigh in or rbd or rados developers. Is the alignment loss in Option 1 acceptable, or should we rather pursue Option 2 or 3?
It would be crucial if others could run the benchmark script [2] and share the results with me to come to a data driven result. (The script creates RADOS objects, performs I/O, outputs 4 JSON result files, and cleans up. Default size is 16GB, but larger runs would be greatly appreciated).
Links: [1] PR: https://github.com/ceph/ceph/pull/66969 [2] Benchmark Script: https://gist.github.com/Greenpepper15/6a0a0eb1ce33645cfcb55b0be9a05a52
Thanks, David Mohren - Clyso