How OSD encryption affects latency/iops on NVMe, SSD and HDD
Hi all, For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some info on this subject but there isn't much detail available. From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast NVMe. Using aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys. Best, Tri Hoang
On Sat, Sep 26, 2020 at 04:50:34PM +0000, tri@postix.net wrote:
Hi all,
For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some info on this subject but there isn't much detail available.
From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast NVMe. Using aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys.
There's two things to point out as improvements for you. 1. CloudFlare's writeup about reducing latency in dm-crypt earlier this year: https://blog.cloudflare.com/speeding-up-linux-disk-encryption/ 2. An internal observation, that I don't think was well published yet, that disabling CONFIG_CRYPTO_STATS may provide significant CPU reduction (CPU time spent in crypto_stats_* specifically). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Some tests on dmcrypted (aes-xts-plain64, 512 bit) vs non-dmcrypted on a small SAS SSD drive. Latencies are reported at 99.9 percentile fio 4k, direct, sync, QD1 ========================== WRITE READ IOPS LATENCIES(us) IOPS LATENCIES(us) Base 17.5k 85 20.4k 79 Encrypted 5.58k 685 10.2k 206 fio 4k, direct, sync, QD32 ========================== WRITE READ IOPS LATENCIES(us) IOPS LATENCIES(us) Base 65.2k 1156 93.4k 742 Encrypted 52.7k 2442 65.2k 1123 fio 4k, direct, sync, QD128 ========================== WRITE READ IOPS LATENCIES(us) IOPS LATENCIES(us) Base 65.6k 4686 94.6k 2835 Encrypted 55.9k 12780 74.7k 3687 fio 4k, direct, sync, QD1, jobs=8 =================================== WRITE READ IOPS LATENCIES(us) IOPS LATENCIES(us) Base 51.6k 1336 53.7k 273 Encrypted 24.8k 1205 43.7k 367 It looks like the biggest encryption penalty is at 4k, QD1. So perhaps the journal is most impacted (block.db and wal.db). Since the iops on the journal limit an OSD's overall iops, the real impact is at 4k/QD1. I suspect that if one could tolerate an unencrypted journal with an encrypted data OSD (as in bluestore), the overall penalty could be lower, perhaps around 20-30%. The penalty could be much larger with faster (RAM, nvme) drives. Cloudflare did a similar test and found that it could be as much as 7x. September 26, 2020 12:50 PM, tri@postix.net wrote:
Hi all,
For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some info on this subject but there isn't much detail available.
From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast NVMe. Using aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys.
Best,
Tri Hoang _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I did also some testing, but was more surprised how much cputime kworker and dmcrypt-write(?) instances are taking. Is there some way to get fio output realtime to influx or prometheus so you can view it with load together? -----Original Message----- From: tri@postix.net [mailto:tri@postix.net] Sent: maandag 28 september 2020 16:07 To: ceph-users@ceph.io Subject: [ceph-users] Re: How OSD encryption affects latency/iops on NVMe, SSD and HDD Some tests on dmcrypted (aes-xts-plain64, 512 bit) vs non-dmcrypted on a small SAS SSD drive. Latencies are reported at 99.9 percentile fio 4k, direct, sync, QD1 ========================== WRITE READ IOPS LATENCIES(us) IOPS LATENCIES(us) Base 17.5k 85 20.4k 79 Encrypted 5.58k 685 10.2k 206 fio 4k, direct, sync, QD32 ========================== WRITE READ IOPS LATENCIES(us) IOPS LATENCIES(us) Base 65.2k 1156 93.4k 742 Encrypted 52.7k 2442 65.2k 1123 fio 4k, direct, sync, QD128 ========================== WRITE READ IOPS LATENCIES(us) IOPS LATENCIES(us) Base 65.6k 4686 94.6k 2835 Encrypted 55.9k 12780 74.7k 3687 fio 4k, direct, sync, QD1, jobs=8 =================================== WRITE READ IOPS LATENCIES(us) IOPS LATENCIES(us) Base 51.6k 1336 53.7k 273 Encrypted 24.8k 1205 43.7k 367 It looks like the biggest encryption penalty is at 4k, QD1. So perhaps the journal is most impacted (block.db and wal.db). Since the iops on the journal limit an OSD's overall iops, the real impact is at 4k/QD1. I suspect that if one could tolerate an unencrypted journal with an encrypted data OSD (as in bluestore), the overall penalty could be lower, perhaps around 20-30%. The penalty could be much larger with faster (RAM, nvme) drives. Cloudflare did a similar test and found that it could be as much as 7x. September 26, 2020 12:50 PM, tri@postix.net wrote:
Hi all,
For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some info on this subject but there isn't much detail available.
From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast NVMe. Using aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys.
Best,
Tri Hoang _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Marc Roos
-
Robin H. Johnson
-
tri＠postix.net