26 Sep
2020
26 Sep
'20
9:50 p.m.
Hi all,
For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD and
HDD vs non-encrypted OSDs? I tried to find some info on this subject but there isn't
much detail available.
From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast
NVMe. Using aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys.
Best,
Tri Hoang
27 Sep
27 Sep
3:55 a.m.
On Sat, Sep 26, 2020 at 04:50:34PM +0000, tri(a)postix.net wrote:
Hi all,
For those who use encryption on your OSDs, what effect do you see on
your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some
info on this subject but there isn't much detail available.
>From experience, dmcrypt is CPU-bound and becomes a bottleneck when
>used on very fast NVMe. Using aes-xts, one can only expect around
>1600-2000GB/s with 256/512 bit keys.
There's two things to point out as improvements for you.
1. CloudFlare's writeup about reducing latency in dm-crypt earlier this
year:
https://blog.cloudflare.com/speeding-up-linux-disk-encryption/
2. An internal observation, that I don't think was well published yet,
that disabling CONFIG_CRYPTO_STATS may provide significant CPU reduction
(CPU time spent in crypto_stats_* specifically).
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2(a)gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
28 Sep
28 Sep
7:06 p.m.
Some tests on dmcrypted (aes-xts-plain64, 512 bit) vs non-dmcrypted on a small SAS SSD
drive. Latencies are reported at 99.9 percentile
fio 4k, direct, sync, QD1
==========================
WRITE READ
IOPS LATENCIES(us) IOPS LATENCIES(us)
Base 17.5k 85 20.4k 79
Encrypted 5.58k 685 10.2k 206
fio 4k, direct, sync, QD32
==========================
WRITE READ
IOPS LATENCIES(us) IOPS LATENCIES(us)
Base 65.2k 1156 93.4k 742
Encrypted 52.7k 2442 65.2k 1123
fio 4k, direct, sync, QD128
==========================
WRITE READ
IOPS LATENCIES(us) IOPS LATENCIES(us)
Base 65.6k 4686 94.6k 2835
Encrypted 55.9k 12780 74.7k 3687
fio 4k, direct, sync, QD1, jobs=8
===================================
WRITE READ
IOPS LATENCIES(us) IOPS LATENCIES(us)
Base 51.6k 1336 53.7k 273
Encrypted 24.8k 1205 43.7k 367
It looks like the biggest encryption penalty is at 4k, QD1. So perhaps the journal is most
impacted (block.db and wal.db). Since the iops on the journal limit an OSD's overall
iops, the real impact is at 4k/QD1. I suspect that if one could tolerate an unencrypted
journal with an encrypted data OSD (as in bluestore), the overall penalty could be lower,
perhaps around 20-30%.
The penalty could be much larger with faster (RAM, nvme) drives. Cloudflare did a similar
test and found that it could be as much as 7x.
September 26, 2020 12:50 PM, tri(a)postix.net wrote:
> Hi all,
>
> For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD
and HDD vs
> non-encrypted OSDs? I tried to find some info on this subject but there isn't
much detail
> available.
>
> From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast
NVMe. Using
> aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys.
>
> Best,
>
> Tri Hoang
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:12 p.m.
I did also some testing, but was more surprised how much cputime kworker
and dmcrypt-write(?) instances are taking. Is there some way to get fio
output realtime to influx or prometheus so you can view it with load
together?
-----Original Message-----
From: tri(a)postix.net [mailto:tri@postix.net]
Sent: maandag 28 september 2020 16:07
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: How OSD encryption affects latency/iops on
NVMe, SSD and HDD
Some tests on dmcrypted (aes-xts-plain64, 512 bit) vs non-dmcrypted on a
small SAS SSD drive. Latencies are reported at 99.9 percentile
fio 4k, direct, sync, QD1
==========================
WRITE READ
IOPS LATENCIES(us) IOPS LATENCIES(us)
Base 17.5k 85 20.4k 79
Encrypted 5.58k 685 10.2k 206
fio 4k, direct, sync, QD32
==========================
WRITE READ
IOPS LATENCIES(us) IOPS LATENCIES(us)
Base 65.2k 1156 93.4k 742
Encrypted 52.7k 2442 65.2k 1123
fio 4k, direct, sync, QD128
==========================
WRITE READ
IOPS LATENCIES(us) IOPS LATENCIES(us)
Base 65.6k 4686 94.6k 2835
Encrypted 55.9k 12780 74.7k 3687
fio 4k, direct, sync, QD1, jobs=8
===================================
WRITE READ
IOPS LATENCIES(us) IOPS LATENCIES(us)
Base 51.6k 1336 53.7k 273
Encrypted 24.8k 1205 43.7k 367
It looks like the biggest encryption penalty is at 4k, QD1. So perhaps
the journal is most impacted (block.db and wal.db). Since the iops on
the journal limit an OSD's overall iops, the real impact is at 4k/QD1. I
suspect that if one could tolerate an unencrypted journal with an
encrypted data OSD (as in bluestore), the overall penalty could be
lower, perhaps around 20-30%.
The penalty could be much larger with faster (RAM, nvme) drives.
Cloudflare did a similar test and found that it could be as much as 7x.
September 26, 2020 12:50 PM, tri(a)postix.net wrote:
Hi all,
For those who use encryption on your OSDs, what effect do you see on
your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some
info on this subject but there isn't much detail available.
From experience, dmcrypt is CPU-bound and becomes a bottleneck when
used on very fast NVMe. Using aes-xts, one can only expect around
1600-2000GB/s
with 256/512 bit keys.
Best,
Tri Hoang
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
1299
days inactive
1301
days old
3 comments
3 participants
participants (3)
-
Marc Roos -
Robin H. Johnson -
tri＠postix.net