Encryption per user Howto
Dear Ceph folks, Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store. Does anyone know (or have experience) how to do with CephFS and Ceph RGW? Any suggestionns or comments are highly appreciated, best regards, Samuel huxiaoyu@horebdata.cn
Hello Samuel, On Sun, May 21, 2023 at 3:48 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Ceph folks,
Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store.
Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
For CephFS, this is unachievable. For RGW, please use Vault for storing encryption keys. Don't forget about the proper high-availability setup. Use an AppRole to manage tokens. Use Vault Agent as a proxy that adds the token to requests issued by RGWs. Then create a bucket for each user and set the encryption policy for this bucket using the PutBucketEncryption API that is available through AWS CLI. Either SSE-S3 or SSE-KMS will work for you. SSE-S3 is easier to manage. Each object will then be encrypted using a different key derived from its name and a per-bucket master key which never leaves Vault. Note that users will be able to create additional buckets by themselves, and they won't be encrypted, so tell them either not to do that or to encrypt the new buckets similarly. -- Alexander E. Patrakov
Dear Alexander, Thanks a lot for helpful comments and insights. Regarding CephFS and RGW, Per user seems to be daunting and complex. What if encryption on the server side without per user requirment? would it be relatively easy to achieve, and how? best regards, Samuel huxiaoyu@horebdata.cn From: Alexander E. Patrakov Date: 2023-05-21 15:44 To: huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Encryption per user Howto Hello Samuel, On Sun, May 21, 2023 at 3:48 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Ceph folks,
Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store.
Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
For CephFS, this is unachievable. For RGW, please use Vault for storing encryption keys. Don't forget about the proper high-availability setup. Use an AppRole to manage tokens. Use Vault Agent as a proxy that adds the token to requests issued by RGWs. Then create a bucket for each user and set the encryption policy for this bucket using the PutBucketEncryption API that is available through AWS CLI. Either SSE-S3 or SSE-KMS will work for you. SSE-S3 is easier to manage. Each object will then be encrypted using a different key derived from its name and a per-bucket master key which never leaves Vault. Note that users will be able to create additional buckets by themselves, and they won't be encrypted, so tell them either not to do that or to encrypt the new buckets similarly. -- Alexander E. Patrakov
rgw supports the 3 flavors of S3 Server-Side Encryption, along with the PutBucketEncryption api for per-bucket default encryption. you can find the docs in https://docs.ceph.com/en/quincy/radosgw/encryption/ On Mon, May 22, 2023 at 10:49 AM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Alexander,
Thanks a lot for helpful comments and insights. Regarding CephFS and RGW, Per user seems to be daunting and complex.
What if encryption on the server side without per user requirment? would it be relatively easy to achieve, and how?
best regards,
Samuel
huxiaoyu@horebdata.cn
From: Alexander E. Patrakov Date: 2023-05-21 15:44 To: huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Encryption per user Howto Hello Samuel,
On Sun, May 21, 2023 at 3:48 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Ceph folks,
Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store.
Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
For CephFS, this is unachievable.
For RGW, please use Vault for storing encryption keys. Don't forget about the proper high-availability setup. Use an AppRole to manage tokens. Use Vault Agent as a proxy that adds the token to requests issued by RGWs. Then create a bucket for each user and set the encryption policy for this bucket using the PutBucketEncryption API that is available through AWS CLI. Either SSE-S3 or SSE-KMS will work for you. SSE-S3 is easier to manage. Each object will then be encrypted using a different key derived from its name and a per-bucket master key which never leaves Vault.
Note that users will be able to create additional buckets by themselves, and they won't be encrypted, so tell them either not to do that or to encrypt the new buckets similarly.
-- Alexander E. Patrakov
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 5/21/23 15:44, Alexander E. Patrakov wrote:
Hello Samuel,
On Sun, May 21, 2023 at 3:48 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Ceph folks,
Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store.
Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
For CephFS, this is unachievable.
For a couple of years already, work is being done to have fscrypt support for CephFS [1]. When that work ends up in mainline kernel (and distro kernels at some point) this will be possible. Gr. Stefan [1]: https://lwn.net/Articles/829448/
Hi, Stefan, Thanks a lot for the message. It seems that client-side encryption (or per use) is still on the way and not ready yet for today. Are there practical methods to implement encryption for CephFS with today' technique? e.g using LUKS or other tools? Kind regards, Samuel huxiaoyu@horebdata.cn From: Stefan Kooman Date: 2023-05-22 17:19 To: Alexander E. Patrakov; huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Re: Encryption per user Howto On 5/21/23 15:44, Alexander E. Patrakov wrote:
Hello Samuel,
On Sun, May 21, 2023 at 3:48 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Ceph folks,
Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store.
Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
For CephFS, this is unachievable.
For a couple of years already, work is being done to have fscrypt support for CephFS [1]. When that work ends up in mainline kernel (and distro kernels at some point) this will be possible. Gr. Stefan [1]: https://lwn.net/Articles/829448/
Hm, this thread is confusing in the context of S3 client-side encryption means - the user is responsible to encrypt the data with their own keys before submitting it. As far as I'm aware, client-side encryption doesn't require any specific server support - it's a function of the client SDK used which provides the convenience of encrypting your data before upload and decryptiing it after download - https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingClientSideEncrypt... But you can always encrypt your data and then upload it via RGW, there is nothing anywhere that prevents that since uploaded objects are just a sequence of bytes, meta data won't be encrypted then You can also do server-side encryption by bringing your own keys - https://docs.ceph.com/en/quincy/radosgw/encryption/#customer-provided-keys I suspect you're asking for server-side encryption with keys managed by ceph on a per-user basis? On Tue, 23 May 2023 at 03:28, huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Hi, Stefan,
Thanks a lot for the message. It seems that client-side encryption (or per use) is still on the way and not ready yet for today.
Are there practical methods to implement encryption for CephFS with today' technique? e.g using LUKS or other tools?
Kind regards,
Samuel
huxiaoyu@horebdata.cn
Hello Samuel,
On Sun, May 21, 2023 at 3:48 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Ceph folks,
Recently one of our clients approached us with a request on encrpytion
From: Stefan Kooman Date: 2023-05-22 17:19 To: Alexander E. Patrakov; huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Re: Encryption per user Howto On 5/21/23 15:44, Alexander E. Patrakov wrote: per user, i.e. using individual encrytion key for each user and encryption files and object store.
Does anyone know (or have experience) how to do with CephFS and Ceph
RGW?
For CephFS, this is unachievable.
For a couple of years already, work is being done to have fscrypt support for CephFS [1]. When that work ends up in mainline kernel (and distro kernels at some point) this will be possible.
Gr. Stefan
[1]: https://lwn.net/Articles/829448/
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Indeed, the question is on server-side encryption with keys managed by ceph on a per-user basis From: Christian Wuerdig Date: 2023-05-23 00:51 To: huxiaoyu@horebdata.cn CC: Stefan Kooman; ceph-users Subject: Re: [ceph-users] Re: Encryption per user Howto Hm, this thread is confusing in the context of S3 client-side encryption means - the user is responsible to encrypt the data with their own keys before submitting it. As far as I'm aware, client-side encryption doesn't require any specific server support - it's a function of the client SDK used which provides the convenience of encrypting your data before upload and decryptiing it after download - https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingClientSideEncrypt... But you can always encrypt your data and then upload it via RGW, there is nothing anywhere that prevents that since uploaded objects are just a sequence of bytes, meta data won't be encrypted then You can also do server-side encryption by bringing your own keys - https://docs.ceph.com/en/quincy/radosgw/encryption/#customer-provided-keys I suspect you're asking for server-side encryption with keys managed by ceph on a per-user basis? On Tue, 23 May 2023 at 03:28, huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote: Hi, Stefan, Thanks a lot for the message. It seems that client-side encryption (or per use) is still on the way and not ready yet for today. Are there practical methods to implement encryption for CephFS with today' technique? e.g using LUKS or other tools? Kind regards, Samuel huxiaoyu@horebdata.cn From: Stefan Kooman Date: 2023-05-22 17:19 To: Alexander E. Patrakov; huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Re: Encryption per user Howto On 5/21/23 15:44, Alexander E. Patrakov wrote:
Hello Samuel,
On Sun, May 21, 2023 at 3:48 PM huxiaoyu@horebdata.cn <huxiaoyu@horebdata.cn> wrote:
Dear Ceph folks,
Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store.
Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
For CephFS, this is unachievable.
For a couple of years already, work is being done to have fscrypt support for CephFS [1]. When that work ends up in mainline kernel (and distro kernels at some point) this will be possible. Gr. Stefan [1]: https://lwn.net/Articles/829448/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 23.05.23 08:42, huxiaoyu@horebdata.cn wrote:
Indeed, the question is on server-side encryption with keys managed by ceph on a per-user basis
What kind of security to you want to achieve with encryption keys stored on the server side? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Hi all, jumping on this thread as we have requests for which per-client fs mount encryption makes a lot of sense:
What kind of security to you want to achieve with encryption keys stored on the server side?
One of the use cases is if a user requests a share with encryption at rest. Since encryption has an unavoidable performance impact, it is impractical to make 100% of users pay for the requirements that only 1% of users really have. Instead of all-OSD back-end encryption hitting everyone for little reason, encrypting only some user-buckets/fs-shares on the front-end application level will ensure that the data is encrypted at rest. It may very well not serve any other purpose, but these are requests we get. If I could provide an encryption key to a ceph-fs kernel at mount time, this requirement could be solved very elegantly on a per-user (request) basis and only making users who want it pay with performance penalties. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Tuesday, May 23, 2023 6:35 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Encryption per user Howto On 23.05.23 08:42, huxiaoyu@horebdata.cn wrote:
Indeed, the question is on server-side encryption with keys managed by ceph on a per-user basis
What kind of security to you want to achieve with encryption keys stored on the server side? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 5/26/23 12:26, Frank Schilder wrote:
It may very well not serve any other purpose, but these are requests we get. If I could provide an encryption key to a ceph-fs kernel at mount time, this requirement could be solved very elegantly on a per-user (request) basis and only making users who want it pay with performance penalties.
I understand this use case. But this would still mean that the client encrypts the data. In your case the CephFS mount or with S3 the rados-gateway. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert.
But this would still mean that the client encrypts the data.
Yes and as far as I understood this would be fine for the original request as well. Maybe this might sound confusing, but here is my terminology for that: I don't count the RGW daemon as a storage server, in my terminology its a storage gateway, which in itself is a client of the rados back-end store. Hence, I count encryption on a gateway as client-sided. For RGW the natural place to have keys for such encryption would be the gateway (which was called server-sided in an earlier e-mail), while for cephfs if would be on the machine that does the actual FS mount. For the kclient, this would be the host itself and when using ganesha, it would have to be in the VFS config on the NFS gateway. All these I count under client-sided keys while others might consider a gateway as server-sided. Note that client is not the same as user. The key point here is, that ordinary (end-) users will in none of these cases be aware of the encryption or able to bypass it. It happens transparently. It is still on application level and, therefore, can be applied selectively. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, May 26, 2023 1:29 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Encryption per user Howto On 5/26/23 12:26, Frank Schilder wrote:
It may very well not serve any other purpose, but these are requests we get. If I could provide an encryption key to a ceph-fs kernel at mount time, this requirement could be solved very elegantly on a per-user (request) basis and only making users who want it pay with performance penalties.
I understand this use case. But this would still mean that the client encrypts the data. In your case the CephFS mount or with S3 the rados-gateway. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Frank, On Fri, May 26, 2023 at 6:27 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
jumping on this thread as we have requests for which per-client fs mount encryption makes a lot of sense:
What kind of security to you want to achieve with encryption keys stored on the server side?
One of the use cases is if a user requests a share with encryption at rest. Since encryption has an unavoidable performance impact, it is impractical to make 100% of users pay for the requirements that only 1% of users really have. Instead of all-OSD back-end encryption hitting everyone for little reason, encrypting only some user-buckets/fs-shares on the front-end application level will ensure that the data is encrypted at rest.
I would disagree about the unavoidable performance impact of at-rest encryption of OSDs. Read the CloudFlare blog article which shows how they make the encryption impact on their (non-Ceph) drives negligible: https://blog.cloudflare.com/speeding-up-linux-disk-encryption/. The main part of their improvements (the ability to disable dm-crypt workqueues) is already in the mainline kernel. There is also a Ceph pull request that disables dm-crypt workqueues on certain drives: https://github.com/ceph/ceph/pull/49554 While the other part of the performance enhancements authored by CloudFlare (namely, the "xtsproxy" module) is not mainlined yet, I hope that some equivalent solution will find its way into the official kernel sooner or later. In summary: just encrypt everything.
It may very well not serve any other purpose, but these are requests we get. If I could provide an encryption key to a ceph-fs kernel at mount time, this requirement could be solved very elegantly on a per-user (request) basis and only making users who want it pay with performance penalties.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Tuesday, May 23, 2023 6:35 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Encryption per user Howto
On 23.05.23 08:42, huxiaoyu@horebdata.cn wrote:
Indeed, the question is on server-side encryption with keys managed by ceph on a per-user basis
What kind of security to you want to achieve with encryption keys stored on the server side?
Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de
Tel: 030-405051-43 Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov
On Sat, May 27, 2023 at 5:09 AM Alexander E. Patrakov <patrakov@gmail.com> wrote:
Hello Frank,
On Fri, May 26, 2023 at 6:27 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
jumping on this thread as we have requests for which per-client fs mount encryption makes a lot of sense:
What kind of security to you want to achieve with encryption keys stored on the server side?
One of the use cases is if a user requests a share with encryption at rest. Since encryption has an unavoidable performance impact, it is impractical to make 100% of users pay for the requirements that only 1% of users really have. Instead of all-OSD back-end encryption hitting everyone for little reason, encrypting only some user-buckets/fs-shares on the front-end application level will ensure that the data is encrypted at rest.
I would disagree about the unavoidable performance impact of at-rest encryption of OSDs. Read the CloudFlare blog article which shows how they make the encryption impact on their (non-Ceph) drives negligible: https://blog.cloudflare.com/speeding-up-linux-disk-encryption/. The main part of their improvements (the ability to disable dm-crypt workqueues) is already in the mainline kernel. There is also a Ceph pull request that disables dm-crypt workqueues on certain drives: https://github.com/ceph/ceph/pull/49554
While the other part of the performance enhancements authored by CloudFlare (namely, the "xtsproxy" module) is not mainlined yet, I hope that some equivalent solution will find its way into the official kernel sooner or later.
In summary: just encrypt everything.
As a follow-up, if you disagree with the advice to encrypt everything, please note that CephFS allows one to place certain directories on a separate pool. Therefore, you can create a separate device class for encrypted OSDs, create a pool that uses this device class, and put the directories owned by your premium users onto this pool. Documentation: https://docs.ceph.com/en/latest/cephfs/file-layouts/
It may very well not serve any other purpose, but these are requests we get. If I could provide an encryption key to a ceph-fs kernel at mount time, this requirement could be solved very elegantly on a per-user (request) basis and only making users who want it pay with performance penalties.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Tuesday, May 23, 2023 6:35 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Encryption per user Howto
On 23.05.23 08:42, huxiaoyu@horebdata.cn wrote:
Indeed, the question is on server-side encryption with keys managed by ceph on a per-user basis
What kind of security to you want to achieve with encryption keys stored on the server side?
Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de
Tel: 030-405051-43 Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov
-- Alexander E. Patrakov
On 5/26/23 23:09, Alexander E. Patrakov wrote:
Hello Frank,
On Fri, May 26, 2023 at 6:27 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
jumping on this thread as we have requests for which per-client fs mount encryption makes a lot of sense:
What kind of security to you want to achieve with encryption keys stored on the server side?
One of the use cases is if a user requests a share with encryption at rest. Since encryption has an unavoidable performance impact, it is impractical to make 100% of users pay for the requirements that only 1% of users really have. Instead of all-OSD back-end encryption hitting everyone for little reason, encrypting only some user-buckets/fs-shares on the front-end application level will ensure that the data is encrypted at rest.
I would disagree about the unavoidable performance impact of at-rest encryption of OSDs. Read the CloudFlare blog article which shows how they make the encryption impact on their (non-Ceph) drives negligible: https://blog.cloudflare.com/speeding-up-linux-disk-encryption/. The main part of their improvements (the ability to disable dm-crypt workqueues) is already in the mainline kernel. There is also a Ceph pull request that disables dm-crypt workqueues on certain drives: https://github.com/ceph/ceph/pull/49554
Indeed. With the bypass workqueue option enabled for flash devices the overhead of crypto is really low. Here a partial repost from an email I send earlier I repeated the tests from Cloudflare and could draw the same conclusions: TL;DR: performance is increased a lot and less CPU is used. Some fio 4k write, iodepth=1, performance numbers on a Samsung PM983 3.84 TB drive )Ubuntu 22.04 with HWE kernel, 5.15.0-52-generic, AMD EPYC 7302P 16-Core Processor, C-state pinning, CPU performance mode on, Samsung PM 983 firmware: EDA5702Q): Unencrypted NVMe: write: IOPS=63.3k, BW=247MiB/s (259MB/s)(62.6GiB/259207msec); 0 zone resets clat (nsec): min=13190, max=56400, avg=15397.89, stdev=1506.45 lat (nsec): min=13250, max=56940, avg=15462.03, stdev=1507.88 Encrypted (without no_write_workqueue / no_read_workqueue): write: IOPS=34.8k, BW=136MiB/s (143MB/s)(47.4GiB/357175msec); 0 zone resets clat (usec): min=24, max=1221, avg=28.12, stdev= 2.98 lat (usec): min=24, max=1221, avg=28.37, stdev= 2.99 Encrypted (with no_write_workqueue / no_read_workqueue enabled): write: IOPS=55.7k, BW=218MiB/s (228MB/s)(57.3GiB/269574msec); 0 zone resets clat (nsec): min=15710, max=87090, avg=17550.99, stdev=875.72 lat (nsec): min=15770, max=87150, avg=17614.82, stdev=876.85 So encryption does have a performance impact, but the added latency is only a few micro seconds. And these tests are on NVMe drives, not Ceph OSDS. Compared to the latency Ceph itself adds to (client) IO this seems negligible. At least, when the work queues are bypassed, otherwise a lot of CPU seems to be involved (loads of kcryptd threads). And that might hurt max performance on a system (especially if CPU bound). So, today I did a comparison on a production cluster while draining an OSD with 10 concurrent backfills: without no_write_workqueue / no_read_workqueue: 32 krcryptd threads each doing on average 5.5% CPU, and dmcrypt write thread doing ~ 9% CPU. So that's almost 2 CPU cores with no_write_workqueue / no_read_workqueue: dmcrypt / cryptd threads do not even show up in top ... So if encryption is important, even for a subset of your users, I'm pretty sure you can enable it and won't have a negative impact. It does require reprovisioning of all your OSDs ... which is not a small feat. Although this thread started with "per user" encryption. If your users do not trust your Ceph cluster ... client side encryption (i.e. CephFS fscrypt) with a key _they_ manage is still the only way to go. Gr. Stefan
Stefan, how do you have this implemented? Earlier this year I submitted https://tracker.ceph.com/issues/58569 asking to enable just this.
On Jun 2, 2023, at 10:09, Stefan Kooman <stefan@bit.nl> wrote:
On 5/26/23 23:09, Alexander E. Patrakov wrote:
Hello Frank, On Fri, May 26, 2023 at 6:27 PM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
jumping on this thread as we have requests for which per-client fs mount encryption makes a lot of sense:
What kind of security to you want to achieve with encryption keys stored on the server side?
One of the use cases is if a user requests a share with encryption at rest. Since encryption has an unavoidable performance impact, it is impractical to make 100% of users pay for the requirements that only 1% of users really have. Instead of all-OSD back-end encryption hitting everyone for little reason, encrypting only some user-buckets/fs-shares on the front-end application level will ensure that the data is encrypted at rest.
I would disagree about the unavoidable performance impact of at-rest encryption of OSDs. Read the CloudFlare blog article which shows how they make the encryption impact on their (non-Ceph) drives negligible: https://blog.cloudflare.com/speeding-up-linux-disk-encryption/. The main part of their improvements (the ability to disable dm-crypt workqueues) is already in the mainline kernel. There is also a Ceph pull request that disables dm-crypt workqueues on certain drives: https://github.com/ceph/ceph/pull/49554
Indeed. With the bypass workqueue option enabled for flash devices the overhead of crypto is really low. Here a partial repost from an email I send earlier
I repeated the tests from Cloudflare and could draw the same conclusions: TL;DR: performance is increased a lot and less CPU is used. Some fio 4k write, iodepth=1, performance numbers on a Samsung PM983 3.84 TB drive )Ubuntu 22.04 with HWE kernel, 5.15.0-52-generic, AMD EPYC 7302P 16-Core Processor, C-state pinning, CPU performance mode on, Samsung PM 983 firmware: EDA5702Q):
Unencrypted NVMe:
write: IOPS=63.3k, BW=247MiB/s (259MB/s)(62.6GiB/259207msec); 0 zone resets clat (nsec): min=13190, max=56400, avg=15397.89, stdev=1506.45 lat (nsec): min=13250, max=56940, avg=15462.03, stdev=1507.88
Encrypted (without no_write_workqueue / no_read_workqueue):
write: IOPS=34.8k, BW=136MiB/s (143MB/s)(47.4GiB/357175msec); 0 zone resets clat (usec): min=24, max=1221, avg=28.12, stdev= 2.98 lat (usec): min=24, max=1221, avg=28.37, stdev= 2.99
Encrypted (with no_write_workqueue / no_read_workqueue enabled):
write: IOPS=55.7k, BW=218MiB/s (228MB/s)(57.3GiB/269574msec); 0 zone resets clat (nsec): min=15710, max=87090, avg=17550.99, stdev=875.72 lat (nsec): min=15770, max=87150, avg=17614.82, stdev=876.85
So encryption does have a performance impact, but the added latency is only a few micro seconds. And these tests are on NVMe drives, not Ceph OSDS. Compared to the latency Ceph itself adds to (client) IO this seems negligible. At least, when the work queues are bypassed, otherwise a lot of CPU seems to be involved (loads of kcryptd threads). And that might hurt max performance on a system (especially if CPU bound).
So, today I did a comparison on a production cluster while draining an OSD with 10 concurrent backfills:
without no_write_workqueue / no_read_workqueue: 32 krcryptd threads each doing on average 5.5% CPU, and dmcrypt write thread doing ~ 9% CPU. So that's almost 2 CPU cores
with no_write_workqueue / no_read_workqueue: dmcrypt / cryptd threads do not even show up in top ...
So if encryption is important, even for a subset of your users, I'm pretty sure you can enable it and won't have a negative impact. It does require reprovisioning of all your OSDs ... which is not a small feat. Although this thread started with "per user" encryption. If your users do not trust your Ceph cluster ... client side encryption (i.e. CephFS fscrypt) with a key _they_ manage is still the only way to go.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/2/23 16:33, Anthony D'Atri wrote:
Stefan, how do you have this implemented? Earlier this year I submitted https://tracker.ceph.com/issues/58569 <https://tracker.ceph.com/issues/58569> asking to enable just this.
Lol, I have never seen that tracker otherwise I would have informed you about it. I see the PR and tracker are updated by you / Joshua, thanks for that.. So yes, we have this implemented and running in production (currently re-provisioning all OSDs). It's a locally patched 16.2.11 ceph-volume for that matter. The PR [1] needs some fixing (I need to sit down and make it happen, just so many other things that take up my time). But then this would be enabled by default for flash devices (non-rotational). If used with cryptsetup 2.4.x also the appropriate sector size is used (based on the physical sector size). We use 4K on NVMe. Added benefit of using cryptsetup 2.4.x is that is uses Argon2id as PBKDF for LUKS2. We created a backport of cryptsetup 2.4.3 for use in Ubuntu Focal (based on Jammy) [2]. We are converting our whole cluster using LUKS2 with the work queues bypassed. For the nodes that have been converted already it works just fine. So, as multiple users seem to be waiting for this to be available in Ceph ... I should hurry up and make sure the PR gets in proper shape and merged in main. Gr. Stefan [1]: https://github.com/ceph/ceph/pull/49554 [2]: https://obit.bit.nl/ubuntu/focal/cryptsetup/
Hello Stefan, On Fri, Jun 2, 2023 at 11:12 PM Stefan Kooman <stefan@bit.nl> wrote:
On 6/2/23 16:33, Anthony D'Atri wrote:
Stefan, how do you have this implemented? Earlier this year I submitted https://tracker.ceph.com/issues/58569 <https://tracker.ceph.com/issues/58569> asking to enable just this.
Lol, I have never seen that tracker otherwise I would have informed you about it. I see the PR and tracker are updated by you / Joshua, thanks for that..
So yes, we have this implemented and running in production (currently re-provisioning all OSDs). It's a locally patched 16.2.11 ceph-volume for that matter. The PR [1] needs some fixing (I need to sit down and make it happen, just so many other things that take up my time). But then this would be enabled by default for flash devices (non-rotational). If used with cryptsetup 2.4.x also the appropriate sector size is used (based on the physical sector size). We use 4K on NVMe.
Added benefit of using cryptsetup 2.4.x is that is uses Argon2id as PBKDF for LUKS2.
We created a backport of cryptsetup 2.4.3 for use in Ubuntu Focal (based on Jammy) [2].
We are converting our whole cluster using LUKS2 with the work queues bypassed. For the nodes that have been converted already it works just fine. So, as multiple users seem to be waiting for this to be available in Ceph ... I should hurry up and make sure the PR gets in proper shape and merged in main.
Thanks for the report. However, I would like to take back a part of my previous response, where I informed you about the "xtsproxy" kernel module. Please don't try to use it. Reason: I recently filed a bug for its inclusion into the Zen kernel, available for Arch Linux users, and the result is that the resulting system stopped booting for some users. So a proper backport is required, even though the Cloudflare patch applies as-is. https://github.com/zen-kernel/zen-kernel/issues/306 https://github.com/zen-kernel/zen-kernel/issues/310 -- Alexander E. Patrakov
Hi Stefan, there are still users with large HDD installations and I think this will not change anytime soon. What is the impact of encryption with the new settings for HDD? Is it as bad as their continued omission from any statement suggests? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Friday, June 2, 2023 5:11 PM To: Anthony D'Atri; ceph-users@ceph.io Subject: [ceph-users] Re: Encryption per user Howto On 6/2/23 16:33, Anthony D'Atri wrote:
Stefan, how do you have this implemented? Earlier this year I submitted https://tracker.ceph.com/issues/58569 <https://tracker.ceph.com/issues/58569> asking to enable just this.
Lol, I have never seen that tracker otherwise I would have informed you about it. I see the PR and tracker are updated by you / Joshua, thanks for that.. So yes, we have this implemented and running in production (currently re-provisioning all OSDs). It's a locally patched 16.2.11 ceph-volume for that matter. The PR [1] needs some fixing (I need to sit down and make it happen, just so many other things that take up my time). But then this would be enabled by default for flash devices (non-rotational). If used with cryptsetup 2.4.x also the appropriate sector size is used (based on the physical sector size). We use 4K on NVMe. Added benefit of using cryptsetup 2.4.x is that is uses Argon2id as PBKDF for LUKS2. We created a backport of cryptsetup 2.4.3 for use in Ubuntu Focal (based on Jammy) [2]. We are converting our whole cluster using LUKS2 with the work queues bypassed. For the nodes that have been converted already it works just fine. So, as multiple users seem to be waiting for this to be available in Ceph ... I should hurry up and make sure the PR gets in proper shape and merged in main. Gr. Stefan [1]: https://github.com/ceph/ceph/pull/49554 [2]: https://obit.bit.nl/ubuntu/focal/cryptsetup/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/6/23 14:26, Frank Schilder wrote:
Hi Stefan,
there are still users with large HDD installations and I think this will not change anytime soon. What is the impact of encryption with the new settings for HDD? Is it as bad as their continued omission from any statement suggests?
We only tested flash, as we don't have any spinners to test on. So not sure what the impact would be. From the blog post I got that the queuing was there to mainly help workloads with spinning disks. But all that was valid in 2015. It specifically mentions benefit for CFQ scheduler, that since has been removed from the linux kernel. So, it would be interesting to know if it is still needed on spinners when used with Ceph ... Gr. Stefan
Yes, would be interesting. I understood that it mainly helps with buffered writes, but ceph is using direct IO for writes and that's where bypassing the queues helps. Are there detailed instructions somewhere how to set up a host to disable the queues? I don't have time to figure this out myself. It should be detailed enough so that I just need to edit some configs, reboot et voila. I have a number of new hosts to deploy and I could use one of these to run a test. They have a mix of NVMe, SSD and HDD and I can run fio benchmarks before deploying OSDs in the way you did. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Tuesday, June 6, 2023 3:20 PM To: Frank Schilder; Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Encryption per user Howto On 6/6/23 14:26, Frank Schilder wrote:
Hi Stefan,
there are still users with large HDD installations and I think this will not change anytime soon. What is the impact of encryption with the new settings for HDD? Is it as bad as their continued omission from any statement suggests?
We only tested flash, as we don't have any spinners to test on. So not sure what the impact would be. From the blog post I got that the queuing was there to mainly help workloads with spinning disks. But all that was valid in 2015. It specifically mentions benefit for CFQ scheduler, that since has been removed from the linux kernel. So, it would be interesting to know if it is still needed on spinners when used with Ceph ... Gr. Stefan
On 6/6/23 15:33, Frank Schilder wrote:
Yes, would be interesting. I understood that it mainly helps with buffered writes, but ceph is using direct IO for writes and that's where bypassing the queues helps.
Yeah, that makes sense.
Are there detailed instructions somewhere how to set up a host to disable the queues? I don't have time to figure this out myself. It should be detailed enough so that I just need to edit some configs, reboot et voila.
Do you want instructions for package based Ceph install, or container based? I tested it with both deployment types. Container based (Cephadm)) is a little bit more involved, but certainly doable.
I have a number of new hosts to deploy and I could use one of these to run a test. They have a mix of NVMe, SSD and HDD and I can run fio benchmarks before deploying OSDs in the way you did
That would be great, Gr. Stefan .
Hi Stefan, bare metal. I just need to know what kernel version and how to configure the new queue parameters (I guess its kernel boot parameters). I will do a fio test to the raw block device first, I think this is what you posted? I can probably try these settings on our test cluster, which is entirely SAS HDD based. If there is anything I need to do when deploying an OSD, the bare metal instructions are good. I actually start all daemons myself, its containerized, but with custom startup scripts. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Wednesday, June 7, 2023 8:01 AM To: Frank Schilder; Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Encryption per user Howto On 6/6/23 15:33, Frank Schilder wrote:
Yes, would be interesting. I understood that it mainly helps with buffered writes, but ceph is using direct IO for writes and that's where bypassing the queues helps.
Yeah, that makes sense.
Are there detailed instructions somewhere how to set up a host to disable the queues? I don't have time to figure this out myself. It should be detailed enough so that I just need to edit some configs, reboot et voila.
Do you want instructions for package based Ceph install, or container based? I tested it with both deployment types. Container based (Cephadm)) is a little bit more involved, but certainly doable.
I have a number of new hosts to deploy and I could use one of these to run a test. They have a mix of NVMe, SSD and HDD and I can run fio benchmarks before deploying OSDs in the way you did
That would be great, Gr. Stefan .
Hi Stefan, sorry, forgot. Block device is almost certainly LVM with dmcrypt - unless you have another way of using encryption with ceph OSDs. I can compare LVM with LVM+dmcrypt(default/new) and possibly also raw /dev/sd? performance. If LVM+dmcrypt shows good results, I will also try it with our test cluster. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Wednesday, June 7, 2023 10:03 AM To: Stefan Kooman; Anthony D'Atri; ceph-users@ceph.io Subject: [ceph-users] Re: Encryption per user Howto Hi Stefan, bare metal. I just need to know what kernel version and how to configure the new queue parameters (I guess its kernel boot parameters). I will do a fio test to the raw block device first, I think this is what you posted? I can probably try these settings on our test cluster, which is entirely SAS HDD based. If there is anything I need to do when deploying an OSD, the bare metal instructions are good. I actually start all daemons myself, its containerized, but with custom startup scripts. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Wednesday, June 7, 2023 8:01 AM To: Frank Schilder; Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Encryption per user Howto On 6/6/23 15:33, Frank Schilder wrote:
Yes, would be interesting. I understood that it mainly helps with buffered writes, but ceph is using direct IO for writes and that's where bypassing the queues helps.
Yeah, that makes sense.
Are there detailed instructions somewhere how to set up a host to disable the queues? I don't have time to figure this out myself. It should be detailed enough so that I just need to edit some configs, reboot et voila.
Do you want instructions for package based Ceph install, or container based? I tested it with both deployment types. Container based (Cephadm)) is a little bit more involved, but certainly doable.
I have a number of new hosts to deploy and I could use one of these to run a test. They have a mix of NVMe, SSD and HDD and I can run fio benchmarks before deploying OSDs in the way you did
That would be great, Gr. Stefan . _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/7/23 12:57, Frank Schilder wrote:
Hi Stefan,
sorry, forgot. Block device is almost certainly LVM with dmcrypt - unless you have another way of using encryption with ceph OSDs.
I can compare LVM with LVM+dmcrypt(default/new) and possibly also raw /dev/sd? performance. If LVM+dmcrypt shows good results, I will also try it with our test cluster.
We use it with OSDs prepared with ceph-volume. Do you use ceph-volume? So in this case the logical volume (LV) is encrypted and used by ceph. Gr. Stefan
Hi Stefan, yes, ceph-volume OSDs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Wednesday, June 7, 2023 1:38 PM To: Frank Schilder; Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Encryption per user Howto On 6/7/23 12:57, Frank Schilder wrote:
Hi Stefan,
sorry, forgot. Block device is almost certainly LVM with dmcrypt - unless you have another way of using encryption with ceph OSDs.
I can compare LVM with LVM+dmcrypt(default/new) and possibly also raw /dev/sd? performance. If LVM+dmcrypt shows good results, I will also try it with our test cluster.
We use it with OSDs prepared with ceph-volume. Do you use ceph-volume? So in this case the logical volume (LV) is encrypted and used by ceph. Gr. Stefan
On 6/7/23 14:22, Frank Schilder wrote:
Hi Stefan,
yes, ceph-volume OSDs.
Requirements: Kernel version requirement and higher: 5.9 cryptsetup: 2.3.4 and higher. Preferably 2.4.x (automatic alignment of sector size based on physical disk properties). RAW device: cryptsetup luksFormat /dev/device LVM device: cryptsetup luksFormat /dev/LVM/LV-here (this is what ceph-volume uses) Afterwards unlock the device with "cryptsetup open": cryptsetup open /dev/sdz name_that_makes_sense_for_you To use it with ceph-volume, the following changes can be made, here a diff against 16.2.11: --- before: /usr/lib/python3/dist-packages/ceph_volume/util/disk.py +++ after: /home/stefan/git/bit-ceph/configs/osd/usr/lib/python3/dist-packages/ceph_volume/util/disk.py @@ -194,6 +194,26 @@ devices.append(_lsblk_parser(line)) return devices + +def is_rotational(device): + """ + Returns true if a device is rotational or not. + """ + labels = ['ROTA'] + command = ['lsblk', '-P', '-p', '-o', ','.join(labels), device] + out, err, rc = process.call(command) + device = [] + for line in out: + device.append(_lsblk_parser(line)) + + for i in device: + if type(i) is dict and "ROTA" in i.keys(): + if i['ROTA'] == '1': + rotational = True + else: + rotational = False + + return rotational def udevadm_property(device, properties=[]): @@ -900,6 +920,10 @@ metadata['sectorsize'] = get_file_contents(sysdir + "/queue/logical_block_size", fallback_sectorsize) + fallback_rotational = '1' + metadata['rotational'] = get_file_contents(sysdir + + "/queue/rotational", + fallback_rotational) metadata['size'] = float(size) * 512 metadata['human_readable_size'] = human_readable_size(metadata['size']) metadata['path'] = diskname --- before: /usr/lib/python3/dist-packages/ceph_volume/util/encryption.py +++ after: /home/stefan/git/bit-ceph/configs/osd/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py @@ -1,11 +1,12 @@ import base64 import os import logging +import subprocess from ceph_volume import process, conf, terminal from ceph_volume.util import constants, system from ceph_volume.util.device import Device from .prepare import write_keyring -from .disk import lsblk, device_family, get_part_entry_type +from .disk import lsblk, device_family, get_part_entry_type, is_rotational logger = logging.getLogger(__name__) mlogger = terminal.MultiLogger(__name__) @@ -23,10 +24,52 @@ if key_size not in ['256', '512']: logger.warning(("Invalid value set for osd_dmcrypt_key_size ({}). " - "Falling back to {}bits".format(key_size, default_key_size))) + "Falling back to {} bits".format(key_size, default_key_size))) return default_key_size return key_size + +def execCmd(cmdline): + cmd = subprocess.Popen(cmdline, shell=True, stdout=subprocess.PIPE) + stdout, _ = cmd.communicate() + return stdout.decode("utf-8") + +def bypass_workqueues(device): + """ + 1) Check if cryptsetup has support for no_read_workqueue. + If so, it also supports no_write_workqueue. + cryptsetup has support for this since 2.3.4 + + 2) Also check if we are using a flash (non-rotational) device + or not. I.e. Spinning media might still benefit from queuing. + + Only if both 1) and 2) are true we do _not_ want to use work queues + Note: + crypt version 1.22 and higher have support for this. Example: + modprobe dm-crypt + dmsetup targets + + integrity v1.10.0 + crypt v1.23.0 + striped v1.6.0 + linear v1.4.0 + error v1.5.0 + + If dm-crypt does not have support for no_read_workqueue / + no_write_workqueue the options will be discarded and + logged by the linux kernel: + + device-mapper: table: major:minor: crypt: Invalid feature arguments + device-mapper: ioctl: error adding target to table + + The encrypted device will come online without the options active. + """ + cryptsetup_help = execCmd("cryptsetup --help 2>/dev/null") + + if '--perf-no_read_workqueue' not in cryptsetup_help or is_rotational(device): + return [] + + return ['--perf-no_read_workqueue', '--perf-no_write_workqueue'] def create_dmcrypt_key(): """ @@ -78,7 +121,8 @@ '--type', 'plain', '--key-size', '256', ] - + for extra_opts in bypass_workqueues(device): + command.insert(1, extra_opts) process.call(command, stdin=key, terminal_verbose=True, show_command=True) @@ -103,6 +147,8 @@ device, mapping, ] + for extra_opts in bypass_workqueues(device): + command.insert(1, extra_opts) process.call(command, stdin=key, terminal_verbose=True, show_command=True) But in your case you might want to change the logic a bit so regardless of rotational media or not the queues are bypassed (make the ROTA function always return true for example). To change Ceph container images ... I used this quick and dirty procedure: docker run -it ceph-image bash Make the above changes to ceph-volume exit from container docker ps -a Look for the id of the most recently used container (that you just exited) docker commit container-hash alternative_name_of_container_image if you want to upload to registry: docker image tag alternative_name_of_container_image docker_registry_url:5000/alternative_name_of_container_image docker image push docker_registry_url:5000/alternative_name_of_container_image for cephadm users: docker image ls --digests <- look for the sha256 hash of the image ceph config-key set config/global/container_image docker_registry_url:5000/alternative_name_of_container_image:latest@sha256:the-hash-here redeploy containers with this new image Let me know if you need other info to make this work for you. Looking forward to the results. Gr. Stefan
On 5/22/23 17:28, huxiaoyu@horebdata.cn wrote:
Hi, Stefan,
Thanks a lot for the message. It seems that client-side encryption (or per use) is still on the way and not ready yet for today.
Are there practical methods to implement encryption for CephFS with today' technique? e.g using LUKS or other tools?
Not that I know of. LUKS encryption can be done on the OSD level, but that does not seem what you are looking for. Gr. Stefan
participants (8)
-
Alexander E. Patrakov
-
Anthony D'Atri
-
Casey Bodley
-
Christian Wuerdig
-
Frank Schilder
-
huxiaoyu＠horebdata.cn
-
Robert Sander
-
Stefan Kooman