Proposal: Encrypted Namespaces
This proposal might be a bit thin on details, but I would love to have some feedback and gauge the broader community's and developer's interest, as well as to poke holes in the current idea. All comments welcome. -Joao MOTIVATION ---------- Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys. The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary. While there are very good arguments for ensuring this encryption is performed client-side, such that each client actively controls their own secrets, a server-side approach has several other benefits that may outweigh a client-side approach. On the one hand, * encrypting server side means encrypting N times, depending on replication size and scheme; * the secrets keyring will be centralized, likely in the monitor, much like what we do for dmcrypt; even though encrypted. * on-the-wire data will still need to rely on msgr2 encryption; even though one could argue that this will likely happen regardless of whether a client- or server-side approach is being used. But on the other, 1. encryption becomes transparent for the client, avoiding the effort of implementing such schemes in client libraries and kernel drivers; 2. tighter control over the unit of data being encrypted, reducing the load of encrypting a whole object versus a disk block in bluestore. 3. older clients will be able to support encryption out of the box, given they will have no idea their data is being encrypted, nor on how that is happening. CHOOSING NAMESPACES -------------------- While investigating where and how per-tenant encryption could be implemented, two other ideas were on the table: 1. on a per-client basis, relying on cephx entities, with an encryption key per-client, or a shared key amongst several clients; this key would be kept encrypted in the monitor's kv store with the entity's cephx key. 2. on a per-pool basis. The first one would definitely be feasible, but potentially tricky to implement just right, without too many exceptions or involvement of other portions of the stack. E.g., dealing with metadata could become tricky. Then again, there wasn't one reason that could not be addressed and become a showstopper. As for 2., it would definitely be the easiest to implement: pool is created with an 'encrypted' flag on, key is kept in the monitors, OSDs encrypt any object belonging to that pool. The problem with this option, however, is how coarse-grained it is. If we really wanted a per-tenant approach, one would have to ensure one pool per tenant. Not necessarily a big deal if a lot of potentially small pools is fine. This idea was scrapped in favour of encrypting namespaces instead. Given RADOS already has the concept of a namespace, it might just be the ideal medium to implement such an approach, as we get the best of the two options above: we can get a smaller-grained access than a pool, but still with the same capabilities of limiting access by entity through caps. We also get to have multiple namespaces in a single pool should we choose to do so. All the while, the concept is high-level enough that the effort of implementing the actual encryption scheme might be performed in a select, handful of places, without the need for a lot (maybe, any) particular exceptions or corner cases. APPROACH --------- It is important to note that there are several implementations details, especially on "how exactly this is going to happen", that have not been fully figured out. Essentially, the objective is to ensure that objects from a given namespace are always encrypted or decrypted by bluestore when writing or reading the data. The hope that performing at this level will allow us to 1. ensure the operation is performed at the disk block size, ensuring that small writes, or partial writes, will not require a rewrite of the whole object; same goes for reads. 2. avoid dealing with all the mechanics involving objects and other operations over them, and focus solely on their data and metadata. Secret distribution is expected to be done by the monitors, at the OSDs request. In an ideal world, the OSDs would know exactly which namespaces they might have to encrypt/decrypt, based on pools they currently hold, and request keys for those before hand, such that they don't have to request a key from the monitor when an operation arrives. This would not only require us to become a bit more aware of namespaces, but keeping these keys cached might require the osd to keep them encrypted in memory. What to use for that is something that hasn't been much thought about -- maybe we could get away with using the osd's cephx key. As for the namespaces, in their current form we don't have much (any?) information about them. Access to an object in a namespace is based on prior knowledge of that namespace and the object's name. We currently don't have statistics on namespaces, nor are we able to know whether an OSD keeps any object belonging to a namespace _before_ an operation on such an object is handled. Even though it's not particularly _required_ to get more out of namespaces than we currently have, it would definitely be ideal if we ended up with the ability to 1) have statistics out of namespaces, as it would imperative if we're using them for tenants; and 2) able to cache ahead keys for namespaces an osd might have to handle (read, namespaces living in a pool with PGs mapped to a given osd).
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
This proposal might be a bit thin on details, but I would love to have some feedback and gauge the broader community's and developer's interest, as well as to poke holes in the current idea.
All comments welcome.
-Joao
MOTIVATION ----------
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary.
I would be interested to hear the tenant use-case where they trust the backing storage system (Ceph) with all things encryption and don't have any effective control over the keys / ciphers / rotation policies / etc. If you have a vulnerability that exposes the current OSD dm-crypt keys, I would think it would be possible to get the per-namespace keys though a similar vector if they are stored effectively side-by-side?
While there are very good arguments for ensuring this encryption is performed client-side, such that each client actively controls their own secrets, a server-side approach has several other benefits that may outweigh a client-side approach.
On the one hand,
* encrypting server side means encrypting N times, depending on replication size and scheme; * the secrets keyring will be centralized, likely in the monitor, much like what we do for dmcrypt; even though encrypted. * on-the-wire data will still need to rely on msgr2 encryption; even though one could argue that this will likely happen regardless of whether a client- or server-side approach is being used.
But on the other,
1. encryption becomes transparent for the client, avoiding the effort of implementing such schemes in client libraries and kernel drivers;
Just an FYI: krbd supports client-side via dm-crypt, kernel CephFS is actively looking to incorporate fscrypt, librbd can utilize QEMU-layered LUKS for many use-cases and work is in-progress on built-in librbd client-side encryption. RGW has had client-side encryption for a while.
2. tighter control over the unit of data being encrypted, reducing the load of encrypting a whole object versus a disk block in bluestore.
RBD client-side encryption doesn't rely on the underlying object size (512 bytes for dm-crypt I think and looking at 4KiB blocks for the librbd built-in encryption). I can't speak for CephFS+fscrypt, but I suspect it wouldn't require re-encrypting the full file or backing object (probably 4KiB page).
3. older clients will be able to support encryption out of the box, given they will have no idea their data is being encrypted, nor on how that is happening.
CHOOSING NAMESPACES --------------------
While investigating where and how per-tenant encryption could be implemented, two other ideas were on the table:
1. on a per-client basis, relying on cephx entities, with an encryption key per-client, or a shared key amongst several clients; this key would be kept encrypted in the monitor's kv store with the entity's cephx key.
2. on a per-pool basis.
The first one would definitely be feasible, but potentially tricky to implement just right, without too many exceptions or involvement of other portions of the stack. E.g., dealing with metadata could become tricky. Then again, there wasn't one reason that could not be addressed and become a showstopper.
As for 2., it would definitely be the easiest to implement: pool is created with an 'encrypted' flag on, key is kept in the monitors, OSDs encrypt any object belonging to that pool. The problem with this option, however, is how coarse-grained it is. If we really wanted a per-tenant approach, one would have to ensure one pool per tenant. Not necessarily a big deal if a lot of potentially small pools is fine. This idea was scrapped in favour of encrypting namespaces instead.
Given RADOS already has the concept of a namespace, it might just be the ideal medium to implement such an approach, as we get the best of the two options above: we can get a smaller-grained access than a pool, but still with the same capabilities of limiting access by entity through caps. We also get to have multiple namespaces in a single pool should we choose to do so. All the while, the concept is high-level enough that the effort of implementing the actual encryption scheme might be performed in a select, handful of places, without the need for a lot (maybe, any) particular exceptions or corner cases.
APPROACH ---------
It is important to note that there are several implementations details, especially on "how exactly this is going to happen", that have not been fully figured out.
Essentially, the objective is to ensure that objects from a given namespace are always encrypted or decrypted by bluestore when writing or reading the data. The hope that performing at this level will allow us to
1. ensure the operation is performed at the disk block size, ensuring that small writes, or partial writes, will not require a rewrite of the whole object; same goes for reads.
2. avoid dealing with all the mechanics involving objects and other operations over them, and focus solely on their data and metadata.
Secret distribution is expected to be done by the monitors, at the OSDs request. In an ideal world, the OSDs would know exactly which namespaces they might have to encrypt/decrypt, based on pools they currently hold, and request keys for those before hand, such that they don't have to request a key from the monitor when an operation arrives. This would not only require us to become a bit more aware of namespaces, but keeping these keys cached might require the osd to keep them encrypted in memory. What to use for that is something that hasn't been much thought about -- maybe we could get away with using the osd's cephx key.
As for the namespaces, in their current form we don't have much (any?) information about them. Access to an object in a namespace is based on prior knowledge of that namespace and the object's name. We currently don't have statistics on namespaces, nor are we able to know whether an OSD keeps any object belonging to a namespace _before_ an operation on such an object is handled.
Even though it's not particularly _required_ to get more out of namespaces than we currently have, it would definitely be ideal if we ended up with the ability to 1) have statistics out of namespaces, as it would imperative if we're using them for tenants; and 2) able to cache ahead keys for namespaces an osd might have to handle (read, namespaces living in a pool with PGs mapped to a given osd).
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason
On 20/08/04 09:04AM, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary.
I would be interested to hear the tenant use-case where they trust the backing storage system (Ceph) with all things encryption and don't have any effective control over the keys / ciphers / rotation policies / etc. If you have a vulnerability that exposes the current OSD dm-crypt keys, I would think it would be possible to get the per-namespace keys though a similar vector if they are stored effectively side-by-side?
The idea here was not to berate the current at-rest scheme, nor propose something as better than, but a use case where it is used instead of. Maybe I'm being naive, but the trade-off of having the storage system handling the namespace keys is not much different than having it handling the dmcrypt keys. I'm in no way saying that Ceph handling the secrets is better than having the clients doing their own encryption; it's just a different use case being addressed.
While there are very good arguments for ensuring this encryption is performed client-side, such that each client actively controls their own secrets, a server-side approach has several other benefits that may outweigh a client-side approach.
On the one hand,
* encrypting server side means encrypting N times, depending on replication size and scheme; * the secrets keyring will be centralized, likely in the monitor, much like what we do for dmcrypt; even though encrypted. * on-the-wire data will still need to rely on msgr2 encryption; even though one could argue that this will likely happen regardless of whether a client- or server-side approach is being used.
But on the other,
1. encryption becomes transparent for the client, avoiding the effort of implementing such schemes in client libraries and kernel drivers;
Just an FYI: krbd supports client-side via dm-crypt, kernel CephFS is actively looking to incorporate fscrypt, librbd can utilize QEMU-layered LUKS for many use-cases and work is in-progress on built-in librbd client-side encryption. RGW has had client-side encryption for a while.
Very much aware. To be frank, this came up mostly to address cephfs encryption, but the approach seemed generic enough to write it up as such. As for cephfs, I've been following the fscrypt discussion, within reason and what's available in the ticket, and it didn't seem particularly in conflict with this proposal (other than potential duplication of efforts).
2. tighter control over the unit of data being encrypted, reducing the load of encrypting a whole object versus a disk block in bluestore.
RBD client-side encryption doesn't rely on the underlying object size (512 bytes for dm-crypt I think and looking at 4KiB blocks for the librbd built-in encryption). I can't speak for CephFS+fscrypt, but I suspect it wouldn't require re-encrypting the full file or backing object (probably 4KiB page).
TIL. Thanks :) -Joao
3. older clients will be able to support encryption out of the box, given they will have no idea their data is being encrypted, nor on how that is happening.
CHOOSING NAMESPACES --------------------
While investigating where and how per-tenant encryption could be implemented, two other ideas were on the table:
1. on a per-client basis, relying on cephx entities, with an encryption key per-client, or a shared key amongst several clients; this key would be kept encrypted in the monitor's kv store with the entity's cephx key.
2. on a per-pool basis.
The first one would definitely be feasible, but potentially tricky to implement just right, without too many exceptions or involvement of other portions of the stack. E.g., dealing with metadata could become tricky. Then again, there wasn't one reason that could not be addressed and become a showstopper.
As for 2., it would definitely be the easiest to implement: pool is created with an 'encrypted' flag on, key is kept in the monitors, OSDs encrypt any object belonging to that pool. The problem with this option, however, is how coarse-grained it is. If we really wanted a per-tenant approach, one would have to ensure one pool per tenant. Not necessarily a big deal if a lot of potentially small pools is fine. This idea was scrapped in favour of encrypting namespaces instead.
Given RADOS already has the concept of a namespace, it might just be the ideal medium to implement such an approach, as we get the best of the two options above: we can get a smaller-grained access than a pool, but still with the same capabilities of limiting access by entity through caps. We also get to have multiple namespaces in a single pool should we choose to do so. All the while, the concept is high-level enough that the effort of implementing the actual encryption scheme might be performed in a select, handful of places, without the need for a lot (maybe, any) particular exceptions or corner cases.
APPROACH ---------
It is important to note that there are several implementations details, especially on "how exactly this is going to happen", that have not been fully figured out.
Essentially, the objective is to ensure that objects from a given namespace are always encrypted or decrypted by bluestore when writing or reading the data. The hope that performing at this level will allow us to
1. ensure the operation is performed at the disk block size, ensuring that small writes, or partial writes, will not require a rewrite of the whole object; same goes for reads.
2. avoid dealing with all the mechanics involving objects and other operations over them, and focus solely on their data and metadata.
Secret distribution is expected to be done by the monitors, at the OSDs request. In an ideal world, the OSDs would know exactly which namespaces they might have to encrypt/decrypt, based on pools they currently hold, and request keys for those before hand, such that they don't have to request a key from the monitor when an operation arrives. This would not only require us to become a bit more aware of namespaces, but keeping these keys cached might require the osd to keep them encrypted in memory. What to use for that is something that hasn't been much thought about -- maybe we could get away with using the osd's cephx key.
As for the namespaces, in their current form we don't have much (any?) information about them. Access to an object in a namespace is based on prior knowledge of that namespace and the object's name. We currently don't have statistics on namespaces, nor are we able to know whether an OSD keeps any object belonging to a namespace _before_ an operation on such an object is handled.
Even though it's not particularly _required_ to get more out of namespaces than we currently have, it would definitely be ideal if we ended up with the ability to 1) have statistics out of namespaces, as it would imperative if we're using them for tenants; and 2) able to cache ahead keys for namespaces an osd might have to handle (read, namespaces living in a pool with PGs mapped to a given osd).
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Tue, Aug 4, 2020 at 11:55 AM Joao Eduardo Luis <joao@suse.de> wrote:
On 20/08/04 09:04AM, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary.
I would be interested to hear the tenant use-case where they trust the backing storage system (Ceph) with all things encryption and don't have any effective control over the keys / ciphers / rotation policies / etc. If you have a vulnerability that exposes the current OSD dm-crypt keys, I would think it would be possible to get the per-namespace keys though a similar vector if they are stored effectively side-by-side?
The idea here was not to berate the current at-rest scheme, nor propose something as better than, but a use case where it is used instead of. Maybe I'm being naive, but the trade-off of having the storage system handling the namespace keys is not much different than having it handling the dmcrypt keys.
I'm in no way saying that Ceph handling the secrets is better than having the clients doing their own encryption; it's just a different use case being addressed.
Totally understand. I am just honestly interested if this solves a known issue for Ceph users (regulatory or otherwise). i.e. would it check all the same boxes to implement pool-level encryption vs namespace-level encryption?
While there are very good arguments for ensuring this encryption is performed client-side, such that each client actively controls their own secrets, a server-side approach has several other benefits that may outweigh a client-side approach.
On the one hand,
* encrypting server side means encrypting N times, depending on replication size and scheme; * the secrets keyring will be centralized, likely in the monitor, much like what we do for dmcrypt; even though encrypted. * on-the-wire data will still need to rely on msgr2 encryption; even though one could argue that this will likely happen regardless of whether a client- or server-side approach is being used.
But on the other,
1. encryption becomes transparent for the client, avoiding the effort of implementing such schemes in client libraries and kernel drivers;
Just an FYI: krbd supports client-side via dm-crypt, kernel CephFS is actively looking to incorporate fscrypt, librbd can utilize QEMU-layered LUKS for many use-cases and work is in-progress on built-in librbd client-side encryption. RGW has had client-side encryption for a while.
Very much aware. To be frank, this came up mostly to address cephfs encryption, but the approach seemed generic enough to write it up as such.
As for cephfs, I've been following the fscrypt discussion, within reason and what's available in the ticket, and it didn't seem particularly in conflict with this proposal (other than potential duplication of efforts).
2. tighter control over the unit of data being encrypted, reducing the load of encrypting a whole object versus a disk block in bluestore.
RBD client-side encryption doesn't rely on the underlying object size (512 bytes for dm-crypt I think and looking at 4KiB blocks for the librbd built-in encryption). I can't speak for CephFS+fscrypt, but I suspect it wouldn't require re-encrypting the full file or backing object (probably 4KiB page).
TIL. Thanks :)
-Joao
3. older clients will be able to support encryption out of the box, given they will have no idea their data is being encrypted, nor on how that is happening.
CHOOSING NAMESPACES --------------------
While investigating where and how per-tenant encryption could be implemented, two other ideas were on the table:
1. on a per-client basis, relying on cephx entities, with an encryption key per-client, or a shared key amongst several clients; this key would be kept encrypted in the monitor's kv store with the entity's cephx key.
2. on a per-pool basis.
The first one would definitely be feasible, but potentially tricky to implement just right, without too many exceptions or involvement of other portions of the stack. E.g., dealing with metadata could become tricky. Then again, there wasn't one reason that could not be addressed and become a showstopper.
As for 2., it would definitely be the easiest to implement: pool is created with an 'encrypted' flag on, key is kept in the monitors, OSDs encrypt any object belonging to that pool. The problem with this option, however, is how coarse-grained it is. If we really wanted a per-tenant approach, one would have to ensure one pool per tenant. Not necessarily a big deal if a lot of potentially small pools is fine. This idea was scrapped in favour of encrypting namespaces instead.
Given RADOS already has the concept of a namespace, it might just be the ideal medium to implement such an approach, as we get the best of the two options above: we can get a smaller-grained access than a pool, but still with the same capabilities of limiting access by entity through caps. We also get to have multiple namespaces in a single pool should we choose to do so. All the while, the concept is high-level enough that the effort of implementing the actual encryption scheme might be performed in a select, handful of places, without the need for a lot (maybe, any) particular exceptions or corner cases.
APPROACH ---------
It is important to note that there are several implementations details, especially on "how exactly this is going to happen", that have not been fully figured out.
Essentially, the objective is to ensure that objects from a given namespace are always encrypted or decrypted by bluestore when writing or reading the data. The hope that performing at this level will allow us to
1. ensure the operation is performed at the disk block size, ensuring that small writes, or partial writes, will not require a rewrite of the whole object; same goes for reads.
2. avoid dealing with all the mechanics involving objects and other operations over them, and focus solely on their data and metadata.
Secret distribution is expected to be done by the monitors, at the OSDs request. In an ideal world, the OSDs would know exactly which namespaces they might have to encrypt/decrypt, based on pools they currently hold, and request keys for those before hand, such that they don't have to request a key from the monitor when an operation arrives. This would not only require us to become a bit more aware of namespaces, but keeping these keys cached might require the osd to keep them encrypted in memory. What to use for that is something that hasn't been much thought about -- maybe we could get away with using the osd's cephx key.
As for the namespaces, in their current form we don't have much (any?) information about them. Access to an object in a namespace is based on prior knowledge of that namespace and the object's name. We currently don't have statistics on namespaces, nor are we able to know whether an OSD keeps any object belonging to a namespace _before_ an operation on such an object is handled.
Even though it's not particularly _required_ to get more out of namespaces than we currently have, it would definitely be ideal if we ended up with the ability to 1) have statistics out of namespaces, as it would imperative if we're using them for tenants; and 2) able to cache ahead keys for namespaces an osd might have to handle (read, namespaces living in a pool with PGs mapped to a given osd).
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason
On 8/4/20 10:02 AM, Jason Dillaman wrote:
On Tue, Aug 4, 2020 at 11:55 AM Joao Eduardo Luis <joao@suse.de> wrote:
On 20/08/04 09:04AM, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary.
I would be interested to hear the tenant use-case where they trust the backing storage system (Ceph) with all things encryption and don't have any effective control over the keys / ciphers / rotation policies / etc. If you have a vulnerability that exposes the current OSD dm-crypt keys, I would think it would be possible to get the per-namespace keys though a similar vector if they are stored effectively side-by-side?
The idea here was not to berate the current at-rest scheme, nor propose something as better than, but a use case where it is used instead of. Maybe I'm being naive, but the trade-off of having the storage system handling the namespace keys is not much different than having it handling the dmcrypt keys.
I'm in no way saying that Ceph handling the secrets is better than having the clients doing their own encryption; it's just a different use case being addressed.
Totally understand. I am just honestly interested if this solves a known issue for Ceph users (regulatory or otherwise). i.e. would it check all the same boxes to implement pool-level encryption vs namespace-level encryption?
Indeed, pools could even be used today (one pool on dmcrypted drives, another on drives without encryption) for similar security properties.
On 20/08/04 01:12PM, Josh Durgin wrote:
On 8/4/20 10:02 AM, Jason Dillaman wrote:
On Tue, Aug 4, 2020 at 11:55 AM Joao Eduardo Luis <joao@suse.de> wrote:
On 20/08/04 09:04AM, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary.
I would be interested to hear the tenant use-case where they trust the backing storage system (Ceph) with all things encryption and don't have any effective control over the keys / ciphers / rotation policies / etc. If you have a vulnerability that exposes the current OSD dm-crypt keys, I would think it would be possible to get the per-namespace keys though a similar vector if they are stored effectively side-by-side?
The idea here was not to berate the current at-rest scheme, nor propose something as better than, but a use case where it is used instead of. Maybe I'm being naive, but the trade-off of having the storage system handling the namespace keys is not much different than having it handling the dmcrypt keys.
I'm in no way saying that Ceph handling the secrets is better than having the clients doing their own encryption; it's just a different use case being addressed.
Totally understand. I am just honestly interested if this solves a known issue for Ceph users (regulatory or otherwise). i.e. would it check all the same boxes to implement pool-level encryption vs namespace-level encryption?
Indeed, pools could even be used today (one pool on dmcrypted drives, another on drives without encryption) for similar security properties.
This is the main driver for this proposal, actually. Indeed, having a policy of some pools going into dmcrypted osds vs others going into non-encrypted osds works. However, the problem we've been seeing is that there's demand for encrypting those pools per-tenant, with different keys; some of this may have legal implications. Now, the real issue is not with being unable to achieve this with pools, but the scale. Mapping a handful of tenants to specific pools may be feasible, but go a bit above that and you will end up with a complex crush hierarchy, and maybe not enough osds to ensure the mentioned requirements. And you'd end up with a lot of (potentially small) pools. The idea of encrypting namespaces instead aims at addressing this scenario. -Joao
On Tue, 2020-08-04 at 09:04 -0400, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
This proposal might be a bit thin on details, but I would love to have some feedback and gauge the broader community's and developer's interest, as well as to poke holes in the current idea.
All comments welcome.
-Joao
MOTIVATION ----------
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary.
I would be interested to hear the tenant use-case where they trust the backing storage system (Ceph) with all things encryption and don't have any effective control over the keys / ciphers / rotation policies / etc. If you have a vulnerability that exposes the current OSD dm-crypt keys, I would think it would be possible to get the per-namespace keys though a similar vector if they are stored effectively side-by-side?
Agreed. If I were a cloud tenant, I'd not be thrilled at a scheme that required me to trust the OSD to encrypt my cleartext data for me. You might want to take a step back and consider that there are really two problems you have to deal with: 1/ encryption: where and how do I perform the encryption of the data? I think doing this as close to the edges as possible would be best, as it means fewer trusted parties. Conceptually, that makes this part fairly straightforward. Just run the crypto over the appropriate buffers before you send and just after you receive. You will need to do things like ensure that someone with a bad key can't corrupt data in an encrypted namespace. 2/ key management: how do I get keys to feed into the crypto engine? This is the hard part, IMO. While their scheme isn't perfect, you may want to look closely at how Linux fscrypt works. Basically, each fs has a master key and then you derive keys for the individual inodes from that. Every inode has a nonce generated, that's used to ensure that (e.g.) two identical files don't have identical encrypted contents. The master keys are stored encrypted themselves, and the filesystem has a number of ways that you can set up to unlock them -- passwords, hard tokens, etc. Note that this part requires special tools to set up the keys. You'll probably want to aim for some sort of similar hierarchy of keys here, I think. You may need a "special" per-object xattr or something to store nonces, and you'll need to think about where they get generated. One question : are you planning to encrypt object names too? Ideally, you want to allow any client-generated data to be encrypted. In a filesystem, that's basically filenames and contents. An object store is probably pretty similar in that regard.
While there are very good arguments for ensuring this encryption is performed client-side, such that each client actively controls their own secrets, a server-side approach has several other benefits that may outweigh a client-side approach.
On the one hand,
* encrypting server side means encrypting N times, depending on replication size and scheme; * the secrets keyring will be centralized, likely in the monitor, much like what we do for dmcrypt; even though encrypted. * on-the-wire data will still need to rely on msgr2 encryption; even though one could argue that this will likely happen regardless of whether a client- or server-side approach is being used.
But on the other,
1. encryption becomes transparent for the client, avoiding the effort of implementing such schemes in client libraries and kernel drivers;
Just an FYI: krbd supports client-side via dm-crypt, kernel CephFS is actively looking to incorporate fscrypt, librbd can utilize QEMU-layered LUKS for many use-cases and work is in-progress on built-in librbd client-side encryption. RGW has had client-side encryption for a while.
2. tighter control over the unit of data being encrypted, reducing the load of encrypting a whole object versus a disk block in bluestore.
RBD client-side encryption doesn't rely on the underlying object size (512 bytes for dm-crypt I think and looking at 4KiB blocks for the librbd built-in encryption). I can't speak for CephFS+fscrypt, but I suspect it wouldn't require re-encrypting the full file or backing object (probably 4KiB page).
For fscrypt, the client basically just encrypts/decrypts file data on a per-block basis. For ceph, that means we'll just operate on it a page at a time. fwiw, the data path looks reasonably simple to deal with. The hard part is dealing with encrypted filenames.
3. older clients will be able to support encryption out of the box, given they will have no idea their data is being encrypted, nor on how that is happening.
CHOOSING NAMESPACES --------------------
While investigating where and how per-tenant encryption could be implemented, two other ideas were on the table:
1. on a per-client basis, relying on cephx entities, with an encryption key per-client, or a shared key amongst several clients; this key would be kept encrypted in the monitor's kv store with the entity's cephx key.
2. on a per-pool basis.
The first one would definitely be feasible, but potentially tricky to implement just right, without too many exceptions or involvement of other portions of the stack. E.g., dealing with metadata could become tricky. Then again, there wasn't one reason that could not be addressed and become a showstopper.
As for 2., it would definitely be the easiest to implement: pool is created with an 'encrypted' flag on, key is kept in the monitors, OSDs encrypt any object belonging to that pool. The problem with this option, however, is how coarse-grained it is. If we really wanted a per-tenant approach, one would have to ensure one pool per tenant. Not necessarily a big deal if a lot of potentially small pools is fine. This idea was scrapped in favour of encrypting namespaces instead.
Given RADOS already has the concept of a namespace, it might just be the ideal medium to implement such an approach, as we get the best of the two options above: we can get a smaller-grained access than a pool, but still with the same capabilities of limiting access by entity through caps. We also get to have multiple namespaces in a single pool should we choose to do so. All the while, the concept is high-level enough that the effort of implementing the actual encryption scheme might be performed in a select, handful of places, without the need for a lot (maybe, any) particular exceptions or corner cases.
APPROACH ---------
It is important to note that there are several implementations details, especially on "how exactly this is going to happen", that have not been fully figured out.
Essentially, the objective is to ensure that objects from a given namespace are always encrypted or decrypted by bluestore when writing or reading the data. The hope that performing at this level will allow us to
1. ensure the operation is performed at the disk block size, ensuring that small writes, or partial writes, will not require a rewrite of the whole object; same goes for reads.
2. avoid dealing with all the mechanics involving objects and other operations over them, and focus solely on their data and metadata.
Secret distribution is expected to be done by the monitors, at the OSDs request. In an ideal world, the OSDs would know exactly which namespaces they might have to encrypt/decrypt, based on pools they currently hold, and request keys for those before hand, such that they don't have to request a key from the monitor when an operation arrives. This would not only require us to become a bit more aware of namespaces, but keeping these keys cached might require the osd to keep them encrypted in memory. What to use for that is something that hasn't been much thought about -- maybe we could get away with using the osd's cephx key.
As for the namespaces, in their current form we don't have much (any?) information about them. Access to an object in a namespace is based on prior knowledge of that namespace and the object's name. We currently don't have statistics on namespaces, nor are we able to know whether an OSD keeps any object belonging to a namespace _before_ an operation on such an object is handled.
Even though it's not particularly _required_ to get more out of namespaces than we currently have, it would definitely be ideal if we ended up with the ability to 1) have statistics out of namespaces, as it would imperative if we're using them for tenants; and 2) able to cache ahead keys for namespaces an osd might have to handle (read, namespaces living in a pool with PGs mapped to a given osd).
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jeff Layton <jlayton@redhat.com>
On 20/08/04 01:16PM, Jeff Layton wrote:
On Tue, 2020-08-04 at 09:04 -0400, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary.
I would be interested to hear the tenant use-case where they trust the backing storage system (Ceph) with all things encryption and don't have any effective control over the keys / ciphers / rotation policies / etc. If you have a vulnerability that exposes the current OSD dm-crypt keys, I would think it would be possible to get the per-namespace keys though a similar vector if they are stored effectively side-by-side?
Agreed. If I were a cloud tenant, I'd not be thrilled at a scheme that required me to trust the OSD to encrypt my cleartext data for me.
You might want to take a step back and consider that there are really two problems you have to deal with:
1/ encryption: where and how do I perform the encryption of the data?
I think doing this as close to the edges as possible would be best, as it means fewer trusted parties. Conceptually, that makes this part fairly straightforward. Just run the crypto over the appropriate buffers before you send and just after you receive. You will need to do things like ensure that someone with a bad key can't corrupt data in an encrypted namespace.
That is a very fair point, and I don't disagree. I do think that having the client handling their own encryption, and being in control of their secrets, is the safest approach. However, the use cases brought to us have been somewhat like what we have with at-rest encryption with dmcrypt: the provider controls the keys, and retiring a tenant is simply a matter of destroying their key. Mind you, this is something we could achieve with pools, if the granularity wasn't so coarse IMO.
2/ key management: how do I get keys to feed into the crypto engine?
This is the hard part, IMO. While their scheme isn't perfect, you may want to look closely at how Linux fscrypt works. Basically, each fs has a master key and then you derive keys for the individual inodes from that. Every inode has a nonce generated, that's used to ensure that (e.g.) two identical files don't have identical encrypted contents.
Definitely agree. In this whole proposal, it's the encryption portion (a small detail, I know) that scares me the most, and is something that needs proper ironing before this could even be phantomed to be implemented.
The master keys are stored encrypted themselves, and the filesystem has a number of ways that you can set up to unlock them -- passwords, hard tokens, etc. Note that this part requires special tools to set up the keys.
You'll probably want to aim for some sort of similar hierarchy of keys here, I think. You may need a "special" per-object xattr or something to store nonces, and you'll need to think about where they get generated.
One question : are you planning to encrypt object names too? Ideally, you want to allow any client-generated data to be encrypted. In a filesystem, that's basically filenames and contents. An object store is probably pretty similar in that regard.
That would be the plan. The intent would be to encrypt pretty much anything that is encryptable, data and metadata. From the object store, the only thing the client needs to know is the object the data is stored in. AFAIU, this is currently achieved by hashing the namespace and the object name. From the storage backend's perspective, accessing an object in an encrypted namespace means encrypting or decrypting whatever the client wants. I'm not entirely sure how feasible it will be to encrypt all on-disk metadata, but after speaking with Igor about this I came out under the impression that it would be feasible to do so.
While there are very good arguments for ensuring this encryption is performed client-side, such that each client actively controls their own secrets, a server-side approach has several other benefits that may outweigh a client-side approach.
On the one hand,
* encrypting server side means encrypting N times, depending on replication size and scheme; * the secrets keyring will be centralized, likely in the monitor, much like what we do for dmcrypt; even though encrypted. * on-the-wire data will still need to rely on msgr2 encryption; even though one could argue that this will likely happen regardless of whether a client- or server-side approach is being used.
But on the other,
1. encryption becomes transparent for the client, avoiding the effort of implementing such schemes in client libraries and kernel drivers;
Just an FYI: krbd supports client-side via dm-crypt, kernel CephFS is actively looking to incorporate fscrypt, librbd can utilize QEMU-layered LUKS for many use-cases and work is in-progress on built-in librbd client-side encryption. RGW has had client-side encryption for a while.
2. tighter control over the unit of data being encrypted, reducing the load of encrypting a whole object versus a disk block in bluestore.
RBD client-side encryption doesn't rely on the underlying object size (512 bytes for dm-crypt I think and looking at 4KiB blocks for the librbd built-in encryption). I can't speak for CephFS+fscrypt, but I suspect it wouldn't require re-encrypting the full file or backing object (probably 4KiB page).
For fscrypt, the client basically just encrypts/decrypts file data on a per-block basis. For ceph, that means we'll just operate on it a page at a time.
fwiw, the data path looks reasonably simple to deal with. The hard part is dealing with encrypted filenames.
I'm guessing these filenames are being kept somewhere in an object in the object store? Would this also pose issues with xattrs and other metadata? Solely as an exercise, should the client encrypt the data and write it to the cluster, and write the metadata to an encrypted namespace, I believe the object itself could simply be encrypted as a whole, transparently for the client. Granted, at this point I'm not entirely sure how hard it would be to, or if it's even feasible to, encrypt the internal metadata used by bluestore, but from my conversation with Igor it seemed that encrypting the object's metadata would be feasible. -Joao
On Wed, 2020-08-05 at 11:22 +0000, Joao Eduardo Luis wrote:
On 20/08/04 01:16PM, Jeff Layton wrote:
On Tue, 2020-08-04 at 09:04 -0400, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
Is it really valuable to allow different entities to use different keys, when the OSDs have access to all of them? If the OSD's master store of keys is compromised then all of the tenant data is compromised. This wouldn't necessarily be the case with a scheme where the encryption is done on the clients.
The intent here is to allow different tenants to have their data encrypted at rest, independently, and without necessarily relying on full osd encryption. This way one could have anywhere between a handful to dozens or hundreds of tenants with their data encrypted on disk, while not having to maintain full at-rest encryption should the administrator consider it too cumbersome or unnecessary.
I would be interested to hear the tenant use-case where they trust the backing storage system (Ceph) with all things encryption and don't have any effective control over the keys / ciphers / rotation policies / etc. If you have a vulnerability that exposes the current OSD dm-crypt keys, I would think it would be possible to get the per-namespace keys though a similar vector if they are stored effectively side-by-side?
Agreed. If I were a cloud tenant, I'd not be thrilled at a scheme that required me to trust the OSD to encrypt my cleartext data for me.
You might want to take a step back and consider that there are really two problems you have to deal with:
1/ encryption: where and how do I perform the encryption of the data?
I think doing this as close to the edges as possible would be best, as it means fewer trusted parties. Conceptually, that makes this part fairly straightforward. Just run the crypto over the appropriate buffers before you send and just after you receive. You will need to do things like ensure that someone with a bad key can't corrupt data in an encrypted namespace.
That is a very fair point, and I don't disagree. I do think that having the client handling their own encryption, and being in control of their secrets, is the safest approach.
However, the use cases brought to us have been somewhat like what we have with at-rest encryption with dmcrypt: the provider controls the keys, and retiring a tenant is simply a matter of destroying their key.
Mind you, this is something we could achieve with pools, if the granularity wasn't so coarse IMO.
I guess I'm not 100% clear on the use-case for this. Are there really folks just interested in knowing that the OSD will store the data encrypted and don't really care that it may deal with unencrypted data in its memory? Is this being driven by some sort of regulatory requirement?
2/ key management: how do I get keys to feed into the crypto engine?
This is the hard part, IMO. While their scheme isn't perfect, you may want to look closely at how Linux fscrypt works. Basically, each fs has a master key and then you derive keys for the individual inodes from that. Every inode has a nonce generated, that's used to ensure that (e.g.) two identical files don't have identical encrypted contents.
Definitely agree. In this whole proposal, it's the encryption portion (a small detail, I know) that scares me the most, and is something that needs proper ironing before this could even be phantomed to be implemented.
The master keys are stored encrypted themselves, and the filesystem has a number of ways that you can set up to unlock them -- passwords, hard tokens, etc. Note that this part requires special tools to set up the keys.
You'll probably want to aim for some sort of similar hierarchy of keys here, I think. You may need a "special" per-object xattr or something to store nonces, and you'll need to think about where they get generated.
One question : are you planning to encrypt object names too? Ideally, you want to allow any client-generated data to be encrypted. In a filesystem, that's basically filenames and contents. An object store is probably pretty similar in that regard.
That would be the plan. The intent would be to encrypt pretty much anything that is encryptable, data and metadata. From the object store, the only thing the client needs to know is the object the data is stored in. AFAIU, this is currently achieved by hashing the namespace and the object name. From the storage backend's perspective, accessing an object in an encrypted namespace means encrypting or decrypting whatever the client wants.
I'm not entirely sure how feasible it will be to encrypt all on-disk metadata, but after speaking with Igor about this I came out under the impression that it would be feasible to do so.
While there are very good arguments for ensuring this encryption is performed client-side, such that each client actively controls their own secrets, a server-side approach has several other benefits that may outweigh a client-side approach.
On the one hand,
* encrypting server side means encrypting N times, depending on replication size and scheme; * the secrets keyring will be centralized, likely in the monitor, much like what we do for dmcrypt; even though encrypted. * on-the-wire data will still need to rely on msgr2 encryption; even though one could argue that this will likely happen regardless of whether a client- or server-side approach is being used.
But on the other,
1. encryption becomes transparent for the client, avoiding the effort of implementing such schemes in client libraries and kernel drivers;
Just an FYI: krbd supports client-side via dm-crypt, kernel CephFS is actively looking to incorporate fscrypt, librbd can utilize QEMU-layered LUKS for many use-cases and work is in-progress on built-in librbd client-side encryption. RGW has had client-side encryption for a while.
2. tighter control over the unit of data being encrypted, reducing the load of encrypting a whole object versus a disk block in bluestore.
RBD client-side encryption doesn't rely on the underlying object size (512 bytes for dm-crypt I think and looking at 4KiB blocks for the librbd built-in encryption). I can't speak for CephFS+fscrypt, but I suspect it wouldn't require re-encrypting the full file or backing object (probably 4KiB page).
For fscrypt, the client basically just encrypts/decrypts file data on a per-block basis. For ceph, that means we'll just operate on it a page at a time.
fwiw, the data path looks reasonably simple to deal with. The hard part is dealing with encrypted filenames.
I'm guessing these filenames are being kept somewhere in an object in the object store? Would this also pose issues with xattrs and other metadata?
No, with fscrypt, the filenames are encrypted in place, such that the filenames themselves are encrypted and you need a key to see their unencrypted forms in a readdir(). What is sort of cool is that you can still access the tree without a key at all, but all you see is encrypted filenames, and you can't do anything with the files (other than unlink(), assuming you have the correct permissions). The main downside of fscrypt is that it can't encrypt metadata at all. Stuff like the mtime or file size is under the purview of the filesystem (the Ceph MDS in our case). It'd also be dangerous to (e.g.) show a scrambled mode or ownership for the file, as you might not be able to predict how applications might behave.
Solely as an exercise, should the client encrypt the data and write it to the cluster, and write the metadata to an encrypted namespace, I believe the object itself could simply be encrypted as a whole, transparently for the client.
Encrypting metadata is going to pretty much be impossible from a solely client-side solution. The client doesn't have much control over stuff like mtime. That's all managed on the server side.
Granted, at this point I'm not entirely sure how hard it would be to, or if it's even feasible to, encrypt the internal metadata used by bluestore, but from my conversation with Igor it seemed that encrypting the object's metadata would be feasible.
Yeah, if you have the OSD doing the encryption for you, then it could (theoretically) store encrypted metadata too. One thing you should consider as well: What will you do if someone has a bunch of encrypted data and the key is (somehow) lost? You'll need some way to be able to blow away old objects that can't be accessed anymore. -- Jeff Layton <jlayton@redhat.com>
On 20/08/05 08:29AM, Jeff Layton wrote:
On Wed, 2020-08-05 at 11:22 +0000, Joao Eduardo Luis wrote:
On 20/08/04 01:16PM, Jeff Layton wrote:
On Tue, 2020-08-04 at 09:04 -0400, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
Is it really valuable to allow different entities to use different keys, when the OSDs have access to all of them? If the OSD's master store of keys is compromised then all of the tenant data is compromised. This wouldn't necessarily be the case with a scheme where the encryption is done on the clients.
I mean, this is what we're doing for dmcrypt anyway, is it not? I don't think the idea of encrypting namespaces should be seen as the ultimate alternative to encrypting client-side, but as a mechanism similar to what we do with at-rest encryption with dmcrypt, except on subdivisions of a pool.
That is a very fair point, and I don't disagree. I do think that having the client handling their own encryption, and being in control of their secrets, is the safest approach.
However, the use cases brought to us have been somewhat like what we have with at-rest encryption with dmcrypt: the provider controls the keys, and retiring a tenant is simply a matter of destroying their key.
Mind you, this is something we could achieve with pools, if the granularity wasn't so coarse IMO.
I guess I'm not 100% clear on the use-case for this.
Are there really folks just interested in knowing that the OSD will store the data encrypted and don't really care that it may deal with unencrypted data in its memory? Is this being driven by some sort of regulatory requirement?
AFAIK, this is already the case for the at-rest encryption. The data is still unencrypted in memory, and folks are just fine as long as the data is encrypted at rest. The principle is essentially the same, I think.
I'm guessing these filenames are being kept somewhere in an object in the object store? Would this also pose issues with xattrs and other metadata?
No, with fscrypt, the filenames are encrypted in place, such that the filenames themselves are encrypted and you need a key to see their unencrypted forms in a readdir().
What is sort of cool is that you can still access the tree without a key at all, but all you see is encrypted filenames, and you can't do anything with the files (other than unlink(), assuming you have the correct permissions).
The main downside of fscrypt is that it can't encrypt metadata at all. Stuff like the mtime or file size is under the purview of the filesystem (the Ceph MDS in our case). It'd also be dangerous to (e.g.) show a scrambled mode or ownership for the file, as you might not be able to predict how applications might behave.
Solely as an exercise, should the client encrypt the data and write it to the cluster, and write the metadata to an encrypted namespace, I believe the object itself could simply be encrypted as a whole, transparently for the client.
Encrypting metadata is going to pretty much be impossible from a solely client-side solution. The client doesn't have much control over stuff like mtime. That's all managed on the server side.
Granted, at this point I'm not entirely sure how hard it would be to, or if it's even feasible to, encrypt the internal metadata used by bluestore, but from my conversation with Igor it seemed that encrypting the object's metadata would be feasible.
Yeah, if you have the OSD doing the encryption for you, then it could (theoretically) store encrypted metadata too.
One thing you should consider as well: What will you do if someone has a bunch of encrypted data and the key is (somehow) lost? You'll need some way to be able to blow away old objects that can't be accessed anymore.
That is one of the reasons why I think namespaces need to become a bit more self-aware: should one need to remove all objects belonging to a namespace, at the moment, without knowing the objects beforehand, AFAIK one will need to traverse a pool to find all the objects one by one. But to your point, say that we are encrypting a namespace and the encryption key is lost; that namespace is essentially useless now. The only reasonable action is to remove the namespace's objects (although I'd leave that decision to the administrator), even if, in the worst case scenario, one has to iterate over a pool and check object by object whether it belongs to a namespace to be destroyed.
On Wed, Aug 05, 2020 at 02:38:09PM +0000, Joao Eduardo Luis wrote:
On 20/08/05 08:29AM, Jeff Layton wrote:
On Wed, 2020-08-05 at 11:22 +0000, Joao Eduardo Luis wrote:
On 20/08/04 01:16PM, Jeff Layton wrote:
On Tue, 2020-08-04 at 09:04 -0400, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@suse.de> wrote:
Even though we currently have at-rest encryption, ensuring data security on the physical device, this is currently on an OSD-basis, and it is too coarse-grained to allow different entities/clients/tenants to have their data encrypted with different keys.
Is it really valuable to allow different entities to use different keys, when the OSDs have access to all of them? If the OSD's master store of keys is compromised then all of the tenant data is compromised. This wouldn't necessarily be the case with a scheme where the encryption is done on the clients.
I mean, this is what we're doing for dmcrypt anyway, is it not?
I don't think the idea of encrypting namespaces should be seen as the ultimate alternative to encrypting client-side, but as a mechanism similar to what we do with at-rest encryption with dmcrypt, except on subdivisions of a pool.
That is a very fair point, and I don't disagree. I do think that having the client handling their own encryption, and being in control of their secrets, is the safest approach.
However, the use cases brought to us have been somewhat like what we have with at-rest encryption with dmcrypt: the provider controls the keys, and retiring a tenant is simply a matter of destroying their key.
Mind you, this is something we could achieve with pools, if the granularity wasn't so coarse IMO.
I guess I'm not 100% clear on the use-case for this.
Are there really folks just interested in knowing that the OSD will store the data encrypted and don't really care that it may deal with unencrypted data in its memory? Is this being driven by some sort of regulatory requirement?
Just to expand on Joao's points a little. Afaiu the use case can be considered as a quick and safe data delete for tenants. Delete might be the wrong term here but "make data inaccessible" is a bit clunky. I think something regulatory plays a role here but essentially one requirement this can address is to quickly and reliably make a tenants data inaccessible (by deleting the encryption key). I think this probably shouldn't be labeled as encryption in the sense of keeping data confidential in the presence of adversarial agents, but rather encryption can be looked at as a means to an end...safely deleting data at a certain time.
AFAIK, this is already the case for the at-rest encryption. The data is still unencrypted in memory, and folks are just fine as long as the data is encrypted at rest. The principle is essentially the same, I think.
I'm guessing these filenames are being kept somewhere in an object in the object store? Would this also pose issues with xattrs and other metadata?
No, with fscrypt, the filenames are encrypted in place, such that the filenames themselves are encrypted and you need a key to see their unencrypted forms in a readdir().
What is sort of cool is that you can still access the tree without a key at all, but all you see is encrypted filenames, and you can't do anything with the files (other than unlink(), assuming you have the correct permissions).
The main downside of fscrypt is that it can't encrypt metadata at all. Stuff like the mtime or file size is under the purview of the filesystem (the Ceph MDS in our case). It'd also be dangerous to (e.g.) show a scrambled mode or ownership for the file, as you might not be able to predict how applications might behave.
Solely as an exercise, should the client encrypt the data and write it to the cluster, and write the metadata to an encrypted namespace, I believe the object itself could simply be encrypted as a whole, transparently for the client.
Encrypting metadata is going to pretty much be impossible from a solely client-side solution. The client doesn't have much control over stuff like mtime. That's all managed on the server side.
Granted, at this point I'm not entirely sure how hard it would be to, or if it's even feasible to, encrypt the internal metadata used by bluestore, but from my conversation with Igor it seemed that encrypting the object's metadata would be feasible.
Yeah, if you have the OSD doing the encryption for you, then it could (theoretically) store encrypted metadata too.
One thing you should consider as well: What will you do if someone has a bunch of encrypted data and the key is (somehow) lost? You'll need some way to be able to blow away old objects that can't be accessed anymore.
That is one of the reasons why I think namespaces need to become a bit more self-aware: should one need to remove all objects belonging to a namespace, at the moment, without knowing the objects beforehand, AFAIK one will need to traverse a pool to find all the objects one by one.
But to your point, say that we are encrypting a namespace and the encryption key is lost; that namespace is essentially useless now. The only reasonable action is to remove the namespace's objects (although I'd leave that decision to the administrator), even if, in the worst case scenario, one has to iterate over a pool and check object by object whether it belongs to a namespace to be destroyed. _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
On 2020-08-06T18:55:29, Jan Fajerski <jfajerski@suse.com> wrote: Hi all, let me try to add a few comments on the use case.
Just to expand on Joao's points a little. Afaiu the use case can be considered as a quick and safe data delete for tenants. Delete might be the wrong term here but "make data inaccessible" is a bit clunky. I think something regulatory plays a role here but essentially one requirement this can address is to quickly and reliably make a tenants data inaccessible (by deleting the encryption key). I think this probably shouldn't be labeled as encryption in the sense of keeping data confidential in the presence of adversarial agents, but rather encryption can be looked at as a means to an end...safely deleting data at a certain time.
So, yes. Dynamic multi-tenancy. This includes the ability to provision but also deprovision differently encrypted namespaces. Doing this at the full pool (consisting of dedicated OSDs) level is too coarse. It'd require significant overhead in terms of OSD layout/deployment, CRUSH map editing etc. Pools in general are a fairly clunky concept in Ceph. If all my tenants have similar redundancy needs but aren't supposed to share the same namespace and have different data amounts, I don't want a million pools. Hence, namespaces as a more lightweight construct (even if there's parts missing that make namespaces truly useful as "virtual pools", there's hopefully more coming ;-). There's a minor benefit for tenant isolation - if, by accident, data is exposed, unless the client also supplied the right key as part of their CephX auth, perhaps they can't access it. So even if they steal the OSD key, maybe they've not stolen the tenant key. But that's somewhat of a thinner security blanket rather than an actual bunker. Though I'd like to submit for consideration that Amazon's S3 "SSE" threat model is pretty much similar. As for client side encryption: that is, indeed, an option. It's not mutually exclusive. But from what I understand, not necessarily easier, and requires modifying all clients and making sure they're compatible. (For RBD, btw, this can be done via storage classes in k8s, which can be independently encrypted. But this is exclusive access, not shared data.) For shared file (CephFS), which is our key focus here so far, client side encryption would be hard - and file/directory names and attribute values can easily be considered risky data. (None of the encrypted file system layers supports a shared file system underneath.) We could fairly easily stand up an MDS set that's pointed at a (potentially encrypted) dedicated namespace though. Also, if the performance needs otherwise require unencrypted data, this allows for encrypting only a subset of data w/o full OSD encryption. I understand and agree that it's not perfectly as awesome as full, consistent client-side encryption for everything. After long discussions, though, this is better than the current status, and seemed less difficult than full client side encryption, while ticking all the customer requirements and regulatory arguments. If we can more easily add the client-side encryption to the RADOS client protocol (so it works consistently with RBD, CephFS data and metadata paths, RGW, etc), that'd be awesome. I think. Regards, Lars -- SUSE Software Solutions Germany GmbH, MD: Felix Imendörffer, HRB 36809 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
participants (6)
-
Jan Fajerski
-
Jason Dillaman
-
Jeff Layton
-
Joao Eduardo Luis
-
Josh Durgin
-
Lars Marowsky-Bree