rgw: compression+encryption bug in multisite
hey Shilpa and team, early in the reef cycle, https://github.com/ceph/ceph/pull/46188 was contributed to support the combination of server-side compression and encryption on the same object data. only recently did we catch a regression in multisite, where such objects fail to replicate and can cause crashes. this bug, tracked in https://tracker.ceph.com/issues/57905, was just fixed and backported for reef in https://github.com/ceph/ceph/pull/52297. this was a regression in reef, so i was planning to treat it as a blocker in that backport, i added a warning to the original release note: RGW: Compression is now supported for objects uploaded with Server-Side Encryption. When both are enabled, compression is applied before encryption. WARNING: In a multisite configuration, objects that are both compressed and encrypted will not replicate correctly to Pacific or Quincy. Upgrade all zones to Reef before enabling compression. it occurs to me that we might add a new 'compress-encrypted' feature flag to the zonegroup (similar to the 'resharding' flag in reef) to prevent this combination of compression+encryption until all zones upgrade and enable it. do you think that's worth doing, or is a release note sufficient?
i opened https://github.com/ceph/ceph/pull/52300 to require a 'compress-encrypted' zonegroup feature for this, and updated the documentation and release note accordingly On Mon, Jul 3, 2023 at 2:11 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Shilpa and team,
early in the reef cycle, https://github.com/ceph/ceph/pull/46188 was contributed to support the combination of server-side compression and encryption on the same object data. only recently did we catch a regression in multisite, where such objects fail to replicate and can cause crashes. this bug, tracked in https://tracker.ceph.com/issues/57905, was just fixed and backported for reef in https://github.com/ceph/ceph/pull/52297. this was a regression in reef, so i was planning to treat it as a blocker
in that backport, i added a warning to the original release note:
RGW: Compression is now supported for objects uploaded with Server-Side Encryption. When both are enabled, compression is applied before encryption. WARNING: In a multisite configuration, objects that are both compressed and encrypted will not replicate correctly to Pacific or Quincy. Upgrade all zones to Reef before enabling compression.
it occurs to me that we might add a new 'compress-encrypted' feature flag to the zonegroup (similar to the 'resharding' flag in reef) to prevent this combination of compression+encryption until all zones upgrade and enable it. do you think that's worth doing, or is a release note sufficient?
Please note that compression before encryption is considered a security breach. I would not implement this without a clear warning and specific user approval. Regards, Josh On Mon, Jul 3, 2023 at 10:54 PM Casey Bodley <cbodley@redhat.com> wrote:
i opened https://github.com/ceph/ceph/pull/52300 to require a 'compress-encrypted' zonegroup feature for this, and updated the documentation and release note accordingly
On Mon, Jul 3, 2023 at 2:11 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Shilpa and team,
early in the reef cycle, https://github.com/ceph/ceph/pull/46188 was contributed to support the combination of server-side compression and encryption on the same object data. only recently did we catch a regression in multisite, where such objects fail to replicate and can cause crashes. this bug, tracked in https://tracker.ceph.com/issues/57905, was just fixed and backported for reef in https://github.com/ceph/ceph/pull/52297. this was a regression in reef, so i was planning to treat it as a blocker
in that backport, i added a warning to the original release note:
RGW: Compression is now supported for objects uploaded with Server-Side Encryption. When both are enabled, compression is applied before encryption. WARNING: In a multisite configuration, objects that are both compressed and encrypted will not replicate correctly to Pacific or Quincy. Upgrade all zones to Reef before enabling compression.
it occurs to me that we might add a new 'compress-encrypted' feature flag to the zonegroup (similar to the 'resharding' flag in reef) to prevent this combination of compression+encryption until all zones upgrade and enable it. do you think that's worth doing, or is a release note sufficient?
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
How so? Encrypting first would I suspect confound the compressibility of data.
Please note that compression before encryption is considered a security breach. I would not implement this without a clear warning and specific user approval.
Regards,
Josh
On Mon, Jul 3, 2023 at 10:54 PM Casey Bodley <cbodley@redhat.com <mailto:cbodley@redhat.com>> wrote:
i opened https://github.com/ceph/ceph/pull/52300 to require a 'compress-encrypted' zonegroup feature for this, and updated the documentation and release note accordingly
On Mon, Jul 3, 2023 at 2:11 PM Casey Bodley <cbodley@redhat.com <mailto:cbodley@redhat.com>> wrote:
hey Shilpa and team,
early in the reef cycle, https://github.com/ceph/ceph/pull/46188 was contributed to support the combination of server-side compression and encryption on the same object data. only recently did we catch a regression in multisite, where such objects fail to replicate and can cause crashes. this bug, tracked in https://tracker.ceph.com/issues/57905, was just fixed and backported for reef in https://github.com/ceph/ceph/pull/52297. this was a regression in reef, so i was planning to treat it as a blocker
in that backport, i added a warning to the original release note:
RGW: Compression is now supported for objects uploaded with Server-Side Encryption. When both are enabled, compression is applied before encryption. WARNING: In a multisite configuration, objects that are both compressed and encrypted will not replicate correctly to Pacific or Quincy. Upgrade all zones to Reef before enabling compression.
it occurs to me that we might add a new 'compress-encrypted' feature flag to the zonegroup (similar to the 'resharding' flag in reef) to prevent this combination of compression+encryption until all zones upgrade and enable it. do you think that's worth doing, or is a release note sufficient?
_______________________________________________ Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io>
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
thanks Josh, i wasn't aware of attacks in this space. a naive search came up with a blog post from minio about a 'compression-ratio side channel' at https://blog.min.io/c-e-compression-encryption/. is that the same kind of issue you're concerned about here? without more time to evaluate this, i'm tempted to disable the feature for reef On Tue, Jul 4, 2023 at 3:13 AM Josh Salomon <jsalomon@redhat.com> wrote:
Please note that compression before encryption is considered a security breach. I would not implement this without a clear warning and specific user approval.
Regards,
Josh
On Mon, Jul 3, 2023 at 10:54 PM Casey Bodley <cbodley@redhat.com> wrote:
i opened https://github.com/ceph/ceph/pull/52300 to require a 'compress-encrypted' zonegroup feature for this, and updated the documentation and release note accordingly
On Mon, Jul 3, 2023 at 2:11 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Shilpa and team,
early in the reef cycle, https://github.com/ceph/ceph/pull/46188 was contributed to support the combination of server-side compression and encryption on the same object data. only recently did we catch a regression in multisite, where such objects fail to replicate and can cause crashes. this bug, tracked in https://tracker.ceph.com/issues/57905, was just fixed and backported for reef in https://github.com/ceph/ceph/pull/52297. this was a regression in reef, so i was planning to treat it as a blocker
in that backport, i added a warning to the original release note:
RGW: Compression is now supported for objects uploaded with Server-Side Encryption. When both are enabled, compression is applied before encryption. WARNING: In a multisite configuration, objects that are both compressed and encrypted will not replicate correctly to Pacific or Quincy. Upgrade all zones to Reef before enabling compression.
it occurs to me that we might add a new 'compress-encrypted' feature flag to the zonegroup (similar to the 'resharding' flag in reef) to prevent this combination of compression+encryption until all zones upgrade and enable it. do you think that's worth doing, or is a release note sufficient?
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
The "active" strategy outlined here seems to be pretty strange. The attacker's data needs to be combined into the unencrypted data being sent via S3, so I guess it has compromised the S3 client environment or dataset already in some way? I'm not sure if that qualifies as a direct attack on compression+encryption in the S3 service. In general, it seems like whether to compress+encrypt should be a policy decision, not something arbitrarily chosen by the S3 implementation (as this article claims minio does). Matt On Wed, Jul 5, 2023 at 10:24 AM Casey Bodley <cbodley@redhat.com> wrote:
thanks Josh, i wasn't aware of attacks in this space. a naive search came up with a blog post from minio about a 'compression-ratio side channel' at https://blog.min.io/c-e-compression-encryption/. is that the same kind of issue you're concerned about here?
without more time to evaluate this, i'm tempted to disable the feature for reef
On Tue, Jul 4, 2023 at 3:13 AM Josh Salomon <jsalomon@redhat.com> wrote:
Please note that compression before encryption is considered a security
breach. I would not implement this without a clear warning and specific user approval.
Regards,
Josh
On Mon, Jul 3, 2023 at 10:54 PM Casey Bodley <cbodley@redhat.com> wrote:
i opened https://github.com/ceph/ceph/pull/52300 to require a 'compress-encrypted' zonegroup feature for this, and updated the documentation and release note accordingly
On Mon, Jul 3, 2023 at 2:11 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Shilpa and team,
early in the reef cycle, https://github.com/ceph/ceph/pull/46188 was contributed to support the combination of server-side compression and encryption on the same object data. only recently did we catch a regression in multisite, where such objects fail to replicate and can cause crashes. this bug, tracked in https://tracker.ceph.com/issues/57905, was just fixed and backported for reef in https://github.com/ceph/ceph/pull/52297. this was a regression in reef, so i was planning to treat it as a blocker
in that backport, i added a warning to the original release note:
RGW: Compression is now supported for objects uploaded with Server-Side Encryption. When both are enabled, compression is applied before encryption. WARNING: In a multisite configuration, objects that are both compressed and encrypted will not replicate correctly to Pacific or Quincy. Upgrade all zones to Reef before enabling compression.
it occurs to me that we might add a new 'compress-encrypted' feature flag to the zonegroup (similar to the 'resharding' flag in reef) to prevent this combination of compression+encryption until all zones upgrade and enable it. do you think that's worth doing, or is a release note sufficient?
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
On Wed, Jul 5, 2023 at 10:38 AM Matt Benjamin <mbenjami@redhat.com> wrote:
The "active" strategy outlined here seems to be pretty strange. The attacker's data needs to be combined into the unencrypted data being sent via S3, so I guess it has compromised the S3 client environment or dataset already in some way? I'm not sure if that qualifies as a direct attack on compression+encryption in the S3 service.
the active version does sound a bit contrived. but the passive version does leak some information about the data at rest, and server-side encryption is meant to protect against storage admins. any rados client with read access to the data pool can calculate these compression ratios to test whether two same-sized s3 objects might contain different data
In general, it seems like whether to compress+encrypt should be a policy decision, not something arbitrarily chosen by the S3 implementation (as this article claims minio does).
agreed, but the default should be off, and we'd need to understand and clearly document the tradeoffs. the current 'zonegroup feature' mechanism defaults to on for new zones, so https://github.com/ceph/ceph/pull/52300 isn't sufficient as-is
Matt
On Wed, Jul 5, 2023 at 10:24 AM Casey Bodley <cbodley@redhat.com> wrote:
thanks Josh, i wasn't aware of attacks in this space. a naive search came up with a blog post from minio about a 'compression-ratio side channel' at https://blog.min.io/c-e-compression-encryption/. is that the same kind of issue you're concerned about here?
without more time to evaluate this, i'm tempted to disable the feature for reef
On Tue, Jul 4, 2023 at 3:13 AM Josh Salomon <jsalomon@redhat.com> wrote:
Please note that compression before encryption is considered a security breach. I would not implement this without a clear warning and specific user approval.
Regards,
Josh
On Mon, Jul 3, 2023 at 10:54 PM Casey Bodley <cbodley@redhat.com> wrote:
i opened https://github.com/ceph/ceph/pull/52300 to require a 'compress-encrypted' zonegroup feature for this, and updated the documentation and release note accordingly
On Mon, Jul 3, 2023 at 2:11 PM Casey Bodley <cbodley@redhat.com> wrote:
hey Shilpa and team,
early in the reef cycle, https://github.com/ceph/ceph/pull/46188 was contributed to support the combination of server-side compression and encryption on the same object data. only recently did we catch a regression in multisite, where such objects fail to replicate and can cause crashes. this bug, tracked in https://tracker.ceph.com/issues/57905, was just fixed and backported for reef in https://github.com/ceph/ceph/pull/52297. this was a regression in reef, so i was planning to treat it as a blocker
in that backport, i added a warning to the original release note:
RGW: Compression is now supported for objects uploaded with Server-Side Encryption. When both are enabled, compression is applied before encryption. WARNING: In a multisite configuration, objects that are both compressed and encrypted will not replicate correctly to Pacific or Quincy. Upgrade all zones to Reef before enabling compression.
it occurs to me that we might add a new 'compress-encrypted' feature flag to the zonegroup (similar to the 'resharding' flag in reef) to prevent this combination of compression+encryption until all zones upgrade and enable it. do you think that's worth doing, or is a release note sufficient?
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Hi, We had exactly the same problem with the on wire compression and encryption. The item that triggered my research was the same blog post from Minio. I also thought that this attack vector is strange and that some simple mitigations can be taken to protect against such attacks, so I asked a security expert in RedHat and the answer was very clear - any form of compression and encryption (at least when the compression algorithm is known) is considered a security breach. For the on-wire compression, our solution is to give a warning before enabling compression and encryption together and let the user decide - this is because the compression on the wire is meant basically to save money and we don't want to block it. When the entire network is encrypted (on the network level, not in msgr2) we allow compression since the encryption is not on the Ceph level. I suggest consulting with product managers/customers. The business case for on-wire compression is different that the one for the at rest compression and I would claim that the compression at rest should be stronger than the on-wire, but this is not dev decision, it is a business decision. Regards, Josh On Wed, Jul 5, 2023 at 6:29 PM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Jul 5, 2023 at 10:38 AM Matt Benjamin <mbenjami@redhat.com> wrote:
The "active" strategy outlined here seems to be pretty strange. The
attacker's data needs to be combined into the unencrypted data being sent via S3, so I guess it has compromised the S3 client environment or dataset already in some way? I'm not sure if that qualifies as a direct attack on compression+encryption in the S3 service.
the active version does sound a bit contrived. but the passive version does leak some information about the data at rest, and server-side encryption is meant to protect against storage admins. any rados client with read access to the data pool can calculate these compression ratios to test whether two same-sized s3 objects might contain different data
In general, it seems like whether to compress+encrypt should be a policy decision, not something arbitrarily chosen by the S3 implementation (as this article claims minio does).
agreed, but the default should be off, and we'd need to understand and clearly document the tradeoffs. the current 'zonegroup feature' mechanism defaults to on for new zones, so https://github.com/ceph/ceph/pull/52300 isn't sufficient as-is
Matt
On Wed, Jul 5, 2023 at 10:24 AM Casey Bodley <cbodley@redhat.com> wrote:
thanks Josh, i wasn't aware of attacks in this space. a naive search came up with a blog post from minio about a 'compression-ratio side channel' at https://blog.min.io/c-e-compression-encryption/. is that the same kind of issue you're concerned about here?
without more time to evaluate this, i'm tempted to disable the feature
for reef
On Tue, Jul 4, 2023 at 3:13 AM Josh Salomon <jsalomon@redhat.com>
wrote:
Please note that compression before encryption is considered a
security breach. I would not implement this without a clear warning and specific user approval.
Regards,
Josh
On Mon, Jul 3, 2023 at 10:54 PM Casey Bodley <cbodley@redhat.com>
wrote:
i opened https://github.com/ceph/ceph/pull/52300 to require a 'compress-encrypted' zonegroup feature for this, and updated the documentation and release note accordingly
On Mon, Jul 3, 2023 at 2:11 PM Casey Bodley <cbodley@redhat.com>
wrote:
hey Shilpa and team,
early in the reef cycle, https://github.com/ceph/ceph/pull/46188
was
contributed to support the combination of server-side compression and encryption on the same object data. only recently did we catch a regression in multisite, where such objects fail to replicate and can cause crashes. this bug, tracked in https://tracker.ceph.com/issues/57905, was just fixed and backported for reef in https://github.com/ceph/ceph/pull/52297. this was a regression in reef, so i was planning to treat it as a blocker
in that backport, i added a warning to the original release note:
RGW: Compression is now supported for objects uploaded with Server-Side Encryption. When both are enabled, compression is applied before encryption. WARNING: In a multisite configuration, objects that are both compressed and encrypted will not replicate correctly to Pacific or Quincy. Upgrade all zones to Reef before enabling compression.
it occurs to me that we might add a new 'compress-encrypted' feature flag to the zonegroup (similar to the 'resharding' flag in reef) to prevent this combination of compression+encryption until all zones upgrade and enable it. do you think that's worth doing, or is a release note sufficient?
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
thanks Josh, i updated https://github.com/ceph/ceph/pull/52300 to leave this 'compress-encrypted' feature disabled by default, and tried to outline the security considerations in a 'warning' in the documentation On Thu, Jul 6, 2023 at 4:03 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi,
We had exactly the same problem with the on wire compression and encryption. The item that triggered my research was the same blog post from Minio. I also thought that this attack vector is strange and that some simple mitigations can be taken to protect against such attacks, so I asked a security expert in RedHat and the answer was very clear - any form of compression and encryption (at least when the compression algorithm is known) is considered a security breach. For the on-wire compression, our solution is to give a warning before enabling compression and encryption together and let the user decide - this is because the compression on the wire is meant basically to save money and we don't want to block it. When the entire network is encrypted (on the network level, not in msgr2) we allow compression since the encryption is not on the Ceph level.
I suggest consulting with product managers/customers. The business case for on-wire compression is different that the one for the at rest compression and I would claim that the compression at rest should be stronger than the on-wire, but this is not dev decision, it is a business decision.
Regards,
Josh
On Wed, Jul 5, 2023 at 6:29 PM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Jul 5, 2023 at 10:38 AM Matt Benjamin <mbenjami@redhat.com> wrote:
The "active" strategy outlined here seems to be pretty strange. The attacker's data needs to be combined into the unencrypted data being sent via S3, so I guess it has compromised the S3 client environment or dataset already in some way? I'm not sure if that qualifies as a direct attack on compression+encryption in the S3 service.
the active version does sound a bit contrived. but the passive version does leak some information about the data at rest, and server-side encryption is meant to protect against storage admins. any rados client with read access to the data pool can calculate these compression ratios to test whether two same-sized s3 objects might contain different data
In general, it seems like whether to compress+encrypt should be a policy decision, not something arbitrarily chosen by the S3 implementation (as this article claims minio does).
agreed, but the default should be off, and we'd need to understand and clearly document the tradeoffs. the current 'zonegroup feature' mechanism defaults to on for new zones, so https://github.com/ceph/ceph/pull/52300 isn't sufficient as-is
Matt
On Wed, Jul 5, 2023 at 10:24 AM Casey Bodley <cbodley@redhat.com> wrote:
thanks Josh, i wasn't aware of attacks in this space. a naive search came up with a blog post from minio about a 'compression-ratio side channel' at https://blog.min.io/c-e-compression-encryption/. is that the same kind of issue you're concerned about here?
without more time to evaluate this, i'm tempted to disable the feature for reef
On Tue, Jul 4, 2023 at 3:13 AM Josh Salomon <jsalomon@redhat.com> wrote:
Please note that compression before encryption is considered a security breach. I would not implement this without a clear warning and specific user approval.
Regards,
Josh
On Mon, Jul 3, 2023 at 10:54 PM Casey Bodley <cbodley@redhat.com> wrote:
i opened https://github.com/ceph/ceph/pull/52300 to require a 'compress-encrypted' zonegroup feature for this, and updated the documentation and release note accordingly
On Mon, Jul 3, 2023 at 2:11 PM Casey Bodley <cbodley@redhat.com> wrote: > > hey Shilpa and team, > > early in the reef cycle, https://github.com/ceph/ceph/pull/46188 was > contributed to support the combination of server-side compression and > encryption on the same object data. only recently did we catch a > regression in multisite, where such objects fail to replicate and can > cause crashes. this bug, tracked in > https://tracker.ceph.com/issues/57905, was just fixed and backported > for reef in https://github.com/ceph/ceph/pull/52297. this was a > regression in reef, so i was planning to treat it as a blocker > > in that backport, i added a warning to the original release note: > > RGW: Compression is now supported for objects uploaded with > Server-Side Encryption. When both are enabled, compression is applied > before encryption. > WARNING: In a multisite configuration, objects that are both > compressed and encrypted will not replicate correctly to Pacific or > Quincy. Upgrade all zones to Reef before enabling compression. > > it occurs to me that we might add a new 'compress-encrypted' feature > flag to the zonegroup (similar to the 'resharding' flag in reef) to > prevent this combination of compression+encryption until all zones > upgrade and enable it. do you think that's worth doing, or is a > release note sufficient? _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
participants (4)
-
Anthony D'Atri
-
Casey Bodley
-
Josh Salomon
-
Matt Benjamin