Important: RGW multisite bug may silently corrupt encrypted objects on replication
Our downstream QE team recently observed an md5 mismatch of replicated objects when testing rgw's server-side encryption in multisite. This corruption is specific to s3 multipart uploads, and only affects the replicated copy - the original object remains intact. The bug likely affects Ceph releases all the way back to Luminous where server-side encryption was first introduced. To expand on the cause of this corruption: Encryption of multipart uploads requires special handling around the part boundaries, because each part is uploaded and encrypted separately. In multisite, objects are replicated in their encrypted form, and multipart uploads are replicated as a single part. As a result, the replicated copy loses its knowledge about the original part boundaries required to decrypt the data correctly. We don't have a fix yet, but we're tracking it in https://tracker.ceph.com/issues/46062. The fix will only modify the replication logic, so won't repair any objects that have already replicated incorrectly. We'll need to develop a radosgw-admin command to search for affected objects and reschedule their replication. In the meantime, I can only advise multisite users to avoid using encryption for multipart uploads. If you'd like to scan your cluster for existing encrypted multipart uploads, you can identify them with a s3 HeadObject request. The response would include a x-amz-server-side-encryption header, and the ETag header value (with "s removed) would be longer than 32 characters (multipart ETags are in the special form "<md5sum>-<num parts>"). Take care not to delete the corrupted replicas, because an active-active multisite configuration would go on to delete the original copy.
Hello Casey, Thanks for the information! Can you please confirm that this is only an issue when using “rgw_crypt_default_encryption_key” config opt that says “testing only” in the documentation [1] to enable encryption and not when using Barbican or Vault as KMS or using SSE-C with the S3 API? [1] https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-for...
On 26 May 2023, at 22:45, Casey Bodley <cbodley@redhat.com> wrote:
Our downstream QE team recently observed an md5 mismatch of replicated objects when testing rgw's server-side encryption in multisite. This corruption is specific to s3 multipart uploads, and only affects the replicated copy - the original object remains intact. The bug likely affects Ceph releases all the way back to Luminous where server-side encryption was first introduced.
To expand on the cause of this corruption: Encryption of multipart uploads requires special handling around the part boundaries, because each part is uploaded and encrypted separately. In multisite, objects are replicated in their encrypted form, and multipart uploads are replicated as a single part. As a result, the replicated copy loses its knowledge about the original part boundaries required to decrypt the data correctly.
We don't have a fix yet, but we're tracking it in https://tracker.ceph.com/issues/46062. The fix will only modify the replication logic, so won't repair any objects that have already replicated incorrectly. We'll need to develop a radosgw-admin command to search for affected objects and reschedule their replication.
In the meantime, I can only advise multisite users to avoid using encryption for multipart uploads. If you'd like to scan your cluster for existing encrypted multipart uploads, you can identify them with a s3 HeadObject request. The response would include a x-amz-server-side-encryption header, and the ETag header value (with "s removed) would be longer than 32 characters (multipart ETags are in the special form "<md5sum>-<num parts>"). Take care not to delete the corrupted replicas, because an active-active multisite configuration would go on to delete the original copy. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, May 30, 2023 at 8:22 AM Tobias Urdin <tobias.urdin@binero.com> wrote:
Hello Casey,
Thanks for the information!
Can you please confirm that this is only an issue when using “rgw_crypt_default_encryption_key” config opt that says “testing only” in the documentation [1] to enable encryption and not when using Barbican or Vault as KMS or using SSE-C with the S3 API?
unfortunately, all flavors of server-side encryption (SSE-C, SSE-KMS, SSE-S3, and rgw_crypt_default_encryption_key) are affected by this bug, as they share the same encryption logic. the main difference is where they get the key
[1] https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-for...
On 26 May 2023, at 22:45, Casey Bodley <cbodley@redhat.com> wrote:
Our downstream QE team recently observed an md5 mismatch of replicated objects when testing rgw's server-side encryption in multisite. This corruption is specific to s3 multipart uploads, and only affects the replicated copy - the original object remains intact. The bug likely affects Ceph releases all the way back to Luminous where server-side encryption was first introduced.
To expand on the cause of this corruption: Encryption of multipart uploads requires special handling around the part boundaries, because each part is uploaded and encrypted separately. In multisite, objects are replicated in their encrypted form, and multipart uploads are replicated as a single part. As a result, the replicated copy loses its knowledge about the original part boundaries required to decrypt the data correctly.
We don't have a fix yet, but we're tracking it in https://tracker.ceph.com/issues/46062. The fix will only modify the replication logic, so won't repair any objects that have already replicated incorrectly. We'll need to develop a radosgw-admin command to search for affected objects and reschedule their replication.
In the meantime, I can only advise multisite users to avoid using encryption for multipart uploads. If you'd like to scan your cluster for existing encrypted multipart uploads, you can identify them with a s3 HeadObject request. The response would include a x-amz-server-side-encryption header, and the ETag header value (with "s removed) would be longer than 32 characters (multipart ETags are in the special form "<md5sum>-<num parts>"). Take care not to delete the corrupted replicas, because an active-active multisite configuration would go on to delete the original copy. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Casey, Understood, thanks! That means that the original copy in the site that it was uploaded to is still safe as long as that copy is not removed, and no underlying changes below RadosGW in the Ceph storage could corrupt the original copy? Best regards Tobias On 30 May 2023, at 14:48, Casey Bodley <cbodley@redhat.com> wrote: On Tue, May 30, 2023 at 8:22 AM Tobias Urdin <tobias.urdin@binero.com<mailto:tobias.urdin@binero.com>> wrote: Hello Casey, Thanks for the information! Can you please confirm that this is only an issue when using “rgw_crypt_default_encryption_key” config opt that says “testing only” in the documentation [1] to enable encryption and not when using Barbican or Vault as KMS or using SSE-C with the S3 API? unfortunately, all flavors of server-side encryption (SSE-C, SSE-KMS, SSE-S3, and rgw_crypt_default_encryption_key) are affected by this bug, as they share the same encryption logic. the main difference is where they get the key [1] https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-for... On 26 May 2023, at 22:45, Casey Bodley <cbodley@redhat.com> wrote: Our downstream QE team recently observed an md5 mismatch of replicated objects when testing rgw's server-side encryption in multisite. This corruption is specific to s3 multipart uploads, and only affects the replicated copy - the original object remains intact. The bug likely affects Ceph releases all the way back to Luminous where server-side encryption was first introduced. To expand on the cause of this corruption: Encryption of multipart uploads requires special handling around the part boundaries, because each part is uploaded and encrypted separately. In multisite, objects are replicated in their encrypted form, and multipart uploads are replicated as a single part. As a result, the replicated copy loses its knowledge about the original part boundaries required to decrypt the data correctly. We don't have a fix yet, but we're tracking it in https://tracker.ceph.com/issues/46062. The fix will only modify the replication logic, so won't repair any objects that have already replicated incorrectly. We'll need to develop a radosgw-admin command to search for affected objects and reschedule their replication. In the meantime, I can only advise multisite users to avoid using encryption for multipart uploads. If you'd like to scan your cluster for existing encrypted multipart uploads, you can identify them with a s3 HeadObject request. The response would include a x-amz-server-side-encryption header, and the ETag header value (with "s removed) would be longer than 32 characters (multipart ETags are in the special form "<md5sum>-<num parts>"). Take care not to delete the corrupted replicas, because an active-active multisite configuration would go on to delete the original copy. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
On Wed, May 31, 2023 at 7:24 AM Tobias Urdin <tobias.urdin@binero.com> wrote:
Hello Casey,
Understood, thanks!
That means that the original copy in the site that it was uploaded to is still safe as long as that copy is not removed, and no underlying changes below RadosGW in the Ceph storage could corrupt the original copy?
right, the original multipart upload remains intact and can be decrypted successfully as i noted above, take care not to delete or modify any replicas that were corrupted. replication is bidirectional by default, so those changes would sync back and delete/overwrite the original copy
Best regards Tobias
On 30 May 2023, at 14:48, Casey Bodley <cbodley@redhat.com> wrote:
On Tue, May 30, 2023 at 8:22 AM Tobias Urdin <tobias.urdin@binero.com<mailto:tobias.urdin@binero.com>> wrote:
Hello Casey,
Thanks for the information!
Can you please confirm that this is only an issue when using “rgw_crypt_default_encryption_key” config opt that says “testing only” in the documentation [1] to enable encryption and not when using Barbican or Vault as KMS or using SSE-C with the S3 API?
unfortunately, all flavors of server-side encryption (SSE-C, SSE-KMS, SSE-S3, and rgw_crypt_default_encryption_key) are affected by this bug, as they share the same encryption logic. the main difference is where they get the key
[1] https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-for...
On 26 May 2023, at 22:45, Casey Bodley <cbodley@redhat.com> wrote:
Our downstream QE team recently observed an md5 mismatch of replicated objects when testing rgw's server-side encryption in multisite. This corruption is specific to s3 multipart uploads, and only affects the replicated copy - the original object remains intact. The bug likely affects Ceph releases all the way back to Luminous where server-side encryption was first introduced.
To expand on the cause of this corruption: Encryption of multipart uploads requires special handling around the part boundaries, because each part is uploaded and encrypted separately. In multisite, objects are replicated in their encrypted form, and multipart uploads are replicated as a single part. As a result, the replicated copy loses its knowledge about the original part boundaries required to decrypt the data correctly.
We don't have a fix yet, but we're tracking it in https://tracker.ceph.com/issues/46062. The fix will only modify the replication logic, so won't repair any objects that have already replicated incorrectly. We'll need to develop a radosgw-admin command to search for affected objects and reschedule their replication.
In the meantime, I can only advise multisite users to avoid using encryption for multipart uploads. If you'd like to scan your cluster for existing encrypted multipart uploads, you can identify them with a s3 HeadObject request. The response would include a x-amz-server-side-encryption header, and the ETag header value (with "s removed) would be longer than 32 characters (multipart ETags are in the special form "<md5sum>-<num parts>"). Take care not to delete the corrupted replicas, because an active-active multisite configuration would go on to delete the original copy. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Casey Bodley
-
Tobias Urdin