26 May
2023
26 May
'23
10:45 p.m.
Our downstream QE team recently observed an md5 mismatch of replicated
objects when testing rgw's server-side encryption in multisite. This
corruption is specific to s3 multipart uploads, and only affects the
replicated copy - the original object remains intact. The bug likely
affects Ceph releases all the way back to Luminous where server-side
encryption was first introduced.
To expand on the cause of this corruption: Encryption of multipart
uploads requires special handling around the part boundaries, because
each part is uploaded and encrypted separately. In multisite, objects
are replicated in their encrypted form, and multipart uploads are
replicated as a single part. As a result, the replicated copy loses
its knowledge about the original part boundaries required to decrypt
the data correctly.
We don't have a fix yet, but we're tracking it in
https://tracker.ceph.com/issues/46062. The fix will only modify the
replication logic, so won't repair any objects that have already
replicated incorrectly. We'll need to develop a radosgw-admin command
to search for affected objects and reschedule their replication.
In the meantime, I can only advise multisite users to avoid using
encryption for multipart uploads. If you'd like to scan your cluster
for existing encrypted multipart uploads, you can identify them with a
s3 HeadObject request. The response would include a
x-amz-server-side-encryption header, and the ETag header value (with
"s removed) would be longer than 32 characters (multipart ETags are in
the special form "<md5sum>-<num parts>"). Take care not to delete
the
corrupted replicas, because an active-active multisite configuration
would go on to delete the original copy.
30 May
30 May
2:21 p.m.
Hello Casey,
Thanks for the information!
Can you please confirm that this is only an issue when using
“rgw_crypt_default_encryption_key”
config opt that says “testing only” in the documentation [1] to enable encryption and not
when using
Barbican or Vault as KMS or using SSE-C with the S3 API?
[1]
https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo…
On 26 May 2023, at 22:45, Casey Bodley
<cbodley(a)redhat.com> wrote:
Our downstream QE team recently observed an md5 mismatch of replicated
objects when testing rgw's server-side encryption in multisite. This
corruption is specific to s3 multipart uploads, and only affects the
replicated copy - the original object remains intact. The bug likely
affects Ceph releases all the way back to Luminous where server-side
encryption was first introduced.
To expand on the cause of this corruption: Encryption of multipart
uploads requires special handling around the part boundaries, because
each part is uploaded and encrypted separately. In multisite, objects
are replicated in their encrypted form, and multipart uploads are
replicated as a single part. As a result, the replicated copy loses
its knowledge about the original part boundaries required to decrypt
the data correctly.
We don't have a fix yet, but we're tracking it in
https://tracker.ceph.com/issues/46062. The fix will only modify the
replication logic, so won't repair any objects that have already
replicated incorrectly. We'll need to develop a radosgw-admin command
to search for affected objects and reschedule their replication.
In the meantime, I can only advise multisite users to avoid using
encryption for multipart uploads. If you'd like to scan your cluster
for existing encrypted multipart uploads, you can identify them with a
s3 HeadObject request. The response would include a
x-amz-server-side-encryption header, and the ETag header value (with
"s removed) would be longer than 32 characters (multipart ETags are in
the special form "<md5sum>-<num parts>"). Take care not to delete
the
corrupted replicas, because an active-active multisite configuration
would go on to delete the original copy.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:48 p.m.
On Tue, May 30, 2023 at 8:22 AM Tobias Urdin <tobias.urdin(a)binero.com> wrote:
Hello Casey,
Thanks for the information!
Can you please confirm that this is only an issue when using
“rgw_crypt_default_encryption_key”
config opt that says “testing only” in the documentation [1] to enable encryption and not
when using
Barbican or Vault as KMS or using SSE-C with the S3 API?
unfortunately, all flavors of server-side encryption (SSE-C, SSE-KMS,
SSE-S3, and rgw_crypt_default_encryption_key) are affected by this
bug, as they share the same encryption logic. the main difference is
where they get the key
[1]
https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo…
On 26 May 2023, at 22:45, Casey Bodley
<cbodley(a)redhat.com> wrote:
Our downstream QE team recently observed an md5 mismatch of replicated
objects when testing rgw's server-side encryption in multisite. This
corruption is specific to s3 multipart uploads, and only affects the
replicated copy - the original object remains intact. The bug likely
affects Ceph releases all the way back to Luminous where server-side
encryption was first introduced.
To expand on the cause of this corruption: Encryption of multipart
uploads requires special handling around the part boundaries, because
each part is uploaded and encrypted separately. In multisite, objects
are replicated in their encrypted form, and multipart uploads are
replicated as a single part. As a result, the replicated copy loses
its knowledge about the original part boundaries required to decrypt
the data correctly.
We don't have a fix yet, but we're tracking it in
https://tracker.ceph.com/issues/46062. The fix will only modify the
replication logic, so won't repair any objects that have already
replicated incorrectly. We'll need to develop a radosgw-admin command
to search for affected objects and reschedule their replication.
In the meantime, I can only advise multisite users to avoid using
encryption for multipart uploads. If you'd like to scan your cluster
for existing encrypted multipart uploads, you can identify them with a
s3 HeadObject request. The response would include a
x-amz-server-side-encryption header, and the ETag header value (with
"s removed) would be longer than 32 characters (multipart ETags are in
the special form "<md5sum>-<num parts>"). Take care not to delete
the
corrupted replicas, because an active-active multisite configuration
would go on to delete the original copy.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
31 May
31 May
1:23 p.m.
Hello Casey,
Understood, thanks!
That means that the original copy in the site that it was uploaded to is still
safe as long as that copy is not removed, and no underlying changes below
RadosGW in the Ceph storage could corrupt the original copy?
Best regards
Tobias
On 30 May 2023, at 14:48, Casey Bodley <cbodley(a)redhat.com> wrote:
On Tue, May 30, 2023 at 8:22 AM Tobias Urdin
<tobias.urdin@binero.com<mailto:tobias.urdin@binero.com>> wrote:
Hello Casey,
Thanks for the information!
Can you please confirm that this is only an issue when using
“rgw_crypt_default_encryption_key”
config opt that says “testing only” in the documentation [1] to enable encryption and not
when using
Barbican or Vault as KMS or using SSE-C with the S3 API?
unfortunately, all flavors of server-side encryption (SSE-C, SSE-KMS,
SSE-S3, and rgw_crypt_default_encryption_key) are affected by this
bug, as they share the same encryption logic. the main difference is
where they get the key
[1]
https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo…
On 26 May 2023, at 22:45, Casey Bodley <cbodley(a)redhat.com> wrote:
Our downstream QE team recently observed an md5 mismatch of replicated
objects when testing rgw's server-side encryption in multisite. This
corruption is specific to s3 multipart uploads, and only affects the
replicated copy - the original object remains intact. The bug likely
affects Ceph releases all the way back to Luminous where server-side
encryption was first introduced.
To expand on the cause of this corruption: Encryption of multipart
uploads requires special handling around the part boundaries, because
each part is uploaded and encrypted separately. In multisite, objects
are replicated in their encrypted form, and multipart uploads are
replicated as a single part. As a result, the replicated copy loses
its knowledge about the original part boundaries required to decrypt
the data correctly.
We don't have a fix yet, but we're tracking it in
https://tracker.ceph.com/issues/46062. The fix will only modify the
replication logic, so won't repair any objects that have already
replicated incorrectly. We'll need to develop a radosgw-admin command
to search for affected objects and reschedule their replication.
In the meantime, I can only advise multisite users to avoid using
encryption for multipart uploads. If you'd like to scan your cluster
for existing encrypted multipart uploads, you can identify them with a
s3 HeadObject request. The response would include a
x-amz-server-side-encryption header, and the ETag header value (with
"s removed) would be longer than 32 characters (multipart ETags are in
the special form "<md5sum>-<num parts>"). Take care not to delete
the
corrupted replicas, because an active-active multisite configuration
would go on to delete the original copy.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
3:06 p.m.
On Wed, May 31, 2023 at 7:24 AM Tobias Urdin <tobias.urdin(a)binero.com> wrote:
Hello Casey,
Understood, thanks!
That means that the original copy in the site that it was uploaded to is still
safe as long as that copy is not removed, and no underlying changes below
RadosGW in the Ceph storage could corrupt the original copy?
right, the original multipart upload remains intact and can be
decrypted successfully
as i noted above, take care not to delete or modify any replicas that
were corrupted. replication is bidirectional by default, so those
changes would sync back and delete/overwrite the original copy
Best regards
Tobias
On 30 May 2023, at 14:48, Casey Bodley <cbodley(a)redhat.com> wrote:
On Tue, May 30, 2023 at 8:22 AM Tobias Urdin
<tobias.urdin@binero.com<mailto:tobias.urdin@binero.com>> wrote:
Hello Casey,
Thanks for the information!
Can you please confirm that this is only an issue when using
“rgw_crypt_default_encryption_key”
config opt that says “testing only” in the documentation [1] to enable encryption and not
when using
Barbican or Vault as KMS or using SSE-C with the S3 API?
unfortunately, all flavors of server-side encryption (SSE-C, SSE-KMS,
SSE-S3, and rgw_crypt_default_encryption_key) are affected by this
bug, as they share the same encryption logic. the main difference is
where they get the key
[1]
https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo…
On 26 May 2023, at 22:45, Casey Bodley <cbodley(a)redhat.com> wrote:
Our downstream QE team recently observed an md5 mismatch of replicated
objects when testing rgw's server-side encryption in multisite. This
corruption is specific to s3 multipart uploads, and only affects the
replicated copy - the original object remains intact. The bug likely
affects Ceph releases all the way back to Luminous where server-side
encryption was first introduced.
To expand on the cause of this corruption: Encryption of multipart
uploads requires special handling around the part boundaries, because
each part is uploaded and encrypted separately. In multisite, objects
are replicated in their encrypted form, and multipart uploads are
replicated as a single part. As a result, the replicated copy loses
its knowledge about the original part boundaries required to decrypt
the data correctly.
We don't have a fix yet, but we're tracking it in
https://tracker.ceph.com/issues/46062. The fix will only modify the
replication logic, so won't repair any objects that have already
replicated incorrectly. We'll need to develop a radosgw-admin command
to search for affected objects and reschedule their replication.
In the meantime, I can only advise multisite users to avoid using
encryption for multipart uploads. If you'd like to scan your cluster
for existing encrypted multipart uploads, you can identify them with a
s3 HeadObject request. The response would include a
x-amz-server-side-encryption header, and the ETag header value (with
"s removed) would be longer than 32 characters (multipart ETags are in
the special form "<md5sum>-<num parts>"). Take care not to delete
the
corrupted replicas, because an active-active multisite configuration
would go on to delete the original copy.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
340
days inactive
345
days old
4 comments
2 participants
participants (2)
-
Casey Bodley -
Tobias Urdin