Upgraded to Quincy 17.2.7: some S3 buckets inaccessible
Hi everybody, we upgraded our containerized Red Hat Pacific cluster to the latest Quincy release (Community Edition). The upgrade itself went fine, the cluster is HEALTH_OK, all daemons run the upgraded version: ---- %< ---- $ ceph -s cluster: id: 68675a58-cf09-4ebd-949c-b9fcc4f2264e health: HEALTH_OK services: mon: 5 daemons, quorum node02,node03,node04,node05,node01 (age 25h) mgr: node03.ztlair(active, since 25h), standbys: node01.koymku, node04.uvxgvp, node02.znqnhg, node05.iifmpc osd: 408 osds: 408 up (since 22h), 408 in (since 7d) rgw: 19 daemons active (19 hosts, 1 zones) data: pools: 11 pools, 8481 pgs objects: 236.99M objects, 544 TiB usage: 1.6 PiB used, 838 TiB / 2.4 PiB avail pgs: 8385 active+clean 79 active+clean+scrubbing+deep 17 active+clean+scrubbing io: client: 42 MiB/s rd, 439 MiB/s wr, 2.15k op/s rd, 1.64k op/s wr --- $ ceph versions | jq .overall { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 437 } ---- >% ---- After all the daemons were upgraded we started noticing some RGW buckets which are inaccessible. s3cmd failed with NoSuchKey: ---- %< ---- $ s3cmd la -l ERROR: S3 error: 404 (NoSuchKey) ---- >% ---- The buckets still exists according to "radosgw-admin bucket list". Out of the ~600 buckets, 13 buckets are unaccessible at the moment: ---- %< ---- $ radosgw-admin bucket radoslist --tenant xy --uid xy --bucket xy 2024-04-03T12:13:40.607+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed 2024-04-03T12:13:40.609+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed ---- >% ---- The affected buckets are comparatively large, around 4 - 7 TB, but not all buckets of that size are affected. Using "rados -p rgw.buckets.data ls" it seems like all the objects are still there, although "rados -p rgw.buckets.data get objectname -" only prints unusable (?) binary data, even for objects of intact buckets. Overall we're facing around 60 TB of customer data which are just gone at the moment. Is there a way to recover from this situation or further narrowing down the root cause of the problem? Kind regards, Lorenz
On Wed, Apr 3, 2024 at 11:58 AM Lorenz Bausch <info@lorenzbausch.de> wrote:
Hi everybody,
we upgraded our containerized Red Hat Pacific cluster to the latest Quincy release (Community Edition).
i'm afraid this is not an upgrade path that we try to test or support. Red Hat makes its own decisions about what to backport into its releases. my understanding is that Red Hat's pacific-based 5.3 release includes all of the rgw multisite resharding changes which were not introduced upstream until the Reef release. this includes changes to data formats that an upstream Quincy release would not understand. in this case, you might have more luck upgrading to Reef?
The upgrade itself went fine, the cluster is HEALTH_OK, all daemons run the upgraded version:
---- %< ---- $ ceph -s cluster: id: 68675a58-cf09-4ebd-949c-b9fcc4f2264e health: HEALTH_OK
services: mon: 5 daemons, quorum node02,node03,node04,node05,node01 (age 25h) mgr: node03.ztlair(active, since 25h), standbys: node01.koymku, node04.uvxgvp, node02.znqnhg, node05.iifmpc osd: 408 osds: 408 up (since 22h), 408 in (since 7d) rgw: 19 daemons active (19 hosts, 1 zones)
data: pools: 11 pools, 8481 pgs objects: 236.99M objects, 544 TiB usage: 1.6 PiB used, 838 TiB / 2.4 PiB avail pgs: 8385 active+clean 79 active+clean+scrubbing+deep 17 active+clean+scrubbing
io: client: 42 MiB/s rd, 439 MiB/s wr, 2.15k op/s rd, 1.64k op/s wr
---
$ ceph versions | jq .overall { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 437 } ---- >% ----
After all the daemons were upgraded we started noticing some RGW buckets which are inaccessible. s3cmd failed with NoSuchKey:
---- %< ---- $ s3cmd la -l ERROR: S3 error: 404 (NoSuchKey) ---- >% ----
The buckets still exists according to "radosgw-admin bucket list". Out of the ~600 buckets, 13 buckets are unaccessible at the moment:
---- %< ---- $ radosgw-admin bucket radoslist --tenant xy --uid xy --bucket xy 2024-04-03T12:13:40.607+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed 2024-04-03T12:13:40.609+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed ---- >% ----
The affected buckets are comparatively large, around 4 - 7 TB, but not all buckets of that size are affected.
Using "rados -p rgw.buckets.data ls" it seems like all the objects are still there, although "rados -p rgw.buckets.data get objectname -" only prints unusable (?) binary data, even for objects of intact buckets.
Overall we're facing around 60 TB of customer data which are just gone at the moment. Is there a way to recover from this situation or further narrowing down the root cause of the problem?
Kind regards, Lorenz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
to expand on this diagnosis: with multisite resharding, we changed how buckets name/locate their bucket index shard objects. any buckets that were resharded under this Red Hat pacific release would be using the new object names. after upgrading to the Quincy release, rgw would look at the wrong object names when trying to list those buckets. 404 NoSuchKey is the response i would expect in that case On Wed, Apr 3, 2024 at 12:20 PM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Apr 3, 2024 at 11:58 AM Lorenz Bausch <info@lorenzbausch.de> wrote:
Hi everybody,
we upgraded our containerized Red Hat Pacific cluster to the latest Quincy release (Community Edition).
i'm afraid this is not an upgrade path that we try to test or support. Red Hat makes its own decisions about what to backport into its releases. my understanding is that Red Hat's pacific-based 5.3 release includes all of the rgw multisite resharding changes which were not introduced upstream until the Reef release. this includes changes to data formats that an upstream Quincy release would not understand. in this case, you might have more luck upgrading to Reef?
The upgrade itself went fine, the cluster is HEALTH_OK, all daemons run the upgraded version:
---- %< ---- $ ceph -s cluster: id: 68675a58-cf09-4ebd-949c-b9fcc4f2264e health: HEALTH_OK
services: mon: 5 daemons, quorum node02,node03,node04,node05,node01 (age 25h) mgr: node03.ztlair(active, since 25h), standbys: node01.koymku, node04.uvxgvp, node02.znqnhg, node05.iifmpc osd: 408 osds: 408 up (since 22h), 408 in (since 7d) rgw: 19 daemons active (19 hosts, 1 zones)
data: pools: 11 pools, 8481 pgs objects: 236.99M objects, 544 TiB usage: 1.6 PiB used, 838 TiB / 2.4 PiB avail pgs: 8385 active+clean 79 active+clean+scrubbing+deep 17 active+clean+scrubbing
io: client: 42 MiB/s rd, 439 MiB/s wr, 2.15k op/s rd, 1.64k op/s wr
---
$ ceph versions | jq .overall { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 437 } ---- >% ----
After all the daemons were upgraded we started noticing some RGW buckets which are inaccessible. s3cmd failed with NoSuchKey:
---- %< ---- $ s3cmd la -l ERROR: S3 error: 404 (NoSuchKey) ---- >% ----
The buckets still exists according to "radosgw-admin bucket list". Out of the ~600 buckets, 13 buckets are unaccessible at the moment:
---- %< ---- $ radosgw-admin bucket radoslist --tenant xy --uid xy --bucket xy 2024-04-03T12:13:40.607+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed 2024-04-03T12:13:40.609+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed ---- >% ----
The affected buckets are comparatively large, around 4 - 7 TB, but not all buckets of that size are affected.
Using "rados -p rgw.buckets.data ls" it seems like all the objects are still there, although "rados -p rgw.buckets.data get objectname -" only prints unusable (?) binary data, even for objects of intact buckets.
Overall we're facing around 60 TB of customer data which are just gone at the moment. Is there a way to recover from this situation or further narrowing down the root cause of the problem?
Kind regards, Lorenz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Casey, thank you so much for analysis! We tested the upgraded intensively, but the buckets in our test environment were probably too small to get dynamically resharded.
after upgrading to the Quincy release, rgw would look at the wrong object names when trying to list those buckets. As we're currently running Quincy, do you think objects/bucket indexes might already be altered in a way which makes them also unusable for Reef?
Kind regards, Lorenz
On Wed, Apr 3, 2024 at 3:09 PM Lorenz Bausch <info@lorenzbausch.de> wrote:
Hi Casey,
thank you so much for analysis! We tested the upgraded intensively, but the buckets in our test environment were probably too small to get dynamically resharded.
after upgrading to the Quincy release, rgw would look at the wrong object names when trying to list those buckets. As we're currently running Quincy, do you think objects/bucket indexes might already be altered in a way which makes them also unusable for Reef?
for multisite resharding support, the bucket instance metadata now stores an additional 'layout' structure which contains all of the information necessary to locate its bucket index objects. on reshard, the Red Hat pacific release would have stored that information with the bucket. the upstream Reef release should be able to interpret that layout data correctly however, if the Quincy release overwrites that bucket instance metadata (via an operation like PutBucketAcl, PutBucketPolicy, etc), the corresponding layout information would be erased such that an upgrade to Reef would not be able to find the real bucket index shard objects
Kind regards, Lorenz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you again Casey for putting us on the right track regarding the changes in multisite resharding support. When going through the various changelogs we didn't pay too much attention to those changes as this cluster doesn't use any multisite features. We have now upgraded to Reef and all buckets except one are usable again. The rados objects seem to still exist, so we might have luck with something like rgw-restore-bucket-index. On 03/04/2024 21:43, Casey Bodley wrote:
On Wed, Apr 3, 2024 at 3:09 PM Lorenz Bausch <info@lorenzbausch.de> wrote:
Hi Casey,
thank you so much for analysis! We tested the upgraded intensively, but the buckets in our test environment were probably too small to get dynamically resharded.
after upgrading to the Quincy release, rgw would look at the wrong object names when trying to list those buckets. As we're currently running Quincy, do you think objects/bucket indexes might already be altered in a way which makes them also unusable for Reef?
for multisite resharding support, the bucket instance metadata now stores an additional 'layout' structure which contains all of the information necessary to locate its bucket index objects. on reshard, the Red Hat pacific release would have stored that information with the bucket. the upstream Reef release should be able to interpret that layout data correctly
however, if the Quincy release overwrites that bucket instance metadata (via an operation like PutBucketAcl, PutBucketPolicy, etc), the corresponding layout information would be erased such that an upgrade to Reef would not be able to find the real bucket index shard objects
Kind regards, Lorenz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey there, thanks. We had/have the same problem and I was debugging it for a while now as well. We will test if the upgrade solves the problem on our side as well. Thanks again for the thread. Kind regards Thomas On 4/4/24 7:52 PM, Lorenz Bausch wrote:
Thank you again Casey for putting us on the right track regarding the changes in multisite resharding support. When going through the various changelogs we didn't pay too much attention to those changes as this cluster doesn't use any multisite features.
We have now upgraded to Reef and all buckets except one are usable again. The rados objects seem to still exist, so we might have luck with something like rgw-restore-bucket-index.
On 03/04/2024 21:43, Casey Bodley wrote:
On Wed, Apr 3, 2024 at 3:09 PM Lorenz Bausch <info@lorenzbausch.de> wrote:
Hi Casey,
thank you so much for analysis! We tested the upgraded intensively, but the buckets in our test environment were probably too small to get dynamically resharded.
after upgrading to the Quincy release, rgw would look at the wrong object names when trying to list those buckets. As we're currently running Quincy, do you think objects/bucket indexes might already be altered in a way which makes them also unusable for Reef?
for multisite resharding support, the bucket instance metadata now stores an additional 'layout' structure which contains all of the information necessary to locate its bucket index objects. on reshard, the Red Hat pacific release would have stored that information with the bucket. the upstream Reef release should be able to interpret that layout data correctly
however, if the Quincy release overwrites that bucket instance metadata (via an operation like PutBucketAcl, PutBucketPolicy, etc), the corresponding layout information would be erased such that an upgrade to Reef would not be able to find the real bucket index shard objects
Kind regards, Lorenz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Thomas Schneider IT.SERVICES Wissenschaftliche Informationsversorgung Ruhr-Universität Bochum | 44780 Bochum Telefon: +49 234 32 23939 http://www.it-services.rub.de/
participants (3)
-
Casey Bodley
-
Lorenz Bausch
-
Thomas Schneider