Public Swift bucket with Openstack Keystone integration - not working in quincy/reef
Hi, Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308. Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga. I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine. With rgw_swift_account_in_url = true and proper endpoints: "https://rgw.test/swift/v1/AUTH_%(project_id)s" ticking public access in horizon properly sets ACL on the bucket according to swift client: swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 however still getting 404 NoSuchBucket error Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you. Best regards, Bartosz Bezak
To back Bartosz with this issue, I've pretty much the same issue with Pacific 16.2.13 and I think the whole root cause for both Bartosz's issue and mine is the same. Each time we call for an object, the radosGW nodes answer with a 404 NoSuchKey error 50% of time (on a 3 node gateway) and it's 100% time reproducible. If the request reach the rgw01 node, it end up with a 200 code and it serve the request correctly as long as: 1°/ - The request call for a whole path (foo/index.html, bar/baz.html, etc). 2°/ - The request call for the root of the bucket (swiftURL/bucket/), the gateway then correctly transform the trailing slash and serve the index.html file found. BUT it fail with the 404 NoSuchKey error for any request that call for: 1°/ - The request call for a path with trailing slash (swiftURL/bucket/foo/) beyond the root path. On rgw02 and rgw03 nodes, it end up with a 200 code and it serve the request correctly as long as: 1°/ - The request call for a whole path (swiftURL/bucket/index.html, swiftURL/bucket/foo/index.html). BUT it fail with the 404 NoSuchKey error for any requests that call for: 1°/ - The request call for a path with trailing slash (swiftURL/bucket/foo/) 2°/ - The request call for the root of the bucket (swiftURL/bucket/), then the gateway DO NOT honor the web-index directive and return a 404 NoSuchKey error. For complementary information, here are the result of the bucket stat command and our radosGW configuration: Swift stat on bucket: https://paste.opendev.org/show/bMbFDXWT71oDU1T6bhRh/ RadosGW config: https://paste.opendev.org/show/bdUkFZdGKF3fgMkqmSg7/ Our platform is an Openstack YOGA With 3 controllers, all CP services are redundant, working and behind a keepalived VIP/HAproxy layer. Hope someone will have a brilliant idea here. Feel free to poke if you need any further information. Le ven. 26 avr. 2024 à 13:51, Bartosz Bezak <bartosz@stackhpc.com> a écrit :
Hi,
Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308.
Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga.
I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine.
With rgw_swift_account_in_url = true and proper endpoints: " https://rgw.test/swift/v1/AUTH_%(project_id)s"
ticking public access in horizon properly sets ACL on the bucket according to swift client:
swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8
however still getting 404 NoSuchBucket error
Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you.
Best regards, Bartosz Bezak
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Assuming both Ceph and OSP are healthy enough, to me your issue looks pretty close to this one: https://tracker.ceph.com/issues/63684 There is also a parent issue 56029 you might want to look in. If you hit one of those bugs, an upgrade to the version containing a fix might help. Good luck, Vladimir. Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: Gaël THEROND <gael.therond@bitswalk.com> Sent: Wednesday, July 31, 2024 10:32:54 AM To: Bartosz Bezak <bartosz@stackhpc.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Public Swift bucket with Openstack Keystone integration - not working in quincy/reef To back Bartosz with this issue, I've pretty much the same issue with Pacific 16.2.13 and I think the whole root cause for both Bartosz's issue and mine is the same. Each time we call for an object, the radosGW nodes answer with a 404 NoSuchKey error 50% of time (on a 3 node gateway) and it's 100% time reproducible. If the request reach the rgw01 node, it end up with a 200 code and it serve the request correctly as long as: 1°/ - The request call for a whole path (foo/index.html, bar/baz.html, etc). 2°/ - The request call for the root of the bucket (swiftURL/bucket/), the gateway then correctly transform the trailing slash and serve the index.html file found. BUT it fail with the 404 NoSuchKey error for any request that call for: 1°/ - The request call for a path with trailing slash (swiftURL/bucket/foo/) beyond the root path. On rgw02 and rgw03 nodes, it end up with a 200 code and it serve the request correctly as long as: 1°/ - The request call for a whole path (swiftURL/bucket/index.html, swiftURL/bucket/foo/index.html). BUT it fail with the 404 NoSuchKey error for any requests that call for: 1°/ - The request call for a path with trailing slash (swiftURL/bucket/foo/) 2°/ - The request call for the root of the bucket (swiftURL/bucket/), then the gateway DO NOT honor the web-index directive and return a 404 NoSuchKey error. For complementary information, here are the result of the bucket stat command and our radosGW configuration: Swift stat on bucket: https://paste.opendev.org/show/bMbFDXWT71oDU1T6bhRh/ RadosGW config: https://paste.opendev.org/show/bdUkFZdGKF3fgMkqmSg7/ Our platform is an Openstack YOGA With 3 controllers, all CP services are redundant, working and behind a keepalived VIP/HAproxy layer. Hope someone will have a brilliant idea here. Feel free to poke if you need any further information. Le ven. 26 avr. 2024 à 13:51, Bartosz Bezak <bartosz@stackhpc.com> a écrit :
Hi,
Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308.
Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga.
I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine.
With rgw_swift_account_in_url = true and proper endpoints: " https://rgw.test/swift/v1/AUTH_%(project_id)s"
ticking public access in horizon properly sets ACL on the bucket according to swift client:
swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8
however still getting 404 NoSuchBucket error
Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you.
Best regards, Bartosz Bezak
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I looked at the rados logs, not a single segfault was found, the gateway just return a 404 NoSuchKey as if it didn't recognized the swift #anonymous access_key or we're hitting the infamous bug of the distribute cache not being invoked again as for Nautilus release. Which indeed looks like what I described. I'll fill a bug on the tracker so we could track those issues. Le mer. 31 juil. 2024 à 17:19, Vladimir Sigunov <vladimir.sigunov@gmail.com> a écrit :
Assuming both Ceph and OSP are healthy enough, to me your issue looks pretty close to this one: https://tracker.ceph.com/issues/63684
There is also a parent issue 56029 you might want to look in.
If you hit one of those bugs, an upgrade to the version containing a fix might help.
Good luck, Vladimir.
Get Outlook for Android <https://aka.ms/AAb9ysg> ------------------------------ *From:* Gaël THEROND <gael.therond@bitswalk.com> *Sent:* Wednesday, July 31, 2024 10:32:54 AM *To:* Bartosz Bezak <bartosz@stackhpc.com> *Cc:* ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* [ceph-users] Re: Public Swift bucket with Openstack Keystone integration - not working in quincy/reef
To back Bartosz with this issue, I've pretty much the same issue with Pacific 16.2.13 and I think the whole root cause for both Bartosz's issue and mine is the same.
Each time we call for an object, the radosGW nodes answer with a 404 NoSuchKey error 50% of time (on a 3 node gateway) and it's 100% time reproducible.
If the request reach the rgw01 node, it end up with a 200 code and it serve the request correctly as long as: 1°/ - The request call for a whole path (foo/index.html, bar/baz.html, etc). 2°/ - The request call for the root of the bucket (swiftURL/bucket/), the gateway then correctly transform the trailing slash and serve the index.html file found.
BUT it fail with the 404 NoSuchKey error for any request that call for: 1°/ - The request call for a path with trailing slash (swiftURL/bucket/foo/) beyond the root path.
On rgw02 and rgw03 nodes, it end up with a 200 code and it serve the request correctly as long as: 1°/ - The request call for a whole path (swiftURL/bucket/index.html, swiftURL/bucket/foo/index.html).
BUT it fail with the 404 NoSuchKey error for any requests that call for: 1°/ - The request call for a path with trailing slash (swiftURL/bucket/foo/) 2°/ - The request call for the root of the bucket (swiftURL/bucket/), then the gateway DO NOT honor the web-index directive and return a 404 NoSuchKey error.
For complementary information, here are the result of the bucket stat command and our radosGW configuration:
Swift stat on bucket: https://paste.opendev.org/show/bMbFDXWT71oDU1T6bhRh/
RadosGW config: https://paste.opendev.org/show/bdUkFZdGKF3fgMkqmSg7/
Our platform is an Openstack YOGA With 3 controllers, all CP services are redundant, working and behind a keepalived VIP/HAproxy layer.
Hope someone will have a brilliant idea here. Feel free to poke if you need any further information.
Le ven. 26 avr. 2024 à 13:51, Bartosz Bezak <bartosz@stackhpc.com> a écrit :
Hi,
Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308.
Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga.
I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine.
With rgw_swift_account_in_url = true and proper endpoints: " https://rgw.test/swift/v1/AUTH_%(project_id)s"
ticking public access in horizon properly sets ACL on the bucket according to swift client:
swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8
however still getting 404 NoSuchBucket error
Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you.
Best regards, Bartosz Bezak
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Bartosz Bezak
-
Gaël THEROND
-
Vladimir Sigunov