Ceph RadosGW & OpenStack swift problem

List overview All Threads
Download

newer

older

Latency spike investigations on...

OSDs in pool full : can't restart...

Mika Saari

5 Jan 2021 5 Jan '21

12:33 p.m.

Hi, Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW to work. I have managed to get the RadosGW working. I can manage it through a dashboard and use aws s3 client to create new buckets etc. When trying to use swift I get errors. Not sure how to continue to track the problem here. Any tips are welcome. Thank you very much, -Mika ------- What I have done and what are the results. Some data changed manually ------- What I have done: At OpenStack Side: 1) openstack user create --domain default --password-prompt swift 2) openstack role add --project service --user swift admin 3) openstack endpoint create --region RegionOne object-store public http://ceph1/swift/v1/AUTH_%$project_id$s 4) openstack endpoint create --region RegionOne object-store internal http://ceph1/swift/v1/AUTH_%$project_id$s 5) openstack endpoint create --region RegionOne object-store admin http://ceph1/swift/v1 At Ceph side: 1) ceph config set mgr rgw_keystone_api_version 3 2) ceph config set mgr rgw_keystone_url http://controller:5000 3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://controller:5000/v3/auth/tokens DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80 "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i http://ceph1/swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: http://ceph1/swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 401 Unauthorized b'AccessDenied'

Show replies by date

Wissem MIMOUNA

5 Jan 5 Jan

12:49 p.m.

Hi, Which version of OpenStack do you have ? I guess , since Usurri ( or may be even before ) swift authentification through keystone require the account in url . You have to add this option in "/etc/ceph/ceph.conf" , section rgw "rgw swift account in url = true" or do it via setting directly . Also , I noticed you did this ==> 3) ceph config set mgr rgw_keystone_accepted_admin_roles xxxx || I think , you should use the option "rgw keystone accepted roles xxxx" instead. Regards -----Message d'origine----- De : Mika Saari <mika.saari(a)gmail.com> Envoyé : mardi 5 janvier 2021 10:03 À : ceph-users(a)ceph.io Objet : [ceph-users] Ceph RadosGW & OpenStack swift problem Hi, Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW to work. I have managed to get the RadosGW working. I can manage it through a dashboard and use aws s3 client to create new buckets etc. When trying to use swift I get errors. Not sure how to continue to track the problem here. Any tips are welcome. Thank you very much, -Mika ------- What I have done and what are the results. Some data changed manually ------- What I have done: At OpenStack Side: 1) openstack user create --domain default --password-prompt swift 2) openstack role add --project service --user swift admin 3) openstack endpoint create --region RegionOne object-store public https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-… 4) openstack endpoint create --region RegionOne object-store internal https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-… 5) openstack endpoint create --region RegionOne object-store admin https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1&d=Dw… At Ceph side: 1) ceph config set mgr rgw_keystone_api_version 3 2) ceph config set mgr rgw_keystone_url https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d… 3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000_v3_au… DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80 "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5Fa… -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5Fa… 401 Unauthorized b'AccessDenied' _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Mika Saari

9:30 p.m.

Hi, I am using indeed OpenStack Ussuri release. I changed the "gw swift account in url = true" directly with ceph config set ... command. Also checked that rgw_keystone_accepted_roles is correctly set and not the admin one. Also tested disabling rgw_keystone_verify_ssl. Should radosgw communicate with keystone somehow? I can not see my ceph-cluster requesting anything from keystone through any interface (tcpdump checked this one). I have tested restarting the radosgw with command "ceph orch restart rgw.default.ou" and seems that it brings the container down and up. Not sure though it is enough to bring the settings in use.q Current status is: 1) swift command seems to be able to authenticate with keystone at the very beginning, this is done in the client side. 2) swift command makes a request to radosgw and gets 401 INFO:swiftclient:REQ: curl -i <radosgw url here>/swift/v1/AUTH_<some id here>/test3 -X POST -H "X-Auth-Token: <token here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized Thanks a lot again, -Mika On Tue, Jan 5, 2021 at 11:19 AM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote:

...

Mika Saari

7 Jan 7 Jan

2:05 p.m.

Hi, I have added debug_rgw 20 to configuration. When checking docker logs -f <radosgw container id> I get this error for my radowgw request (swift post test3 --debug) Would there be a way to get more debug information from radosgw to solve this 401 problem ? Thanks a lot, -Mika --- clip clip ---- debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== starting new request req=0x7f1b5b32a6b0 ===== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 op->ERRORHANDLER: err_no=-1 new_err_no=-1 debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== req done req=0x7f1b5b32a6b0 op status=0 http_status=401 latency=0s ====== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 beast: 0x7f1b5b32a6b0: 10.0.2.10 - - [2021-01-07T10:32:42.269372+0000] "POST /swift/v1/AUTH_50f0ce372a4a4ed6a41126852358f097/test3 HTTP/1.1" 401 12 - "python-swiftclient-3.9.0" - --- clip clip ---- On Tue, Jan 5, 2021 at 8:00 PM Mika Saari <mika.saari(a)gmail.com> wrote:

...

Wissem MIMOUNA

2:46 p.m.

Hi, The radosgw should have a dedicated user (different from you swift user) for authentifiation with keystone ( openstack) in the project "service" and you should also add the role "_member_" in the rgw_keystone_accepted_roles. Regards -----Message d'origine----- De : Mika Saari <mika.saari(a)gmail.com> Envoyé : jeudi 7 janvier 2021 11:35 À : ceph-users(a)ceph.io Objet : [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, I have added debug_rgw 20 to configuration. When checking docker logs -f <radosgw container id> I get this error for my radowgw request (swift post test3 --debug) Would there be a way to get more debug information from radosgw to solve this 401 problem ? Thanks a lot, -Mika --- clip clip ---- debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== starting new request req=0x7f1b5b32a6b0 ===== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 op->ERRORHANDLER: err_no=-1 new_err_no=-1 debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== req done req=0x7f1b5b32a6b0 op status=0 http_status=401 latency=0s ====== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 beast: 0x7f1b5b32a6b0: 10.0.2.10 - - [2021-01-07T10:32:42.269372+0000] "POST /swift/v1/AUTH_50f0ce372a4a4ed6a41126852358f097/test3 HTTP/1.1" 401 12 - "python-swiftclient-3.9.0" - --- clip clip ---- On Tue, Jan 5, 2021 at 8:00 PM Mika Saari <mika.saari(a)gmail.com> wrote:

...

Hi, I am using indeed OpenStack Ussuri release. I changed the "gw swift account in url = true" directly with ceph config set ... command. Also checked that rgw_keystone_accepted_roles is correctly set and not the admin one. Also tested disabling rgw_keystone_verify_ssl. Should radosgw communicate with keystone somehow? I can not see my ceph-cluster requesting anything from keystone through any interface (tcpdump checked this one). I have tested restarting the radosgw with command "ceph orch restart rgw.default.ou" and seems that it brings the container down and up. Not sure though it is enough to bring the settings in use.q Current status is: 1) swift command seems to be able to authenticate with keystone at the very beginning, this is done in the client side. 2) swift command makes a request to radosgw and gets 401 INFO:swiftclient:REQ: curl -i <radosgw url here>/swift/v1/AUTH_<some id here>/test3 -X POST -H "X-Auth-Token: here><token " -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized Thanks a lot again, -Mika On Tue, Jan 5, 2021 at 11:19 AM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote:

Hi, Which version of OpenStack do you have ? I guess , since Usurri ( or may be even before ) swift authentification through keystone require the account in url . You have to add this option in "/etc/ceph/ceph.conf" , section rgw "rgw swift account in url = true" or do it via setting directly . Also , I noticed you did this ==> 3) ceph config set mgr rgw_keystone_accepted_admin_roles xxxx || I think , you should use the option "rgw keystone accepted roles xxxx" instead. Regards -----Message d'origine----- De : Mika Saari <mika.saari(a)gmail.com> Envoyé : mardi 5 janvier 2021 10:03 À : ceph-users(a)ceph.io Objet : [ceph-users] Ceph RadosGW & OpenStack swift problem Hi, Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW to work. I have managed to get the RadosGW working. I can manage it through a dashboard and use aws s3 client to create new buckets etc. When trying to use swift I get errors. Not sure how to continue to track the problem here. Any tips are welcome. Thank you very much, -Mika ------- What I have done and what are the results. Some data changed manually ------- What I have done: At OpenStack Side: 1) openstack user create --domain default --password-prompt swift 2) openstack role add --project service --user swift admin 3) openstack endpoint create --region RegionOne object-store public https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-… 4) openstack endpoint create --region RegionOne object-store internal https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-… 5) openstack endpoint create --region RegionOne object-store admin https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1&d= DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Ktt b6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7eX4&s=bm67b3lMVeLC 3sNvuyufFCe3AksJgfIgeI8SDorhHMU&e= At Ceph side: 1) ceph config set mgr rgw_keystone_api_version 3 2) ceph config set mgr rgw_keystone_url https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d… 3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000 _v3_auth_tokens&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KK a6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7 eX4&s=-98qpMcc8sdRTdN7AwNPIyGsIK1GaFvi_SC5GtZGUpY&e= DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80 "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= 401 Unauthorized b'AccessDenied' _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Mika Saari

3:15 p.m.

Hi, Adding below what I tested. Do you see from this what I am doing wrong? Thank you very much, -Mika --clip clip-- OPENSTACK SIDE: [root@controller ~]# openstack user create --domain default --password-prompt rgwswift User Password: Repeat User Password: +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 85a86ec5c0264302b0471fd147042e0b | | name | rgwswift | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+ [root@controller ~]# openstack role add --project service --user rgwswift admin CEPH SIDE: [root@ceph1 ~]# ceph config set mgr rgw_keystone_accepted_roles "admin, _member_, Member, member, creator" [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_user rgwswift [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_project service [root@ceph1 ~]# ceph orch restart rgw.default.ou restart rgw.default.ou.ceph1.gxblht from host 'ceph1' CLIENT SIDE: $ . swift-openrc Where swift-openrc is like this: export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=service export OS_USERNAME=rgwswift export OS_PASSWORD=rgwswiftpw export OS_AUTH_URL=http://controller:5000/v3 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 $ swift stat --debug Problem like earlier. First the swift client authenticates to the keystone and that works. Second it tries to contact radosgw, and that gives 401. Checked the rgw_process.cc : process_request and seems that there is no more debug information in the source. I assume the row int ret = client_io->init(g_ceph_context); gives < 0 which causes the process_request to return out with abort_early. On Thu, Jan 7, 2021 at 1:16 PM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote:

...

Hi, I am using indeed OpenStack Ussuri release. I changed the "gw swift account in url = true" directly with ceph config set ... command. Also checked that rgw_keystone_accepted_roles is correctly set and not the admin one. Also tested disabling rgw_keystone_verify_ssl. Should radosgw communicate with keystone somehow? I can not see my ceph-cluster requesting anything from keystone through any interface (tcpdump checked this one). I have tested restarting the radosgw with command "ceph orch restart rgw.default.ou" and seems that it brings the container down and up. Not sure though it is enough to bring the settings in use.q Current status is: 1) swift command seems to be able to authenticate with keystone at the very beginning, this is done in the client side. 2) swift command makes a request to radosgw and gets 401 INFO:swiftclient:REQ: curl -i <radosgw url here>/swift/v1/AUTH_<some id here>/test3 -X POST -H "X-Auth-Token: here><token " -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized Thanks a lot again, -Mika On Tue, Jan 5, 2021 at 11:19 AM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote: > Hi, > > Which version of OpenStack do you have ? I guess , since Usurri ( or > may be even before ) swift authentification through keystone require > the account in url . You have to add this option in > "/etc/ceph/ceph.conf" , section rgw "rgw swift account in url = true"

or do it via setting directly

> . Also , I noticed you did this ==> 3) ceph config set mgr > rgw_keystone_accepted_admin_roles xxxx || I think , you should use > the option "rgw keystone accepted roles xxxx" instead. > > Regards > > -----Message d'origine----- > De : Mika Saari <mika.saari(a)gmail.com> Envoyé : mardi 5 janvier 2021 > 10:03 À : ceph-users(a)ceph.io Objet : [ceph-users] Ceph RadosGW & > OpenStack swift problem > > Hi, > > Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW to

work.

> I have managed to get the RadosGW working. I can manage it through a > dashboard and use aws s3 client to create new buckets etc. When > trying to use swift I get errors. > > Not sure how to continue to track the problem here. Any tips are > welcome. > > Thank you very much, > -Mika > > ------- What I have done and what are the results. Some data changed > manually ------- > What I have done: > At OpenStack Side: > 1) openstack user create --domain default --password-prompt swift > 2) openstack role add --project service --user swift admin > 3) openstack endpoint create --region RegionOne object-store > public

https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-…

> 4) openstack endpoint create --region RegionOne object-store > internal >

https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-…

> 5) openstack endpoint create --region RegionOne object-store > admin > https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1&d= > DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Ktt > b6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7eX4&s=bm67b3lMVeLC > 3sNvuyufFCe3AksJgfIgeI8SDorhHMU&e= > > At Ceph side: > 1) ceph config set mgr rgw_keystone_api_version 3 > 2) ceph config set mgr rgw_keystone_url >

https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d…

3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000 _v3_auth_tokens&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KK a6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7 eX4&s=-98qpMcc8sdRTdN7AwNPIyGsIK1GaFvi_SC5GtZGUpY&e= DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80 "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= 401 Unauthorized b'AccessDenied' _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Wissem MIMOUNA

4:38 p.m.

The user rgwswift only for radosgw config ( do not use it in your file openrc ) use swift user instead . Also , keep the default project to admin ( os_project_name ) . Rgds De : Mika Saari <mika.saari(a)gmail.com> Envoyé : jeudi 7 janvier 2021 12:45 À : Wissem MIMOUNA <wissem.mimouna(a)fiducialcloud.fr> Cc : ceph-users(a)ceph.io Objet : Re: [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, Adding below what I tested. Do you see from this what I am doing wrong? Thank you very much, -Mika --clip clip-- OPENSTACK SIDE: [root@controller ~]# openstack user create --domain default --password-prompt rgwswift User Password: Repeat User Password: +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 85a86ec5c0264302b0471fd147042e0b | | name | rgwswift | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+ [root@controller ~]# openstack role add --project service --user rgwswift admin CEPH SIDE: [root@ceph1 ~]# ceph config set mgr rgw_keystone_accepted_roles "admin, _member_, Member, member, creator" [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_user rgwswift [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_project service [root@ceph1 ~]# ceph orch restart rgw.default.ou restart rgw.default.ou.ceph1.gxblht from host 'ceph1' CLIENT SIDE: $ . swift-openrc Where swift-openrc is like this: export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=service export OS_USERNAME=rgwswift export OS_PASSWORD=rgwswiftpw export OS_AUTH_URL=http://controller:5000/v3<https://urldefense.proofpoint.com/… export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 $ swift stat --debug Problem like earlier. First the swift client authenticates to the keystone and that works. Second it tries to contact radosgw, and that gives 401. Checked the rgw_process.cc : process_request and seems that there is no more debug information in the source. I assume the row int ret = client_io->init(g_ceph_context); gives < 0 which causes the process_request to return out with abort_early. On Thu, Jan 7, 2021 at 1:16 PM Wissem MIMOUNA <wissem.mimouna@fiducialcloud.fr<mailto:wissem.mimouna@fiducialcloud.fr>> wrote: Hi, The radosgw should have a dedicated user (different from you swift user) for authentifiation with keystone ( openstack) in the project "service" and you should also add the role "_member_" in the rgw_keystone_accepted_roles. Regards -----Message d'origine----- De : Mika Saari <mika.saari@gmail.com<mailto:mika.saari@gmail.com>> Envoyé : jeudi 7 janvier 2021 11:35 À : ceph-users@ceph.io<mailto:ceph-users@ceph.io> Objet : [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, I have added debug_rgw 20 to configuration. When checking docker logs -f <radosgw container id> I get this error for my radowgw request (swift post test3 --debug) Would there be a way to get more debug information from radosgw to solve this 401 problem ? Thanks a lot, -Mika --- clip clip ---- debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== starting new request req=0x7f1b5b32a6b0 ===== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 op->ERRORHANDLER: err_no=-1 new_err_no=-1 debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== req done req=0x7f1b5b32a6b0 op status=0 http_status=401 latency=0s ====== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 beast: 0x7f1b5b32a6b0: 10.0.2.10 - - [2021-01-07T10:32:42.269372+0000] "POST /swift/v1/AUTH_50f0ce372a4a4ed6a41126852358f097/test3 HTTP/1.1" 401 12 - "python-swiftclient-3.9.0" - --- clip clip ---- On Tue, Jan 5, 2021 at 8:00 PM Mika Saari <mika.saari@gmail.com<mailto:mika.saari@gmail.com>> wrote:

...

Hi, I am using indeed OpenStack Ussuri release. I changed the "gw swift account in url = true" directly with ceph config set ... command. Also checked that rgw_keystone_accepted_roles is correctly set and not the admin one. Also tested disabling rgw_keystone_verify_ssl. Should radosgw communicate with keystone somehow? I can not see my ceph-cluster requesting anything from keystone through any interface (tcpdump checked this one). I have tested restarting the radosgw with command "ceph orch restart rgw.default.ou" and seems that it brings the container down and up. Not sure though it is enough to bring the settings in use.q Current status is: 1) swift command seems to be able to authenticate with keystone at the very beginning, this is done in the client side. 2) swift command makes a request to radosgw and gets 401 INFO:swiftclient:REQ: curl -i <radosgw url here>/swift/v1/AUTH_<some id here>/test3 -X POST -H "X-Auth-Token: here><token " -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized Thanks a lot again, -Mika On Tue, Jan 5, 2021 at 11:19 AM Wissem MIMOUNA < wissem.mimouna@fiducialcloud.fr<mailto:wissem.mimouna@fiducialcloud.fr>> wrote:

Hi, Which version of OpenStack do you have ? I guess , since Usurri ( or may be even before ) swift authentification through keystone require the account in url . You have to add this option in "/etc/ceph/ceph.conf" , section rgw "rgw swift account in url = true" or do it via setting directly . Also , I noticed you did this ==> 3) ceph config set mgr rgw_keystone_accepted_admin_roles xxxx || I think , you should use the option "rgw keystone accepted roles xxxx" instead. Regards -----Message d'origine----- De : Mika Saari <mika.saari@gmail.com<mailto:mika.saari@gmail.com>> Envoyé : mardi 5 janvier 2021 10:03 À : ceph-users@ceph.io<mailto:ceph-users@ceph.io> Objet : [ceph-users] Ceph RadosGW & OpenStack swift problem Hi, Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW to work. I have managed to get the RadosGW working. I can manage it through a dashboard and use aws s3 client to create new buckets etc. When trying to use swift I get errors. Not sure how to continue to track the problem here. Any tips are welcome. Thank you very much, -Mika ------- What I have done and what are the results. Some data changed manually ------- What I have done: At OpenStack Side: 1) openstack user create --domain default --password-prompt swift 2) openstack role add --project service --user swift admin 3) openstack endpoint create --region RegionOne object-store public https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-… 4) openstack endpoint create --region RegionOne object-store internal https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-… 5) openstack endpoint create --region RegionOne object-store admin https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1&d= DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Ktt b6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7eX4&s=bm67b3lMVeLC 3sNvuyufFCe3AksJgfIgeI8SDorhHMU&e= At Ceph side: 1) ceph config set mgr rgw_keystone_api_version 3 2) ceph config set mgr rgw_keystone_url https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d… 3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000 _v3_auth_tokens&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KK a6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7 eX4&s=-98qpMcc8sdRTdN7AwNPIyGsIK1GaFvi_SC5GtZGUpY&e= DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000<https://urldefense.p… "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80<https://urldefense.proofpoi… "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= 401 Unauthorized b'AccessDenied' _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>

Mika Saari

6:32 p.m.

Hi, Changed switch-openrc and verified the project to be "admin". Unfortunately problems stills. I think I have configured the Ceph now somehow wrong with command ceph config set mgr rgw_keystone_url http://controller:5000 It probably should be something like ceph config set client.radosgw.gateway rgw_keystone_url http:/controllerc:5000 I am not sure about this though. I tested configuring these parameters to /etc/ceph/ceph.conf as well, but not sure if those will affect inside docker containers. It seems that radosgw won't trigger any communication towards keystone. Will continue with this. Thanks, -Mika On Thu, Jan 7, 2021 at 3:08 PM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote:

...

The user rgwswift only for radosgw config ( do not use it in your file openrc ) use swift user instead . Also , keep the default project to admin ( os_project_name ) . Rgds *De :* Mika Saari <mika.saari(a)gmail.com> *Envoyé :* jeudi 7 janvier 2021 12:45 *À :* Wissem MIMOUNA <wissem.mimouna(a)fiducialcloud.fr> *Cc :* ceph-users(a)ceph.io *Objet :* Re: [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, Adding below what I tested. Do you see from this what I am doing wrong? Thank you very much, -Mika --clip clip-- OPENSTACK SIDE: [root@controller ~]# openstack user create --domain default --password-prompt rgwswift User Password: Repeat User Password: +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 85a86ec5c0264302b0471fd147042e0b | | name | rgwswift | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+ [root@controller ~]# openstack role add --project service --user rgwswift admin CEPH SIDE: [root@ceph1 ~]# ceph config set mgr rgw_keystone_accepted_roles "admin, _member_, Member, member, creator" [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_user rgwswift [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_project service [root@ceph1 ~]# ceph orch restart rgw.default.ou restart rgw.default.ou.ceph1.gxblht from host 'ceph1' CLIENT SIDE: $ . swift-openrc Where swift-openrc is like this: export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=service export OS_USERNAME=rgwswift export OS_PASSWORD=rgwswiftpw export OS_AUTH_URL=http://controller:5000/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000_v3&d=DwMFaQ&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=oc3C1TP2mMYCukAjjobWV7SPwto-zVeUvBG-JgRS3SI&s=xYsKH127snVkstwVzGM-ha6td0BdcY5-XQxutKOxNto&e=> export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 $ swift stat --debug Problem like earlier. First the swift client authenticates to the keystone and that works. Second it tries to contact radosgw, and that gives 401. Checked the rgw_process.cc : process_request and seems that there is no more debug information in the source. I assume the row int ret = client_io->init(g_ceph_context); gives < 0 which causes the process_request to return out with abort_early. On Thu, Jan 7, 2021 at 1:16 PM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote: Hi, The radosgw should have a dedicated user (different from you swift user) for authentifiation with keystone ( openstack) in the project "service" and you should also add the role "_member_" in the rgw_keystone_accepted_roles. Regards -----Message d'origine----- De : Mika Saari <mika.saari(a)gmail.com> Envoyé : jeudi 7 janvier 2021 11:35 À : ceph-users(a)ceph.io Objet : [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, I have added debug_rgw 20 to configuration. When checking docker logs -f <radosgw container id> I get this error for my radowgw request (swift post test3 --debug) Would there be a way to get more debug information from radosgw to solve this 401 problem ? Thanks a lot, -Mika --- clip clip ---- debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== starting new request req=0x7f1b5b32a6b0 ===== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 op->ERRORHANDLER: err_no=-1 new_err_no=-1 debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== req done req=0x7f1b5b32a6b0 op status=0 http_status=401 latency=0s ====== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 beast: 0x7f1b5b32a6b0: 10.0.2.10 - - [2021-01-07T10:32:42.269372+0000] "POST /swift/v1/AUTH_50f0ce372a4a4ed6a41126852358f097/test3 HTTP/1.1" 401 12 - "python-swiftclient-3.9.0" - --- clip clip ---- On Tue, Jan 5, 2021 at 8:00 PM Mika Saari <mika.saari(a)gmail.com> wrote:

Hi, I am using indeed OpenStack Ussuri release. I changed the "gw swift account in url = true" directly with ceph config set ... command. Also checked that rgw_keystone_accepted_roles is correctly set and not the admin one. Also tested disabling rgw_keystone_verify_ssl. Should radosgw communicate with keystone somehow? I can not see my ceph-cluster requesting anything from keystone through any interface (tcpdump checked this one). I have tested restarting the radosgw with command "ceph orch restart rgw.default.ou" and seems that it brings the container down and up. Not sure though it is enough to bring the settings in use.q Current status is: 1) swift command seems to be able to authenticate with keystone at the very beginning, this is done in the client side. 2) swift command makes a request to radosgw and gets 401 INFO:swiftclient:REQ: curl -i <radosgw url here>/swift/v1/AUTH_<some id here>/test3 -X POST -H "X-Auth-Token: here><token " -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized Thanks a lot again, -Mika On Tue, Jan 5, 2021 at 11:19 AM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote: > Hi, > > Which version of OpenStack do you have ? I guess , since Usurri ( or > may be even before ) swift authentification through keystone require > the account in url . You have to add this option in > "/etc/ceph/ceph.conf" , section rgw "rgw swift account in url = true"

or do it via setting directly

work.

https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-…

> 4) openstack endpoint create --region RegionOne object-store > internal >

https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-…

https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d…

<https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d=DwMFaQ&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=oc3C1TP2mMYCukAjjobWV7SPwto-zVeUvBG-JgRS3SI&s=D3W7JtLCq7AbYLGXj1Tm-RTLE4w95svqucaeAg87aeE&e=> "POST

> /v3/auth/tokens HTTP/1.1" 201 7032 > > . some openstack data here . > > DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): > ceph1:80 > DEBUG:urllib3.connectionpool:http://ceph1:80

<https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1-3A80&d=DwMFaQ&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=oc3C1TP2mMYCukAjjobWV7SPwto-zVeUvBG-JgRS3SI&s=vfsbb-sSKs_VnT0vrT_MZRnADOCDvRh0208AgDEvLeo&e=> "POST

/swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= 401 Unauthorized b'AccessDenied' _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Wissem MIMOUNA

9:08 p.m.

Hi, The user rgwswift should have the role admin in the project service . This user should be used in ceph to authenticate other users via keystone . What the following command show : openstack role assignment list –user rgwswift --project service –names Rgds De : Mika Saari <mika.saari(a)gmail.com> Envoyé : jeudi 7 janvier 2021 16:02 À : Wissem MIMOUNA <wissem.mimouna(a)fiducialcloud.fr> Cc : ceph-users(a)ceph.io Objet : Re: [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, Changed switch-openrc and verified the project to be "admin". Unfortunately problems stills. I think I have configured the Ceph now somehow wrong with command ceph config set mgr rgw_keystone_url http://controller:5000<https://urldefense.proofpoint.com/v2/url?u=http-3… It probably should be something like ceph config set client.radosgw.gateway rgw_keystone_url http:/controllerc:5000 I am not sure about this though. I tested configuring these parameters to /etc/ceph/ceph.conf as well, but not sure if those will affect inside docker containers. It seems that radosgw won't trigger any communication towards keystone. Will continue with this. Thanks, -Mika On Thu, Jan 7, 2021 at 3:08 PM Wissem MIMOUNA <wissem.mimouna@fiducialcloud.fr<mailto:wissem.mimouna@fiducialcloud.fr>> wrote: The user rgwswift only for radosgw config ( do not use it in your file openrc ) use swift user instead . Also , keep the default project to admin ( os_project_name ) . Rgds De : Mika Saari <mika.saari@gmail.com<mailto:mika.saari@gmail.com>> Envoyé : jeudi 7 janvier 2021 12:45 À : Wissem MIMOUNA <wissem.mimouna@fiducialcloud.fr<mailto:wissem.mimouna@fiducialcloud.fr>> Cc : ceph-users@ceph.io<mailto:ceph-users@ceph.io> Objet : Re: [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, Adding below what I tested. Do you see from this what I am doing wrong? Thank you very much, -Mika --clip clip-- OPENSTACK SIDE: [root@controller ~]# openstack user create --domain default --password-prompt rgwswift User Password: Repeat User Password: +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 85a86ec5c0264302b0471fd147042e0b | | name | rgwswift | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+ [root@controller ~]# openstack role add --project service --user rgwswift admin CEPH SIDE: [root@ceph1 ~]# ceph config set mgr rgw_keystone_accepted_roles "admin, _member_, Member, member, creator" [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_user rgwswift [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_project service [root@ceph1 ~]# ceph orch restart rgw.default.ou restart rgw.default.ou.ceph1.gxblht from host 'ceph1' CLIENT SIDE: $ . swift-openrc Where swift-openrc is like this: export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=service export OS_USERNAME=rgwswift export OS_PASSWORD=rgwswiftpw export OS_AUTH_URL=http://controller:5000/v3<https://urldefense.proofpoint.com/… export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 $ swift stat --debug Problem like earlier. First the swift client authenticates to the keystone and that works. Second it tries to contact radosgw, and that gives 401. Checked the rgw_process.cc : process_request and seems that there is no more debug information in the source. I assume the row int ret = client_io->init(g_ceph_context); gives < 0 which causes the process_request to return out with abort_early. On Thu, Jan 7, 2021 at 1:16 PM Wissem MIMOUNA <wissem.mimouna@fiducialcloud.fr<mailto:wissem.mimouna@fiducialcloud.fr>> wrote: Hi, The radosgw should have a dedicated user (different from you swift user) for authentifiation with keystone ( openstack) in the project "service" and you should also add the role "_member_" in the rgw_keystone_accepted_roles. Regards -----Message d'origine----- De : Mika Saari <mika.saari@gmail.com<mailto:mika.saari@gmail.com>> Envoyé : jeudi 7 janvier 2021 11:35 À : ceph-users@ceph.io<mailto:ceph-users@ceph.io> Objet : [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, I have added debug_rgw 20 to configuration. When checking docker logs -f <radosgw container id> I get this error for my radowgw request (swift post test3 --debug) Would there be a way to get more debug information from radosgw to solve this 401 problem ? Thanks a lot, -Mika --- clip clip ---- debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== starting new request req=0x7f1b5b32a6b0 ===== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 op->ERRORHANDLER: err_no=-1 new_err_no=-1 debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== req done req=0x7f1b5b32a6b0 op status=0 http_status=401 latency=0s ====== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 beast: 0x7f1b5b32a6b0: 10.0.2.10 - - [2021-01-07T10:32:42.269372+0000] "POST /swift/v1/AUTH_50f0ce372a4a4ed6a41126852358f097/test3 HTTP/1.1" 401 12 - "python-swiftclient-3.9.0" - --- clip clip ---- On Tue, Jan 5, 2021 at 8:00 PM Mika Saari <mika.saari@gmail.com<mailto:mika.saari@gmail.com>> wrote:

...

Hi, I am using indeed OpenStack Ussuri release. I changed the "gw swift account in url = true" directly with ceph config set ... command. Also checked that rgw_keystone_accepted_roles is correctly set and not the admin one. Also tested disabling rgw_keystone_verify_ssl. Should radosgw communicate with keystone somehow? I can not see my ceph-cluster requesting anything from keystone through any interface (tcpdump checked this one). I have tested restarting the radosgw with command "ceph orch restart rgw.default.ou" and seems that it brings the container down and up. Not sure though it is enough to bring the settings in use.q Current status is: 1) swift command seems to be able to authenticate with keystone at the very beginning, this is done in the client side. 2) swift command makes a request to radosgw and gets 401 INFO:swiftclient:REQ: curl -i <radosgw url here>/swift/v1/AUTH_<some id here>/test3 -X POST -H "X-Auth-Token: here><token " -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized Thanks a lot again, -Mika On Tue, Jan 5, 2021 at 11:19 AM Wissem MIMOUNA < wissem.mimouna@fiducialcloud.fr<mailto:wissem.mimouna@fiducialcloud.fr>> wrote:

Hi, Which version of OpenStack do you have ? I guess , since Usurri ( or may be even before ) swift authentification through keystone require the account in url . You have to add this option in "/etc/ceph/ceph.conf" , section rgw "rgw swift account in url = true" or do it via setting directly . Also , I noticed you did this ==> 3) ceph config set mgr rgw_keystone_accepted_admin_roles xxxx || I think , you should use the option "rgw keystone accepted roles xxxx" instead. Regards -----Message d'origine----- De : Mika Saari <mika.saari@gmail.com<mailto:mika.saari@gmail.com>> Envoyé : mardi 5 janvier 2021 10:03 À : ceph-users@ceph.io<mailto:ceph-users@ceph.io> Objet : [ceph-users] Ceph RadosGW & OpenStack swift problem Hi, Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW to work. I have managed to get the RadosGW working. I can manage it through a dashboard and use aws s3 client to create new buckets etc. When trying to use swift I get errors. Not sure how to continue to track the problem here. Any tips are welcome. Thank you very much, -Mika ------- What I have done and what are the results. Some data changed manually ------- What I have done: At OpenStack Side: 1) openstack user create --domain default --password-prompt swift 2) openstack role add --project service --user swift admin 3) openstack endpoint create --region RegionOne object-store public https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-… 4) openstack endpoint create --region RegionOne object-store internal https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-… 5) openstack endpoint create --region RegionOne object-store admin https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1&d= DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Ktt b6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7eX4&s=bm67b3lMVeLC 3sNvuyufFCe3AksJgfIgeI8SDorhHMU&e= At Ceph side: 1) ceph config set mgr rgw_keystone_api_version 3 2) ceph config set mgr rgw_keystone_url https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d… 3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000 _v3_auth_tokens&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KK a6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7 eX4&s=-98qpMcc8sdRTdN7AwNPIyGsIK1GaFvi_SC5GtZGUpY&e= DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000<https://urldefense.p… "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80<https://urldefense.proofpoi… "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoESZ0 &e= 401 Unauthorized b'AccessDenied' _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>

Mika Saari

10:46 p.m.

...

Hi, The user rgwswift should have the role admin in the project service . This user should be used in ceph to authenticate other users via keystone . What the following command show : openstack role assignment list –user rgwswift --project service –names Rgds *De :* Mika Saari <mika.saari(a)gmail.com> *Envoyé :* jeudi 7 janvier 2021 16:02 *À :* Wissem MIMOUNA <wissem.mimouna(a)fiducialcloud.fr> *Cc :* ceph-users(a)ceph.io *Objet :* Re: [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, Changed switch-openrc and verified the project to be "admin". Unfortunately problems stills. I think I have configured the Ceph now somehow wrong with command ceph config set mgr rgw_keystone_url http://controller:5000 <https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d=DwMFaQ&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=7KmstXtim2KMemw_3lT7T-OQwpy1vDI3u0gXdEEX6XY&s=K3qDOYJzxdBcrTQ9r1rXuhGl1TigBUlgxw6dC34lymw&e=> It probably should be something like ceph config set client.radosgw.gateway rgw_keystone_url http:/controllerc:5000 I am not sure about this though. I tested configuring these parameters to /etc/ceph/ceph.conf as well, but not sure if those will affect inside docker containers. It seems that radosgw won't trigger any communication towards keystone. Will continue with this. Thanks, -Mika On Thu, Jan 7, 2021 at 3:08 PM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote: The user rgwswift only for radosgw config ( do not use it in your file openrc ) use swift user instead . Also , keep the default project to admin ( os_project_name ) . Rgds *De :* Mika Saari <mika.saari(a)gmail.com> *Envoyé :* jeudi 7 janvier 2021 12:45 *À :* Wissem MIMOUNA <wissem.mimouna(a)fiducialcloud.fr> *Cc :* ceph-users(a)ceph.io *Objet :* Re: [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, Adding below what I tested. Do you see from this what I am doing wrong? Thank you very much, -Mika --clip clip-- OPENSTACK SIDE: [root@controller ~]# openstack user create --domain default --password-prompt rgwswift User Password: Repeat User Password: +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 85a86ec5c0264302b0471fd147042e0b | | name | rgwswift | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+ [root@controller ~]# openstack role add --project service --user rgwswift admin CEPH SIDE: [root@ceph1 ~]# ceph config set mgr rgw_keystone_accepted_roles "admin, _member_, Member, member, creator" [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_user rgwswift [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_project service [root@ceph1 ~]# ceph orch restart rgw.default.ou restart rgw.default.ou.ceph1.gxblht from host 'ceph1' CLIENT SIDE: $ . swift-openrc Where swift-openrc is like this: export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=service export OS_USERNAME=rgwswift export OS_PASSWORD=rgwswiftpw export OS_AUTH_URL=http://controller:5000/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000_v3&d=DwMFaQ&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=oc3C1TP2mMYCukAjjobWV7SPwto-zVeUvBG-JgRS3SI&s=xYsKH127snVkstwVzGM-ha6td0BdcY5-XQxutKOxNto&e=> export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 $ swift stat --debug Problem like earlier. First the swift client authenticates to the keystone and that works. Second it tries to contact radosgw, and that gives 401. Checked the rgw_process.cc : process_request and seems that there is no more debug information in the source. I assume the row int ret = client_io->init(g_ceph_context); gives < 0 which causes the process_request to return out with abort_early. On Thu, Jan 7, 2021 at 1:16 PM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote: Hi, The radosgw should have a dedicated user (different from you swift user) for authentifiation with keystone ( openstack) in the project "service" and you should also add the role "_member_" in the rgw_keystone_accepted_roles. Regards -----Message d'origine----- De : Mika Saari <mika.saari(a)gmail.com> Envoyé : jeudi 7 janvier 2021 11:35 À : ceph-users(a)ceph.io Objet : [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, I have added debug_rgw 20 to configuration. When checking docker logs -f <radosgw container id> I get this error for my radowgw request (swift post test3 --debug) Would there be a way to get more debug information from radosgw to solve this 401 problem ? Thanks a lot, -Mika --- clip clip ---- debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== starting new request req=0x7f1b5b32a6b0 ===== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 op->ERRORHANDLER: err_no=-1 new_err_no=-1 debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== req done req=0x7f1b5b32a6b0 op status=0 http_status=401 latency=0s ====== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 beast: 0x7f1b5b32a6b0: 10.0.2.10 - - [2021-01-07T10:32:42.269372+0000] "POST /swift/v1/AUTH_50f0ce372a4a4ed6a41126852358f097/test3 HTTP/1.1" 401 12 - "python-swiftclient-3.9.0" - --- clip clip ---- On Tue, Jan 5, 2021 at 8:00 PM Mika Saari <mika.saari(a)gmail.com> wrote:

Hi, I am using indeed OpenStack Ussuri release. I changed the "gw swift account in url = true" directly with ceph config set ... command. Also checked that rgw_keystone_accepted_roles is correctly set and not the admin one. Also tested disabling rgw_keystone_verify_ssl. Should radosgw communicate with keystone somehow? I can not see my ceph-cluster requesting anything from keystone through any interface (tcpdump checked this one). I have tested restarting the radosgw with command "ceph orch restart rgw.default.ou" and seems that it brings the container down and up. Not sure though it is enough to bring the settings in use.q Current status is: 1) swift command seems to be able to authenticate with keystone at the very beginning, this is done in the client side. 2) swift command makes a request to radosgw and gets 401 INFO:swiftclient:REQ: curl -i <radosgw url here>/swift/v1/AUTH_<some id here>/test3 -X POST -H "X-Auth-Token: here><token " -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized Thanks a lot again, -Mika On Tue, Jan 5, 2021 at 11:19 AM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote: > Hi, > > Which version of OpenStack do you have ? I guess , since Usurri ( or > may be even before ) swift authentification through keystone require > the account in url . You have to add this option in > "/etc/ceph/ceph.conf" , section rgw "rgw swift account in url = true"

or do it via setting directly

work.

https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-…

> 4) openstack endpoint create --region RegionOne object-store > internal >

https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUTH-5F-…

https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000&d…

> /v3/auth/tokens HTTP/1.1" 201 7032 > > . some openstack data here . > > DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): > ceph1:80 > DEBUG:urllib3.connectionpool:http://ceph1:80

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Tony Liu

8 Jan 8 Jan

6:57 a.m.

Is swift service endpoint created in OpenStack? Tony

...

-----Original Message----- From: Mika Saari <mika.saari(a)gmail.com> Sent: Thursday, January 7, 2021 3:45 AM To: Wissem MIMOUNA <wissem.mimouna(a)fiducialcloud.fr> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, Adding below what I tested. Do you see from this what I am doing wrong? Thank you very much, -Mika --clip clip-- OPENSTACK SIDE: [root@controller ~]# openstack user create --domain default --password- prompt rgwswift User Password: Repeat User Password: +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 85a86ec5c0264302b0471fd147042e0b | | name | rgwswift | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+ [root@controller ~]# openstack role add --project service --user rgwswift admin CEPH SIDE: [root@ceph1 ~]# ceph config set mgr rgw_keystone_accepted_roles "admin, _member_, Member, member, creator" [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_user rgwswift [root@ceph1 ~]# ceph config set mgr rgw_keystone_admin_project service [root@ceph1 ~]# ceph orch restart rgw.default.ou restart rgw.default.ou.ceph1.gxblht from host 'ceph1' CLIENT SIDE: $ . swift-openrc Where swift-openrc is like this: export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=service export OS_USERNAME=rgwswift export OS_PASSWORD=rgwswiftpw export OS_AUTH_URL=http://controller:5000/v3 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 $ swift stat --debug Problem like earlier. First the swift client authenticates to the keystone and that works. Second it tries to contact radosgw, and that gives 401. Checked the rgw_process.cc : process_request and seems that there is no more debug information in the source. I assume the row int ret = client_io->init(g_ceph_context); gives < 0 which causes the process_request to return out with abort_early. On Thu, Jan 7, 2021 at 1:16 PM Wissem MIMOUNA < wissem.mimouna(a)fiducialcloud.fr> wrote:

Hi, The radosgw should have a dedicated user (different from you swift user) for authentifiation with keystone ( openstack) in the project "service" and you should also add the role "_member_" in the

rgw_keystone_accepted_roles.

Regards -----Message d'origine----- De : Mika Saari <mika.saari(a)gmail.com> Envoyé : jeudi 7 janvier 2021 11:35 À : ceph-users(a)ceph.io Objet : [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi, I have added debug_rgw 20 to configuration. When checking docker logs -f <radosgw container id> I get this error for my radowgw request (swift post test3 --debug) Would there be a way to get more debug information from radosgw to solve this 401 problem ? Thanks a lot, -Mika --- clip clip ---- debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== starting new request req=0x7f1b5b32a6b0 ===== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 op->ERRORHANDLER: err_no=-1 new_err_no=-1 debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== req done req=0x7f1b5b32a6b0 op status=0 http_status=401 latency=0s ====== debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 beast: 0x7f1b5b32a6b0: 10.0.2.10 - - [2021-01-07T10:32:42.269372+0000] "POST /swift/v1/AUTH_50f0ce372a4a4ed6a41126852358f097/test3 HTTP/1.1" 401 12 - "python-swiftclient-3.9.0" - --- clip clip ---- On Tue, Jan 5, 2021 at 8:00 PM Mika Saari <mika.saari(a)gmail.com> wrote: > Hi, > > I am using indeed OpenStack Ussuri release. I changed the "gw > swift account in url = true" directly with ceph config set ... > command. Also checked that rgw_keystone_accepted_roles is correctly > set and not the admin one. Also tested disabling

rgw_keystone_verify_ssl.

> > Should radosgw communicate with keystone somehow? I can not see my > ceph-cluster requesting anything from keystone through any interface > (tcpdump checked this one). I have tested restarting the radosgw > with command "ceph orch restart rgw.default.ou" and seems that it > brings the container down and up. Not sure though it is enough to > bring the settings in use.q > > Current status is: > 1) swift command seems to be able to authenticate with keystone > at the very beginning, this is done in the client side. > 2) swift command makes a request to radosgw and gets 401 > INFO:swiftclient:REQ: curl -i <radosgw url > here>/swift/v1/AUTH_<some id here>/test3 -X POST -H "X-Auth-Token: > here><token " -H "Content-Length: 0" > INFO:swiftclient:RESP STATUS: 401 Unauthorized > > Thanks a lot again, > -Mika > > On Tue, Jan 5, 2021 at 11:19 AM Wissem MIMOUNA < > wissem.mimouna(a)fiducialcloud.fr> wrote: > >> Hi, >> >> Which version of OpenStack do you have ? I guess , since Usurri ( >> or may be even before ) swift authentification through keystone >> require the account in url . You have to add this option in >> "/etc/ceph/ceph.conf" , section rgw "rgw swift account in url =

true"

or do it via setting directly

> . Also , I noticed you did this ==> 3) ceph config set mgr > rgw_keystone_accepted_admin_roles xxxx || I think , you should use > the option "rgw keystone accepted roles xxxx" instead. > > Regards > > -----Message d'origine----- > De : Mika Saari <mika.saari(a)gmail.com> Envoyé : mardi 5 janvier > 2021 > 10:03 À : ceph-users(a)ceph.io Objet : [ceph-users] Ceph RadosGW & > OpenStack swift problem > > Hi, > > Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW > to

work. >> I have managed to get the RadosGW working. I can manage it through >> a dashboard and use aws s3 client to create new buckets etc. When >> trying to use swift I get errors. >> >> Not sure how to continue to track the problem here. Any tips are >> welcome. >> >> Thank you very much, >> -Mika >> >> ------- What I have done and what are the results. Some data >> changed manually ------- >> What I have done: >> At OpenStack Side: >> 1) openstack user create --domain default --password-prompt

swift

> 2) openstack role add --project service --user swift admin > 3) openstack endpoint create --region RegionOne object-store > public

https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_AUT H-5F-25-255C-28project-5Fid-255C-29s&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U4 6oD9d1KMRwdpbF9VLg7eX4&s=-1FtdhjTcNA8jPSUoyoUfsPl5uqTqu4I_ThTOJNLjtg&e =

> 4) openstack endpoint create --region RegionOne object-store > internal >

> 5) openstack endpoint create --region RegionOne object-store > admin > https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1& > d= > DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9K > tt > b6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7eX4&s=bm67b3lMVe > LC > 3sNvuyufFCe3AksJgfIgeI8SDorhHMU&e= > > At Ceph side: > 1) ceph config set mgr rgw_keystone_api_version 3 > 2) ceph config set mgr rgw_keystone_url >

https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A5000& d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kt tb6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VLg7eX4&s=lyXWyh-BXrik PWqWM3dcPW4ZofvjiAxnq-nXsjifnEw&e=

3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A50 00 _v3_auth_tokens&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6 KK a6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VL g7 eX4&s=-98qpMcc8sdRTdN7AwNPIyGsIK1GaFvi_SC5GtZGUpY&e= DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80 "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_ AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZt yA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw 1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoES Z0 &e= -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_ AU TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZt yA &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw 1U 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoES Z0 &e= 401 Unauthorized b'AccessDenied' _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Mika Saari

10:32 a.m.

Hi all, Thanks a lot for the tips. I finally found out what was the problem. The "WHO" field in the "ceph config set" was not correct. I was using "client.radosgw.gateway" but after using "ceph config dump" I found out that when radosgw was executed up it had generated "client.rgw.default" named "WHO" field with some parameters, which was the correct one to use. After using commands in ceph/radosgw machine, everything started to work: ceph config set client.rgw.default rgw_keystone_url http://controller:5000 ceph config set client.rgw.default rgw_keystone_admin_domain default ceph config set client.rgw.default rgw_keystone_admin_password rgwswiftpw ceph config set client.rgw.default rgw_keystone_admin_project service ceph config set client.rgw.default rgw_keystone_admin_user rgwswift ceph config set client.rgw.default rgw_keystone_api_version 3 ceph config set client.rgw.default rgw_swift_account_in_url true After these commands RadosGW started to communicate to keystone, created accounts to radosgw users and also bucket creation started to work. These all with OpenStack Ussuri & Ceph Octopus 15.2.8. Summary would be: 1) Check correct "WHO" with ceph config dump after the radosgw is running 2) Create correct users to openstack 3) Start using Thanks a lot once more for all the help ! -Mika On Fri, Jan 8, 2021 at 5:27 AM Tony Liu <tonyliu0592(a)hotmail.com> wrote:

...

Is swift service endpoint created in OpenStack? Tony

Hi, The radosgw should have a dedicated user (different from you swift user) for authentifiation with keystone ( openstack) in the project "service" and you should also add the role "_member_" in the

rgw_keystone_accepted_roles. > > Regards > > -----Message d'origine----- > De : Mika Saari <mika.saari(a)gmail.com> Envoyé : jeudi 7 janvier 2021 > 11:35 À : ceph-users(a)ceph.io Objet : [ceph-users] Re: Ceph RadosGW & > OpenStack swift problem > > Hi, > > I have added debug_rgw 20 to configuration. When checking docker > logs -f <radosgw container id> I get this error for my radowgw request > (swift post > test3 --debug) > > Would there be a way to get more debug information from radosgw to > solve this 401 problem ? > > Thanks a lot, > -Mika > > --- clip clip ---- > debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== starting new > request req=0x7f1b5b32a6b0 ===== debug 2021-01-07T10:32:42.269+0000 > 7f1ae111b700 1 op->ERRORHANDLER: > err_no=-1 new_err_no=-1 > debug 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 ====== req done > req=0x7f1b5b32a6b0 op status=0 http_status=401 latency=0s ====== debug > 2021-01-07T10:32:42.269+0000 7f1ae111b700 1 beast: 0x7f1b5b32a6b0: > 10.0.2.10 - - [2021-01-07T10:32:42.269372+0000] "POST > /swift/v1/AUTH_50f0ce372a4a4ed6a41126852358f097/test3 HTTP/1.1" 401 12 > - "python-swiftclient-3.9.0" - > --- clip clip ---- > > > On Tue, Jan 5, 2021 at 8:00 PM Mika Saari <mika.saari(a)gmail.com>

wrote:

> Hi, > > I am using indeed OpenStack Ussuri release. I changed the "gw > swift account in url = true" directly with ceph config set ... > command. Also checked that rgw_keystone_accepted_roles is correctly > set and not the admin one. Also tested disabling

rgw_keystone_verify_ssl.

true"

or do it via setting directly

> . Also , I noticed you did this ==> 3) ceph config set mgr > rgw_keystone_accepted_admin_roles xxxx || I think , you should use > the option "rgw keystone accepted roles xxxx" instead. > > Regards > > -----Message d'origine----- > De : Mika Saari <mika.saari(a)gmail.com> Envoyé : mardi 5 janvier > 2021 > 10:03 À : ceph-users(a)ceph.io Objet : [ceph-users] Ceph RadosGW & > OpenStack swift problem > > Hi, > > Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW > to

swift

> 2) openstack role add --project service --user swift admin > 3) openstack endpoint create --region RegionOne object-store > public

> 4) openstack endpoint create --region RegionOne object-store > internal >

> 3) ceph config set mgr rgw_keystone_accepted_admin_roles admin > 4) ceph config set mgr rgw_keystone_admin_user swift > 5) ceph config set mgr rgw_keystone_admin_password swift_test > 6) ceph config set mgr rgw_keystone_admin_domain default > 7) ceph config set mgr rgw_keystone_admin_project service > for project I have tested different projects e.g. service and > admin > > Now when testing the API using swift client I get next: > 1) swift post test3 --debug > > DEBUG:keystoneclient.auth.identity.v3.base:Making authentication > request to > https://urldefense.proofpoint.com/v2/url?u=http-3A__controller-3A50 > 00 > _v3_auth_tokens&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZtyA&r=h1fIFv3Ydv-kdH6 > KK > a6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw1U46oD9d1KMRwdpbF9VL > g7 eX4&s=-98qpMcc8sdRTdN7AwNPIyGsIK1GaFvi_SC5GtZGUpY&e= > DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): > controller:5000 > DEBUG:urllib3.connectionpool:http://controller:5000 "POST > /v3/auth/tokens HTTP/1.1" 201 7032 > > . some openstack data here . > > DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): > ceph1:80 > DEBUG:urllib3.connectionpool:http://ceph1:80 "POST > /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 > INFO:swiftclient:REQ: curl -i > > https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_ > AU > TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZt > yA > &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw > 1U > 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoES > Z0 > &e= > -X POST -H > "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" > INFO:swiftclient:RESP STATUS: 401 Unauthorized > > and finally I get > Container POST failed: > > https://urldefense.proofpoint.com/v2/url?u=http-3A__ceph1_swift_v1_ > AU > TH-5Fadsfasdfasdfasdfasdfasdf_test3&d=DwICAg&c=1tDFxPZjcWEmlmmx4CZt > yA > &r=h1fIFv3Ydv-kdH6KKa6lmB20LbjUiXP9Kttb6tTs__E&m=EmlYLMTNHaWmSJrApw > 1U > 46oD9d1KMRwdpbF9VLg7eX4&s=g1inMAENxiOpxc4L8FlmbLypegdcQwgH8drm6aoES > Z0 > &e= > 401 Unauthorized > b'AccessDenied' > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send > an email to ceph-users-leave(a)ceph.io >

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Bujack, Stefan

14 Jan 14 Jan

11:39 a.m.

Hello, thank you very much. Your debugging helped me a lot finding a solution for my own problem with keystone and radosgw. Greets Stefan ----- Original Message ----- From: "Mika Saari" <mika.saari(a)gmail.com> To: "ceph-users" <ceph-users(a)ceph.io> Sent: Friday, 8 January, 2021 08:02:31 Subject: [ceph-users] Re: Ceph RadosGW & OpenStack swift problem Hi all, Thanks a lot for the tips. I finally found out what was the problem. The "WHO" field in the "ceph config set" was not correct. I was using "client.radosgw.gateway" but after using "ceph config dump" I found out that when radosgw was executed up it had generated "client.rgw.default" named "WHO" field with some parameters, which was the correct one to use. After using commands in ceph/radosgw machine, everything started to work: ceph config set client.rgw.default rgw_keystone_url http://controller:5000 ceph config set client.rgw.default rgw_keystone_admin_domain default ceph config set client.rgw.default rgw_keystone_admin_password rgwswiftpw ceph config set client.rgw.default rgw_keystone_admin_project service ceph config set client.rgw.default rgw_keystone_admin_user rgwswift ceph config set client.rgw.default rgw_keystone_api_version 3 ceph config set client.rgw.default rgw_swift_account_in_url true After these commands RadosGW started to communicate to keystone, created accounts to radosgw users and also bucket creation started to work. These all with OpenStack Ussuri & Ceph Octopus 15.2.8. Summary would be: 1) Check correct "WHO" with ceph config dump after the radosgw is running 2) Create correct users to openstack 3) Start using Thanks a lot once more for all the help ! -Mika On Fri, Jan 8, 2021 at 5:27 AM Tony Liu <tonyliu0592(a)hotmail.com> wrote:

...

Is swift service endpoint created in OpenStack? Tony

Hi, The radosgw should have a dedicated user (different from you swift user) for authentifiation with keystone ( openstack) in the project "service" and you should also add the role "_member_" in the

wrote:

rgw_keystone_verify_ssl.

true"

or do it via setting directly

> . Also , I noticed you did this ==> 3) ceph config set mgr > rgw_keystone_accepted_admin_roles xxxx || I think , you should use > the option "rgw keystone accepted roles xxxx" instead. > > Regards > > -----Message d'origine----- > De : Mika Saari <mika.saari(a)gmail.com> Envoyé : mardi 5 janvier > 2021 > 10:03 À : ceph-users(a)ceph.io Objet : [ceph-users] Ceph RadosGW & > OpenStack swift problem > > Hi, > > Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW > to

swift

> 2) openstack role add --project service --user swift admin > 3) openstack endpoint create --region RegionOne object-store > public

> 4) openstack endpoint create --region RegionOne object-store > internal >

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1216

days inactive

1225

days old

ceph-users@ceph.io

Manage subscription

12 comments

4 participants

tags (0)

participants (4)

Bujack, Stefan
Mika Saari
Tony Liu
Wissem MIMOUNA