Need help integrating radosgw with keystone for openstack swift
Hello, I am struggling to integrate ceph radosgw as obejctstore in openstack swift via keystone. Could someone please have a look at my configs and help finding the issue? Many thanks ins advance. ceph version 14.2.11 nautilus (stable) [root@ciosmon06 ~]# cat /etc/ceph/ceph.conf [global] fsid = someid mon_initial_members = ciosmon06, ciosmon07, ciosmon08, ciosmon09, ciosmon10 mon_host = 10.254.32.32,10.254.32.33,10.254.32.34,10.254.32.35,10.254.32.36 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 10.254.32.0/24 cluster_network = 10.254.36.0/24 [client.radosgw.gateway] debug ms = 1 debug rgw = 20 rgw host = 10.254.32.32 rgw keystone verify ssl = false rgw keystone api version = 3 rgw keystone url = https://keystone-intern.desy.de:5000 rgw keystone admin domain = default rgw keystone admin project = service rgw keystone admin user = swift rgw keystone admin password = password rgw s3 auth use keystone = true rgw swift account in url = true rgw keystone implicit tenants = true rgw swift versioning enabled = true rgw enable apis = swift, s3 rgw keystone accepted roles = admin, _member_, member, swiftoperator rgw keystone accepted admin roles = admin, _member_ keyring = /etc/ceph/client.rgw.ciosmon06.keyring [root@ciosmon06 ~]# systemctl status ceph-radosgw@rgw.ciosmon06.service ● ceph-radosgw@rgw.ciosmon06.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-10-21 17:22:04 CEST; 3min 36s ago Main PID: 3372096 (radosgw) CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.ciosmon06.service └─3372096 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ciosmon06 --setuser ceph --setgroup ceph Oct 21 17:22:04 ciosmon06.desy.de systemd[1]: Started Ceph rados gateway. [root@ciosmon06 ~]# netstat -ntlup Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 3372096/radosgw tcp6 0 0 :::7480 :::* LISTEN 3372096/radosgw [root@ciosmon06 ceph]# ls -la total 32 drwxr-xr-x 2 root root 4096 Oct 21 18:18 . drwxr-xr-x. 93 root root 12288 Sep 24 05:25 .. -rw------- 1 root root 63 Feb 24 2020 ceph.client.admin.keyring -rw-r--r-- 1 root root 1036 Oct 21 18:18 ceph.conf -rw------- 1 root root 71 Oct 21 16:38 client.rgw.ciosmon06.keyring [root@ciosmon06 ~]# cat /etc/ceph/client.rgw.ciosmon06.keyring [client.rgw.ciosmon06] key = somekey [root@ciosmon06 ~]# ceph auth ls client.rgw.ciosmon06 key: somekey caps: [mon] allow rw caps: [osd] allow rwx Openstack ussuri on ubuntu 20.04 root@keystone:~# openstack endpoint list +----------------------------------+-----------+--------------+---------------------+---------+-----------+------------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+---------------------+---------+-----------+------------------------------------------------------------+ | 0ee9c91af2424e33a91a4c118b693301 | RegionOne | swift | object-store | True | internal | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | 1fe9f91db7f942d2a8cb2163bd09772d | RegionOne | keystone | identity | True | admin | https://keystone-intern.desy.de:5000/v3/ | | 4719a266432f45bda380c52486421e62 | RegionOne | swift | object-store | True | public | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | 95a292a5584d4e2c9f8177af3a312494 | RegionOne | keystone | identity | True | public | https://keystone-intern.desy.de:5000/v3/ | | e68b3990e74447bfa35a5d6aa66ca2aa | RegionOne | swift | object-store | True | admin | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | efc1df3cb21344bfb74d8e345e267606 | RegionOne | keystone | identity | True | internal | https://keystone-intern.desy.de:5000/v3/ | root@keystone:~# openstack service list +----------------------------------+-----------+---------------------+ | ID | Name | Type | +----------------------------------+-----------+---------------------+ | 4881557156f844df87d13e427d02b74a | swift | object-store | | 6d928ad49d954a179e37dc17f96c4b1e | keystone | identity | root@keystone:~# openstack user list +----------------------------------+-----------+ | ID | Name | +----------------------------------+-----------+ | ca94085253944ba3803f9d3202172b80 | admin | | ea025c992c224b5f8ca5fea0334eaf15 | swift | +----------------------------------+-----------+ root@keystone:~# openstack role assignment list --names --project service +-------+-------------------+-------+-----------------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+-------------------+-------+-----------------+--------+--------+-----------+ | admin | swift@Default | | service@Default | | | False | +-------+-------------------+-------+-----------------+--------+--------+-----------+ Tests: root@it-build:~# openstack ec2 credentials create +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access | bc6f1ad9064d4097ad7a323c8368780b | | links | {'self': 'https://keystone-intern.desy.de:5000/v3/users/926c750033e668f0af2330b1c7c723a05b86fa393655369fdb1a5622ae65dac8/credentials/OS-EC2/bc6f1ad9064d4097ad7a323c8368780b'} | | project_id | 286f5d2b38ae4595ba9ff8129e754f54 | | secret | c6d9a58220fd4491ba8654d6d3b3c4dc | | trust_id | None | | user_id | 926c750033e668f0af2330b1c7c723a05b86fa393655369fdb1a5622ae65dac8 | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ root@it-build:~# /usr/local/bin/aws configure AWS Access Key ID [****************e90c]: bc6f1ad9064d4097ad7a323c8368780b AWS Secret Access Key [****************6ba4]: c6d9a58220fd4491ba8654d6d3b3c4dc Default region name [default]: Default output format [None]: root@it-build:~# /usr/local/bin/aws --endpoint='http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s' s3 ls s3:// expected string or bytes-like object [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-21 18:24:24.710 7efeabf69700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:24:24.713 7efeabf69700 1 ====== req done req=0x562f3dbb08f0 op status=-2024 http_status=301 latency=0.003s ====== root@keystone:~# openstack role assignment list --names --project 286f5d2b38ae4595ba9ff8129e754f54 | grep swiftoperator | swiftoperator | myusername@closedbeta | | closedbeta@closedbeta | | | False | root@it-build:~# cat myusername-openrc-closedbeta export OS_PROJECT_DOMAIN_NAME=closedbeta export OS_USER_DOMAIN_NAME=closedbeta export OS_PROJECT_NAME=closedbeta export OS_USERNAME=myusername export OS_PASSWORD=password export OS_AUTH_URL=https://keystone-intern.desy.de:5000 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 export OS_VOLUME_API_VERSION=3 root@it-build:~# openstack container list Unrecognized schema in response body. (HTTP 401) (Request-ID: tx000000000000000000005-005f9060f7-26173f-default) [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-21 18:25:26.868 7efea9f65700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:25:26.868 7efea9f65700 1 ====== req done req=0x562f3dbb08f0 op status=0 http_status=401 latency=0s ====== 2020-10-21 18:25:27.088 7efeadf6d700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:25:27.088 7efeadf6d700 1 ====== req done req=0x562f3dbb08f0 op status=0 http_status=401 latency=0s ====== Greets Stefan Bujack
Hi, in our setup (ceph 15.2.4, openstack train) the swift endpoint URLs are different, e.g. # openstack endpoint list --service swift +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------------------------------+ | 521a556e391c40cc8d242f0f61a22812 | RegionOne | swift | object-store | True | public | https://s3.<redacted>/swift/v1 | .... And a somewhat related personal opinion: do not use swift. The API requires using openstack credentials, and in many cases these credentials are the main user credentials used for accessing openstack (there are other methods, but most users are not aware of this). If instances want to access data in the object storage, you have to store the credentials in the instance. If an instance is exposed to the internet, it may be attacked and broken into; as a result the openstack credentials might end up in the wrong hands. I'm not sure whether using other methods like application credentials can reduce the problem e.g. by restricting them to certain services. But you can encourage users to use the S3 interface instead. S3 credentials can be created in the openstack web interface and by command line; they are scoped to a certain project only, and if you do not use some AWS compatibility layer they can _only_ be used for authentication in the S3 API. It's probably still a problem if they are stolen, but it is not as worse as openstack credentials... Just my 0.02 euro Regards, Burkhard
Hy, I tried your endpoint configuration but with the same outcome. Maybe I am missing something.... I also don't know if I am testing the right way. But thank you for your answer and your help. Greets Stefan Bujack root@keystone:~# openstack endpoint list | grep swift | 0ee9c91af2424e33a91a4c118b693301 | RegionOne | swift | object-store | True | internal | http://ciosmon06.desy.de:7480/swift/v1/ | | 4719a266432f45bda380c52486421e62 | RegionOne | swift | object-store | True | public | http://ciosmon06.desy.de:7480/swift/v1/ | | e68b3990e74447bfa35a5d6aa66ca2aa | RegionOne | swift | object-store | True | admin | http://ciosmon06.desy.de:7480/swift/v1/ | root@it-build:~# openstack container list Unrecognized schema in response body. (HTTP 401) (Request-ID: tx000000000000000000007-005f914731-26173f-default) [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-22 10:47:45.535 7efea6f5f700 1 ====== req done req=0x562f3de148f0 op status=0 http_status=401 latency=0.000999999s ====== 2020-10-22 10:47:45.798 7efea675e700 1 ====== starting new request req=0x562f3de148f0 ===== 2020-10-22 10:47:45.798 7efea675e700 1 ====== req done req=0x562f3de148f0 op status=0 http_status=401 latency=0s ====== root@it-build:~# openstack ec2 credentials create +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access | 91fe4a54ac4547b2a127fc4599bd7580 | | links | {'self': 'https://keystone-intern.desy.de:5000/v3/users/926c750033e668f0af2330b1c7c723a05b86fa393655369fdb1a5622ae65dac8/credentials/OS-EC2/91fe4a54ac4547b2a127fc4599bd7580'} | | project_id | 286f5d2b38ae4595ba9ff8129e754f54 | | secret | e8e0035d228743cfb40083d84d6f3580 | | trust_id | None | | user_id | 926c750033e668f0af2330b1c7c723a05b86fa393655369fdb1a5622ae65dac8 | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ root@it-build:~# /usr/local/bin/aws configure AWS Access Key ID [****************780b]: 91fe4a54ac4547b2a127fc4599bd7580 AWS Secret Access Key [****************c4dc]: e8e0035d228743cfb40083d84d6f3580 Default region name [default]: Default output format [None]: root@it-build:~# /usr/local/bin/aws --endpoint='http://ciosmon06.desy.de:7480/swift/v1/' s3 ls s3:// An error occurred (404) when calling the ListBuckets operation: Not Found root@it-build:~# /usr/local/bin/aws --endpoint='http://ciosmon06.desy.de:7480' s3 ls s3:// An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: Unknown [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-22 10:49:57.886 7efea2f57700 1 ====== starting new request req=0x562f3de248f0 ===== 2020-10-22 10:49:57.888 7efea2f57700 1 ====== req done req=0x562f3de248f0 op status=-2 http_status=404 latency=0.002s ====== 2020-10-22 10:50:22.344 7efea0752700 1 ====== starting new request req=0x562f3de488f0 ===== 2020-10-22 10:50:22.346 7efea0752700 1 ====== req done req=0x562f3de488f0 op status=0 http_status=403 latency=0.002s ====== ----- Original Message ----- From: "Burkhard Linke" <Burkhard.Linke@computational.bio.uni-giessen.de> To: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, 22 October, 2020 10:11:22 Subject: [ceph-users] Re: Need help integrating radosgw with keystone for openstack swift Hi, in our setup (ceph 15.2.4, openstack train) the swift endpoint URLs are different, e.g. # openstack endpoint list --service swift +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------------------------------+ | 521a556e391c40cc8d242f0f61a22812 | RegionOne | swift | object-store | True | public | https://s3.<redacted>/swift/v1 | .... And a somewhat related personal opinion: do not use swift. The API requires using openstack credentials, and in many cases these credentials are the main user credentials used for accessing openstack (there are other methods, but most users are not aware of this). If instances want to access data in the object storage, you have to store the credentials in the instance. If an instance is exposed to the internet, it may be attacked and broken into; as a result the openstack credentials might end up in the wrong hands. I'm not sure whether using other methods like application credentials can reduce the problem e.g. by restricting them to certain services. But you can encourage users to use the S3 interface instead. S3 credentials can be created in the openstack web interface and by command line; they are scoped to a certain project only, and if you do not use some AWS compatibility layer they can _only_ be used for authentication in the S3 API. It's probably still a problem if they are stolen, but it is not as worse as openstack credentials... Just my 0.02 euro Regards, Burkhard _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Bujack, Stefan
-
Burkhard Linke