For the sake of the archive and future readers: I think we now have an explanation for this issue. Our cloud is one of the few remaining OpenStack deploys which predates the use of UUIDs for OpenStack tenant names; instead our project ids are typically the same as project names. Radosgw checks project ids and rejects any that contain characters other than letters, numbers, and underscores[0]. So that check is actively rejecting many of our projects, including all with - in their names. IMO that check is wrong (see discussion of a similar issue at [1]) but in the meantime we're exploring various terrible workarounds. On the off-chance that anyone reading this has encountered and fixed this same issue, please reach out! -Andrew [0] https://github.com/ceph/ceph/commit/d50ef542372f541ac9411f655cddd5fcab4dceac [1] https://review.opendev.org/c/openstack/cinder/+/864585 On 7/10/23 2:59 PM, Andrew Bogott wrote:
I'm in the process of adding the radosgw service to our OpenStack cloud and hoping to re-use keystone for discovery and auth. Things seem to work fine with many keystone tenants, but as soon as we try to do something in a project with a '-' in its name everything fails.
Here's an example, using the openstack swift cli:
root@cloudcontrol2001-dev:~# OS_PROJECT_ID="testlabs" openstack container create 'makethiscontainer' +---------------+-------------------+----------------------------------------------------+
| account | container | x-trans-id | +---------------+-------------------+----------------------------------------------------+
| AUTH_testlabs | makethiscontainer | tx0000008c311dbda86c695-0064ac5fad-6927acd-default | +---------------+-------------------+----------------------------------------------------+
root@cloudcontrol2001-dev:~# OS_PROJECT_ID="service" openstack container create 'makethiscontainer' +--------------+-------------------+----------------------------------------------------+
| account | container | x-trans-id | +--------------+-------------------+----------------------------------------------------+
| AUTH_service | makethiscontainer | tx00000b341a22866f65e44-0064ac5fb7-6927acd-default | +--------------+-------------------+----------------------------------------------------+
root@cloudcontrol2001-dev:~# OS_PROJECT_ID="admin-monitoring" openstack container create 'makethiscontainer' Bad Request (HTTP 400) (Request-ID: tx00000f7326bb541b4d2a9-0064ac5fc2-6927acd-default)
Before I dive into the source code, is this a known issue and/or something I can configure? Dash-named-projects work fine in keystone and seem to also work fine with standalone rados; I assume the issue is somewhere in the communication between the two. I suspected the implicit user creation code, but that seems to be working properly:
# radosgw-admin user list [ "cloudvirt-canary$cloudvirt-canary", "testlabs$testlabs", "paws-dev$paws-dev", "andrewtestproject$andrewtestproject", "admin-monitoring$admin-monitoring", "taavi-test-project$taavi-test-project", "admin$admin", "taavitestproject$taavitestproject", "bastioninfra-codfw1dev$bastioninfra-codfw1dev", ]
Here is the radosgw section of my ceph.conf:
[client.radosgw]
host = 10.192.20.9 keyring = /etc/ceph/ceph.client.radosgw.keyring rgw frontends = "civetweb port=18080" rgw_keystone_verify_ssl = false rgw_keystone_api_version = 3 rgw_keystone_url = https://openstack.codfw1dev.wikimediacloud.org:25000 rgw_keystone_accepted_roles = 'reader, admin, member' rgw_keystone_implicit_tenants = true rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = (redacted) rgw_s3_auth_use_keystone = true rgw_swift_account_in_url = true
rgw_user_default_quota_max_objects = 4096 rgw_user_default_quota_max_size = 8589934592
And here's a debug log of a failed transaction:
https://phabricator.wikimedia.org/P49539
Thanks in advance!