For the sake of the archive and future readers: I think we now have an explanation for this issue. Our cloud is one of the few remaining OpenStack deploys which predates the use of UUIDs for OpenStack tenant names; instead our project ids are typically the same as project names. Radosgw checks project ids and rejects any that contain characters other than letters, numbers, and underscores[0]. So that check is actively rejecting many of our projects, including all with - in their names. IMO that check is wrong (see discussion of a similar issue at [1]) but in the meantime we're exploring various terrible workarounds. On the off-chance that anyone reading this has encountered and fixed this same issue, please reach out! -Andrew [0] https://github.com/ceph/ceph/commit/d50ef542372f541ac9411f655cddd5fcab4dceac [1] https://review.opendev.org/c/openstack/cinder/+/864585 On 7/10/23 2:59 PM, Andrew Bogott wrote:

I'm in the process of adding the radosgw service to our OpenStack cloud and hoping to re-use keystone for discovery and auth. Things seem to work fine with many keystone tenants, but as soon as we try to do something in a project with a '-' in its name everything fails.

Here's an example, using the openstack swift cli:

root@cloudcontrol2001-dev:~# OS_PROJECT_ID="testlabs" openstack container create 'makethiscontainer' +---------------+-------------------+----------------------------------------------------+

| account       | container         | x-trans-id                                         | +---------------+-------------------+----------------------------------------------------+

| AUTH_testlabs | makethiscontainer | tx0000008c311dbda86c695-0064ac5fad-6927acd-default | +---------------+-------------------+----------------------------------------------------+

root@cloudcontrol2001-dev:~# OS_PROJECT_ID="service" openstack container create 'makethiscontainer' +--------------+-------------------+----------------------------------------------------+

| account      | container         | x-trans-id                                         | +--------------+-------------------+----------------------------------------------------+

| AUTH_service | makethiscontainer | tx00000b341a22866f65e44-0064ac5fb7-6927acd-default | +--------------+-------------------+----------------------------------------------------+

root@cloudcontrol2001-dev:~# OS_PROJECT_ID="admin-monitoring" openstack container create 'makethiscontainer' Bad Request (HTTP 400) (Request-ID: tx00000f7326bb541b4d2a9-0064ac5fc2-6927acd-default)

Before I dive into the source code, is this a known issue and/or something I can configure? Dash-named-projects work fine in keystone and seem to also work fine with standalone rados; I assume the issue is somewhere in the communication between the two. I suspected the implicit user creation code, but that seems to be working properly:

# radosgw-admin user list [     "cloudvirt-canary$cloudvirt-canary",     "testlabs$testlabs",     "paws-dev$paws-dev",     "andrewtestproject$andrewtestproject",     "admin-monitoring$admin-monitoring",     "taavi-test-project$taavi-test-project",     "admin$admin",     "taavitestproject$taavitestproject",     "bastioninfra-codfw1dev$bastioninfra-codfw1dev", ]

Here is the radosgw section of my ceph.conf:

[client.radosgw]

    host = 10.192.20.9     keyring = /etc/ceph/ceph.client.radosgw.keyring     rgw frontends = "civetweb port=18080"     rgw_keystone_verify_ssl = false     rgw_keystone_api_version = 3     rgw_keystone_url = https://openstack.codfw1dev.wikimediacloud.org:25000     rgw_keystone_accepted_roles = 'reader, admin, member'     rgw_keystone_implicit_tenants = true     rgw_keystone_admin_domain = default     rgw_keystone_admin_project = service     rgw_keystone_admin_user = swift     rgw_keystone_admin_password = (redacted)     rgw_s3_auth_use_keystone = true     rgw_swift_account_in_url = true

    rgw_user_default_quota_max_objects = 4096     rgw_user_default_quota_max_size = 8589934592

And here's a debug log of a failed transaction:

    https://phabricator.wikimedia.org/P49539

Thanks in advance!