Ceph RGW Barbican Integration — Tenant isolation issue (Barbican keys from one project usable by another)
Hi Team, Lab Environment * Ceph: 19.2.x (Squid) * OpenStack: Caracal * OS: Ubuntu 22.04 LTS * Deployment: `cephadm` --- Setup Summary We have a working Ceph RGW + Barbican integration with: * Keystone auth: `rgw_s3_auth_use_keystone = true` * SSE-KMS backend: `rgw_crypt_s3_kms_backend = barbican` * Single RGW Barbican service user under a single project: ``` rgw_keystone_barbican_user = rgw-barbican rgw_keystone_barbican_project = service rgw_keystone_barbican_domain = default rgw_barbican_url = http://controller:9311 rgw_crypt_s3_kms_backend = barbican rgw_crypt_s3_kms_encryption_keys = {"coke":"<COKE_KEY_ID>","pepsi":"<PEPSI_KEY_ID>"} rgw_crypt_require_ssl = false ``` Each tenant (e.g., Coke, Pepsi) has: * Its own OpenStack project * Separate EC2 credentials * Its own Barbican AES-256 key (with `rgw-barbican` added to the secret ACL for read) S3 operations work for both tenants using their own EC2 credentials. --- Observed Issue Cross-tenant Barbican key usage is allowed: | Tenant | Key Used | Result | | ----------------- | ---------- | -------------- | | Coke → Coke key | ✅ Success | expected | | Pepsi → Pepsi key | ✅ Success | expected | | Coke → Pepsi key | ⚠️ Success | unexpected | | Pepsi → Coke key | ⚠️ Success | unexpected | If a client supplies any valid Barbican key UUID (one that `rgw-barbican` can read via ACL), RGW accepts it—even if the key belongs to a different project. This suggests RGW does not validate that the Barbican key is owned by (or scoped to) the same project as the authenticated S3 user. --- Minimal Repro 1. Create projects `coke` and `pepsi`; create per-project EC2 creds. 2. Store one Barbican key per project; grant `rgw-barbican` read ACL on both secrets. 3. Configure RGW as above with the `rgw_crypt_s3_kms_encryption_keys` mapping. 4. Using Coke EC2 creds, run `put-object` with Pepsi key UUID via `--ssekms-key-id <PEPSI_KEY_ID>`. → Upload succeeds; encryption uses Pepsi key. --- What We Need * A way for RGW to validate that the SSE-KMS key used in the request belongs to the same OpenStack project (tenant) as the authenticated S3 user. * Cross-project use of KMS keys should fail with `AccessDenied` (or similar). --- Questions for Guidance 1. Is there a supported (or planned) way to restrict RGW so the Barbican key must belong to the same project as the authenticated S3 user? 2. Can RGW be configured to derive the project scope from the S3 auth context when requesting the key from Barbican (instead of always using the fixed `rgw_keystone_barbican_project` for the token)? --- Best Regards, Vignesh
Hello o/ Interesting find, this is somewhat complex and it’s not clear to me if this was intentional during the design or not. A Barbican secret is in fact owned by a Keystone user and not by the project, even if the secret is “placed in the project” nobody in the project actually has access to the secret except for the owner unless the project-access is set to True on the secret in Barbican. This is why a user with access to the project needs to create a Barbican ACL that allows the rgw_keystone_barbican_user configured user read access on the secret regardless of project. So while the secret in theory might be "in the project", it’s not sure a certain that the secret itself can be used by anybody in the project, which I assume also is somewhat of a flaw on the RGW implementation, as this was assumed during design(?). I do not have enough history or can’t find any specification when the feature was designed if any of this was already considered, perhaps Casey (cc) has history on the topic? So if we go by the route that the we wan’t to restrict – we need to take into account that, due to it’s long history and how this works now – any S3 consumer (keystone user accessing the S3 API) might be using a Barbican secret of a project it’s not a member of, as read acl on the secret has been shared to the rgw_keystone_barbican_user user. So we need to be thorough here as this, in it’s current form, will be a breaking change. I don’t know if you can actually retrieve the encrypted data from S3, and as such, be privy access to data encrypted by a secret and then use that for bruteforcing the secret itself, one potential attack vector I see here is that if you also have the Swift API enabled you can just download the object using the Swift API and it will be encrypted and thus get access to the encrypted blob of the object. /Tobias
On 17 Oct 2025, at 10:21, Vignesh Varma <vignesh.varma.g@stackbill.com> wrote:
Hi Team,
Lab Environment
* Ceph: 19.2.x (Squid) * OpenStack: Caracal * OS: Ubuntu 22.04 LTS * Deployment: `cephadm`
---
Setup Summary
We have a working Ceph RGW + Barbican integration with:
* Keystone auth: `rgw_s3_auth_use_keystone = true` * SSE-KMS backend: `rgw_crypt_s3_kms_backend = barbican` * Single RGW Barbican service user under a single project:
``` rgw_keystone_barbican_user = rgw-barbican rgw_keystone_barbican_project = service rgw_keystone_barbican_domain = default rgw_barbican_url = http://controller:9311 rgw_crypt_s3_kms_backend = barbican rgw_crypt_s3_kms_encryption_keys = {"coke":"<COKE_KEY_ID>","pepsi":"<PEPSI_KEY_ID>"} rgw_crypt_require_ssl = false ```
Each tenant (e.g., Coke, Pepsi) has:
* Its own OpenStack project * Separate EC2 credentials * Its own Barbican AES-256 key (with `rgw-barbican` added to the secret ACL for read)
S3 operations work for both tenants using their own EC2 credentials.
---
Observed Issue
Cross-tenant Barbican key usage is allowed:
| Tenant | Key Used | Result | | ----------------- | ---------- | -------------- | | Coke → Coke key | ✅ Success | expected | | Pepsi → Pepsi key | ✅ Success | expected | | Coke → Pepsi key | ⚠️ Success | unexpected | | Pepsi → Coke key | ⚠️ Success | unexpected |
If a client supplies any valid Barbican key UUID (one that `rgw-barbican` can read via ACL), RGW accepts it—even if the key belongs to a different project. This suggests RGW does not validate that the Barbican key is owned by (or scoped to) the same project as the authenticated S3 user.
---
Minimal Repro
1. Create projects `coke` and `pepsi`; create per-project EC2 creds. 2. Store one Barbican key per project; grant `rgw-barbican` read ACL on both secrets. 3. Configure RGW as above with the `rgw_crypt_s3_kms_encryption_keys` mapping. 4. Using Coke EC2 creds, run `put-object` with Pepsi key UUID via `--ssekms-key-id <PEPSI_KEY_ID>`. → Upload succeeds; encryption uses Pepsi key.
---
What We Need
* A way for RGW to validate that the SSE-KMS key used in the request belongs to the same OpenStack project (tenant) as the authenticated S3 user. * Cross-project use of KMS keys should fail with `AccessDenied` (or similar).
---
Questions for Guidance
1. Is there a supported (or planned) way to restrict RGW so the Barbican key must belong to the same project as the authenticated S3 user? 2. Can RGW be configured to derive the project scope from the S3 auth context when requesting the key from Barbican (instead of always using the fixed `rgw_keystone_barbican_project` for the token)?
---
Best Regards, Vignesh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi There, This effectively means that the final layer of isolation currently depends on the OpenStack project scope. To strengthen this boundary, I would like to suggest an enhancement on the Ceph side: introducing an optional configuration flag, for example rgw_crypt_s3_kms_verify_project = true. When enabled, RGW would validate that the Barbican secret's project_id matches the authenticated S3 user's project before performing encryption or decryption. This would prevent cross tenant key usage while remaining backward compatible for deployments that deliberately share a Barbican service user. In cases where the project information cannot be determined, RGW should fail safely and deny the request. I would be interested to hear other perspectives on this proposal. Best, _ Senol On 2025-10-17 10:57, Tobias Urdin - Binero wrote:
Hello o/
Interesting find, this is somewhat complex and it's not clear to me if this was intentional during the design or not.
A Barbican secret is in fact owned by a Keystone user and not by the project, even if the secret is "placed in the project" nobody in the project actually has access to the secret except for the owner unless the project-access is set to True on the secret in Barbican.
This is why a user with access to the project needs to create a Barbican ACL that allows the rgw_keystone_barbican_user configured user read access on the secret regardless of project.
So while the secret in theory might be "in the project", it's not sure a certain that the secret itself can be used by anybody in the project, which I assume also is somewhat of a flaw on the RGW implementation, as this was assumed during design(?).
I do not have enough history or can't find any specification when the feature was designed if any of this was already considered, perhaps Casey (cc) has history on the topic?
So if we go by the route that the we wan't to restrict - we need to take into account that, due to it's long history and how this works now - any S3 consumer (keystone user accessing the S3 API) might be using a Barbican secret of a project it's not a member of, as read acl on the secret has been shared to the rgw_keystone_barbican_user user.
So we need to be thorough here as this, in it's current form, will be a breaking change.
I don't know if you can actually retrieve the encrypted data from S3, and as such, be privy access to data encrypted by a secret and then use that for bruteforcing the secret itself, one potential attack vector I see here is that if you also have the Swift API enabled you can just download the object using the Swift API and it will be encrypted and thus get access to the encrypted blob of the object.
/Tobias
On 17 Oct 2025, at 10:21, Vignesh Varma <vignesh.varma.g@stackbill.com> wrote:
Hi Team,
Lab Environment
* Ceph: 19.2.x (Squid) * OpenStack: Caracal * OS: Ubuntu 22.04 LTS * Deployment: `cephadm`
---
Setup Summary
We have a working Ceph RGW + Barbican integration with:
* Keystone auth: `rgw_s3_auth_use_keystone = true` * SSE-KMS backend: `rgw_crypt_s3_kms_backend = barbican` * Single RGW Barbican service user under a single project:
``` rgw_keystone_barbican_user = rgw-barbican rgw_keystone_barbican_project = service rgw_keystone_barbican_domain = default rgw_barbican_url = http://controller:9311 rgw_crypt_s3_kms_backend = barbican rgw_crypt_s3_kms_encryption_keys = {"coke":"<COKE_KEY_ID>","pepsi":"<PEPSI_KEY_ID>"} rgw_crypt_require_ssl = false ```
Each tenant (e.g., Coke, Pepsi) has:
* Its own OpenStack project * Separate EC2 credentials * Its own Barbican AES-256 key (with `rgw-barbican` added to the secret ACL for read)
S3 operations work for both tenants using their own EC2 credentials.
---
Observed Issue
Cross-tenant Barbican key usage is allowed:
| Tenant | Key Used | Result | | ----------------- | ---------- | -------------- | | Coke → Coke key | ✅ Success | expected | | Pepsi → Pepsi key | ✅ Success | expected | | Coke → Pepsi key | ⚠️ Success | unexpected | | Pepsi → Coke key | ⚠️ Success | unexpected |
If a client supplies any valid Barbican key UUID (one that `rgw-barbican` can read via ACL), RGW accepts it--even if the key belongs to a different project. This suggests RGW does not validate that the Barbican key is owned by (or scoped to) the same project as the authenticated S3 user.
---
Minimal Repro
1. Create projects `coke` and `pepsi`; create per-project EC2 creds. 2. Store one Barbican key per project; grant `rgw-barbican` read ACL on both secrets. 3. Configure RGW as above with the `rgw_crypt_s3_kms_encryption_keys` mapping. 4. Using Coke EC2 creds, run `put-object` with Pepsi key UUID via `--ssekms-key-id <PEPSI_KEY_ID>`. → Upload succeeds; encryption uses Pepsi key.
---
What We Need
* A way for RGW to validate that the SSE-KMS key used in the request belongs to the same OpenStack project (tenant) as the authenticated S3 user. * Cross-project use of KMS keys should fail with `AccessDenied` (or similar).
---
Questions for Guidance
1. Is there a supported (or planned) way to restrict RGW so the Barbican key must belong to the same project as the authenticated S3 user? 2. Can RGW be configured to derive the project scope from the S3 auth context when requesting the key from Barbican (instead of always using the fixed `rgw_keystone_barbican_project` for the token)?
---
Best Regards, Vignesh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Senol and Tobias, Thank you for the detailed response and for outlining the proposed enhancement. The suggestion to introduce a configuration flag such as: ``` rgw_crypt_s3_kms_verify_project = true ``` is exactly what’s needed to close the current gap in **cross-tenant key isolation** for RGW–Barbican integrations. As confirmed during our lab testing, RGW currently relies solely on the OpenStack project scope for isolation and does not verify whether the **Barbican secret’s project_id** matches the authenticated **S3 user’s project**. This allows cross-project key usage when a shared Barbican service user has read access to multiple secrets — a scenario that poses a clear multi-tenant boundary risk in production environments. We would like to **formally request that this enhancement be considered for integration** into Ceph’s upcoming release. Implementing this verification logic would ensure: * Stronger tenant isolation at the RGW layer. * Safe and predictable key usage in multi-project deployments. * Backward compatibility for existing shared Barbican setups. Appreciate the community’s support and attention to this critical security improvement. Best Regards, Vignesh Varma G ----- Original Message ----- From: senol@kubedo.com To: "ceph-users" <ceph-users@ceph.io> Sent: Monday, October 20, 2025 10:05:59 PM Subject: [ceph-users] Re: Ceph RGW Barbican Integration — Tenant isolation issue (Barbican keys from one project usable by another) Hi There, This effectively means that the final layer of isolation currently depends on the OpenStack project scope. To strengthen this boundary, I would like to suggest an enhancement on the Ceph side: introducing an optional configuration flag, for example rgw_crypt_s3_kms_verify_project = true. When enabled, RGW would validate that the Barbican secret's project_id matches the authenticated S3 user's project before performing encryption or decryption. This would prevent cross tenant key usage while remaining backward compatible for deployments that deliberately share a Barbican service user. In cases where the project information cannot be determined, RGW should fail safely and deny the request. I would be interested to hear other perspectives on this proposal. Best, _ Senol On 2025-10-17 10:57, Tobias Urdin - Binero wrote:
Hello o/
Interesting find, this is somewhat complex and it's not clear to me if this was intentional during the design or not.
A Barbican secret is in fact owned by a Keystone user and not by the project, even if the secret is "placed in the project" nobody in the project actually has access to the secret except for the owner unless the project-access is set to True on the secret in Barbican.
This is why a user with access to the project needs to create a Barbican ACL that allows the rgw_keystone_barbican_user configured user read access on the secret regardless of project.
So while the secret in theory might be "in the project", it's not sure a certain that the secret itself can be used by anybody in the project, which I assume also is somewhat of a flaw on the RGW implementation, as this was assumed during design(?).
I do not have enough history or can't find any specification when the feature was designed if any of this was already considered, perhaps Casey (cc) has history on the topic?
So if we go by the route that the we wan't to restrict - we need to take into account that, due to it's long history and how this works now - any S3 consumer (keystone user accessing the S3 API) might be using a Barbican secret of a project it's not a member of, as read acl on the secret has been shared to the rgw_keystone_barbican_user user.
So we need to be thorough here as this, in it's current form, will be a breaking change.
I don't know if you can actually retrieve the encrypted data from S3, and as such, be privy access to data encrypted by a secret and then use that for bruteforcing the secret itself, one potential attack vector I see here is that if you also have the Swift API enabled you can just download the object using the Swift API and it will be encrypted and thus get access to the encrypted blob of the object.
/Tobias
On 17 Oct 2025, at 10:21, Vignesh Varma <vignesh.varma.g@stackbill.com> wrote:
Hi Team,
Lab Environment
* Ceph: 19.2.x (Squid) * OpenStack: Caracal * OS: Ubuntu 22.04 LTS * Deployment: `cephadm`
---
Setup Summary
We have a working Ceph RGW + Barbican integration with:
* Keystone auth: `rgw_s3_auth_use_keystone = true` * SSE-KMS backend: `rgw_crypt_s3_kms_backend = barbican` * Single RGW Barbican service user under a single project:
``` rgw_keystone_barbican_user = rgw-barbican rgw_keystone_barbican_project = service rgw_keystone_barbican_domain = default rgw_barbican_url = http://controller:9311 rgw_crypt_s3_kms_backend = barbican rgw_crypt_s3_kms_encryption_keys = {"coke":"<COKE_KEY_ID>","pepsi":"<PEPSI_KEY_ID>"} rgw_crypt_require_ssl = false ```
Each tenant (e.g., Coke, Pepsi) has:
* Its own OpenStack project * Separate EC2 credentials * Its own Barbican AES-256 key (with `rgw-barbican` added to the secret ACL for read)
S3 operations work for both tenants using their own EC2 credentials.
---
Observed Issue
Cross-tenant Barbican key usage is allowed:
| Tenant | Key Used | Result | | ----------------- | ---------- | -------------- | | Coke → Coke key | ✅ Success | expected | | Pepsi → Pepsi key | ✅ Success | expected | | Coke → Pepsi key | ⚠️ Success | unexpected | | Pepsi → Coke key | ⚠️ Success | unexpected |
If a client supplies any valid Barbican key UUID (one that `rgw-barbican` can read via ACL), RGW accepts it--even if the key belongs to a different project. This suggests RGW does not validate that the Barbican key is owned by (or scoped to) the same project as the authenticated S3 user.
---
Minimal Repro
1. Create projects `coke` and `pepsi`; create per-project EC2 creds. 2. Store one Barbican key per project; grant `rgw-barbican` read ACL on both secrets. 3. Configure RGW as above with the `rgw_crypt_s3_kms_encryption_keys` mapping. 4. Using Coke EC2 creds, run `put-object` with Pepsi key UUID via `--ssekms-key-id <PEPSI_KEY_ID>`. → Upload succeeds; encryption uses Pepsi key.
---
What We Need
* A way for RGW to validate that the SSE-KMS key used in the request belongs to the same OpenStack project (tenant) as the authenticated S3 user. * Cross-project use of KMS keys should fail with `AccessDenied` (or similar).
---
Questions for Guidance
1. Is there a supported (or planned) way to restrict RGW so the Barbican key must belong to the same project as the authenticated S3 user? 2. Can RGW be configured to derive the project scope from the S3 auth context when requesting the key from Barbican (instead of always using the fixed `rgw_keystone_barbican_project` for the token)?
---
Best Regards, Vignesh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
senol＠kubedo.com
-
Tobias Urdin - Binero
-
Vignesh Varma