Hi, thanks for your feedback! Eugen Block via ceph-users <ceph-users@ceph.io> writes:
that sounds a bit impractical to create a new pool for each tenant regarding PG and pool management (adding and possibly removing data pools from the CephFS), depending on how many tenants you have or plan to have.
In our setup each K8S cluster would be on tenant. So the pools would be on the subvolumegroup level and we expect to go ahead with less than ten pools for the foreseeable future.
What I just tested was to manually encrypt a directory in a test cluster with fscrypt, that seems to work as intended. But I don't have anything with k8s to test it at the moment.
From what I have tested, the problem seems to be located in ceph-csi where the encryption is initially set up. The code tries to allocates an exclusive lock on an object representing the volume to prevent a race condition with multiple clients sharing the volume. The relevant code looks like this: ioctx, err := volOptions.GetConnection().GetIoctx(volOptions.MetadataPool) if err != nil { log.ErrorLog(ctx, "Failed to create ioctx: %s", err) return err } lock := iolock.NewLock(ioctx, objectUUID, lockName, lockCookie, lockDesc, lockDuration) err = lock.LockExclusive(ctx) if err != nil { log.ErrorLog(ctx, "failed to create lock for volume ID %s: %v", volID, err) return err } The error message on the ceph/csi pods clearly shows that the LockExclusive call fails with “Operation not permitted”. I failed to find the place in the relevant code, where the namespace for this operation is set, so it looks like the default namespace is used. But the capabilities only allow a specific namespace and so this could be the root cause. I tried to use the following code to create an object in the metadata pool and lock it. I used the same user with the same capabilities: % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put foobar /dev/null % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock get foobar lock1 --lock-cookie=cookie --lock-duration=60 --lock-type=exclusive % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test lock info foobar lock1 {"name":"lock1","type":"exclusive","tag":"","lockers":[{"name":"client.2690590","cookie":"cookie","description":"","expiration":"2026-07-31T08:21:36.190835+0200","addr":"172.21.70.10:0/2292729948"}]} So the capabilities seem to be sufficient for the lock operation. But leaving out the namespace when creating the lock will show the exact same error message that the ceph-csi shows: % rados --id=k8s-test --pool=cephfs.meta --namespace=k8s-test put foobar /dev/null % rados --id=k8s-test --pool=cephfs.meta lock get foobar lock1 --lock-cookie=cookie --lock-duration=60 --lock-type=exclusive ERROR: failed locking: (1) Operation not permitted error 1: (1) Operation not permitted -- Stefan