Hi Frédéric I haven't tested anything yet. On the ceph slack it was suggested that the overhead would scale as O(log(n)), and the suggestion was other things (e.g. cluster maps) would dominate (which I feel means it's not a major source of worry). The use-case(s) are mainly focused on cephfs and giving outside-of-cluster clients (i.e. standard linux clients of various distros and versions, but also k8s clusters with pods that may or may not need to see specific subtrees, with some needing read/write and others which only need read-only) specific access to only what they need (with unix permissions/acls mixed in as needed). While we could give broader permissions and rely on mounts being configured correctly (e.g. only mounting specific subpaths which are going to vary across the clients), it would seem safer to be more precise with our cephx setup, which then implies not being able to reuse the same key across clients (which we probably wouldn't want to do anyway because we'd likely not have full control over all clients). Regards James ________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Monday, 9 June 2025 8:21 PM To: James Tocknell Cc: ceph-users Subject: [ceph-users] Re: Limits on the number of cephx keys used Hi James, I don't recall reading anything related to performance degradation due to an excessive number of keyrings generated in a Ceph cluster. Could you elaborate on your use case? E.g. why you would have so many clients not able to share a common keyring. FWIW, the orchestrator can take care of pushing ceph.conf and keyrings on clients based on a host label [1]. This might help your case. Best regards, Frédéric. [1] https://docs.ceph.com/en/latest/cephadm/operations/#client-keyrings-and-configs<https://docs.ceph.com/en/latest/cephadm/operations/#client-keyrings-and-configs> ----- Le 6 Juin 25, à 5:28, James Tocknell james.tocknell@mq.edu.au a écrit :

Hi All

As far as I can see, there is no guidance on the number of cephx keys that can be in use at one time. Is there a number at which ceph becomes much slower e.g. 100, 10000, 1000000? I'm wondering how best to manage keys across many clients (let's say 1000s for now), most of which won't actually be connected at the same time.

Regards James _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Den tis 10 juni 2025 kl 19:39 skrev Gregory Farnum <gfarnum@redhat.com>:

I'm sure at some scale on some hardware it is possible to run into bottlenecks, but no reported issues with scaling come to mind.

CephX keys are durably stored in the monitor's RocksDB instance, which it uses to store all of its data. This scales well but not infinitely, but I don't think we've run into monitor issues with scaling after some early teething issues and switching from LevelDB to RocksDB.

Also, its a key-value store right? So you would normally do lookups based on a known key, and you get the value (ie, password+caps) back. In this case it would not matter much if it was a 1000 keys or a million, it would do the lookups really fast in both cases. Only commands like "ceph auth list" would have to traverse it from 0 .. 1000. So while the keys (with pw and caps) might eat a kB or two of ram, I would not guess there are practical limits on size, unless you do crazy numbers of keys, and the first issue you get might be mon RAM running out or adding an entry to the list takes a long while, more than "auth takes too long for client.something6789".

If there is an issue, it's definitely closer to 1 million keys than to 1 thousand.

Yeah. -- May the most significant bit of your life be positive.