Hey Cephers, i was investigating some other issue, when I stumbled across this. I am not sure, if this is "as intended" or faulty. This is a cephadm cluster on reef 18.2.4, containerized with docker. The ceph-crash module states that it cant find its key and that it cant access RADOS. Pre-face, this is how its key looks like and what its caps are (dont worry about leaking the key, its a Test-Cluster): client.crash.bi-ubu-srv-ceph2-01 key: AQBi5CBnedEKORAAgaTwkiqO7KJ+wzu8+EGXEQ== caps: [mgr] profile crash caps: [mon] profile crash Looks like it should. As per documentation: https://docs.ceph.com/en/reef/mgr/crash/#automated-collection The key on the node/directory of this ceph-crash container is the same. This is the docker logs of the container running ceph-crash. INFO:ceph-crash:pinging cluster to exercise our key 2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 AuthRegistry(0x7f7ab4067810) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 AuthRegistry(0x7f7abb1ae0c0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-10-29T13:34:28.265+0000 7f7ab99ac640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7aba1ad640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7aba9ae640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 13] RADOS permission denied (error connecting to the cluster) INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s As we can see here, it says that it couldn't find its key. After some online research, I found that some names are not good/interpreted correctly. Well, just to test, i've changed the unit.run file of ceph-crash-container and restarted it. This is the part I've changed: From /var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/ceph.client.crash.bi-ubu-srv-ceph2-01.keyring To /var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/keyring Then the logs look different, but the permissions denied error is still there. INFO:ceph-crash:pinging cluster to exercise our key 2024-10-29T13:36:42.273+0000 7f1d04b49640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2024-10-29T13:36:42.277+0000 7f1cff7fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2024-10-29T13:36:42.281+0000 7f1cfffff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s Can anybody tell me if this is "normal"? I think this is suspicious. Because for example, if i do ceph crash ls, it does not yield anything, while there is definitely something on a node inside the folder /var/lib/ceph/xyz/crash/ Thanks in advance and best wishes!