This is a common error on my system (Pacific). It appears that there is internal confusion as to where the crash support stuff lives - whether it's new-style (administered and under /var/lib/ceph/fsid) or legacy style (/var/lib/ceph). One way to fake it out was to manually created a minimal crash infrastructure (keyring) under /var/lib/ceph/crash and assign appropriate access rights. Tim On 10/29/24 10:41, mailing-lists wrote:
Hey Cephers,
i was investigating some other issue, when I stumbled across this. I am not sure, if this is "as intended" or faulty. This is a cephadm cluster on reef 18.2.4, containerized with docker.
The ceph-crash module states that it cant find its key and that it cant access RADOS.
Pre-face, this is how its key looks like and what its caps are (dont worry about leaking the key, its a Test-Cluster): client.crash.bi-ubu-srv-ceph2-01 key: AQBi5CBnedEKORAAgaTwkiqO7KJ+wzu8+EGXEQ== caps: [mgr] profile crash caps: [mon] profile crash
Looks like it should. As per documentation: https://docs.ceph.com/en/reef/mgr/crash/#automated-collection
The key on the node/directory of this ceph-crash container is the same.
This is the docker logs of the container running ceph-crash.
INFO:ceph-crash:pinging cluster to exercise our key 2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 AuthRegistry(0x7f7ab4067810) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 AuthRegistry(0x7f7abb1ae0c0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-10-29T13:34:28.265+0000 7f7ab99ac640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7aba1ad640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7aba9ae640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 13] RADOS permission denied (error connecting to the cluster) INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
As we can see here, it says that it couldn't find its key. After some online research, I found that some names are not good/interpreted correctly. Well, just to test, i've changed the unit.run file of ceph-crash-container and restarted it.
This is the part I've changed:
From
/var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/ceph.client.crash.bi-ubu-srv-ceph2-01.keyring
To
/var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/keyring
Then the logs look different, but the permissions denied error is still there.
INFO:ceph-crash:pinging cluster to exercise our key 2024-10-29T13:36:42.273+0000 7f1d04b49640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2024-10-29T13:36:42.277+0000 7f1cff7fe640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2024-10-29T13:36:42.281+0000 7f1cfffff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
Can anybody tell me if this is "normal"?
I think this is suspicious. Because for example, if i do ceph crash ls, it does not yield anything, while there is definitely something on a node inside the folder /var/lib/ceph/xyz/crash/
Thanks in advance and best wishes!
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io