Hello, recently I checked my server logs for error events and got some hits for the ceph crash service, deployed with cephadm. When restarted, crash service logs to journald: auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory AuthRegistry(0x7f21a0068da0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory AuthRegistry(0x7f21a861bff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 13] RADOS permission denied (error connecting to the cluster) The image is mounted with: -v /var/lib/ceph/7644057a-00f6-11f0-9a0c-eac00fed9338/crash.virt-master3/keyring:/etc/ceph/ceph.client.crash.virt-master3.keyring so I assume there should be a key available, but crash daemon searches for a admin keyring instead of "client.crash.virt-master3". Any advice how to fix this error? I see these error logs on all machines where crash service is deployed. I've tried a redeployment without any effect. # ceph -v ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable) # systemctl restart ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service # journalctl -u ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service -ef Output: Apr 12 18:36:24 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1402272]: *** Interrupted with signal 15 *** Apr 12 18:36:24 virt-master3 podman[1408122]: 2025-04-12 18:36:24.992684396 +0200 CEST m=+0.043782773 container died 6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_REF=squid, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, io.buildah.version=1.33.7, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20250124, OSD_FLAVOR=default) Apr 12 18:36:25 virt-master3 podman[1408122]: 2025-04-12 18:36:25.016797163 +0200 CEST m=+0.067895534 container remove 6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20250124, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default, org.label-schema.license=GPLv2, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>) Apr 12 18:36:25 virt-master3 bash[1408104]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3 Apr 12 18:36:25 virt-master3 systemd[1]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service: Deactivated successfully. Apr 12 18:36:25 virt-master3 systemd[1]: Stopped ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338. Apr 12 18:36:25 virt-master3 systemd[1]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service: Consumed 1.540s CPU time. Apr 12 18:36:25 virt-master3 systemd[1]: Starting ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338... Apr 12 18:36:25 virt-master3 podman[1408259]: Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.498991532 +0200 CEST m=+0.071935136 container create bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.build-date=20250124, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>) Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.462816456 +0200 CEST m=+0.035760071 image pull quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.589479015 +0200 CEST m=+0.162422619 container init bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.build-date=20250124, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=squid, io.buildah.version=1.33.7) Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.595018205 +0200 CEST m=+0.167961840 container start bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20250124, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.vendor=CentOS, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) Apr 12 18:36:25 virt-master3 bash[1408259]: bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 Apr 12 18:36:25 virt-master3 systemd[1]: Started ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338. Apr 12 18:36:25 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: INFO:ceph-crash:pinging cluster to exercise our key Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 AuthRegistry(0x7f21a0068da0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 AuthRegistry(0x7f21a861bff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a5b91640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a6392640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: [errno 13] RADOS permission denied (error connecting to the cluster) Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher