I ran into this same puzzling behavior and never resolved it. I *think* it is a benign bug, that can be ignored. Here's what I found. The crash service first attempts to ping the cluster to exercise the key. "pinging cluster" is accomplished with a `ceph -s`. # v18.2.4:src/ceph-crash.in 109 def main(): 110 global auth_names 111 112 # run as unprivileged ceph user 113 drop_privs() 114 115 # exit code 0 on SIGINT, SIGTERM 116 signal.signal(signal.SIGINT, handler) 117 signal.signal(signal.SIGTERM, handler) 118 119 args = parse_args() 120 if args.log_level == 'DEBUG': 121 log.setLevel(logging.DEBUG) 122 123 postdir = os.path.join(args.path, 'posted') 124 if args.name: 125 auth_names = [args.name] 126 127 while not os.path.isdir(postdir): 128 log.error("directory %s does not exist; please create" % postdir) 129 time.sleep(30) 130 131 log.info("pinging cluster to exercise our key") 132 pr = subprocess.Popen(args=['timeout', '30', 'ceph', '-s']) 133 pr.wait() That part seems to be broken. Notice it drops privileges, to ceph:ceph. Which the eventual intent of processing the crash report to crash/posted. (global auth_names below). I think, what is happening, is when the attempt to process a crash report fails, because of directory ownership, it then tries to use the client.admin key to see if the cluster is even up. That key cant be found, and so you get the errors. (my best guess) Note the global auth_names. # v18.2.4:src/ceph-crash.in 19 auth_names = ['client.crash.%s' % socket.gethostname(), 20 'client.crash', 21 'client.admin'] 22 23 24 def parse_args(): 25 parser = argparse.ArgumentParser() 26 parser.add_argument( 27 '-p', '--path', default='/var/lib/ceph/crash', 28 help='base path to monitor for crash dumps') 29 parser.add_argument( 30 '-d', '--delay', default=10.0, type=float, 31 help='minutes to delay between scans (0 to exit after one)', 32 ) 33 parser.add_argument( 34 '--name', '-n', 35 help='ceph name to authenticate as ' 36 '(default: try client.crash, client.admin)') 37 parser.add_argument( 38 '--log-level', '-l', 39 help='log level output (default: INFO), support INFO or DEBUG') 40 41 return parser.parse_args() I came across this when we had a soft crash of a ceph node which resulted in an incomplete crash directory being created. Once we removed that incomplete directory /var/lib/ceph/<fsid>/crash/<incomplete crash directory here> The errors went away. In our situation, there were other complete crash directories which needed to be processed. But when it ran into the incomplete directory it just looped the error youre seeing. Once we deleted the incomplete crash directory the complete directories were processed. Not sure if the same problem exists for you. It looks like there was discussion about it in the past[1], and a tracker report filed[2], but nothing follows. You can also use TOP to see if there is a zombie process running as USER: ceph, with PPID running ceph-crash. If so, I think you might be in the situation below. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/5JYWVOK3NFGX... [2] https://tracker.ceph.com/issues/64102 There is a related problem (I believe fixed in v18.2.5), concerning directory ownership for the crash directories. For whatever reason, whenever a non-ceph daemon is deployed (e.g., alertmanager, prometheus, et al.), cephadm is hard coded to first set the crash directory ownership according to the daemon being deployed. So, if the altertmanager daemon runs as nobody:nobody, then when cephadm deploys alertmanager if first goes in and changes the crash directory ownership to nobody:nobody. For whatever reason, it does this always, whenever a non-ceph daemon is being deployed (e.g., all the monitoring type daemons). This can be very frustrating. But the simple fix is to just manually set those directories back to ceph:ceph. And remove any incomplete crash reports. If you don't, the crash service is effectively broken after deploying any non-ceph daemons. To test, node-exporter is a good one, because it touches every host. Or, you could try just grafana or prometheus, to limit the work in fixing it again. # ceph-18.2.4/src/cephadm/cephadm.py 2775 def make_data_dir_base(fsid, data_dir, uid, gid): 2776 # type: (str, str, int, int) -> str 2777 data_dir_base = os.path.join(data_dir, fsid) 2778 makedirs(data_dir_base, uid, gid, DATA_DIR_MODE) 2779 makedirs(os.path.join(data_dir_base, 'crash'), uid, gid, DATA_DIR_MODE) 2780 makedirs(os.path.join(data_dir_base, 'crash', 'posted'), uid, gid, 2781 DATA_DIR_MODE) 2782 return data_dir_base We confirmed this theory by hardcoding the uid, gid, to 167, 167 (ceph:ceph) in a modified cephadm, and then redeployed node-exporter. The crash directories didnt get changed. This was a slightly more elegant solution, which proved our theory, but then for whatever reason our ganesha daemons wouldnt start. 2775 def make_data_dir_base(fsid, data_dir, daemon_type, uid, gid): 2776 # type: (str, str, int, int) -> str 2777 data_dir_base = os.path.join(data_dir, fsid) 2778 makedirs(data_dir_base, uid, gid, DATA_DIR_MODE) 2779 if daemon_type not in Monitoring.components.keys(): 2780 makedirs(os.path.join(data_dir_base, 'crash'), uid, gid, DATA_DIR_MODE) 2781 makedirs(os.path.join(data_dir_base, 'crash', 'posted'), uid, gid, 2782 DATA_DIR_MODE) 2783 return data_dir_base 2784 2785 2786 def make_data_dir(ctx, fsid, daemon_type, daemon_id, uid=None, gid=None): 2787 # type: (CephadmContext, str, str, Union[int, str], Optional[int], Optional[int]) -> str 2788 if uid is None or gid is None: 2789 uid, gid = extract_uid_gid(ctx) 2790 make_data_dir_base(fsid, ctx.data_dir, daemon_type, uid, gid) 2791 data_dir = get_data_dir(fsid, ctx.data_dir, daemon_type, daemon_id) 2792 makedirs(data_dir, uid, gid, DATA_DIR_MODE) 2793 return data_dir We gave up at that point, and noticed there might be a fix in the latest v18.2.5 released a few days ago. https://github.com/ceph/ceph/pull/58458 Oddly, it is listed as a "debian" fix, I hope it is just a general fix for the crash directory ownership issue. -Robert On 4/13/25 12:16, Daniel Vogelbacher wrote:
Hello,
recently I checked my server logs for error events and got some hits for the ceph crash service, deployed with cephadm.
When restarted, crash service logs to journald:
auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/ etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory AuthRegistry(0x7f21a0068da0) no keyring found at /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ ceph/keyring.bin, disabling cephx auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/ etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory AuthRegistry(0x7f21a861bff0) no keyring found at /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ ceph/keyring.bin, disabling cephx monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] monclient: authenticate NOTE: no keyring found; disabled cephx authentication [errno 13] RADOS permission denied (error connecting to the cluster)
The image is mounted with:
-v /var/lib/ceph/7644057a-00f6-11f0-9a0c-eac00fed9338/crash.virt- master3/keyring:/etc/ceph/ceph.client.crash.virt-master3.keyring
so I assume there should be a key available, but crash daemon searches for a admin keyring instead of "client.crash.virt-master3".
Any advice how to fix this error? I see these error logs on all machines where crash service is deployed. I've tried a redeployment without any effect.
# ceph -v ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)
# systemctl restart ceph-7644057a-00f6-11f0-9a0c- eac00fed9338@crash.virt-master3.service
# journalctl -u ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt- master3.service -ef
Output:
Apr 12 18:36:24 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1402272]: *** Interrupted with signal 15 *** Apr 12 18:36:24 virt-master3 podman[1408122]: 2025-04-12 18:36:24.992684396 +0200 CEST m=+0.043782773 container died 6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b (image=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_REF=squid, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, io.buildah.version=1.33.7, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20250124, OSD_FLAVOR=default) Apr 12 18:36:25 virt-master3 podman[1408122]: 2025-04-12 18:36:25.016797163 +0200 CEST m=+0.067895534 container remove 6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b (image=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20250124, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default, org.label-schema.license=GPLv2, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>) Apr 12 18:36:25 virt-master3 bash[1408104]: ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3 Apr 12 18:36:25 virt-master3 systemd[1]: ceph-7644057a-00f6-11f0-9a0c- eac00fed9338@crash.virt-master3.service: Deactivated successfully. Apr 12 18:36:25 virt-master3 systemd[1]: Stopped ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338. Apr 12 18:36:25 virt-master3 systemd[1]: ceph-7644057a-00f6-11f0-9a0c- eac00fed9338@crash.virt-master3.service: Consumed 1.540s CPU time. Apr 12 18:36:25 virt-master3 systemd[1]: Starting ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338... Apr 12 18:36:25 virt-master3 podman[1408259]: Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.498991532 +0200 CEST m=+0.071935136 container create bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.build-date=20250124, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>) Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.462816456 +0200 CEST m=+0.035760071 image pull quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.589479015 +0200 CEST m=+0.162422619 container init bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.build-date=20250124, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=squid, io.buildah.version=1.33.7) Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12 18:36:25.595018205 +0200 CEST m=+0.167961840 container start bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 (image=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de, name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20250124, io.buildah.version=1.33.7, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.vendor=CentOS, CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) Apr 12 18:36:25 virt-master3 bash[1408259]: bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63 Apr 12 18:36:25 virt-master3 systemd[1]: Started ceph-7644057a-00f6-11f0-9a0c-eac00fed9338@crash.virt-master3.service - Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338. Apr 12 18:36:25 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: INFO:ceph-crash:pinging cluster to exercise our key Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a keyring on /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ ceph/keyring.bin: (2) No such file or directory Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 AuthRegistry(0x7f21a0068da0) no keyring found at /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ ceph/keyring.bin, disabling cephx Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a keyring on /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ ceph/keyring.bin: (2) No such file or directory Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 AuthRegistry(0x7f21a861bff0) no keyring found at /etc/ceph/ ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ ceph/keyring.bin, disabling cephx Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a5b91640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a6392640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: 2025-04-12T16:36:26.131+0000 7f21a861d640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: [errno 13] RADOS permission denied (error connecting to the cluster) Apr 12 18:36:26 virt-master3 ceph-7644057a-00f6-11f0-9a0c-eac00fed9338- crash-virt-master3[1408301]: INFO:ceph-crash:monitoring path /var/lib/ ceph/crash, delay 600s