Dear all Today I realized that the syslog on the mon/mgr modules was complaining about missing credentials for the crash client So. following what is reported in: https://docs.ceph.com/en/quincy/mgr/crash/ I did: root@ceph-mon-01 ~]# ceph auth get-or-create client.crash mon 'profile crash' mgr 'profile crash' [client.crash] key = AQAqNKtmny3yNRAALoVRCgPju2Epc+BPfqOSdw== and on the 3 mon/mgr nodes I created the file /etc/ceph/ceph.client.crash.keyring root@ceph-mon-01 ~]# ll /etc/ceph/ceph.client.crash.keyring -rw-r--r-- 1 ceph ceph 64 Aug 1 09:21 /etc/ceph/ceph.client.crash.keyring [root@ceph-mon-01 ~]# cat /etc/ceph/ceph.client.crash.keyring [client.crash] key = AQAqNKtmny3yNRAALoVRCgPju2Epc+BPfqOSdw== But it is still complaining in the syslog [*] Do I need to also create a client.crash.$hostname ? I have to admit that the doc is not fully clear to me .... At any rate now ceph shows: [root@ceph-mon-01 ~]# ceph health detail HEALTH_WARN 8 mgr modules have recently crashed [WRN] RECENT_MGR_MODULE_CRASH: 8 mgr modules have recently crashed mgr module k8sevents crashed in daemon mgr.ceph-mon-01 on host ceph-mon-01.cloud.pd.infn.it at 2024-07-23T08:53:52.543451Z mgr module k8sevents crashed in daemon mgr.ceph-mon-01 on host ceph-mon-01.cloud.pd.infn.it at 2024-07-23T09:09:17.703402Z mgr module k8sevents crashed in daemon mgr.ceph-mon-02 on host ceph-mon-02.cloud.pd.infn.it at 2024-07-22T13:44:18.779525Z mgr module k8sevents crashed in daemon mgr.ceph-mon-02 on host ceph-mon-02.cloud.pd.infn.it at 2024-07-22T13:57:24.207309Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-07-22T10:42:42.002880Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-07-22T11:09:21.639576Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-08-01T06:02:01.139099Z mgr module k8sevents crashed in daemon mgr.ceph-mon-03 on host ceph-mon-03.cloud.pd.infn.it at 2024-08-01T07:45:16.249664Z but that module is disabled [**]. So I am a bit confused Thanks, Massimo [**] [root@ceph-mon-01 ~]# ceph mgr module ls MODULE balancer on (always on) crash on (always on) devicehealth on (always on) orchestrator on (always on) pg_autoscaler on (always on) progress on (always on) rbd_support on (always on) status on (always on) telemetry on (always on) volumes on (always on) dashboard on restful on alerts - cephadm - influx - insights - iostat - k8sevents - localpool - mds_autoscaler - mirroring - nfs - osd_perf_query - osd_support - prometheus - rgw - rook - selftest - snap_schedule - stats - telegraf - test_orchestrator - zabbix - [root@ceph-mon-01 ~]# ceph mgr module disable k8sevents module 'k8sevents' is already disabled [root@ceph-mon-01 ~]# [*] Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: WARNING:ceph-crash:post /var/lib/ceph/crash/2024-07-23T08:53:52.543451Z_da7be91d-5025-4bd0-b6a9-8fe2a1d3b71d as client.crash.ceph-mon-01.cloud.pd.infn.it failed: 2024-08-01T09:26:39.599+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.599+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44b4060ac0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44b408c3d0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:39.604+0200 7f44ba8ae640 -1 AuthRegistry(0x7f44ba8ad0c0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:39 ceph-mon-01 ceph-crash[1399]: [errno 2] RADOS object not found (error connecting to the cluster) Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: WARNING:ceph-crash:post /var/lib/ceph/crash/2024-07-23T09:09:17.703402Z_911fe89d-8f28-4f0e-8280-09b2e8a2e342 as client.crash.ceph-mon-01.cloud.pd.infn.it failed: 2024-08-01T09:26:40.277+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.277+0200 7fdfa4314640 -1 AuthRegistry(0x7fdf9c060ac0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.282+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.282+0200 7fdfa4314640 -1 AuthRegistry(0x7fdf9c08c3d0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.283+0200 7fdfa4314640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: 2024-08-01T09:26:40.283+0200 7fdfa4314640 -1 AuthRegistry(0x7fdfa43130c0) no keyring found at /etc/ceph/ceph.client.crash.ceph-mon-01.cloud.pd.infn.it.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Aug 1 09:26:40 ceph-mon-01 ceph-crash[1399]: [errno 2] RADOS object not found (error connecting to the cluster)