Hi all, not sure it's ok to ask questions here as we are only a little Theater where I manage the data of our sound and video files. I set up a ceph cluster as second backup location on bullseye with quincy which worked flawless. All runs smooth but I thought I would have to do a step forward to get on track. Yes I know: never change a running system. :-D After updating all three nodes I forget to set the ceph packages on hold so they were updated to reef with trixie. No way to downgrade ...I tried a lot....! So I went forward. After all all 3 nodes are back online running reef my cluster seems to work, my cephfs share is reachable by my client machine, backups are working. This is my ceph status output: cluster: id: c5986871-ab9f-4365-8113-a1854fa07786 health: HEALTH_WARN 15 mgr modules have failed dependencies services: mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 19h) mgr: ceph1(active, since 18h), standbys: ceph2, ceph3 mds: 1/1 daemons up, 2 standby osd: 24 osds: 24 up (since 19h), 24 in (since 22M) data: volumes: 1/1 healthy pools: 3 pools, 1041 pgs objects: 3.34M objects, 11 TiB usage: 32 TiB used, 56 TiB / 87 TiB avail pgs: 1041 active+clean Problem: dashboard is not working any more and I have this erre/warn: 15 mgr modules have failed dependencies The cluster was made without podman/containers. ceph-crash gives me a flood of this errors: ... WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T11:51:45.411885Z_738ee242-f7bd-4f6c-bd90-d96a08e06bf6 as client.admin failed: 2025-08-22T06:55:50.234+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.234+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff728064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff728068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.238+0000 7ff72cea96c0 -1 AuthRegistry(0x7ff72cea8410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.crash.ceph1 failed: 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81ac0640c8) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81ac0686c0) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.318+0000 7f81b3ea96c0 -1 AuthRegistry(0x7f81b3ea8410) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.crash failed: 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5b9c064068) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5b9c068540) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.398+0000 7f5ba2ea96c0 -1 AuthRegistry(0x7f5ba2ea8410) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:54:08.817091Z_34eaeca9-7883-4b93-b75d-83102c113baa as client.admin failed: 2025-08-22T06:55:50.478+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.478+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc844064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc844068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.482+0000 7fc8494fe6c0 -1 AuthRegistry(0x7fc8494fd410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.crash.ceph1 failed: 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d40640c8) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d40686c0) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.562+0000 7f83d9aa96c0 -1 AuthRegistry(0x7f83d9aa8410) no keyring found at /etc/ceph/ceph.client.crash.ceph1.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.crash failed: 2025-08-22T06:55:50.642+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.642+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a30064068) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a30068540) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.646+0000 7f2a37aa96c0 -1 AuthRegistry(0x7f2a37aa8410) no keyring found at /etc/ceph/ceph.client.crash.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) WARNING:ceph-crash:post /var/lib/ceph/crash/2025-08-21T10:28:14.730789Z_ad2ef583-85b7-4cf0-a2fd-2fa3b775a7a2 as client.admin failed: 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa800064068) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa800068540) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-22T06:55:50.726+0000 7fa805ea96c0 -1 AuthRegistry(0x7fa805ea8410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx [errno 2] RADOS object not found (error connecting to the cluster) ...... In /etc/ceph I have ceph.client.admin.keyring ceph.conf rbdmap Maye there is a way to fix this but if it's to complicated I will think about a complete re-install. I'm not an admin but an amateur with some linux knowledge. If there is hope and someone could support to solev this I would be very happy! Regards, Christian Peters