Hi, The network my 4-node cluster uses broke down after a driver issue a week ago. Now, as the network resumed normal operation, but my Ceph 19 cluster first said HEALTH_WARN and informed me of a lengthy recovery process, but some one hour later, I only found this error message: mixtile@blade3n1:~$ sudo ceph -s [sudo] password for mixtile: 2026-05-25T13:35:51.685+0200 ffff9701f180 0 monclient(hunting): authenticate timed out after 300 [errno 110] RADOS timed out (error connecting to the cluster) A restart of all nodes did *not* help. Even worse: The Docker containers with the various processes (mon, mrg, crash,…) started disappearing one by one! Here is what remained: mixtile@blade3n1:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2039b18ba392 quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 47 minutes ago Up 47 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-node-exporter-blade3n1 16cb4a6822f2 af0c5903e901 "/usr/bin/ceph-crash…" 47 minutes ago Up 47 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-crash-blade3n1 mixtile@blade3n2:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 42361a694abf quay.io/prometheus/prometheus:v2.51.0 "/bin/prometheus --c…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-prometheus-blade3n2 88b085d000a8 quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-node-exporter-blade3n2 7bb17808bdd8 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager -…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-alertmanager-blade3n2 a43cfe36da29 quay.io/ceph/grafana:10.4.0 "/run.sh" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-grafana-blade3n2 a95140b707ee af0c5903e901 "/usr/bin/ceph-crash…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-crash-blade3n2 38d4ca4035b0 af0c5903e901 "/usr/bin/ceph-mon -…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-mon-blade3n2 mixtile@blade3n3:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d664dfe30bd8 af0c5903e901 "/usr/bin/ceph-crash…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-crash-blade3n3 a64ac00dc28b quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-node-exporter-blade3n3 f7de98403a10 netdata/netdata "/usr/sbin/run.sh" mixtile@blade3n4:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d437cec7d6bf quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 54 minutes ago Up 53 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-node-exporter-blade3n4 c6d5ac595857 af0c5903e901 "/usr/bin/ceph-crash…" 54 minutes ago Up 53 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-crash-blade3n4 As you can see, there are much less Ceph-related processes than expected. The rest hasn't only crashed: In fact, the corresponding images have also disappeared! Pulling the missing containers didn't work: mixtile@blade3n1:~$ docker run ceph-mon Unable to find image 'ceph-mon:latest' locally docker: Error response from daemon: pull access denied for ceph-mon, repository does not exist or may require 'docker login' This is my exact system and Ceph version BTW: mixtile@blade3n1:~$ ceph -v ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable) mixtile@blade3n1:~$ uname -a Linux blade3n1 6.1.0-1027-rockchip #27 SMP Sun Apr 27 01:54:34 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux The drives my data are stored on seem to be still there (as lsblk said), but as all most OSD processes are gone, I can no longer access them. I've got four hosts, of which #1 is the admin node. #2 also hosts Ganesha NFS for external clients. So: What can I do to bring my cluster back to life without endangering my data? Thank you. Kind regards Jacek Rużyczka