Hi, this sound a bit like a "classic" quorum loss, resulting in cascading daemon failures. There seems to be one surviving MON (blade3n2), that could be a good starting point for disaster recovery by reducing the monmap to this one MON. This should give you back qourum and a working cluster. You might need to run 'systemctl reset-failed...' to let systemd start the containers. Although the product has been discontinued, the section of the SUSE docs [0] is still relevant ("Restoring the MONs quorum"). Back up the mon store of each node before you do this, just in case. The procedure itself has worked many times for me, but maybe there's an easier way, especially if you're not too familiar with cephadm or this procedure. But before you do that, do you have MON logs with an explanation why they refuse to start? Regarding Ceph images, your cluster uses af0c5903e901 for the Ceph services, what does 'docker images | grep af0c5903e901' show? I doubt it's "ceph-mon:latest", at least I haven't seen those in use with cephadm (I have the impression that this is a "regular" cephadm cluster). Regards, Eugen [0] https://documentation.suse.com/pt-br/ses/7.1/html/ses-all/bp-troubleshooting... Zitat von Jacek Rużyczka via ceph-users <ceph-users@ceph.io>:

Hi,

The network my 4-node cluster uses broke down after a driver issue a week ago. Now, as the network resumed normal operation, but my Ceph 19 cluster first said HEALTH_WARN and informed me of a lengthy recovery process, but some one hour later, I only found this error message:

mixtile@blade3n1:~$ sudo ceph -s [sudo] password for mixtile: 2026-05-25T13:35:51.685+0200 ffff9701f180 0 monclient(hunting): authenticate timed out after 300 [errno 110] RADOS timed out (error connecting to the cluster)

A restart of all nodes did *not* help. Even worse: The Docker containers with the various processes (mon, mrg, crash,…) started disappearing one by one! Here is what remained:

mixtile@blade3n1:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2039b18ba392 quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 47 minutes ago Up 47 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-node-exporter-blade3n1 16cb4a6822f2 af0c5903e901 "/usr/bin/ceph-crash…" 47 minutes ago Up 47 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-crash-blade3n1

mixtile@blade3n2:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 42361a694abf quay.io/prometheus/prometheus:v2.51.0 "/bin/prometheus --c…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-prometheus-blade3n2 88b085d000a8 quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-node-exporter-blade3n2 7bb17808bdd8 quay.io/prometheus/alertmanager:v0.25.0 "/bin/alertmanager -…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-alertmanager-blade3n2 a43cfe36da29 quay.io/ceph/grafana:10.4.0 "/run.sh" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-grafana-blade3n2 a95140b707ee af0c5903e901 "/usr/bin/ceph-crash…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-crash-blade3n2 38d4ca4035b0 af0c5903e901 "/usr/bin/ceph-mon -…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-mon-blade3n2

mixtile@blade3n3:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d664dfe30bd8 af0c5903e901 "/usr/bin/ceph-crash…" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-crash-blade3n3 a64ac00dc28b quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 2 hours ago Up 2 hours ceph-8aad3073-39a1-11f1-bf6 e-f2704a1efa9b-node-exporter-blade3n3 f7de98403a10 netdata/netdata "/usr/sbin/run.sh"

mixtile@blade3n4:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d437cec7d6bf quay.io/prometheus/node-exporter:v1.7.0 "/bin/node_exporter …" 54 minutes ago Up 53 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-node-exporter-blade3n4 c6d5ac595857 af0c5903e901 "/usr/bin/ceph-crash…" 54 minutes ago Up 53 minutes ceph-8aad3073-39a1-11f1-b f6e-f2704a1efa9b-crash-blade3n4

As you can see, there are much less Ceph-related processes than expected. The rest hasn't only crashed: In fact, the corresponding images have also disappeared! Pulling the missing containers didn't work:

mixtile@blade3n1:~$ docker run ceph-mon Unable to find image 'ceph-mon:latest' locally docker: Error response from daemon: pull access denied for ceph-mon, repository does not exist or may require 'docker login'

This is my exact system and Ceph version BTW:

mixtile@blade3n1:~$ ceph -v ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)

mixtile@blade3n1:~$ uname -a Linux blade3n1 6.1.0-1027-rockchip #27 SMP Sun Apr 27 01:54:34 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

The drives my data are stored on seem to be still there (as lsblk said), but as all most OSD processes are gone, I can no longer access them. I've got four hosts, of which #1 is the admin node. #2 also hosts Ganesha NFS for external clients.

So: What can I do to bring my cluster back to life without endangering my data? Thank you.

Kind regards Jacek Rużyczka _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io