Hi folks, I've now been struggling for five weeks with my broken Ceph cluster (4 nodes on Ubuntu 24, containerised with Docker) and can't find a solution, not even with help from Google's AI. Here is what happened: 1. The cluster FS was no longer accessible, and even ceph -s no longer worked (error 110). The MON count had gone down to 2, and an hour later, all 4 OSD daemons were also gone. The corresponding containers had also disappeared (docker ps -a no longer displayed them) together with their logs. 2. A week later, one of the nodes got frozen. Accessing its serial console returned a bunch of low-level errors you can read in this forum thread if you wish: https://community.mixtile.com/t/node-3-unreachable-100-cpu-load/941/6 3. It turned out that some essential files on the nodes (like ceph.conf or keyrings) had simply disappeared. Further research revealed trouble with the internal flash storage (eMMC) of the nodes. So, the data on them was apparently corrupt, what forced me to ressort to "safer" eMMC settings, re-flash them, and re-install Ceph (with the FSID of the old cluster, of course). After that, 4 MONs and 2 MGRs worked, but: 4. Re-integrating the existing OSDs (on the NVMe SSDs, not on the internal eMMCs) did not work: Eugen's script https://heiterbiswolkig.blogs.nde.ag/2026/06/28/cephadm-recover-mon-store-us... ended me up with a "No such file or directory" error. Google's AI made a bunch of suggestions, which did not help either (mostly due to "Permission denied" errors). It also told me to shut down one of the MONs, extract a monmap, and inject it into the new cluster. But the "permission denied" errors persisted for at least two weeks until we could get rid of them. 5. Two OSDs could get revivif <https://dict.leo.org/englisch-deutsch/revivify>ied, but ceph -s marked them as "in", but "down" whatever we did. Extracting the keys from the OSDs and adding them to the config of the "new" cluster didn't help. So I got stuck in a dead end. 6. Now, Google's AI wants me to extract the raw data (which is hopefully still complete) from the drives for whatever purpose and store it on another computer, but I'm afraid that doing this won't help me either. So I need some reliable expert advice in order to bring my cluster back to life. What shall I do here? Are the files in my cluster maybe also corrupted by the abovementioned incident? Thank you. Regards Jacek