CEPHADM_FAILED_DAEMON: OSDs in error state but containers running fine (19.2.3/Squid)
Hi all, Ceph 19.2.3 (Squid), cephadm-deployed, 528 OSDs. After a drive failure on one host (sdc, ~10 min of SCSI retry stalls at boot), ceph health detail now reports 13 CEPHADM_FAILED_DAEMON warnings on that host: daemon osd.266 on ceph-bh03 is in error state daemon osd.267 on ceph-bh03 is in error state ... (13 total) ceph orch ps confirms the same 13 as error. But checking each container directly: podman ps -a --filter name=osd.275 ... Up 3 days 12 of the 13 containers are running fine, no restarts, clean RocksDB stats in their logs. Only 1 (the OSD that actually sat on the dead drive) has no container at all. So it looks like cephadm's daemon-state cache went stale for the 12 healthy ones. Has anyone seen this before? Is "ceph mgr fail" the right way to force a clean orchestrator re-sync here, or is there a more targeted fix (per-daemon redeploy, restarting the cephadm mgr module)? Thanks, Soroush -- Soroush Pasandideh | Universität Osnabrück - Rechenzentrum | 0541 969 2336
participants (1)
-
Sroush Pasandideh