Thanks, Adam. Before I purged the nodes again, I looked at the current output of 'ceph orch ps', and indeed, there are still orphaned OSD daemons: # ceph orch ps --daemon-type osd NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.0 host5 running (17h) 7m ago 17h 201M 4096M 19.2.2 4892a7ef541b 0d3e81e80f8e osd.0 host6 stopped 7m ago 18h - 4096M <unknown> <unknown> <unknown> osd.1 host7 running (17h) 7m ago 18h 246M 4096M 19.2.2 4892a7ef541b 4b81e06eaea7 osd.2 host5 stopped 7m ago 18h - 4096M <unknown> <unknown> <unknown> I didn't look at that output before. I'll try to clear that state and then repeat the test. Zitat von Adam King <adking@redhat.com>:
The daemons cephadm "knows" about is actually just based on the contents of the /var/lib/ceph/<fsid>/ directory on each given host cephadm is managing. If osd.6 was present, got removed by the host drain process, and then its daemon directory was still on the host / there was still a container running for osd.6, it sounds like a bug with the drain process and sounds worthy of a ticket (I think this is what happened based on what you said). If I'm misreading and this was a manual removal of osd.6 then it could be possible that cephadm either hadn't checked the host for what daemons were there since that removal happened (could verify by checking the REFRESHED column of `ceph orch ps`, osd.6 should be listed there if you got this error) or that removal process didn't clean up the daemon directory, in which case I wouldn't consider it to have been a bug. Assuming it's the former case and you can show what was actually left on the host for osd.6 or you have a consistent way to reproduce the failed removal, I can take a look.
On Fri, Jul 25, 2025 at 8:01 AM Eugen Block <eblock@nde.ag> wrote:
Hi *,
an unexpected issue occurred today, at least twice, so it seems kind of reproducable. I've been preparing a demo in a (virtual) lab cluster (19.2.2) and wanted to drain multiple hosts. The first time I didn't pay much attention, but the draining seemed stuck (kind of a common issue these days), so I intervened and cleaned up until I got into a healthy state, all good. Then I did my thing, changed the crush tree, added the removed hosts again, cephadm created the OSDs, backfill finished successfully.
Now I wanted to reset the cluster again to my starting point, so I issued the drain command again for multiple hosts (each host has 2 OSDs):
# for i in {5..8}; do ceph orch host drain host$i; done
This time all OSDs were drained successfully (I watched 'ceph orch osd rm status'), so I wanted to remove the hosts, but it failed:
# for i in {5..8}; do ceph orch host rm host$i --rm-crush-entry; done Removed host 'host5' Removed host 'host6' Removed host 'host7' Error EINVAL: Not allowed to remove host8 from cluster. The following daemons are running in the host: type id -------------------- --------------- osd 6
Please run 'ceph orch host drain host8' to remove daemons from host
But there was nothing to drain anymore, osd.6 was already successfully removed from the crush tree. But on host8 there was still a daemon I had to clean up manually:
host8:~ # cephadm rm-daemon --name osd.6 --fsid 543967bc-e586-32b8-bd2c-2d8b8b168f02 --force
I compared the cephadm.log files (3 out of 4 to-be-drained hosts were successfully drained) and on host8 the command rm-daemon was never executed (until I did manually). Is this a known issue? It doesn't seem to happen with only on host, at least I didn't notice in the past. Should I create a tracker for this?
Thanks, Eugen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io