I have found these two threads https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VHZ7IJ7PAL7L... https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NG5QVRTVCLLY... but I didn't exactly figure out what to do. I've since found remnants of removed cephfs-cluster-node-{0,1,2} in crush map buckets. Which I removed with no effect on health detail. I found out that ceph dashboard lists the non-existent cephfs-cluster-node-2 among ceph hosts (while orch host ls doesn't). On the other hand, device ls-by-host cephfs-cluster-node-2 lists an entry with device name coinciding with a drive on the live X host, with daemons listing the mon.X. Meanwhile device ls-by-host X lists among other things the exact same entry with the difference that in the dev column there is the actual device nvme0n1 whereas with cephfs-cluster-node-2 the column was empty root@X:~# cephadm shell -- ceph device ls-by-host cephfs-cluster-node-2 DEVICE DEV DAEMONS EXPECTED FAILURE Samsung_SSD_970_PRO_1TB_S462NF0M310269L mon.X root@X:~# cephadm shell -- ceph device ls-by-host X DEVICE DEV DAEMONS EXPECTED FAILURE Samsung_SSD_850_EVO_250GB_S2R6NX0J423123P sdb osd.3 Samsung_SSD_860_EVO_1TB_S3Z9NY0M431048H mon.Y Samsung_SSD_970_PRO_1TB_S462NF0M310269L nvme0n1 mon.X This output is confusing since it lists mon.Y also when asked for host X. I will continue investigating. If anyone has any hints what to try or where to look, I would be very grateful. Jakub On Sun, 17 Nov 2024, 15:34 Tim Holloway, <timh@mousetech.com> wrote:
I think I can count 5 sources that Ceph can query to report/display/control its resources.
1. The /dec/ceph/ceph.conf file. Mostly supplanted bt the Ceph configuration database.
2. The ceph configuration database. A namelesskey/value store internal to a ceph filesystem. It's distributed (no fixed location), accessed by Ceph commands and APIs/
3. Legacy Ceph resources/ Stuff found under a host's /var/lib/ceph directory.
4. Managed Ceph resources. Stuff found under a host's /var/lib/ceph/{fsid} diirectory.
5. The live machine state of Ceph. Since this not only can vary from host to host, but also service to service, I don't think that this is considered to be an authoritative source of information.
Compounding this is that current releases of Ceph can all too easily end up in a "forbidden" state where you may have, for example a legacy OSD.6 and a managed OSD.6 on the same host. In such a case, system is generally operable, but functionally corrupt and ideally should be corrected to remove the redundant resource.
The real issue is that depending on what Ceph interface you're querying (or "ceph health" is querying!), you don't always get your answer from a single authoritative source, so you'll get conflicting results and annoying error reports. The "stray daemon" condition is an especially egregious example of this, and it's not only possible because of a false detection from one of the above sources, but also, I think can come from "dead" daemons being referenced in CRUSH.
You might want to run through this lists's history for "phantom host" postings made by me back around this past June because I was absolutely plagued with them. Eugen Block helped me eventually purge them all.
Regards, Tim
On Sat, 2024-11-16 at 21:42 +0100, Jakub Daniel wrote:
Hello,
I'm pretty new to ceph deployment. I have setup my first cephfs cluster using cephadm. Initially, I deployed ceph in 3 virtualbox instances that I called cephfs-cluster-node-{0, 1, 2} just to test things. Later, I added 5 more real hardware nodes. Later I decided I'd remove the virtualboxes, so I drained the osds and removed the hosts. Suddenly, ceph status detail started reporting
HEALTH_WARN 1 stray host(s) with 1 daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by cephadm stray host cephfs-cluster-node-2 has 1 stray daemons: ['mon.X']
The cephfs-cluster-node-2 is no longer listed among hosts, it is (and has been for tens of hours) offline (powered down). The mon.X doesn't even belong to that node, it is one of the real hardware nodes. I am unaware of mon.X ever running on cephfs-cluster-node-2 (never noticed it among systemd units).
Where does cephadm shell -- ceph status detail come to the conclusion there is something stray? How can I address this?
Thank you for any insights Jakub _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io