Hi, thank you Eugen and Tim.
did you fail the mgr?
I think I didn't.
Or how exactly did you drain that host?
``` cephadm shell -- ceph orch host drain cephfs-cluster-node-2 cephadm shell -- ceph orch host rm cephfs-cluster-node-2 ```
`ceph config-key get mgr/cephadm/host.cephfs-cluster-node-2 | jq`
Error ENOENT:
Is the VM still reachable?
No. It wasn't reachable for majority of the time the warning was displayed.
Remove the directory so cephadm forgets about it.
What directory do you mean? I have since made the VM reachable again and removed it again a few times. I even tried removing the mon.X that was running on X while (and ceph knew about it) while ceph was complaining that mon.X is running on cephfs-cluster-node-2. I disabled and re-enabled the stray warnings. I found some custom config pertaining to cephfs-cluster-node-0 (which `health detail` doesn't report as stray; namely: `osd host:cephfs-cluster-node-0 basic osd_memory_target 5829100748`). I tried enabling mgr module cli, to query list_servers, which I believe is behind what `health detail` queries, but ceph refused to enable that module. So I queried restful `/server`. I ran `grep -R cephfs-cluster-node-2` in /var/lib/ceph*/ and found a few logs and binary files that matched, among them some of the osd dirs. I tried to drop one of the osds and re-add it later. None of these things seemed to have any immediate effect. But suddenly after days of warning about stray host with a daemon (that does not even belong to that host), the warning disappeared. I am afraid I may have done some other things out of desperation that I do not recall at the moment. I hope the stray host + stray daemon warning does not return. Thanks again! Jakub On Mon, 18 Nov 2024 at 16:11, Eugen Block <eblock@nde.ag> wrote:
Hi,
just to be safe, did you fail the mgr? If not, try 'ceph mgr fail' and see if it still reports that information. It sounds like you didn't clean up your virtual MON after you drained the OSDs. Or how exactly did you drain that host? If you run 'ceph orch host drain {host}' the orchestrator will remove all daemons, not only OSDs. I assume that there are still some entries in the ceph config-key database where the 'device ls-by-host' output comes from, I haven't verified that though. Is this key present?
ceph config-key get mgr/cephadm/host.cephfs-cluster-node-2 | jq
Is the VM still reachable? If it is, check the MON directory under /var/lib/ceph/{FSID}/mon.X and remove it. You can also first check with 'cephadm ls --no-detail' if cephadm believes that there's a MON daemon located. Remove the directory so cephadm forgets about it.
Zitat von Jakub Daniel <jakub.daniel@gmail.com>:
I have found these two threads
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VHZ7IJ7PAL7L...
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NG5QVRTVCLLY...
but I didn't exactly figure out what to do. I've since found remnants of removed cephfs-cluster-node-{0,1,2} in crush map buckets. Which I removed with no effect on health detail. I found out that ceph dashboard lists
non-existent cephfs-cluster-node-2 among ceph hosts (while orch host ls doesn't). On the other hand, device ls-by-host cephfs-cluster-node-2
the lists
an entry with device name coinciding with a drive on the live X host, with daemons listing the mon.X. Meanwhile device ls-by-host X lists among other things the exact same entry with the difference that in the dev column there is the actual device nvme0n1 whereas with cephfs-cluster-node-2 the column was empty
root@X:~# cephadm shell -- ceph device ls-by-host cephfs-cluster-node-2 DEVICE DEV DAEMONS EXPECTED FAILURE Samsung_SSD_970_PRO_1TB_S462NF0M310269L mon.X
root@X:~# cephadm shell -- ceph device ls-by-host X DEVICE DEV DAEMONS EXPECTED FAILURE Samsung_SSD_850_EVO_250GB_S2R6NX0J423123P sdb osd.3 Samsung_SSD_860_EVO_1TB_S3Z9NY0M431048H mon.Y Samsung_SSD_970_PRO_1TB_S462NF0M310269L nvme0n1 mon.X
This output is confusing since it lists mon.Y also when asked for host X.
I will continue investigating. If anyone has any hints what to try or where to look, I would be very grateful.
Jakub
On Sun, 17 Nov 2024, 15:34 Tim Holloway, <timh@mousetech.com> wrote:
I think I can count 5 sources that Ceph can query to report/display/control its resources.
1. The /dec/ceph/ceph.conf file. Mostly supplanted bt the Ceph configuration database.
2. The ceph configuration database. A namelesskey/value store internal to a ceph filesystem. It's distributed (no fixed location), accessed by Ceph commands and APIs/
3. Legacy Ceph resources/ Stuff found under a host's /var/lib/ceph directory.
4. Managed Ceph resources. Stuff found under a host's /var/lib/ceph/{fsid} diirectory.
5. The live machine state of Ceph. Since this not only can vary from host to host, but also service to service, I don't think that this is considered to be an authoritative source of information.
Compounding this is that current releases of Ceph can all too easily end up in a "forbidden" state where you may have, for example a legacy OSD.6 and a managed OSD.6 on the same host. In such a case, system is generally operable, but functionally corrupt and ideally should be corrected to remove the redundant resource.
The real issue is that depending on what Ceph interface you're querying (or "ceph health" is querying!), you don't always get your answer from a single authoritative source, so you'll get conflicting results and annoying error reports. The "stray daemon" condition is an especially egregious example of this, and it's not only possible because of a false detection from one of the above sources, but also, I think can come from "dead" daemons being referenced in CRUSH.
You might want to run through this lists's history for "phantom host" postings made by me back around this past June because I was absolutely plagued with them. Eugen Block helped me eventually purge them all.
Regards, Tim
On Sat, 2024-11-16 at 21:42 +0100, Jakub Daniel wrote:
Hello,
I'm pretty new to ceph deployment. I have setup my first cephfs cluster using cephadm. Initially, I deployed ceph in 3 virtualbox instances that I called cephfs-cluster-node-{0, 1, 2} just to test things. Later, I added 5 more real hardware nodes. Later I decided I'd remove the virtualboxes, so I drained the osds and removed the hosts. Suddenly, ceph status detail started reporting
HEALTH_WARN 1 stray host(s) with 1 daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by cephadm stray host cephfs-cluster-node-2 has 1 stray daemons: ['mon.X']
The cephfs-cluster-node-2 is no longer listed among hosts, it is (and has been for tens of hours) offline (powered down). The mon.X doesn't even belong to that node, it is one of the real hardware nodes. I am unaware of mon.X ever running on cephfs-cluster-node-2 (never noticed it among systemd units).
Where does cephadm shell -- ceph status detail come to the conclusion there is something stray? How can I address this?
Thank you for any insights Jakub _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io