Hi Eugen, It's gone now, although similar artefacts seems to linger. The reason it's gone is that I've been upgrading all my machines to AlamLinux 8 from CentOS 7 and AlmaLinux 7, as one is already EOL and the other is within days of it. Rather than upgrade-in-place, I chose to nuke/replace the entire system disks and provision from scratch. It helped me clean up my network and get rid of years of cruft. Ceph helped a lot there, since I'd do one machine at a time, and since the provisioning data is on Ceph, it was always available even as individual machines went up and down. I lost the phantom host, although for a while one of the newer OSDs gave me issues. The container would die while starting claiming that the OSD block (badly-quoted) was "already in user". I believe this happened right after I moved the _admin node to that machine. I finally got the failed machine back online by manually stopping the systemd service, waiting a while, then starting (not restarting) it. But some other nodes may have been rebooted in the interim, so it's hard to be certain what actually made it happy. Annoyingly, the dashboard and OSD tree listed the failed node as "up" and "in" even thoiug "ceph orch ps" showed it as "error". I couldn't persuade it to go down and out, or I would have destroyed and re-created it. I did clear up a major mess, though. My original install/admin machine was littered with dead and mangled objects. Two long-deleted OSDs left traces, and there was a mix of pre-cephadm components (including) OSDs and newer stuff. I did discover a node still running Octopus which I plan to upgrade today, but overall things look pretty clean, excepting the ever- frustrating "too many PGs per OSD". If autotuning was supposed to auto- fix this, it's not doing so, even though autotuning is switched on. Manual changes don't seem to take either. Going back to the phantom host situation, one thing I have seen is that on the dashboard, the hosts display lists OSDs that have been deleted as belonging to that machine. "ceph osd tree" and the OSD view disagree and show neither the phantom host nor the deleted OSDs. Just to recap, the original phantom host was a non-ceph node that accidentally got sucked in when I did a host add with the wrong IP address. It then claimed to own another host's OSD. Thanks, Tim On Tue, 2024-07-09 at 06:08 +0000, Eugen Block wrote:
Hi Tim,
is this still an issue? If it is, I recommend to add some more details so it's easier to follow your train of thought.
ceph osd tree ceph -s ceph health detail ceph orch host ls
And then please point out which host you're trying to get rid of. I would deal with the rgw thing later. Is it possible, that the phantom host actually had OSDs on it? Maybe that needs to be cleaned up first. I had something similar on a customer cluster recently where we hunted failing OSDs but it turned out they were removed quite a while ago, just not properly cleaned up yet on the filesystem.
Thanks, Eugen
Zitat von Tim Holloway <timholloway34@gmail.com>:
It's getting worse.
As many may be aware, the venerable CentOS 7 OS is hitting end-of- life in a matter of days.
The easiest way to upgrade my serves has been to simply create an alternate disk with the new OS, turn my provisioning system loose on it, yank the old OS system disk and jack in the new one.
However, Ceph is another matter. For that part, the simplest thing to do is to destroy the Ceph node(s) on the affected box, do the OS upgrade, then re-create the nodes.
But now I have even MORE strays. The OSD on my box lives on in Ceph in the dashboard host view even though the documented removal procedures were followed and the VM itself was destroyed.
Further, this last node is an RGW node and I cannot remove it from the RGW configuration. It not only shows on the dashboard, it also lists as still active on the command line and as entries in the config database no matter what I do.
I really need some solution to this, as it's a major chokepoint in the upgrade process
Tim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io