----- Le 26 Fév 25, à 16:40, Tim Holloway timh@mousetech.com a écrit :
Thanks. I did resolve that problem, though I haven't had a chance to update until now.
I had already attempted to use ceph orch to remove the daemons, but they didn't succeed.
Fortunately, I was able to bring the host online, which allowed the scheduled removals to complete. I confirmed everything was drained, again removed the host from inventory and powered down.
Still got complaints from cephadm about the decommissioned host.
I took a break - impatience and ceph don't mix - and came back to address the next problem. which was lots of stuck PGs. Either because cephadm timed out or something kicked in when I started randomly rebooting OSDs. the host complaint finally disappeared. End of story.
Now for what sent me down that path.
I had 2 OSDs on one server and felt that that was probably not a good idea, so I marked one for deletion. 4 days later it was still in "destroying" state. More concerning, all signs indicated that despite having been reweighted to 0, the "destroying" OSD was still an essential participant and no indication that its PGs were being relocared to active servers. Shutting down the "destroying" OSD would immediately trigger a re-allocation panic, but that didn't clean anything. The re-allocation would proceed at a furious pace, then slowly stall out and hang, and the system was degraded. Restarting the OSD brought the PG inventory back up, but stuff still wasn't moving off the OSD,
Right about that time I decommissioned the questionable host.
Finally, I did a "ceph orch rm osd.x", and terminated the "destroying" permanently, making it finally disappear from the OSD tree list.
I also deleted a number of OSD pools that are (hopefully) not going to be missed.
Kicking and randomly repeatedly rebooting the other OSDs finally cleared all the stuck OSDs, some of which hadn't resolved in over 2 days.
So at the moment, it's either rebalancing the cleaned-up OSDs or in a loop thinking that it is.
Since you deleted some pools, it's probably the upmap balancer rebalancing PGs across the OSDs.
And the PG/per-OSD count seems way too high,
How much is it right now? With what hardware?
but the auto-sized doesn't seem to want to do anything about that.
If the PG autoscaler is enabled you could try adjusting per pool settings [1] and see if the # of PGs decreases. If disabled you could manually reduce the number of PGs on the remaining pools to lower the PG/OSD ratio. Regards, Frédéric.
Of course, the whole shebang has been unavailable to clients this whole week because of that.
I've been considering upgrading to reef, but recent posts regarding issues resembling what I've been going through are making me pause.
Again, thanks! Tim
On Wed, 2025-02-26 at 13:57 +0100, Frédéric Nass wrote:
Hi Tim,
If you can't bring the host back online so that cephadm can remove these services itself, I guess you'll have to clean up the mess by:
- removing these services from the cluster (for example with a 'ceph mon remove {mon-id}' for the monitor) - forcing their removal from the orchestrator with the --force option on the commands 'ceph orch daemon rm <names>' and 'ceph orch host rm <hostname>'. If the --force option doesn't help, then looking into/editing/removing ceph-config keys like 'mgr/cephadm/inventory' and 'mgr/cephadm/host.ceph07.internal.mousetech.com' that 'ceph config-key dump' output shows might help.
Regards, Frédéric.
----- Le 25 Fév 25, à 16:42, Tim Holloway timh@mousetech.com a écrit :
Ack. Another fine mess.
I was trying to clean things up and the process of tossing around OSD's kept getting me reports of slow responses and hanging PG operations.
This is Ceph Pacific, by the way.
I found a deprecated server that claimed to have an OSD even though it didn't show in either "ceph osd tree" or the dashboard OSD list. I suspect that a lot of the grief came from it attempting to use resources that weren't always seen as resources.
I shut down the server's OSD (removed the daemon using ceph orch), then foolishly deleted the server from the inventory without doing a drain first.
Now cephadmin hates me (key not found), and there are still an MDS and MON listed as ceph orch ls daemons even after I powered the host off.
I cannot do a ceph orch daemon delete because there's no longer an IP address available to the daemon delete, and I cannot clear the cephadmin queue:
[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'ceph07.internal.mousetech.com'
Any suggestions? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io