demons being stray?
Hi guys. Is there a way to "clean" those up, both orderly & not manners would be ok. -> $ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.02539 root default -3 0.34180 host podster1 9 hdd 0.04880 osd.9 up 1.00000 1.00000 10 hdd 0.29300 osd.10 up 1.00000 1.00000 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 up 1.00000 1.00000 4 hdd 0.29300 osd.4 up 1.00000 1.00000 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 up 1.00000 1.00000 5 hdd 0.29300 osd.5 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000 3 0 osd.3 down 0 1.00000 6 0 osd.6 down 0 1.00000 7 0 osd.7 down 0 1.00000 8 0 osd.8 down 0 1.00000 ID CLASS WEIGHT TYPE NAME -1 1.02539 root default -3 0.34180 host podster1 9 hdd 0.04880 osd.9 10 hdd 0.29300 osd.10 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 4 hdd 0.29300 osd.4 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 5 hdd 0.29300 osd.5 -> $ ceph node ls ... "osd": { "podster1.mine.priv": [ 0, 3, 6, 7, 8, 9, 10 ], "podster2.mine.priv": [ 2, 4 ], "podster3.mine.priv": [ 1, 5 ] }, ... many thanks, L.
Hi, maybe you could add some more details, for example if your cluster is managed by cephadm. If so, you can remove orphaned OSDs with the orchestrator: ceph orch osd rm 0 3 6 7 8 (force might be required, zap-osds flag can be also helpful) Or if the orchestrator isn’t able to do that, you can use cephadm locally on the node: ceph osd purge 0 cephadm rm-daemon --name osd.0 Does that help? Zitat von lejeczek <peljasz@yahoo.co.uk>:
Hi guys.
Is there a way to "clean" those up, both orderly & not manners would be ok.
-> $ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.02539 root default -3 0.34180 host podster1 9 hdd 0.04880 osd.9 up 1.00000 1.00000 10 hdd 0.29300 osd.10 up 1.00000 1.00000 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 up 1.00000 1.00000 4 hdd 0.29300 osd.4 up 1.00000 1.00000 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 up 1.00000 1.00000 5 hdd 0.29300 osd.5 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000 3 0 osd.3 down 0 1.00000 6 0 osd.6 down 0 1.00000 7 0 osd.7 down 0 1.00000 8 0 osd.8 down 0 1.00000
ID CLASS WEIGHT TYPE NAME -1 1.02539 root default -3 0.34180 host podster1 9 hdd 0.04880 osd.9 10 hdd 0.29300 osd.10 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 4 hdd 0.29300 osd.4 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 5 hdd 0.29300 osd.5
-> $ ceph node ls ... "osd": { "podster1.mine.priv": [ 0, 3, 6, 7, 8, 9, 10 ], "podster2.mine.priv": [ 2, 4 ], "podster3.mine.priv": [ 1, 5 ] }, ...
many thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-> $ ceph orch osd rm 0 3 6 7 8 Unable to find OSDs: ['0', '3', '6', '7', '8'] -> $ ceph osd purge 0 purged osd.0 # !! <= seems this here does the trick, after which /ceph mgr fail/ and health OK. -> $ cephadm rm-daemon --name osd.0 --force --fsid 9f4f9dba-72c7-11f0-8052-525400519d29 Non-zero exit code 3 from systemctl status ceph-9f4f9dba-72c7-11f0-8052-525400519d29@osd.0.service systemctl: stdout ○ ceph-9f4f9dba-72c7-11f0-8052-525400519d29@osd.0.service - Ceph osd.0 for 9f4f9dba-72c7-11f0-8052-525400519d29 systemctl: stdout Loaded: loaded (/etc/systemd/system/ceph-9f4f9dba-72c7-11f0-8052-525400519d29@.service; disabled; preset: disabled) systemctl: stdout Active: inactive (dead) ERROR: Daemon not found: osd.0. See `cephadm ls` many thanks!
btw. On the host with stray daemons I find: -> $ llar /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/removed/ total 36 drwx------. 3 ceph ceph 4096 Aug 17 01:04 mon.podster1.mine.priv_2025-08-18T15:44:42.323358Z drwx------. 3 ceph ceph 4096 Aug 21 09:45 mon.podster1_2025-08-22T09:57:59.336223Z drwx------. 3 ceph ceph 4096 Aug 30 18:29 mon.podster1_2025-09-01T07:33:11.090257Z drwx------. 3 ceph ceph 4096 Sep 16 02:18 mon.podster1_2025-09-16T09:01:59.990439Z drwx------. 3 ceph ceph 4096 Nov 1 14:50 mon.podster1_2025-11-02T16:19:50.366693Z drwx------. 3 ceph ceph 4096 Nov 3 19:48 mon.podster1_2025-11-04T09:14:16.327796Z drwx------. 2 ceph ceph 4096 Nov 4 09:47 osd.6_2025-11-04T09:52:05.968466Z drwx------. 2 ceph ceph 4096 Nov 4 10:00 osd.7_2025-11-04T10:03:27.957622Z drwx------. 2 ceph ceph 4096 Nov 4 10:00 osd.8_2025-11-04T10:03:35.668456Z I removed a few prior to messaging here, in hope that might help but it did not. I also tried other /cmds/ earlier, which failed as I showed - I only missed *purge *Does that folder serve any purpose? thanks, L.
That folder contains all the daemons that the orchestrator removed. It can have a purpose, for example for Prometheus data, or even the Mon db store. But it also might fill up the local disk space, and I’m not aware of any retention setting, so one has to keep an eye on that. Zitat von lejeczek <peljasz@yahoo.co.uk>:
btw. On the host with stray daemons I find:
-> $ llar /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/removed/ total 36 drwx------. 3 ceph ceph 4096 Aug 17 01:04 mon.podster1.mine.priv_2025-08-18T15:44:42.323358Z drwx------. 3 ceph ceph 4096 Aug 21 09:45 mon.podster1_2025-08-22T09:57:59.336223Z drwx------. 3 ceph ceph 4096 Aug 30 18:29 mon.podster1_2025-09-01T07:33:11.090257Z drwx------. 3 ceph ceph 4096 Sep 16 02:18 mon.podster1_2025-09-16T09:01:59.990439Z drwx------. 3 ceph ceph 4096 Nov 1 14:50 mon.podster1_2025-11-02T16:19:50.366693Z drwx------. 3 ceph ceph 4096 Nov 3 19:48 mon.podster1_2025-11-04T09:14:16.327796Z drwx------. 2 ceph ceph 4096 Nov 4 09:47 osd.6_2025-11-04T09:52:05.968466Z drwx------. 2 ceph ceph 4096 Nov 4 10:00 osd.7_2025-11-04T10:03:27.957622Z drwx------. 2 ceph ceph 4096 Nov 4 10:00 osd.8_2025-11-04T10:03:35.668456Z
I removed a few prior to messaging here, in hope that might help but it did not. I also tried other /cmds/ earlier, which failed as I showed - I only missed *purge *Does that folder serve any purpose? thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
lejeczek