Zitat von Holger Naundorf <naundorf@rz.uni-kiel.de>:
Hello, title should of course be "orchestrator behaving strangely"
I did give a mgr restart another try (for the last OSD removal - which also did not work - I did already restart the mgr without effect)
there is no (immediate- i.e after ~10min) effect now as well - or should I reissue the OSD rm command as well?
Is there something in the queue (ceph orch osd rm status)? Sometimes the queue clears after a mgr restart, so it might be necessary to restart the rm command as well.
Regards, Holger
On 27.06.25 12:26, Eugen Block wrote:
Hi,
have you retried it after restarting/failing the mgr?
ceph mgr fail
Quite often this (still) helps.
Zitat von Holger Naundorf <naundorf@rz.uni-kiel.de>:
Hello, we are running a ceph cluster at version:
ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
and since a few weeks the orchestrator started to misbehave - up to now we could not identify any root cause, so I am fishing in the community to see if there are any hints.
Problems:
An OSD removal (for disk replacement) gets stuck in the 'purge' step:
ceph orch osd rm 406 --replace
root@aadm01:~# ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 406 acn07 done, waiting for purge 0 True False True 2025-06-25 09:18:07.650734+00:00
(now for more than 24h in this state)
At the same time the orchestrator is not restarting OSD daemons - i.e. an 'ceph orch daemon restart osd.xxx' claims its queuing uo the restart, but it never happens. Other services continue to be controlled correctly via 'ceph orch ...'
If anyone has an idea where to poke around or can match this to some known problem - I would appreciate any pointers.
Regards, Holger
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de