Hi, I would probably stop the upgrade to continue, this might be blocking cephadm. Then try again to redeploy a daemon, if it still fails check the cephadm.log(s) on the respective servers as well as the active mgr log. Regards, Eugen Zitat von Thomas Widhalm <widhalmt@widhalm.or.at>:
Hi,
As you might know, I have a problem with MDS not starting. During the investigation with your help I found another issue that might be related.
I can plan to restart, redeploy, reconfigure services via cephadm or dashboard just as I want but services won't react. I only see the action to be scheduled but that's all.
2023-04-13T17:11:15.690698+0000 mgr.ceph04.qaexpv (mgr.74906907) 37184 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph05.pqxmvt 2023-04-13T17:11:20.746743+0000 mgr.ceph04.qaexpv (mgr.74906907) 37190 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph06.rrxmks 2023-04-13T17:11:24.971226+0000 mgr.ceph04.qaexpv (mgr.74906907) 37195 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph07.omdisd
It's the same for other daemons/services. I changed placement rules, scheduled changes, failed mgr, even rebooted hosts. I even was desperate enough to delete files for services from hosts before rebooting hoping I could trigger a manual redeploy.
All I see are the same MDS stuck in "error" state. I removed them via "ceph orch rm" but they are still there. When I reissue the command it fails saying that the service isn't there.
"ceph orch ps" still lists them.
mds.mds01.ceph03.xqwdjy ceph03 error 2d ago 2M - - <unknown> <unknown> <unknown> mds.mds01.ceph04.hcmvae ceph04 error 2d ago 2d - - <unknown> <unknown> <unknown> mds.mds01.ceph05.pqxmvt ceph05 error 2d ago 10M - - <unknown> <unknown> <unknown> mds.mds01.ceph06.rrxmks ceph06 error 2d ago 10w - - <unknown> <unknown> <unknown> mds.mds01.ceph07.omdisd ceph07 error 2d ago 3M - - <unknown> <unknown> <unknown>
Any idea how I can get rid of them? Or redeploy them?
Additionally I'm just in the middle of an upgrade.
{ "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon", "osd" ], "progress": "18/40 daemons upgraded", "message": "Upgrade paused", "is_paused": true }
I paused it on purpose to allow for manipulation of daemons.
Cheers, Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io