Hi, As you might know, I have a problem with MDS not starting. During the investigation with your help I found another issue that might be related. I can plan to restart, redeploy, reconfigure services via cephadm or dashboard just as I want but services won't react. I only see the action to be scheduled but that's all. 2023-04-13T17:11:15.690698+0000 mgr.ceph04.qaexpv (mgr.74906907) 37184 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph05.pqxmvt 2023-04-13T17:11:20.746743+0000 mgr.ceph04.qaexpv (mgr.74906907) 37190 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph06.rrxmks 2023-04-13T17:11:24.971226+0000 mgr.ceph04.qaexpv (mgr.74906907) 37195 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph07.omdisd It's the same for other daemons/services. I changed placement rules, scheduled changes, failed mgr, even rebooted hosts. I even was desperate enough to delete files for services from hosts before rebooting hoping I could trigger a manual redeploy. All I see are the same MDS stuck in "error" state. I removed them via "ceph orch rm" but they are still there. When I reissue the command it fails saying that the service isn't there. "ceph orch ps" still lists them. mds.mds01.ceph03.xqwdjy ceph03 error 2d ago 2M - - <unknown> <unknown> <unknown> mds.mds01.ceph04.hcmvae ceph04 error 2d ago 2d - - <unknown> <unknown> <unknown> mds.mds01.ceph05.pqxmvt ceph05 error 2d ago 10M - - <unknown> <unknown> <unknown> mds.mds01.ceph06.rrxmks ceph06 error 2d ago 10w - - <unknown> <unknown> <unknown> mds.mds01.ceph07.omdisd ceph07 error 2d ago 3M - - <unknown> <unknown> <unknown> Any idea how I can get rid of them? Or redeploy them? Additionally I'm just in the middle of an upgrade. { "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon", "osd" ], "progress": "18/40 daemons upgraded", "message": "Upgrade paused", "is_paused": true } I paused it on purpose to allow for manipulation of daemons. Cheers, Thomas