Hi, I have some orchestrator issues on our cluster running 16.2.9 with rgw only services. We first noticed these issues a few weeks ago when adding new hosts to the cluster - the orch was not detecting the new drives to build the osd containers for them. Debugging the mgr logs, I noticed that the mgr was crashing due to the dashboard module. I disabled the dashboard module and the new drives were detected and added to the cluster. Now we have other similar issues : we have a failed drive. The failure was detected , the osd was marked as down and the rebalancing is finished. I want to remove the failed osd from the cluster but it looks like the orch is not working : - I launched the osd removal with 'ceph orch osd rm 92 --force' where 92 is the osd id in question - I checked the progress but nothing happens even after a few days : ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT *92 node10 started 0 False True False* - the osd process is stopped on that host and from the orch side I can see this : ceph orch ps --daemon_type osd --daemon_id 92 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID osd.92 node10 error 11h ago 4w - 4096M <unknown> <unknown> - I have the same long refresh interval on other osds as well. I know it should be 10 minutes or so : NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.93 node09 running (4w) 11h ago 4w 5573M 4096M 16.2.9 3520ead5eb19 d2f658e9e37b osd: *116 osds: 115 up* (since 11d), 115 in (since 11d) 90 hdd 16.37109 1.00000 16 TiB 4.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 24.66 0.97 146 up * 92 hdd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down* 94 hdd 16.37109 1.00000 16 TiB 4.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 24.66 0.97 146 up - I activated debug 20 on the mgr but I can't see any errors or other clues regarding the osd removal. I also switched to the standby manager with 'ceph mgr fail'. The mgr switch works but still nothing happens - It's not only the osd removal thing. I also tried to deploy new rgw services by applying rgw labels on 2 new hosts , we have specs for building rgw containers when detecting the label. Again, nothing happens. I'm planning to upgrade to 16.2.11 to see if this solves the issues but I'm not very confident, I didn't see anything regarding this in the changelogs. Is there anything else I can try to debug this issue ? Thanks.