I have a pre-production cluster running on Ubuntu 22.04 hosts using cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which went completely smoothly. We are a government facility so we are required to enable FIPS mode in the OS. I believe this last fact may be the cause of the following: I wanted to enable mgmt-gateway so I made a service definition file to enable it and applied it. It appeared to work, but then “ceph health detail” showed: HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported The mgmt-gateway service was started on all 3 nodes however. $ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.alfred alfred running (4m) 3m ago 4m 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (4m) 3m ago 4m 4452k - 1.26.1 c2348906c6ff 329705c23aad I then attempted to remove the service definition with ceph orch rm, but it only succeeded in removing it on 1 of the 3 nodes. The process is still running on the other 2, and the warning is still present. “ceph mgr fail && ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it comes right back, and the process is still present on 2 nodes (as shown by “ceph orch ps”). $ ceph mgr fail && ceph orch rm mgmt-gateway Removed service mgmt-gateway $ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (15h) 15h ago 15h 4448k - 1.26.1 c2348906c6ff 329705c23aad $ ceph health detail HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported Despite the 2 processes running, the configured service IP does not respond to a ping or requests from curl. Since we were getting ready to move this cluster into production I decided to look for help here rather than proceed experimenting further and risk mucking it up worse. How can I (a) diagnose what’s actually going wrong; (b) recover the cephadm module to get back to normal with no warnings; and less importantly, (c) is there a way to get mgmt-gateway working in this environment? A little googling and AI querying suggests it may be an issue with certs/ciphers so maybe I can provide my own? (Yes I know I could set up my own proxy; I was just trying to use the built-in tools where possible.) Thanks in advance.