Hi Steve, I'd recommend (if possible) disabling FIPS, then forcing a mgr fail-over to recover cephadm control. After that, you should be able to use cephadm commands to fix/remove the failing services. Best, Redo. On Wed, Apr 29, 2026 at 4:20 PM Barber, Steve W. (Fed) via ceph-users < ceph-users@ceph.io> wrote:

Thank you Redo!

Do you have any suggestions for how I can back out of the current situation to get rid of the health warning and the orphaned mgmt-gateway services without making a mess of things?

Steve

From: Redouane Kachach via ceph-users <ceph-users@ceph.io> Date: Tuesday, April 28, 2026 at 4:25 AM To: Barber, Steve W. (Fed) <steve.barber@nist.gov> Cc: Ceph Users <ceph-users@ceph.io> Subject: [EXTERNAL] [ceph-users] Re: [EXTERNAL] Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken

Hi Steve,

I saw that you already opened a ticket for the issue (thanks):

https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.ceph.com%2Fissues%2F76185&data=05%7C02%7Csteve.barber%40nist.gov%7Cd7f3679df7f54a3cc64508dea4ffb6e5%7C2ab5d82fd8fa4797a93e054655c61dec%7C0%7C0%7C639129615369743473%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=78HdQke5beaeIBmY1sZxKORgPr5Hcto1otdvR4Vx%2B%2BM%3D&reserved=0 <https://tracker.ceph.com/issues/76185>.

This problem is basically related to the env (FIPS is enabled). A PR with a candidate fix is under-review:

https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fceph%2Fceph%2Fpull%2F68638&data=05%7C02%7Csteve.barber%40nist.gov%7Cd7f3679df7f54a3cc64508dea4ffb6e5%7C2ab5d82fd8fa4797a93e054655c61dec%7C0%7C0%7C639129615369772151%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XRByuCtfdEElL0iUmkkK%2BJO5RGHes56dpOyy4KSqeqQ%3D&reserved=0 <https://github.com/ceph/ceph/pull/68638>.

We will backport this change to Tentacle.

Best, Redo.

On Mon, Apr 27, 2026 at 5:02 PM Barber, Steve W. (Fed) < steve.barber@nist.gov> wrote:

...

Hi Redo,

The spec was very simple:

service_type: mgmt-gateway placement: #label: mgmt hosts: - alfred - cyclops - mask spec: virtual_ip: (redacted)/24 port: 443

As far as active mgr logs, at this point I’m not seeing anything unusual in the logs from the active mgr host. If I attempt to remove the service again (it was already theoretically removed) I get:

$ ceph orch rm mgmt-gateway Invalid service 'mgmt-gateway'. Use 'ceph orch ls' to list available services.

That generated these lines in the log:

... Apr 27 10:35:03 alfred bash[3568828]: debug 2026-04-27T14:35:03.592+0000 7f21e7e80640 0 log_channel(audit) log [DBG] : from=‘mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch Apr 27 10:35:04 alfred bash[3568828]: audit 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch Apr 27 10:35:04 alfred bash[3568828]: audit 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch Apr 27 10:35:06 alfred bash[3566093]: debug 2026-04-27T14:35:06.596+0000 7f4324bbe640 0 log_channel(audit) log [DBG] : from=‘client.42505047 -' entity='client.admin' cmd=[{"prefix": "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: dispatch Apr 27 10:35:07 alfred bash[3568828]: audit 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : audit [DBG] from='client.42505047 -' entity='client.admin' cmd=[{"prefix": "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: dispatch Apr 27 10:35:07 alfred bash[3568828]: audit 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : audit [DBG] from='client.42505047 -' entity='client.admin' cmd=[{"prefix": "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: dispatch Apr 27 10:35:07 alfred bash[3568828]: cephadm 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : cephadm [INF] Remove service mgmt-gateway Apr 27 10:35:07 alfred bash[3568828]: cephadm 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : cephadm [INF] Remove service mgmt.-gateway Apr 27 10:35:18 alfred bash[3568828]: debug 2026-04-27T14:35:18.596+0000 7f21e7e80640 0 log_channel(audit) log [DBG] : from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch Apr 27 10:35:19 alfred bash[3568828]: audit 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch Apr 27 10:35:19 alfred bash[3568828]: audit 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch ...

But otherwise everything else looks pretty normal.

This is still with 2 of the 3 mon/mgr nodes running the mgmt-gateway service: $ ceph orch ps | egrep 'mgr|mon|mgmt' mgmt-gateway.cyclops cyclops running (6d) 5d ago 6d 6188k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (6d) 5d ago 6d 5980k - 1.26.1 c2348906c6ff 329705c23aad mgr.alfred.afvnok alfred *:8443,9283,8765 running (2w) 5d ago 7M 564M - 20.2.1 fb63cba66eea 875883d08efc mgr.cyclops.nifeym cyclops *:8443,9283,8765 running (2w) 5d ago 7M 463M - 20.2.1 fb63cba66eea 676ec1753348 mgr.mask.yfbdpf mask *:8443,9283,8765 running (2w) 5d ago 7M 486M - 20.2.1 fb63cba66eea 6a1bdd37997c mon.alfred alfred running (2w) 5d ago 7M 525M 2048M 20.2.1 fb63cba66eea ce5bbb75d650 mon.cyclops cyclops running (2w) 5d ago 7M 571M 2048M 20.2.1 fb63cba66eea e675cf6714e4 mon.mask mask running (2w) 5d ago 7M 510M 2048M 20.2.1 fb63cba66eea 9a034ab6f132

And I still have the same health error.

Let me know if there’s anything more specific you want me to show.

Steve

*From: *Redouane Kachach <rkachach@redhat.com> *Date: *Friday, April 24, 2026 at 4:32 AM *To: *Barber, Steve W. (Fed) <steve.barber@nist.gov> *Cc: *Ceph Users <ceph-users@ceph.io> *Subject: *[EXTERNAL] Re: [ceph-users] Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken

Hi Steve,

Can you plz share the spec you used to configure the mgmt-gateway and the active mgr logs?

Best, Redo.

On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < ceph-users@ceph.io> wrote:

I have a pre-production cluster running on Ubuntu 22.04 hosts using cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which went completely smoothly. We are a government facility so we are required to enable FIPS mode in the OS. I believe this last fact may be the cause of the following:

I wanted to enable mgmt-gateway so I made a service definition file to enable it and applied it. It appeared to work, but then “ceph health detail” showed:

HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported

The mgmt-gateway service was started on all 3 nodes however.

$ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.alfred alfred running (4m) 3m ago 4m 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (4m) 3m ago 4m 4452k - 1.26.1 c2348906c6ff 329705c23aad

I then attempted to remove the service definition with ceph orch rm, but it only succeeded in removing it on 1 of the 3 nodes. The process is still running on the other 2, and the warning is still present. “ceph mgr fail && ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it comes right back, and the process is still present on 2 nodes (as shown by “ceph orch ps”).

$ ceph mgr fail && ceph orch rm mgmt-gateway Removed service mgmt-gateway

$ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (15h) 15h ago 15h 4448k - 1.26.1 c2348906c6ff 329705c23aad

$ ceph health detail HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported

Despite the 2 processes running, the configured service IP does not respond to a ping or requests from curl.

Since we were getting ready to move this cluster into production I decided to look for help here rather than proceed experimenting further and risk mucking it up worse. How can I (a) diagnose what’s actually going wrong; (b) recover the cephadm module to get back to normal with no warnings; and less importantly, (c) is there a way to get mgmt-gateway working in this environment? A little googling and AI querying suggests it may be an issue with certs/ciphers so maybe I can provide my own? (Yes I know I could set up my own proxy; I was just trying to use the built-in tools where possible.)

Thanks in advance.

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io