- ceph-users - lists.ceph.io

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Eugen Block 24 Apr '26

24 Apr '26

Yeah, the message > [ 171.485542] ceph: No mds server is up or the cluster is laggy can be misleading sometimes. But I'd recommend to create a new thread since the original issue is resolved. Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: >> >> It seems like removing packages did the trick, and there are up OSDs. > > > Apparently yes. I've already placed some test files in the cluster, and the > test went well. I've also added the cluster as a mount in /etc/fstab. The > mount itself was performed after mount -a, but not on startup. > > There is only one thing I've just found in the syslog at startup: > > [ 150.710312] ceph: loaded (mds proto 32) > [ 150.745343] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state > V1_BANN > ER) > [ 151.016656] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state > V1_BANN > ER) > [ 151.520015] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state > V1_BANN > ER) > [ 157.642865] libceph: mon3 (1)10.20.0.14:6789 session established > [ 157.645794] libceph: client274103 fsid > 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > [ 157.646023] libceph: mds0 (1)10.20.0.11:6801 socket error on write > [ 157.919274] libceph: mds0 (1)10.20.0.11:6801 socket error on write > [ 158.426046] libceph: mds0 (1)10.20.0.11:6801 socket error on write > [ 171.485479] ceph: mds0 reconnect start > [ 171.485542] ceph: No mds server is up or the cluster is laggy > > This contradicts the output of ceph -s, which still says, HEALTH_OK. > > Am Fr., 24. Apr. 2026 um 00:38 Uhr schrieb Anthony D'Atri via ceph-users < > ceph-users(a)ceph.io>: > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 24 Apr '26

24 Apr '26

> > It seems like removing packages did the trick, and there are up OSDs. Apparently yes. I've already placed some test files in the cluster, and the test went well. I've also added the cluster as a mount in /etc/fstab. The mount itself was performed after mount -a, but not on startup. There is only one thing I've just found in the syslog at startup: [ 150.710312] ceph: loaded (mds proto 32) [ 150.745343] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state V1_BANN ER) [ 151.016656] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state V1_BANN ER) [ 151.520015] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state V1_BANN ER) [ 157.642865] libceph: mon3 (1)10.20.0.14:6789 session established [ 157.645794] libceph: client274103 fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b [ 157.646023] libceph: mds0 (1)10.20.0.11:6801 socket error on write [ 157.919274] libceph: mds0 (1)10.20.0.11:6801 socket error on write [ 158.426046] libceph: mds0 (1)10.20.0.11:6801 socket error on write [ 171.485479] ceph: mds0 reconnect start [ 171.485542] ceph: No mds server is up or the cluster is laggy This contradicts the output of ceph -s, which still says, HEALTH_OK. Am Fr., 24. Apr. 2026 um 00:38 Uhr schrieb Anthony D'Atri via ceph-users < ceph-users(a)ceph.io>: > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Frédéric Nass 23 Apr '26

23 Apr '26

Hi Anthony, Actually, ceph-bluestore-tool and other useful tools are embedded in the ceph container image (even tools like smartctl), so the only command the operator might want to have on a host is cephadm, installed either from cephadm package when available, or downloaded via curl/wget, or the operator could just chmod +x /var/lib/ceph/$(ceph fsid)/cephadm.* that is pushed by the orchestrator on every node and run it. Cheers, Frédéric. -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass(a)clyso.com Le jeu. 23 avr. 2026 à 03:06, Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> a écrit : > > > > On Apr 22, 2026, at 12:04 PM, Jacek Rużyczka via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > Hi Eugen, > > Hi Frédéric, > > > > One after the other, please: > > > > Do you have the ceph-osd package installed on the servers? It's known > >> to be a potential issue when trying to use cephadm while ceph packages > >> are installed (except for cephadm, of course, and maybe ceph-common). > > My understanding is that the orchestrator uses a different copy of cephadm > than what the package installs, in order to avoid conflicts. > > Issues with `ceph-osd` installed are problematic, though, as that's neeed > to get ceph-bluestore-tool. Is there a tracker for this? > > > And just one thought on the number of monitors: It's not useful to > >> have 4 MONs, an odd number is recommended. With 4 MONs you'd still > >> lose quorum when two MONs go down, so it doesn't offer any more > >> resiliency than 3 MONs. > > > > > > Shall I reduce it to 3? > > If you have at least 5 hosts, I would increase to 5. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: pg in 'backfill_wait+backfill_toofull' (and backfilling not converging)
by Wesley Dillingham 23 Apr '26

23 Apr '26

You should increase the number of PGs in that pool. From what I can see these OSDs only have about 12 PGs/OSD and you want to shoot for closer to 100 (usually over IMO), increase that pools pg_num as the pg autoscaler is telling you. Respectfully, *Wes Dillingham* On Thu, Apr 23, 2026 at 1:38 PM Massimo Sgaravatto via ceph-users < ceph-users(a)ceph.io> wrote: > Dear all > > I (try to) manage a ceph cluster which is used for different use cases: > cephfs, block storage for OpenStack and for a proxmox cluster, object > storage. > > I have a hdd-cephfs device class used only for the OSDs for the cephfs data > pool (cephfs_data). > > > There has been a massive writing to the cephfs file system and so the pool > and one OSD of that device class became 'nearfull' > > To try to address the problem I then moved 3 osds to the hdd-cephfs device > class. > > During the rebalance: > > - there was a short down of one node hosting one of the OSD belonging to > the hdd-cephfs device class > - many files in the cephfs file system got deleted > > > Now I have an on going backfill that doesn't seem to converge: > > pgs: 8824966/525185097 objects misplaced (1.680%) > 2580 active+clean > 17 active+remapped+backfill_wait > 13 active+clean+scrubbing+deep > 13 active+clean+scrubbing > 2 active+remapped+backfilling > 1 active+remapped+backfill_wait+backfill_toofull > > > and one pg is is in active+remapped+backfill_wait+backfill_toofull > > > I am also told that the pool needs more pgs: > > [root@ceph-mon-01 ~]# ceph health detail > HEALTH_WARN Low space hindering backfill (add storage if this doesn't > resolve itself): 1 pg backfill_toofull; 1 pools have too few placement > group\ > s > [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this > doesn't resolve itself): 1 pg backfill_toofull > pg 60.4 is active+remapped+backfill_wait+backfill_toofull, acting > [10,189,110] > [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups > Pool cephfs_data has 32 placement groups, should have 128 > > > These are the details of the pool: > > [root@ceph-mon-01 ~]# ceph df | grep 'cephfs_data ' > cephfs_data 60 32 16 TiB 9.48M 48 TiB 47.39 > 18 TiB > [root@ceph-mon-01 ~]# > > > pool 60 'cephfs_data' replicated size 3 min_size 2 crush_rule 11 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change > 4259332\ > flags hashpspool stripe_width 0 application cephfs read_balance_score 1.88 > > > and this [*] is the status of the OSDs belonging to that device class. > > As I said, the backfilling has been running for several days and doesn't > seem to get finalized (the number of misplaced objects even increases from > time to time, maybe because of the balancer ?) > > > > Should I simply be patient and wait for the end of the backfilling ? > What about the pg in active+remapped+backfill_wait+backfill_toofull ? > Should it "fix" by itself ? > > Or should I increase now the number of pgs of that pool ? > > Any suggestion will be highly appreciated ! > > Thanks, Massimo > > > > > [*] > > [root@ceph-mon-01 ~]# ceph osd df | grep cephfs > 70 hdd-cephfs 9.18259 1.00000 9.2 TiB 1.8 TiB 1.7 TiB 92 KiB > 2.6 GiB 7.4 TiB 19.08 0.59 3 up > 110 hdd-cephfs 11.00099 1.00000 11 TiB 6.5 TiB 6.4 TiB 114 KiB > 9.8 GiB 4.5 TiB 59.36 1.85 13 up > 130 hdd-cephfs 14.63959 1.00000 15 TiB 7.0 TiB 6.9 TiB 11 MiB > 10 GiB 7.6 TiB 47.98 1.49 14 up > 0 hdd-cephfs 18.26270 1.00000 18 TiB 5.9 TiB 5.8 TiB 10 MiB > 9.8 GiB 12 TiB 32.42 1.01 12 up > 189 hdd-cephfs 21.91499 1.00000 22 TiB 6.0 TiB 5.9 TiB 10 MiB > 10 GiB 16 TiB 27.31 0.85 12 up > 10 hdd-cephfs 14.72659 1.00000 15 TiB 6.0 TiB 5.9 TiB 10 MiB > 9.9 GiB 8.7 TiB 40.93 1.27 12 up > 25 hdd-cephfs 14.72659 1.00000 15 TiB 6.1 TiB 5.9 TiB 10 MiB > 10 GiB 8.6 TiB 41.56 1.29 12 up > 26 hdd-cephfs 14.72659 1.00000 15 TiB 1.3 TiB 1.1 TiB 10 MiB > 2.5 GiB 13 TiB 8.60 0.27 2 up > 170 hdd-cephfs 14.72659 1.00000 15 TiB 8.0 TiB 7.8 TiB 10 MiB > 13 GiB 6.8 TiB 54.10 1.68 16 up > 171 hdd-cephfs 14.72659 1.00000 15 TiB 368 GiB 190 GiB 11 MiB > 1.3 GiB 14 TiB 2.44 0.08 0 up > [root@ceph-mon-01 ~]# > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Eugen Block 23 Apr '26

23 Apr '26

Oh, apparently the last messages weren’t in reply to the list. It seems like removing packages did the trick, and there are up OSDs. Zitat von Eugen Block <eblock(a)nde.ag>: > Yeah, looks like it worked. :-) > > The IDs are part of the crush tree, you don’t need to worry about > them. I assume that you could also revert the unmanaged step (Ceph > orch set-managed …) and the rest of the OSDs would probably be > created as well. > > Zitat von Jacek Rużyczka <jacek.ruzyczka(a)googlemail.com>: > >>> >>> I would aim at keeping all nodes in the same state, so yes, remove >>> them from all Ceph nodes. >> >> >> Yep. >> >> Maybe before you reboot, zap the drives first, the >>> ones the orchestrator might have tried to build and failed. >> >> >> Already done that. >> >> ceph orch set-unmanaged osd.<your_osd_service> >> >> >> Do you mean this: >> >> root@blade3n1:/# ceph orch set-unmanaged osd.all-available-devices >> Set unmanaged to True for service osd.all-available-devices >> >> To deploy a single OSD you can run: >> >> >>> ceph orch daemon add osd <host> /dev/sdX >> >> >> So something like this: >> >> mixtile@blade3n1:~$ sudo ceph orch daemon add osd blade3n1:/dev/nvme0n1 >> Created osd(s) 3 on host 'blade3n1' >> >> But: >> >> mixtile@blade3n1:~$ sudo ceph orch daemon add osd blade3n2:/dev/nvme0n1 >> Created no osd(s) on host blade3n2; already created? >> >> Before I could disable the OSD process, it must have started creating OSDs >> on the other hosts: >> >> mixtile@blade3n1:~$ sudo ceph osd tree >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> -1 29.80798 root default >> -9 7.45200 host blade3n1 >> 3 ssd 7.45200 osd.3 up 1.00000 1.00000 >> -7 7.45200 host blade3n2 >> 2 ssd 7.45200 osd.2 up 1.00000 1.00000 >> -5 7.45200 host blade3n3 >> 1 ssd 7.45200 osd.1 up 1.00000 1.00000 >> -3 7.45200 host blade3n4 >> 0 ssd 7.45200 osd.0 up 1.00000 1.00000 >> >> But: What do the negative IDs mean? Yesterday, I only had the root >> default with >> ID -1. At least, the health check now says, HEALTH_OK: >> >> mixtile@blade3n1:~$ sudo ceph -s >> cluster: >> id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b >> health: HEALTH_OK >> >> services: >> mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) >> mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv >> mds: 1/1 daemons up, 1 standby >> osd: 4 osds: 4 up (since 10m), 4 in (since 60m) >> >> data: >> volumes: 1/1 healthy >> pools: 3 pools, 145 pgs >> objects: 24 objects, 579 KiB >> usage: 248 MiB used, 30 TiB / 30 TiB avail >> pgs: 145 active+clean >> >> So: Do I now have a working Ceph cluster?

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Frédéric Nass 23 Apr '26

23 Apr '26

Yes, Jacek. That cephadm file is the one the orchestrator pushes to every node and works with. But as Eugen said, that doesn't matter much. Could you share the output of 'id ceph' (on the Linux host) and 'id ceph' (in the container), and check for any UID mismatch? And follow Eugen's advice to remove the ceph packages? Best, Frédéric. -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass(a)clyso.com Le jeu. 23 avr. 2026 à 15:35, Jacek Rużyczka via ceph-users < ceph-users(a)ceph.io> a écrit : > Hi Frédéric, > > Do you mean this file? At least it's the only cephadm I've found in the > folder you told me: > > mixtile@blade3n2:~$ sudo ls -al > /var/lib/ceph/8aad3073-39a1-11f1-bf6e-f2704a1efa > 9b | grep cephadm > -rw-r--r-- 1 root root 787672 Apr 16 16:52 cephadm > .1a8853661a9c1798390b8e8 > d13c27688c1b1327a075745af2ee40ac466f0ac36 > > Regards > Jacek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: squid 19.2.4 QE validation status
by Laura Flores 23 Apr '26

23 Apr '26

The rerun of the rados suite on Build 3 looks good. Did not do a full analysis, but I do not see anything related to the ceph-volume blocker. RADOS approved. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Venky Shankar via dev <dev(a)ceph.io> Sent: Thursday, April 23, 2026 12:24 PM To: Yuri Weinstein <yweinste(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: squid 19.2.4 QE validation status Hi Yuri, On Tue, Apr 14, 2026 at 1:01 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky fs approved. failures are https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_proje… > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA -- Cheers, Venky

1 0

pg in 'backfill_wait+backfill_toofull' (and backfilling not converging)
by Massimo Sgaravatto 23 Apr '26

23 Apr '26

Dear all I (try to) manage a ceph cluster which is used for different use cases: cephfs, block storage for OpenStack and for a proxmox cluster, object storage. I have a hdd-cephfs device class used only for the OSDs for the cephfs data pool (cephfs_data). There has been a massive writing to the cephfs file system and so the pool and one OSD of that device class became 'nearfull' To try to address the problem I then moved 3 osds to the hdd-cephfs device class. During the rebalance: - there was a short down of one node hosting one of the OSD belonging to the hdd-cephfs device class - many files in the cephfs file system got deleted Now I have an on going backfill that doesn't seem to converge: pgs: 8824966/525185097 objects misplaced (1.680%) 2580 active+clean 17 active+remapped+backfill_wait 13 active+clean+scrubbing+deep 13 active+clean+scrubbing 2 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull and one pg is is in active+remapped+backfill_wait+backfill_toofull I am also told that the pool needs more pgs: [root@ceph-mon-01 ~]# ceph health detail HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; 1 pools have too few placement group\ s [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull pg 60.4 is active+remapped+backfill_wait+backfill_toofull, acting [10,189,110] [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool cephfs_data has 32 placement groups, should have 128 These are the details of the pool: [root@ceph-mon-01 ~]# ceph df | grep 'cephfs_data ' cephfs_data 60 32 16 TiB 9.48M 48 TiB 47.39 18 TiB [root@ceph-mon-01 ~]# pool 60 'cephfs_data' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 4259332\ flags hashpspool stripe_width 0 application cephfs read_balance_score 1.88 and this [*] is the status of the OSDs belonging to that device class. As I said, the backfilling has been running for several days and doesn't seem to get finalized (the number of misplaced objects even increases from time to time, maybe because of the balancer ?) Should I simply be patient and wait for the end of the backfilling ? What about the pg in active+remapped+backfill_wait+backfill_toofull ? Should it "fix" by itself ? Or should I increase now the number of pgs of that pool ? Any suggestion will be highly appreciated ! Thanks, Massimo [*] [root@ceph-mon-01 ~]# ceph osd df | grep cephfs 70 hdd-cephfs 9.18259 1.00000 9.2 TiB 1.8 TiB 1.7 TiB 92 KiB 2.6 GiB 7.4 TiB 19.08 0.59 3 up 110 hdd-cephfs 11.00099 1.00000 11 TiB 6.5 TiB 6.4 TiB 114 KiB 9.8 GiB 4.5 TiB 59.36 1.85 13 up 130 hdd-cephfs 14.63959 1.00000 15 TiB 7.0 TiB 6.9 TiB 11 MiB 10 GiB 7.6 TiB 47.98 1.49 14 up 0 hdd-cephfs 18.26270 1.00000 18 TiB 5.9 TiB 5.8 TiB 10 MiB 9.8 GiB 12 TiB 32.42 1.01 12 up 189 hdd-cephfs 21.91499 1.00000 22 TiB 6.0 TiB 5.9 TiB 10 MiB 10 GiB 16 TiB 27.31 0.85 12 up 10 hdd-cephfs 14.72659 1.00000 15 TiB 6.0 TiB 5.9 TiB 10 MiB 9.9 GiB 8.7 TiB 40.93 1.27 12 up 25 hdd-cephfs 14.72659 1.00000 15 TiB 6.1 TiB 5.9 TiB 10 MiB 10 GiB 8.6 TiB 41.56 1.29 12 up 26 hdd-cephfs 14.72659 1.00000 15 TiB 1.3 TiB 1.1 TiB 10 MiB 2.5 GiB 13 TiB 8.60 0.27 2 up 170 hdd-cephfs 14.72659 1.00000 15 TiB 8.0 TiB 7.8 TiB 10 MiB 13 GiB 6.8 TiB 54.10 1.68 16 up 171 hdd-cephfs 14.72659 1.00000 15 TiB 368 GiB 190 GiB 11 MiB 1.3 GiB 14 TiB 2.44 0.08 0 up [root@ceph-mon-01 ~]#

1 0

Re: squid 19.2.4 QE validation status
by Venky Shankar 23 Apr '26

23 Apr '26

Hi Yuri, On Tue, Apr 14, 2026 at 1:01 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75029#note-5 > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky fs approved. failures are https://tracker.ceph.com/projects/cephfs/wiki/Squid#Squid-v1924 > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA -- Cheers, Venky

1 0

Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Barber, Steve W. (Fed) 23 Apr '26

23 Apr '26

I have a pre-production cluster running on Ubuntu 22.04 hosts using cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which went completely smoothly. We are a government facility so we are required to enable FIPS mode in the OS. I believe this last fact may be the cause of the following: I wanted to enable mgmt-gateway so I made a service definition file to enable it and applied it. It appeared to work, but then “ceph health detail” showed: HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported The mgmt-gateway service was started on all 3 nodes however. $ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.alfred alfred running (4m) 3m ago 4m 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (4m) 3m ago 4m 4452k - 1.26.1 c2348906c6ff 329705c23aad I then attempted to remove the service definition with ceph orch rm, but it only succeeded in removing it on 1 of the 3 nodes. The process is still running on the other 2, and the warning is still present. “ceph mgr fail && ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it comes right back, and the process is still present on 2 nodes (as shown by “ceph orch ps”). $ ceph mgr fail && ceph orch rm mgmt-gateway Removed service mgmt-gateway $ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (15h) 15h ago 15h 4448k - 1.26.1 c2348906c6ff 329705c23aad $ ceph health detail HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported Despite the 2 processes running, the configured service IP does not respond to a ping or requests from curl. Since we were getting ready to move this cluster into production I decided to look for help here rather than proceed experimenting further and risk mucking it up worse. How can I (a) diagnose what’s actually going wrong; (b) recover the cephadm module to get back to normal with no warnings; and less importantly, (c) is there a way to get mgmt-gateway working in this environment? A little googling and AI querying suggests it may be an issue with certs/ciphers so maybe I can provide my own? (Yes I know I could set up my own proxy; I was just trying to use the built-in tools where possible.) Thanks in advance.

1 0