- ceph-users - lists.ceph.io

Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Barber, Steve W. (Fed) 23 Apr '26

23 Apr '26

I have a pre-production cluster running on Ubuntu 22.04 hosts using cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which went completely smoothly. We are a government facility so we are required to enable FIPS mode in the OS. I believe this last fact may be the cause of the following: I wanted to enable mgmt-gateway so I made a service definition file to enable it and applied it. It appeared to work, but then “ceph health detail” showed: HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported The mgmt-gateway service was started on all 3 nodes however. $ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.alfred alfred running (4m) 3m ago 4m 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (4m) 3m ago 4m 4452k - 1.26.1 c2348906c6ff 329705c23aad I then attempted to remove the service definition with ceph orch rm, but it only succeeded in removing it on 1 of the 3 nodes. The process is still running on the other 2, and the warning is still present. “ceph mgr fail && ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it comes right back, and the process is still present on 2 nodes (as shown by “ceph orch ps”). $ ceph mgr fail && ceph orch rm mgmt-gateway Removed service mgmt-gateway $ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (15h) 15h ago 15h 4448k - 1.26.1 c2348906c6ff 329705c23aad $ ceph health detail HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported Despite the 2 processes running, the configured service IP does not respond to a ping or requests from curl. Since we were getting ready to move this cluster into production I decided to look for help here rather than proceed experimenting further and risk mucking it up worse. How can I (a) diagnose what’s actually going wrong; (b) recover the cephadm module to get back to normal with no warnings; and less importantly, (c) is there a way to get mgmt-gateway working in this environment? A little googling and AI querying suggests it may be an issue with certs/ciphers so maybe I can provide my own? (Yes I know I could set up my own proxy; I was just trying to use the built-in tools where possible.) Thanks in advance.

1 0

Ceph Day London CFP - Call for Papers
by Lee Sanders 23 Apr '26

23 Apr '26

Hi everyone! Just a reminder the CFP for Ceph Days London closes on Friday 24th April (tomorrow) at around noon UK time, if you are interested in being a speaker at this event please submit your talk here: https://ceph.io/en/community/events/2026/ceph-days-london/ Thank you! Lee Sanders Lee Sanders. Performance Architect, IBM Master Inventor, Ceph Development Email: mailto:ljsanders@uk.ibm.com CSOA10, IBM Hursley Park Badges: https://www.credly.com/users/lee-sanders/badges Hursley, SO21 2JN United Kingdom Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN

1 0

squid 19.2.4 QE validation status
by Yuri Weinstein 23 Apr '26

23 Apr '26

Details of this release are summarized here: https://tracker.ceph.com/issues/75029#note-5 Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/squid-x: Laura, Patrick upgrade/reef-x: Laura, Patrick upgrade/telemetry-upgrade: Laura, Patrick upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 3

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Eugen Block 23 Apr '26

23 Apr '26

I'd recommend to remove these: - ceph-base - ceph-mds - ceph-osd - ceph-volume Don't bother too much about the cephadm binary right now, it's useful to know but not that relevant to get your cluster into a proper state. Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > OK, this is what I've got on all four nodes: > > mixtile@blade3n2:~$ sudo apt list --installed | grep ceph > [sudo] password for mixtile: > > WARNING: apt does not have a stable CLI interface. Use with caution in > scripts. > > ceph-base/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed,automatic] > ceph-common/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed] > ceph-mds/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed,automatic] > ceph-osd/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed] > ceph-volume/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 all > [installed] > cephadm/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed] > libcephfs2/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed,automatic] > python3-ceph-argparse/noble-updates,noble-security,now > 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] > python3-ceph-common/noble-updates,noble-security,now > 19.2.3-0ubuntu0.24.04.3 all [installed,automatic] > python3-cephfs/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 > arm64 [installed,automatic] > > As you can see, most packages have been installed automatically, probably > because of a dep. *Which of them shall I remove on hosts #2, #3, and #4?* > > And: Unfortunately, I've only got 4 hosts. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 23 Apr '26

23 Apr '26

Hi Frédéric, Do you mean this file? At least it's the only cephadm I've found in the folder you told me: mixtile@blade3n2:~$ sudo ls -al /var/lib/ceph/8aad3073-39a1-11f1-bf6e-f2704a1efa 9b | grep cephadm -rw-r--r-- 1 root root 787672 Apr 16 16:52 cephadm .1a8853661a9c1798390b8e8 d13c27688c1b1327a075745af2ee40ac466f0ac36 Regards Jacek

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 23 Apr '26

23 Apr '26

OK, this is what I've got on all four nodes: mixtile@blade3n2:~$ sudo apt list --installed | grep ceph [sudo] password for mixtile: WARNING: apt does not have a stable CLI interface. Use with caution in scripts. ceph-base/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] ceph-common/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed] ceph-mds/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] ceph-osd/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed] ceph-volume/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 all [installed] cephadm/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed] libcephfs2/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] python3-ceph-argparse/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] python3-ceph-common/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 all [installed,automatic] python3-cephfs/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] As you can see, most packages have been installed automatically, probably because of a dep. *Which of them shall I remove on hosts #2, #3, and #4?* And: Unfortunately, I've only got 4 hosts.

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 23 Apr '26

23 Apr '26

IIRC, I think I was able to reproduce mon sync issues back then on Quiny or Octopus on fresh (virtual) deployments, so I don't think it's related to the cluster age. :-) But it's been too long, I remember only vaguely. What I remember is I never could pinpoint to a specific root cause. And I also remember other users having issues as well, reported here on the list. So it seems like a broken upgrade path to Q, I would say. I guess we're lucky that we mostly upgraded our own and our customers clusters from N to P, then P to R. We usually skip one release, except last year when we upgraded our own cluster from R to S. Zitat von Janne Johansson via ceph-users <ceph-users(a)ceph.io>: >> Correct! It's an old cluster initialy deployed with ceph-deploy in the >> Hammer days and upgraded since then. Right now is manually managed (but >> one of the targets at the end of the upgrade is to switch to cephadm). > >> We didn't wait for a long time before stopping the new MON a regain >> "control" of the cluster, maybe 10 minutes max... >> >> > - Inspecting debug logs on the syncing MON could help finding the root >> > cause. >> >> With "syncing MON" you mean the MON which is just added to the cluster, >> right? Collecting this kind of logs is on the to-do list, but blocked >> because the "log problem" mentioned bellow. > > Sorry for coming in late, but we also had troubles upgrading a cluster > from Quincy from Octopus, which coincided with moving from CentOS to > Ubuntu. In our case, the Oct mons started crashing and would not start > (crashing before being able to join) as soon as I had two Quincy mons > joined. At the time, we had little choice than to just upgrade the Oct > hosts to Q and then they would join. > > A potentially interesting similarity is the age of our cluster, ours > is from Infernalis or Jewel at the latest, and then upgraded in many > steps. > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Upgrade problem from Octopus to Quincy
by Janne Johansson 23 Apr '26

23 Apr '26

> Correct! It's an old cluster initialy deployed with ceph-deploy in the > Hammer days and upgraded since then. Right now is manually managed (but > one of the targets at the end of the upgrade is to switch to cephadm). > We didn't wait for a long time before stopping the new MON a regain > "control" of the cluster, maybe 10 minutes max... > > > - Inspecting debug logs on the syncing MON could help finding the root > > cause. > > With "syncing MON" you mean the MON which is just added to the cluster, > right? Collecting this kind of logs is on the to-do list, but blocked > because the "log problem" mentioned bellow. Sorry for coming in late, but we also had troubles upgrading a cluster from Quincy from Octopus, which coincided with moving from CentOS to Ubuntu. In our case, the Oct mons started crashing and would not start (crashing before being able to join) as soon as I had two Quincy mons joined. At the time, we had little choice than to just upgrade the Oct hosts to Q and then they would join. A potentially interesting similarity is the age of our cluster, ours is from Infernalis or Jewel at the latest, and then upgraded in many steps. -- May the most significant bit of your life be positive.

1 0

Re: squid 19.2.4 QE validation status
by wissem mimouna 23 Apr '26

23 Apr '26

Hi , Sorry for the inconvience , I'm just wondering if you have any ideas of when this PR https://github.com/ceph/ceph/pull/66501 will be available in a stable release ? Thanks On 22.04.2026 01:06, Yuri Weinstein via ceph-users wrote: > We need to merge two PRs: > https://github.com/ceph/ceph/pull/68526 > https://github.com/ceph/ceph/pull/68528 > and rerun upgrades > > On Tue, Apr 21, 2026 at 2:29 PM Laura Flores <lflores(a)ibm.com> wrote: > >> upgrade/quincy-x and upgrade/squid-x are approved: >> https://tracker.ceph.com/projects/rados/wiki/SQUID#v1924-httpstrackercephco… >> >> The upgrade/squid-p2p tests need to be rerun with >> https://github.com/ceph/ceph/pull/68526 included. >> >> Thanks, >> Laura >> >> *Laura Flores* >> Software Engineer; Ceph Storage >> 1 708 738 8804 Mobile >> She/Her/Hers >> >> IBM >> ------------------------------ >> *From:* Afreen via ceph-users <ceph-users(a)ceph.io> >> *Sent:* Tuesday, April 21, 2026 12:55 PM >> *To:* Yuri Weinstein <yweinste(a)redhat.com>; Redouane Kachach < >> rkachach(a)redhat.com> >> *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> >> *Subject:* [EXTERNAL] [ceph-users] Re: squid 19.2.4 QE validation status >> >> * Dashboard approved.* >> >> On Tue, Apr 14, 2026 at 1:00 AM Yuri Weinstein via dev <dev(a)ceph.io> >> wrote: >> >>> Details of this release are summarized here: >>> >>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… >>> Release Notes - TBD >>> LRC upgrade - N/A >>> >>> Seeking approvals/reviews for: >>> >>> smoke: PASSED >>> >>> rados: Radek, Laura, Afreen, Travis >>> rgw: Adam Emerson >>> fs: Venky >>> orch/ceph-volume: Redouane & Guillaume >>> rbd: Ilya >>> krbd: Ilya >>> >>> upgrade/squid-x: Laura, Patrick >>> upgrade/reef-x: Laura, Patrick >>> upgrade/telemetry-upgrade: Laura, Patrick >>> upgrade/squid-p2p: Laura, Patrick >>> >>> upgrade-clients:client-upgrade-reef-squid-squid: PASSED >>> upgrade-clients:client-upgrade-quincy-squid-squid: PASSED >>> >>> perf-basic: PASSED >>> crimson-rados: N/A >>> powercycle: PASSED >>> >>> TIA >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Eugen Block 23 Apr '26

23 Apr '26

Hi, a quick look into the code [0] shows (note that I'm not a developer): def set_mirror_info(local_cluster_id, local_fsid, remote_fs): log.info(f'setting {local_cluster_id}::{local_fsid} on remote') try: remote_fs.setxattr('/', 'ceph.mirror.info', f'cluster_id={local_cluster_id} fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE) except cephfs.Error as e: if e.errno == errno.EEXIST: My interpretation is that the setxattr command fails on the remote site. Could this be an auth caps issue? Did you verify the keyrings and their permissions? This is the only lead I got. Regards, Eugen [0] https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20cbd6a/… Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>: > Hi, > > This is issue with CephFS snapshot mirroring setup between two > clusters and would appreciate your assistance. > > Environment: > > * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) > [cid:image001.png@01DCD199.2933E2E0] > * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) > [cid:image002.png@01DCD199.2933E2E0] > * Filesystem: cephfs > * Issue: Unable to import peer bootstrap token > > Problem Description: > When attempting to import a peer bootstrap token using: > ceph fs snapshot mirror peer_bootstrap import cephfs <token> > I receive the error: > Error EEXIST: already an active peer mgr.server reply reply (17) > File exists already an active peer > However, ceph fs snapshot mirror peer_list cephfs returns an empty > list [], indicating no peers are configured. > > Steps Performed: > Initial Setup: > 1. Enabled mirroring module: ceph mgr module enable mirroring > 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror > 3. Enabled mirroring on filesystem: ceph fs snapshot mirror > enable cephfs > 4. Created remote user and bootstrap token on backup cluster > 5. Verified mirroring daemons are running: ceph fs snapshot > mirror daemon status > Troubleshooting Attempts: > 1. Reset filesystem mirroring: > ceph fs snapshot mirror disable cephfs > ceph fs snapshot mirror enable cephfs > 2. Module reset: > ceph mgr module disable mirroring > ceph mgr module enable mirroring > 3. Service recreation: > ceph orch rm cephfs-mirror > ceph orch apply cephfs-mirror > 4. Checked for stale configuration: > o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror > o ceph fs snapshot mirror dirmap cephfs returns empty > o ceph fs snapshot mirror status cephfs shows no active mirroring > > Current Status: > > * Mirroring daemons: Running and visible in daemon status > * Peer list: Empty [] > * Filesystem mirroring: Enabled > * Error persists: EEXIST despite no visible peers > > Diagnostic Information: > > * MGR logs show: mgr.server reply reply (17) File exists already > an active peer > * No connectivity issues between clusters (ports 6789, 3300, > 6800+ tested) > * Both clusters are healthy with no other issues > > Regards, > Anantha

1 0