April 2026 - ceph-users - lists.ceph.io

Re: Best "modern" approach to drain/fill a OSD server
by Janne Johansson 05 Jun '26

05 Jun '26

> If it can be useful, I'm thinking of organizing an OSD balancing happy > hour call in the next month or so. > Maybe mid-May or so on a Friday afternoon (EU time) / morning (US time). Also interested. -- May the most significant bit of your life be positive.

3 2

Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Adiga, Anantha 07 May '26

07 May '26

Hi Eugen, Thank you so much. Yes, that fixed the issue: removed the peer uuid from the primary cluster. This removed the attribute from the backup cluster as well. Then import peer bootstrap token worked. Regards, Anantha -----Original Message----- From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Thursday, April 23, 2026 12:56 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty Hi, a quick look into the code [0] shows (note that I'm not a developer): def set_mirror_info(local_cluster_id, local_fsid, remote_fs): log.info(f'setting {local_cluster_id}::{local_fsid} on remote') try: remote_fs.setxattr('/', 'ceph.mirror.info', f'cluster_id={local_cluster_id} fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE) except cephfs.Error as e: if e.errno == errno.EEXIST: My interpretation is that the setxattr command fails on the remote site. Could this be an auth caps issue? Did you verify the keyrings and their permissions? This is the only lead I got. Regards, Eugen [0] https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20cbd6a/… Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>: > Hi, > > This is issue with CephFS snapshot mirroring setup between two > clusters and would appreciate your assistance. > > Environment: > > * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) > [cid:image001.png@01DCD199.2933E2E0] > * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) > [cid:image002.png@01DCD199.2933E2E0] > * Filesystem: cephfs > * Issue: Unable to import peer bootstrap token > > Problem Description: > When attempting to import a peer bootstrap token using: > ceph fs snapshot mirror peer_bootstrap import cephfs <token> I receive > the error: > Error EEXIST: already an active peer mgr.server reply reply (17) File > exists already an active peer However, ceph fs snapshot mirror > peer_list cephfs returns an empty list [], indicating no peers are > configured. > > Steps Performed: > Initial Setup: > 1. Enabled mirroring module: ceph mgr module enable mirroring > 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror > 3. Enabled mirroring on filesystem: ceph fs snapshot mirror > enable cephfs > 4. Created remote user and bootstrap token on backup cluster > 5. Verified mirroring daemons are running: ceph fs snapshot > mirror daemon status > Troubleshooting Attempts: > 1. Reset filesystem mirroring: > ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable > cephfs > 2. Module reset: > ceph mgr module disable mirroring > ceph mgr module enable mirroring > 3. Service recreation: > ceph orch rm cephfs-mirror > ceph orch apply cephfs-mirror > 4. Checked for stale configuration: > o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror > o ceph fs snapshot mirror dirmap cephfs returns empty > o ceph fs snapshot mirror status cephfs shows no active mirroring > > Current Status: > > * Mirroring daemons: Running and visible in daemon status > * Peer list: Empty [] > * Filesystem mirroring: Enabled > * Error persists: EEXIST despite no visible peers > > Diagnostic Information: > > * MGR logs show: mgr.server reply reply (17) File exists already > an active peer > * No connectivity issues between clusters (ports 6789, 3300, > 6800+ tested) > * Both clusters are healthy with no other issues > > Regards, > Anantha _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: [EXTERNAL] Re: [EXTERNAL] Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Redouane Kachach 30 Apr '26

30 Apr '26

Hi Steve, I'd recommend (if possible) disabling FIPS, then forcing a mgr fail-over to recover cephadm control. After that, you should be able to use cephadm commands to fix/remove the failing services. Best, Redo. On Wed, Apr 29, 2026 at 4:20 PM Barber, Steve W. (Fed) via ceph-users < ceph-users(a)ceph.io> wrote: > Thank you Redo! > > Do you have any suggestions for how I can back out of the current > situation to get rid of the health warning and the orphaned mgmt-gateway > services without making a mess of things? > > Steve > > From: Redouane Kachach via ceph-users <ceph-users(a)ceph.io> > Date: Tuesday, April 28, 2026 at 4:25 AM > To: Barber, Steve W. (Fed) <steve.barber(a)nist.gov> > Cc: Ceph Users <ceph-users(a)ceph.io> > Subject: [EXTERNAL] [ceph-users] Re: [EXTERNAL] Re: Attempt to enable > mgmt-gateway on v20.2.1 failed, and now cephadm module is broken > > Hi Steve, > > I saw that you already opened a ticket for the issue (thanks): > > > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.c… > <https://tracker.ceph.com/issues/76185>. > > This problem is basically related to the env (FIPS is enabled). A PR with a > candidate fix is under-review: > > > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co… > <https://github.com/ceph/ceph/pull/68638>. > > We will backport this change to Tentacle. > > Best, > Redo. > > > > On Mon, Apr 27, 2026 at 5:02 PM Barber, Steve W. (Fed) < > steve.barber(a)nist.gov> wrote: > > > Hi Redo, > > > > The spec was very simple: > > > > service_type: mgmt-gateway > > placement: > > #label: mgmt > > hosts: > > - alfred > > - cyclops > > - mask > > spec: > > virtual_ip: (redacted)/24 > > port: 443 > > > > As far as active mgr logs, at this point I’m not seeing anything unusual > > in the logs from the active mgr host. If I attempt to remove the service > > again (it was already theoretically removed) I get: > > > > $ ceph orch rm mgmt-gateway > > Invalid service 'mgmt-gateway'. Use 'ceph orch ls' to list available > > services. > > > > That generated these lines in the log: > > > > ... > > Apr 27 10:35:03 alfred bash[3568828]: debug 2026-04-27T14:35:03.592+0000 > > 7f21e7e80640 0 log_channel(audit) log [DBG] : from=‘mgr.42485063 > (alfred’s > > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd > blocklist > > ls", "format": "json"} : dispatch > > Apr 27 10:35:04 alfred bash[3568828]: audit > > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > > Apr 27 10:35:04 alfred bash[3568828]: audit > > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > > Apr 27 10:35:06 alfred bash[3566093]: debug 2026-04-27T14:35:06.596+0000 > > 7f4324bbe640 0 log_channel(audit) log [DBG] : from=‘client.42505047 -' > > entity='client.admin' cmd=[{"prefix": "orch rm", "service_name": > > "mgmt-gateway", "target": ["mon-mgr", ""]}]: dispatch > > Apr 27 10:35:07 alfred bash[3568828]: audit > > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > > audit [DBG] from='client.42505047 -' entity='client.admin' > cmd=[{"prefix": > > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > > dispatch > > Apr 27 10:35:07 alfred bash[3568828]: audit > > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > > audit [DBG] from='client.42505047 -' entity='client.admin' > cmd=[{"prefix": > > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > > dispatch > > Apr 27 10:35:07 alfred bash[3568828]: cephadm > > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > > cephadm [INF] Remove service mgmt-gateway > > Apr 27 10:35:07 alfred bash[3568828]: cephadm > > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > > cephadm [INF] Remove service mgmt.-gateway > > Apr 27 10:35:18 alfred bash[3568828]: debug 2026-04-27T14:35:18.596+0000 > > 7f21e7e80640 0 log_channel(audit) log [DBG] : from='mgr.42485063 > (alfred’s > > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd > blocklist > > ls", "format": "json"} : dispatch > > Apr 27 10:35:19 alfred bash[3568828]: audit > > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > > Apr 27 10:35:19 alfred bash[3568828]: audit > > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > > ... > > > > But otherwise everything else looks pretty normal. > > > > This is still with 2 of the 3 mon/mgr nodes running the mgmt-gateway > > service: > > $ ceph orch ps | egrep 'mgr|mon|mgmt' > > mgmt-gateway.cyclops cyclops running (6d) 5d ago > > 6d 6188k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (6d) 5d ago > > 6d 5980k - 1.26.1 c2348906c6ff 329705c23aad > > mgr.alfred.afvnok alfred *:8443,9283,8765 running (2w) 5d ago > > 7M 564M - 20.2.1 fb63cba66eea 875883d08efc > > mgr.cyclops.nifeym cyclops *:8443,9283,8765 running (2w) 5d ago > > 7M 463M - 20.2.1 fb63cba66eea 676ec1753348 > > mgr.mask.yfbdpf mask *:8443,9283,8765 running (2w) 5d ago > > 7M 486M - 20.2.1 fb63cba66eea 6a1bdd37997c > > mon.alfred alfred running (2w) 5d ago > > 7M 525M 2048M 20.2.1 fb63cba66eea ce5bbb75d650 > > mon.cyclops cyclops running (2w) 5d ago > > 7M 571M 2048M 20.2.1 fb63cba66eea e675cf6714e4 > > mon.mask mask running (2w) 5d ago > > 7M 510M 2048M 20.2.1 fb63cba66eea 9a034ab6f132 > > > > And I still have the same health error. > > > > Let me know if there’s anything more specific you want me to show. > > > > Steve > > > > *From: *Redouane Kachach <rkachach(a)redhat.com> > > *Date: *Friday, April 24, 2026 at 4:32 AM > > *To: *Barber, Steve W. (Fed) <steve.barber(a)nist.gov> > > *Cc: *Ceph Users <ceph-users(a)ceph.io> > > *Subject: *[EXTERNAL] Re: [ceph-users] Attempt to enable mgmt-gateway on > > v20.2.1 failed, and now cephadm module is broken > > > > Hi Steve, > > > > Can you plz share the spec you used to configure the mgmt-gateway and the > > active mgr logs? > > > > Best, > > Redo. > > > > On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > > I have a pre-production cluster running on Ubuntu 22.04 hosts using > > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, > which > > went completely smoothly. We are a government facility so we are required > > to enable FIPS mode in the OS. I believe this last fact may be the cause > of > > the following: > > > > I wanted to enable mgmt-gateway so I made a service definition file to > > enable it and applied it. It appeared to work, but then “ceph health > > detail” showed: > > > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > > unsupported > > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > > routines] unsupported > > Module 'cephadm' has failed: [digital envelope routines] unsupported > > > > The mgmt-gateway service was started on all 3 nodes however. > > > > $ ceph orch ps --service_name mgmt-gateway > > NAME HOST PORTS STATUS REFRESHED AGE MEM > > USE MEM LIM VERSION IMAGE ID CONTAINER ID > > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (4m) 3m ago 4m > > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > > > I then attempted to remove the service definition with ceph orch rm, but > > it only succeeded in removing it on 1 of the 3 nodes. The process is > still > > running on the other 2, and the warning is still present. “ceph mgr fail > && > > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > > comes right back, and the process is still present on 2 nodes (as shown > by > > “ceph orch ps”). > > > > $ ceph mgr fail && ceph orch rm mgmt-gateway > > Removed service mgmt-gateway > > > > $ ceph orch ps --service_name mgmt-gateway > > NAME HOST PORTS STATUS REFRESHED AGE MEM > > USE MEM LIM VERSION IMAGE ID CONTAINER ID > > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (15h) 15h ago 15h > > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > > > $ ceph health detail > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > > unsupported > > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > > routines] unsupported > > Module 'cephadm' has failed: [digital envelope routines] unsupported > > > > Despite the 2 processes running, the configured service IP does not > > respond to a ping or requests from curl. > > > > Since we were getting ready to move this cluster into production I > decided > > to look for help here rather than proceed experimenting further and risk > > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > > (b) recover the cephadm module to get back to normal with no warnings; > and > > less importantly, (c) is there a way to get mgmt-gateway working in this > > environment? A little googling and AI querying suggests it may be an > issue > > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > > up my own proxy; I was just trying to use the built-in tools where > > possible.) > > > > Thanks in advance. > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Anthony D'Atri 30 Apr '26

30 Apr '26

Apologies, I could swear I saw an example float by. I might be conflating LC with the new tiering. > On Apr 29, 2026, at 9:17 PM, Matthew H via ceph-users <ceph-users(a)ceph.io> wrote: > > > s3 lifecycle rules are based on object age, which only reflects the > time since last write. so i'm afraid you won't be able to transition > objects based on read activity

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Matthew H 30 Apr '26

30 Apr '26

Hi Vishu, There is no symmetric, read-triggered promotion between two local storage classes as Casey stated. The closest mechanism RGW has today is the cloud-tier read-through path: - cloud-transition transitions objects out to an S3-compatible endpoint and leaves a stub head object behind. https://docs.ceph.com/en/tentacle/radosgw/cloud-transition/ - cloud-restore brings them back, either via an explicit RestoreObject call (temporary or permanent), or transparently on GET if the tier-config has allow_read_through=true. With read-through enabled, a regular GET on a transitioned stub triggers a restore automatically. https://docs.ceph.com/en/tentacle/radosgw/cloud-restore/ That's the closest fit for your "active again -> promote back" requirement. Which in your case would require two separate Ceph clusters configured with RGW, one with NVMe and the other with HDD. ________________________________ From: Casey Bodley via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, April 29, 2026 8:22 AM To: Vishnu Bhaskar <vishnukb(a)acceleronlabs.com> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD On Wed, Apr 29, 2026 at 7:02 AM Vishnu Bhaskar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I have a question regarding object lifecycle management in Ceph RGW. > > Is it possible to use RGW lifecycle policies to automatically move objects > between different storage tiers for example, from a hot tier (NVMe) to a > cold tier (HDD) based on inactivity over a period of time? > > Additionally, if those objects become active again (i.e., are accessed > frequently after being moved to the cold tier), is there a mechanism to > automatically move them back to the original hot tier (NVMe)? > > I may be misunderstanding how lifecycle management works in RGW, so please > correct me if that’s the case. If this type of tiering is supported, could > you point me toward the recommended approach or configuration? s3 lifecycle rules are based on object age, which only reflects the time since last write. so i'm afraid you won't be able to transition objects based on read activity > > Thanks and Regards > Vishnu Bhaskar > Acceleron Labs Pvt Ltd > Bangalore, India > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [EXTERNAL] Re: [EXTERNAL] Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Barber, Steve W. (Fed) 29 Apr '26

29 Apr '26

Thank you Redo! Do you have any suggestions for how I can back out of the current situation to get rid of the health warning and the orphaned mgmt-gateway services without making a mess of things? Steve From: Redouane Kachach via ceph-users <ceph-users(a)ceph.io> Date: Tuesday, April 28, 2026 at 4:25 AM To: Barber, Steve W. (Fed) <steve.barber(a)nist.gov> Cc: Ceph Users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] Re: [EXTERNAL] Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken Hi Steve, I saw that you already opened a ticket for the issue (thanks): https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.c…<https://tracker.ceph.com/issues/76185>. This problem is basically related to the env (FIPS is enabled). A PR with a candidate fix is under-review: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co…<https://github.com/ceph/ceph/pull/68638>. We will backport this change to Tentacle. Best, Redo. On Mon, Apr 27, 2026 at 5:02 PM Barber, Steve W. (Fed) < steve.barber(a)nist.gov> wrote: > Hi Redo, > > The spec was very simple: > > service_type: mgmt-gateway > placement: > #label: mgmt > hosts: > - alfred > - cyclops > - mask > spec: > virtual_ip: (redacted)/24 > port: 443 > > As far as active mgr logs, at this point I’m not seeing anything unusual > in the logs from the active mgr host. If I attempt to remove the service > again (it was already theoretically removed) I get: > > $ ceph orch rm mgmt-gateway > Invalid service 'mgmt-gateway'. Use 'ceph orch ls' to list available > services. > > That generated these lines in the log: > > ... > Apr 27 10:35:03 alfred bash[3568828]: debug 2026-04-27T14:35:03.592+0000 > 7f21e7e80640 0 log_channel(audit) log [DBG] : from=‘mgr.42485063 (alfred’s > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist > ls", "format": "json"} : dispatch > Apr 27 10:35:04 alfred bash[3568828]: audit > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:04 alfred bash[3568828]: audit > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:06 alfred bash[3566093]: debug 2026-04-27T14:35:06.596+0000 > 7f4324bbe640 0 log_channel(audit) log [DBG] : from=‘client.42505047 -' > entity='client.admin' cmd=[{"prefix": "orch rm", "service_name": > "mgmt-gateway", "target": ["mon-mgr", ""]}]: dispatch > Apr 27 10:35:07 alfred bash[3568828]: audit > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > audit [DBG] from='client.42505047 -' entity='client.admin' cmd=[{"prefix": > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > dispatch > Apr 27 10:35:07 alfred bash[3568828]: audit > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > audit [DBG] from='client.42505047 -' entity='client.admin' cmd=[{"prefix": > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > dispatch > Apr 27 10:35:07 alfred bash[3568828]: cephadm > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > cephadm [INF] Remove service mgmt-gateway > Apr 27 10:35:07 alfred bash[3568828]: cephadm > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > cephadm [INF] Remove service mgmt.-gateway > Apr 27 10:35:18 alfred bash[3568828]: debug 2026-04-27T14:35:18.596+0000 > 7f21e7e80640 0 log_channel(audit) log [DBG] : from='mgr.42485063 (alfred’s > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist > ls", "format": "json"} : dispatch > Apr 27 10:35:19 alfred bash[3568828]: audit > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:19 alfred bash[3568828]: audit > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > ... > > But otherwise everything else looks pretty normal. > > This is still with 2 of the 3 mon/mgr nodes running the mgmt-gateway > service: > $ ceph orch ps | egrep 'mgr|mon|mgmt' > mgmt-gateway.cyclops cyclops running (6d) 5d ago > 6d 6188k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (6d) 5d ago > 6d 5980k - 1.26.1 c2348906c6ff 329705c23aad > mgr.alfred.afvnok alfred *:8443,9283,8765 running (2w) 5d ago > 7M 564M - 20.2.1 fb63cba66eea 875883d08efc > mgr.cyclops.nifeym cyclops *:8443,9283,8765 running (2w) 5d ago > 7M 463M - 20.2.1 fb63cba66eea 676ec1753348 > mgr.mask.yfbdpf mask *:8443,9283,8765 running (2w) 5d ago > 7M 486M - 20.2.1 fb63cba66eea 6a1bdd37997c > mon.alfred alfred running (2w) 5d ago > 7M 525M 2048M 20.2.1 fb63cba66eea ce5bbb75d650 > mon.cyclops cyclops running (2w) 5d ago > 7M 571M 2048M 20.2.1 fb63cba66eea e675cf6714e4 > mon.mask mask running (2w) 5d ago > 7M 510M 2048M 20.2.1 fb63cba66eea 9a034ab6f132 > > And I still have the same health error. > > Let me know if there’s anything more specific you want me to show. > > Steve > > *From: *Redouane Kachach <rkachach(a)redhat.com> > *Date: *Friday, April 24, 2026 at 4:32 AM > *To: *Barber, Steve W. (Fed) <steve.barber(a)nist.gov> > *Cc: *Ceph Users <ceph-users(a)ceph.io> > *Subject: *[EXTERNAL] Re: [ceph-users] Attempt to enable mgmt-gateway on > v20.2.1 failed, and now cephadm module is broken > > Hi Steve, > > Can you plz share the spec you used to configure the mgmt-gateway and the > active mgr logs? > > Best, > Redo. > > On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < > ceph-users(a)ceph.io> wrote: > > I have a pre-production cluster running on Ubuntu 22.04 hosts using > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which > went completely smoothly. We are a government facility so we are required > to enable FIPS mode in the OS. I believe this last fact may be the cause of > the following: > > I wanted to enable mgmt-gateway so I made a service definition file to > enable it and applied it. It appeared to work, but then “ceph health > detail” showed: > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > The mgmt-gateway service was started on all 3 nodes however. > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (4m) 3m ago 4m > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > I then attempted to remove the service definition with ceph orch rm, but > it only succeeded in removing it on 1 of the 3 nodes. The process is still > running on the other 2, and the warning is still present. “ceph mgr fail && > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > comes right back, and the process is still present on 2 nodes (as shown by > “ceph orch ps”). > > $ ceph mgr fail && ceph orch rm mgmt-gateway > Removed service mgmt-gateway > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (15h) 15h ago 15h > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > $ ceph health detail > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > Despite the 2 processes running, the configured service IP does not > respond to a ping or requests from curl. > > Since we were getting ready to move this cluster into production I decided > to look for help here rather than proceed experimenting further and risk > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > (b) recover the cephadm module to get back to normal with no warnings; and > less importantly, (c) is there a way to get mgmt-gateway working in this > environment? A little googling and AI querying suggests it may be an issue > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > up my own proxy; I was just trying to use the built-in tools where > possible.) > > Thanks in advance. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Casey Bodley 29 Apr '26

29 Apr '26

On Wed, Apr 29, 2026 at 7:02 AM Vishnu Bhaskar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I have a question regarding object lifecycle management in Ceph RGW. > > Is it possible to use RGW lifecycle policies to automatically move objects > between different storage tiers for example, from a hot tier (NVMe) to a > cold tier (HDD) based on inactivity over a period of time? > > Additionally, if those objects become active again (i.e., are accessed > frequently after being moved to the cold tier), is there a mechanism to > automatically move them back to the original hot tier (NVMe)? > > I may be misunderstanding how lifecycle management works in RGW, so please > correct me if that’s the case. If this type of tiering is supported, could > you point me toward the recommended approach or configuration? s3 lifecycle rules are based on object age, which only reflects the time since last write. so i'm afraid you won't be able to transition objects based on read activity > > Thanks and Regards > Vishnu Bhaskar > Acceleron Labs Pvt Ltd > Bangalore, India > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Vishnu Bhaskar 29 Apr '26

29 Apr '26

Hi FYI Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India On Wed, Apr 29, 2026 at 6:41 PM Julien Laurenceau via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > I confirm you can setup tiering between two differents S3 pools on the > same ceph cluster with squid. > > You have to create 2 differents pools and placement groups relying on > your two type of physical storage drives. You may use the same RGW > client and use the S3 storage class attribute or you may also create a > new RGW client dedicated to each pool. > > I use S3 bucket policy for this and you should prefer using standard AWS > names of storage class ("GLACIER" or "STANDARD_IA"...) to do this easily > using the aws cli. > > > On 4/29/26 14:18, Anthony D'Atri via ceph-users wrote: > > > >> Is it possible to use RGW lifecycle policies to automatically move > objects > >> between different storage tiers for example, from a hot tier (NVMe) to a > >> cold tier (HDD) based on inactivity over a period of time? > > Absolutely. This is one key application of S3 storage classes. Note > that LC policies are run once a day by default, so with a heavy load you > may wish to deploy a set of RGWs that are not user-facing, but which only > handle LC tasks, and disable LC processing for the user-facing cohort of > daemons. > > > > > >> Additionally, if those objects become active again (i.e., are accessed > >> frequently after being moved to the cold tier), is there a mechanism to > >> automatically move them back to the original hot tier (NVMe)? > > But of course! Transparent tiering is available from Squid, at least > when offloading cold objects to a different endpoint. I'm not positive that > this works natively within a single cluster, but it may well. You could > though most likely still use LC policies, albeit in a less elegant fashion. > > > >> I may be misunderstanding how lifecycle management works in RGW, so > please > >> correct me if that’s the case. If this type of tiering is supported, > could > >> you point me toward the recommended approach or configuration? > > The following awesome blog posts from Daniel Parkes will guide you. When > deploying a cold tier, it is ideal to leave the first STANDARD storage > class as the default so that HEAD objects are always on faster storage. > They don't take much space, but this accelerates various operations. > > > > https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/ > > > > Depending on your object sizes and workload, consider replication vs > erasure coding. When using EC for your slower -- or even faster -- storage > class, be sure to select a value of m >= 2. Profiles wider than 8+3 may > increase space amp and cost write, scrub, and recovery performance. Always > choose (k+m < the number of CRUSH failure domains. > > > > Writes to a replicated pool will generally be faster, but EC makes more > efficient use of underlying storage. > > > > This is nuanced, though for the smallest objects. Check out Mark > Nelson's most excellent spreadsheet that shows this visually: > > > > > https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUTo… > > > > Through the Squid release, RADOS always writes full stripes, so if you > are using EC, the very smallest objects will consume more underlying raw > storage than one might think. The wider the EC profile, the more pronounced > this effect. If you have a large proportion of tiny objects, say below 256 > - 512 KiB, this space amplification may be significant. If the number of > very small objects is dwarfed by the number * size of larger objects, this > is amortized into the noise floor. If not, you can consider client headers > or Lua scripting to assort objects to appropriate storage classes at > ingest. Check out these presentations by Frederic Nass and Steve U: > > > > > https://ceph.io/en/news/blog/2024/auto-tiering-ceph-object-storage-part-1/ > > https://www.youtube.com/watch?v=qKd2Po2hTe4 > > > > Note that with the Tentacle release, bucket data pools with the EC > optimizations flag significantly reduce this space amplification, lowering > the size threshold for special handling and reducing the need to worry > about it. > > > > > > https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/ > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Julien Laurenceau 29 Apr '26

29 Apr '26

Hi, I confirm you can setup tiering between two differents S3 pools on the same ceph cluster with squid. You have to create 2 differents pools and placement groups relying on your two type of physical storage drives. You may use the same RGW client and use the S3 storage class attribute or you may also create a new RGW client dedicated to each pool. I use S3 bucket policy for this and you should prefer using standard AWS names of storage class ("GLACIER" or "STANDARD_IA"...) to do this easily using the aws cli. On 4/29/26 14:18, Anthony D'Atri via ceph-users wrote: > >> Is it possible to use RGW lifecycle policies to automatically move objects >> between different storage tiers for example, from a hot tier (NVMe) to a >> cold tier (HDD) based on inactivity over a period of time? > Absolutely. This is one key application of S3 storage classes. Note that LC policies are run once a day by default, so with a heavy load you may wish to deploy a set of RGWs that are not user-facing, but which only handle LC tasks, and disable LC processing for the user-facing cohort of daemons. > > >> Additionally, if those objects become active again (i.e., are accessed >> frequently after being moved to the cold tier), is there a mechanism to >> automatically move them back to the original hot tier (NVMe)? > But of course! Transparent tiering is available from Squid, at least when offloading cold objects to a different endpoint. I'm not positive that this works natively within a single cluster, but it may well. You could though most likely still use LC policies, albeit in a less elegant fashion. > >> I may be misunderstanding how lifecycle management works in RGW, so please >> correct me if that’s the case. If this type of tiering is supported, could >> you point me toward the recommended approach or configuration? > The following awesome blog posts from Daniel Parkes will guide you. When deploying a cold tier, it is ideal to leave the first STANDARD storage class as the default so that HEAD objects are always on faster storage. They don't take much space, but this accelerates various operations. > > https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/ > > Depending on your object sizes and workload, consider replication vs erasure coding. When using EC for your slower -- or even faster -- storage class, be sure to select a value of m >= 2. Profiles wider than 8+3 may increase space amp and cost write, scrub, and recovery performance. Always choose (k+m < the number of CRUSH failure domains. > > Writes to a replicated pool will generally be faster, but EC makes more efficient use of underlying storage. > > This is nuanced, though for the smallest objects. Check out Mark Nelson's most excellent spreadsheet that shows this visually: > > https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUTo… > > Through the Squid release, RADOS always writes full stripes, so if you are using EC, the very smallest objects will consume more underlying raw storage than one might think. The wider the EC profile, the more pronounced this effect. If you have a large proportion of tiny objects, say below 256 - 512 KiB, this space amplification may be significant. If the number of very small objects is dwarfed by the number * size of larger objects, this is amortized into the noise floor. If not, you can consider client headers or Lua scripting to assort objects to appropriate storage classes at ingest. Check out these presentations by Frederic Nass and Steve U: > > https://ceph.io/en/news/blog/2024/auto-tiering-ceph-object-storage-part-1/ > https://www.youtube.com/watch?v=qKd2Po2hTe4 > > Note that with the Tentacle release, bucket data pools with the EC optimizations flag significantly reduce this space amplification, lowering the size threshold for special handling and reducing the need to worry about it. > > > https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/ > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Anthony D'Atri 29 Apr '26

29 Apr '26

> > Is it possible to use RGW lifecycle policies to automatically move objects > between different storage tiers for example, from a hot tier (NVMe) to a > cold tier (HDD) based on inactivity over a period of time? Absolutely. This is one key application of S3 storage classes. Note that LC policies are run once a day by default, so with a heavy load you may wish to deploy a set of RGWs that are not user-facing, but which only handle LC tasks, and disable LC processing for the user-facing cohort of daemons. > Additionally, if those objects become active again (i.e., are accessed > frequently after being moved to the cold tier), is there a mechanism to > automatically move them back to the original hot tier (NVMe)? But of course! Transparent tiering is available from Squid, at least when offloading cold objects to a different endpoint. I'm not positive that this works natively within a single cluster, but it may well. You could though most likely still use LC policies, albeit in a less elegant fashion. > I may be misunderstanding how lifecycle management works in RGW, so please > correct me if that’s the case. If this type of tiering is supported, could > you point me toward the recommended approach or configuration? The following awesome blog posts from Daniel Parkes will guide you. When deploying a cold tier, it is ideal to leave the first STANDARD storage class as the default so that HEAD objects are always on faster storage. They don't take much space, but this accelerates various operations. https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/ Depending on your object sizes and workload, consider replication vs erasure coding. When using EC for your slower -- or even faster -- storage class, be sure to select a value of m >= 2. Profiles wider than 8+3 may increase space amp and cost write, scrub, and recovery performance. Always choose (k+m < the number of CRUSH failure domains. Writes to a replicated pool will generally be faster, but EC makes more efficient use of underlying storage. This is nuanced, though for the smallest objects. Check out Mark Nelson's most excellent spreadsheet that shows this visually: https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUTo… Through the Squid release, RADOS always writes full stripes, so if you are using EC, the very smallest objects will consume more underlying raw storage than one might think. The wider the EC profile, the more pronounced this effect. If you have a large proportion of tiny objects, say below 256 - 512 KiB, this space amplification may be significant. If the number of very small objects is dwarfed by the number * size of larger objects, this is amortized into the noise floor. If not, you can consider client headers or Lua scripting to assort objects to appropriate storage classes at ingest. Check out these presentations by Frederic Nass and Steve U: https://ceph.io/en/news/blog/2024/auto-tiering-ceph-object-storage-part-1/ https://www.youtube.com/watch?v=qKd2Po2hTe4 Note that with the Tentacle release, bucket data pools with the EC optimizations flag significantly reduce this space amplification, lowering the size threshold for special handling and reducing the need to worry about it. https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/

1 0

Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Vishnu Bhaskar 29 Apr '26

29 Apr '26

Hi, I have a question regarding object lifecycle management in Ceph RGW. Is it possible to use RGW lifecycle policies to automatically move objects between different storage tiers for example, from a hot tier (NVMe) to a cold tier (HDD) based on inactivity over a period of time? Additionally, if those objects become active again (i.e., are accessed frequently after being moved to the cold tier), is there a mechanism to automatically move them back to the original hot tier (NVMe)? I may be misunderstanding how lifecycle management works in RGW, so please correct me if that’s the case. If this type of tiering is supported, could you point me toward the recommended approach or configuration? Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Harald Staub 29 Apr '26

29 Apr '26

This does not use upmap. With the swap-bucket command, all the data of the source node (and some more) becomes "misplaced". I would expect it to work less direct in case lots of upmap remappings are in place. On 4/29/26 04:44, Wannes Smet via ceph-users wrote: > Does swap-bucket use upmap? If so, wouldn't that mean that a lot of PGs no longer end up where CRUSH would have placed them? > ________________________________ > From: Eugen Block via ceph-users <ceph-users(a)ceph.io> > Sent: Tuesday, April 28, 2026 21:40 > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Best "modern" approach to drain/fill a OSD server > > +1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > +1 for swap-bucket, we've been using that as well successfully. > > What we also sometimes do, especially when time is a bit more > critical, is to purge OSDs of an entire host (we also did this with > two hosts at the same time due to high resiliency) and then let Ceph > recover. But I generally don't recommend that to unexperienced > operators. > > > Zitat von Harald Staub via ceph-users <ceph-users(a)ceph.io>: > >> Not recently, but a few years ago, I used the "Whole host >> replacement" using "swap-bucket": >> https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/operation… >> >> This worked really nicely. With source and destination about the >> same host CRUSH weight, most PGs are moved directly from source to >> destination. So also the copy speed was more or less the same most >> of the time. >> >> Cheers, >> Harry >> >> On 4/28/26 11:08, Denis Polom via ceph-users wrote: >>> you can control max_backfills and some others if you set >>> osd_mclock_override_recovery_settings to true >>> >>> On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>>>> Another approach is to set the flags, destroy the OSDs on a >>>>> single host, redeploy them, set your backfill/recovery throttles >>>>> down to 1, and unset the flags. >>>> This is harder with mclock right? With wpq you can still control >>>> max_backfills, but not with mclock unless I am mistaken. >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Iztok Gregori 29 Apr '26

29 Apr '26

On 27/04/26 17:42, Iztok Gregori via ceph-users wrote: >>> I assume that 'target_max_misplaced_ratio' is a key parameter to >>> avoid large data movement in a given time, there is more to look into? >> >> Yes. Setting that lower than default can help avoid certain >> intermediate states that can result in a fairly-full cluster's OSDs >> going nearfull/backfillfull. > > The documentation say about 'target_max_misplaced_ratio' "...the > balancer will incrementally move a small fraction of unbalanced PGs in > order to improve distribution. This fraction will not exceed a certain > threshold that defaults to 5%...". does it mean that off all the PGs > that has to be moved out/in of the node(s), maximum 5% will be moved at > the same time? It seems that in my case (Ceph Quincy) 'target_max_misplaced_ratio' is looking for misplaced objects, not PGs (as I thought). I activated the balancer on a unbalanced cluster, set the ratio to 0.005000 (0.5%) and watch the bytes fly. I was expecting that maximum 0.5% of PGs (~70 PGs) will be in backfilling, but the number almost doubled (~130 PGs, > 1% of PGs) until the balancer status said: "Too many objects (0.005030 > 0.005000) are misplaced; try again later" I re-checked the latest documentation and the PGs are explicitly mention when talking about the ratio: "...The maximum percentage of PGs to remap (move) in a single phase defaults to 5%..." https://docs.ceph.com/en/latest/rados/operations/balancer/#throttling Is this the default on newer version of Ceph (I'm running Quincy in a upgrade path to Squid) or the documentation is not correct? Thanks Iztok Gregori

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Anthony D'Atri 29 Apr '26

29 Apr '26

I believe it’s an in-place edit of the CRUSH map. A powerful tool if you have a whole and fungible failure domain of undeployed hardware available. Last year I helped someone who did, and wanted to avoid long delays when transitioning their systems. > On Apr 28, 2026, at 10:42 PM, Wannes Smet via ceph-users <ceph-users(a)ceph.io> wrote: > > Does swap-bucket use upmap? If so, wouldn't that mean that a lot of PGs no longer end up where CRUSH would have placed them? > ________________________________ > From: Eugen Block via ceph-users <ceph-users(a)ceph.io> > Sent: Tuesday, April 28, 2026 21:40 > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Best "modern" approach to drain/fill a OSD server > > +1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > +1 for swap-bucket, we've been using that as well successfully. > > What we also sometimes do, especially when time is a bit more > critical, is to purge OSDs of an entire host (we also did this with > two hosts at the same time due to high resiliency) and then let Ceph > recover. But I generally don't recommend that to unexperienced > operators. > > > Zitat von Harald Staub via ceph-users <ceph-users(a)ceph.io>: > >> Not recently, but a few years ago, I used the "Whole host >> replacement" using "swap-bucket": >> https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/operation… >> >> This worked really nicely. With source and destination about the >> same host CRUSH weight, most PGs are moved directly from source to >> destination. So also the copy speed was more or less the same most >> of the time. >> >> Cheers, >> Harry >> >>> On 4/28/26 11:08, Denis Polom via ceph-users wrote: >>> you can control max_backfills and some others if you set >>> osd_mclock_override_recovery_settings to true >>> >>> On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>>>> Another approach is to set the flags, destroy the OSDs on a >>>>> single host, redeploy them, set your backfill/recovery throttles >>>>> down to 1, and unset the flags. >>>> This is harder with mclock right? With wpq you can still control >>>> max_backfills, but not with mclock unless I am mistaken. >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Wannes Smet 29 Apr '26

29 Apr '26

Does swap-bucket use upmap? If so, wouldn't that mean that a lot of PGs no longer end up where CRUSH would have placed them? ________________________________ From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, April 28, 2026 21:40 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Best "modern" approach to drain/fill a OSD server +1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd +1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) and then let Ceph recover. But I generally don't recommend that to unexperienced operators. Zitat von Harald Staub via ceph-users <ceph-users(a)ceph.io>: > Not recently, but a few years ago, I used the "Whole host > replacement" using "swap-bucket": > https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/operation… > > This worked really nicely. With source and destination about the > same host CRUSH weight, most PGs are moved directly from source to > destination. So also the copy speed was more or less the same most > of the time. > > Cheers, > Harry > > On 4/28/26 11:08, Denis Polom via ceph-users wrote: >> you can control max_backfills and some others if you set >> osd_mclock_override_recovery_settings to true >> >> On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>>> Another approach is to set the flags, destroy the OSDs on a >>>> single host, redeploy them, set your backfill/recovery throttles >>>> down to 1, and unset the flags. >>> This is harder with mclock right? With wpq you can still control >>> max_backfills, but not with mclock unless I am mistaken. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Eugen Block 28 Apr '26

28 Apr '26

+1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) and then let Ceph recover. But I generally don't recommend that to unexperienced operators. Zitat von Harald Staub via ceph-users <ceph-users(a)ceph.io>: > Not recently, but a few years ago, I used the "Whole host > replacement" using "swap-bucket": > https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/ > > This worked really nicely. With source and destination about the > same host CRUSH weight, most PGs are moved directly from source to > destination. So also the copy speed was more or less the same most > of the time. > > Cheers, > Harry > > On 4/28/26 11:08, Denis Polom via ceph-users wrote: >> you can control max_backfills and some others if you set >> osd_mclock_override_recovery_settings to true >> >> On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>>> Another approach is to set the flags, destroy the OSDs on a >>>> single host, redeploy them, set your backfill/recovery throttles >>>> down to 1, and unset the flags. >>> This is harder with mclock right? With wpq you can still control >>> max_backfills, but not with mclock unless I am mistaken. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Harald Staub 28 Apr '26

28 Apr '26

Not recently, but a few years ago, I used the "Whole host replacement" using "swap-bucket": https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/ This worked really nicely. With source and destination about the same host CRUSH weight, most PGs are moved directly from source to destination. So also the copy speed was more or less the same most of the time. Cheers, Harry On 4/28/26 11:08, Denis Polom via ceph-users wrote: > you can control max_backfills and some others if you set > osd_mclock_override_recovery_settings to true > > On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>> Another approach is to set the flags, destroy the OSDs on a single >>> host, redeploy them, set your backfill/recovery throttles down to 1, >>> and unset the flags. >> This is harder with mclock right? With wpq you can still control >> max_backfills, but not with mclock unless I am mistaken. >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Dan van der Ster 28 Apr '26

28 Apr '26

Hello! I'm glad to see that this info is still proving useful to the community. Here's a pretty comprehensive document explaining the typical upmap-remapped workflow: https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ If it can be useful, I'm thinking of organizing an OSD balancing happy hour call in the next month or so. Maybe mid-May or so on a Friday afternoon (EU time) / morning (US time). Thanks! Dan -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster(a)clyso.com On Mon, Apr 27, 2026 at 7:11 PM Iztok Gregori via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi to all! > > I'm searching the wisdom of the community on how to drain/fill an OSD > server (like for removal, bulk disk replacing, filestore to bluestore > conversion...) without impacting on the performance of the cluster. > > We are using the 'ceph-gentle-reweight' script, which is running fine, > but we are interested to know if there is a newer approach for this job. > > Searching the web I found a pdf of the talk of Dan van der Ster about > "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about > using the 'balancer' in 'upmap' mode (and a "magic" script) to move the > PGs from/to the OSDs in the background. > > So the balancer upmap approach can be used to drain a node following > this procedure: > - set balancer tunables (like 'target_max_misplaced_ratio') > - activate the balancer in upmap mode (if not already active) > - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain > - run the magic script 'upmap-remapped.py' > - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain > - do stuff on the node (remove, reinstall etc etc). > > But what is the implication for the whole cluster, how this background > job affects its performance? > > I assume that 'target_max_misplaced_ratio' is a key parameter to avoid > large data movement in a given time, there is more to look into? > > And there are other approaches for the job? > > Thanks a lot! > Iztok > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Best "modern" approach to drain/fill a OSD server
by Denis Polom 28 Apr '26

28 Apr '26

you can control max_backfills and some others if you set osd_mclock_override_recovery_settings to true On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >> Another approach is to set the flags, destroy the OSDs on a single host, redeploy them, set your backfill/recovery throttles down to 1, and unset the flags. > This is harder with mclock right? With wpq you can still control > max_backfills, but not with mclock unless I am mistaken. >

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Janne Johansson 28 Apr '26

28 Apr '26

> >> I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into? > > > > Yes. Setting that lower than default can help avoid certain intermediate states that can result in a fairly-full cluster's OSDs going nearfull/backfillfull. > > The documentation say about 'target_max_misplaced_ratio' "...the > balancer will incrementally move a small fraction of unbalanced PGs in > order to improve distribution. This fraction will not exceed a certain > threshold that defaults to 5%...". does it mean that off all the PGs > that has to be moved out/in of the node(s), maximum 5% will be moved at > the same time? Max 5% (I usually set it lower) of the misplaced-but-upmapped PGs will be un-upmapped and go into "backfill" or "backfill_wait". Since 50% is kind of a large number on large clusters, this means a lot of the PGs will be in backfill_wait. This isn't a huge problem in terms of movement, since other factors (like max_backfill when combined with wpq scheduler at least) will limit how many transfers are being done, but having 12 backfilling and 120 backfill_wait will just mean those in backfill_wait will not see scrubbing done to them while waiting, so I usually set the max_misplaced to a smaller number like 0.1% for larger clusters or pools with large number of PGs, which still leaves enough PGs that actually does move, but not too many lingering in _wait state so they can get their scrubs done as expected. The balancer runs every minute or so, so if the _wait queue dips, it will only take some 30-60 seconds for the balancer to make sure 0.1% is again heading for movement. -- May the most significant bit of your life be positive.

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Janne Johansson 28 Apr '26

28 Apr '26

> Another approach is to set the flags, destroy the OSDs on a single host, redeploy them, set your backfill/recovery throttles down to 1, and unset the flags. This is harder with mclock right? With wpq you can still control max_backfills, but not with mclock unless I am mistaken. -- May the most significant bit of your life be positive.

1 0

Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Redouane Kachach 28 Apr '26

28 Apr '26

Hi Steve, Can you plz share the spec you used to configure the mgmt-gateway and the active mgr logs? Best, Redo. On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < ceph-users(a)ceph.io> wrote: > I have a pre-production cluster running on Ubuntu 22.04 hosts using > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which > went completely smoothly. We are a government facility so we are required > to enable FIPS mode in the OS. I believe this last fact may be the cause of > the following: > > I wanted to enable mgmt-gateway so I made a service definition file to > enable it and applied it. It appeared to work, but then “ceph health > detail” showed: > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > The mgmt-gateway service was started on all 3 nodes however. > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (4m) 3m ago 4m > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > I then attempted to remove the service definition with ceph orch rm, but > it only succeeded in removing it on 1 of the 3 nodes. The process is still > running on the other 2, and the warning is still present. “ceph mgr fail && > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > comes right back, and the process is still present on 2 nodes (as shown by > “ceph orch ps”). > > $ ceph mgr fail && ceph orch rm mgmt-gateway > Removed service mgmt-gateway > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (15h) 15h ago 15h > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > $ ceph health detail > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > Despite the 2 processes running, the configured service IP does not > respond to a ping or requests from curl. > > Since we were getting ready to move this cluster into production I decided > to look for help here rather than proceed experimenting further and risk > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > (b) recover the cephadm module to get back to normal with no warnings; and > less importantly, (c) is there a way to get mgmt-gateway working in this > environment? A little googling and AI querying suggests it may be an issue > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > up my own proxy; I was just trying to use the built-in tools where > possible.) > > Thanks in advance. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 2

Re: Best "modern" approach to drain/fill a OSD server
by Wannes Smet 28 Apr '26

28 Apr '26

Hi Dan, Speaking for myself, it could definitely be useful. Kr, Wannes ________________________________ From: Dan van der Ster via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, April 28, 2026 02:59 To: Iztok Gregori <iztok.gregori(a)elettra.eu> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Best "modern" approach to drain/fill a OSD server Hello! I'm glad to see that this info is still proving useful to the community. Here's a pretty comprehensive document explaining the typical upmap-remapped workflow: https: //urldefense. com/v3/__https: //docs. clyso. com/docs/kb/rados/adding-hosts-osds/__;!!FtrhtPsWDhZ6tw!EYA3GHD_cjvUgVuwVS1tqn71YZ0peFcj2lpFMCLJHnptXIVYM9BIQSIAsnUxfEjRbdIYZwvr9YnzWNqA$[docs[. ]clyso[. ]com] ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello! I'm glad to see that this info is still proving useful to the community. Here's a pretty comprehensive document explaining the typical upmap-remapped workflow: https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/rados/adding-hos… If it can be useful, I'm thinking of organizing an OSD balancing happy hour call in the next month or so. Maybe mid-May or so on a Friday afternoon (EU time) / morning (US time). Thanks! Dan -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://urldefense.com/v3/__https://analyzer.clyso.com/__;!!FtrhtPsWDhZ6tw!… https://urldefense.com/v3/__https://clyso.com__;!!FtrhtPsWDhZ6tw!EYA3GHD_cj… | dan.vanderster(a)clyso.com On Mon, Apr 27, 2026 at 7:11 PM Iztok Gregori via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi to all! > > I'm searching the wisdom of the community on how to drain/fill an OSD > server (like for removal, bulk disk replacing, filestore to bluestore > conversion...) without impacting on the performance of the cluster. > > We are using the 'ceph-gentle-reweight' script, which is running fine, > but we are interested to know if there is a newer approach for this job. > > Searching the web I found a pdf of the talk of Dan van der Ster about > "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about > using the 'balancer' in 'upmap' mode (and a "magic" script) to move the > PGs from/to the OSDs in the background. > > So the balancer upmap approach can be used to drain a node following > this procedure: > - set balancer tunables (like 'target_max_misplaced_ratio') > - activate the balancer in upmap mode (if not already active) > - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain > - run the magic script 'upmap-remapped.py' > - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain > - do stuff on the node (remove, reinstall etc etc). > > But what is the implication for the whole cluster, how this background > job affects its performance? > > I assume that 'target_max_misplaced_ratio' is a key parameter to avoid > large data movement in a given time, there is more to look into? > > And there are other approaches for the job? > > Thanks a lot! > Iztok > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Anthony D'Atri 27 Apr '26

27 Apr '26

> Hi to all! > > I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion...) without impacting on the performance of the cluster. You'll impact performance no matter what you do, but to differing degrees. The least impact is making the least change: set noout destroy one OSD redeploy that OSD unset noout wait for recovery At scale that doesn't, well, scale. > We are using the 'ceph-gentle-reweight' script, which is running fine, but we are interested to know if there is a newer approach for this job. What Ceph release are you running? That approach works, but will take a long time and you need to have enough free capacity on other nodes to hold that node's share of data. > > Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in 'upmap' mode If you're running, say, Mimic or later, you typically would already have the balancer on; upmap mode is the default. Run ceph balancer status When using upmaps, you want to avoid "ceph osd reweight-by-utilization" and ensure that the REWEIGHT column of "ceph osd df" or "ceph osd tree" is all 1.0000 > (and a "magic" script) to move the PGs from/to the OSDs in the background. > > So the balancer upmap approach can be used to drain a node following this procedure: Be sure to disable backfill and rebalancing, you want no backfilling in progress when you run upmap-remapped.py > - set balancer tunables (like 'target_max_misplaced_ratio') > - activate the balancer in upmap mode (if not already active) > - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain > - run the magic script 'upmap-remapped.py' > - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain > - do stuff on the node (remove, reinstall etc etc). If you are solving for safety over time, sure. Especially if you have replicated size=2 pools or EC N+1. Another approach is to set the flags, destroy the OSDs on a single host, redeploy them, set your backfill/recovery throttles down to 1, and unset the flags. This will be somewhat more risky as you're reducing redundancy until the recovery completes, and more will happen in parallel. Whether you can tolerate that depends in part on your hardware, CRUSH topology, how many hosts, replication vs EC, etc. > > But what is the implication for the whole cluster, how this background job affects its performance? We need more info. > > I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into? Yes. Setting that lower than default can help avoid certain intermediate states that can result in a fairly-full cluster's OSDs going nearfull/backfillfull. > > And there are other approaches for the job? > > Thanks a lot! > Iztok > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

CSC Meeting Minutes April 27th
by Patrick Donnelly 27 Apr '26

27 Apr '26

Good day, Here are the meeting minutes for today's short call: * [Anthony M and Emmanuel Ameh] Meet the next Tech Writer * [Anthony M] CSC/Board Q2 Meeting will be on Mon. Jun 1, 8:00 - 9:00 am PDT (we will invite the board to the CSC Meeting) Please add topics to the meeting pad - https://pad.ceph.com/p/CSC_-_board-meeting-Q2 by May 25th * [Anthony M] Voting for CSC Co-Leads ends Friday, May 1st at 11:59 pm PDT Have a good week, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Wannes Smet 27 Apr '26

27 Apr '26

Thanks for bringing this presentation to my attention. I'm in a similar place atm. where I'll likely introduce a change in our cluster which will cause "massive" data reloction. The "leap of faith" drawing seems very recognizable. I think here's the recording of the presentation. I'll definitily watch that one: https://www.youtube.com/watch?v=6PQYHlerJ8k&list=PLrBUGiINAakMz4n_oOeiueeUk… ________________________________ From: Iztok Gregori via ceph-users <ceph-users(a)ceph.io> Sent: Monday, April 27, 2026 12:12 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Best "modern" approach to drain/fill a OSD server Hi to all! I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion. . . ) without impacting on the performance of the cluster. We are using the 'ceph-gentle-reweight' ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi to all! I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion...) without impacting on the performance of the cluster. We are using the 'ceph-gentle-reweight' script, which is running fine, but we are interested to know if there is a newer approach for this job. Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in 'upmap' mode (and a "magic" script) to move the PGs from/to the OSDs in the background. So the balancer upmap approach can be used to drain a node following this procedure: - set balancer tunables (like 'target_max_misplaced_ratio') - activate the balancer in upmap mode (if not already active) - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain - run the magic script 'upmap-remapped.py' - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain - do stuff on the node (remove, reinstall etc etc). But what is the implication for the whole cluster, how this background job affects its performance? I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into? And there are other approaches for the job? Thanks a lot! Iztok _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Best "modern" approach to drain/fill a OSD server
by Iztok Gregori 27 Apr '26

27 Apr '26

Hi to all! I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion...) without impacting on the performance of the cluster. We are using the 'ceph-gentle-reweight' script, which is running fine, but we are interested to know if there is a newer approach for this job. Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in 'upmap' mode (and a "magic" script) to move the PGs from/to the OSDs in the background. So the balancer upmap approach can be used to drain a node following this procedure: - set balancer tunables (like 'target_max_misplaced_ratio') - activate the balancer in upmap mode (if not already active) - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain - run the magic script 'upmap-remapped.py' - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain - do stuff on the node (remove, reinstall etc etc). But what is the implication for the whole cluster, how this background job affects its performance? I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into? And there are other approaches for the job? Thanks a lot! Iztok

1 0

Re: Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])
by Flemming Frandsen 27 Apr '26

27 Apr '26

let's encrypt certs, yes. On Fri, 24 Apr 2026 at 16:26, Konstantin Shalygin via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > Just guessing - it's Lets Encrypt certs? > Or may be something InternalCA when certificate issues by golang > application? > > > k > Sent from my iPhone > > > On 24 Apr 2026, at 15:54, Flemming Frandsen via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > I have a tentacle 20.2.1 cluster managed by cephadm, where I have > > configured a certificate for the rgw using something like: > > > > service_type: ingress > > service_id: rgw.default > > service_name: ingress.rgw.default > > placement: > > count: 3 > > label: rgw > > spec: > > backend_service: rgw.default > > first_virtual_router_id: 50 > > frontend_port: 443 > > monitor_port: 8080 > > virtual_ip: 10.0.40.3/24 > > networks: > > - 10.0.40.0/24 > > ssl_cert: | > > -----BEGIN CERTIFICATE----- > > *snip* > > -----END CERTIFICATE----- > > -----BEGIN CERTIFICATE----- > > *snip* > > -----END CERTIFICATE----- > > -----BEGIN RSA PRIVATE KEY----- > > *snip* > > -----END RSA PRIVATE KEY----- > > > > This initially worked perfectly, but I have just rotated the certificate > by > > generating the yaml with a fresh private key and associated certificate > > then installing it with: > > ceph orch apply -i new.yaml > > > > Clients accessing the rgw ingress get the correct certificate and work > > perfectly, both before and after the key rotation. > > > > > > I have a script that does this for me daily, and until the certificate > was > > rotated, everything was fine, but then I got this error: > > > > # ceph orch certmgr cert check > > > > Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > > valid (error: Invalid cert/key pair: [('x509 certificate routines', '', > > 'key values mismatch')]) > > > > After messing with the certificate a bunch I now get a completely > > different, but also worrying error: > > > > # ceph orch certmgr cert check > > - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > > valid (error: Invalid private key: [('DECODER routines', '', > > 'unsupported')]) > > > > The documentation offers no help: > > https://docs.ceph.com/en/tentacle/cephadm/certmgr/ > > > > The error seems to come from this code: > > > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > > > So the error is an amputated exception from somewhere in openssl but the > > actual exception is eaten by the code, so there is no way for me to debug > > the situation: > > > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > > > I have tried installing the old certificate and key, that only leads the > > the Invalid private key: [('DECODER routines', '', 'unsupported')] error. > > > > I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does > > nothing. > > > > I have even tried copying the key and cert pem files into the mgr > container > > and verified them using the openssl installed in the container. > > > > The certificates are generated by cert-manager, running in k8s and a > > cronjob that periodically pushes the new key/cert to the ceph cluster via > > ssh. > > > > Has anyone seen these types of errors before? > > > > -- > > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

2 1

Re: could not find secret_id
by Eugen Block 24 Apr '26

24 Apr '26

Hi, how did you end up in that situation (did something happen in the cluster)? Is it affecting only OSDs or other daemons as well? We did investigate such an issue one and a half years ago for a customer. They added hosts to one of the two clusters, at that time the customer had two sites, with rbd mirroring enabled. And I don't recall the exact details, we did solve it in an online meeting and I don't think anyone wrote a protocoll. And the session definitely wasn't recored. :-) It might have some remainder on the new host from previous tests, or some misconfiguration, I really can't tell anymore. I think in the end we ended up removing the OSDs, zapping them and adding them back. Of course, it could be a different root cause, not sure which role the mon leader can play in this. But I thought I'd share this tiny bit at least. :-) Regards, Eugen Zitat von Andrea Martra via ceph-users <ceph-users(a)ceph.io>: > Hi all, > I'm reaching back to this old thread from December 2023 because > we're investigating a very similar pattern at a customer site > running Ceph Reef 18.2.x. Logs show identical shape: > > AuthRegistry get_handler peer_type 4 method 2 ... > cephx: verify_authorizer decrypted service osd secret_id=3524 > auth: could not find secret_id=3524 > auth: dump_rotating: (local range ~971-973) > cephx: verify_authorizer could not get service secret ... > > Affecting 4 OSDs on 7 out of 39 nodes, persistent over days > (15M+ log lines in 3 days). No clock skew, quorum stable. > > @xiaowenhao111 or anyone else who hit this: did you ever > find a root cause or a reliable fix? > The original thread > didn't seem to reach a conclusion. Specifically: > > - Did restarting the affected OSDs clear it, or did the > pattern come back? > - Was a MON leader failover enough, or did you need to > rebuild anything? > - Any tracker issue filed for this? > > We're trying to reproduce on a test cluster (Reef 18.2.7) > by isolating a node hosting MON leader + OSDs via network > for 3+ hours, to match the scenario described in the > original thread. Happy to share findings back to the list. > > Thanks, > Andrea > > Il 21/12/23 02:42, xiaowenhao111 ha scritto: >> >> >> Hello ceph-users: >> >> First ,sorry my english ... >> >> >> I find a bug (16.2.10 <tel:16210> and 18.2.1), but i do not know why >> >> test steps: >> >> 1、config all hosts contain chronyd service >> >> 2、cephadm --image quay.io/ceph/ceph:v16.2.10 >> <http://quay.io/ceph/ceph:v16.2.10> bootstrap >> --dashboard-password-noupdate --mon-ip 10.40.10.200 >> <http://10.40.10.200> --cluster-network=10.40.10.0/24 >> <http://10.40.10.0/24> --skip-pull --allow-overwrite >> --skip-monitoring-stack --allow-mismatched-release >> >> 3、add hosts >> >> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node2 >> >> ceph orch host add node2 10.40.10.202 <tel:2104010202> --labels=_admin >> >> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node-1 >> >> ceph orch host add node-1 10.40.10.202 <tel:1104010202> --labels=_admin >> >> 4、add osds: >> >> ceph orch daemon add osd node1:/dev/sda >> >> ceph orch daemon add osd node1:/dev/sdb >> >> ceph orch daemon add osd node1:/dev/sdc >> >> ceph orch daemon add osd node1:/dev/sdd >> >> ceph orch daemon add osd node2:/dev/sda >> >> ceph orch daemon add osd node2:/dev/sdb >> >> ceph orch daemon add osd node2:/dev/sdc >> >> ceph orch daemon add osd node2:/dev/sdd >> >> 5、ceph osd pool create test_pool 512 512 <tel:512512> --size=3 >> >> 6、disconnect node1 (which is monitor leader) cluster and public network >> >> ifdown enp35s0f0 >> >> ifdown enp35s0f1 >> >> 7、stay overnight（more than three hours） >> >> 8、connect node1 cluster and public network >> >> ifup enp35s0f0 >> >> ifup enp35s0f1 >> >> >> then, the osds of node1 call get_auth_session_key from mon.node1 >> which is not new leader，get the wrong secret_id=4. >> >> the others report error "could not find secret_id=4" more than 40 minutes >> >> >> >> >> the ceph.log content of node1: >> >> >> 2023-12-20 <tel:20231220>T09:48:30.199224 <tel:30199224>+0800 >> mon.node1 (mon.0) 1891 <tel:01891> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-20 <tel:20231220>T09:49:37.816260 <tel:37816260>+0800 >> mon.node1 (mon.0) 1901 <tel:01901> : cluster [INF] Health check >> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >> >> 2023-12-20 <tel:20231220>T09:49:44.222046 <tel:44222046>+0800 >> mon.node1 (mon.0) 1902 <tel:01902> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-20 <tel:20231220>T09:50:00.000090 <tel:00000090>+0800 >> mon.node1 (mon.0) 1905 <tel:01905> : cluster [WRN] overall >> HEALTH_WARN Failed to place 1 daemon(s); 3 failed cephadm >> daemon(s); 1 pool(s) do not have an application enabled >> >> 2023-12-20 <tel:20231220>T09:50:51.916464 <tel:51916464>+0800 >> mon.node1 (mon.0) 1912 <tel:01912> : cluster [INF] Health check >> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >> >> 2023-12-20 <tel:20231220>T09:51:07.127982 <tel:07127982>+0800 >> mon.node1 (mon.0) 1916 <tel:01916> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0 >> (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >> election >> >> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new >> leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >> >> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >> >> >> the ceph.log content of node2: >> >> >> 2023-12-21 <tel:20231221>T01:10:01.065051 <tel:01065051>+0800 >> mon.node2 (mon.1) 11522 <tel:111522> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-21 <tel:20231221>T01:11:11.092640 <tel:11092640>+0800 >> mon.node2 (mon.1) 11531 <tel:111531> : cluster [INF] Health check >> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >> >> 2023-12-21 <tel:20231221>T01:11:19.197355 <tel:19197355>+0800 >> mon.node2 (mon.1) 11532 <tel:111532> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-21 <tel:20231221>T01:12:34.469403 <tel:34469403>+0800 >> mon.node2 (mon.1) 11547 <tel:111547> : cluster [INF] Health check >> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) >> >> 2023-12-21 <tel:20231221>T01:12:42.476793 <tel:42476793>+0800 >> mon.node2 (mon.1) 11549 <tel:111549> : cluster [WRN] Health check >> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) >> >> 2023-12-21 <tel:20231221>T01:13:55.234163 <tel:55234163>+0800 >> mon.node2 (mon.1) 11572 <tel:111572> : cluster [INF] osd.5 marked >> itself dead as of e91 >> >> 2023-12-21 <tel:20231221>T01:13:55.236348 <tel:55236348>+0800 >> mon.node2 (mon.1) 11573 <tel:111573> : cluster [INF] osd.4 marked >> itself dead as of e91 >> >> 2023-12-21 <tel:20231221>T01:13:55.242209 <tel:55242209>+0800 >> mon.node2 (mon.1) 11574 <tel:111574> : cluster [INF] osd.0 marked >> itself dead as of e91 >> >> 2023-12-21 <tel:20231221>T01:13:55.233774 <tel:55233774>+0800 osd.5 >> (osd.5) 3 : cluster [WRN] Monitor daemon marked osd.5 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:13:55.236038 <tel:55236038>+0800 osd.4 >> (osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:14:00.109136 <tel:109136>+0800 >> mon.node2 (mon.1) 11589 <tel:111589> : cluster [WRN] Health check >> update: Degraded data redundancy: 1/8 objects degraded (12.500 >> <tel:12500>%), 1 pg degraded, 188 pgs undersized (PG_DEGRADED) >> >> 2023-12-21 <tel:20231221>T01:14:00.311644 <tel:311644>+0800 >> mon.node2 (mon.1) 11590 <tel:111590> : cluster [INF] osd.3 marked >> itself dead as of e94 >> >> 2023-12-21 <tel:20231221>T01:14:00.311410 <tel:311410>+0800 osd.3 >> (osd.3) 3 : cluster [WRN] Monitor daemon marked osd.3 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:14:01.922553 <tel:01922553>+0800 >> mgr.node1.apderh (mgr.34343) 1 <tel:343431> : cluster [ERR] Failed >> to load ceph-mgr modules: k8sevents >> >> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0 >> (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it >> is still running >> >> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800 >> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800 >> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800 >> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor >> election >> >> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800 >> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 >> calling monitor election >> >> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800 >> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new >> leader, mons node1,node2,node-1 in quorum (ranks 0,1,2) >> >> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800 >> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check >> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1) >> >> >> >> the osd log of node1: >> >> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.202:6804 <http://10.40.10.202:6804> osd.6 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.202:6812 <http://10.40.10.202:6812> osd.7 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.202:6820 <http://10.40.10.202:6820> osd.8 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.202:6828 <http://10.40.10.202:6828> osd.9 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: reset >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: 0x562066 >> <tel:09562066>bbc000 handle_response ret = 0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: got initial server >> challenge 5fd5d9eca256ae69 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: validate_tickets: >> want=53 need=0 have=53 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have 0 >> need 53 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: build_request >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 20 <tel:864020> cephx client: old ticket len=96 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: get auth session key: >> client_challenge 7ca3f7c3ef30b2bf >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: 0x562066 >> <tel:09562066>bbc000 handle_response ret = 0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: get_auth_session_key >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply got >> 1 keys >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id auth >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply got >> encrypted ticket >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket.secret_id=2 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> service auth secret_id 2 session_key >> AQDIIINlUlSmKBAAvL5rKyyIb6vt89rxUGlsxg== validity=259200.000000 >> <tel:259200000000> >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket expires=2023-12-24 <tel:20231224>T01:13:44.682217 >> <tel:44682217>+0800 renew_after=2023-12-23 >> <tel:20231223>T07:13:44.682217 <tel:44682217>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: got connection bl 84 >> and extra tickets 550 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: got connection_secret >> 64 bytes >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply got >> 3 keys >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mon >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket.secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> service mon secret_id 4 session_key >> AQDIIINlWvamKBAAxOaZsuCMdgutax1n2ZRbZQ== validity=3600.000000 >> <tel:3600000000> >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682256 >> <tel:44682256>+0800 renew_after=2023-12-21 >> <tel:20231221>T01:58:44.682256 <tel:44682256>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id osd >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket.secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> service osd secret_id 4 session_key >> AQDIIINlXAinKBAAOwTlViH3CJRgSvj5Kn9qtA== validity=3600.000000 >> <tel:3600000000> >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682270 >> <tel:44682270>+0800 renew_after=2023-12-21 >> <tel:20231221>T01:58:44.682270 <tel:44682270>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mgr >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket.secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> service mgr secret_id 4 session_key >> AQDIIINlUiGnKBAAr6WBaSawzVyDoqEs8cENdg== validity=3600.000000 >> <tel:3600000000> >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply >> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682286 >> <tel:44682286>+0800 renew_after=2023-12-21 >> <tel:20231221>T01:58:44.682286 <tel:44682286>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: got extra service_tickets >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have >> 53 need 0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have >> 53 need 0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 20 <tel:864020> cephx client: need_tickets: want=53 >> have=53 need=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 -1 <tel:86401> monclient: _check_auth_rotating >> possible clock skew, rotating keys expired way too early (before >> 2023-12-21 <tel:20231221>T00:13:44.682376 <tel:44682376>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 >> 7fcd2bed8640 10 <tel:864010> cephx client: build_rotating_request >> >> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.201:6804 <http://10.40.10.201:6804> osd.1 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.201:6812 <http://10.40.10.201:6812> osd.2 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 >> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from >> 10.40.10.200:6814 <http://10.40.10.200:6814> osd.3 ever on either >> front or back, first ping sent 2023-12-21 >> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest deadline >> 2023-12-21 <tel:20231221>T01:13:43.148457 <tel:43148457>+0800) >> >> >> >> >> the osd log of node2: >> >> >> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >> 7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have 53 >> need 0 >> >> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >> 7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53 >> have=53 need=0 >> >> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >> 7fa573a6d640 10 <tel:64010> auth: dump_rotating: >> >> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 >> 7fa573a6d640 10 <tel:64010> auth: id 17 >> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21 >> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 >> 7fa573a6d640 10 <tel:64010> auth: id 18 >> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21 >> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 >> 7fa573a6d640 10 <tel:64010> auth: id 19 >> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21 >> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 20 <tel:58129364020> AuthRegistry(0x7ffc86c1e7b0) >> get_handler peer_type 4 method 2 cluster_methods [2] >> service_methods [2] client_methods [2] >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> cephx: verify_authorizer >> decrypted service osd secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 0 <tel:5812936400> auth: could not find secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> auth: dump_rotating: >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> auth: id 17 >> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21 >> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> auth: id 18 >> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21 >> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 10 <tel:58129364010> auth: id 19 >> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21 >> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800 >> >> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 >> 7fa581293640 0 <tel:5812936400> cephx: verify_authorizer could not >> get service secret for service osd secret_id=4 >> >> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 >> 7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have 53 >> need 0 >> >> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 >> 7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53 >> have=53 need=0 >> >> >> >> the mon log of node1: >> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> start_session >> entity_name=mgr.node1.apderh global_id=14184 >> <tel:14184> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >> start_session server_challenge 524ce5eea3e4dd75 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >> handle_request get_auth_session_key for mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >> checking key: req.key=f41952 <tel:41952>c0f02e6c0f >> expected_key=f41952 <tel:41952>c0f02e6c0f >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >> checking old_ticket: secret_id=2 len=112, old_ticket_may_be_omitted=0 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh: >> decoded old_ticket: global_id=14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: >> allowing reclaim of global_id 14184 <tel:14184> (valid ticket >> presented, will encrypt new ticket) >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply >> encoding 1 tickets with secret >> AQBmKIJlvHeYJRAAf5h+auT4c9Ilo/nlUqRJlg== >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> auth secret_id 2 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >> name=mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >> of caps=3 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: adding >> key for service mon >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: adding >> key for service mds >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >> name=mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >> of caps=3 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: adding >> key for service osd >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >> name=mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >> of caps=3 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh: adding >> key for service mgr >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps: >> name=mgr.node1.apderh >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num >> of caps=3 >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply >> encoding 4 tickets with secret >> AQDIIINldY1mGRAApH04rNRZtuyAUrQn5PNG7A== >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> mon secret_id 4 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> mds secret_id 4 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> osd secret_id 4 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 >> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service >> mgr secret_id 4 ticket_info.ticket.name >> <http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id >> 14184 <tel:14184> >> >> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command >> mon_command({"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"} v 0) >> v1 >> >> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800 >> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >> from='mgr.14184 10.40.10.200:0/2514043697 >> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >> cmd=[{"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}]: >> dispatch >> >> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command >> mon_command({"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"} >> v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800 >> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] : >> from='mgr.14184 10.40.10.200:0/2514043697 >> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh' >> cmd=[{"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}]: >> dispatch >> >> >> >> the mon log of node2: >> >> >> 2023-12-21 <tel:20231221>T01:13:10.292 <tel:10292>+0800 >> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >> {"time_micros": 1703092390293629, "job": 1145, "event": >> "table_file_deletion", "file_number": 2272} >> >> 2023-12-21 <tel:20231221>T01:13:10.295 <tel:10295>+0800 >> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >> {"time_micros": 1703092390296385, "job": 1145, "event": >> "table_file_deletion", "file_number": 2270} >> >> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1 >> {"time_micros": 1703092390304703, "job": 1145, "event": >> "table_file_deletion", "file_number": 2269} >> >> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800 >> 7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442 >> <http://db_impl_compaction_flush.cc:1442>] [default] Manual >> compaction starting >> >> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"} >> v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}]: >> dispatch >> >> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"} v 0) >> v1 >> >> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix":"config >> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}]: >> dispatch >> >> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:51.142 <tel:51142>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix": "config dump", "format": "json"} v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:51.143 <tel:51143>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix": "config dump", "format": "json"}]: dispatch >> >> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch >> >> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command({"prefix": "auth get", "entity": "client.admin"} v 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800 >> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch >> >> 2023-12-21 <tel:20231221>T01:13:52.247 <tel:52247>+0800 >> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command >> mon_command([{prefix=config-key set, key=mgr/cephadm/host.node1}] v >> 0) v1 >> >> 2023-12-21 <tel:20231221>T01:13:52.260 <tel:52260>+0800 >> 7f88b96c1640 0 <tel:16400> log_channel(audit) log [INF] : >> from='mgr.14252 10.40.10.201:0/357060750 >> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud' >> >> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 >> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91 >> _set_new_cache_sizes cache_size:1020054731 >> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc: >> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536> >> >> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 >> 7f88bd6c9640 20 <tel:964020> cephx keyserver: >> prepare_rotating_update before: data.rotating_ver=17 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88beecc640 10 <tel:64010> start_session entity_name=osd.0 >> global_id=14205 <tel:14205> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88beecc640 10 <tel:64010> cephx server osd.0: start_session >> server_challenge dba982c2bced072d >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.3 >> global_id=14379 <tel:14379> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88bf6cd640 10 <tel:64010> cephx server osd.3: start_session >> server_challenge e9db92f3f5af188c >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88beecc640 10 <tel:64010> start_session >> entity_name=mgr.node1.apderh global_id=14190 >> <tel:14190> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88beecc640 10 <tel:64010> cephx server mgr.node1.apderh: >> start_session server_challenge b9809292 <tel:9809292>ea2e62f6 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.5 >> global_id=24253 <tel:24253> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 >> 7f88bf6cd640 10 <tel:64010> cephx server osd.5: start_session >> server_challenge 1be9f89b4bd9cdbc >> >> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >> 7f88b9ec2640 10 <tel:264010> start_session entity_name=osd.4 >> global_id=14406 <tel:14406> is_new_global_id=0 >> >> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >> 7f88b9ec2640 10 <tel:264010> cephx server osd.4: start_session >> server_challenge cca4694d496c39a8 >> >> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 >> 7f88beecc640 10 <tel:64010> cephx server osd.0: handle_request >> get_auth_session_key for osd.0 >> >> >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io > > -- > -- > > Andrea Martra > Senior Manager > +39 393 9048451 > > Open Technologies Sas > Via Michele Buniva, 26 - 10064 Pinerolo (TO) > Web:http://www.oscct.it > > ** Open Source Cloud Computing Technologies ** > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Joshua Blanch 24 Apr '26

24 Apr '26

@Redo I think it might have something to do with how we are serializing the certs. https://github.com/ceph/ceph/blob/3dc6a60ce03606649480fd6341c0f0841d28c101/… it might be related to FIPS mode being stricter and requiring PKCS8 instead of TraditionalOpenSSL class PrivateFormat(utils.Enum): PKCS8 = "PKCS8" TraditionalOpenSSL = "TraditionalOpenSSL" Raw = "Raw" OpenSSH = "OpenSSH" https://cryptography.io/en/3.4.5/hazmat/primitives/asymmetric/serialization… On Fri, 24 Apr 2026 at 01:34, Redouane Kachach via ceph-users < ceph-users(a)ceph.io> wrote: > Hi Steve, > > Can you plz share the spec you used to configure the mgmt-gateway and the > active mgr logs? > > Best, > Redo. > > On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < > ceph-users(a)ceph.io> wrote: > > > I have a pre-production cluster running on Ubuntu 22.04 hosts using > > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, > which > > went completely smoothly. We are a government facility so we are required > > to enable FIPS mode in the OS. I believe this last fact may be the cause > of > > the following: > > > > I wanted to enable mgmt-gateway so I made a service definition file to > > enable it and applied it. It appeared to work, but then “ceph health > > detail” showed: > > > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > > unsupported > > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > > routines] unsupported > > Module 'cephadm' has failed: [digital envelope routines] unsupported > > > > The mgmt-gateway service was started on all 3 nodes however. > > > > $ ceph orch ps --service_name mgmt-gateway > > NAME HOST PORTS STATUS REFRESHED AGE MEM > > USE MEM LIM VERSION IMAGE ID CONTAINER ID > > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (4m) 3m ago 4m > > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > > > I then attempted to remove the service definition with ceph orch rm, but > > it only succeeded in removing it on 1 of the 3 nodes. The process is > still > > running on the other 2, and the warning is still present. “ceph mgr fail > && > > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > > comes right back, and the process is still present on 2 nodes (as shown > by > > “ceph orch ps”). > > > > $ ceph mgr fail && ceph orch rm mgmt-gateway > > Removed service mgmt-gateway > > > > $ ceph orch ps --service_name mgmt-gateway > > NAME HOST PORTS STATUS REFRESHED AGE MEM > > USE MEM LIM VERSION IMAGE ID CONTAINER ID > > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (15h) 15h ago 15h > > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > > > $ ceph health detail > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > > unsupported > > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > > routines] unsupported > > Module 'cephadm' has failed: [digital envelope routines] unsupported > > > > Despite the 2 processes running, the configured service IP does not > > respond to a ping or requests from curl. > > > > Since we were getting ready to move this cluster into production I > decided > > to look for help here rather than proceed experimenting further and risk > > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > > (b) recover the cephadm module to get back to normal with no warnings; > and > > less importantly, (c) is there a way to get mgmt-gateway working in this > > environment? A little googling and AI querying suggests it may be an > issue > > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > > up my own proxy; I was just trying to use the built-in tools where > > possible.) > > > > Thanks in advance. > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])
by Konstantin Shalygin 24 Apr '26

24 Apr '26

Hi, Just guessing - it's Lets Encrypt certs? Or may be something InternalCA when certificate issues by golang application? k Sent from my iPhone > On 24 Apr 2026, at 15:54, Flemming Frandsen via ceph-users <ceph-users(a)ceph.io> wrote: > > I have a tentacle 20.2.1 cluster managed by cephadm, where I have > configured a certificate for the rgw using something like: > > service_type: ingress > service_id: rgw.default > service_name: ingress.rgw.default > placement: > count: 3 > label: rgw > spec: > backend_service: rgw.default > first_virtual_router_id: 50 > frontend_port: 443 > monitor_port: 8080 > virtual_ip: 10.0.40.3/24 > networks: > - 10.0.40.0/24 > ssl_cert: | > -----BEGIN CERTIFICATE----- > *snip* > -----END CERTIFICATE----- > -----BEGIN CERTIFICATE----- > *snip* > -----END CERTIFICATE----- > -----BEGIN RSA PRIVATE KEY----- > *snip* > -----END RSA PRIVATE KEY----- > > This initially worked perfectly, but I have just rotated the certificate by > generating the yaml with a fresh private key and associated certificate > then installing it with: > ceph orch apply -i new.yaml > > Clients accessing the rgw ingress get the correct certificate and work > perfectly, both before and after the key rotation. > > > I have a script that does this for me daily, and until the certificate was > rotated, everything was fine, but then I got this error: > > # ceph orch certmgr cert check > > Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > valid (error: Invalid cert/key pair: [('x509 certificate routines', '', > 'key values mismatch')]) > > After messing with the certificate a bunch I now get a completely > different, but also worrying error: > > # ceph orch certmgr cert check > - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > valid (error: Invalid private key: [('DECODER routines', '', > 'unsupported')]) > > The documentation offers no help: > https://docs.ceph.com/en/tentacle/cephadm/certmgr/ > > The error seems to come from this code: > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > So the error is an amputated exception from somewhere in openssl but the > actual exception is eaten by the code, so there is no way for me to debug > the situation: > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > I have tried installing the old certificate and key, that only leads the > the Invalid private key: [('DECODER routines', '', 'unsupported')] error. > > I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does > nothing. > > I have even tried copying the key and cert pem files into the mgr container > and verified them using the openssl installed in the container. > > The certificates are generated by cert-manager, running in k8s and a > cronjob that periodically pushes the new key/cert to the ceph cluster via > ssh. > > Has anyone seen these types of errors before? > > -- > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])
by Redouane Kachach 24 Apr '26

24 Apr '26

Hi Flemming, certmgr at the moment doesn't support fullchain PEM certificates: https://tracker.ceph.com/issues/75710 Can you plz paste the spec you are currently using that is leading to the issue). Right now the format of cert/key has to be something like: service_type: rgw service_id: myrgw spec: ssl: true ssl_key: | -----BEGIN PRIVATE KEY----- MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQCM58IqqXGTCJof ..... QsJ69XOQV9Os37zG2/hELQ1ycBs2Jw== -----END PRIVATE KEY----- ssl_cert: | -----BEGIN CERTIFICATE----- MIIF4TCCA8mgAwIBAgIUI1F5+FeKTeF5J+xYLdkEDP9J0OMwDQYJKoZIhvcNAQEL .... fhTw47C8Dry+2bOaYQPe6qvoFQ+x -----END CERTIFICATE----- Best, Redo. On Fri, Apr 24, 2026 at 2:52 PM Flemming Frandsen via ceph-users < ceph-users(a)ceph.io> wrote: > I have a tentacle 20.2.1 cluster managed by cephadm, where I have > configured a certificate for the rgw using something like: > > service_type: ingress > service_id: rgw.default > service_name: ingress.rgw.default > placement: > count: 3 > label: rgw > spec: > backend_service: rgw.default > first_virtual_router_id: 50 > frontend_port: 443 > monitor_port: 8080 > virtual_ip: 10.0.40.3/24 > networks: > - 10.0.40.0/24 > ssl_cert: | > -----BEGIN CERTIFICATE----- > *snip* > -----END CERTIFICATE----- > -----BEGIN CERTIFICATE----- > *snip* > -----END CERTIFICATE----- > -----BEGIN RSA PRIVATE KEY----- > *snip* > -----END RSA PRIVATE KEY----- > > This initially worked perfectly, but I have just rotated the certificate by > generating the yaml with a fresh private key and associated certificate > then installing it with: > ceph orch apply -i new.yaml > > Clients accessing the rgw ingress get the correct certificate and work > perfectly, both before and after the key rotation. > > > I have a script that does this for me daily, and until the certificate was > rotated, everything was fine, but then I got this error: > > # ceph orch certmgr cert check > > Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > valid (error: Invalid cert/key pair: [('x509 certificate routines', '', > 'key values mismatch')]) > > After messing with the certificate a bunch I now get a completely > different, but also worrying error: > > # ceph orch certmgr cert check > - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not > valid (error: Invalid private key: [('DECODER routines', '', > 'unsupported')]) > > The documentation offers no help: > https://docs.ceph.com/en/tentacle/cephadm/certmgr/ > > The error seems to come from this code: > > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > So the error is an amputated exception from somewhere in openssl but the > actual exception is eaten by the code, so there is no way for me to debug > the situation: > > https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… > > I have tried installing the old certificate and key, that only leads the > the Invalid private key: [('DECODER routines', '', 'unsupported')] error. > > I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does > nothing. > > I have even tried copying the key and cert pem files into the mgr container > and verified them using the openssl installed in the container. > > The certificates are generated by cert-manager, running in k8s and a > cronjob that periodically pushes the new key/cert to the ceph cluster via > ssh. > > Has anyone seen these types of errors before? > > -- > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')])
by Flemming Frandsen 24 Apr '26

24 Apr '26

I have a tentacle 20.2.1 cluster managed by cephadm, where I have configured a certificate for the rgw using something like: service_type: ingress service_id: rgw.default service_name: ingress.rgw.default placement: count: 3 label: rgw spec: backend_service: rgw.default first_virtual_router_id: 50 frontend_port: 443 monitor_port: 8080 virtual_ip: 10.0.40.3/24 networks: - 10.0.40.0/24 ssl_cert: | -----BEGIN CERTIFICATE----- *snip* -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- *snip* -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- *snip* -----END RSA PRIVATE KEY----- This initially worked perfectly, but I have just rotated the certificate by generating the yaml with a fresh private key and associated certificate then installing it with: ceph orch apply -i new.yaml Clients accessing the rgw ingress get the correct certificate and work perfectly, both before and after the key rotation. I have a script that does this for me daily, and until the certificate was rotated, everything was fine, but then I got this error: # ceph orch certmgr cert check Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid cert/key pair: [('x509 certificate routines', '', 'key values mismatch')]) After messing with the certificate a bunch I now get a completely different, but also worrying error: # ceph orch certmgr cert check - Certificate 'ingress_ssl_cert (ingress.rgw.default)' (user-made) is not valid (error: Invalid private key: [('DECODER routines', '', 'unsupported')]) The documentation offers no help: https://docs.ceph.com/en/tentacle/cephadm/certmgr/ The error seems to come from this code: https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… So the error is an amputated exception from somewhere in openssl but the actual exception is eaten by the code, so there is no way for me to debug the situation: https://github.com/ceph/ceph/blob/main/src/python-common/ceph/cryptotools/i… I have tried installing the old certificate and key, that only leads the the Invalid private key: [('DECODER routines', '', 'unsupported')] error. I have tried re-formatting the pkcs#1 RSA key as PKCS#8, but that does nothing. I have even tried copying the key and cert pem files into the mgr container and verified them using the openssl installed in the container. The certificates are generated by cert-manager, running in k8s and a cronjob that periodically pushes the new key/cert to the ceph cluster via ssh. Has anyone seen these types of errors before? -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Eugen Block 24 Apr '26

24 Apr '26

Yeah, the message > [ 171.485542] ceph: No mds server is up or the cluster is laggy can be misleading sometimes. But I'd recommend to create a new thread since the original issue is resolved. Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: >> >> It seems like removing packages did the trick, and there are up OSDs. > > > Apparently yes. I've already placed some test files in the cluster, and the > test went well. I've also added the cluster as a mount in /etc/fstab. The > mount itself was performed after mount -a, but not on startup. > > There is only one thing I've just found in the syslog at startup: > > [ 150.710312] ceph: loaded (mds proto 32) > [ 150.745343] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state > V1_BANN > ER) > [ 151.016656] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state > V1_BANN > ER) > [ 151.520015] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state > V1_BANN > ER) > [ 157.642865] libceph: mon3 (1)10.20.0.14:6789 session established > [ 157.645794] libceph: client274103 fsid > 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > [ 157.646023] libceph: mds0 (1)10.20.0.11:6801 socket error on write > [ 157.919274] libceph: mds0 (1)10.20.0.11:6801 socket error on write > [ 158.426046] libceph: mds0 (1)10.20.0.11:6801 socket error on write > [ 171.485479] ceph: mds0 reconnect start > [ 171.485542] ceph: No mds server is up or the cluster is laggy > > This contradicts the output of ceph -s, which still says, HEALTH_OK. > > Am Fr., 24. Apr. 2026 um 00:38 Uhr schrieb Anthony D'Atri via ceph-users < > ceph-users(a)ceph.io>: > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 24 Apr '26

24 Apr '26

> > It seems like removing packages did the trick, and there are up OSDs. Apparently yes. I've already placed some test files in the cluster, and the test went well. I've also added the cluster as a mount in /etc/fstab. The mount itself was performed after mount -a, but not on startup. There is only one thing I've just found in the syslog at startup: [ 150.710312] ceph: loaded (mds proto 32) [ 150.745343] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state V1_BANN ER) [ 151.016656] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state V1_BANN ER) [ 151.520015] libceph: mon1 (1)10.20.0.12:6789 socket closed (con state V1_BANN ER) [ 157.642865] libceph: mon3 (1)10.20.0.14:6789 session established [ 157.645794] libceph: client274103 fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b [ 157.646023] libceph: mds0 (1)10.20.0.11:6801 socket error on write [ 157.919274] libceph: mds0 (1)10.20.0.11:6801 socket error on write [ 158.426046] libceph: mds0 (1)10.20.0.11:6801 socket error on write [ 171.485479] ceph: mds0 reconnect start [ 171.485542] ceph: No mds server is up or the cluster is laggy This contradicts the output of ceph -s, which still says, HEALTH_OK. Am Fr., 24. Apr. 2026 um 00:38 Uhr schrieb Anthony D'Atri via ceph-users < ceph-users(a)ceph.io>: > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Frédéric Nass 23 Apr '26

23 Apr '26

Hi Anthony, Actually, ceph-bluestore-tool and other useful tools are embedded in the ceph container image (even tools like smartctl), so the only command the operator might want to have on a host is cephadm, installed either from cephadm package when available, or downloaded via curl/wget, or the operator could just chmod +x /var/lib/ceph/$(ceph fsid)/cephadm.* that is pushed by the orchestrator on every node and run it. Cheers, Frédéric. -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass(a)clyso.com Le jeu. 23 avr. 2026 à 03:06, Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> a écrit : > > > > On Apr 22, 2026, at 12:04 PM, Jacek Rużyczka via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > Hi Eugen, > > Hi Frédéric, > > > > One after the other, please: > > > > Do you have the ceph-osd package installed on the servers? It's known > >> to be a potential issue when trying to use cephadm while ceph packages > >> are installed (except for cephadm, of course, and maybe ceph-common). > > My understanding is that the orchestrator uses a different copy of cephadm > than what the package installs, in order to avoid conflicts. > > Issues with `ceph-osd` installed are problematic, though, as that's neeed > to get ceph-bluestore-tool. Is there a tracker for this? > > > And just one thought on the number of monitors: It's not useful to > >> have 4 MONs, an odd number is recommended. With 4 MONs you'd still > >> lose quorum when two MONs go down, so it doesn't offer any more > >> resiliency than 3 MONs. > > > > > > Shall I reduce it to 3? > > If you have at least 5 hosts, I would increase to 5. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: pg in 'backfill_wait+backfill_toofull' (and backfilling not converging)
by Wesley Dillingham 23 Apr '26

23 Apr '26

You should increase the number of PGs in that pool. From what I can see these OSDs only have about 12 PGs/OSD and you want to shoot for closer to 100 (usually over IMO), increase that pools pg_num as the pg autoscaler is telling you. Respectfully, *Wes Dillingham* On Thu, Apr 23, 2026 at 1:38 PM Massimo Sgaravatto via ceph-users < ceph-users(a)ceph.io> wrote: > Dear all > > I (try to) manage a ceph cluster which is used for different use cases: > cephfs, block storage for OpenStack and for a proxmox cluster, object > storage. > > I have a hdd-cephfs device class used only for the OSDs for the cephfs data > pool (cephfs_data). > > > There has been a massive writing to the cephfs file system and so the pool > and one OSD of that device class became 'nearfull' > > To try to address the problem I then moved 3 osds to the hdd-cephfs device > class. > > During the rebalance: > > - there was a short down of one node hosting one of the OSD belonging to > the hdd-cephfs device class > - many files in the cephfs file system got deleted > > > Now I have an on going backfill that doesn't seem to converge: > > pgs: 8824966/525185097 objects misplaced (1.680%) > 2580 active+clean > 17 active+remapped+backfill_wait > 13 active+clean+scrubbing+deep > 13 active+clean+scrubbing > 2 active+remapped+backfilling > 1 active+remapped+backfill_wait+backfill_toofull > > > and one pg is is in active+remapped+backfill_wait+backfill_toofull > > > I am also told that the pool needs more pgs: > > [root@ceph-mon-01 ~]# ceph health detail > HEALTH_WARN Low space hindering backfill (add storage if this doesn't > resolve itself): 1 pg backfill_toofull; 1 pools have too few placement > group\ > s > [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this > doesn't resolve itself): 1 pg backfill_toofull > pg 60.4 is active+remapped+backfill_wait+backfill_toofull, acting > [10,189,110] > [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups > Pool cephfs_data has 32 placement groups, should have 128 > > > These are the details of the pool: > > [root@ceph-mon-01 ~]# ceph df | grep 'cephfs_data ' > cephfs_data 60 32 16 TiB 9.48M 48 TiB 47.39 > 18 TiB > [root@ceph-mon-01 ~]# > > > pool 60 'cephfs_data' replicated size 3 min_size 2 crush_rule 11 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change > 4259332\ > flags hashpspool stripe_width 0 application cephfs read_balance_score 1.88 > > > and this [*] is the status of the OSDs belonging to that device class. > > As I said, the backfilling has been running for several days and doesn't > seem to get finalized (the number of misplaced objects even increases from > time to time, maybe because of the balancer ?) > > > > Should I simply be patient and wait for the end of the backfilling ? > What about the pg in active+remapped+backfill_wait+backfill_toofull ? > Should it "fix" by itself ? > > Or should I increase now the number of pgs of that pool ? > > Any suggestion will be highly appreciated ! > > Thanks, Massimo > > > > > [*] > > [root@ceph-mon-01 ~]# ceph osd df | grep cephfs > 70 hdd-cephfs 9.18259 1.00000 9.2 TiB 1.8 TiB 1.7 TiB 92 KiB > 2.6 GiB 7.4 TiB 19.08 0.59 3 up > 110 hdd-cephfs 11.00099 1.00000 11 TiB 6.5 TiB 6.4 TiB 114 KiB > 9.8 GiB 4.5 TiB 59.36 1.85 13 up > 130 hdd-cephfs 14.63959 1.00000 15 TiB 7.0 TiB 6.9 TiB 11 MiB > 10 GiB 7.6 TiB 47.98 1.49 14 up > 0 hdd-cephfs 18.26270 1.00000 18 TiB 5.9 TiB 5.8 TiB 10 MiB > 9.8 GiB 12 TiB 32.42 1.01 12 up > 189 hdd-cephfs 21.91499 1.00000 22 TiB 6.0 TiB 5.9 TiB 10 MiB > 10 GiB 16 TiB 27.31 0.85 12 up > 10 hdd-cephfs 14.72659 1.00000 15 TiB 6.0 TiB 5.9 TiB 10 MiB > 9.9 GiB 8.7 TiB 40.93 1.27 12 up > 25 hdd-cephfs 14.72659 1.00000 15 TiB 6.1 TiB 5.9 TiB 10 MiB > 10 GiB 8.6 TiB 41.56 1.29 12 up > 26 hdd-cephfs 14.72659 1.00000 15 TiB 1.3 TiB 1.1 TiB 10 MiB > 2.5 GiB 13 TiB 8.60 0.27 2 up > 170 hdd-cephfs 14.72659 1.00000 15 TiB 8.0 TiB 7.8 TiB 10 MiB > 13 GiB 6.8 TiB 54.10 1.68 16 up > 171 hdd-cephfs 14.72659 1.00000 15 TiB 368 GiB 190 GiB 11 MiB > 1.3 GiB 14 TiB 2.44 0.08 0 up > [root@ceph-mon-01 ~]# > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Eugen Block 23 Apr '26

23 Apr '26

Oh, apparently the last messages weren’t in reply to the list. It seems like removing packages did the trick, and there are up OSDs. Zitat von Eugen Block <eblock(a)nde.ag>: > Yeah, looks like it worked. :-) > > The IDs are part of the crush tree, you don’t need to worry about > them. I assume that you could also revert the unmanaged step (Ceph > orch set-managed …) and the rest of the OSDs would probably be > created as well. > > Zitat von Jacek Rużyczka <jacek.ruzyczka(a)googlemail.com>: > >>> >>> I would aim at keeping all nodes in the same state, so yes, remove >>> them from all Ceph nodes. >> >> >> Yep. >> >> Maybe before you reboot, zap the drives first, the >>> ones the orchestrator might have tried to build and failed. >> >> >> Already done that. >> >> ceph orch set-unmanaged osd.<your_osd_service> >> >> >> Do you mean this: >> >> root@blade3n1:/# ceph orch set-unmanaged osd.all-available-devices >> Set unmanaged to True for service osd.all-available-devices >> >> To deploy a single OSD you can run: >> >> >>> ceph orch daemon add osd <host> /dev/sdX >> >> >> So something like this: >> >> mixtile@blade3n1:~$ sudo ceph orch daemon add osd blade3n1:/dev/nvme0n1 >> Created osd(s) 3 on host 'blade3n1' >> >> But: >> >> mixtile@blade3n1:~$ sudo ceph orch daemon add osd blade3n2:/dev/nvme0n1 >> Created no osd(s) on host blade3n2; already created? >> >> Before I could disable the OSD process, it must have started creating OSDs >> on the other hosts: >> >> mixtile@blade3n1:~$ sudo ceph osd tree >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> -1 29.80798 root default >> -9 7.45200 host blade3n1 >> 3 ssd 7.45200 osd.3 up 1.00000 1.00000 >> -7 7.45200 host blade3n2 >> 2 ssd 7.45200 osd.2 up 1.00000 1.00000 >> -5 7.45200 host blade3n3 >> 1 ssd 7.45200 osd.1 up 1.00000 1.00000 >> -3 7.45200 host blade3n4 >> 0 ssd 7.45200 osd.0 up 1.00000 1.00000 >> >> But: What do the negative IDs mean? Yesterday, I only had the root >> default with >> ID -1. At least, the health check now says, HEALTH_OK: >> >> mixtile@blade3n1:~$ sudo ceph -s >> cluster: >> id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b >> health: HEALTH_OK >> >> services: >> mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) >> mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv >> mds: 1/1 daemons up, 1 standby >> osd: 4 osds: 4 up (since 10m), 4 in (since 60m) >> >> data: >> volumes: 1/1 healthy >> pools: 3 pools, 145 pgs >> objects: 24 objects, 579 KiB >> usage: 248 MiB used, 30 TiB / 30 TiB avail >> pgs: 145 active+clean >> >> So: Do I now have a working Ceph cluster?

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Frédéric Nass 23 Apr '26

23 Apr '26

Yes, Jacek. That cephadm file is the one the orchestrator pushes to every node and works with. But as Eugen said, that doesn't matter much. Could you share the output of 'id ceph' (on the Linux host) and 'id ceph' (in the container), and check for any UID mismatch? And follow Eugen's advice to remove the ceph packages? Best, Frédéric. -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass(a)clyso.com Le jeu. 23 avr. 2026 à 15:35, Jacek Rużyczka via ceph-users < ceph-users(a)ceph.io> a écrit : > Hi Frédéric, > > Do you mean this file? At least it's the only cephadm I've found in the > folder you told me: > > mixtile@blade3n2:~$ sudo ls -al > /var/lib/ceph/8aad3073-39a1-11f1-bf6e-f2704a1efa > 9b | grep cephadm > -rw-r--r-- 1 root root 787672 Apr 16 16:52 cephadm > .1a8853661a9c1798390b8e8 > d13c27688c1b1327a075745af2ee40ac466f0ac36 > > Regards > Jacek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: squid 19.2.4 QE validation status
by Laura Flores 23 Apr '26

23 Apr '26

The rerun of the rados suite on Build 3 looks good. Did not do a full analysis, but I do not see anything related to the ceph-volume blocker. RADOS approved. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Venky Shankar via dev <dev(a)ceph.io> Sent: Thursday, April 23, 2026 12:24 PM To: Yuri Weinstein <yweinste(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: squid 19.2.4 QE validation status Hi Yuri, On Tue, Apr 14, 2026 at 1:01 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky fs approved. failures are https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_proje… > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA -- Cheers, Venky

1 0

pg in 'backfill_wait+backfill_toofull' (and backfilling not converging)
by Massimo Sgaravatto 23 Apr '26

23 Apr '26

Dear all I (try to) manage a ceph cluster which is used for different use cases: cephfs, block storage for OpenStack and for a proxmox cluster, object storage. I have a hdd-cephfs device class used only for the OSDs for the cephfs data pool (cephfs_data). There has been a massive writing to the cephfs file system and so the pool and one OSD of that device class became 'nearfull' To try to address the problem I then moved 3 osds to the hdd-cephfs device class. During the rebalance: - there was a short down of one node hosting one of the OSD belonging to the hdd-cephfs device class - many files in the cephfs file system got deleted Now I have an on going backfill that doesn't seem to converge: pgs: 8824966/525185097 objects misplaced (1.680%) 2580 active+clean 17 active+remapped+backfill_wait 13 active+clean+scrubbing+deep 13 active+clean+scrubbing 2 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull and one pg is is in active+remapped+backfill_wait+backfill_toofull I am also told that the pool needs more pgs: [root@ceph-mon-01 ~]# ceph health detail HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; 1 pools have too few placement group\ s [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull pg 60.4 is active+remapped+backfill_wait+backfill_toofull, acting [10,189,110] [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool cephfs_data has 32 placement groups, should have 128 These are the details of the pool: [root@ceph-mon-01 ~]# ceph df | grep 'cephfs_data ' cephfs_data 60 32 16 TiB 9.48M 48 TiB 47.39 18 TiB [root@ceph-mon-01 ~]# pool 60 'cephfs_data' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 4259332\ flags hashpspool stripe_width 0 application cephfs read_balance_score 1.88 and this [*] is the status of the OSDs belonging to that device class. As I said, the backfilling has been running for several days and doesn't seem to get finalized (the number of misplaced objects even increases from time to time, maybe because of the balancer ?) Should I simply be patient and wait for the end of the backfilling ? What about the pg in active+remapped+backfill_wait+backfill_toofull ? Should it "fix" by itself ? Or should I increase now the number of pgs of that pool ? Any suggestion will be highly appreciated ! Thanks, Massimo [*] [root@ceph-mon-01 ~]# ceph osd df | grep cephfs 70 hdd-cephfs 9.18259 1.00000 9.2 TiB 1.8 TiB 1.7 TiB 92 KiB 2.6 GiB 7.4 TiB 19.08 0.59 3 up 110 hdd-cephfs 11.00099 1.00000 11 TiB 6.5 TiB 6.4 TiB 114 KiB 9.8 GiB 4.5 TiB 59.36 1.85 13 up 130 hdd-cephfs 14.63959 1.00000 15 TiB 7.0 TiB 6.9 TiB 11 MiB 10 GiB 7.6 TiB 47.98 1.49 14 up 0 hdd-cephfs 18.26270 1.00000 18 TiB 5.9 TiB 5.8 TiB 10 MiB 9.8 GiB 12 TiB 32.42 1.01 12 up 189 hdd-cephfs 21.91499 1.00000 22 TiB 6.0 TiB 5.9 TiB 10 MiB 10 GiB 16 TiB 27.31 0.85 12 up 10 hdd-cephfs 14.72659 1.00000 15 TiB 6.0 TiB 5.9 TiB 10 MiB 9.9 GiB 8.7 TiB 40.93 1.27 12 up 25 hdd-cephfs 14.72659 1.00000 15 TiB 6.1 TiB 5.9 TiB 10 MiB 10 GiB 8.6 TiB 41.56 1.29 12 up 26 hdd-cephfs 14.72659 1.00000 15 TiB 1.3 TiB 1.1 TiB 10 MiB 2.5 GiB 13 TiB 8.60 0.27 2 up 170 hdd-cephfs 14.72659 1.00000 15 TiB 8.0 TiB 7.8 TiB 10 MiB 13 GiB 6.8 TiB 54.10 1.68 16 up 171 hdd-cephfs 14.72659 1.00000 15 TiB 368 GiB 190 GiB 11 MiB 1.3 GiB 14 TiB 2.44 0.08 0 up [root@ceph-mon-01 ~]#

1 0

Re: squid 19.2.4 QE validation status
by Venky Shankar 23 Apr '26

23 Apr '26

Hi Yuri, On Tue, Apr 14, 2026 at 1:01 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75029#note-5 > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky fs approved. failures are https://tracker.ceph.com/projects/cephfs/wiki/Squid#Squid-v1924 > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA -- Cheers, Venky

1 0

Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Barber, Steve W. (Fed) 23 Apr '26

23 Apr '26

I have a pre-production cluster running on Ubuntu 22.04 hosts using cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which went completely smoothly. We are a government facility so we are required to enable FIPS mode in the OS. I believe this last fact may be the cause of the following: I wanted to enable mgmt-gateway so I made a service definition file to enable it and applied it. It appeared to work, but then “ceph health detail” showed: HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported The mgmt-gateway service was started on all 3 nodes however. $ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.alfred alfred running (4m) 3m ago 4m 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (4m) 3m ago 4m 4452k - 1.26.1 c2348906c6ff 329705c23aad I then attempted to remove the service definition with ceph orch rm, but it only succeeded in removing it on 1 of the 3 nodes. The process is still running on the other 2, and the warning is still present. “ceph mgr fail && ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it comes right back, and the process is still present on 2 nodes (as shown by “ceph orch ps”). $ ceph mgr fail && ceph orch rm mgmt-gateway Removed service mgmt-gateway $ ceph orch ps --service_name mgmt-gateway NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h 4492k - 1.26.1 c2348906c6ff 68ce6d58327a mgmt-gateway.mask mask running (15h) 15h ago 15h 4448k - 1.26.1 c2348906c6ff 329705c23aad $ ceph health detail HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] unsupported [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope routines] unsupported Module 'cephadm' has failed: [digital envelope routines] unsupported Despite the 2 processes running, the configured service IP does not respond to a ping or requests from curl. Since we were getting ready to move this cluster into production I decided to look for help here rather than proceed experimenting further and risk mucking it up worse. How can I (a) diagnose what’s actually going wrong; (b) recover the cephadm module to get back to normal with no warnings; and less importantly, (c) is there a way to get mgmt-gateway working in this environment? A little googling and AI querying suggests it may be an issue with certs/ciphers so maybe I can provide my own? (Yes I know I could set up my own proxy; I was just trying to use the built-in tools where possible.) Thanks in advance.

1 0

Ceph Day London CFP - Call for Papers
by Lee Sanders 23 Apr '26

23 Apr '26

Hi everyone! Just a reminder the CFP for Ceph Days London closes on Friday 24th April (tomorrow) at around noon UK time, if you are interested in being a speaker at this event please submit your talk here: https://ceph.io/en/community/events/2026/ceph-days-london/ Thank you! Lee Sanders Lee Sanders. Performance Architect, IBM Master Inventor, Ceph Development Email: mailto:ljsanders@uk.ibm.com CSOA10, IBM Hursley Park Badges: https://www.credly.com/users/lee-sanders/badges Hursley, SO21 2JN United Kingdom Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN

1 0

squid 19.2.4 QE validation status
by Yuri Weinstein 23 Apr '26

23 Apr '26

Details of this release are summarized here: https://tracker.ceph.com/issues/75029#note-5 Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/squid-x: Laura, Patrick upgrade/reef-x: Laura, Patrick upgrade/telemetry-upgrade: Laura, Patrick upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 3

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Eugen Block 23 Apr '26

23 Apr '26

I'd recommend to remove these: - ceph-base - ceph-mds - ceph-osd - ceph-volume Don't bother too much about the cephadm binary right now, it's useful to know but not that relevant to get your cluster into a proper state. Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > OK, this is what I've got on all four nodes: > > mixtile@blade3n2:~$ sudo apt list --installed | grep ceph > [sudo] password for mixtile: > > WARNING: apt does not have a stable CLI interface. Use with caution in > scripts. > > ceph-base/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed,automatic] > ceph-common/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed] > ceph-mds/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed,automatic] > ceph-osd/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed] > ceph-volume/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 all > [installed] > cephadm/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed] > libcephfs2/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 > [installed,automatic] > python3-ceph-argparse/noble-updates,noble-security,now > 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] > python3-ceph-common/noble-updates,noble-security,now > 19.2.3-0ubuntu0.24.04.3 all [installed,automatic] > python3-cephfs/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 > arm64 [installed,automatic] > > As you can see, most packages have been installed automatically, probably > because of a dep. *Which of them shall I remove on hosts #2, #3, and #4?* > > And: Unfortunately, I've only got 4 hosts. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 23 Apr '26

23 Apr '26

Hi Frédéric, Do you mean this file? At least it's the only cephadm I've found in the folder you told me: mixtile@blade3n2:~$ sudo ls -al /var/lib/ceph/8aad3073-39a1-11f1-bf6e-f2704a1efa 9b | grep cephadm -rw-r--r-- 1 root root 787672 Apr 16 16:52 cephadm .1a8853661a9c1798390b8e8 d13c27688c1b1327a075745af2ee40ac466f0ac36 Regards Jacek

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 23 Apr '26

23 Apr '26

OK, this is what I've got on all four nodes: mixtile@blade3n2:~$ sudo apt list --installed | grep ceph [sudo] password for mixtile: WARNING: apt does not have a stable CLI interface. Use with caution in scripts. ceph-base/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] ceph-common/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed] ceph-mds/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] ceph-osd/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed] ceph-volume/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 all [installed] cephadm/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed] libcephfs2/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] python3-ceph-argparse/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] python3-ceph-common/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 all [installed,automatic] python3-cephfs/noble-updates,noble-security,now 19.2.3-0ubuntu0.24.04.3 arm64 [installed,automatic] As you can see, most packages have been installed automatically, probably because of a dep. *Which of them shall I remove on hosts #2, #3, and #4?* And: Unfortunately, I've only got 4 hosts.

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 23 Apr '26

23 Apr '26

IIRC, I think I was able to reproduce mon sync issues back then on Quiny or Octopus on fresh (virtual) deployments, so I don't think it's related to the cluster age. :-) But it's been too long, I remember only vaguely. What I remember is I never could pinpoint to a specific root cause. And I also remember other users having issues as well, reported here on the list. So it seems like a broken upgrade path to Q, I would say. I guess we're lucky that we mostly upgraded our own and our customers clusters from N to P, then P to R. We usually skip one release, except last year when we upgraded our own cluster from R to S. Zitat von Janne Johansson via ceph-users <ceph-users(a)ceph.io>: >> Correct! It's an old cluster initialy deployed with ceph-deploy in the >> Hammer days and upgraded since then. Right now is manually managed (but >> one of the targets at the end of the upgrade is to switch to cephadm). > >> We didn't wait for a long time before stopping the new MON a regain >> "control" of the cluster, maybe 10 minutes max... >> >> > - Inspecting debug logs on the syncing MON could help finding the root >> > cause. >> >> With "syncing MON" you mean the MON which is just added to the cluster, >> right? Collecting this kind of logs is on the to-do list, but blocked >> because the "log problem" mentioned bellow. > > Sorry for coming in late, but we also had troubles upgrading a cluster > from Quincy from Octopus, which coincided with moving from CentOS to > Ubuntu. In our case, the Oct mons started crashing and would not start > (crashing before being able to join) as soon as I had two Quincy mons > joined. At the time, we had little choice than to just upgrade the Oct > hosts to Q and then they would join. > > A potentially interesting similarity is the age of our cluster, ours > is from Infernalis or Jewel at the latest, and then upgraded in many > steps. > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Upgrade problem from Octopus to Quincy
by Janne Johansson 23 Apr '26

23 Apr '26

> Correct! It's an old cluster initialy deployed with ceph-deploy in the > Hammer days and upgraded since then. Right now is manually managed (but > one of the targets at the end of the upgrade is to switch to cephadm). > We didn't wait for a long time before stopping the new MON a regain > "control" of the cluster, maybe 10 minutes max... > > > - Inspecting debug logs on the syncing MON could help finding the root > > cause. > > With "syncing MON" you mean the MON which is just added to the cluster, > right? Collecting this kind of logs is on the to-do list, but blocked > because the "log problem" mentioned bellow. Sorry for coming in late, but we also had troubles upgrading a cluster from Quincy from Octopus, which coincided with moving from CentOS to Ubuntu. In our case, the Oct mons started crashing and would not start (crashing before being able to join) as soon as I had two Quincy mons joined. At the time, we had little choice than to just upgrade the Oct hosts to Q and then they would join. A potentially interesting similarity is the age of our cluster, ours is from Infernalis or Jewel at the latest, and then upgraded in many steps. -- May the most significant bit of your life be positive.

1 0

Re: squid 19.2.4 QE validation status
by wissem mimouna 23 Apr '26

23 Apr '26

Hi , Sorry for the inconvience , I'm just wondering if you have any ideas of when this PR https://github.com/ceph/ceph/pull/66501 will be available in a stable release ? Thanks On 22.04.2026 01:06, Yuri Weinstein via ceph-users wrote: > We need to merge two PRs: > https://github.com/ceph/ceph/pull/68526 > https://github.com/ceph/ceph/pull/68528 > and rerun upgrades > > On Tue, Apr 21, 2026 at 2:29 PM Laura Flores <lflores(a)ibm.com> wrote: > >> upgrade/quincy-x and upgrade/squid-x are approved: >> https://tracker.ceph.com/projects/rados/wiki/SQUID#v1924-httpstrackercephco… >> >> The upgrade/squid-p2p tests need to be rerun with >> https://github.com/ceph/ceph/pull/68526 included. >> >> Thanks, >> Laura >> >> *Laura Flores* >> Software Engineer; Ceph Storage >> 1 708 738 8804 Mobile >> She/Her/Hers >> >> IBM >> ------------------------------ >> *From:* Afreen via ceph-users <ceph-users(a)ceph.io> >> *Sent:* Tuesday, April 21, 2026 12:55 PM >> *To:* Yuri Weinstein <yweinste(a)redhat.com>; Redouane Kachach < >> rkachach(a)redhat.com> >> *Cc:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> >> *Subject:* [EXTERNAL] [ceph-users] Re: squid 19.2.4 QE validation status >> >> * Dashboard approved.* >> >> On Tue, Apr 14, 2026 at 1:00 AM Yuri Weinstein via dev <dev(a)ceph.io> >> wrote: >> >>> Details of this release are summarized here: >>> >>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… >>> Release Notes - TBD >>> LRC upgrade - N/A >>> >>> Seeking approvals/reviews for: >>> >>> smoke: PASSED >>> >>> rados: Radek, Laura, Afreen, Travis >>> rgw: Adam Emerson >>> fs: Venky >>> orch/ceph-volume: Redouane & Guillaume >>> rbd: Ilya >>> krbd: Ilya >>> >>> upgrade/squid-x: Laura, Patrick >>> upgrade/reef-x: Laura, Patrick >>> upgrade/telemetry-upgrade: Laura, Patrick >>> upgrade/squid-p2p: Laura, Patrick >>> >>> upgrade-clients:client-upgrade-reef-squid-squid: PASSED >>> upgrade-clients:client-upgrade-quincy-squid-squid: PASSED >>> >>> perf-basic: PASSED >>> crimson-rados: N/A >>> powercycle: PASSED >>> >>> TIA >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Eugen Block 23 Apr '26

23 Apr '26

Hi, a quick look into the code [0] shows (note that I'm not a developer): def set_mirror_info(local_cluster_id, local_fsid, remote_fs): log.info(f'setting {local_cluster_id}::{local_fsid} on remote') try: remote_fs.setxattr('/', 'ceph.mirror.info', f'cluster_id={local_cluster_id} fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE) except cephfs.Error as e: if e.errno == errno.EEXIST: My interpretation is that the setxattr command fails on the remote site. Could this be an auth caps issue? Did you verify the keyrings and their permissions? This is the only lead I got. Regards, Eugen [0] https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20cbd6a/… Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>: > Hi, > > This is issue with CephFS snapshot mirroring setup between two > clusters and would appreciate your assistance. > > Environment: > > * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) > [cid:image001.png@01DCD199.2933E2E0] > * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) > [cid:image002.png@01DCD199.2933E2E0] > * Filesystem: cephfs > * Issue: Unable to import peer bootstrap token > > Problem Description: > When attempting to import a peer bootstrap token using: > ceph fs snapshot mirror peer_bootstrap import cephfs <token> > I receive the error: > Error EEXIST: already an active peer mgr.server reply reply (17) > File exists already an active peer > However, ceph fs snapshot mirror peer_list cephfs returns an empty > list [], indicating no peers are configured. > > Steps Performed: > Initial Setup: > 1. Enabled mirroring module: ceph mgr module enable mirroring > 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror > 3. Enabled mirroring on filesystem: ceph fs snapshot mirror > enable cephfs > 4. Created remote user and bootstrap token on backup cluster > 5. Verified mirroring daemons are running: ceph fs snapshot > mirror daemon status > Troubleshooting Attempts: > 1. Reset filesystem mirroring: > ceph fs snapshot mirror disable cephfs > ceph fs snapshot mirror enable cephfs > 2. Module reset: > ceph mgr module disable mirroring > ceph mgr module enable mirroring > 3. Service recreation: > ceph orch rm cephfs-mirror > ceph orch apply cephfs-mirror > 4. Checked for stale configuration: > o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror > o ceph fs snapshot mirror dirmap cephfs returns empty > o ceph fs snapshot mirror status cephfs shows no active mirroring > > Current Status: > > * Mirroring daemons: Running and visible in daemon status > * Peer list: Empty [] > * Filesystem mirroring: Enabled > * Error persists: EEXIST despite no visible peers > > Diagnostic Information: > > * MGR logs show: mgr.server reply reply (17) File exists already > an active peer > * No connectivity issues between clusters (ports 6789, 3300, > 6800+ tested) > * Both clusters are healthy with no other issues > > Regards, > Anantha

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Anthony D'Atri 23 Apr '26

23 Apr '26

> On Apr 22, 2026, at 12:04 PM, Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Eugen, > Hi Frédéric, > > One after the other, please: > > Do you have the ceph-osd package installed on the servers? It's known >> to be a potential issue when trying to use cephadm while ceph packages >> are installed (except for cephadm, of course, and maybe ceph-common). My understanding is that the orchestrator uses a different copy of cephadm than what the package installs, in order to avoid conflicts. Issues with `ceph-osd` installed are problematic, though, as that's neeed to get ceph-bluestore-tool. Is there a tracker for this? > And just one thought on the number of monitors: It's not useful to >> have 4 MONs, an odd number is recommended. With 4 MONs you'd still >> lose quorum when two MONs go down, so it doesn't offer any more >> resiliency than 3 MONs. > > > Shall I reduce it to 3? If you have at least 5 hosts, I would increase to 5. >

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 22 Apr '26

22 Apr '26

Hi, I'm glad you could resolve the issue. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > On 15/04/26 12:29, Iztok Gregori via ceph-users wrote: >> On 15/04/26 12:08, Eugen Block via ceph-users wrote: >>> Yeah, looks like it. I just compared the output with a fresh >>> Tentacle install, I only enabled log-to-file during bootstrap, so >>> log_to_file is true in my cluster as well. There's no >>> leveldb_log_to_ceph_log option anymore since the default has been >>> rocksdb for a while now. >> >> I put 'log_to_journald' to true and journalctl started to log. A >> lot (with debug=20) :-) >> >> Now I'm ready to debug the joining problem. I will keep you posted. >> > > Just an update, the restart of the Octopus daemon didn't had an > effect, one the new monitors were started they still stayed in the > "electing" phase causing the 'ceph ...' commands to hang. Not a lot > in the logs, or nothing that I was able to point out. > > After that we decided to stop the Octopus daemon and ran the cluster > with the remaining 2 Quincy ones. This did the job. > > Once the Octopus monitor wasn't running any more the new Quincy > monitors rapidly joined the cluster. And now we are able to continue > our upgrade path to Squid/Tentacle and cephadm. > > I'm not 100% sure of what was wrong, it seems that in an not > completely upgraded Quincy monitor cluster is not possible to add > new monitors. Once the monitors were all the same version, it > started to work again. > > > Thanks a lot for your help! > > Cheers > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 22 Apr '26

22 Apr '26

Hi Eugen, Hi Frédéric, One after the other, please: Do you have the ceph-osd package installed on the servers? It's known > to be a potential issue when trying to use cephadm while ceph packages > are installed (except for cephadm, of course, and maybe ceph-common). Yes, ceph-osd is installed on all nodes. This must have happened automatically when setting them up. Is this wrong? Is it possible that there are still remainders from previous attempts > and not everything was properly cleaned up? I've looked for remainders of old installations. In fact, there was one, a ruin, which remained from an unfinished bootstrap (because my whole cluster had broken down). I've removed them, but no use. Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints. Yo. In fact, I found out about my issue in the logfile, but it doesn't say, *which *file can't be accessed due to lack of permission. And just one thought on the number of monitors: It's not useful to > have 4 MONs, an odd number is recommended. With 4 MONs you'd still > lose quorum when two MONs go down, so it doesn't offer any more > resiliency than 3 MONs. Shall I reduce it to 3? And: In addition to Eugen's recommendations, could you also check for > conflicting 'ceph' user UIDs on the host and in containers (where the ceph > UID is 167)? Just want to rule out this possibility [1]. I've seen user #167 several times in /var/lib/ceph and its subfolders without knowing who he in fact is. It's not user ceph. Regards Jacek

2 2

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Frédéric Nass 22 Apr '26

22 Apr '26

Hi Jacek, Hi Eugen, In addition to Eugen's recommendations, could you also check for conflicting 'ceph' user UIDs on the host and in containers (where the ceph UID is 167)? Just want to rule out this possibility [1]. Best, Frédéric. [1] https://tracker.ceph.com/issues/63319 -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Check our online Config Diff Tool, it's great! https://analyzer.clyso.com/#/analyzer/config-diff https://clyso.com | frederic.nass(a)clyso.com Le mar. 21 avr. 2026 à 22:20, Eugen Block via ceph-users <ceph-users(a)ceph.io> a écrit : > Hi, > > do you have the ceph-osd package installed on the servers? It's known > to be a potential issue when trying to use cephadm while ceph packages > are installed (except for cephadm, of course, and maybe ceph-common). > > Is it possible that there are still remainders from previous attempts > and not everything was properly cleaned up? > > The host directory /var/lib/ceph/osd is not relevant for cephadm, it > maps /var/lib/ceph/{FSID}/osd/ceph-{OSD_ID} into the container to > /var/lib/ceph/osd/ceph-{OSD_ID}. So the permissions should be okay if > you didn't modify them within the container. > > How exactly did you zap the drives? Did you reboot after zapping? What > usually works for me is 'cephadm ceph-volume lvm zap --destroy > /dev/sdX' or via orchestrator 'ceph orch device zap {HOST} /dev/sdX'. > Sometimes zapping isn't enough and I need to reboot (but usually VMs > when I've been experimenting). > > Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints. > > What's the output of 'ceph orch device ls'? > > And just one thought on the number of monitors: It's not useful to > have 4 MONs, an odd number is recommended. With 4 MONs you'd still > lose quorum when two MONs go down, so it doesn't offer any more > resiliency than 3 MONs. > > Regards, > Eugen > > Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > > > Hi, > > > > I've got the following issue when trying to build the OSD store on a > 4-node > > Ceph cluster: > > > > mixtile@blade3n1:~$ sudo ceph orch apply osd --all-available-devices > > Scheduled osd.all-available-devices update... > > > > Unfortunately, I *always *get this error message in the cephadm log: > > > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr Running > > command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore > > --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - > > --osdspec-affinity all-available-devices --osd-data > > /var/lib/ceph/osd/ceph-3/ --osd-uuid 3a8350dd-023a-49ae-8e4e-86b51082aec2 > > --setuser ceph --setgroup ceph > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:16.656+0000 ffffba0b0040 -1 > > bluestore(/var/lib/ceph/osd/ceph-3//block) No valid bdev label found > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:16.962+0000 ffffba0b0040 -1 > > bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 > > /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 > > /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 OSD::mkfs: ObjectStore::mkfs > > failed with error (13) Permission denied > > 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr > stderr: > > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 ** ERROR: error creating > > empty object store in /var/lib/ceph/osd/ceph-3/: (13) Permission denied > > 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr --> Was > > unable to complete a new OSD, will rollback changes > > > > I've checked the permissions of /var/lib/ceph/osd on all nodes, but can't > > find a mistake there: > > > > mixtile@blade3n1:~$ sudo ls -al /var/lib/ceph/osd > > total 8 > > drwxrwxrwx 2 ceph ceph 4096 Apr 21 15:57 . > > drwxr-x--- 14 ceph ceph 4096 Apr 21 14:54 .. > > > > Isn't ceph the correct user here? I've also added user ceph to group > disk, > > so that the drive /dev/nvme0n1 can be accessed directly by ceph, but no > use: > > > > mixtile@blade3n1:~$ ls -al /dev/nvme0n1 > > brw-rw---- 1 root disk 259, 0 Apr 21 17:05 /dev/nvme0n1 > > > > I've got four active hosts in my cluster: > > > > mixtile@blade3n1:~$ sudo ceph orch host ls > > HOST ADDR LABELS STATUS > > blade3n1 10.20.0.11 _admin > > blade3n2 10.20.0.12 > > blade3n3 10.20.0.13 > > blade3n4 10.20.0.14 > > 4 hosts in cluster > > > > At Reddit, they suggest zapping all drives. I've done that, but no use. > > > > Of course, the cluster can't work in such a situation: > > > > mixtile@blade3n1:~$ sudo ceph -s > > cluster: > > id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > > health: HEALTH_WARN > > Failed to apply 1 service(s): osd.all-available-devices > > 1 MDSs report slow metadata IOs > > Reduced data availability: 2 pgs inactive > > OSD count 1 < osd_pool_default_size 3 > > > > services: > > mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) > > mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv > > mds: 1/1 daemons up, 1 standby > > osd: 1 osds: 0 up, 0 in (since 7m) > > > > data: > > volumes: 1/1 healthy > > pools: 2 pools, 2 pgs > > objects: 0 objects, 0 B > > usage: 0 B used, 0 B / 0 B avail > > pgs: 100.000% pgs unknown > > 2 unknown > > > > progress: > > Global Recovery Event (0s) > > [............................] > > > > What's wrong here? I've got Ceph 19.2.3 Squid (stable) on Ubuntu 24.04.4 > > LTS (aarch64). Thank you. > > > > Regards > > Jacek Rużyczka > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 22 Apr '26

22 Apr '26

On 15/04/26 12:29, Iztok Gregori via ceph-users wrote: > On 15/04/26 12:08, Eugen Block via ceph-users wrote: >> Yeah, looks like it. I just compared the output with a fresh Tentacle >> install, I only enabled log-to-file during bootstrap, so log_to_file >> is true in my cluster as well. There's no leveldb_log_to_ceph_log >> option anymore since the default has been rocksdb for a while now. > > I put 'log_to_journald' to true and journalctl started to log. A lot > (with debug=20) :-) > > Now I'm ready to debug the joining problem. I will keep you posted. > Just an update, the restart of the Octopus daemon didn't had an effect, one the new monitors were started they still stayed in the "electing" phase causing the 'ceph ...' commands to hang. Not a lot in the logs, or nothing that I was able to point out. After that we decided to stop the Octopus daemon and ran the cluster with the remaining 2 Quincy ones. This did the job. Once the Octopus monitor wasn't running any more the new Quincy monitors rapidly joined the cluster. And now we are able to continue our upgrade path to Squid/Tentacle and cephadm. I'm not 100% sure of what was wrong, it seems that in an not completely upgraded Quincy monitor cluster is not possible to add new monitors. Once the monitors were all the same version, it started to work again. Thanks a lot for your help! Cheers Iztok

1 0

CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Adiga, Anantha 22 Apr '26

22 Apr '26

Hi, This is issue with CephFS snapshot mirroring setup between two clusters and would appreciate your assistance. Environment: * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) [cid:image001.png@01DCD199.2933E2E0] * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) [cid:image002.png@01DCD199.2933E2E0] * Filesystem: cephfs * Issue: Unable to import peer bootstrap token Problem Description: When attempting to import a peer bootstrap token using: ceph fs snapshot mirror peer_bootstrap import cephfs <token> I receive the error: Error EEXIST: already an active peer mgr.server reply reply (17) File exists already an active peer However, ceph fs snapshot mirror peer_list cephfs returns an empty list [], indicating no peers are configured. Steps Performed: Initial Setup: 1. Enabled mirroring module: ceph mgr module enable mirroring 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror 3. Enabled mirroring on filesystem: ceph fs snapshot mirror enable cephfs 4. Created remote user and bootstrap token on backup cluster 5. Verified mirroring daemons are running: ceph fs snapshot mirror daemon status Troubleshooting Attempts: 1. Reset filesystem mirroring: ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable cephfs 2. Module reset: ceph mgr module disable mirroring ceph mgr module enable mirroring 3. Service recreation: ceph orch rm cephfs-mirror ceph orch apply cephfs-mirror 4. Checked for stale configuration: o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror o ceph fs snapshot mirror dirmap cephfs returns empty o ceph fs snapshot mirror status cephfs shows no active mirroring Current Status: * Mirroring daemons: Running and visible in daemon status * Peer list: Empty [] * Filesystem mirroring: Enabled * Error persists: EEXIST despite no visible peers Diagnostic Information: * MGR logs show: mgr.server reply reply (17) File exists already an active peer * No connectivity issues between clusters (ports 6789, 3300, 6800+ tested) * Both clusters are healthy with no other issues Regards, Anantha

1 0

Re: squid 19.2.4 QE validation status
by Laura Flores 21 Apr '26

21 Apr '26

upgrade/quincy-x and upgrade/squid-x are approved: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1924-httpstrackercephco… The upgrade/squid-p2p tests need to be rerun with https://github.com/ceph/ceph/pull/68526 included. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Afreen via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, April 21, 2026 12:55 PM To: Yuri Weinstein <yweinste(a)redhat.com>; Redouane Kachach <rkachach(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] Re: squid 19.2.4 QE validation status * Dashboard approved.* On Tue, Apr 14, 2026 at 1:00 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > Details of this release are summarized here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: ceph orch apply osd error: creating empty object store: (13) Permission denied
by Eugen Block 21 Apr '26

21 Apr '26

Hi, do you have the ceph-osd package installed on the servers? It's known to be a potential issue when trying to use cephadm while ceph packages are installed (except for cephadm, of course, and maybe ceph-common). Is it possible that there are still remainders from previous attempts and not everything was properly cleaned up? The host directory /var/lib/ceph/osd is not relevant for cephadm, it maps /var/lib/ceph/{FSID}/osd/ceph-{OSD_ID} into the container to /var/lib/ceph/osd/ceph-{OSD_ID}. So the permissions should be okay if you didn't modify them within the container. How exactly did you zap the drives? Did you reboot after zapping? What usually works for me is 'cephadm ceph-volume lvm zap --destroy /dev/sdX' or via orchestrator 'ceph orch device zap {HOST} /dev/sdX'. Sometimes zapping isn't enough and I need to reboot (but usually VMs when I've been experimenting). Did you inspect the logs? /var/log/ceph/cephadm.log could show some hints. What's the output of 'ceph orch device ls'? And just one thought on the number of monitors: It's not useful to have 4 MONs, an odd number is recommended. With 4 MONs you'd still lose quorum when two MONs go down, so it doesn't offer any more resiliency than 3 MONs. Regards, Eugen Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I've got the following issue when trying to build the OSD store on a 4-node > Ceph cluster: > > mixtile@blade3n1:~$ sudo ceph orch apply osd --all-available-devices > Scheduled osd.all-available-devices update... > > Unfortunately, I *always *get this error message in the cephadm log: > > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr Running > command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore > --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - > --osdspec-affinity all-available-devices --osd-data > /var/lib/ceph/osd/ceph-3/ --osd-uuid 3a8350dd-023a-49ae-8e4e-86b51082aec2 > --setuser ceph --setgroup ceph > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:16.656+0000 ffffba0b0040 -1 > bluestore(/var/lib/ceph/osd/ceph-3//block) No valid bdev label found > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:16.962+0000 ffffba0b0040 -1 > bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 > /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 > /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied > 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 OSD::mkfs: ObjectStore::mkfs > failed with error (13) Permission denied > 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: > 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 ** ERROR: error creating > empty object store in /var/lib/ceph/osd/ceph-3/: (13) Permission denied > 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr --> Was > unable to complete a new OSD, will rollback changes > > I've checked the permissions of /var/lib/ceph/osd on all nodes, but can't > find a mistake there: > > mixtile@blade3n1:~$ sudo ls -al /var/lib/ceph/osd > total 8 > drwxrwxrwx 2 ceph ceph 4096 Apr 21 15:57 . > drwxr-x--- 14 ceph ceph 4096 Apr 21 14:54 .. > > Isn't ceph the correct user here? I've also added user ceph to group disk, > so that the drive /dev/nvme0n1 can be accessed directly by ceph, but no use: > > mixtile@blade3n1:~$ ls -al /dev/nvme0n1 > brw-rw---- 1 root disk 259, 0 Apr 21 17:05 /dev/nvme0n1 > > I've got four active hosts in my cluster: > > mixtile@blade3n1:~$ sudo ceph orch host ls > HOST ADDR LABELS STATUS > blade3n1 10.20.0.11 _admin > blade3n2 10.20.0.12 > blade3n3 10.20.0.13 > blade3n4 10.20.0.14 > 4 hosts in cluster > > At Reddit, they suggest zapping all drives. I've done that, but no use. > > Of course, the cluster can't work in such a situation: > > mixtile@blade3n1:~$ sudo ceph -s > cluster: > id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > health: HEALTH_WARN > Failed to apply 1 service(s): osd.all-available-devices > 1 MDSs report slow metadata IOs > Reduced data availability: 2 pgs inactive > OSD count 1 < osd_pool_default_size 3 > > services: > mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) > mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv > mds: 1/1 daemons up, 1 standby > osd: 1 osds: 0 up, 0 in (since 7m) > > data: > volumes: 1/1 healthy > pools: 2 pools, 2 pgs > objects: 0 objects, 0 B > usage: 0 B used, 0 B / 0 B avail > pgs: 100.000% pgs unknown > 2 unknown > > progress: > Global Recovery Event (0s) > [............................] > > What's wrong here? I've got Ceph 19.2.3 Squid (stable) on Ubuntu 24.04.4 > LTS (aarch64). Thank you. > > Regards > Jacek Rużyczka > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: squid 19.2.4 QE validation status
by Afreen 21 Apr '26

21 Apr '26

* Dashboard approved.* On Tue, Apr 14, 2026 at 1:00 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75029#note-5 > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA >

1 0

ceph orch apply osd error: creating empty object store: (13) Permission denied
by Jacek Rużyczka 21 Apr '26

21 Apr '26

Hi, I've got the following issue when trying to build the OSD store on a 4-node Ceph cluster: mixtile@blade3n1:~$ sudo ceph orch apply osd --all-available-devices Scheduled osd.all-available-devices update... Unfortunately, I *always *get this error message in the cephadm log: 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid 3a8350dd-023a-49ae-8e4e-86b51082aec2 --setuser ceph --setgroup ceph 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:16.656+0000 ffffba0b0040 -1 bluestore(/var/lib/ceph/osd/ceph-3//block) No valid bdev label found 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:16.962+0000 ffffba0b0040 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 bdev(0xaaaae79c7400 /var/lib/ceph/osd/ceph-3//block) open open got: (13) Permission denied 2026-04-21 16:54:29,562 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr stderr: 2026-04-21T14:54:17.769+0000 ffffba0b0040 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3/: (13) Permission denied 2026-04-21 16:54:29,563 ffff8c5b9720 INFO /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes I've checked the permissions of /var/lib/ceph/osd on all nodes, but can't find a mistake there: mixtile@blade3n1:~$ sudo ls -al /var/lib/ceph/osd total 8 drwxrwxrwx 2 ceph ceph 4096 Apr 21 15:57 . drwxr-x--- 14 ceph ceph 4096 Apr 21 14:54 .. Isn't ceph the correct user here? I've also added user ceph to group disk, so that the drive /dev/nvme0n1 can be accessed directly by ceph, but no use: mixtile@blade3n1:~$ ls -al /dev/nvme0n1 brw-rw---- 1 root disk 259, 0 Apr 21 17:05 /dev/nvme0n1 I've got four active hosts in my cluster: mixtile@blade3n1:~$ sudo ceph orch host ls HOST ADDR LABELS STATUS blade3n1 10.20.0.11 _admin blade3n2 10.20.0.12 blade3n3 10.20.0.13 blade3n4 10.20.0.14 4 hosts in cluster At Reddit, they suggest zapping all drives. I've done that, but no use. Of course, the cluster can't work in such a situation: mixtile@blade3n1:~$ sudo ceph -s cluster: id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b health: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 1 MDSs report slow metadata IOs Reduced data availability: 2 pgs inactive OSD count 1 < osd_pool_default_size 3 services: mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv mds: 1/1 daemons up, 1 standby osd: 1 osds: 0 up, 0 in (since 7m) data: volumes: 1/1 healthy pools: 2 pools, 2 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 2 unknown progress: Global Recovery Event (0s) [............................] What's wrong here? I've got Ceph 19.2.3 Squid (stable) on Ubuntu 24.04.4 LTS (aarch64). Thank you. Regards Jacek Rużyczka

1 0

Re: squid 19.2.4 QE validation status
by Redouane Kachach 21 Apr '26

21 Apr '26

Orch approved On Mon, Apr 20, 2026 at 8:45 AM Guillaume ABRIOUX via dev <dev(a)ceph.io> wrote: > Hello Yuri, > > ceph-volume approved. > > > Thanks! > > -- > > Guillaume Abrioux > > Software Engineer > *From: *Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> > *Date: *Monday, 13 April 2026 at 21:25 > *To: *dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> > *Subject: *[EXTERNAL] [ceph-users] squid 19.2.4 QE validation status > > Details of this release are summarized here: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > Unless otherwise stated above: > > Compagnie IBM France > Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex > RCS Nanterre 552 118 465 > Forme Sociale : S.A.S. > Capital Social : 664 614 175,50 € > SIRET : 552 118 465 03644 - Code NAF 6203Z >

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Anthony D'Atri 20 Apr '26

20 Apr '26

> The only reason setting mode=off is the observed ping-pong. Ah, I didn't catch that you were actively experiencing that. > The index pool is small (3 GiB stored) and PG merge/split is fast (~3 minutes). Right after adjusting pg_num (to 32/64) the health warning flips into the opposite, several times, back and forth. Do you set the mode to on for all pools? Does every pool use a CRUSH rule that specifies a device class? > The SSD-OSD's size is only 32 GiB, 4 per host and 3 hosts. The small size and count was chosen out of unawareness (pre-created LVs). Remind me please, how many OSDs total? The index pool does benefit from more vs larger OSDs, though if you're shaving that device class with other purposes it can be nuanced. If > Could this be something that confuses the autoscaler? For sure the autoscaler isn't great at very small clusters in some cases, but I wouldn't expected flapping. > >> What you're thinking of is the per-pool threshold. Historically this has defaulted to 3.0 : the autoscaler won't make a change until the calculated value varies by more than a factor of 3 from the current setting. I think of this as hysteresis, but there's probably a better term. There's been some fairly recent work to do this more surgically, which I _think_ removes the per-pool threshold, but in any event, I believe that recent autoscaler releases are fairly immune to this flapping. > > Thanks for clarification on BIAS and threshold. Yes I meant threshold, but looks like threshold is obsolete long ago (can't find it in the docs[1])? Curious, could be that we never documented it upstream. IBM FWIW documents this: https://www.ibm.com/docs/en/storage-ceph/9.9.0?topic=groups-configuring-pla… (IBM 9 is Tentacle, 8 is Squid) > At least for me the autoscaler seems not immune to flapping. What's your `ceph df`? Do you set any of the knobs to non-default values? I would think that your buckets.data pool would show a higher RATIO, but the bulk flag on it may be a factor. I've yet to fully wrap my head around how the bulk flag actually works, but it might be interesting to set it on the index pool too to see what happens with the autoscaler enabled. You might try # ceph osd pool set default.rgw.buckets.index pg_num_min 64 > > Thanks, Sascha. > > [1] https://docs.ceph.com/en/latest/rados/operations/pools/#setting-pool-values > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Sascha Lucas 20 Apr '26

20 Apr '26

Hi Anthony, On Sat, 18 Apr 2026, Anthony D'Atri via ceph-users wrote: > You might just periodically set them to warn, capture the info, and turn > it back off. But I do have to ask, is there a specific reason you > aren't using the autoscaler? It's much better than it used to be, and > there are really great improvements in the pipe. The only reason setting mode=off is the observed ping-pong. The index pool is small (3 GiB stored) and PG merge/split is fast (~3 minutes). Right after adjusting pg_num (to 32/64) the health warning flips into the opposite, several times, back and forth. The SSD-OSD's size is only 32 GiB, 4 per host and 3 hosts. The small size and count was chosen out of unawareness (pre-created LVs). Cloud this be something that confuses the autoscaler? > What you're thinking of is the per-pool threshold. Historically this > has defaulted to 3.0 : the autoscaler won't make a change until the > calculated value varies by more than a factor of 3 from the current > setting. I think of this as hysteresis, but there's probably a better > term. There's been some fairly recent work to do this more surgically, > which I _think_ removes the per-pool threshold, but in any event, I > believe that recent autoscaler releases are fairly immune to this > flapping. Thanks for clarification on BIAS and threshold. Yes I meant threshold, but looks like threshold is obsolete long ago (can't find it in the docs[1])? At least for me the autoscaler seems not immune to flapping. Thanks, Sascha. [1] https://docs.ceph.com/en/latest/rados/operations/pools/#setting-pool-values

1 0

Re: Tentacle, hybrid-OSD bug and making space for block.db
by Marek Szuba 20 Apr '26

20 Apr '26

On 2026-04-17 15:18, Anthony D'Atri wrote: > I suspect that the answer might be something ilke: > > * Use the tool to migrate to the slow device > * Compact it there (if the tool doesn't compact as a function of the migration) > * Migrate it to a smaller offload slice Judging from Igor's response, looks like this is indeed the way to go here. Oh well, fortunately we aren't short on space just yet so I think we'll save ourselves the hassle and just keep waiting for whichever Tentacle release ends up incorporating the fix. Thank you for your suggestions! -- MS

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Marek Szuba 20 Apr '26

20 Apr '26

Dear everyone, Update, on Friday we decided to apply blunt force to the problem (this still being a prototype cluster for the time being) and started rebooting nodes, one by one, again. After the second rebooted node had come back online the misplaced percentage jumped by about 10 percent but we finally began to see 'ceph -s' report recovery traffic - and when I checked this morning all PGs were back to normal. Just in case this is helps anyone figure out what did happen, so that we can avoid this / fix it faster in the future: * CRUSH: we do group our hosts into buckets corresponding to the server rooms they are located in (of which there are three). Apart from that, nothing non-default; * EC pools - yes, we have got some of these. See below. * pool and CRUSH-rule details: pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 545 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 50.00 pool 2 'cephfs.fs1.meta' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 548 lfor 0/0/153 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 3.02 pool 3 'cephfs.fs1.data' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 38714 lfor 0/0/201 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application cephfs read_balance_score 1.08 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 546 lfor 0/0/233 flags hashpspool stripe_width 0 application nfs read_balance_score 2.99 pool 5 'cephfs.fs1.repl1' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 38714 lfor 0/0/264 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application cephfs read_balance_score 1.13 pool 8 '.rgw.root' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20219 lfor 0/0/569 flags hashpspool stripe_width 0 application rgw read_balance_score 4.52 pool 9 'default.rgw.log' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20221 lfor 0/0/571 flags hashpspool stripe_width 0 application rgw read_balance_score 2.99 pool 10 'default.rgw.control' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20226 lfor 0/0/571 flags hashpspool stripe_width 0 application rgw read_balance_score 4.49 pool 11 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20231 lfor 0/0/573 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 2.99 pool 15 '.smb' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20233 lfor 0/0/6089 flags hashpspool stripe_width 0 application smb read_balance_score 4.50 pool 16 'cephfs.fs1.ec1' erasure profile isa_rsv_4p2_host_hdd size 6 min_size 5 crush_rule 4 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 38714 lfor 0/0/9461 flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 16384 fast_read 1 application cephfs Where the relevant CRUSH rules are: * 4 - erasure, on hard drives only, host-level failure domain * 6 - replicated, on NVMe drives only, room-level failure domain -- MS

1 0

Re: squid 19.2.4 QE validation status
by Guillaume ABRIOUX 20 Apr '26

20 Apr '26

Hello Yuri, ceph-volume approved. Thanks! -- Guillaume Abrioux Software Engineer From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Date: Monday, 13 April 2026 at 21:25 To: dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] squid 19.2.4 QE validation status Details of this release are summarized here: https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/squid-x: Laura, Patrick upgrade/reef-x: Laura, Patrick upgrade/telemetry-upgrade: Laura, Patrick upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Anthony D'Atri 18 Apr '26

18 Apr '26

> Hi Anthony, > > On Thu, 16 Apr 2026, Anthony D'Atri via ceph-users wrote: > >> I might suggest either enabling the autoscaler or setting the mode to off. > > I confirm, setting the autoscaler mode from warn to off removes the health warning. Until now I had always used mode=warn as a last resort for the absentminded admin (i.e. cluster expansion / OSD addition). That's one of the reasons the PG autoscaler was implemented, it is very, very easy to neglect the need to recalculate, and computers should serve US after all. > What I miss from previews ceph versions is, that autoscale-status shows the suggested PG count in the column "NEW PG_NUM", like shown in the docs[1]. You might just periodically set them to warn, capture the info, and turn it back off. But I do have to ask, is there a specific reason you aren't using the autoscaler? It's much better than it used to be, and there are really great improvements in the pipe. > And regarding the ping-pong with TOO_MANY / TOO_FEW. I understand that BIAS=4.0 for the index pool should prevent oscillating suggestions? The BIAS value means "give this pool more PGs than the amount of data would otherwise indicate". You'll see it on RGW index and some other metadata pools, which benefit from more sharding than does bulk data. What you're thinking of is the per-pool threshold. Historically this has defaulted to 3.0 : the autoscaler won't make a change until the calculated value varies by more than a factor of 3 from the current setting. I think of this as hysteresis, but there's probably a better term. There's been some fairly recent work to do this more surgically, which I _think_ removes the per-pool threshold, but in any event, I believe that recent autoscaler releases are fairly immune to this flapping. > > Thanks, Sascha. > > [1] https://docs.ceph.com/en/latest/rados/operations/placement-groups/#viewing-… > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

April Ceph Tech Talk
by Anthony Middleton 17 Apr '26

17 Apr '26

The next Ceph Tech Talk is on Wednesday, April 22, at 9 am PDT / 16:00 UTC, and is titled “MAAS as a Backend: Provisioning Infrastructure for Teuthology Suites,” a technical deep dive into how MaaS is now integrated with Teuthology to provision and manage bare-metal infrastructure on demand. Invite is on the Ceph Community Calendar: https://t.ly/CephTechTalks If you want to hear a topic at an upcoming session, email me at amiddleton(a)linuxfoundation.org. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Tentacle, hybrid-OSD bug and making space for block.db
by Igor Fedotov 17 Apr '26

17 Apr '26

Hi Marek, the only way to do that is to migrate data from DB volume(s) to main ones using ceph-volume(which will remove DB ones as well) and reallocate/reattach them back. Thanks, Igor On 4/17/2026 5:11 PM, Marek Szuba via ceph-users wrote: > Hello again, > > Having just tried to add additional hard drives to our production Ceph > cluster (i.e. not the one currently experiencing problems with misplaced > PGs), I have been forcibly reminded of the "orch puts OSD block.db and > data on the same device" orchestrator bug in Squid and Tentacle. > > With the relevant ticket having only reached "pending backport" stage 2 > days ago, I figured I would just follow the workaround of assembling the > new OSDs by hand rather than wait for whichever Tentacle patch release > ends up incorporating the fix. There is, however, a snag - turns out the > orchestrator has already allocated all the space on the DB SSDs to > existing OSDs! > > Is it possible to manually shrink block.db LVs so that I can fit another > one of them in? Having had a cursory look at documentation, seems one > can use ceph-bluestore-tool to _extend_ OSD storage - but have yet to > see any mention of the possibility of shrinking it. > > Thanks in advance. >

1 0

Re: squid 19.2.4 QE validation status
by Laura Flores 17 Apr '26

17 Apr '26

The RADOS suite is approved: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1924-httpstrackercephco… Upgrade suites are pending the inclusion of https://tracker.ceph.com/issues/76028, and will need to be rerun. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Laura Flores via dev <dev(a)ceph.io> Sent: Thursday, April 16, 2026 4:08 PM To: Yuri Weinstein <yweinste(a)redhat.com>; Travis Nielsen <tnielsen(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] RE: squid 19.2.4 QE validation status I reviewed the RADOS results. Will consult with Radek and update this thread by tomorrow. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM From: Travis Nielsen via dev <dev@ ceph. io> Sent: I reviewed the RADOS results. Will consult with Radek and update this thread by tomorrow. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Travis Nielsen via dev <dev(a)ceph.io> Sent: Thursday, April 16, 2026 12:56 PM To: Yuri Weinstein <yweinste(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: squid 19.2.4 QE validation status Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9: 01 AM Yuri Weinstein via dev <dev@ ceph. io> wrote: We got only rbd, krbd approved. Still seeking approvals On Mon, Apr 13, 2026 at Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9:01 AM Yuri Weinstein via dev <dev(a)ceph.io<mailto:dev@ceph.io>> wrote: We got only rbd, krbd approved. Still seeking approvals On Mon, Apr 13, 2026 at 12:24 PM Yuri Weinstein <yweinste(a)redhat.com<mailto:yweinste@redhat.com>> wrote: Details of this release are summarized here: https://tracker.ceph.com/issues/75029#note-5<https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/squid-x: Laura, Patrick upgrade/reef-x: Laura, Patrick upgrade/telemetry-upgrade: Laura, Patrick upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Anthony D'Atri 17 Apr '26

17 Apr '26

What does `ceph osd tree` look like? How many hosts? `ceph osd crush rule dump` `ceph osd pool ls detail` I'm wondering if your CRUSH rule for at least one pool requires more failure domains than the cluster provides. Also, in past releases there have been issues with the mclock scheduler on HDD and especially EC pools, improved by reverting to wpq. I thought Tentacle improved this but have not tested.

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Malte Stroem 17 Apr '26

17 Apr '26

Thanks! What about your pools? Show us: ceph osd pool ls detail Do you use erasure coded pools? What about your crush rules? Did you set up something special inside? Best, Malte On 4/17/26 15:44, Marek Szuba via ceph-users wrote: > Many thank to those who have responded! Will comment on everything in > this message, for brevity. > > > On 2026-04-14 17:00, Eugen Block via ceph-users wrote: > >> First thing coming to mind is: ceph mgr fail >> >> If you already did that something else is going on. > > I have indeed already tried it, several times in fact. Seems to have no > effect. > > > On 2026-04-14 17:41, tobias tempel wrote: > >> you write "all the daemons which should be up and running ARE up and >> running. In particular, all the expected OSDs appear to be >> operational". Does that mean, that not all OSDs are running? > No, all OSDs are expected to be running at present so what I meant by > the above was that all were - indeed, still are - up. Should have made > it more clear, apologies. > > > On 2026-04-14 18:05, Malte Stroem wrote: > > > add storage to the cluster. > > Is this expected to help if the cluster is nowhere near full? According > to 'ceph osd df' even the most heavily utilised OSDs are only about 20% > full, and most of them are half that or less. > > > Please show us: > > > > ceph -s > > As of a few minutes ago, this shows: > > cluster: > id: (REDACTED) > health: HEALTH_OK > > services: > mon: 5 daemons, quorum cdev01,cdev02,cdev03,cdev06,cdev08(age 3d) > [leader: cdev01] > mgr: cdev04.lmasuw(active, since 2d), standbys: cdev01.dgatgl, > cdev08.nxaiqd > mds: 1/1 daemons up, 2 standby > osd: 88 osds: 88 up (since 2d), 88 in (since 9w); 1448 remapped pgs > rgw: 6 daemons active (3 hosts, 1 zones) > > data: > volumes: 1/1 healthy > pools: 13 pools, 2641 pgs > objects: 98.57M objects, 27 TiB > usage: 105 TiB used, 846 TiB / 951 TiB avail > pgs: 76576200/314890284 objects misplaced (24.318%) > 1448 active+clean+remapped > 1189 active+clean > 2 active+clean+scrubbing+deep > 2 active+clean+scrubbing > > io: > client: 986 KiB/s rd, 0 B/s wr, 120 op/s rd, 0 op/s wr > > > Nb. The number of misplaced objects HAS in fact decreased slightly (by > about 0.3%) since two days ago so something IS happening, if extremely > slowly. >

1 0

Tentacle, hybrid-OSD bug and making space for block.db
by Marek Szuba 17 Apr '26

17 Apr '26

Hello again, Having just tried to add additional hard drives to our production Ceph cluster (i.e. not the one currently experiencing problems with misplaced PGs), I have been forcibly reminded of the "orch puts OSD block.db and data on the same device" orchestrator bug in Squid and Tentacle. With the relevant ticket having only reached "pending backport" stage 2 days ago, I figured I would just follow the workaround of assembling the new OSDs by hand rather than wait for whichever Tentacle patch release ends up incorporating the fix. There is, however, a snag - turns out the orchestrator has already allocated all the space on the DB SSDs to existing OSDs! Is it possible to manually shrink block.db LVs so that I can fit another one of them in? Having had a cursory look at documentation, seems one can use ceph-bluestore-tool to _extend_ OSD storage - but have yet to see any mention of the possibility of shrinking it. Thanks in advance. -- MS

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Marek Szuba 17 Apr '26

17 Apr '26

Many thank to those who have responded! Will comment on everything in this message, for brevity. On 2026-04-14 17:00, Eugen Block via ceph-users wrote: > First thing coming to mind is: ceph mgr fail > > If you already did that something else is going on. I have indeed already tried it, several times in fact. Seems to have no effect. On 2026-04-14 17:41, tobias tempel wrote: > you write "all the daemons which should be up and running ARE up and > running. In particular, all the expected OSDs appear to be > operational". Does that mean, that not all OSDs are running? No, all OSDs are expected to be running at present so what I meant by the above was that all were - indeed, still are - up. Should have made it more clear, apologies. On 2026-04-14 18:05, Malte Stroem wrote: > add storage to the cluster. Is this expected to help if the cluster is nowhere near full? According to 'ceph osd df' even the most heavily utilised OSDs are only about 20% full, and most of them are half that or less. > Please show us: > > ceph -s As of a few minutes ago, this shows: cluster: id: (REDACTED) health: HEALTH_OK services: mon: 5 daemons, quorum cdev01,cdev02,cdev03,cdev06,cdev08(age 3d) [leader: cdev01] mgr: cdev04.lmasuw(active, since 2d), standbys: cdev01.dgatgl, cdev08.nxaiqd mds: 1/1 daemons up, 2 standby osd: 88 osds: 88 up (since 2d), 88 in (since 9w); 1448 remapped pgs rgw: 6 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 2641 pgs objects: 98.57M objects, 27 TiB usage: 105 TiB used, 846 TiB / 951 TiB avail pgs: 76576200/314890284 objects misplaced (24.318%) 1448 active+clean+remapped 1189 active+clean 2 active+clean+scrubbing+deep 2 active+clean+scrubbing io: client: 986 KiB/s rd, 0 B/s wr, 120 op/s rd, 0 op/s wr Nb. The number of misplaced objects HAS in fact decreased slightly (by about 0.3%) since two days ago so something IS happening, if extremely slowly. -- MS

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Sascha Lucas 17 Apr '26

17 Apr '26

Hi Anthony, On Thu, 16 Apr 2026, Anthony D'Atri via ceph-users wrote: > I might suggest either enabling the autoscaler or setting the mode to off. I confirm, setting the autoscaler mode from warn to off removes the health warning. Until now I had always used mode=warn as a last resort for the absentminded admin (i.e. cluster expansion / OSD addition). What I miss from previews ceph versions is, that autoscale-status shows the suggested PG count in the column "NEW PG_NUM", like shown in the docs[1]. And regarding the ping-pong with TOO_MANY / TOO_FEW. I understand that BIAS=4.0 for the index pool should prevent oscillating suggestions? Thanks, Sascha. [1] https://docs.ceph.com/en/latest/rados/operations/placement-groups/#viewing-…

1 0

Re: squid 19.2.4 QE validation status
by Laura Flores 16 Apr '26

16 Apr '26

I reviewed the RADOS results. Will consult with Radek and update this thread by tomorrow. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Travis Nielsen via dev <dev(a)ceph.io> Sent: Thursday, April 16, 2026 12:56 PM To: Yuri Weinstein <yweinste(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: squid 19.2.4 QE validation status Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9: 01 AM Yuri Weinstein via dev <dev@ ceph. io> wrote: We got only rbd, krbd approved. Still seeking approvals On Mon, Apr 13, 2026 at Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9:01 AM Yuri Weinstein via dev <dev(a)ceph.io<mailto:dev@ceph.io>> wrote: We got only rbd, krbd approved. Still seeking approvals On Mon, Apr 13, 2026 at 12:24 PM Yuri Weinstein <yweinste(a)redhat.com<mailto:yweinste@redhat.com>> wrote: Details of this release are summarized here: https://tracker.ceph.com/issues/75029#note-5<https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/squid-x: Laura, Patrick upgrade/reef-x: Laura, Patrick upgrade/telemetry-upgrade: Laura, Patrick upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 0

Re: squid 19.2.4 QE validation status
by Travis Nielsen 16 Apr '26

16 Apr '26

Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9:01 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > We got only rbd, krbd approved. > > Still seeking approvals > > On Mon, Apr 13, 2026 at 12:24 PM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/75029#note-5 >> >> Release Notes - TBD >> LRC upgrade - N/A >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Redouane & Guillaume >> rbd: Ilya >> krbd: Ilya >> >> upgrade/squid-x: Laura, Patrick >> upgrade/reef-x: Laura, Patrick >> upgrade/telemetry-upgrade: Laura, Patrick >> upgrade/squid-p2p: Laura, Patrick >> >> upgrade-clients:client-upgrade-reef-squid-squid: PASSED >> upgrade-clients:client-upgrade-quincy-squid-squid: PASSED >> >> perf-basic: PASSED >> crimson-rados: N/A >> powercycle: PASSED >> >> TIA >> >

1 0

Re: squid 19.2.4 QE validation status
by Adam Emerson 16 Apr '26

16 Apr '26

On 13/04/2026, Yuri Weinstein via dev wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75029#note-5 > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson RGW Approved. Thank you!

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Anthony D'Atri 16 Apr '26

16 Apr '26

Hmmm, I thought this was fixed by Squid. I've seen Pool wombats has 32 placement groups, should have 32 with Pacific. I might suggest either enabling the autoscaler or setting the mode to off. When enabling, setting mon_max_pg_per_osd 500 mon_target_pg_per_osd 300 will likely give you reasonable results. > On Apr 16, 2026, at 10:07 AM, Sascha Lucas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > having a small ceph-19.2.3 cluster for RGW workload. index-pool is on separate SSD class. PG autoscaler is enabled in warning mode only. > > Cluster health is: > > # ceph health detail > HEALTH_WARN 1 pools have too many placement groups > [WRN] POOL_TOO_MANY_PGS: 1 pools have too many placement groups > Pool default.rgw.buckets.index has 64 placement groups, should have 32 > > When merging PGs down to 32, the opposite is shown: > > # ceph health detail > HEALTH_WARN 1 pools have too few placement groups > [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups > Pool default.rgw.buckets.index has 32 placement groups, should have 64 > > It seems not the autoscaler who raises health warning here (NEW PG_NUM is empty): > > # ceph osd pool autoscale-status > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK > .mgr 23812k 3.0 384.0G 0.0002 1.0 1 warn False > .rgw.root 20480 3.0 384.0G 0.0000 1.0 32 warn False > default.rgw.log 23905k 3.0 384.0G 0.0002 1.0 32 warn False > default.rgw.control 0 3.0 384.0G 0.0000 1.0 32 warn False > default.rgw.meta 575.2k 3.0 384.0G 0.0000 4.0 32 warn False > default.rgw.buckets.index 3017M 3.0 384.0G 0.0230 4.0 64 warn False > default.rgw.buckets.non-ec 0 3.0 384.0G 0.0000 1.0 32 warn False > default.rgw.buckets.data 10678G 3.0 374.2T 0.0836 1.0 1024 warn True > > Anyone knowing from where this health warning is coming and how to stop the ping-pong? > > Thanks, Sascha. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Sascha Lucas 16 Apr '26

16 Apr '26

Hi, having a small ceph-19.2.3 cluster for RGW workload. index-pool is on separate SSD class. PG autoscaler is enabled in warning mode only. Cluster health is: # ceph health detail HEALTH_WARN 1 pools have too many placement groups [WRN] POOL_TOO_MANY_PGS: 1 pools have too many placement groups Pool default.rgw.buckets.index has 64 placement groups, should have 32 When merging PGs down to 32, the opposite is shown: # ceph health detail HEALTH_WARN 1 pools have too few placement groups [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool default.rgw.buckets.index has 32 placement groups, should have 64 It seems not the autoscaler who raises health warning here (NEW PG_NUM is empty): # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 23812k 3.0 384.0G 0.0002 1.0 1 warn False .rgw.root 20480 3.0 384.0G 0.0000 1.0 32 warn False default.rgw.log 23905k 3.0 384.0G 0.0002 1.0 32 warn False default.rgw.control 0 3.0 384.0G 0.0000 1.0 32 warn False default.rgw.meta 575.2k 3.0 384.0G 0.0000 4.0 32 warn False default.rgw.buckets.index 3017M 3.0 384.0G 0.0230 4.0 64 warn False default.rgw.buckets.non-ec 0 3.0 384.0G 0.0000 1.0 32 warn False default.rgw.buckets.data 10678G 3.0 374.2T 0.0836 1.0 1024 warn True Anyone knowing from where this health warning is coming and how to stop the ping-pong? Thanks, Sascha.

1 0

Re: Changing EC profile
by Daniel Parkes 16 Apr '26

16 Apr '26

Hi, Yes, we have it in the docs, https://docs.ceph.com/en/latest/radosgw/placement/#placement-targets : [image: image.png] On Wed, Apr 15, 2026 at 2:06 PM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > It didn't work in my tests without a realm, so I don't believe grok > :-D But I'd love to be proven wrong. > > Zitat von Rok Jaklič <rjaklic(a)gmail.com>: > > > There is: > > > > [root@ctplmon1 ~]# radosgw-admin realm list > > { > > "default_info": "", > > "realms": [] > > } > > [root@ctplmon1 ~]# radosgw-admin realm get-default > > No default realm is set > > [root@ctplmon1 ~]# radosgw-admin period list > > { > > "periods": [] > > } > > [root@ctplmon1 ~]# radosgw-admin zonegroup list > > { > > "default_info": "b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578", > > "zonegroups": [ > > "default" > > ] > > } > > [root@ctplmon1 ~]# radosgw-admin zone list > > { > > "default_info": "c2c70444-7a41-4acd-a0d0-9f87d324ec72", > > "zones": [ > > "default" > > ] > > } > > > > Since there is no realm set, I can skip radosgw-admin period update > > --commit entirely if grok is to be believed? > > > > On Wed, Apr 15, 2026 at 11:54 AM Eugen Block via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> From my understanding the new placement will not be available if the > >> period can't be updated. Your zonegroup/zone configuration seems off > >> compared to the test cluster. Is there a default zonegroup/zone? > >> > >> Zitat von Rok Jaklič via ceph-users <ceph-users(a)ceph.io>: > >> > >> > So I've created a little plan on how to create a new profile and > assign > >> new > >> > S3 users to EC 8+3 pool. Idea is that all new s3 users are using a new > >> > pool, and old ones pool with EC 3+2 and data is intact. > >> > > >> > --- > >> > ceph osd erasure-code-profile set ec-83-profile k=8 m=3 > >> > crush-failure-domain=host > >> > ceph osd pool create rgw.buckets.ec-83.data 256 erasure ec-83-profile > >> > ceph osd pool application enable rgw.buckets.ec-83.data rgw > >> > > >> > radosgw-admin zonegroup placement add --placement-id=ec-83-placement > >> > > >> > radosgw-admin zone placement add \ > >> > --placement-id ec-83-placement \ > >> > --data-pool rgw.buckets.ec-83.data \ > >> > --index-pool default.rgw.buckets.index \ > >> > --data-extra-pool default.rgw.buckets.non-ec \ > >> > --placement-index-type 0 > >> > > >> > radosgw-admin period update --commit > >> > > >> > # restart rgw > >> > systemctl restart ceph-radosgw.target > >> > # verify the new placement > >> > radosgw-admin zone get | grep -A 20 "ec-83-placement" > >> > > >> > # for new s3 users, set the default placement > >> > radosgw-admin user create --uid=<user> --tenant=<tenant> > >> > --placement-id=ec-83-placement > >> > --- > >> > > >> > It seems to work on "test cluster", however I get an error on: > >> > > >> > [root@a8-mon1 ~]# radosgw-admin period update --commit > >> > cannot commit period: period does not have a master zone of a master > >> > zonegroup > >> > failed to commit period: (22) Invalid argument > >> > > >> > Do I actually need to run this command or is "rgw restart" enough? > >> > > >> > Any comments appriciated. > >> > > >> > Rok > >> > > >> > > >> > On Fri, Apr 3, 2026 at 12:42 PM Rok Jaklič via ceph-users < > >> > ceph-users(a)ceph.io> wrote: > >> > > >> >> Hi, > >> >> > >> >> we expanded the cluster to 12 nodes with 28 osds, but we are still > >> running > >> >> on EC 3+2, since we started cluster with 6 nodes. We would like to > >> change > >> >> it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too > >> much > >> >> CPU/network capacities? Should I use isa/isa-l instead of jerasure? > >> >> > >> >> Can anyone share experience with 8+3? > >> >> > >> >> Kind regards, > >> >> Rok > >> >> _______________________________________________ > >> >> ceph-users mailing list -- ceph-users(a)ceph.io > >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> >> > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users(a)ceph.io > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

3 5

Re: Upgrade path from Pacific to Quincy (and to Reef)
by Eugen Block 15 Apr '26

15 Apr '26

Thanks for sharing, I was not aware of that. I must admit, it’s been a while since we upgraded our own or our customer’s clusters to Reef, we usually try to stay near supported releases. Anyway, I don’t have a clear recommendation which Quincy version to upgrade to. We skipped it as did almost all of our customers (several of them came from SUSE Enterprise Storage which ended with version Pacific, so they upgraded to Reef). One other customer was on Quincy (came from Octopus) until recently, they didn’t have any issues with Q, although I don’t recall the exact versions of Q they had in use. I can try to look it up tomorrow if that helps. Zitat von J-P Methot via ceph-users <ceph-users(a)ceph.io>: > From the "Known issues" section of the Reef 18.2.8 release notes > (https://ceph.io/en/news/blog/2026/v18-2-8-reef-released/) : > > "During QA for v18.2.8, it was found that there was a bug for > upgrades from Pacific to Reef. Pacific OSDs (and other Ceph daemons) > were still using a deprecated connection feature bit that was > adopted to indicate a Reef OSD. This can cause a > OSD_UPGRADE_FINISHED warning before all OSDs are actually upgraded > to Reef. There are no known issues associated with Pacific and Reef > OSDs interoperating where Pacific OSDs are "advertising" Reef > compatibility; however, out of an abundance of caution, we no longer > recommend upgrading from Pacific to Reef directly." > > On 4/15/26 16:02, Eugen Block via ceph-users wrote: >> Why is it not recommended to upgrade from Pacific to Reef? We’ve >> done that multiple times, I’m quite surprised by that statement. >> >> Zitat von J-P Methot via ceph-users <ceph-users(a)ceph.io>: >> >>> Hi, >>> >>> We have a fairly old Ceph Pacific cephadm cluster that we would >>> like to upgrade to an active release. Since upgrading from Pacific >>> to Reef is not recommended, we will need to upgrade to Quincy >>> first. However, there seems to be bugs with Pacific to Quincy >>> upgrades for several Quincy releases. Would it be best to upgrade >>> Quincy 17.2.8, which has a bluestore regression bug, or to 17.2.7 >>> which seems to also have caused issues to other users? I >>> understand that 17.2.9 is not obtainable, as there was no >>> container image built for it. We like to run our ceph versions for >>> a few weeks inbetween upgrades, so the cluster needs to remain >>> stable. >>> >>> >>> -- >>> Mes salutations distinguées | Best regards >>> Jean-Philippe MÉTHOT (Il, Lui | He/Him) >>> Administrateur Infra sénior | Senior Infra administrator >>> PlanetHoster >>> >>> AVIS: Ce courriel est confidentiel et peut être l'objet de secret >>> professionnel. Tout usage ou diffusion de celui-ci par une >>> personne autre que son destinataire est illégal. Si vous recevez >>> ce courriel par erreur, veuillez nous en aviser sans délai et le >>> supprimer de votre système. >>> >>> This message is confidential. It may also be privileged or >>> otherwise protected by work product immunity or other legal rules. >>> If you have received it by mistake, please let us know by e-mail >>> reply and delete it from your system. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Mes salutations distinguées | Best regards > Jean-Philippe MÉTHOT (Il, Lui | He/Him) > Administrateur Infra sénior | Senior Infra administrator > PlanetHoster > > AVIS: Ce courriel est confidentiel et peut être l'objet de secret > professionnel. Tout usage ou diffusion de celui-ci par une personne > autre que son destinataire est illégal. Si vous recevez ce courriel > par erreur, veuillez nous en aviser sans délai et le supprimer de > votre système. > > This message is confidential. It may also be privileged or otherwise > protected by work product immunity or other legal rules. If you have > received it by mistake, please let us know by e-mail reply and > delete it from your system. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Upgrade path from Pacific to Quincy (and to Reef)
by J-P Methot 15 Apr '26

15 Apr '26

From the "Known issues" section of the Reef 18.2.8 release notes (https://ceph.io/en/news/blog/2026/v18-2-8-reef-released/) : "During QA for v18.2.8, it was found that there was a bug for upgrades from Pacific to Reef. Pacific OSDs (and other Ceph daemons) were still using a deprecated connection feature bit that was adopted to indicate a Reef OSD. This can cause a OSD_UPGRADE_FINISHED warning before all OSDs are actually upgraded to Reef. There are no known issues associated with Pacific and Reef OSDs interoperating where Pacific OSDs are "advertising" Reef compatibility; however, out of an abundance of caution, we no longer recommend upgrading from Pacific to Reef directly." On 4/15/26 16:02, Eugen Block via ceph-users wrote: > Why is it not recommended to upgrade from Pacific to Reef? We’ve done > that multiple times, I’m quite surprised by that statement. > > Zitat von J-P Methot via ceph-users <ceph-users(a)ceph.io>: > >> Hi, >> >> We have a fairly old Ceph Pacific cephadm cluster that we would like >> to upgrade to an active release. Since upgrading from Pacific to Reef >> is not recommended, we will need to upgrade to Quincy first. However, >> there seems to be bugs with Pacific to Quincy upgrades for several >> Quincy releases. Would it be best to upgrade Quincy 17.2.8, which has >> a bluestore regression bug, or to 17.2.7 which seems to also have >> caused issues to other users? I understand that 17.2.9 is not >> obtainable, as there was no container image built for it. We like to >> run our ceph versions for a few weeks inbetween upgrades, so the >> cluster needs to remain stable. >> >> >> -- >> Mes salutations distinguées | Best regards >> Jean-Philippe MÉTHOT (Il, Lui | He/Him) >> Administrateur Infra sénior | Senior Infra administrator >> PlanetHoster >> >> AVIS: Ce courriel est confidentiel et peut être l'objet de secret >> professionnel. Tout usage ou diffusion de celui-ci par une personne >> autre que son destinataire est illégal. Si vous recevez ce courriel >> par erreur, veuillez nous en aviser sans délai et le supprimer de >> votre système. >> >> This message is confidential. It may also be privileged or otherwise >> protected by work product immunity or other legal rules. If you have >> received it by mistake, please let us know by e-mail reply and delete >> it from your system. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Mes salutations distinguées | Best regards Jean-Philippe MÉTHOT (Il, Lui | He/Him) Administrateur Infra sénior | Senior Infra administrator PlanetHoster AVIS: Ce courriel est confidentiel et peut être l'objet de secret professionnel. Tout usage ou diffusion de celui-ci par une personne autre que son destinataire est illégal. Si vous recevez ce courriel par erreur, veuillez nous en aviser sans délai et le supprimer de votre système. This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system.

1 0

Re: Upgrade path from Pacific to Quincy (and to Reef)
by Eugen Block 15 Apr '26

15 Apr '26

Why is it not recommended to upgrade from Pacific to Reef? We’ve done that multiple times, I’m quite surprised by that statement. Zitat von J-P Methot via ceph-users <ceph-users(a)ceph.io>: > Hi, > > We have a fairly old Ceph Pacific cephadm cluster that we would like > to upgrade to an active release. Since upgrading from Pacific to > Reef is not recommended, we will need to upgrade to Quincy first. > However, there seems to be bugs with Pacific to Quincy upgrades for > several Quincy releases. Would it be best to upgrade Quincy 17.2.8, > which has a bluestore regression bug, or to 17.2.7 which seems to > also have caused issues to other users? I understand that 17.2.9 is > not obtainable, as there was no container image built for it. We > like to run our ceph versions for a few weeks inbetween upgrades, so > the cluster needs to remain stable. > > > -- > Mes salutations distinguées | Best regards > Jean-Philippe MÉTHOT (Il, Lui | He/Him) > Administrateur Infra sénior | Senior Infra administrator > PlanetHoster > > AVIS: Ce courriel est confidentiel et peut être l'objet de secret > professionnel. Tout usage ou diffusion de celui-ci par une personne > autre que son destinataire est illégal. Si vous recevez ce courriel > par erreur, veuillez nous en aviser sans délai et le supprimer de > votre système. > > This message is confidential. It may also be privileged or otherwise > protected by work product immunity or other legal rules. If you have > received it by mistake, please let us know by e-mail reply and > delete it from your system. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Upgrade path from Pacific to Quincy (and to Reef)
by J-P Methot 15 Apr '26

15 Apr '26

Hi, We have a fairly old Ceph Pacific cephadm cluster that we would like to upgrade to an active release. Since upgrading from Pacific to Reef is not recommended, we will need to upgrade to Quincy first. However, there seems to be bugs with Pacific to Quincy upgrades for several Quincy releases. Would it be best to upgrade Quincy 17.2.8, which has a bluestore regression bug, or to 17.2.7 which seems to also have caused issues to other users? I understand that 17.2.9 is not obtainable, as there was no container image built for it. We like to run our ceph versions for a few weeks inbetween upgrades, so the cluster needs to remain stable. -- Mes salutations distinguées | Best regards Jean-Philippe MÉTHOT (Il, Lui | He/Him) Administrateur Infra sénior | Senior Infra administrator PlanetHoster AVIS: Ce courriel est confidentiel et peut être l'objet de secret professionnel. Tout usage ou diffusion de celui-ci par une personne autre que son destinataire est illégal. Si vous recevez ce courriel par erreur, veuillez nous en aviser sans délai et le supprimer de votre système. This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system.

1 0

Announcing go-ceph v0.39.0
by John Mulligan 15 Apr '26

15 Apr '26

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.39.0 Enhancements include additions to the smb admin package, and a new command builder package. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project.

1 0

Re: Upgrade to tentacle failed! (Now recovered, upgrade paused)
by Johan 15 Apr '26

15 Apr '26

I realized I can't manually upgrade specific daemon-types when upgrade already in progress (paused in this case). I therefore decided to resume the upgrade from the dashboard and all has gone well, the cluster is now fully upgraded. /Johan Den 2026-04-15 kl. 13:47, skrev Johan via ceph-users: > Im running two private ceph clusters. Both are cephadm/docker on debian. > > The first cluster was upgraded without problem to latest tentacle > (19.2.3 -> 20.2.1) using > ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.1 > > Today I tried to upgrade the second cluster. During the upgrade, when a > couple of managers had been upgraded, the upgrade process stopped with > the error message > Orchestrator is unavailable: Module not found > and the suggestion I should enable it (ceph mgr module enable > orchestrator). From here Im not exactly certain about the errors/ > messages I received, but I could not enable the module (it already was) > and several pages on the dashboard didn't give any information as it > apparently dependent on the orchestrator. > > As I neither were near the servers nor had the time to deal with the > failure I chose to shutdown the servers. I have now restarted the > servers and after a while the cluster got up again. > > The dashboard (Administration/Upgrade) now shows version 20.2.1 and > "Upgrade is paused". Two managers (incl the active) are upgraded and the > third is old version(19.2.3). Everything else seems old version > (logical, mgr is upgraded first and the error occured during mgr upgrade). > > When I got the cluster running again > ceph orch uprade status > showed > > { > "in_progress": true, > "target_image": "quay.io/ceph/ > ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337", > "services_complete": [], > "which": "Upgrading all daemon types on all hosts", > "progress": "2/52 daemons upgraded", > "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an > unexpected exception", > "is_paused": true > } > > > Any suggestions regarding the continued upgrade? Resume the upgrade from > the dashboard? First manually upgrade the last manager and then resume > the normal upgrade process? or... ? > > Anyone seen anything like this before or understand the reason for the > failure? > > /Johan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Bluestore deferred writes (Squid)
by Igor Fedotov 15 Apr '26

15 Apr '26

Hi Magnus, Here is the brief overview for the relevant perf counters: - 'issued deferred writes' denotes amount of user write ops which have got deferred processing and have been persisted at Key-Value DB (The first stage of deferred write procedure). - 'submitted deferred writes' denotes amount of writes which have been submitted to main device to reach their final disk location. (the second stage of deferred write procedure). Additionally there is the last stage which is staged deferred writes cleanup from KV. But there is no relevant counter to track this state though at the moment. Frankly speaking I recall no modifications to bluestore_prefer_deferred_size_hdd happened to Squid. The only not-that-recent PR which might look relevant is https://github.com/ceph/ceph/pull/48490 but it's been backported back to Pacific and Quincy. May be you had had another upgrade(s) prior to R->S one recently which triggered the regression? It would be great to get more information on the issue including perf counter dumps and more details on that performance degradation and write patters which are affected. Thanks, Igor On 4/14/2026 5:00 PM, Magnus Grönlund via ceph-users wrote: > Hi All, > > We are running a 5 year old 15 node Ceph cluster with spinners for data and > NVMe for DB/Wal. > After the recent upgrade to Squid (from Reef) some users reported issues > with fluctuating (often bad) disk performance especially affecting k8s etcd. > They point to the change to bluestore_prefer_deferred_size_hdd claiming > that it's detrimental to the "synchronous write patter" of escd. > They also point to submitted_deferred_writes and issued_deferred_writes > claiming that there is a massive and growing backlog of deferred writes. > > I'm not 100% correct that this is the issue but I'm having a hard time > finding any relevant information (bad google-fu?) especially about what > submitted and issued deferred writes actually represents and why the > difference between issued - submitted keeps growing. > Anyone got some insights to share? > > Thanks in advance > Magnus > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Changing EC profile
by Rok Jaklič 15 Apr '26

15 Apr '26

There is: [root@ctplmon1 ~]# radosgw-admin realm list { "default_info": "", "realms": [] } [root@ctplmon1 ~]# radosgw-admin realm get-default No default realm is set [root@ctplmon1 ~]# radosgw-admin period list { "periods": [] } [root@ctplmon1 ~]# radosgw-admin zonegroup list { "default_info": "b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578", "zonegroups": [ "default" ] } [root@ctplmon1 ~]# radosgw-admin zone list { "default_info": "c2c70444-7a41-4acd-a0d0-9f87d324ec72", "zones": [ "default" ] } Since there is no realm set, I can skip radosgw-admin period update --commit entirely if grok is to be believed? On Wed, Apr 15, 2026 at 11:54 AM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > From my understanding the new placement will not be available if the > period can't be updated. Your zonegroup/zone configuration seems off > compared to the test cluster. Is there a default zonegroup/zone? > > Zitat von Rok Jaklič via ceph-users <ceph-users(a)ceph.io>: > > > So I've created a little plan on how to create a new profile and assign > new > > S3 users to EC 8+3 pool. Idea is that all new s3 users are using a new > > pool, and old ones pool with EC 3+2 and data is intact. > > > > --- > > ceph osd erasure-code-profile set ec-83-profile k=8 m=3 > > crush-failure-domain=host > > ceph osd pool create rgw.buckets.ec-83.data 256 erasure ec-83-profile > > ceph osd pool application enable rgw.buckets.ec-83.data rgw > > > > radosgw-admin zonegroup placement add --placement-id=ec-83-placement > > > > radosgw-admin zone placement add \ > > --placement-id ec-83-placement \ > > --data-pool rgw.buckets.ec-83.data \ > > --index-pool default.rgw.buckets.index \ > > --data-extra-pool default.rgw.buckets.non-ec \ > > --placement-index-type 0 > > > > radosgw-admin period update --commit > > > > # restart rgw > > systemctl restart ceph-radosgw.target > > # verify the new placement > > radosgw-admin zone get | grep -A 20 "ec-83-placement" > > > > # for new s3 users, set the default placement > > radosgw-admin user create --uid=<user> --tenant=<tenant> > > --placement-id=ec-83-placement > > --- > > > > It seems to work on "test cluster", however I get an error on: > > > > [root@a8-mon1 ~]# radosgw-admin period update --commit > > cannot commit period: period does not have a master zone of a master > > zonegroup > > failed to commit period: (22) Invalid argument > > > > Do I actually need to run this command or is "rgw restart" enough? > > > > Any comments appriciated. > > > > Rok > > > > > > On Fri, Apr 3, 2026 at 12:42 PM Rok Jaklič via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Hi, > >> > >> we expanded the cluster to 12 nodes with 28 osds, but we are still > running > >> on EC 3+2, since we started cluster with 6 nodes. We would like to > change > >> it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too > much > >> CPU/network capacities? Should I use isa/isa-l instead of jerasure? > >> > >> Can anyone share experience with 8+3? > >> > >> Kind regards, > >> Rok > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Upgrade to tentacle failed! (Now recovered, upgrade paused)
by Johan 15 Apr '26

15 Apr '26

Im running two private ceph clusters. Both are cephadm/docker on debian. The first cluster was upgraded without problem to latest tentacle (19.2.3 -> 20.2.1) using ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.1 Today I tried to upgrade the second cluster. During the upgrade, when a couple of managers had been upgraded, the upgrade process stopped with the error message Orchestrator is unavailable: Module not found and the suggestion I should enable it (ceph mgr module enable orchestrator). From here Im not exactly certain about the errors/messages I received, but I could not enable the module (it already was) and several pages on the dashboard didn't give any information as it apparently dependent on the orchestrator. As I neither were near the servers nor had the time to deal with the failure I chose to shutdown the servers. I have now restarted the servers and after a while the cluster got up again. The dashboard (Administration/Upgrade) now shows version 20.2.1 and "Upgrade is paused". Two managers (incl the active) are upgraded and the third is old version(19.2.3). Everything else seems old version (logical, mgr is upgraded first and the error occured during mgr upgrade). When I got the cluster running again ceph orch uprade status showed { "in_progress": true, "target_image": "quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337", "services_complete": [], "which": "Upgrading all daemon types on all hosts", "progress": "2/52 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } Any suggestions regarding the continued upgrade? Resume the upgrade from the dashboard? First manually upgrade the last manager and then resume the normal upgrade process? or... ? Anyone seen anything like this before or understand the reason for the failure? /Johan

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 15 Apr '26

15 Apr '26

On 15/04/26 12:08, Eugen Block via ceph-users wrote: > Yeah, looks like it. I just compared the output with a fresh Tentacle > install, I only enabled log-to-file during bootstrap, so log_to_file is > true in my cluster as well. There's no leveldb_log_to_ceph_log option > anymore since the default has been rocksdb for a while now. I put 'log_to_journald' to true and journalctl started to log. A lot (with debug=20) :-) Now I'm ready to debug the joining problem. I will keep you posted. Thanks Iztok

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 15 Apr '26

15 Apr '26

Yeah, looks like it. I just compared the output with a fresh Tentacle install, I only enabled log-to-file during bootstrap, so log_to_file is true in my cluster as well. There's no leveldb_log_to_ceph_log option anymore since the default has been rocksdb for a while now. Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi! > > On 15/04/26 11:11, Eugen Block via ceph-users wrote: >> Hi, >> >> Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: >> > > [cut] > >>>> So your cluster currently has quorum with 2 out of 3 MONs on >>>> Quincy already, correct? Restarting the mon on node3 shouldn't be >>>> dangerous, I think. You'd still have quorum with node4 and node5, >>>> even if node3 wouldn't be able to join. If it doesn't join as >>>> Octopus, you could still upgrade it and retry with Quincy. >>> >>> Yes. I was just reluctant to do so because in the case it doesn't >>> join our room to manoeuvre will be reduced. Plus the fact that >>> node3 is the only daemon which is still logging something... >> >> oh I didn't realize there was only one left logging something. I >> thought only the newly to be joined MON didn't log. In that case I >> would also try to find the root cause for that. > > It could easily be, because this is an old cluster which was > upgraded multiple times, that "default" log settings changed in the > meantime. If I query the daemon for 'mon_status' and search for > log_to I see the following: > > # ceph daemon mon.node4 config show | grep log_to > "clog_to_graylog": "false", > "clog_to_graylog_host": "127.0.0.1", > "clog_to_graylog_port": "12201", > "clog_to_monitors": "default=true", > "clog_to_syslog": "false", > "clog_to_syslog_facility": "default=daemon audit=local0", > "clog_to_syslog_level": "info", > "leveldb_log_to_ceph_log": "true", > "log_to_file": "true", > "log_to_graylog": "false", > "log_to_journald": "false", > "log_to_stderr": "false", > "log_to_syslog": "false", > "mon_cluster_log_to_file": "true", > "mon_cluster_log_to_graylog": "false", > "mon_cluster_log_to_graylog_host": "127.0.0.1", > "mon_cluster_log_to_graylog_port": "12201", > "mon_cluster_log_to_journald": "false", > "mon_cluster_log_to_stderr": "false", > "mon_cluster_log_to_syslog": "default=false", > "mon_cluster_log_to_syslog_facility": "daemon", > "mon_cluster_log_to_syslog_level": "info", > "rocksdb_log_to_ceph_log": "true", > > > If I understand correctly I don't see the log in journald because > '"log_to_journald": "false"' (in fact, on the Octopus node there is > no "log_to_journald" option). Is this the correct default behaviour? > > > >>>> What I'm wondering about, why did you remove the MONs you wanted >>>> to upgrade from Ceph? Is that recommended somewhere? Maybe you >>>> need to run 'ceph mon add <name> <addr>' as well to let node1 or >>>> node2 join the monmap again? I'm not sure, it has been a long >>>> time doing this kind of stuff. ;-) But I don't think I ever >>>> removed a MON before upgrading it, not even when reinstalling the >>>> OS. >>> >>> No, we didn't follow any recommendations, it seemed reasonable to >>> remove a monitor before reinstalling the OS, a fresh start looked >>> better than a backup/restore of '/var/lib/ceph/'. Maybe we were >>> wrong. For the next upgrades (node3-5) we could think not to do >>> that and see what it happens. >> >> I don't exactly remember, tbh, but I expect that it would suffice >> to have the /var/lib/ceph/mon/ceph-nodeX directory with the correct >> permissions present in order for it to re-sync with the other MONs. >> But please double-check that, I'm really not sure. If I have the >> time, I will try to test that in a lab environment. > > Thank you! I will check it before doing anything strange. > > BTW, I remember that another reason for the MONs removal was that we > needed to change the hostname (not the IP address) and we taught > that a fresh install would be better. > > [cut] > >>>>>> - Inspecting debug logs on the syncing MON could help finding >>>>>> the root cause. >>>>> >>>>> With "syncing MON" you mean the MON which is just added to the >>>>> cluster, right? Collecting this kind of logs is on the to-do >>>>> list, but blocked because the "log problem" mentioned bellow. >>>> >>>> Yes, that's what I meant. The missing logs are a problem, of >>>> course. It does sound familiar, but I don't have a good idea >>>> right now except for looking at directory and file permissions, >>>> apparmor/selinux things, dmesg, stuff like that. >>> >>> >>> Yep. My biggest fear right now is that lack of logging from Quincy >>> monitor daemons could mean that they don't work properly and if we >>> restart the Octopus one (or it doesn't want to rejoin the quorum) >>> we will have a not functioning cluster. We need to investigate >>> further into it. >> >> I totally understand your hesitation, before I'd do that I would >> also try to rule out anything I can. But at the same time it would >> be surprising, given your cluster shows a healthy quorum right now. >> But yeah, investigate further before moving forward is the safer >> choice here. >> > > Probably you are right. I'll be cautiuos. > > Thank you for your help! > Iztok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Changing EC profile
by Eugen Block 15 Apr '26

15 Apr '26

From my understanding the new placement will not be available if the period can't be updated. Your zonegroup/zone configuration seems off compared to the test cluster. Is there a default zonegroup/zone? Zitat von Rok Jaklič via ceph-users <ceph-users(a)ceph.io>: > So I've created a little plan on how to create a new profile and assign new > S3 users to EC 8+3 pool. Idea is that all new s3 users are using a new > pool, and old ones pool with EC 3+2 and data is intact. > > --- > ceph osd erasure-code-profile set ec-83-profile k=8 m=3 > crush-failure-domain=host > ceph osd pool create rgw.buckets.ec-83.data 256 erasure ec-83-profile > ceph osd pool application enable rgw.buckets.ec-83.data rgw > > radosgw-admin zonegroup placement add --placement-id=ec-83-placement > > radosgw-admin zone placement add \ > --placement-id ec-83-placement \ > --data-pool rgw.buckets.ec-83.data \ > --index-pool default.rgw.buckets.index \ > --data-extra-pool default.rgw.buckets.non-ec \ > --placement-index-type 0 > > radosgw-admin period update --commit > > # restart rgw > systemctl restart ceph-radosgw.target > # verify the new placement > radosgw-admin zone get | grep -A 20 "ec-83-placement" > > # for new s3 users, set the default placement > radosgw-admin user create --uid=<user> --tenant=<tenant> > --placement-id=ec-83-placement > --- > > It seems to work on "test cluster", however I get an error on: > > [root@a8-mon1 ~]# radosgw-admin period update --commit > cannot commit period: period does not have a master zone of a master > zonegroup > failed to commit period: (22) Invalid argument > > Do I actually need to run this command or is "rgw restart" enough? > > Any comments appriciated. > > Rok > > > On Fri, Apr 3, 2026 at 12:42 PM Rok Jaklič via ceph-users < > ceph-users(a)ceph.io> wrote: > >> Hi, >> >> we expanded the cluster to 12 nodes with 28 osds, but we are still running >> on EC 3+2, since we started cluster with 6 nodes. We would like to change >> it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much >> CPU/network capacities? Should I use isa/isa-l instead of jerasure? >> >> Can anyone share experience with 8+3? >> >> Kind regards, >> Rok >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 15 Apr '26

15 Apr '26

Hi! On 15/04/26 11:11, Eugen Block via ceph-users wrote: > Hi, > > Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > [cut] >>> So your cluster currently has quorum with 2 out of 3 MONs on Quincy >>> already, correct? Restarting the mon on node3 shouldn't be dangerous, >>> I think. You'd still have quorum with node4 and node5, even if node3 >>> wouldn't be able to join. If it doesn't join as Octopus, you could >>> still upgrade it and retry with Quincy. >> >> Yes. I was just reluctant to do so because in the case it doesn't join >> our room to manoeuvre will be reduced. Plus the fact that node3 is the >> only daemon which is still logging something... > > oh I didn't realize there was only one left logging something. I thought > only the newly to be joined MON didn't log. In that case I would also > try to find the root cause for that. It could easily be, because this is an old cluster which was upgraded multiple times, that "default" log settings changed in the meantime. If I query the daemon for 'mon_status' and search for log_to I see the following: # ceph daemon mon.node4 config show | grep log_to "clog_to_graylog": "false", "clog_to_graylog_host": "127.0.0.1", "clog_to_graylog_port": "12201", "clog_to_monitors": "default=true", "clog_to_syslog": "false", "clog_to_syslog_facility": "default=daemon audit=local0", "clog_to_syslog_level": "info", "leveldb_log_to_ceph_log": "true", "log_to_file": "true", "log_to_graylog": "false", "log_to_journald": "false", "log_to_stderr": "false", "log_to_syslog": "false", "mon_cluster_log_to_file": "true", "mon_cluster_log_to_graylog": "false", "mon_cluster_log_to_graylog_host": "127.0.0.1", "mon_cluster_log_to_graylog_port": "12201", "mon_cluster_log_to_journald": "false", "mon_cluster_log_to_stderr": "false", "mon_cluster_log_to_syslog": "default=false", "mon_cluster_log_to_syslog_facility": "daemon", "mon_cluster_log_to_syslog_level": "info", "rocksdb_log_to_ceph_log": "true", If I understand correctly I don't see the log in journald because '"log_to_journald": "false"' (in fact, on the Octopus node there is no "log_to_journald" option). Is this the correct default behaviour? >>> What I'm wondering about, why did you remove the MONs you wanted to >>> upgrade from Ceph? Is that recommended somewhere? Maybe you need to >>> run 'ceph mon add <name> <addr>' as well to let node1 or node2 join >>> the monmap again? I'm not sure, it has been a long time doing this >>> kind of stuff. ;-) But I don't think I ever removed a MON before >>> upgrading it, not even when reinstalling the OS. >> >> No, we didn't follow any recommendations, it seemed reasonable to >> remove a monitor before reinstalling the OS, a fresh start looked >> better than a backup/restore of '/var/lib/ceph/'. Maybe we were wrong. >> For the next upgrades (node3-5) we could think not to do that and see >> what it happens. > > I don't exactly remember, tbh, but I expect that it would suffice to > have the /var/lib/ceph/mon/ceph-nodeX directory with the correct > permissions present in order for it to re-sync with the other MONs. But > please double-check that, I'm really not sure. If I have the time, I > will try to test that in a lab environment. Thank you! I will check it before doing anything strange. BTW, I remember that another reason for the MONs removal was that we needed to change the hostname (not the IP address) and we taught that a fresh install would be better. [cut] >>>>> - Inspecting debug logs on the syncing MON could help finding the >>>>> root cause. >>>> >>>> With "syncing MON" you mean the MON which is just added to the >>>> cluster, right? Collecting this kind of logs is on the to-do list, >>>> but blocked because the "log problem" mentioned bellow. >>> >>> Yes, that's what I meant. The missing logs are a problem, of course. >>> It does sound familiar, but I don't have a good idea right now except >>> for looking at directory and file permissions, apparmor/selinux >>> things, dmesg, stuff like that. >> >> >> Yep. My biggest fear right now is that lack of logging from Quincy >> monitor daemons could mean that they don't work properly and if we >> restart the Octopus one (or it doesn't want to rejoin the quorum) we >> will have a not functioning cluster. We need to investigate further >> into it. > > I totally understand your hesitation, before I'd do that I would also > try to rule out anything I can. But at the same time it would be > surprising, given your cluster shows a healthy quorum right now. But > yeah, investigate further before moving forward is the safer choice here. > Probably you are right. I'll be cautiuos. Thank you for your help! Iztok

1 0

Re: Upgrade problem from Octopus to Quincy
by Eugen Block 15 Apr '26

15 Apr '26

Hi, Zitat von Iztok Gregori via ceph-users <ceph-users(a)ceph.io>: > Hi, > > comments inline. > > On 15/04/26 09:31, Eugen Block via ceph-users wrote: > > [cut] > >>>> - Can you share the output of 'ceph mon dump'? >>> >>> Sure! >>> >>> # ceph mon dump >>> epoch 18 >>> fsid 5bed1bf6-123f-4597-aaaa-d77931228548 >>> last_changed 2026-04-08T20:23:59.781295+0200 >>> created 2018-01-09T10:20:36.351967+0100 >>> min_mon_release 15 (octopus) >>> election_strategy: 1 >>> 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 >>> 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 >>> 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 >>> dumped monmap epoch 18 >> >> So your cluster currently has quorum with 2 out of 3 MONs on Quincy >> already, correct? Restarting the mon on node3 shouldn't be >> dangerous, I think. You'd still have quorum with node4 and node5, >> even if node3 wouldn't be able to join. If it doesn't join as >> Octopus, you could still upgrade it and retry with Quincy. > > Yes. I was just reluctant to do so because in the case it doesn't > join our room to manoeuvre will be reduced. Plus the fact that node3 > is the only daemon which is still logging something... oh I didn't realize there was only one left logging something. I thought only the newly to be joined MON didn't log. In that case I would also try to find the root cause for that. >> What I'm wondering about, why did you remove the MONs you wanted to >> upgrade from Ceph? Is that recommended somewhere? Maybe you need to >> run 'ceph mon add <name> <addr>' as well to let node1 or node2 join >> the monmap again? I'm not sure, it has been a long time doing this >> kind of stuff. ;-) But I don't think I ever removed a MON before >> upgrading it, not even when reinstalling the OS. > > No, we didn't follow any recommendations, it seemed reasonable to > remove a monitor before reinstalling the OS, a fresh start looked > better than a backup/restore of '/var/lib/ceph/'. Maybe we were > wrong. For the next upgrades (node3-5) we could think not to do that > and see what it happens. I don't exactly remember, tbh, but I expect that it would suffice to have the /var/lib/ceph/mon/ceph-nodeX directory with the correct permissions present in order for it to re-sync with the other MONs. But please double-check that, I'm really not sure. If I have the time, I will try to test that in a lab environment. >> Just for completeness sake, could you also share 'ceph -s' and >> 'ceph versions'? > > Sure! > > # ceph -s > cluster: > id: 5bed1bf6-123f-4597-aaa-d77931228548 > health: HEALTH_OK > > services: > mon: 3 daemons, quorum node3,node4,node5 (age 6d) > mgr: node1(active, since 6d), standbys: node2 > osd: 852 osds: 849 up (since 2w), 849 in (since 8w) > > data: > pools: 11 pools, 13825 pgs > objects: 447.25M objects, 1.7 PiB > usage: 5.0 PiB used, 3.7 PiB / 8.7 PiB avail > pgs: 13817 active+clean > 8 active+clean+scrubbing+deep > > io: > client: 3.0 MiB/s rd, 53 MiB/s wr, 35 op/s rd, 1.05k op/s wr > cache: 0 op/s promote > > # ceph versions > { > "mon": { > "ceph version 15.2.16 > (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 1, > "ceph version 17.2.6 > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2 > }, > "mgr": { > "ceph version 17.2.9 > (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 > }, > "osd": { > "ceph version 15.2.16 > (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 179, > "ceph version 15.2.17 > (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670 > }, > "mds": {}, > "overall": { > "ceph version 15.2.16 > (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 180, > "ceph version 15.2.17 > (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670, > "ceph version 17.2.6 > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2, > "ceph version 17.2.9 > (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 > } > } > > >>>> - There were a couple of reports back at O/P versions about MONs >>>> not joining the quorum. Sometimes it worked after hours, >>>> sometimes it didn't, and I'm still not aware of a real solution. >>>> One of our customers has been facing severe MON sync delay, >>>> sometimes taking around 40 minutes until a MON joined quorum. It >>>> turned out to be caused by some paxos default settings, requring >>>> the MONs to do a full sync instead of a diff sync. This has to do >>>> with millions of snapshots they have in their cluster. What made >>>> it worse was the MON's filesystems were on HDDs, increasing the >>>> sync time as well. >>> >>> We didn't wait for a long time before stopping the new MON a >>> regain "control" of the cluster, maybe 10 minutes max... >>> >>>> - Inspecting debug logs on the syncing MON could help finding the >>>> root cause. >>> >>> With "syncing MON" you mean the MON which is just added to the >>> cluster, right? Collecting this kind of logs is on the to-do list, >>> but blocked because the "log problem" mentioned bellow. >> >> Yes, that's what I meant. The missing logs are a problem, of >> course. It does sound familiar, but I don't have a good idea right >> now except for looking at directory and file permissions, >> apparmor/selinux things, dmesg, stuff like that. > > > Yep. My biggest fear right now is that lack of logging from Quincy > monitor daemons could mean that they don't work properly and if we > restart the Octopus one (or it doesn't want to rejoin the quorum) we > will have a not functioning cluster. We need to investigate further > into it. I totally understand your hesitation, before I'd do that I would also try to rule out anything I can. But at the same time it would be surprising, given your cluster shows a healthy quorum right now. But yeah, investigate further before moving forward is the safer choice here. > Iztok > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Changing EC profile
by Rok Jaklič 15 Apr '26

15 Apr '26

So I've created a little plan on how to create a new profile and assign new S3 users to EC 8+3 pool. Idea is that all new s3 users are using a new pool, and old ones pool with EC 3+2 and data is intact. --- ceph osd erasure-code-profile set ec-83-profile k=8 m=3 crush-failure-domain=host ceph osd pool create rgw.buckets.ec-83.data 256 erasure ec-83-profile ceph osd pool application enable rgw.buckets.ec-83.data rgw radosgw-admin zonegroup placement add --placement-id=ec-83-placement radosgw-admin zone placement add \ --placement-id ec-83-placement \ --data-pool rgw.buckets.ec-83.data \ --index-pool default.rgw.buckets.index \ --data-extra-pool default.rgw.buckets.non-ec \ --placement-index-type 0 radosgw-admin period update --commit # restart rgw systemctl restart ceph-radosgw.target # verify the new placement radosgw-admin zone get | grep -A 20 "ec-83-placement" # for new s3 users, set the default placement radosgw-admin user create --uid=<user> --tenant=<tenant> --placement-id=ec-83-placement --- It seems to work on "test cluster", however I get an error on: [root@a8-mon1 ~]# radosgw-admin period update --commit cannot commit period: period does not have a master zone of a master zonegroup failed to commit period: (22) Invalid argument Do I actually need to run this command or is "rgw restart" enough? Any comments appriciated. Rok On Fri, Apr 3, 2026 at 12:42 PM Rok Jaklič via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > we expanded the cluster to 12 nodes with 28 osds, but we are still running > on EC 3+2, since we started cluster with 6 nodes. We would like to change > it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much > CPU/network capacities? Should I use isa/isa-l instead of jerasure? > > Can anyone share experience with 8+3? > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade problem from Octopus to Quincy
by Iztok Gregori 15 Apr '26

15 Apr '26

Hi, comments inline. On 15/04/26 09:31, Eugen Block via ceph-users wrote: [cut] >>> - Can you share the output of 'ceph mon dump'? >> >> Sure! >> >> # ceph mon dump >> epoch 18 >> fsid 5bed1bf6-123f-4597-aaaa-d77931228548 >> last_changed 2026-04-08T20:23:59.781295+0200 >> created 2018-01-09T10:20:36.351967+0100 >> min_mon_release 15 (octopus) >> election_strategy: 1 >> 0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3 >> 1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4 >> 2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5 >> dumped monmap epoch 18 > > So your cluster currently has quorum with 2 out of 3 MONs on Quincy > already, correct? Restarting the mon on node3 shouldn't be dangerous, I > think. You'd still have quorum with node4 and node5, even if node3 > wouldn't be able to join. If it doesn't join as Octopus, you could still > upgrade it and retry with Quincy. Yes. I was just reluctant to do so because in the case it doesn't join our room to manoeuvre will be reduced. Plus the fact that node3 is the only daemon which is still logging something... > What I'm wondering about, why did you remove the MONs you wanted to > upgrade from Ceph? Is that recommended somewhere? Maybe you need to run > 'ceph mon add <name> <addr>' as well to let node1 or node2 join the > monmap again? I'm not sure, it has been a long time doing this kind of > stuff. ;-) But I don't think I ever removed a MON before upgrading it, > not even when reinstalling the OS. No, we didn't follow any recommendations, it seemed reasonable to remove a monitor before reinstalling the OS, a fresh start looked better than a backup/restore of '/var/lib/ceph/'. Maybe we were wrong. For the next upgrades (node3-5) we could think not to do that and see what it happens. > Just for completeness sake, could you also share 'ceph -s' and 'ceph > versions'? Sure! # ceph -s cluster: id: 5bed1bf6-123f-4597-aaa-d77931228548 health: HEALTH_OK services: mon: 3 daemons, quorum node3,node4,node5 (age 6d) mgr: node1(active, since 6d), standbys: node2 osd: 852 osds: 849 up (since 2w), 849 in (since 8w) data: pools: 11 pools, 13825 pgs objects: 447.25M objects, 1.7 PiB usage: 5.0 PiB used, 3.7 PiB / 8.7 PiB avail pgs: 13817 active+clean 8 active+clean+scrubbing+deep io: client: 3.0 MiB/s rd, 53 MiB/s wr, 35 op/s rd, 1.05k op/s wr cache: 0 op/s promote # ceph versions { "mon": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 1, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2 }, "mgr": { "ceph version 17.2.9 (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 }, "osd": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 179, "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670 }, "mds": {}, "overall": { "ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)": 180, "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 670, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 2, "ceph version 17.2.9 (69bf48f20731a4b0d742613f6c6335ccb54dd217) quincy (stable)": 2 } } >>> - There were a couple of reports back at O/P versions about MONs not >>> joining the quorum. Sometimes it worked after hours, sometimes it >>> didn't, and I'm still not aware of a real solution. One of our >>> customers has been facing severe MON sync delay, sometimes taking >>> around 40 minutes until a MON joined quorum. It turned out to be >>> caused by some paxos default settings, requring the MONs to do a full >>> sync instead of a diff sync. This has to do with millions of >>> snapshots they have in their cluster. What made it worse was the >>> MON's filesystems were on HDDs, increasing the sync time as well. >> >> We didn't wait for a long time before stopping the new MON a regain >> "control" of the cluster, maybe 10 minutes max... >> >>> - Inspecting debug logs on the syncing MON could help finding the >>> root cause. >> >> With "syncing MON" you mean the MON which is just added to the >> cluster, right? Collecting this kind of logs is on the to-do list, but >> blocked because the "log problem" mentioned bellow. > > Yes, that's what I meant. The missing logs are a problem, of course. It > does sound familiar, but I don't have a good idea right now except for > looking at directory and file permissions, apparmor/selinux things, > dmesg, stuff like that. Yep. My biggest fear right now is that lack of logging from Quincy monitor daemons could mean that they don't work properly and if we restart the Octopus one (or it doesn't want to rejoin the quorum) we will have a not functioning cluster. We need to investigate further into it. Iztok

1 0