- ceph-users - lists.ceph.io

Re: [EXTERNAL] Re: [EXTERNAL] Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Redouane Kachach 30 Apr '26

30 Apr '26

Hi Steve, I'd recommend (if possible) disabling FIPS, then forcing a mgr fail-over to recover cephadm control. After that, you should be able to use cephadm commands to fix/remove the failing services. Best, Redo. On Wed, Apr 29, 2026 at 4:20 PM Barber, Steve W. (Fed) via ceph-users < ceph-users(a)ceph.io> wrote: > Thank you Redo! > > Do you have any suggestions for how I can back out of the current > situation to get rid of the health warning and the orphaned mgmt-gateway > services without making a mess of things? > > Steve > > From: Redouane Kachach via ceph-users <ceph-users(a)ceph.io> > Date: Tuesday, April 28, 2026 at 4:25 AM > To: Barber, Steve W. (Fed) <steve.barber(a)nist.gov> > Cc: Ceph Users <ceph-users(a)ceph.io> > Subject: [EXTERNAL] [ceph-users] Re: [EXTERNAL] Re: Attempt to enable > mgmt-gateway on v20.2.1 failed, and now cephadm module is broken > > Hi Steve, > > I saw that you already opened a ticket for the issue (thanks): > > > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.c… > <https://tracker.ceph.com/issues/76185>. > > This problem is basically related to the env (FIPS is enabled). A PR with a > candidate fix is under-review: > > > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co… > <https://github.com/ceph/ceph/pull/68638>. > > We will backport this change to Tentacle. > > Best, > Redo. > > > > On Mon, Apr 27, 2026 at 5:02 PM Barber, Steve W. (Fed) < > steve.barber(a)nist.gov> wrote: > > > Hi Redo, > > > > The spec was very simple: > > > > service_type: mgmt-gateway > > placement: > > #label: mgmt > > hosts: > > - alfred > > - cyclops > > - mask > > spec: > > virtual_ip: (redacted)/24 > > port: 443 > > > > As far as active mgr logs, at this point I’m not seeing anything unusual > > in the logs from the active mgr host. If I attempt to remove the service > > again (it was already theoretically removed) I get: > > > > $ ceph orch rm mgmt-gateway > > Invalid service 'mgmt-gateway'. Use 'ceph orch ls' to list available > > services. > > > > That generated these lines in the log: > > > > ... > > Apr 27 10:35:03 alfred bash[3568828]: debug 2026-04-27T14:35:03.592+0000 > > 7f21e7e80640 0 log_channel(audit) log [DBG] : from=‘mgr.42485063 > (alfred’s > > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd > blocklist > > ls", "format": "json"} : dispatch > > Apr 27 10:35:04 alfred bash[3568828]: audit > > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > > Apr 27 10:35:04 alfred bash[3568828]: audit > > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > > Apr 27 10:35:06 alfred bash[3566093]: debug 2026-04-27T14:35:06.596+0000 > > 7f4324bbe640 0 log_channel(audit) log [DBG] : from=‘client.42505047 -' > > entity='client.admin' cmd=[{"prefix": "orch rm", "service_name": > > "mgmt-gateway", "target": ["mon-mgr", ""]}]: dispatch > > Apr 27 10:35:07 alfred bash[3568828]: audit > > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > > audit [DBG] from='client.42505047 -' entity='client.admin' > cmd=[{"prefix": > > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > > dispatch > > Apr 27 10:35:07 alfred bash[3568828]: audit > > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > > audit [DBG] from='client.42505047 -' entity='client.admin' > cmd=[{"prefix": > > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > > dispatch > > Apr 27 10:35:07 alfred bash[3568828]: cephadm > > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > > cephadm [INF] Remove service mgmt-gateway > > Apr 27 10:35:07 alfred bash[3568828]: cephadm > > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > > cephadm [INF] Remove service mgmt.-gateway > > Apr 27 10:35:18 alfred bash[3568828]: debug 2026-04-27T14:35:18.596+0000 > > 7f21e7e80640 0 log_channel(audit) log [DBG] : from='mgr.42485063 > (alfred’s > > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd > blocklist > > ls", "format": "json"} : dispatch > > Apr 27 10:35:19 alfred bash[3568828]: audit > > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > > Apr 27 10:35:19 alfred bash[3568828]: audit > > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > > ... > > > > But otherwise everything else looks pretty normal. > > > > This is still with 2 of the 3 mon/mgr nodes running the mgmt-gateway > > service: > > $ ceph orch ps | egrep 'mgr|mon|mgmt' > > mgmt-gateway.cyclops cyclops running (6d) 5d ago > > 6d 6188k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (6d) 5d ago > > 6d 5980k - 1.26.1 c2348906c6ff 329705c23aad > > mgr.alfred.afvnok alfred *:8443,9283,8765 running (2w) 5d ago > > 7M 564M - 20.2.1 fb63cba66eea 875883d08efc > > mgr.cyclops.nifeym cyclops *:8443,9283,8765 running (2w) 5d ago > > 7M 463M - 20.2.1 fb63cba66eea 676ec1753348 > > mgr.mask.yfbdpf mask *:8443,9283,8765 running (2w) 5d ago > > 7M 486M - 20.2.1 fb63cba66eea 6a1bdd37997c > > mon.alfred alfred running (2w) 5d ago > > 7M 525M 2048M 20.2.1 fb63cba66eea ce5bbb75d650 > > mon.cyclops cyclops running (2w) 5d ago > > 7M 571M 2048M 20.2.1 fb63cba66eea e675cf6714e4 > > mon.mask mask running (2w) 5d ago > > 7M 510M 2048M 20.2.1 fb63cba66eea 9a034ab6f132 > > > > And I still have the same health error. > > > > Let me know if there’s anything more specific you want me to show. > > > > Steve > > > > *From: *Redouane Kachach <rkachach(a)redhat.com> > > *Date: *Friday, April 24, 2026 at 4:32 AM > > *To: *Barber, Steve W. (Fed) <steve.barber(a)nist.gov> > > *Cc: *Ceph Users <ceph-users(a)ceph.io> > > *Subject: *[EXTERNAL] Re: [ceph-users] Attempt to enable mgmt-gateway on > > v20.2.1 failed, and now cephadm module is broken > > > > Hi Steve, > > > > Can you plz share the spec you used to configure the mgmt-gateway and the > > active mgr logs? > > > > Best, > > Redo. > > > > On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > > I have a pre-production cluster running on Ubuntu 22.04 hosts using > > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, > which > > went completely smoothly. We are a government facility so we are required > > to enable FIPS mode in the OS. I believe this last fact may be the cause > of > > the following: > > > > I wanted to enable mgmt-gateway so I made a service definition file to > > enable it and applied it. It appeared to work, but then “ceph health > > detail” showed: > > > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > > unsupported > > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > > routines] unsupported > > Module 'cephadm' has failed: [digital envelope routines] unsupported > > > > The mgmt-gateway service was started on all 3 nodes however. > > > > $ ceph orch ps --service_name mgmt-gateway > > NAME HOST PORTS STATUS REFRESHED AGE MEM > > USE MEM LIM VERSION IMAGE ID CONTAINER ID > > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (4m) 3m ago 4m > > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > > > I then attempted to remove the service definition with ceph orch rm, but > > it only succeeded in removing it on 1 of the 3 nodes. The process is > still > > running on the other 2, and the warning is still present. “ceph mgr fail > && > > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > > comes right back, and the process is still present on 2 nodes (as shown > by > > “ceph orch ps”). > > > > $ ceph mgr fail && ceph orch rm mgmt-gateway > > Removed service mgmt-gateway > > > > $ ceph orch ps --service_name mgmt-gateway > > NAME HOST PORTS STATUS REFRESHED AGE MEM > > USE MEM LIM VERSION IMAGE ID CONTAINER ID > > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > > mgmt-gateway.mask mask running (15h) 15h ago 15h > > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > > > $ ceph health detail > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > > unsupported > > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > > routines] unsupported > > Module 'cephadm' has failed: [digital envelope routines] unsupported > > > > Despite the 2 processes running, the configured service IP does not > > respond to a ping or requests from curl. > > > > Since we were getting ready to move this cluster into production I > decided > > to look for help here rather than proceed experimenting further and risk > > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > > (b) recover the cephadm module to get back to normal with no warnings; > and > > less importantly, (c) is there a way to get mgmt-gateway working in this > > environment? A little googling and AI querying suggests it may be an > issue > > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > > up my own proxy; I was just trying to use the built-in tools where > > possible.) > > > > Thanks in advance. > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Anthony D'Atri 30 Apr '26

30 Apr '26

Apologies, I could swear I saw an example float by. I might be conflating LC with the new tiering. > On Apr 29, 2026, at 9:17 PM, Matthew H via ceph-users <ceph-users(a)ceph.io> wrote: > > > s3 lifecycle rules are based on object age, which only reflects the > time since last write. so i'm afraid you won't be able to transition > objects based on read activity

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Matthew H 30 Apr '26

30 Apr '26

Hi Vishu, There is no symmetric, read-triggered promotion between two local storage classes as Casey stated. The closest mechanism RGW has today is the cloud-tier read-through path: - cloud-transition transitions objects out to an S3-compatible endpoint and leaves a stub head object behind. https://docs.ceph.com/en/tentacle/radosgw/cloud-transition/ - cloud-restore brings them back, either via an explicit RestoreObject call (temporary or permanent), or transparently on GET if the tier-config has allow_read_through=true. With read-through enabled, a regular GET on a transitioned stub triggers a restore automatically. https://docs.ceph.com/en/tentacle/radosgw/cloud-restore/ That's the closest fit for your "active again -> promote back" requirement. Which in your case would require two separate Ceph clusters configured with RGW, one with NVMe and the other with HDD. ________________________________ From: Casey Bodley via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, April 29, 2026 8:22 AM To: Vishnu Bhaskar <vishnukb(a)acceleronlabs.com> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD On Wed, Apr 29, 2026 at 7:02 AM Vishnu Bhaskar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I have a question regarding object lifecycle management in Ceph RGW. > > Is it possible to use RGW lifecycle policies to automatically move objects > between different storage tiers for example, from a hot tier (NVMe) to a > cold tier (HDD) based on inactivity over a period of time? > > Additionally, if those objects become active again (i.e., are accessed > frequently after being moved to the cold tier), is there a mechanism to > automatically move them back to the original hot tier (NVMe)? > > I may be misunderstanding how lifecycle management works in RGW, so please > correct me if that’s the case. If this type of tiering is supported, could > you point me toward the recommended approach or configuration? s3 lifecycle rules are based on object age, which only reflects the time since last write. so i'm afraid you won't be able to transition objects based on read activity > > Thanks and Regards > Vishnu Bhaskar > Acceleron Labs Pvt Ltd > Bangalore, India > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [EXTERNAL] Re: [EXTERNAL] Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Barber, Steve W. (Fed) 29 Apr '26

29 Apr '26

Thank you Redo! Do you have any suggestions for how I can back out of the current situation to get rid of the health warning and the orphaned mgmt-gateway services without making a mess of things? Steve From: Redouane Kachach via ceph-users <ceph-users(a)ceph.io> Date: Tuesday, April 28, 2026 at 4:25 AM To: Barber, Steve W. (Fed) <steve.barber(a)nist.gov> Cc: Ceph Users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] Re: [EXTERNAL] Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken Hi Steve, I saw that you already opened a ticket for the issue (thanks): https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.c…<https://tracker.ceph.com/issues/76185>. This problem is basically related to the env (FIPS is enabled). A PR with a candidate fix is under-review: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co…<https://github.com/ceph/ceph/pull/68638>. We will backport this change to Tentacle. Best, Redo. On Mon, Apr 27, 2026 at 5:02 PM Barber, Steve W. (Fed) < steve.barber(a)nist.gov> wrote: > Hi Redo, > > The spec was very simple: > > service_type: mgmt-gateway > placement: > #label: mgmt > hosts: > - alfred > - cyclops > - mask > spec: > virtual_ip: (redacted)/24 > port: 443 > > As far as active mgr logs, at this point I’m not seeing anything unusual > in the logs from the active mgr host. If I attempt to remove the service > again (it was already theoretically removed) I get: > > $ ceph orch rm mgmt-gateway > Invalid service 'mgmt-gateway'. Use 'ceph orch ls' to list available > services. > > That generated these lines in the log: > > ... > Apr 27 10:35:03 alfred bash[3568828]: debug 2026-04-27T14:35:03.592+0000 > 7f21e7e80640 0 log_channel(audit) log [DBG] : from=‘mgr.42485063 (alfred’s > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist > ls", "format": "json"} : dispatch > Apr 27 10:35:04 alfred bash[3568828]: audit > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:04 alfred bash[3568828]: audit > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:06 alfred bash[3566093]: debug 2026-04-27T14:35:06.596+0000 > 7f4324bbe640 0 log_channel(audit) log [DBG] : from=‘client.42505047 -' > entity='client.admin' cmd=[{"prefix": "orch rm", "service_name": > "mgmt-gateway", "target": ["mon-mgr", ""]}]: dispatch > Apr 27 10:35:07 alfred bash[3568828]: audit > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > audit [DBG] from='client.42505047 -' entity='client.admin' cmd=[{"prefix": > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > dispatch > Apr 27 10:35:07 alfred bash[3568828]: audit > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > audit [DBG] from='client.42505047 -' entity='client.admin' cmd=[{"prefix": > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > dispatch > Apr 27 10:35:07 alfred bash[3568828]: cephadm > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > cephadm [INF] Remove service mgmt-gateway > Apr 27 10:35:07 alfred bash[3568828]: cephadm > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > cephadm [INF] Remove service mgmt.-gateway > Apr 27 10:35:18 alfred bash[3568828]: debug 2026-04-27T14:35:18.596+0000 > 7f21e7e80640 0 log_channel(audit) log [DBG] : from='mgr.42485063 (alfred’s > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist > ls", "format": "json"} : dispatch > Apr 27 10:35:19 alfred bash[3568828]: audit > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:19 alfred bash[3568828]: audit > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > ... > > But otherwise everything else looks pretty normal. > > This is still with 2 of the 3 mon/mgr nodes running the mgmt-gateway > service: > $ ceph orch ps | egrep 'mgr|mon|mgmt' > mgmt-gateway.cyclops cyclops running (6d) 5d ago > 6d 6188k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (6d) 5d ago > 6d 5980k - 1.26.1 c2348906c6ff 329705c23aad > mgr.alfred.afvnok alfred *:8443,9283,8765 running (2w) 5d ago > 7M 564M - 20.2.1 fb63cba66eea 875883d08efc > mgr.cyclops.nifeym cyclops *:8443,9283,8765 running (2w) 5d ago > 7M 463M - 20.2.1 fb63cba66eea 676ec1753348 > mgr.mask.yfbdpf mask *:8443,9283,8765 running (2w) 5d ago > 7M 486M - 20.2.1 fb63cba66eea 6a1bdd37997c > mon.alfred alfred running (2w) 5d ago > 7M 525M 2048M 20.2.1 fb63cba66eea ce5bbb75d650 > mon.cyclops cyclops running (2w) 5d ago > 7M 571M 2048M 20.2.1 fb63cba66eea e675cf6714e4 > mon.mask mask running (2w) 5d ago > 7M 510M 2048M 20.2.1 fb63cba66eea 9a034ab6f132 > > And I still have the same health error. > > Let me know if there’s anything more specific you want me to show. > > Steve > > *From: *Redouane Kachach <rkachach(a)redhat.com> > *Date: *Friday, April 24, 2026 at 4:32 AM > *To: *Barber, Steve W. (Fed) <steve.barber(a)nist.gov> > *Cc: *Ceph Users <ceph-users(a)ceph.io> > *Subject: *[EXTERNAL] Re: [ceph-users] Attempt to enable mgmt-gateway on > v20.2.1 failed, and now cephadm module is broken > > Hi Steve, > > Can you plz share the spec you used to configure the mgmt-gateway and the > active mgr logs? > > Best, > Redo. > > On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < > ceph-users(a)ceph.io> wrote: > > I have a pre-production cluster running on Ubuntu 22.04 hosts using > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which > went completely smoothly. We are a government facility so we are required > to enable FIPS mode in the OS. I believe this last fact may be the cause of > the following: > > I wanted to enable mgmt-gateway so I made a service definition file to > enable it and applied it. It appeared to work, but then “ceph health > detail” showed: > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > The mgmt-gateway service was started on all 3 nodes however. > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (4m) 3m ago 4m > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > I then attempted to remove the service definition with ceph orch rm, but > it only succeeded in removing it on 1 of the 3 nodes. The process is still > running on the other 2, and the warning is still present. “ceph mgr fail && > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > comes right back, and the process is still present on 2 nodes (as shown by > “ceph orch ps”). > > $ ceph mgr fail && ceph orch rm mgmt-gateway > Removed service mgmt-gateway > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (15h) 15h ago 15h > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > $ ceph health detail > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > Despite the 2 processes running, the configured service IP does not > respond to a ping or requests from curl. > > Since we were getting ready to move this cluster into production I decided > to look for help here rather than proceed experimenting further and risk > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > (b) recover the cephadm module to get back to normal with no warnings; and > less importantly, (c) is there a way to get mgmt-gateway working in this > environment? A little googling and AI querying suggests it may be an issue > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > up my own proxy; I was just trying to use the built-in tools where > possible.) > > Thanks in advance. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Casey Bodley 29 Apr '26

29 Apr '26

On Wed, Apr 29, 2026 at 7:02 AM Vishnu Bhaskar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I have a question regarding object lifecycle management in Ceph RGW. > > Is it possible to use RGW lifecycle policies to automatically move objects > between different storage tiers for example, from a hot tier (NVMe) to a > cold tier (HDD) based on inactivity over a period of time? > > Additionally, if those objects become active again (i.e., are accessed > frequently after being moved to the cold tier), is there a mechanism to > automatically move them back to the original hot tier (NVMe)? > > I may be misunderstanding how lifecycle management works in RGW, so please > correct me if that’s the case. If this type of tiering is supported, could > you point me toward the recommended approach or configuration? s3 lifecycle rules are based on object age, which only reflects the time since last write. so i'm afraid you won't be able to transition objects based on read activity > > Thanks and Regards > Vishnu Bhaskar > Acceleron Labs Pvt Ltd > Bangalore, India > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Vishnu Bhaskar 29 Apr '26

29 Apr '26

Hi FYI Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India On Wed, Apr 29, 2026 at 6:41 PM Julien Laurenceau via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > I confirm you can setup tiering between two differents S3 pools on the > same ceph cluster with squid. > > You have to create 2 differents pools and placement groups relying on > your two type of physical storage drives. You may use the same RGW > client and use the S3 storage class attribute or you may also create a > new RGW client dedicated to each pool. > > I use S3 bucket policy for this and you should prefer using standard AWS > names of storage class ("GLACIER" or "STANDARD_IA"...) to do this easily > using the aws cli. > > > On 4/29/26 14:18, Anthony D'Atri via ceph-users wrote: > > > >> Is it possible to use RGW lifecycle policies to automatically move > objects > >> between different storage tiers for example, from a hot tier (NVMe) to a > >> cold tier (HDD) based on inactivity over a period of time? > > Absolutely. This is one key application of S3 storage classes. Note > that LC policies are run once a day by default, so with a heavy load you > may wish to deploy a set of RGWs that are not user-facing, but which only > handle LC tasks, and disable LC processing for the user-facing cohort of > daemons. > > > > > >> Additionally, if those objects become active again (i.e., are accessed > >> frequently after being moved to the cold tier), is there a mechanism to > >> automatically move them back to the original hot tier (NVMe)? > > But of course! Transparent tiering is available from Squid, at least > when offloading cold objects to a different endpoint. I'm not positive that > this works natively within a single cluster, but it may well. You could > though most likely still use LC policies, albeit in a less elegant fashion. > > > >> I may be misunderstanding how lifecycle management works in RGW, so > please > >> correct me if that’s the case. If this type of tiering is supported, > could > >> you point me toward the recommended approach or configuration? > > The following awesome blog posts from Daniel Parkes will guide you. When > deploying a cold tier, it is ideal to leave the first STANDARD storage > class as the default so that HEAD objects are always on faster storage. > They don't take much space, but this accelerates various operations. > > > > https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/ > > > > Depending on your object sizes and workload, consider replication vs > erasure coding. When using EC for your slower -- or even faster -- storage > class, be sure to select a value of m >= 2. Profiles wider than 8+3 may > increase space amp and cost write, scrub, and recovery performance. Always > choose (k+m < the number of CRUSH failure domains. > > > > Writes to a replicated pool will generally be faster, but EC makes more > efficient use of underlying storage. > > > > This is nuanced, though for the smallest objects. Check out Mark > Nelson's most excellent spreadsheet that shows this visually: > > > > > https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUTo… > > > > Through the Squid release, RADOS always writes full stripes, so if you > are using EC, the very smallest objects will consume more underlying raw > storage than one might think. The wider the EC profile, the more pronounced > this effect. If you have a large proportion of tiny objects, say below 256 > - 512 KiB, this space amplification may be significant. If the number of > very small objects is dwarfed by the number * size of larger objects, this > is amortized into the noise floor. If not, you can consider client headers > or Lua scripting to assort objects to appropriate storage classes at > ingest. Check out these presentations by Frederic Nass and Steve U: > > > > > https://ceph.io/en/news/blog/2024/auto-tiering-ceph-object-storage-part-1/ > > https://www.youtube.com/watch?v=qKd2Po2hTe4 > > > > Note that with the Tentacle release, bucket data pools with the EC > optimizations flag significantly reduce this space amplification, lowering > the size threshold for special handling and reducing the need to worry > about it. > > > > > > https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/ > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Julien Laurenceau 29 Apr '26

29 Apr '26

Hi, I confirm you can setup tiering between two differents S3 pools on the same ceph cluster with squid. You have to create 2 differents pools and placement groups relying on your two type of physical storage drives. You may use the same RGW client and use the S3 storage class attribute or you may also create a new RGW client dedicated to each pool. I use S3 bucket policy for this and you should prefer using standard AWS names of storage class ("GLACIER" or "STANDARD_IA"...) to do this easily using the aws cli. On 4/29/26 14:18, Anthony D'Atri via ceph-users wrote: > >> Is it possible to use RGW lifecycle policies to automatically move objects >> between different storage tiers for example, from a hot tier (NVMe) to a >> cold tier (HDD) based on inactivity over a period of time? > Absolutely. This is one key application of S3 storage classes. Note that LC policies are run once a day by default, so with a heavy load you may wish to deploy a set of RGWs that are not user-facing, but which only handle LC tasks, and disable LC processing for the user-facing cohort of daemons. > > >> Additionally, if those objects become active again (i.e., are accessed >> frequently after being moved to the cold tier), is there a mechanism to >> automatically move them back to the original hot tier (NVMe)? > But of course! Transparent tiering is available from Squid, at least when offloading cold objects to a different endpoint. I'm not positive that this works natively within a single cluster, but it may well. You could though most likely still use LC policies, albeit in a less elegant fashion. > >> I may be misunderstanding how lifecycle management works in RGW, so please >> correct me if that’s the case. If this type of tiering is supported, could >> you point me toward the recommended approach or configuration? > The following awesome blog posts from Daniel Parkes will guide you. When deploying a cold tier, it is ideal to leave the first STANDARD storage class as the default so that HEAD objects are always on faster storage. They don't take much space, but this accelerates various operations. > > https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/ > > Depending on your object sizes and workload, consider replication vs erasure coding. When using EC for your slower -- or even faster -- storage class, be sure to select a value of m >= 2. Profiles wider than 8+3 may increase space amp and cost write, scrub, and recovery performance. Always choose (k+m < the number of CRUSH failure domains. > > Writes to a replicated pool will generally be faster, but EC makes more efficient use of underlying storage. > > This is nuanced, though for the smallest objects. Check out Mark Nelson's most excellent spreadsheet that shows this visually: > > https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUTo… > > Through the Squid release, RADOS always writes full stripes, so if you are using EC, the very smallest objects will consume more underlying raw storage than one might think. The wider the EC profile, the more pronounced this effect. If you have a large proportion of tiny objects, say below 256 - 512 KiB, this space amplification may be significant. If the number of very small objects is dwarfed by the number * size of larger objects, this is amortized into the noise floor. If not, you can consider client headers or Lua scripting to assort objects to appropriate storage classes at ingest. Check out these presentations by Frederic Nass and Steve U: > > https://ceph.io/en/news/blog/2024/auto-tiering-ceph-object-storage-part-1/ > https://www.youtube.com/watch?v=qKd2Po2hTe4 > > Note that with the Tentacle release, bucket data pools with the EC optimizations flag significantly reduce this space amplification, lowering the size threshold for special handling and reducing the need to worry about it. > > > https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/ > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Anthony D'Atri 29 Apr '26

29 Apr '26

> > Is it possible to use RGW lifecycle policies to automatically move objects > between different storage tiers for example, from a hot tier (NVMe) to a > cold tier (HDD) based on inactivity over a period of time? Absolutely. This is one key application of S3 storage classes. Note that LC policies are run once a day by default, so with a heavy load you may wish to deploy a set of RGWs that are not user-facing, but which only handle LC tasks, and disable LC processing for the user-facing cohort of daemons. > Additionally, if those objects become active again (i.e., are accessed > frequently after being moved to the cold tier), is there a mechanism to > automatically move them back to the original hot tier (NVMe)? But of course! Transparent tiering is available from Squid, at least when offloading cold objects to a different endpoint. I'm not positive that this works natively within a single cluster, but it may well. You could though most likely still use LC policies, albeit in a less elegant fashion. > I may be misunderstanding how lifecycle management works in RGW, so please > correct me if that’s the case. If this type of tiering is supported, could > you point me toward the recommended approach or configuration? The following awesome blog posts from Daniel Parkes will guide you. When deploying a cold tier, it is ideal to leave the first STANDARD storage class as the default so that HEAD objects are always on faster storage. They don't take much space, but this accelerates various operations. https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/ Depending on your object sizes and workload, consider replication vs erasure coding. When using EC for your slower -- or even faster -- storage class, be sure to select a value of m >= 2. Profiles wider than 8+3 may increase space amp and cost write, scrub, and recovery performance. Always choose (k+m < the number of CRUSH failure domains. Writes to a replicated pool will generally be faster, but EC makes more efficient use of underlying storage. This is nuanced, though for the smallest objects. Check out Mark Nelson's most excellent spreadsheet that shows this visually: https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUTo… Through the Squid release, RADOS always writes full stripes, so if you are using EC, the very smallest objects will consume more underlying raw storage than one might think. The wider the EC profile, the more pronounced this effect. If you have a large proportion of tiny objects, say below 256 - 512 KiB, this space amplification may be significant. If the number of very small objects is dwarfed by the number * size of larger objects, this is amortized into the noise floor. If not, you can consider client headers or Lua scripting to assort objects to appropriate storage classes at ingest. Check out these presentations by Frederic Nass and Steve U: https://ceph.io/en/news/blog/2024/auto-tiering-ceph-object-storage-part-1/ https://www.youtube.com/watch?v=qKd2Po2hTe4 Note that with the Tentacle release, bucket data pools with the EC optimizations flag significantly reduce this space amplification, lowering the size threshold for special handling and reducing the need to worry about it. https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/

1 0

Question on RGW Lifecycle Management and Tiering Between NVMe and HDD
by Vishnu Bhaskar 29 Apr '26

29 Apr '26

Hi, I have a question regarding object lifecycle management in Ceph RGW. Is it possible to use RGW lifecycle policies to automatically move objects between different storage tiers for example, from a hot tier (NVMe) to a cold tier (HDD) based on inactivity over a period of time? Additionally, if those objects become active again (i.e., are accessed frequently after being moved to the cold tier), is there a mechanism to automatically move them back to the original hot tier (NVMe)? I may be misunderstanding how lifecycle management works in RGW, so please correct me if that’s the case. If this type of tiering is supported, could you point me toward the recommended approach or configuration? Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Harald Staub 29 Apr '26

29 Apr '26

This does not use upmap. With the swap-bucket command, all the data of the source node (and some more) becomes "misplaced". I would expect it to work less direct in case lots of upmap remappings are in place. On 4/29/26 04:44, Wannes Smet via ceph-users wrote: > Does swap-bucket use upmap? If so, wouldn't that mean that a lot of PGs no longer end up where CRUSH would have placed them? > ________________________________ > From: Eugen Block via ceph-users <ceph-users(a)ceph.io> > Sent: Tuesday, April 28, 2026 21:40 > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Best "modern" approach to drain/fill a OSD server > > +1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > +1 for swap-bucket, we've been using that as well successfully. > > What we also sometimes do, especially when time is a bit more > critical, is to purge OSDs of an entire host (we also did this with > two hosts at the same time due to high resiliency) and then let Ceph > recover. But I generally don't recommend that to unexperienced > operators. > > > Zitat von Harald Staub via ceph-users <ceph-users(a)ceph.io>: > >> Not recently, but a few years ago, I used the "Whole host >> replacement" using "swap-bucket": >> https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/operation… >> >> This worked really nicely. With source and destination about the >> same host CRUSH weight, most PGs are moved directly from source to >> destination. So also the copy speed was more or less the same most >> of the time. >> >> Cheers, >> Harry >> >> On 4/28/26 11:08, Denis Polom via ceph-users wrote: >>> you can control max_backfills and some others if you set >>> osd_mclock_override_recovery_settings to true >>> >>> On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>>>> Another approach is to set the flags, destroy the OSDs on a >>>>> single host, redeploy them, set your backfill/recovery throttles >>>>> down to 1, and unset the flags. >>>> This is harder with mclock right? With wpq you can still control >>>> max_backfills, but not with mclock unless I am mistaken. >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0