- ceph-users - lists.ceph.io

Re: Best "modern" approach to drain/fill a OSD server
by Iztok Gregori 29 Apr '26

29 Apr '26

On 27/04/26 17:42, Iztok Gregori via ceph-users wrote: >>> I assume that 'target_max_misplaced_ratio' is a key parameter to >>> avoid large data movement in a given time, there is more to look into? >> >> Yes. Setting that lower than default can help avoid certain >> intermediate states that can result in a fairly-full cluster's OSDs >> going nearfull/backfillfull. > > The documentation say about 'target_max_misplaced_ratio' "...the > balancer will incrementally move a small fraction of unbalanced PGs in > order to improve distribution. This fraction will not exceed a certain > threshold that defaults to 5%...". does it mean that off all the PGs > that has to be moved out/in of the node(s), maximum 5% will be moved at > the same time? It seems that in my case (Ceph Quincy) 'target_max_misplaced_ratio' is looking for misplaced objects, not PGs (as I thought). I activated the balancer on a unbalanced cluster, set the ratio to 0.005000 (0.5%) and watch the bytes fly. I was expecting that maximum 0.5% of PGs (~70 PGs) will be in backfilling, but the number almost doubled (~130 PGs, > 1% of PGs) until the balancer status said: "Too many objects (0.005030 > 0.005000) are misplaced; try again later" I re-checked the latest documentation and the PGs are explicitly mention when talking about the ratio: "...The maximum percentage of PGs to remap (move) in a single phase defaults to 5%..." https://docs.ceph.com/en/latest/rados/operations/balancer/#throttling Is this the default on newer version of Ceph (I'm running Quincy in a upgrade path to Squid) or the documentation is not correct? Thanks Iztok Gregori

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Anthony D'Atri 29 Apr '26

29 Apr '26

I believe it’s an in-place edit of the CRUSH map. A powerful tool if you have a whole and fungible failure domain of undeployed hardware available. Last year I helped someone who did, and wanted to avoid long delays when transitioning their systems. > On Apr 28, 2026, at 10:42 PM, Wannes Smet via ceph-users <ceph-users(a)ceph.io> wrote: > > Does swap-bucket use upmap? If so, wouldn't that mean that a lot of PGs no longer end up where CRUSH would have placed them? > ________________________________ > From: Eugen Block via ceph-users <ceph-users(a)ceph.io> > Sent: Tuesday, April 28, 2026 21:40 > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Best "modern" approach to drain/fill a OSD server > > +1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > +1 for swap-bucket, we've been using that as well successfully. > > What we also sometimes do, especially when time is a bit more > critical, is to purge OSDs of an entire host (we also did this with > two hosts at the same time due to high resiliency) and then let Ceph > recover. But I generally don't recommend that to unexperienced > operators. > > > Zitat von Harald Staub via ceph-users <ceph-users(a)ceph.io>: > >> Not recently, but a few years ago, I used the "Whole host >> replacement" using "swap-bucket": >> https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/operation… >> >> This worked really nicely. With source and destination about the >> same host CRUSH weight, most PGs are moved directly from source to >> destination. So also the copy speed was more or less the same most >> of the time. >> >> Cheers, >> Harry >> >>> On 4/28/26 11:08, Denis Polom via ceph-users wrote: >>> you can control max_backfills and some others if you set >>> osd_mclock_override_recovery_settings to true >>> >>> On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>>>> Another approach is to set the flags, destroy the OSDs on a >>>>> single host, redeploy them, set your backfill/recovery throttles >>>>> down to 1, and unset the flags. >>>> This is harder with mclock right? With wpq you can still control >>>> max_backfills, but not with mclock unless I am mistaken. >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Wannes Smet 29 Apr '26

29 Apr '26

Does swap-bucket use upmap? If so, wouldn't that mean that a lot of PGs no longer end up where CRUSH would have placed them? ________________________________ From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, April 28, 2026 21:40 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Best "modern" approach to drain/fill a OSD server +1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd +1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) and then let Ceph recover. But I generally don't recommend that to unexperienced operators. Zitat von Harald Staub via ceph-users <ceph-users(a)ceph.io>: > Not recently, but a few years ago, I used the "Whole host > replacement" using "swap-bucket": > https://urldefense.com/v3/__https://docs.ceph.com/en/latest/rados/operation… > > This worked really nicely. With source and destination about the > same host CRUSH weight, most PGs are moved directly from source to > destination. So also the copy speed was more or less the same most > of the time. > > Cheers, > Harry > > On 4/28/26 11:08, Denis Polom via ceph-users wrote: >> you can control max_backfills and some others if you set >> osd_mclock_override_recovery_settings to true >> >> On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>>> Another approach is to set the flags, destroy the OSDs on a >>>> single host, redeploy them, set your backfill/recovery throttles >>>> down to 1, and unset the flags. >>> This is harder with mclock right? With wpq you can still control >>> max_backfills, but not with mclock unless I am mistaken. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Eugen Block 28 Apr '26

28 Apr '26

+1 for swap-bucket, we've been using that as well successfully. What we also sometimes do, especially when time is a bit more critical, is to purge OSDs of an entire host (we also did this with two hosts at the same time due to high resiliency) and then let Ceph recover. But I generally don't recommend that to unexperienced operators. Zitat von Harald Staub via ceph-users <ceph-users(a)ceph.io>: > Not recently, but a few years ago, I used the "Whole host > replacement" using "swap-bucket": > https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/ > > This worked really nicely. With source and destination about the > same host CRUSH weight, most PGs are moved directly from source to > destination. So also the copy speed was more or less the same most > of the time. > > Cheers, > Harry > > On 4/28/26 11:08, Denis Polom via ceph-users wrote: >> you can control max_backfills and some others if you set >> osd_mclock_override_recovery_settings to true >> >> On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>>> Another approach is to set the flags, destroy the OSDs on a >>>> single host, redeploy them, set your backfill/recovery throttles >>>> down to 1, and unset the flags. >>> This is harder with mclock right? With wpq you can still control >>> max_backfills, but not with mclock unless I am mistaken. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Harald Staub 28 Apr '26

28 Apr '26

Not recently, but a few years ago, I used the "Whole host replacement" using "swap-bucket": https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/ This worked really nicely. With source and destination about the same host CRUSH weight, most PGs are moved directly from source to destination. So also the copy speed was more or less the same most of the time. Cheers, Harry On 4/28/26 11:08, Denis Polom via ceph-users wrote: > you can control max_backfills and some others if you set > osd_mclock_override_recovery_settings to true > > On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >>> Another approach is to set the flags, destroy the OSDs on a single >>> host, redeploy them, set your backfill/recovery throttles down to 1, >>> and unset the flags. >> This is harder with mclock right? With wpq you can still control >> max_backfills, but not with mclock unless I am mistaken. >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Dan van der Ster 28 Apr '26

28 Apr '26

Hello! I'm glad to see that this info is still proving useful to the community. Here's a pretty comprehensive document explaining the typical upmap-remapped workflow: https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ If it can be useful, I'm thinking of organizing an OSD balancing happy hour call in the next month or so. Maybe mid-May or so on a Friday afternoon (EU time) / morning (US time). Thanks! Dan -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster(a)clyso.com On Mon, Apr 27, 2026 at 7:11 PM Iztok Gregori via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi to all! > > I'm searching the wisdom of the community on how to drain/fill an OSD > server (like for removal, bulk disk replacing, filestore to bluestore > conversion...) without impacting on the performance of the cluster. > > We are using the 'ceph-gentle-reweight' script, which is running fine, > but we are interested to know if there is a newer approach for this job. > > Searching the web I found a pdf of the talk of Dan van der Ster about > "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about > using the 'balancer' in 'upmap' mode (and a "magic" script) to move the > PGs from/to the OSDs in the background. > > So the balancer upmap approach can be used to drain a node following > this procedure: > - set balancer tunables (like 'target_max_misplaced_ratio') > - activate the balancer in upmap mode (if not already active) > - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain > - run the magic script 'upmap-remapped.py' > - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain > - do stuff on the node (remove, reinstall etc etc). > > But what is the implication for the whole cluster, how this background > job affects its performance? > > I assume that 'target_max_misplaced_ratio' is a key parameter to avoid > large data movement in a given time, there is more to look into? > > And there are other approaches for the job? > > Thanks a lot! > Iztok > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Best "modern" approach to drain/fill a OSD server
by Denis Polom 28 Apr '26

28 Apr '26

you can control max_backfills and some others if you set osd_mclock_override_recovery_settings to true On 4/28/26 10:57, Janne Johansson via ceph-users wrote: >> Another approach is to set the flags, destroy the OSDs on a single host, redeploy them, set your backfill/recovery throttles down to 1, and unset the flags. > This is harder with mclock right? With wpq you can still control > max_backfills, but not with mclock unless I am mistaken. >

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Janne Johansson 28 Apr '26

28 Apr '26

> >> I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into? > > > > Yes. Setting that lower than default can help avoid certain intermediate states that can result in a fairly-full cluster's OSDs going nearfull/backfillfull. > > The documentation say about 'target_max_misplaced_ratio' "...the > balancer will incrementally move a small fraction of unbalanced PGs in > order to improve distribution. This fraction will not exceed a certain > threshold that defaults to 5%...". does it mean that off all the PGs > that has to be moved out/in of the node(s), maximum 5% will be moved at > the same time? Max 5% (I usually set it lower) of the misplaced-but-upmapped PGs will be un-upmapped and go into "backfill" or "backfill_wait". Since 50% is kind of a large number on large clusters, this means a lot of the PGs will be in backfill_wait. This isn't a huge problem in terms of movement, since other factors (like max_backfill when combined with wpq scheduler at least) will limit how many transfers are being done, but having 12 backfilling and 120 backfill_wait will just mean those in backfill_wait will not see scrubbing done to them while waiting, so I usually set the max_misplaced to a smaller number like 0.1% for larger clusters or pools with large number of PGs, which still leaves enough PGs that actually does move, but not too many lingering in _wait state so they can get their scrubs done as expected. The balancer runs every minute or so, so if the _wait queue dips, it will only take some 30-60 seconds for the balancer to make sure 0.1% is again heading for movement. -- May the most significant bit of your life be positive.

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Janne Johansson 28 Apr '26

28 Apr '26

> Another approach is to set the flags, destroy the OSDs on a single host, redeploy them, set your backfill/recovery throttles down to 1, and unset the flags. This is harder with mclock right? With wpq you can still control max_backfills, but not with mclock unless I am mistaken. -- May the most significant bit of your life be positive.

1 0

Re: Attempt to enable mgmt-gateway on v20.2.1 failed, and now cephadm module is broken
by Redouane Kachach 28 Apr '26

28 Apr '26

Hi Steve, Can you plz share the spec you used to configure the mgmt-gateway and the active mgr logs? Best, Redo. On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < ceph-users(a)ceph.io> wrote: > I have a pre-production cluster running on Ubuntu 22.04 hosts using > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which > went completely smoothly. We are a government facility so we are required > to enable FIPS mode in the OS. I believe this last fact may be the cause of > the following: > > I wanted to enable mgmt-gateway so I made a service definition file to > enable it and applied it. It appeared to work, but then “ceph health > detail” showed: > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > The mgmt-gateway service was started on all 3 nodes however. > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (4m) 3m ago 4m > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > I then attempted to remove the service definition with ceph orch rm, but > it only succeeded in removing it on 1 of the 3 nodes. The process is still > running on the other 2, and the warning is still present. “ceph mgr fail && > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > comes right back, and the process is still present on 2 nodes (as shown by > “ceph orch ps”). > > $ ceph mgr fail && ceph orch rm mgmt-gateway > Removed service mgmt-gateway > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (15h) 15h ago 15h > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > $ ceph health detail > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > Despite the 2 processes running, the configured service IP does not > respond to a ping or requests from curl. > > Since we were getting ready to move this cluster into production I decided > to look for help here rather than proceed experimenting further and risk > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > (b) recover the cephadm module to get back to normal with no warnings; and > less importantly, (c) is there a way to get mgmt-gateway working in this > environment? A little googling and AI querying suggests it may be an issue > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > up my own proxy; I was just trying to use the built-in tools where > possible.) > > Thanks in advance. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 2