On 27/04/26 16:46, Anthony D'Atri wrote:
Hi to all!
I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion...) without impacting on the performance of the cluster.
You'll impact performance no matter what you do, but to differing degrees.
The least impact is making the least change:
set noout destroy one OSD redeploy that OSD unset noout wait for recovery
At scale that doesn't, well, scale.
We are using the 'ceph-gentle-reweight' script, which is running fine, but we are interested to know if there is a newer approach for this job.
What Ceph release are you running? That approach works, but will take a long time and you need to have enough free capacity on other nodes to hold that node's share of data.
We are running Quincy (on our upgrade path to Squid/Tentacle) and we have enough free space to drain at least one node at the time. We used successfully the gentle-reweight a lot of times, but we are interested to know if there are other, more efficient, ways to drain/fill a node (or group of nodes).
Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in 'upmap' mode
If you're running, say, Mimic or later, you typically would already have the balancer on; upmap mode is the default.
Run
ceph balancer status
When using upmaps, you want to avoid "ceph osd reweight-by-utilization" and ensure that the REWEIGHT column of "ceph osd df" or "ceph osd tree" is all 1.0000
Sure, balancer is already active (in upmap mode) and is doing the job of balancing :-)
(and a "magic" script) to move the PGs from/to the OSDs in the background.
So the balancer upmap approach can be used to drain a node following this procedure:
Be sure to disable backfill and rebalancing, you want no backfilling in progress when you run upmap-remapped.py
I realized (after I sent the email) that you need to have 'norebalancing' active before doinng the upmap-remapped.py, but I didn't thought about 'nobackfilling', so thank you!
- set balancer tunables (like 'target_max_misplaced_ratio') - activate the balancer in upmap mode (if not already active) - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain - run the magic script 'upmap-remapped.py' - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain - do stuff on the node (remove, reinstall etc etc).
If you are solving for safety over time, sure. Especially if you have replicated size=2 pools or EC N+1.
Another approach is to set the flags, destroy the OSDs on a single host, redeploy them, set your backfill/recovery throttles down to 1, and unset the flags. This will be somewhat more risky as you're reducing redundancy until the recovery completes, and more will happen in parallel. Whether you can tolerate that depends in part on your hardware, CRUSH topology, how many hosts, replication vs EC, etc.
Our first approach was prioritize time over safety, but because we overran our maintenance window we switched to safety over time (all the pools in this cluster are replica 3).
But what is the implication for the whole cluster, how this background job affects its performance?
We need more info.
I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into?
Yes. Setting that lower than default can help avoid certain intermediate states that can result in a fairly-full cluster's OSDs going nearfull/backfillfull.
The documentation say about 'target_max_misplaced_ratio' "...the balancer will incrementally move a small fraction of unbalanced PGs in order to improve distribution. This fraction will not exceed a certain threshold that defaults to 5%...". does it mean that off all the PGs that has to be moved out/in of the node(s), maximum 5% will be moved at the same time? Thanks for your help! Iztok