Hi to all!
I'm searching the wisdom of the community on how to drain/fill an OSD server (like for removal, bulk disk replacing, filestore to bluestore conversion...) without impacting on the performance of the cluster.
You'll impact performance no matter what you do, but to differing degrees. The least impact is making the least change: set noout destroy one OSD redeploy that OSD unset noout wait for recovery At scale that doesn't, well, scale.
We are using the 'ceph-gentle-reweight' script, which is running fine, but we are interested to know if there is a newer approach for this job.
What Ceph release are you running? That approach works, but will take a long time and you need to have enough free capacity on other nodes to hold that node's share of data.
Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in 'upmap' mode
If you're running, say, Mimic or later, you typically would already have the balancer on; upmap mode is the default. Run ceph balancer status When using upmaps, you want to avoid "ceph osd reweight-by-utilization" and ensure that the REWEIGHT column of "ceph osd df" or "ceph osd tree" is all 1.0000
(and a "magic" script) to move the PGs from/to the OSDs in the background.
So the balancer upmap approach can be used to drain a node following this procedure:
Be sure to disable backfill and rebalancing, you want no backfilling in progress when you run upmap-remapped.py
- set balancer tunables (like 'target_max_misplaced_ratio') - activate the balancer in upmap mode (if not already active) - 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain - run the magic script 'upmap-remapped.py' - 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain - do stuff on the node (remove, reinstall etc etc).
If you are solving for safety over time, sure. Especially if you have replicated size=2 pools or EC N+1. Another approach is to set the flags, destroy the OSDs on a single host, redeploy them, set your backfill/recovery throttles down to 1, and unset the flags. This will be somewhat more risky as you're reducing redundancy until the recovery completes, and more will happen in parallel. Whether you can tolerate that depends in part on your hardware, CRUSH topology, how many hosts, replication vs EC, etc.
But what is the implication for the whole cluster, how this background job affects its performance?
We need more info.
I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large data movement in a given time, there is more to look into?
Yes. Setting that lower than default can help avoid certain intermediate states that can result in a fairly-full cluster's OSDs going nearfull/backfillfull.
And there are other approaches for the job?
Thanks a lot! Iztok
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io