Hi,
I would be nice if we could just copy the content to the new drive and go from there.
that's exactly what we usually do, we add a new drive and 'pvmove' the contents of the failing drive. The worst thing so far is that the orchestrator still thinks it's /dev/sd{previous_letter}, but I don't care too much about that. Note that we don't use the --all-available-devices to have full control over OSD deployment. But you could pause the orchestrator to prevent it from building a new OSD on that drive until the pvmove finished. Regards, Eugen Zitat von Anthony D'Atri <anthony.datri@gmail.com>:
1. Pulled failed drive ( after troubleshooting of course )
2. Cephadm gui - find OSD, purge osd 3. Wait for rebalance 4. Insert new drive ( let cluster rebalance after it automatically adds the drive as an OSD ) ( yes, we have auto-add on in the clusters )
I imagine with an existing good drive, we would use delete instead of purge, but the process would be the similar, except the drive swap would happen after the data was moved.
You don’t have to wait for the rebalance / backfill / recovery, at least if you do one drive (or failure domain) at a time.
In fact you can be more efficient by not waiting, as deploying the new OSD will short-circuit some of that data movement from the deletion.
Would the replace flag ( or keep OSD option in gui ) allow us to avoid the initial rebalance by backfilling the new drive with the old drives content?
Only if you set the new drive’s CRUSH weight artificially low, to match the old drive’s weight exactly. But when you weight it up fully, data will move anyway.
I would be nice if we could just copy the content to the new drive and go from there.
I get your drift, but there’s a nuance. Because of how CRUSH works, the data that the 20TB OSD will eventually hold will not be a proper superset of what’s on the 4TB OSD today. Data will also shuffle on other OSDs.
Be careful that you only delete / destroy OSDs in a single failure domain at a time, and wait for full recovery before proceeding to the next failure domain. If you are short on capacity, you might want to do a small number of drives in one failure domain, wait for recovery, then move to the next failure domain, as you will only realize additional cluster capacity once you’ve added CRUSH weight to at least 3 failure domains.
We would like to avoid lots of read/write cluster recovery activity if possible sine we could be replacing 40+ drives in each cluster.
Part of the false economy of HDDs :-/. But again, attend to your failure domains. If you are only replacing *some* off the smaller drives, spread that across failure domains or you won’t gain any actual capacity. And be prepared for the larger drives to get a proportionally larger fraction of workload, which can be somewhat mitigated with primary affinity but that’s a bit of an advanced topic.
Related advice: as you add OSDs that are 5x the size of existing OSDs, you run the risk of hitting mon_max_pg_per_osd on the larger OSDs. This defaults to 250. I suggest setting it to 1000 before starting this project, to avoid larger OSDs that won’t activate.
https://github.com/ceph/ceph/pull/60492
Also, you might temporarily disable the balancer and use pgremapper or https://gitlab.cern.ch/ceph/ceph-scripts/blob/master/tools/upmap/upmap-remap... to minimize extra backfill and control its rate. You would use the tool to effective freeze all PG mappings, then destroy / redeploy as many OSDs as you like *within a single failure domain*, and gradually remove the manual upmaps at a rate informed by how much backfill your spinners can handle. There are lots of articles and list posts about this strategy. This lets you leapfrog the transient churn as multiple OSDs are removed / added, and control the thundering herd of recovery that can DoS spinners.
If you’re using the pg autoscaler, I might ensure that all affected pools have the ‘bulk’ flag set in advance, so that you don’t have PG splitting / merging and backfill/recovery going on at the same time.
US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds
UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2mds
US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds
I suspect FWIW that you would benefit from running an RGW on more servers - any of them that have enough CPU / RAM.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io