Personally, when adding drives like this, I set noin (ceph osd set noin), and norebalance (ceph osd set norebalance). Like your situation, we run smaller clusters; our largest cluster only has 18 OSDs. That keeps the cluster from starting data moves until all new drives are in place. Don't forget to unset these values (ceph osd unset noin, ceph osd unset norebalance). There are also values you can tune to control whether user traffic or recovery traffic gets precedent while data is moving. Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Kai Börnert [mailto:kai.boernert@posteo.de] Sent: Tuesday, June 15, 2021 8:20 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Strategy for add new osds Hi, as far as I understand it, you get no real benefit with doing them one by one, as each osd add, can cause a lot of data to be moved to a different osd, even tho you just rebalanced it. The algorithm determining the placement of pg's does not take the current/historic placement into account, so changing anything at this, could cause any amount of data to migrate, with each change Greetings, Kai On 6/15/21 5:06 PM, Jorge JP wrote:

Hello,

I have a ceph cluster with 5 nodes (1 hdd each node). I want to add 5 more drives (hdd) to expand my cluster. What is the best strategy for this?

I will add each drive in each node but is a good strategy add one drive and wait to rebalance the data to new osd for add new osd? or maybe.. I should be add the 5 drives without wait rebalancing and ceph rebalancing the data to all new osd?

Thank you. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I've to say I am reading quite some interesting strategies in this thread and I'd like to shortly take the time to compare them: 1) one by one osd adding - least amount of pg rebalance - will potentially re-re-balance data that has just been distributed with the next OSD phase in - limits the impact if you have a bug in the hdd/ssd series The biggest problem with this approach is that you will re-re-re-balance data over and over again and that will slowdown the process significantly. 2) reweighted phase in - Starting slow with reweighting to a small amount of its potential - Allows to see how the new OSD performs - Needs manual interaction for growing - delays the phase in possibly for "longer than necessary" We use this approach when phasing in multiple, larger OSDs that are from a newer / not so well known series of disks. 3) noin / norebalance based phase in - Interesting approach to delay rebalancing until the "proper/final" new storage is in place - Unclear how much of a difference it makes if you insert the new set of osds within a short timeframe (i.e. adding 1 osd at minute 0, 2nd at minute 1, etc.) 4) All at once / randomly - Least amount of manual tuning - In a way something one "would expect" ceph to do right (but in practice doesn't all the time) - Might (likely) cause short term re-adjustments - Might cause client i/o slowdown (see next point) 5) General slowing down What we actually do in datacenterlight.ch is slowing down phase ins by default via the followign tunings: # Restrain recovery operations so that normal cluster is not affected [osd] osd max backfills = 1 osd recovery max active = 1 osd recovery op priority = 2 This works well in about 90% of the cases for us. Quite an interesting thread, thanks everyone for sharing! Cheers, Nico Anthony D'Atri <anthony.datri@gmail.com> writes:

...

Hi,

as far as I understand it,

you get no real benefit with doing them one by one, as each osd add, can cause a lot of data to be moved to a different osd, even tho you just rebalanced it.

Less than with older releases, but yeah.

I’ve known someone who advised against doing them in parallel because one would — for a time — have PGs with multiple remaps in the acting set. The objection may have been paranoia, I’m not sure.

One compromise is to upweight the new OSDs one node at a time, so the churn is limited to one failure domain at a time.

— aad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Sustainable and modern Infrastructures by ungleich.ch