There have been threads on exactly this. Might depend a bit on your ceph version. We are
running mimic and have no issues doing:
- set noout, norebalance, nobackfill
- add all OSDs (with weight 1)
- wait for peering to complete
- unset all flags and let the rebalance loose
Starting with nautilus there seem to be issues with this procedure. Mainly the peering
phase can cause a collapse of the cluster. In your case, it sounds like you added the
OSDs already. You should be able to do relatively safely:
- set noout, norebalance, nobackfill
- set weight of OSDs to 1 one by one and wait for peering to complete every time
- unset all flags and let the rebalance loose
I believe once the peering succeeded without crashes, the rebalancing will just work fine.
You can easily control how much rebalancing is going on.
I noted that ceph seems to have a strange concept of priority though. I needed to gain
capacity by adding OSDs and ceph was very consequent with moving PGs from the fullest OSDs
last. The opposite of what should happen. Thus, it took ages for additional capacity to
become available and also the backfill too full warnings stayed for all the time. You can
influence this to some degree by using force_recovery commands on PGs on the fullest
OSDs.
Best regards and good luck,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Kristof Coucke <kristof.coucke(a)gmail.com>
Sent: 21 October 2020 13:29:00
To: ceph-users(a)ceph.io
Subject: [ceph-users] Question about expansion existing Ceph cluster - adding OSDs
Hi,
I have a cluster with 182 OSDs, this has been expanded towards 282 OSDs.
Some disks were near full.
The new disks have been added with initial weight = 0.
The original plan was to increase this slowly towards their full weight
using the gentle reweight script. However, this is going way too slow and
I'm also having issues now with "backfill_toofull".
Can I just add all the OSDs with their full weight, or will I get a lot of
issues when I'm doing that?
I know that a lot of PGs will have to be replaced, but increasing the
weight slowly will take a year at the current speed. I'm already playing
with the max backfill to increase the speed, but every time I increase the
weight it will take a lot of time again...
I can face the fact that there will be a performance decrease.
Looking forward to your comments!
Regards,
Kristof
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io