Best way to add OSDs - whole node or one by one?
Hello, I am currently in the process of expanding my Nautilus cluster from 3 nodes (combined OSD/MGR/MON/MDS) to 6 OSD nodes and 3 management nodes. The old and new OSD nodes all have 8 x 12TB HDDs plus NVMe. The front and back networks are 10GB. Last Friday evening I injected a whole new OSD node, increasing the OSD HDDs from 24 to 32. As of this morning the cluster is still re-balancing - with periodic warnings about degraded PGs and missed deep-scrub deadlines. So after 4.5 days my misplaced PGs are down from 33% to 2%. My question: For a cluster of this size, what is the best-practice procedure for adding OSDs? Should I use 'ceph-volume prepare' to layout the new OSDs, but only add them a couple at a time, or should I continue adding whole nodes? Maybe this has to do with a maximum percentage of misplaced PGs. The first new node increased the OSD capacity by 33% and resulted in 33% PG misplacement. The next node will only result in 25% misplacement. If a too high percentage of misplaced PGs negatively impacts rebalancing or data availability, what is a reasonable ceiling for this percentage? Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)
I'm sure there is a "correct" way, but I think it mostly relates to how busy your cluster is, and how tolerant it is of the added load from the backfills. My current modus operandi is to set the noin, noout, nobackfill, norecover, and norebalance flags first. This makes sure that new OSDs don't come in, current OSDs don't go out, it doesn't start backfilling or try to rebalance (yet). Add all of my OSDs. Then unset noin and norebalance. In all of the new OSDs. Let it work out the new crush map so that data isn't constantly in motion moving back and forth as new OSD hosts are added. Inject osd_max_backfills and osd_recovery_max_active to 1 Then unset norecover and nobackfill and noout. Then it should slowly but surely chip away at recovery. During times of lighter load I can ratchet up the max backfills and recovery max actives to a higher level to chug through more of it while iops aren't being burned. I'm sure everyone has their own way, but I've been very comfortable with this approach over the last few years. NOTE: you probably want to make sure that the balancer and the pg_autoscaler are set to off during this, otherwise they might throw backfills on the pile and you will feel like you'll never reach the bottom. Reed
On Mar 10, 2021, at 9:55 AM, Dave Hall <kdhall@binghamton.edu> wrote:
Hello,
I am currently in the process of expanding my Nautilus cluster from 3 nodes (combined OSD/MGR/MON/MDS) to 6 OSD nodes and 3 management nodes. The old and new OSD nodes all have 8 x 12TB HDDs plus NVMe. The front and back networks are 10GB.
Last Friday evening I injected a whole new OSD node, increasing the OSD HDDs from 24 to 32. As of this morning the cluster is still re-balancing - with periodic warnings about degraded PGs and missed deep-scrub deadlines. So after 4.5 days my misplaced PGs are down from 33% to 2%.
My question: For a cluster of this size, what is the best-practice procedure for adding OSDs? Should I use 'ceph-volume prepare' to layout the new OSDs, but only add them a couple at a time, or should I continue adding whole nodes?
Maybe this has to do with a maximum percentage of misplaced PGs. The first new node increased the OSD capacity by 33% and resulted in 33% PG misplacement. The next node will only result in 25% misplacement. If a too high percentage of misplaced PGs negatively impacts rebalancing or data availability, what is a reasonable ceiling for this percentage?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Reed, Thank you. This seems like a very well thought approach. Your note about the balancer and the auto_scaler seem quite relevant as well. I'll give it a try when I add my next two nodes. -Dave -- Dave Hall Binghamton University On Thu, Mar 11, 2021 at 5:53 PM Reed Dier <reed.dier@focusvq.com> wrote:
I'm sure there is a "correct" way, but I think it mostly relates to how busy your cluster is, and how tolerant it is of the added load from the backfills.
My current modus operandi is to set the noin, noout, nobackfill, norecover, and norebalance flags first. This makes sure that new OSDs don't come in, current OSDs don't go out, it doesn't start backfilling or try to rebalance (yet).
Add all of my OSDs.
Then unset noin and norebalance. In all of the new OSDs. Let it work out the new crush map so that data isn't constantly in motion moving back and forth as new OSD hosts are added. Inject osd_max_backfills and osd_recovery_max_active to 1 Then unset norecover and nobackfill and noout.
Then it should slowly but surely chip away at recovery. During times of lighter load I can ratchet up the max backfills and recovery max actives to a higher level to chug through more of it while iops aren't being burned.
I'm sure everyone has their own way, but I've been very comfortable with this approach over the last few years.
NOTE: you probably want to make sure that the balancer and the pg_autoscaler are set to off during this, otherwise they might throw backfills on the pile and you will feel like you'll never reach the bottom.
Reed
On Mar 10, 2021, at 9:55 AM, Dave Hall <kdhall@binghamton.edu> wrote:
Hello,
I am currently in the process of expanding my Nautilus cluster from 3 nodes (combined OSD/MGR/MON/MDS) to 6 OSD nodes and 3 management nodes. The old and new OSD nodes all have 8 x 12TB HDDs plus NVMe. The front and back networks are 10GB.
Last Friday evening I injected a whole new OSD node, increasing the OSD HDDs from 24 to 32. As of this morning the cluster is still re-balancing - with periodic warnings about degraded PGs and missed deep-scrub deadlines. So after 4.5 days my misplaced PGs are down from 33% to 2%.
My question: For a cluster of this size, what is the best-practice procedure for adding OSDs? Should I use 'ceph-volume prepare' to layout the new OSDs, but only add them a couple at a time, or should I continue adding whole nodes?
Maybe this has to do with a maximum percentage of misplaced PGs. The first new node increased the OSD capacity by 33% and resulted in 33% PG misplacement. The next node will only result in 25% misplacement. If a too high percentage of misplaced PGs negatively impacts rebalancing or data availability, what is a reasonable ceiling for this percentage?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dave, Worth just looking at utilisation across your OSD’s. I’ve had Pgs get stuck in backfill-wait-too big when I’ve added new osds. Ceph was unable to move Pg around onto a smaller capacity osd that was quite full. I had to increase the number of pgs (and pg_num) for it to get sorted (and do some reweighting). Reeds plan is a good one. Because my setup has been in quite a state of flux recently, I’ve kept autoscale set to warn and the number of pgs higher than normal for the short term. Cheers A Sent from my iPhone On 12 Mar 2021, at 04:38, Dave Hall <kdhall@binghamton.edu> wrote: Reed, Thank you. This seems like a very well thought approach. Your note about the balancer and the auto_scaler seem quite relevant as well. I'll give it a try when I add my next two nodes. -Dave -- Dave Hall Binghamton University On Thu, Mar 11, 2021 at 5:53 PM Reed Dier <reed.dier@focusvq.com> wrote:
I'm sure there is a "correct" way, but I think it mostly relates to how busy your cluster is, and how tolerant it is of the added load from the backfills.
My current modus operandi is to set the noin, noout, nobackfill, norecover, and norebalance flags first. This makes sure that new OSDs don't come in, current OSDs don't go out, it doesn't start backfilling or try to rebalance (yet).
Add all of my OSDs.
Then unset noin and norebalance. In all of the new OSDs. Let it work out the new crush map so that data isn't constantly in motion moving back and forth as new OSD hosts are added. Inject osd_max_backfills and osd_recovery_max_active to 1 Then unset norecover and nobackfill and noout.
Then it should slowly but surely chip away at recovery. During times of lighter load I can ratchet up the max backfills and recovery max actives to a higher level to chug through more of it while iops aren't being burned.
I'm sure everyone has their own way, but I've been very comfortable with this approach over the last few years.
NOTE: you probably want to make sure that the balancer and the pg_autoscaler are set to off during this, otherwise they might throw backfills on the pile and you will feel like you'll never reach the bottom.
Reed
On Mar 10, 2021, at 9:55 AM, Dave Hall <kdhall@binghamton.edu> wrote:
Hello,
I am currently in the process of expanding my Nautilus cluster from 3 nodes (combined OSD/MGR/MON/MDS) to 6 OSD nodes and 3 management nodes. The old and new OSD nodes all have 8 x 12TB HDDs plus NVMe. The front and back networks are 10GB.
Last Friday evening I injected a whole new OSD node, increasing the OSD HDDs from 24 to 32. As of this morning the cluster is still re-balancing - with periodic warnings about degraded PGs and missed deep-scrub deadlines. So after 4.5 days my misplaced PGs are down from 33% to 2%.
My question: For a cluster of this size, what is the best-practice procedure for adding OSDs? Should I use 'ceph-volume prepare' to layout the new OSDs, but only add them a couple at a time, or should I continue adding whole nodes?
Maybe this has to do with a maximum percentage of misplaced PGs. The first new node increased the OSD capacity by 33% and resulted in 33% PG misplacement. The next node will only result in 25% misplacement. If a too high percentage of misplaced PGs negatively impacts rebalancing or data availability, what is a reasonable ceiling for this percentage?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I thought the balancer and pg_autoscaler are only doing something if all the PG's are in active+clean state? So if there is any backfilling going around it just bails out. Or did you mean during the norecover/nobackfill/noout phase? Kind regards, Caspar Op do 11 mrt. 2021 om 23:54 schreef Reed Dier <reed.dier@focusvq.com>:
I'm sure there is a "correct" way, but I think it mostly relates to how busy your cluster is, and how tolerant it is of the added load from the backfills.
My current modus operandi is to set the noin, noout, nobackfill, norecover, and norebalance flags first. This makes sure that new OSDs don't come in, current OSDs don't go out, it doesn't start backfilling or try to rebalance (yet).
Add all of my OSDs.
Then unset noin and norebalance. In all of the new OSDs. Let it work out the new crush map so that data isn't constantly in motion moving back and forth as new OSD hosts are added. Inject osd_max_backfills and osd_recovery_max_active to 1 Then unset norecover and nobackfill and noout.
Then it should slowly but surely chip away at recovery. During times of lighter load I can ratchet up the max backfills and recovery max actives to a higher level to chug through more of it while iops aren't being burned.
I'm sure everyone has their own way, but I've been very comfortable with this approach over the last few years.
NOTE: you probably want to make sure that the balancer and the pg_autoscaler are set to off during this, otherwise they might throw backfills on the pile and you will feel like you'll never reach the bottom.
Reed
On Mar 10, 2021, at 9:55 AM, Dave Hall <kdhall@binghamton.edu> wrote:
Hello,
I am currently in the process of expanding my Nautilus cluster from 3 nodes (combined OSD/MGR/MON/MDS) to 6 OSD nodes and 3 management nodes. The old and new OSD nodes all have 8 x 12TB HDDs plus NVMe. The front and back networks are 10GB.
Last Friday evening I injected a whole new OSD node, increasing the OSD HDDs from 24 to 32. As of this morning the cluster is still re-balancing - with periodic warnings about degraded PGs and missed deep-scrub deadlines. So after 4.5 days my misplaced PGs are down from 33% to 2%.
My question: For a cluster of this size, what is the best-practice procedure for adding OSDs? Should I use 'ceph-volume prepare' to layout the new OSDs, but only add them a couple at a time, or should I continue adding whole nodes?
Maybe this has to do with a maximum percentage of misplaced PGs. The first new node increased the OSD capacity by 33% and resulted in 33% PG misplacement. The next node will only result in 25% misplacement. If a too high percentage of misplaced PGs negatively impacts rebalancing or data availability, what is a reasonable ceiling for this percentage?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
They shouldn't, but you can have cases where you will have OSDs/pools that may be active+clean, and it will trip a rebalance, etc while other pools/OSDs are still backfilling, potentially further implicating the backfilling OSDs. It may be better now, but peace of mind from knowing I won't be chasing a balancer/autoscaler ghost is easy enough by disabling for the expansion, and then re-enable after fully backfilled. But also helpful during the initial phase, so as to keep things from starting movement before it will get redirected in short time. Reed
On Mar 15, 2021, at 2:57 AM, Caspar Smit <casparsmit@supernas.eu> wrote:
Hi,
I thought the balancer and pg_autoscaler are only doing something if all the PG's are in active+clean state? So if there is any backfilling going around it just bails out.
Or did you mean during the norecover/nobackfill/noout phase?
Kind regards, Caspar
Op do 11 mrt. 2021 om 23:54 schreef Reed Dier <reed.dier@focusvq.com>:
I'm sure there is a "correct" way, but I think it mostly relates to how busy your cluster is, and how tolerant it is of the added load from the backfills.
My current modus operandi is to set the noin, noout, nobackfill, norecover, and norebalance flags first. This makes sure that new OSDs don't come in, current OSDs don't go out, it doesn't start backfilling or try to rebalance (yet).
Add all of my OSDs.
Then unset noin and norebalance. In all of the new OSDs. Let it work out the new crush map so that data isn't constantly in motion moving back and forth as new OSD hosts are added. Inject osd_max_backfills and osd_recovery_max_active to 1 Then unset norecover and nobackfill and noout.
Then it should slowly but surely chip away at recovery. During times of lighter load I can ratchet up the max backfills and recovery max actives to a higher level to chug through more of it while iops aren't being burned.
I'm sure everyone has their own way, but I've been very comfortable with this approach over the last few years.
NOTE: you probably want to make sure that the balancer and the pg_autoscaler are set to off during this, otherwise they might throw backfills on the pile and you will feel like you'll never reach the bottom.
Reed
On Mar 10, 2021, at 9:55 AM, Dave Hall <kdhall@binghamton.edu> wrote:
Hello,
I am currently in the process of expanding my Nautilus cluster from 3 nodes (combined OSD/MGR/MON/MDS) to 6 OSD nodes and 3 management nodes. The old and new OSD nodes all have 8 x 12TB HDDs plus NVMe. The front and back networks are 10GB.
Last Friday evening I injected a whole new OSD node, increasing the OSD HDDs from 24 to 32. As of this morning the cluster is still re-balancing - with periodic warnings about degraded PGs and missed deep-scrub deadlines. So after 4.5 days my misplaced PGs are down from 33% to 2%.
My question: For a cluster of this size, what is the best-practice procedure for adding OSDs? Should I use 'ceph-volume prepare' to layout the new OSDs, but only add them a couple at a time, or should I continue adding whole nodes?
Maybe this has to do with a maximum percentage of misplaced PGs. The first new node increased the OSD capacity by 33% and resulted in 33% PG misplacement. The next node will only result in 25% misplacement. If a too high percentage of misplaced PGs negatively impacts rebalancing or data availability, what is a reasonable ceiling for this percentage?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Andrew Walker-Brown
-
Caspar Smit
-
Dave Hall
-
Reed Dier