Hello, I am currently managing a Ceph cluster that consists of 3 racks, each with 4 OSD nodes. Each node contains 24 OSDs. I plan to add three new nodes, one to each rack, to help alleviate the high OSD utilization. The current highest OSD utilization is 85%. I am concerned about the possibility of any OSD reaching the osd_full_ratio threshold during the rebalancing process. This would cause the cluster to enter a read-only state, which I want to avoid at all costs. I am planning to execute the following commands: ceph orch host add new-node-1 ceph orch host add new-node-2 ceph orch host add new-node-3 ceph osd crush move new-node-1 rack=rack-1 ceph osd crush move new-node-2 rack=rack-2 ceph osd crush move new-node-3 rack=rack-3 ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 1 ceph config set osd osd_recovery_sleep 0.1 ceph orch apply osd --all-available-devices Before proceeding, I would like to ask if the above steps are safe to execute in a cluster with such high utilization. My main concern is whether the rebalancing could cause any OSD to exceed the osd_full_ratio and result in unexpected failures. Any insights or advice on how to safely add these nodes without impacting cluster stability would be greatly appreciated. Thanks!
Hi Sinan, The safest approach would be to use the upmap-remapped.py tool developed by Dan at CERN. See [1] for details. The idea is to leverage the upmap load balancer to progressively migrate the data to the new servers, minimizing performance impact on the cluster and clients. I like to create the OSDs ahead of time on the nodes that I initially place in a root directory called ‘closet’. I then apply the norebalance flag (ceph osd set norebalance), disable the balancer (ceph balancer off), move the new nodes with already provisioned OSDs to their final location (rack), run ./upmap-remapped.py to bring all PGs back to active+clean state, remove the norebalance flag (ceph osd unset norebalance), re-enable the balancer (ceph balancer on) and watch data moving progressively as the upmap balancer executes its plans. Regards, Frédéric. [1] https://docs.clyso.com/blog/adding-capacity-with-upmap-remapped/ ----- Le 17 Mar 25, à 17:51, Sinan Polat sinan86polat@gmail.com a écrit :
Hello,
I am currently managing a Ceph cluster that consists of 3 racks, each with 4 OSD nodes. Each node contains 24 OSDs. I plan to add three new nodes, one to each rack, to help alleviate the high OSD utilization.
The current highest OSD utilization is 85%. I am concerned about the possibility of any OSD reaching the osd_full_ratio threshold during the rebalancing process. This would cause the cluster to enter a read-only state, which I want to avoid at all costs.
I am planning to execute the following commands:
ceph orch host add new-node-1 ceph orch host add new-node-2 ceph orch host add new-node-3
ceph osd crush move new-node-1 rack=rack-1 ceph osd crush move new-node-2 rack=rack-2 ceph osd crush move new-node-3 rack=rack-3
ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 1 ceph config set osd osd_recovery_sleep 0.1
ceph orch apply osd --all-available-devices
Before proceeding, I would like to ask if the above steps are safe to execute in a cluster with such high utilization. My main concern is whether the rebalancing could cause any OSD to exceed the osd_full_ratio and result in unexpected failures.
Any insights or advice on how to safely add these nodes without impacting cluster stability would be greatly appreciated.
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Sinan, Agree on the safe approach to use upmap-remapped.py tool - it can help to reduce the unwanted data movement when new nodes are added. However since these are new nodes being added and not old ones removed/swapped - I suspect not much data movement going above the thresholds. In case you reach those thresholds you can modify them with the following commands and add yourself some headroom: ceph osd set-nearfull-ratio .85 ceph osd set-backfillfull-ratio .90 ceph osd set-full-ratio .95 Increase the values by a small inch but make sure you tune them back down once rebalancing completes. This will allow the cluster to continue with backfilling and I/O requests. Set-nearfull-ration is purely cosmetic but should not be ignored. You can also disable the balancer and use this tool: https://github.com/laimis9133/plankton-swarm Just move a few pgs from the most full OSDs to less full ones manually. This will also give you some headroom for action. Best, Laimis J.
On 18 Mar 2025, at 16:59, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Sinan,
The safest approach would be to use the upmap-remapped.py tool developed by Dan at CERN. See [1] for details.
The idea is to leverage the upmap load balancer to progressively migrate the data to the new servers, minimizing performance impact on the cluster and clients. I like to create the OSDs ahead of time on the nodes that I initially place in a root directory called ‘closet’.
I then apply the norebalance flag (ceph osd set norebalance), disable the balancer (ceph balancer off), move the new nodes with already provisioned OSDs to their final location (rack), run ./upmap-remapped.py to bring all PGs back to active+clean state, remove the norebalance flag (ceph osd unset norebalance), re-enable the balancer (ceph balancer on) and watch data moving progressively as the upmap balancer executes its plans.
Regards, Frédéric.
----- Le 17 Mar 25, à 17:51, Sinan Polat sinan86polat@gmail.com a écrit :
Hello,
I am currently managing a Ceph cluster that consists of 3 racks, each with 4 OSD nodes. Each node contains 24 OSDs. I plan to add three new nodes, one to each rack, to help alleviate the high OSD utilization.
The current highest OSD utilization is 85%. I am concerned about the possibility of any OSD reaching the osd_full_ratio threshold during the rebalancing process. This would cause the cluster to enter a read-only state, which I want to avoid at all costs.
I am planning to execute the following commands:
ceph orch host add new-node-1 ceph orch host add new-node-2 ceph orch host add new-node-3
ceph osd crush move new-node-1 rack=rack-1 ceph osd crush move new-node-2 rack=rack-2 ceph osd crush move new-node-3 rack=rack-3
ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 1 ceph config set osd osd_recovery_sleep 0.1
ceph orch apply osd --all-available-devices
Before proceeding, I would like to ask if the above steps are safe to execute in a cluster with such high utilization. My main concern is whether the rebalancing could cause any OSD to exceed the osd_full_ratio and result in unexpected failures.
Any insights or advice on how to safely add these nodes without impacting cluster stability would be greatly appreciated.
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The safest approach would be to use the upmap-remapped.py tool developed by Dan at CERN. See [1] for details.
The idea is to leverage the upmap load balancer to progressively migrate the data to the new servers, minimizing performance impact on the cluster and clients. I like to create the OSDs ahead of time on the nodes that I initially place in a root directory called ‘closet’.
I then apply the norebalance flag (ceph osd set norebalance), disable the balancer (ceph balancer off), move the new nodes with already provisioned OSDs to their final location (rack), run ./upmap-remapped.py to bring all PGs back to active+clean state, remove the norebalance flag (ceph osd unset norebalance), re-enable the balancer (ceph balancer on) and watch data moving progressively as the upmap balancer executes its plans.
We do exactly that also, sometimes using pgremapper instead of upmap-remapper.py, but the effect is the same. Make the changes with norebalance, upmap the PGs to be happy where they are until we unset norebalance and let the ceph balancer correct it X% at a time. -- May the most significant bit of your life be positive.
Hi. Over the past few days, we've been working on migrating data from machine A to machine B using the pgremapper tool, but we haven’t been able to achieve the expected results. As part of our testing, we set up a small Ceph cluster with several monitors, managers, and servers with OSDs.We applied the flags noout, nobackfill, norecovery, and norebalance, and then added additional servers with OSDs. While Ceph did allocate PG replicas to the newly added OSDs, the actual data didn’t move due to the active flags. We then attempted to use pgremapper to migrate all PGs from one server to the new one, removing or negating the flags in the process. However, we frequently failed to complete the migration of all data/PGs. Are we overlooking something? Does anyone have a reliable, step-by-step procedure we can follow to perform this correctly? Any help would be greatly appreciated! Michal On 3/19/25 08:13, Janne Johansson wrote:
The safest approach would be to use the upmap-remapped.py tool developed by Dan at CERN. See [1] for details.
The idea is to leverage the upmap load balancer to progressively migrate the data to the new servers, minimizing performance impact on the cluster and clients. I like to create the OSDs ahead of time on the nodes that I initially place in a root directory called ‘closet’.
I then apply the norebalance flag (ceph osd set norebalance), disable the balancer (ceph balancer off), move the new nodes with already provisioned OSDs to their final location (rack), run ./upmap-remapped.py to bring all PGs back to active+clean state, remove the norebalance flag (ceph osd unset norebalance), re-enable the balancer (ceph balancer on) and watch data moving progressively as the upmap balancer executes its plans.
We do exactly that also, sometimes using pgremapper instead of upmap-remapper.py, but the effect is the same. Make the changes with norebalance, upmap the PGs to be happy where they are until we unset norebalance and let the ceph balancer correct it X% at a time.
Michal; Is the balancer (upmap balancer) running (ceph balancer status) and undoing some of the upmaps the pgremapper is introducing? Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Sat, Jun 21, 2025 at 2:05 AM Michal Strnad <michal.strnad@cesnet.cz> wrote:
Hi.
Over the past few days, we've been working on migrating data from machine A to machine B using the pgremapper tool, but we haven’t been able to achieve the expected results.
As part of our testing, we set up a small Ceph cluster with several monitors, managers, and servers with OSDs.We applied the flags noout, nobackfill, norecovery, and norebalance, and then added additional servers with OSDs. While Ceph did allocate PG replicas to the newly added OSDs, the actual data didn’t move due to the active flags. We then attempted to use pgremapper to migrate all PGs from one server to the new one, removing or negating the flags in the process. However, we frequently failed to complete the migration of all data/PGs. Are we overlooking something? Does anyone have a reliable, step-by-step procedure we can follow to perform this correctly?
Any help would be greatly appreciated!
Michal
The safest approach would be to use the upmap-remapped.py tool developed by Dan at CERN. See [1] for details.
The idea is to leverage the upmap load balancer to progressively migrate the data to the new servers, minimizing performance impact on the cluster and clients. I like to create the OSDs ahead of time on the nodes
I then apply the norebalance flag (ceph osd set norebalance), disable
On 3/19/25 08:13, Janne Johansson wrote: that I initially place in a root directory called ‘closet’. the balancer (ceph balancer off), move the new nodes with already provisioned OSDs to their final location (rack), run ./upmap-remapped.py to bring all PGs back to active+clean state, remove the norebalance flag (ceph osd unset norebalance), re-enable the balancer (ceph balancer on) and watch data moving progressively as the upmap balancer executes its plans.
We do exactly that also, sometimes using pgremapper instead of upmap-remapper.py, but the effect is the same. Make the changes with norebalance, upmap the PGs to be happy where they are until we unset norebalance and let the ceph balancer correct it X% at a time.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi. Thank you for your reply The balancer is off, so it shouldn't be interfering with that. Do you happen to have a list of steps (maybe commands from the history) on how you launched it? We're trying to figure out what we're missing. Regards, Michal On 6/21/25 15:21, Wesley Dillingham wrote:
Michal;
Is the balancer (upmap balancer) running (ceph balancer status) and undoing some of the upmaps the pgremapper is introducing?
Respectfully,
*Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com
On Sat, Jun 21, 2025 at 2:05 AM Michal Strnad <michal.strnad@cesnet.cz> wrote:
Hi.
Over the past few days, we've been working on migrating data from machine A to machine B using the pgremapper tool, but we haven’t been able to achieve the expected results.
As part of our testing, we set up a small Ceph cluster with several monitors, managers, and servers with OSDs.We applied the flags noout, nobackfill, norecovery, and norebalance, and then added additional servers with OSDs. While Ceph did allocate PG replicas to the newly added OSDs, the actual data didn’t move due to the active flags. We then attempted to use pgremapper to migrate all PGs from one server to the new one, removing or negating the flags in the process. However, we frequently failed to complete the migration of all data/PGs. Are we overlooking something? Does anyone have a reliable, step-by-step procedure we can follow to perform this correctly?
Any help would be greatly appreciated!
Michal
The safest approach would be to use the upmap-remapped.py tool developed by Dan at CERN. See [1] for details.
The idea is to leverage the upmap load balancer to progressively migrate the data to the new servers, minimizing performance impact on the cluster and clients. I like to create the OSDs ahead of time on the nodes
I then apply the norebalance flag (ceph osd set norebalance), disable
On 3/19/25 08:13, Janne Johansson wrote: that I initially place in a root directory called ‘closet’. the balancer (ceph balancer off), move the new nodes with already provisioned OSDs to their final location (rack), run ./upmap-remapped.py to bring all PGs back to active+clean state, remove the norebalance flag (ceph osd unset norebalance), re-enable the balancer (ceph balancer on) and watch data moving progressively as the upmap balancer executes its plans.
We do exactly that also, sometimes using pgremapper instead of upmap-remapper.py, but the effect is the same. Make the changes with norebalance, upmap the PGs to be happy where they are until we unset norebalance and let the ceph balancer correct it X% at a time.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Michal Strnad Storage specialist CESNET a.l.e.
The balancer is the usual way of using a remapper. * Set norebalance * Create the OSDs * Run the remapper so the backfill goes away * Unset norebalance * The balancer then will incrementally remove the extra upmaps, gradually evening out data. What did you do? Did you compute a set of upmaps by hand and apply them?
On Jun 21, 2025, at 11:00 AM, Michal Strnad <michal.strnad@cesnet.cz> wrote:
Hi.
Thank you for your reply
The balancer is off, so it shouldn't be interfering with that. Do you happen to have a list of steps (maybe commands from the history) on how you launched it? We're trying to figure out what we're missing.
Regards, Michal
On 6/21/25 15:21, Wesley Dillingham wrote:
Michal; Is the balancer (upmap balancer) running (ceph balancer status) and undoing some of the upmaps the pgremapper is introducing? Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Sat, Jun 21, 2025 at 2:05 AM Michal Strnad <michal.strnad@cesnet.cz> wrote:
Hi.
Over the past few days, we've been working on migrating data from machine A to machine B using the pgremapper tool, but we haven’t been able to achieve the expected results.
As part of our testing, we set up a small Ceph cluster with several monitors, managers, and servers with OSDs.We applied the flags noout, nobackfill, norecovery, and norebalance, and then added additional servers with OSDs. While Ceph did allocate PG replicas to the newly added OSDs, the actual data didn’t move due to the active flags. We then attempted to use pgremapper to migrate all PGs from one server to the new one, removing or negating the flags in the process. However, we frequently failed to complete the migration of all data/PGs. Are we overlooking something? Does anyone have a reliable, step-by-step procedure we can follow to perform this correctly?
Any help would be greatly appreciated!
Michal
The safest approach would be to use the upmap-remapped.py tool developed by Dan at CERN. See [1] for details.
The idea is to leverage the upmap load balancer to progressively migrate the data to the new servers, minimizing performance impact on the cluster and clients. I like to create the OSDs ahead of time on the nodes
I then apply the norebalance flag (ceph osd set norebalance), disable
On 3/19/25 08:13, Janne Johansson wrote: that I initially place in a root directory called ‘closet’. the balancer (ceph balancer off), move the new nodes with already provisioned OSDs to their final location (rack), run ./upmap-remapped.py to bring all PGs back to active+clean state, remove the norebalance flag (ceph osd unset norebalance), re-enable the balancer (ceph balancer on) and watch data moving progressively as the upmap balancer executes its plans.
We do exactly that also, sometimes using pgremapper instead of upmap-remapper.py, but the effect is the same. Make the changes with norebalance, upmap the PGs to be happy where they are until we unset norebalance and let the ceph balancer correct it X% at a time.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Michal Strnad Storage specialist CESNET a.l.e. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Anthony D'Atri
-
Frédéric Nass
-
Janne Johansson
-
Laimis Juzeliūnas
-
Michal Strnad
-
Sinan Polat
-
Wesley Dillingham