Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience
I'm managing a ceph cluster with +1K OSDs distributed accross 56 host. Untill now the crush rule used is the default replicated rule, but I want to change that in order to implement failure domain on rack level. Ceph version: Pacific 16.2.15 All pools(RBD and CephFS) currently use the default replicated_rule All OSD hosts has 25G network, and spinning disks(HDD) MON DBs is on NVMEs Workload is 24x7x365 The builtin balancer is disabled, and has been for a long time - Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30 Current plan is to * Disable rebalancing and backfilling by executing - ceph osd set norebalance; ceph osd set nobackfill; * Add all 7 Rack to crushmap and distribute the hosts (8 in each.) by using the built in commands like - * ceph osd crush add-bucket rack<#> rack root=default * ceph osd crush move osd-host<#> rack=rack<#> * Create the new rack split rule with command * ceph osd crush rule create-replicated rack_split default rack * Set the rule across all my pools * for p in $(ceph osd lspools | cut -d' ' -f 2) ; do echo $p $(ceph osd pool set $p crush_rule rack_split) ; done * I will probably also be using upmap-remapped.py here.. * Finally enable rebalancing, and backfilling - ceph osd unset norebalance; ceph osd unset nobackfill; However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation. Also I've seen some weird behavior - where Pacific(16) seems to do something different from Quincy(17) - In a test cluster I've tested the plan - On Pasific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue) On Quincy: Data is marked as misplaced - which seems correct. All experience and/or input will be greatly appreciated.
On Jan 17, 2025, at 6:02 AM, Kasper Rasmussen <kasper_steengaard@hotmail.com> wrote:
However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation.
Yep, that’s why you want to use upmap-remapped. Otherwise the thundering herd of data shuffling will DoS your client traffic, esp. since you’re using spinners. Count on pretty much all data moving in the process, and the convergence taking …. maybe a week?
On Pacific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue)
On Quincy: Data is marked as misplaced - which seems correct.
I’m not specifically familiar with such a change, but that could be mainly cosmetic, a function of how the percentage is calculated for objects / PGs that are multiply remapped. In the depths of time I had clusters that would sometimes show a negative number of RADOS objects to recover, it would bounce above and below zero a few times as it converged to 0.
Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30
I used a similar strategy with older releases. Note that this will complicate your transition, as those relative weights are a function of the CRUSH topology, so when the topology changes, likely some reweighted OSDs will get much less than their fair share, and some will get much more. How full is your cluster (ceph df)? It might not be a bad idea to incrementally revert those all to 1.00000 if you have the capacity, and disable the cron job. You’ll also likely want to switch to the balancer module for the upmap-remapped strategy to incrementally move your data around. Did you have it disabled for a specific reason? Updating to Reef before migrating might be to your advantage so that you can benefit from performance and efficiency improvements since Pacific.
Hello Kasper, Please be aware that the current "upmap-remapped" script is flaky. It might just refuse to work, with this message: Error loading remapped pgs This has been traced to the fact that "ceph pg ls remapped -f json" sets its stderr to non-blocking mode, and that is the same file descriptor to which jq (which follows in the pipeline) writes. Thus, jq can get -EAGAIN and terminate prematurely. The problem is tracked as https://tracker.ceph.com/issues/67505 Retrying the script might help. What's worse is that the whole reason for adding jq to the upmap-remapped script is another Ceph bug: it sometimes outputs invalid JSON (containing a literal inf or nan instead of a number), and this became much more common with Reef, as new fields were added that are commonly equal to inf or nan. This is tracked as https://tracker.ceph.com/issues/66215 and has a fix merged in a not-yet-released version. Maybe you should look into alternative tools, like https://github.com/digitalocean/pgremapper On Fri, Jan 17, 2025 at 11:43 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
On Jan 17, 2025, at 6:02 AM, Kasper Rasmussen <kasper_steengaard@hotmail.com> wrote:
However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation.
Yep, that’s why you want to use upmap-remapped. Otherwise the thundering herd of data shuffling will DoS your client traffic, esp. since you’re using spinners. Count on pretty much all data moving in the process, and the convergence taking …. maybe a week?
On Pacific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue)
On Quincy: Data is marked as misplaced - which seems correct.
I’m not specifically familiar with such a change, but that could be mainly cosmetic, a function of how the percentage is calculated for objects / PGs that are multiply remapped.
In the depths of time I had clusters that would sometimes show a negative number of RADOS objects to recover, it would bounce above and below zero a few times as it converged to 0.
Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30
I used a similar strategy with older releases. Note that this will complicate your transition, as those relative weights are a function of the CRUSH topology, so when the topology changes, likely some reweighted OSDs will get much less than their fair share, and some will get much more. How full is your cluster (ceph df)? It might not be a bad idea to incrementally revert those all to 1.00000 if you have the capacity, and disable the cron job. You’ll also likely want to switch to the balancer module for the upmap-remapped strategy to incrementally move your data around. Did you have it disabled for a specific reason?
Updating to Reef before migrating might be to your advantage so that you can benefit from performance and efficiency improvements since Pacific.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
The latest version (since September) switched to using the python rados bindings which not only fixes this problem, but also makes it much faster. It also has a fix I made that orders the upmaps so that data is moved off of OSDs before trying to move data on to them. This helps a lot on clusters with EC pools. Bryan From: Alexander Patrakov <patrakov@gmail.com> Date: Friday, January 17, 2025 at 09:53 To: Anthony D'Atri <anthony.datri@gmail.com> Cc: Kasper Rasmussen <kasper_steengaard@hotmail.com>, ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience !-------------------------------------------------------------------| This Message Is From an Untrusted Sender You have not previously corresponded with this sender. |-------------------------------------------------------------------! Hello Kasper, Please be aware that the current "upmap-remapped" script is flaky. It might just refuse to work, with this message: Error loading remapped pgs This has been traced to the fact that "ceph pg ls remapped -f json" sets its stderr to non-blocking mode, and that is the same file descriptor to which jq (which follows in the pipeline) writes. Thus, jq can get -EAGAIN and terminate prematurely. The problem is tracked as https://urldefense.com/v3/__https://tracker.ceph.com/issues/67505__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h3jtdOyn$<https://urldefense.com/v3/__https:/tracker.ceph.com/issues/67505__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h3jtdOyn$> Retrying the script might help. What's worse is that the whole reason for adding jq to the upmap-remapped script is another Ceph bug: it sometimes outputs invalid JSON (containing a literal inf or nan instead of a number), and this became much more common with Reef, as new fields were added that are commonly equal to inf or nan. This is tracked as https://urldefense.com/v3/__https://tracker.ceph.com/issues/66215__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h5M5tXer$<https://urldefense.com/v3/__https:/tracker.ceph.com/issues/66215__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h5M5tXer$> and has a fix merged in a not-yet-released version. Maybe you should look into alternative tools, like https://urldefense.com/v3/__https://github.com/digitalocean/pgremapper__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h8BgK2LL$<https://urldefense.com/v3/__https:/github.com/digitalocean/pgremapper__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h8BgK2LL$> On Fri, Jan 17, 2025 at 11:43 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
On Jan 17, 2025, at 6:02 AM, Kasper Rasmussen <kasper_steengaard@hotmail.com> wrote:
However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation.
Yep, that’s why you want to use upmap-remapped. Otherwise the thundering herd of data shuffling will DoS your client traffic, esp. since you’re using spinners. Count on pretty much all data moving in the process, and the convergence taking …. maybe a week?
On Pacific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue)
On Quincy: Data is marked as misplaced - which seems correct.
I’m not specifically familiar with such a change, but that could be mainly cosmetic, a function of how the percentage is calculated for objects / PGs that are multiply remapped.
In the depths of time I had clusters that would sometimes show a negative number of RADOS objects to recover, it would bounce above and below zero a few times as it converged to 0.
Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30
I used a similar strategy with older releases. Note that this will complicate your transition, as those relative weights are a function of the CRUSH topology, so when the topology changes, likely some reweighted OSDs will get much less than their fair share, and some will get much more. How full is your cluster (ceph df)? It might not be a bad idea to incrementally revert those all to 1.00000 if you have the capacity, and disable the cron job. You’ll also likely want to switch to the balancer module for the upmap-remapped strategy to incrementally move your data around. Did you have it disabled for a specific reason?
Updating to Reef before migrating might be to your advantage so that you can benefit from performance and efficiency improvements since Pacific.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That’s great to know, Bryan. I’ve seen multiple locations for the code out there, which one is canonical? (Lowercase c)
On Jan 17, 2025, at 3:46 PM, Stillwell, Bryan <bstillwe@akamai.com> wrote:
The latest version (since September) switched to using the python rados bindings which not only fixes this problem, but also makes it much faster. It also has a fix I made that orders the upmaps so that data is moved off of OSDs before trying to move data on to them. This helps a lot on clusters with EC pools.
Bryan
From: Alexander Patrakov <patrakov@gmail.com <mailto:patrakov@gmail.com>> Date: Friday, January 17, 2025 at 09:53 To: Anthony D'Atri <anthony.datri@gmail.com <mailto:anthony.datri@gmail.com>> Cc: Kasper Rasmussen <kasper_steengaard@hotmail.com <mailto:kasper_steengaard@hotmail.com>>, ceph-users@ceph.io <mailto:ceph-users@ceph.io> <ceph-users@ceph.io <mailto:ceph-users@ceph.io>> Subject: [ceph-users] Re: Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience
!-------------------------------------------------------------------| This Message Is From an Untrusted Sender You have not previously corresponded with this sender. |-------------------------------------------------------------------!
Hello Kasper,
Please be aware that the current "upmap-remapped" script is flaky. It might just refuse to work, with this message:
Error loading remapped pgs
This has been traced to the fact that "ceph pg ls remapped -f json" sets its stderr to non-blocking mode, and that is the same file descriptor to which jq (which follows in the pipeline) writes. Thus, jq can get -EAGAIN and terminate prematurely.
The problem is tracked as https://urldefense.com/v3/__https://tracker.ceph.com/issues/67505__;!!GjvTz_... <https://urldefense.com/v3/__https:/tracker.ceph.com/issues/67505__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h3jtdOyn$>
Retrying the script might help.
What's worse is that the whole reason for adding jq to the upmap-remapped script is another Ceph bug: it sometimes outputs invalid JSON (containing a literal inf or nan instead of a number), and this became much more common with Reef, as new fields were added that are commonly equal to inf or nan. This is tracked as https://urldefense.com/v3/__https://tracker.ceph.com/issues/66215__;!!GjvTz_... <https://urldefense.com/v3/__https:/tracker.ceph.com/issues/66215__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h5M5tXer$> and has a fix merged in a not-yet-released version.
Maybe you should look into alternative tools, like https://urldefense.com/v3/__https://github.com/digitalocean/pgremapper__;!!G... <https://urldefense.com/v3/__https:/github.com/digitalocean/pgremapper__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h8BgK2LL$>
On Fri, Jan 17, 2025 at 11:43 PM Anthony D'Atri <anthony.datri@gmail.com <mailto:anthony.datri@gmail.com>> wrote:
On Jan 17, 2025, at 6:02 AM, Kasper Rasmussen <kasper_steengaard@hotmail.com <mailto:kasper_steengaard@hotmail.com>> wrote:
However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation.
Yep, that’s why you want to use upmap-remapped. Otherwise the thundering herd of data shuffling will DoS your client traffic, esp. since you’re using spinners. Count on pretty much all data moving in the process, and the convergence taking …. maybe a week?
On Pacific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue)
On Quincy: Data is marked as misplaced - which seems correct.
I’m not specifically familiar with such a change, but that could be mainly cosmetic, a function of how the percentage is calculated for objects / PGs that are multiply remapped.
In the depths of time I had clusters that would sometimes show a negative number of RADOS objects to recover, it would bounce above and below zero a few times as it converged to 0.
Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30
I used a similar strategy with older releases. Note that this will complicate your transition, as those relative weights are a function of the CRUSH topology, so when the topology changes, likely some reweighted OSDs will get much less than their fair share, and some will get much more. How full is your cluster (ceph df)? It might not be a bad idea to incrementally revert those all to 1.00000 if you have the capacity, and disable the cron job. You’ll also likely want to switch to the balancer module for the upmap-remapped strategy to incrementally move your data around. Did you have it disabled for a specific reason?
Updating to Reef before migrating might be to your advantage so that you can benefit from performance and efficiency improvements since Pacific.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Dan can confirm, but this is what I believe is main repo: https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remap... Bryan From: Anthony D'Atri <anthony.datri@gmail.com> Date: Friday, January 17, 2025 at 15:35 To: Stillwell, Bryan <bstillwe@akamai.com> Cc: Alexander Patrakov <patrakov@gmail.com>, Kasper Rasmussen <kasper_steengaard@hotmail.com>, ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience That’s great to know, Bryan. I’ve seen multiple locations for the code out there, which one is canonical? (Lowercase c) On Jan 17, 2025, at 3: 46 PM, Stillwell, Bryan <bstillwe@ akamai. com> wrote: The latest version (since September) switched ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd That’s great to know, Bryan. I’ve seen multiple locations for the code out there, which one is canonical? (Lowercase c) On Jan 17, 2025, at 3:46 PM, Stillwell, Bryan <bstillwe@akamai.com> wrote: The latest version (since September) switched to using the python rados bindings which not only fixes this problem, but also makes it much faster. It also has a fix I made that orders the upmaps so that data is moved off of OSDs before trying to move data on to them. This helps a lot on clusters with EC pools. Bryan From: Alexander Patrakov <patrakov@gmail.com<mailto:patrakov@gmail.com>> Date: Friday, January 17, 2025 at 09:53 To: Anthony D'Atri <anthony.datri@gmail.com<mailto:anthony.datri@gmail.com>> Cc: Kasper Rasmussen <kasper_steengaard@hotmail.com<mailto:kasper_steengaard@hotmail.com>>, ceph-users@ceph.io<mailto:ceph-users@ceph.io> <ceph-users@ceph.io<mailto:ceph-users@ceph.io>> Subject: [ceph-users] Re: Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience !-------------------------------------------------------------------| This Message Is From an Untrusted Sender You have not previously corresponded with this sender. |-------------------------------------------------------------------! Hello Kasper, Please be aware that the current "upmap-remapped" script is flaky. It might just refuse to work, with this message: Error loading remapped pgs This has been traced to the fact that "ceph pg ls remapped -f json" sets its stderr to non-blocking mode, and that is the same file descriptor to which jq (which follows in the pipeline) writes. Thus, jq can get -EAGAIN and terminate prematurely. The problem is tracked as https://urldefense.com/v3/__https://tracker.ceph.com/issues/67505__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h3jtdOyn$<https://urldefense.com/v3/__https:/tracker.ceph.com/issues/67505__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h3jtdOyn$> Retrying the script might help. What's worse is that the whole reason for adding jq to the upmap-remapped script is another Ceph bug: it sometimes outputs invalid JSON (containing a literal inf or nan instead of a number), and this became much more common with Reef, as new fields were added that are commonly equal to inf or nan. This is tracked as https://urldefense.com/v3/__https://tracker.ceph.com/issues/66215__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h5M5tXer$<https://urldefense.com/v3/__https:/tracker.ceph.com/issues/66215__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h5M5tXer$> and has a fix merged in a not-yet-released version. Maybe you should look into alternative tools, like https://urldefense.com/v3/__https://github.com/digitalocean/pgremapper__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h8BgK2LL$<https://urldefense.com/v3/__https:/github.com/digitalocean/pgremapper__;!!GjvTz_vk!UldZKAbJ2Z9kMh9IMdHxZdGbAmWC6sE3ekqhHQMHb-HchhMen_khX4bU3IQcH2foYQtx9R_4h8BgK2LL$> On Fri, Jan 17, 2025 at 11:43 PM Anthony D'Atri <anthony.datri@gmail.com<mailto:anthony.datri@gmail.com>> wrote:
On Jan 17, 2025, at 6:02 AM, Kasper Rasmussen <kasper_steengaard@hotmail.com<mailto:kasper_steengaard@hotmail.com>> wrote:
However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation.
Yep, that’s why you want to use upmap-remapped. Otherwise the thundering herd of data shuffling will DoS your client traffic, esp. since you’re using spinners. Count on pretty much all data moving in the process, and the convergence taking …. maybe a week?
On Pacific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue)
On Quincy: Data is marked as misplaced - which seems correct.
I’m not specifically familiar with such a change, but that could be mainly cosmetic, a function of how the percentage is calculated for objects / PGs that are multiply remapped.
In the depths of time I had clusters that would sometimes show a negative number of RADOS objects to recover, it would bounce above and below zero a few times as it converged to 0.
Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30
I used a similar strategy with older releases. Note that this will complicate your transition, as those relative weights are a function of the CRUSH topology, so when the topology changes, likely some reweighted OSDs will get much less than their fair share, and some will get much more. How full is your cluster (ceph df)? It might not be a bad idea to incrementally revert those all to 1.00000 if you have the capacity, and disable the cron job. You’ll also likely want to switch to the balancer module for the upmap-remapped strategy to incrementally move your data around. Did you have it disabled for a specific reason?
Updating to Reef before migrating might be to your advantage so that you can benefit from performance and efficiency improvements since Pacific.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
On Pacific - It seems like when data is marked as degraded - no pgs are remapped, and the upmap-remapped.py is consistently returning - "There are no remapped PGs" Also nobackfill and noreabalance has no affect in holding back any remapping (norecover has). The recovering of the degraded files seems to be doing the remapping. So, deploying a new crush-map in Pasific seems to be a big-bang thing with no control handles. Balancing: My cluster has a 55% RAW Used. The balancer was disabled before I took over the cluster, unfortunately I do not have the full history of that - I believe it had something to do with it not working or being way to ineffective Your advice to revert the weights to 1.0000.. is to give the balancer a starting point or? My conclusion for now is, that since an upgrade to Quincy or Reef is already in the pipeline for the cluster, I'll go ahead and do that first before adding racks to my crushmap. ________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Friday, January 17, 2025 16:06 To: Kasper Rasmussen <kasper_steengaard@hotmail.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience On Jan 17, 2025, at 6:02 AM, Kasper Rasmussen <kasper_steengaard@hotmail.com> wrote: However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation. Yep, that’s why you want to use upmap-remapped. Otherwise the thundering herd of data shuffling will DoS your client traffic, esp. since you’re using spinners. Count on pretty much all data moving in the process, and the convergence taking …. maybe a week? On Pacific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue) On Quincy: Data is marked as misplaced - which seems correct. I’m not specifically familiar with such a change, but that could be mainly cosmetic, a function of how the percentage is calculated for objects / PGs that are multiply remapped. In the depths of time I had clusters that would sometimes show a negative number of RADOS objects to recover, it would bounce above and below zero a few times as it converged to 0. Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30 I used a similar strategy with older releases. Note that this will complicate your transition, as those relative weights are a function of the CRUSH topology, so when the topology changes, likely some reweighted OSDs will get much less than their fair share, and some will get much more. How full is your cluster (ceph df)? It might not be a bad idea to incrementally revert those all to 1.00000 if you have the capacity, and disable the cron job. You’ll also likely want to switch to the balancer module for the upmap-remapped strategy to incrementally move your data around. Did you have it disabled for a specific reason? Updating to Reef before migrating might be to your advantage so that you can benefit from performance and efficiency improvements since Pacific.
Hi Alexander Thanks for the heads up on the script, though it seems to me like it works totally fine at least on Quincy, but Ceph Pacifics way of handling changes to the crushmap, and remapping is not compatible. ________________________________ From: Alexander Patrakov <patrakov@gmail.com> Sent: Friday, January 17, 2025 17:51 To: Anthony D'Atri <anthony.datri@gmail.com> Cc: Kasper Rasmussen <kasper_steengaard@hotmail.com>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience Hello Kasper, Please be aware that the current "upmap-remapped" script is flaky. It might just refuse to work, with this message: Error loading remapped pgs This has been traced to the fact that "ceph pg ls remapped -f json" sets its stderr to non-blocking mode, and that is the same file descriptor to which jq (which follows in the pipeline) writes. Thus, jq can get -EAGAIN and terminate prematurely. The problem is tracked as https://tracker.ceph.com/issues/67505 Retrying the script might help. What's worse is that the whole reason for adding jq to the upmap-remapped script is another Ceph bug: it sometimes outputs invalid JSON (containing a literal inf or nan instead of a number), and this became much more common with Reef, as new fields were added that are commonly equal to inf or nan. This is tracked as https://tracker.ceph.com/issues/66215 and has a fix merged in a not-yet-released version. Maybe you should look into alternative tools, like https://github.com/digitalocean/pgremapper On Fri, Jan 17, 2025 at 11:43 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
On Jan 17, 2025, at 6:02 AM, Kasper Rasmussen <kasper_steengaard@hotmail.com> wrote:
However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation.
Yep, that’s why you want to use upmap-remapped. Otherwise the thundering herd of data shuffling will DoS your client traffic, esp. since you’re using spinners. Count on pretty much all data moving in the process, and the convergence taking …. maybe a week?
On Pacific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue)
On Quincy: Data is marked as misplaced - which seems correct.
I’m not specifically familiar with such a change, but that could be mainly cosmetic, a function of how the percentage is calculated for objects / PGs that are multiply remapped.
In the depths of time I had clusters that would sometimes show a negative number of RADOS objects to recover, it would bounce above and below zero a few times as it converged to 0.
Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30
I used a similar strategy with older releases. Note that this will complicate your transition, as those relative weights are a function of the CRUSH topology, so when the topology changes, likely some reweighted OSDs will get much less than their fair share, and some will get much more. How full is your cluster (ceph df)? It might not be a bad idea to incrementally revert those all to 1.00000 if you have the capacity, and disable the cron job. You’ll also likely want to switch to the balancer module for the upmap-remapped strategy to incrementally move your data around. Did you have it disabled for a specific reason?
Updating to Reef before migrating might be to your advantage so that you can benefit from performance and efficiency improvements since Pacific.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
participants (4)
-
Alexander Patrakov
-
Anthony D'Atri
-
Kasper Rasmussen
-
Stillwell, Bryan