Hi all, We moved our Ceph cluster to a new data centre about three months ago, which completely changed its physical topology. I changed the CRUSH map accordingly so that the CRUSH location matches the physical location again and the cluster has been rebalancing ever since. Due to capacity limits, the rebalancing requires rather frequent manual reweights so that individual OSDs don't run full. We started at around 65% of remapped PGs and are now down to 12% (it goes a little faster now that the bulk is done). Unfortunately, the MONs refuse to trim their stores while there are remapped PGs in the cluster, so their disk usage has increased gradually from 600MB to 17GB. I assume the rebalancing will finish before we run out of disk space, but the MON restart times have become unsustainably long and compacts now take up to 20 minutes or so. I'm a bit worried that at some point the MONs will start becoming unstable or worse: the stores may get corrupted. Is there anything that I can do to safely (!) get the MON stores back to a more manageable size even without all PGs being active+clean? I would like to forgo any potential disaster, especially before the holidays. Thanks Janek
Hi, On 17.12.24 14:40, Janek Bevendorff wrote:
Hi all,
We moved our Ceph cluster to a new data centre about three months ago, which completely changed its physical topology. I changed the CRUSH map accordingly so that the CRUSH location matches the physical location again and the cluster has been rebalancing ever since. Due to capacity limits, the rebalancing requires rather frequent manual reweights so that individual OSDs don't run full. We started at around 65% of remapped PGs and are now down to 12% (it goes a little faster now that the bulk is done).
Unfortunately, the MONs refuse to trim their stores while there are remapped PGs in the cluster, so their disk usage has increased gradually from 600MB to 17GB. I assume the rebalancing will finish before we run out of disk space, but the MON restart times have become unsustainably long and compacts now take up to 20 minutes or so. I'm a bit worried that at some point the MONs will start becoming unstable or worse: the stores may get corrupted.
Is there anything that I can do to safely (!) get the MON stores back to a more manageable size even without all PGs being active+clean? I would like to forgo any potential disaster, especially before the holidays.
Just my 0.02 euro... You can use pg-remapper (https://github.com/digitalocean/pgremapper) or similar tools to cancel the remapping; up-map entries will be created that reflect the current state of the cluster. After all currently running backfills are finished your mons should not be blocked anymore. I would also disable the balancer temporarily since it will trigger new backfills for those PG that are not at their optimal locations. After mons are fine again you can just enable the balancer. This requires a ceph release and ceph clients with up-map support. Not tested in real life, but this approach might work. Best regards, Burkhard
Agree with pg-remapper or upmap-remapped approach. One thing to be aware of though is that the Mons will invalidate any upmap which breaks the data placement rules. So for instance if you are moving from host based failure domain to rack based failure domain attempting to upmap the data back to its current location (to shed the remapped state) will mostly be useless as those upmap rules would break the data placement rule (not rack based) and therefore the mon will reject them. I think the approach I would use if you think the above will impact you is to increase the backfill rate to be as fast as possible. The approach depends on if you are using mclock or wpq. Which are you using? Also which version of Ceph are you using? Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Tue, Dec 17, 2024 at 8:52 AM Burkhard Linke < Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
Hi,
On 17.12.24 14:40, Janek Bevendorff wrote:
Hi all,
We moved our Ceph cluster to a new data centre about three months ago, which completely changed its physical topology. I changed the CRUSH map accordingly so that the CRUSH location matches the physical location again and the cluster has been rebalancing ever since. Due to capacity limits, the rebalancing requires rather frequent manual reweights so that individual OSDs don't run full. We started at around 65% of remapped PGs and are now down to 12% (it goes a little faster now that the bulk is done).
Unfortunately, the MONs refuse to trim their stores while there are remapped PGs in the cluster, so their disk usage has increased gradually from 600MB to 17GB. I assume the rebalancing will finish before we run out of disk space, but the MON restart times have become unsustainably long and compacts now take up to 20 minutes or so. I'm a bit worried that at some point the MONs will start becoming unstable or worse: the stores may get corrupted.
Is there anything that I can do to safely (!) get the MON stores back to a more manageable size even without all PGs being active+clean? I would like to forgo any potential disaster, especially before the holidays.
Just my 0.02 euro...
You can use pg-remapper (https://github.com/digitalocean/pgremapper) or similar tools to cancel the remapping; up-map entries will be created that reflect the current state of the cluster. After all currently running backfills are finished your mons should not be blocked anymore. I would also disable the balancer temporarily since it will trigger new backfills for those PG that are not at their optimal locations. After mons are fine again you can just enable the balancer. This requires a ceph release and ceph clients with up-map support.
Not tested in real life, but this approach might work.
Best regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your replies!
You can use pg-remapper (https://github.com/digitalocean/pgremapper) or similar tools to cancel the remapping; up-map entries will be created that reflect the current state of the cluster. After all currently running backfills are finished your mons should not be blocked anymore. I would also disable the balancer temporarily since it will trigger new backfills for those PG that are not at their optimal locations. After mons are fine again you can just enable the balancer. This requires a ceph release and ceph clients with up-map support.
Thanks, I'll try that. The balancer is already disabled, since I figured it only made the problem of individual OSDs filling up beyond their capacity worse while rebalancing was in progress.
So for instance if you are moving from host based failure domain to rack based failure domain attempting to upmap the data back to its current location (to shed the remapped state) will mostly be useless as those upmap rules would break the data placement rule (not rack based) and therefore the mon will reject them. I did change the failure domain for one pool, but apart from that, failure domain remains unchanged. However, the host position within racks has changed. The balancer is configured to use upmaps, but as said before, I disabled it after we moved, so upmaps would be out of date anyway.
I think the approach I would use if you think the above will impact you is to increase the backfill rate to be as fast as possible. The approach depends on if you are using mclock or wpq. Which are you using? Also which version of Ceph are you using? I'm using mlock on Ceph 18.2.4.
Anything else I should keep in mind? Will the manual OSD override weights be a problem when I apply pg-remapper?
You can use pg-remapper (https://github.com/digitalocean/pgremapper) or similar tools to cancel the remapping; up-map entries will be created that reflect the current state of the cluster. After all currently running backfills are finished your mons should not be blocked anymore. I would also disable the balancer temporarily since it will trigger new backfills for those PG that are not at their optimal locations. After mons are fine again you can just enable the balancer. This requires a ceph release and ceph clients with up-map support. Not tested in real life, but this approach might work.
We use that approach at times, just so that there isn't a long long queue of PGs in the remapped state, and as far as I can tell, it is quite safe, You just programmatically tell each PG that there is an upmap entry for it telling it to be exactly where it is now, and then it isn't "misplaced" anymore. When you enable the balancer it will take a percentage of these and just remove their individual upmap entry, and they start to move as needed. If you want to have a small movement, set the max balancer to a really low value, and few PGs will be moving at the same time. If your wpq/mclock settings work ok for you, you can have a large percentage and let the IO scheduler prioritize for you. But as Burkhard says, setting "norebalance" for a moment, having the balancer disabled and then running one of these tools once or twice will make all PGs active+clean where they are, even if that isn't the desired end location for them. This should help your mons a lot, then enable the balancer and unset "norebalance" and let it finish the last PGs you have in the wrong spot. -- May the most significant bit of your life be positive.
Thanks. I tried running the command (dry run for now), but something's not working as expected. Have you ever seen this? $ /root/go/bin/pgremapper cancel-backfill --verbose ** executing: ceph osd dump -f json panic: invalid character 'i' looking for beginning of value goroutine 1 [running]: main.mustParseCephCommand({0xc000b00000?, 0x0?}, {0x0?, 0x0?}, {0x59c9c0?, 0xc00011af30?}) /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/ceph.go:743 +0xe6 main.osdDump() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/ceph.go:517 +0x53 main.mustGetCurrentMappingState() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/mappingstate.go:54 +0x1d main.glob..func9(0x73f540?, {0x5dabc9?, 0x1?, 0x1?}) /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/main.go:133 +0x1ef github.com/spf13/cobra.(*Command).execute(0x73f540, {0xc0001188a0, 0x1, 0x1}) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x663 github.com/spf13/cobra.(*Command).ExecuteC(0x73f040) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x39c github.com/spf13/cobra.(*Command).Execute(...) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897 main.main() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/main.go:740 +0x25 Somehow it's choking here while trying to dumping OSDs: https://github.com/digitalocean/pgremapper/blob/main/ceph.go#L741 There isn't an issue report about this. On 17/12/2024 15:59, Janne Johansson wrote:
You can use pg-remapper (https://github.com/digitalocean/pgremapper) or similar tools to cancel the remapping; up-map entries will be created that reflect the current state of the cluster. After all currently running backfills are finished your mons should not be blocked anymore. I would also disable the balancer temporarily since it will trigger new backfills for those PG that are not at their optimal locations. After mons are fine again you can just enable the balancer. This requires a ceph release and ceph clients with up-map support. Not tested in real life, but this approach might work. We use that approach at times, just so that there isn't a long long queue of PGs in the remapped state, and as far as I can tell, it is quite safe, You just programmatically tell each PG that there is an upmap entry for it telling it to be exactly where it is now, and then it isn't "misplaced" anymore. When you enable the balancer it will take a percentage of these and just remove their individual upmap entry, and they start to move as needed. If you want to have a small movement, set the max balancer to a really low value, and few PGs will be moving at the same time. If your wpq/mclock settings work ok for you, you can have a large percentage and let the IO scheduler prioritize for you. But as Burkhard says, setting "norebalance" for a moment, having the balancer disabled and then running one of these tools once or twice will make all PGs active+clean where they are, even if that isn't the desired end location for them. This should help your mons a lot, then enable the balancer and unset "norebalance" and let it finish the last PGs you have in the wrong spot.
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de
I checked the ceph osd dump json-pretty output and validated it with a little Python script. Turns out, there's this somewhere around line 1200: "read_balance": { "score_acting": inf, "score_stable": inf, "optimal_score": 0, "raw_score_acting": 3, "raw_score_stable": 3, "primary_affinity_weighted": 0.9999845027923584, "average_primary_affinity": 1, "average_primary_affinity_weighted": 1 } The inf values seem to be the problem. These are the only two invalid JSON values in the whole file. Do you happen to know how I can debug/fix this? On 17/12/2024 16:17, Janek Bevendorff wrote:
Thanks. I tried running the command (dry run for now), but something's not working as expected. Have you ever seen this?
$ /root/go/bin/pgremapper cancel-backfill --verbose ** executing: ceph osd dump -f json panic: invalid character 'i' looking for beginning of value
goroutine 1 [running]: main.mustParseCephCommand({0xc000b00000?, 0x0?}, {0x0?, 0x0?}, {0x59c9c0?, 0xc00011af30?}) /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/ceph.go:743 +0xe6 main.osdDump() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/ceph.go:517 +0x53 main.mustGetCurrentMappingState() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/mappingstate.go:54 +0x1d main.glob..func9(0x73f540?, {0x5dabc9?, 0x1?, 0x1?}) /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/main.go:133 +0x1ef github.com/spf13/cobra.(*Command).execute(0x73f540, {0xc0001188a0, 0x1, 0x1}) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x663 github.com/spf13/cobra.(*Command).ExecuteC(0x73f040) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x39c github.com/spf13/cobra.(*Command).Execute(...) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897 main.main() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/main.go:740 +0x25
Somehow it's choking here while trying to dumping OSDs: https://github.com/digitalocean/pgremapper/blob/main/ceph.go#L741
There isn't an issue report about this.
On 17/12/2024 15:59, Janne Johansson wrote:
You can use pg-remapper (https://github.com/digitalocean/pgremapper) or similar tools to cancel the remapping; up-map entries will be created that reflect the current state of the cluster. After all currently running backfills are finished your mons should not be blocked anymore. I would also disable the balancer temporarily since it will trigger new backfills for those PG that are not at their optimal locations. After mons are fine again you can just enable the balancer. This requires a ceph release and ceph clients with up-map support. Not tested in real life, but this approach might work. We use that approach at times, just so that there isn't a long long queue of PGs in the remapped state, and as far as I can tell, it is quite safe, You just programmatically tell each PG that there is an upmap entry for it telling it to be exactly where it is now, and then it isn't "misplaced" anymore. When you enable the balancer it will take a percentage of these and just remove their individual upmap entry, and they start to move as needed. If you want to have a small movement, set the max balancer to a really low value, and few PGs will be moving at the same time. If your wpq/mclock settings work ok for you, you can have a large percentage and let the IO scheduler prioritize for you. But as Burkhard says, setting "norebalance" for a moment, having the balancer disabled and then running one of these tools once or twice will make all PGs active+clean where they are, even if that isn't the desired end location for them. This should help your mons a lot, then enable the balancer and unset "norebalance" and let it finish the last PGs you have in the wrong spot.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de
Looks like there is something wrong with the .mgr pool. All other have proper values. For now I've patched the pgremapper source code to replace the inf values with 0 before unmarshaling the JSON. That at least made the tool work. I guess it's safe to just delete that pool and let the MGRs recreate it?? (is it?) On 17/12/2024 17:01, Janek Bevendorff wrote:
I checked the ceph osd dump json-pretty output and validated it with a little Python script. Turns out, there's this somewhere around line 1200:
"read_balance": { "score_acting": inf, "score_stable": inf, "optimal_score": 0, "raw_score_acting": 3, "raw_score_stable": 3, "primary_affinity_weighted": 0.9999845027923584, "average_primary_affinity": 1, "average_primary_affinity_weighted": 1 }
The inf values seem to be the problem. These are the only two invalid JSON values in the whole file. Do you happen to know how I can debug/fix this?
On 17/12/2024 16:17, Janek Bevendorff wrote:
Thanks. I tried running the command (dry run for now), but something's not working as expected. Have you ever seen this?
$ /root/go/bin/pgremapper cancel-backfill --verbose ** executing: ceph osd dump -f json panic: invalid character 'i' looking for beginning of value
goroutine 1 [running]: main.mustParseCephCommand({0xc000b00000?, 0x0?}, {0x0?, 0x0?}, {0x59c9c0?, 0xc00011af30?}) /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/ceph.go:743 +0xe6 main.osdDump() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/ceph.go:517 +0x53 main.mustGetCurrentMappingState() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/mappingstate.go:54 +0x1d main.glob..func9(0x73f540?, {0x5dabc9?, 0x1?, 0x1?}) /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/main.go:133 +0x1ef github.com/spf13/cobra.(*Command).execute(0x73f540, {0xc0001188a0, 0x1, 0x1}) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x663 github.com/spf13/cobra.(*Command).ExecuteC(0x73f040) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x39c github.com/spf13/cobra.(*Command).Execute(...) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897 main.main() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/main.go:740 +0x25
Somehow it's choking here while trying to dumping OSDs: https://github.com/digitalocean/pgremapper/blob/main/ceph.go#L741
There isn't an issue report about this.
On 17/12/2024 15:59, Janne Johansson wrote:
You can use pg-remapper (https://github.com/digitalocean/pgremapper) or similar tools to cancel the remapping; up-map entries will be created that reflect the current state of the cluster. After all currently running backfills are finished your mons should not be blocked anymore. I would also disable the balancer temporarily since it will trigger new backfills for those PG that are not at their optimal locations. After mons are fine again you can just enable the balancer. This requires a ceph release and ceph clients with up-map support. Not tested in real life, but this approach might work. We use that approach at times, just so that there isn't a long long queue of PGs in the remapped state, and as far as I can tell, it is quite safe, You just programmatically tell each PG that there is an upmap entry for it telling it to be exactly where it is now, and then it isn't "misplaced" anymore. When you enable the balancer it will take a percentage of these and just remove their individual upmap entry, and they start to move as needed. If you want to have a small movement, set the max balancer to a really low value, and few PGs will be moving at the same time. If your wpq/mclock settings work ok for you, you can have a large percentage and let the IO scheduler prioritize for you. But as Burkhard says, setting "norebalance" for a moment, having the balancer disabled and then running one of these tools once or twice will make all PGs active+clean where they are, even if that isn't the desired end location for them. This should help your mons a lot, then enable the balancer and unset "norebalance" and let it finish the last PGs you have in the wrong spot.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de
Hey Janek, Ah, yes, we ran into that invalid json output in https://github.com/digitalocean/ceph_exporter as well. I have a patch I wrote for ceph_exporter that I can port over to pgremapper (that does similar to what your patch does). Josh On Tue, Dec 17, 2024 at 9:38 AM Janek Bevendorff <janek.bevendorff@uni-weimar.de> wrote:
Looks like there is something wrong with the .mgr pool. All other have proper values. For now I've patched the pgremapper source code to replace the inf values with 0 before unmarshaling the JSON. That at least made the tool work. I guess it's safe to just delete that pool and let the MGRs recreate it?? (is it?)
On 17/12/2024 17:01, Janek Bevendorff wrote:
I checked the ceph osd dump json-pretty output and validated it with a little Python script. Turns out, there's this somewhere around line 1200:
"read_balance": { "score_acting": inf, "score_stable": inf, "optimal_score": 0, "raw_score_acting": 3, "raw_score_stable": 3, "primary_affinity_weighted": 0.9999845027923584, "average_primary_affinity": 1, "average_primary_affinity_weighted": 1 }
The inf values seem to be the problem. These are the only two invalid JSON values in the whole file. Do you happen to know how I can debug/fix this?
On 17/12/2024 16:17, Janek Bevendorff wrote:
Thanks. I tried running the command (dry run for now), but something's not working as expected. Have you ever seen this?
$ /root/go/bin/pgremapper cancel-backfill --verbose ** executing: ceph osd dump -f json panic: invalid character 'i' looking for beginning of value
goroutine 1 [running]: main.mustParseCephCommand({0xc000b00000?, 0x0?}, {0x0?, 0x0?}, {0x59c9c0?, 0xc00011af30?}) /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/ceph.go:743 +0xe6 main.osdDump() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/ceph.go:517 +0x53 main.mustGetCurrentMappingState() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/mappingstate.go:54 +0x1d main.glob..func9(0x73f540?, {0x5dabc9?, 0x1?, 0x1?}) /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/main.go:133 +0x1ef github.com/spf13/cobra.(*Command).execute(0x73f540, {0xc0001188a0, 0x1, 0x1}) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x663 github.com/spf13/cobra.(*Command).ExecuteC(0x73f040) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x39c github.com/spf13/cobra.(*Command).Execute(...) /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897 main.main() /root/go/pkg/mod/github.com/digitalocean/pgremapper@v0.0.0-20240313130618-268522c0f6d5/main.go:740 +0x25
Somehow it's choking here while trying to dumping OSDs: https://github.com/digitalocean/pgremapper/blob/main/ceph.go#L741
There isn't an issue report about this.
On 17/12/2024 15:59, Janne Johansson wrote:
You can use pg-remapper (https://github.com/digitalocean/pgremapper) or similar tools to cancel the remapping; up-map entries will be created that reflect the current state of the cluster. After all currently running backfills are finished your mons should not be blocked anymore. I would also disable the balancer temporarily since it will trigger new backfills for those PG that are not at their optimal locations. After mons are fine again you can just enable the balancer. This requires a ceph release and ceph clients with up-map support. Not tested in real life, but this approach might work. We use that approach at times, just so that there isn't a long long queue of PGs in the remapped state, and as far as I can tell, it is quite safe, You just programmatically tell each PG that there is an upmap entry for it telling it to be exactly where it is now, and then it isn't "misplaced" anymore. When you enable the balancer it will take a percentage of these and just remove their individual upmap entry, and they start to move as needed. If you want to have a small movement, set the max balancer to a really low value, and few PGs will be moving at the same time. If your wpq/mclock settings work ok for you, you can have a large percentage and let the IO scheduler prioritize for you. But as Burkhard says, setting "norebalance" for a moment, having the balancer disabled and then running one of these tools once or twice will make all PGs active+clean where they are, even if that isn't the desired end location for them. This should help your mons a lot, then enable the balancer and unset "norebalance" and let it finish the last PGs you have in the wrong spot.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany
Phone: +49 3643 58 3577 www.webis.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Something's not quite right yet. I got the remapped PGs down from > 4000 to around 1300, but there it stops. When I restart the process, I can get it down to around 280, but there it stops and creeps back up afterwards. I have a bunch of these messages in the output: WARNING: pg 100.3d53: conflicting mapping 1068->1051 found when trying to map 187->1068 There's maybe around 70-80 of them (definitely not 280 or 1300), any idea how I can fix that? The messages all point to the same pool (our largest one, I did not change the failure domain for this pool).
Ah, yes, we ran into that invalid json output in https://github.com/digitalocean/ceph_exporter as well. I have a patch I wrote for ceph_exporter that I can port over to pgremapper (that does similar to what your patch does).
That'd be nice! Janek
I think it was mentioned elsewhere in this thread that there are limitations to what upmap can do, especially in significant crush map change situations. It can't violate crush rules (mon-enforced), and if the same OSD shows up multiple times in a backfill then upmap can't deal with it. Creeping back up is a bit odd; if you have the balancer off, any chance there's somehow also a PG split going on? What does 'ceph osd pool ls detail' say? Josh On Tue, Dec 17, 2024 at 10:06 AM Janek Bevendorff <janek.bevendorff@uni-weimar.de> wrote:
Something's not quite right yet. I got the remapped PGs down from > 4000 to around 1300, but there it stops. When I restart the process, I can get it down to around 280, but there it stops and creeps back up afterwards.
I have a bunch of these messages in the output:
WARNING: pg 100.3d53: conflicting mapping 1068->1051 found when trying to map 187->1068
There's maybe around 70-80 of them (definitely not 280 or 1300), any idea how I can fix that? The messages all point to the same pool (our largest one, I did not change the failure domain for this pool).
Ah, yes, we ran into that invalid json output in https://github.com/digitalocean/ceph_exporter as well. I have a patch I wrote for ceph_exporter that I can port over to pgremapper (that does similar to what your patch does).
That'd be nice!
Janek
I think it was mentioned elsewhere in this thread that there are limitations to what upmap can do, especially in significant crush map change situations. It can't violate crush rules (mon-enforced), and if the same OSD shows up multiple times in a backfill then upmap can't deal with it.
The number of warnings was larger the first few times I ran it, but I think this is as low as I can get for now. I also just replaced one HDD with excessive read/write times, which neared the end of its lifespan and kept flip-flopping. So for now I'm waiting for that to backfill again, then I'll try again tomorrow.
Creeping back up is a bit odd; if you have the balancer off, any chance there's somehow also a PG split going on? What does 'ceph osd pool ls detail' say?
Indeed! I did increase the pgp_num of a pool a while back, totally forgot about that. Due to the ongoing rebalancing it was stuck half way, but now suddenly started up again. The current PG number of that pool is not quite final yet, but definitely higher than previously. I'll keep this running over night and see where it settles. Thanks so far! Janek
Josh
On Tue, Dec 17, 2024 at 10:06 AM Janek Bevendorff <janek.bevendorff@uni-weimar.de> wrote:
Something's not quite right yet. I got the remapped PGs down from > 4000 to around 1300, but there it stops. When I restart the process, I can get it down to around 280, but there it stops and creeps back up afterwards.
I have a bunch of these messages in the output:
WARNING: pg 100.3d53: conflicting mapping 1068->1051 found when trying to map 187->1068
There's maybe around 70-80 of them (definitely not 280 or 1300), any idea how I can fix that? The messages all point to the same pool (our largest one, I did not change the failure domain for this pool).
Ah, yes, we ran into that invalid json output in https://github.com/digitalocean/ceph_exporter as well. I have a patch I wrote for ceph_exporter that I can port over to pgremapper (that does similar to what your patch does). That'd be nice!
Janek
-- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de
On 18/12/24 02:30, Janek Bevendorff wrote:
I did increase the pgp_num of a pool a while back, totally forgot about that. Due to the ongoing rebalancing it was stuck half way, but now suddenly started up again. The current PG number of that pool is not quite final yet, but definitely higher than previously.
If you want this to stabilise faster, you could set pg_num (target) to be the value that it is right now. You'll get POOL_PG_NUM_NOT_POWER_OF_TWO warning but you can ignore or mute that for a time.
The replaced OSD is still backfilling, but the MON store size has decreased to 2.3GB already. I'm going to wait for the recovery to finish, then I'll reset all the temporary CRUSH weights, cancel the backfills, and then let the balancer do the rest. Thanks all! On 18/12/2024 03:42, Gregory Orange wrote:
On 18/12/24 02:30, Janek Bevendorff wrote:
I did increase the pgp_num of a pool a while back, totally forgot about that. Due to the ongoing rebalancing it was stuck half way, but now suddenly started up again. The current PG number of that pool is not quite final yet, but definitely higher than previously. If you want this to stabilise faster, you could set pg_num (target) to be the value that it is right now. You'll get POOL_PG_NUM_NOT_POWER_OF_TWO warning but you can ignore or mute that for a time.
participants (6)
-
Burkhard Linke
-
Gregory Orange
-
Janek Bevendorff
-
Janne Johansson
-
Joshua Baergen
-
Wesley Dillingham