Forcibly move PGs from full to empty OSD
Hi, Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too. Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)". Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD? THX
On 3/4/20 11:15 AM, Thomas Schneider wrote:
Hi,
Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)".
Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD?
Yes, you could manually create upmap items to map PGs to a specific OSD and offload another one. This is what the balancer also does. Keep in mind though that you should respect your failure domain (host, rack, etc) when creating these mappings. Wido
THX _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Wido, can you please share some detailed instructions how to do this? And what do you mean with "respect your failure domain"? THX Am 04.03.2020 um 11:27 schrieb Wido den Hollander:
On 3/4/20 11:15 AM, Thomas Schneider wrote:
Hi,
Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)".
Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD? Yes, you could manually create upmap items to map PGs to a specific OSD and offload another one.
This is what the balancer also does. Keep in mind though that you should respect your failure domain (host, rack, etc) when creating these mappings.
Wido
THX _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Thomas, The two commands you're looking for are: ceph osd pg-upmap-items $pg $source_osd $dest_osd and to remove them ceph osd rm-pg-upmap-items $pg You need to pair this with finding which pg's are on your full osd's. I use ceph pg dump and grep for the pool number and osd. With respect to what Wido said about your failure domain, my testing indicated that trying to move a pg to an osd that would violate crush rules for failure domain resulted in an error code in the logs and no movement of the pg. You may wish to test this yourself to be sure. I tested on 3x rep pool with host failure domain. Regards, Rich
Hi Wido, can you please share some detailed instructions how to do this? And what do you mean with "respect your failure domain"? THX Am 04.03.2020 um 11:27 schrieb Wido den Hollander:
On 3/4/20 11:15 AM, Thomas Schneider wrote:
Hi,
Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)".
Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD? Yes, you could manually create upmap items to map PGs to a specific OSD and offload another one.
This is what the balancer also does. Keep in mind though that you should respect your failure domain (host, rack, etc) when creating these mappings.
Wido
THX _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
He means that if eg. you enforce 1 copy of a PG per rack, that any upmaps you enter don’t result in 2 or 3 in the same rack. If your CRUSH poilicy is one copy per *host* the danger is even higher that you could have data become unavailable or even lost in case of a failure.
On Mar 16, 2020, at 7:45 AM, Thomas Schneider <74cmonty@gmail.com> wrote:
Hi Wido,
can you please share some detailed instructions how to do this? And what do you mean with "respect your failure domain"?
THX
Am 04.03.2020 um 11:27 schrieb Wido den Hollander:
On 3/4/20 11:15 AM, Thomas Schneider wrote:
Hi,
Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)".
Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD? Yes, you could manually create upmap items to map PGs to a specific OSD and offload another one.
This is what the balancer also does. Keep in mind though that you should respect your failure domain (host, rack, etc) when creating these mappings.
Wido
THX _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, thanks for this clarification. I'm running a 7-node-cluster and this risk should be managable. Am 16.03.2020 um 16:57 schrieb Anthony D'Atri:
He means that if eg. you enforce 1 copy of a PG per rack, that any upmaps you enter don’t result in 2 or 3 in the same rack. If your CRUSH poilicy is one copy per *host* the danger is even higher that you could have data become unavailable or even lost in case of a failure.
On Mar 16, 2020, at 7:45 AM, Thomas Schneider <74cmonty@gmail.com> wrote:
Hi Wido,
can you please share some detailed instructions how to do this? And what do you mean with "respect your failure domain"?
THX
Am 04.03.2020 um 11:27 schrieb Wido den Hollander:
On 3/4/20 11:15 AM, Thomas Schneider wrote:
Hi,
Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)".
Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD? Yes, you could manually create upmap items to map PGs to a specific OSD and offload another one.
This is what the balancer also does. Keep in mind though that you should respect your failure domain (host, rack, etc) when creating these mappings.
Wido
THX _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Thomas To get the usage: ceph osd df | sort -nk8 #VAR is the ratio to avg util #WEIGHT is CRUSHMAP weight; typically the Disk capacity in TiB #REWEIGHT is temporary (until osd restart or ceph osd set noout) WEIGHT correction for manual rebalance You can use for temporary reweight: ceph osd reweight osd.<ID> <REWEIGHT> or : ceph osd test-reweight-by-utilization <VAR> ceph osd reweight-by-utilization <VAR> You can use for permanent reweight: ceph osd crush reweight osd.<ID> <WEIGHT> To speed up the backfill I use this (warning it decreases client performance): ceph tell 'osd.*' injectargs '--osd_max_backfills 30 --osd_recovery_max_active 45 --osd_recovery_op_priority 10' Then to set to back to default: ceph tell 'osd.*' injectargs '--osd_max_backfills 1 --osd_recovery_max_active 3 --osd_recovery_op_priority 3' Cheers Francois Scheurer ________________________________________ From: Thomas Schneider <74cmonty@gmail.com> Sent: Wednesday, March 4, 2020 11:15 AM To: ceph-users@ceph.io Subject: [ceph-users] Forcibly move PGs from full to empty OSD Hi, Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too. Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)". Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD? THX _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I already use CRUSHMAP weight to manually control the OSD utilization. However this results in a situation where 5-10% of my 336 OSDs have a weight < 1.00000, and this would impact/hinder ceph balancer to work. This means I would need to modify any OSD with weight < 1.00000 first before ceph balancer can start. And this would exceed the treshold of a pool to store new data. Therefore I would prefer to "move PGs manually to empty OSDs". THX Am 04.03.2020 um 11:34 schrieb Scheurer François:
Hi Thomas
To get the usage: ceph osd df | sort -nk8 #VAR is the ratio to avg util #WEIGHT is CRUSHMAP weight; typically the Disk capacity in TiB #REWEIGHT is temporary (until osd restart or ceph osd set noout) WEIGHT correction for manual rebalance
You can use for temporary reweight: ceph osd reweight osd.<ID> <REWEIGHT> or : ceph osd test-reweight-by-utilization <VAR> ceph osd reweight-by-utilization <VAR>
You can use for permanent reweight: ceph osd crush reweight osd.<ID> <WEIGHT>
To speed up the backfill I use this (warning it decreases client performance): ceph tell 'osd.*' injectargs '--osd_max_backfills 30 --osd_recovery_max_active 45 --osd_recovery_op_priority 10' Then to set to back to default: ceph tell 'osd.*' injectargs '--osd_max_backfills 1 --osd_recovery_max_active 3 --osd_recovery_op_priority 3'
Cheers Francois Scheurer
________________________________________ From: Thomas Schneider <74cmonty@gmail.com> Sent: Wednesday, March 4, 2020 11:15 AM To: ceph-users@ceph.io Subject: [ceph-users] Forcibly move PGs from full to empty OSD
Hi,
Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)".
Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD?
THX _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Rich Bade
-
Scheurer François
-
Thomas Schneider
-
Wido den Hollander