4 Mar
2020
4 Mar
'20
1:15 p.m.
Hi,
Ceph balancer is not working correctly; there's an open bug
<https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more
and more warnings about "nearfull osd(s)".
Therefore my question is:
How can I forcibly move PGs from full OSD to empty OSD?
THX
4 Mar
4 Mar
1:27 p.m.
On 3/4/20 11:15 AM, Thomas Schneider wrote:
Hi,
Ceph balancer is not working correctly; there's an open bug
<https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more
and more warnings about "nearfull osd(s)".
Therefore my question is:
How can I forcibly move PGs from full OSD to empty OSD?
Yes, you could manually create upmap items to map PGs to a specific OSD
and offload another one.
This is what the balancer also does. Keep in mind though that you should
respect your failure domain (host, rack, etc) when creating these mappings.
Wido
>
> THX
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
1:57 p.m.
Hi Wido,
can you please share some detailed instructions how to do this?
And what do you mean with "respect your failure domain"?
THX
Am 04.03.2020 um 11:27 schrieb Wido den Hollander:
On 3/4/20 11:15 AM, Thomas Schneider wrote:
Hi,
Ceph balancer is not working correctly; there's an open bug
<https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more
and more warnings about "nearfull osd(s)".
Therefore my question is:
How can I forcibly move PGs from full OSD to empty OSD?
Yes, you could manually
create upmap items to map PGs to a specific OSD
and offload another one.
This is what the balancer also does. Keep in mind though that you should
respect your failure domain (host, rack, etc) when creating these mappings.
Wido
> THX
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> 
10 Mar
10 Mar
4:26 a.m.
Hi Thomas,
The two commands you're looking for are:
ceph osd pg-upmap-items $pg $source_osd $dest_osd
and to remove them
ceph osd rm-pg-upmap-items $pg
You need to pair this with finding which pg's are on your full osd's. I use ceph
pg dump and grep for the pool number and osd.
With respect to what Wido said about your failure domain, my testing indicated that trying
to move a pg to an osd that would violate crush rules for failure domain resulted in an
error code in the logs and no movement of the pg.
You may wish to test this yourself to be sure. I tested on 3x rep pool with host failure
domain.
Regards,
Rich
16 Mar
16 Mar
5:45 p.m.
Hi Wido,
can you please share some detailed instructions how to do this?
And what do you mean with "respect your failure domain"?
THX
Am 04.03.2020 um 11:27 schrieb Wido den Hollander:
On 3/4/20 11:15 AM, Thomas Schneider wrote:
Hi,
Ceph balancer is not working correctly; there's an open bug
<https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more
and more warnings about "nearfull osd(s)".
Therefore my question is:
How can I forcibly move PGs from full OSD to empty OSD?
Yes, you could manually
create upmap items to map PGs to a specific OSD
and offload another one.
This is what the balancer also does. Keep in mind though that you should
respect your failure domain (host, rack, etc) when creating these mappings.
Wido
> THX
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> 
6:57 p.m.
He means that if eg. you enforce 1 copy of a PG per rack, that any upmaps you enter don’t
result in 2 or 3 in the same rack. If your CRUSH poilicy is one copy per *host* the
danger is even higher that you could have data become unavailable or even lost in case of
a failure.
On Mar 16, 2020, at 7:45 AM, Thomas Schneider
<74cmonty(a)gmail.com> wrote:
Hi Wido,
can you please share some detailed instructions how to do this?
And what do you mean with "respect your failure domain"?
THX
Am 04.03.2020 um 11:27 schrieb Wido den Hollander:
On 3/4/20 11:15 AM, Thomas Schneider wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Hi,
Ceph balancer is not working correctly; there's an open bug
<https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more
and more warnings about "nearfull osd(s)".
Therefore my question is:
How can I forcibly move PGs from full OSD to empty OSD?
Yes, you could manually
create upmap items to map PGs to a specific OSD
and offload another one.
This is what the balancer also does. Keep in mind though that you should
respect your failure domain (host, rack, etc) when creating these mappings.
Wido
> THX
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
7:18 p.m.
Hi,
thanks for this clarification.
I'm running a 7-node-cluster and this risk should be managable.
Am 16.03.2020 um 16:57 schrieb Anthony D'Atri:
He means that if eg. you enforce 1 copy of a PG per
rack, that any upmaps you enter don’t result in 2 or 3 in the same rack. If your CRUSH
poilicy is one copy per *host* the danger is even higher that you could have data become
unavailable or even lost in case of a failure.
> On Mar 16, 2020, at 7:45 AM, Thomas Schneider <74cmonty(a)gmail.com> wrote:
>
> Hi Wido,
>
> can you please share some detailed instructions how to do this?
> And what do you mean with "respect your failure domain"?
>
> THX
>
> Am 04.03.2020 um 11:27 schrieb Wido den Hollander:
>> On 3/4/20 11:15 AM, Thomas Schneider wrote:
>>> Hi,
>>>
>>> Ceph balancer is not working correctly; there's an open bug
>>> <https://tracker.ceph.com/issues/43752> report, too.
>>>
>>> Until this issue is not solved, I need a workaround because I get more
>>> and more warnings about "nearfull osd(s)".
>>>
>>> Therefore my question is:
>>> How can I forcibly move PGs from full OSD to empty OSD?
>> Yes, you could manually create upmap items to map PGs to a specific OSD
>> and offload another one.
>>
>> This is what the balancer also does. Keep in mind though that you should
>> respect your failure domain (host, rack, etc) when creating these mappings.
>>
>> Wido
>>
>>> THX
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
4 Mar
4 Mar
1:34 p.m.
Hi Thomas
To get the usage:
ceph osd df | sort -nk8
#VAR is the ratio to avg util
#WEIGHT is CRUSHMAP weight; typically the Disk capacity in TiB
#REWEIGHT is temporary (until osd restart or ceph osd set noout) WEIGHT correction for
manual rebalance
You can use for temporary reweight:
ceph osd reweight osd.<ID> <REWEIGHT>
or :
ceph osd test-reweight-by-utilization <VAR>
ceph osd reweight-by-utilization <VAR>
You can use for permanent reweight:
ceph osd crush reweight osd.<ID> <WEIGHT>
To speed up the backfill I use this (warning it decreases client performance):
ceph tell 'osd.*' injectargs '--osd_max_backfills 30
--osd_recovery_max_active 45 --osd_recovery_op_priority 10'
Then to set to back to default:
ceph tell 'osd.*' injectargs '--osd_max_backfills 1
--osd_recovery_max_active 3 --osd_recovery_op_priority 3'
Cheers
Francois Scheurer
________________________________________
From: Thomas Schneider <74cmonty(a)gmail.com>
Sent: Wednesday, March 4, 2020 11:15 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Forcibly move PGs from full to empty OSD
Hi,
Ceph balancer is not working correctly; there's an open bug
<https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more
and more warnings about "nearfull osd(s)".
Therefore my question is:
How can I forcibly move PGs from full OSD to empty OSD?
THX
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:46 p.m.
Hi,
I already use CRUSHMAP weight to manually control the OSD utilization.
However this results in a situation where 5-10% of my 336 OSDs have a
weight < 1.00000, and this would impact/hinder ceph balancer to work.
This means I would need to modify any OSD with weight < 1.00000 first
before ceph balancer can start. And this would exceed the treshold of a
pool to store new data.
Therefore I would prefer to "move PGs manually to empty OSDs".
THX
Am 04.03.2020 um 11:34 schrieb Scheurer François:
Hi Thomas
To get the usage:
ceph osd df | sort -nk8
#VAR is the ratio to avg util
#WEIGHT is CRUSHMAP weight; typically the Disk capacity in TiB
#REWEIGHT is temporary (until osd restart or ceph osd set noout) WEIGHT correction
for manual rebalance
You can use for temporary reweight:
ceph osd reweight osd.<ID> <REWEIGHT>
or :
ceph osd test-reweight-by-utilization <VAR>
ceph osd reweight-by-utilization <VAR>
You can use for permanent reweight:
ceph osd crush reweight osd.<ID> <WEIGHT>
To speed up the backfill I use this (warning it decreases client performance):
ceph tell 'osd.*' injectargs '--osd_max_backfills 30
--osd_recovery_max_active 45 --osd_recovery_op_priority 10'
Then to set to back to default:
ceph tell 'osd.*' injectargs '--osd_max_backfills 1
--osd_recovery_max_active 3 --osd_recovery_op_priority 3'
Cheers
Francois Scheurer
________________________________________
From: Thomas Schneider <74cmonty(a)gmail.com>
Sent: Wednesday, March 4, 2020 11:15 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Forcibly move PGs from full to empty OSD
Hi,
Ceph balancer is not working correctly; there's an open bug
<https://tracker.ceph.com/issues/43752> report, too.
Until this issue is not solved, I need a workaround because I get more
and more warnings about "nearfull osd(s)".
Therefore my question is:
How can I forcibly move PGs from full OSD to empty OSD?
THX
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1510
days inactive
1522
days old
8 comments
5 participants
participants (5)
-
Anthony D'Atri -
Rich Bade -
Scheurer François -
Thomas Schneider -
Wido den Hollander