22 Sep
2023
22 Sep
'23
12:22 p.m.
Hi Folks,
We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is
about 86% full but the average is 58% full. However, the balancer is skipping a pool on
which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768
(default.rgw.buckets.data pool). However, the autoscaler has been working on this for the
last 20 days, it works through a list of objects that are misplaced but when it gets close
to the end, more objects get added to the list.
This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs
active+remapped+backfilling, one PG completed the backfilling then the list shot up to c.
70,000 objects misplaced with 3 PGs active+remapped+backfilling.
Has anyone come across this behaviour before? If so, what was your remediation?
Thanks in advance for sharing.
Bruno
Cluster details:
3,068 OSDs when all running, c. 60 per storage node
OS: Ubuntu 20.04
Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
Use case:
S3 storage and OpenStack backend, all pools three-way replicated
26 Sep
26 Sep
10:49 a.m.
Hi an idea is to see what
Ceph osd test-reweight-by-utilization
shows.
If it looks usefull you can run the above command without "test"
Hth
Mehmet
Am 22. September 2023 11:22:39 MESZ schrieb bc10(a)sanger.ac.uk:
Hi Folks,
We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is
about 86% full but the average is 58% full. However, the balancer is skipping a pool on
which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768
(default.rgw.buckets.data pool). However, the autoscaler has been working on this for the
last 20 days, it works through a list of objects that are misplaced but when it gets close
to the end, more objects get added to the list.
This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs
active+remapped+backfilling, one PG completed the backfilling then the list shot up to c.
70,000 objects misplaced with 3 PGs active+remapped+backfilling.
Has anyone come across this behaviour before? If so, what was your remediation?
Thanks in advance for sharing.
Bruno
Cluster details:
3,068 OSDs when all running, c. 60 per storage node
OS: Ubuntu 20.04
Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
Use case:
S3 storage and OpenStack backend, all pools three-way replicated
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:32 p.m.
Note that this will adjust override reweight values, which will conflict with balancer
upmaps.
> On Sep 26, 2023, at 3:51 AM, ceph(a)elchaka.de wrote:
>
> Hi an idea is to see what
>
> Ceph osd test-reweight-by-utilization
> shows.
> If it looks usefull you can run the above command without "test"
>
> Hth
> Mehmet
>
> Am 22. September 2023 11:22:39 MESZ schrieb bc10(a)sanger.ac.uk:
>> Hi Folks,
>>
>> We are currently running with one nearfull OSD and 15 nearfull pools. The most
full OSD is about 86% full but the average is 58% full. However, the balancer is skipping
a pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to
32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this
for the last 20 days, it works through a list of objects that are misplaced but when it
gets close to the end, more objects get added to the list.
>>
>> This morning I observed the list get down to c. 7,000 objects misplaced with 2
PGs active+remapped+backfilling, one PG completed the backfilling then the list shot up to
c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling.
>>
>> Has anyone come across this behaviour before? If so, what was your remediation?
>>
>> Thanks in advance for sharing.
>> Bruno
>>
>> Cluster details:
>> 3,068 OSDs when all running, c. 60 per storage node
>> OS: Ubuntu 20.04
>> Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
>>
>> Use case:
>> S3 storage and OpenStack backend, all pools three-way replicated
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
4 Oct
4 Oct
10:40 a.m.
Hi,
you could change the target_max_misplaced_ratio to 1, the balancer has
a default 5% ratio of misplaced objects, see [1] for more information:
ceph config get mgr target_max_misplaced_ratio
0.050000
[1] https://docs.ceph.com/en/latest/rados/operations/balancer/#throttling
Zitat von bc10(a)sanger.ac.uk:
Hi Folks,
We are currently running with one nearfull OSD and 15 nearfull
pools. The most full OSD is about 86% full but the average is 58%
full. However, the balancer is skipping a pool on which the
autoscaler is trying to complete a pg_num reduction from 131,072 to
32,768 (default.rgw.buckets.data pool). However, the autoscaler has
been working on this for the last 20 days, it works through a list
of objects that are misplaced but when it gets close to the end,
more objects get added to the list.
This morning I observed the list get down to c. 7,000 objects
misplaced with 2 PGs active+remapped+backfilling, one PG completed
the backfilling then the list shot up to c. 70,000 objects misplaced
with 3 PGs active+remapped+backfilling.
Has anyone come across this behaviour before? If so, what was your
remediation?
Thanks in advance for sharing.
Bruno
Cluster details:
3,068 OSDs when all running, c. 60 per storage node
OS: Ubuntu 20.04
Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
Use case:
S3 storage and OpenStack backend, all pools three-way replicated
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:55 a.m.
Hi,
we have often seen strange behavior and also interesting pg targets from
pg_autoscaler in the last years.
That's why we disable it globally.
The commands:
ceph osd reweight-by-utilization
ceph osd test-reweight-by-utilization
are from the time before the upmap balancer was introduced and did not
solve the problem in the long run in an active cluster.
That the balancer skips the pool with the rebalancing, I've seen more
than once.
Why pg_autoscaler behaves this way would have to be analyzed in more
detail. As mentioned above, we normally turn it off.
The idea of Eugen would help when the pool rebalancing is done.
Joachim
___________________________________
ceph ambassador DACH
ceph consultant since 2012
Clyso GmbH - Premier Ceph Foundation Member
https://www.clyso.com/
Am 22.09.23 um 11:22 schrieb bc10(a)sanger.ac.uk:
> Hi Folks,
>
> We are currently running with one nearfull OSD and 15 nearfull pools. The most full
OSD is about 86% full but the average is 58% full. However, the balancer is skipping a
pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to
32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this
for the last 20 days, it works through a list of objects that are misplaced but when it
gets close to the end, more objects get added to the list.
>
> This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs
active+remapped+backfilling, one PG completed the backfilling then the list shot up to c.
70,000 objects misplaced with 3 PGs active+remapped+backfilling.
>
> Has anyone come across this behaviour before? If so, what was your remediation?
>
> Thanks in advance for sharing.
> Bruno
229
days inactive
241
days old
4 comments
5 participants
participants (5)
-
Anthony D'Atri -
bc10＠sanger.ac.uk -
ceph＠elchaka.de -
Eugen Block -
Joachim Kraftmayer - ceph ambassador