Balancer blocked as autoscaler not acting on scaling change
Hi Folks, We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is about 86% full but the average is 58% full. However, the balancer is skipping a pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this for the last 20 days, it works through a list of objects that are misplaced but when it gets close to the end, more objects get added to the list. This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs active+remapped+backfilling, one PG completed the backfilling then the list shot up to c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling. Has anyone come across this behaviour before? If so, what was your remediation? Thanks in advance for sharing. Bruno Cluster details: 3,068 OSDs when all running, c. 60 per storage node OS: Ubuntu 20.04 Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive Use case: S3 storage and OpenStack backend, all pools three-way replicated
Hi an idea is to see what Ceph osd test-reweight-by-utilization shows. If it looks usefull you can run the above command without "test" Hth Mehmet Am 22. September 2023 11:22:39 MESZ schrieb bc10@sanger.ac.uk:
Hi Folks,
We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is about 86% full but the average is 58% full. However, the balancer is skipping a pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this for the last 20 days, it works through a list of objects that are misplaced but when it gets close to the end, more objects get added to the list.
This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs active+remapped+backfilling, one PG completed the backfilling then the list shot up to c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling.
Has anyone come across this behaviour before? If so, what was your remediation?
Thanks in advance for sharing. Bruno
Cluster details: 3,068 OSDs when all running, c. 60 per storage node OS: Ubuntu 20.04 Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
Use case: S3 storage and OpenStack backend, all pools three-way replicated _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Note that this will adjust override reweight values, which will conflict with balancer upmaps.
On Sep 26, 2023, at 3:51 AM, ceph@elchaka.de wrote:
Hi an idea is to see what
Ceph osd test-reweight-by-utilization shows. If it looks usefull you can run the above command without "test"
Hth Mehmet
Am 22. September 2023 11:22:39 MESZ schrieb bc10@sanger.ac.uk:
Hi Folks,
We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is about 86% full but the average is 58% full. However, the balancer is skipping a pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this for the last 20 days, it works through a list of objects that are misplaced but when it gets close to the end, more objects get added to the list.
This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs active+remapped+backfilling, one PG completed the backfilling then the list shot up to c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling.
Has anyone come across this behaviour before? If so, what was your remediation?
Thanks in advance for sharing. Bruno
Cluster details: 3,068 OSDs when all running, c. 60 per storage node OS: Ubuntu 20.04 Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
Use case: S3 storage and OpenStack backend, all pools three-way replicated _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, you could change the target_max_misplaced_ratio to 1, the balancer has a default 5% ratio of misplaced objects, see [1] for more information: ceph config get mgr target_max_misplaced_ratio 0.050000 [1] https://docs.ceph.com/en/latest/rados/operations/balancer/#throttling Zitat von bc10@sanger.ac.uk:
Hi Folks,
We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is about 86% full but the average is 58% full. However, the balancer is skipping a pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this for the last 20 days, it works through a list of objects that are misplaced but when it gets close to the end, more objects get added to the list.
This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs active+remapped+backfilling, one PG completed the backfilling then the list shot up to c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling.
Has anyone come across this behaviour before? If so, what was your remediation?
Thanks in advance for sharing. Bruno
Cluster details: 3,068 OSDs when all running, c. 60 per storage node OS: Ubuntu 20.04 Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
Use case: S3 storage and OpenStack backend, all pools three-way replicated _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, we have often seen strange behavior and also interesting pg targets from pg_autoscaler in the last years. That's why we disable it globally. The commands: ceph osd reweight-by-utilization ceph osd test-reweight-by-utilization are from the time before the upmap balancer was introduced and did not solve the problem in the long run in an active cluster. That the balancer skips the pool with the rebalancing, I've seen more than once. Why pg_autoscaler behaves this way would have to be analyzed in more detail. As mentioned above, we normally turn it off. The idea of Eugen would help when the pool rebalancing is done. Joachim ___________________________________ ceph ambassador DACH ceph consultant since 2012 Clyso GmbH - Premier Ceph Foundation Member https://www.clyso.com/ Am 22.09.23 um 11:22 schrieb bc10@sanger.ac.uk:
Hi Folks,
We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is about 86% full but the average is 58% full. However, the balancer is skipping a pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this for the last 20 days, it works through a list of objects that are misplaced but when it gets close to the end, more objects get added to the list.
This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs active+remapped+backfilling, one PG completed the backfilling then the list shot up to c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling.
Has anyone come across this behaviour before? If so, what was your remediation?
Thanks in advance for sharing. Bruno
participants (5)
-
Anthony D'Atri
-
bc10＠sanger.ac.uk
-
ceph＠elchaka.de
-
Eugen Block
-
Joachim Kraftmayer - ceph ambassador