enabling pg_autoscaler on a large production storage?
Hi, I would like to enable the pg_autoscaler on our nautilus cluster. Someone told me that I should be really really careful to NOT have customer impact. Maybe someone can share some experience on this? The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte raw storage where ~600TB raw is used.
Hi, I agree with "someone" -- it's not a good idea to just naively enable pg_autoscaler on an existing cluster with lots of data and active customers. If you're curious about this feature, it would be harmless to start out by enabling it with pg_autoscale_mode = warn on each pool. This way you can see what the autoscaler would do if it were set to *on*. Then you can tweak all the target_size_ratio or target_size_bytes accordingly. BTW, do you have some feeling that your 17000 PGs are currently not correctly proportioned for your cluster? -- Dan On Tue, Jun 16, 2020 at 11:31 AM Boris Behrens <bb@kervyn.de> wrote:
Hi,
I would like to enable the pg_autoscaler on our nautilus cluster. Someone told me that I should be really really careful to NOT have customer impact.
Maybe someone can share some experience on this?
The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte raw storage where ~600TB raw is used. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I read about the "warm" option and we are already discussing this. I don't know if the pgs needs a tuning. I don't know what the impact is and if there will be any difference if we enable it. The, meanwhile gone, last ceph admin made a ticket, and I am not particularly familiar with ceph. So I need to work on this ticket and I try not to trash our ceph storage :-) Am Di., 16. Juni 2020 um 11:39 Uhr schrieb Dan van der Ster <dan@vanderster.com>:
Hi,
I agree with "someone" -- it's not a good idea to just naively enable pg_autoscaler on an existing cluster with lots of data and active customers.
If you're curious about this feature, it would be harmless to start out by enabling it with pg_autoscale_mode = warn on each pool. This way you can see what the autoscaler would do if it were set to *on*. Then you can tweak all the target_size_ratio or target_size_bytes accordingly.
BTW, do you have some feeling that your 17000 PGs are currently not correctly proportioned for your cluster?
-- Dan
On Tue, Jun 16, 2020 at 11:31 AM Boris Behrens <bb@kervyn.de> wrote:
Hi,
I would like to enable the pg_autoscaler on our nautilus cluster. Someone told me that I should be really really careful to NOT have customer impact.
Maybe someone can share some experience on this?
The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte raw storage where ~600TB raw is used. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Could you share the output of ceph osd pool ls detail ? This way we can see how the pools are configured and help recommend if pg_autoscaler is worth enabling. Cheers, Dan On Tue, Jun 16, 2020 at 11:51 AM Boris Behrens <bb@kervyn.de> wrote:
I read about the "warm" option and we are already discussing this.
I don't know if the pgs needs a tuning. I don't know what the impact is and if there will be any difference if we enable it.
The, meanwhile gone, last ceph admin made a ticket, and I am not particularly familiar with ceph. So I need to work on this ticket and I try not to trash our ceph storage :-)
Am Di., 16. Juni 2020 um 11:39 Uhr schrieb Dan van der Ster <dan@vanderster.com>:
Hi,
I agree with "someone" -- it's not a good idea to just naively enable pg_autoscaler on an existing cluster with lots of data and active customers.
If you're curious about this feature, it would be harmless to start out by enabling it with pg_autoscale_mode = warn on each pool. This way you can see what the autoscaler would do if it were set to *on*. Then you can tweak all the target_size_ratio or target_size_bytes accordingly.
BTW, do you have some feeling that your 17000 PGs are currently not correctly proportioned for your cluster?
-- Dan
On Tue, Jun 16, 2020 at 11:31 AM Boris Behrens <bb@kervyn.de> wrote:
Hi,
I would like to enable the pg_autoscaler on our nautilus cluster. Someone told me that I should be really really careful to NOT have customer impact.
Maybe someone can share some experience on this?
The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte raw storage where ~600TB raw is used. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
I did this on my cluster and there was a huge number of pg rebalanced. I think setting this option to 'on' is a good idea if it's a brand new cluster. Dan van der Ster <dan@vanderster.com> 于2020年6月16日周二 下午7:07写道：
Could you share the output of
ceph osd pool ls detail
?
This way we can see how the pools are configured and help recommend if pg_autoscaler is worth enabling.
Cheers, Dan
On Tue, Jun 16, 2020 at 11:51 AM Boris Behrens <bb@kervyn.de> wrote:
I read about the "warm" option and we are already discussing this.
I don't know if the pgs needs a tuning. I don't know what the impact is and if there will be any difference if we enable it.
The, meanwhile gone, last ceph admin made a ticket, and I am not particularly familiar with ceph. So I need to work on this ticket and I try not to trash our ceph storage :-)
Am Di., 16. Juni 2020 um 11:39 Uhr schrieb Dan van der Ster <dan@vanderster.com>:
Hi,
I agree with "someone" -- it's not a good idea to just naively enable pg_autoscaler on an existing cluster with lots of data and active customers.
If you're curious about this feature, it would be harmless to start out by enabling it with pg_autoscale_mode = warn on each pool. This way you can see what the autoscaler would do if it were set to *on*. Then you can tweak all the target_size_ratio or target_size_bytes accordingly.
BTW, do you have some feeling that your 17000 PGs are currently not correctly proportioned for your cluster?
-- Dan
On Tue, Jun 16, 2020 at 11:31 AM Boris Behrens <bb@kervyn.de> wrote:
Hi,
I would like to enable the pg_autoscaler on our nautilus cluster. Someone told me that I should be really really careful to NOT have customer impact.
Maybe someone can share some experience on this?
The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte raw storage where ~600TB raw is used. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
See inline comments Am Di., 16. Juni 2020 um 13:29 Uhr schrieb Zhenshi Zhou <deaderzzs@gmail.com>:
I did this on my cluster and there was a huge number of pg rebalanced. I think setting this option to 'on' is a good idea if it's a brand new cluster.
On our new cluster we enabled them, but not on our primary cluster with the most. Dan van der Ster <dan@vanderster.com> 于2020年6月16日周二 下午7:07写道：
Could you share the output of
ceph osd pool ls detail
pool 1 'pool 1' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 3 'pool 3' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change 2544040 lfor 0/0/1952329 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd pool 4 'pool 4' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 5 'pool 5' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd the mgr module is not enabled yet.
?
This way we can see how the pools are configured and help recommend if pg_autoscaler is worth enabling.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
On Tue, Jun 16, 2020 at 2:00 PM Boris Behrens <bb@kervyn.de> wrote:
See inline comments
Am Di., 16. Juni 2020 um 13:29 Uhr schrieb Zhenshi Zhou <deaderzzs@gmail.com>:
I did this on my cluster and there was a huge number of pg rebalanced. I think setting this option to 'on' is a good idea if it's a brand new cluster.
On our new cluster we enabled them, but not on our primary cluster with the most.
Dan van der Ster <dan@vanderster.com> 于2020年6月16日周二 下午7:07写道：
Could you share the output of
ceph osd pool ls detail
pool 1 'pool 1' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 3 'pool 3' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change 2544040 lfor 0/0/1952329 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd pool 4 'pool 4' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 5 'pool 5' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd
OK now maybe share the output of `ceph df` so we can see how much data is in each pool? Assuming that the majority of your data is in 'pool 3' with 16384 PGs, then your current PG values are just fine. (You should have around 110 PGs per OSD). The pg_autoscaler aims for 100 per OSD and doesn't make changes unless a pool has 4x too few or too many PGs. Unless you are planning to put a large proportion of data into the other pools, I'd leave pg_autoscaler disabled and move on to the next task. -- Dan
the mgr module is not enabled yet.
?
This way we can see how the pools are configured and help recommend if pg_autoscaler is worth enabling.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Oh ok. Because we have two types of ssds. 1.8TB and 3.6TB. The 1.8TB got around 90-100pgs and the 3.6TB around 150-190pgs Here is the output: RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 956 TiB 360 TiB 595 TiB 596 TiB 62.35 TOTAL 956 TiB 360 TiB 595 TiB 596 TiB 62.35 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL pool 1 1 580 GiB 149.31k 1.7 TiB 0.58 96 TiB pool 3 3 208 TiB 61.66M 593 TiB 67.21 96 TiB pool 4 4 7.9 MiB 2.01k 68 MiB 0 96 TiB pool 5 5 19 B 2 36 KiB 0 96 TiB Am Di., 16. Juni 2020 um 14:13 Uhr schrieb Dan van der Ster <dan@vanderster.com>:
On Tue, Jun 16, 2020 at 2:00 PM Boris Behrens <bb@kervyn.de> wrote:
See inline comments Dan van der Ster <dan@vanderster.com> 于2020年6月16日周二 下午7:07写道：
Could you share the output of
ceph osd pool ls detail
pool 1 'pool 1' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 3 'pool 3' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change 2544040 lfor 0/0/1952329 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd pool 4 'pool 4' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 5 'pool 5' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd
OK now maybe share the output of `ceph df` so we can see how much data is in each pool?
Assuming that the majority of your data is in 'pool 3' with 16384 PGs, then your current PG values are just fine. (You should have around 110 PGs per OSD). The pg_autoscaler aims for 100 per OSD and doesn't make changes unless a pool has 4x too few or too many PGs.
Unless you are planning to put a large proportion of data into the other pools, I'd leave pg_autoscaler disabled and move on to the next task.
-- Dan
the mgr module is not enabled yet.
?
This way we can see how the pools are configured and help recommend if pg_autoscaler is worth enabling.
Those df's and PG numbers all look fine to me. I wouldn't start adjusting pg_num now -- leave the autoscaler module disabled. Some might be concerned about having 190 PGs on an OSD, but this is fine provided you have ample memory (at least 3GB per OSD). Cheers, Dan On Tue, Jun 16, 2020 at 2:23 PM Boris Behrens <bb@kervyn.de> wrote:
Oh ok. Because we have two types of ssds. 1.8TB and 3.6TB. The 1.8TB got around 90-100pgs and the 3.6TB around 150-190pgs
Here is the output: RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 956 TiB 360 TiB 595 TiB 596 TiB 62.35 TOTAL 956 TiB 360 TiB 595 TiB 596 TiB 62.35
POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL pool 1 1 580 GiB 149.31k 1.7 TiB 0.58 96 TiB pool 3 3 208 TiB 61.66M 593 TiB 67.21 96 TiB pool 4 4 7.9 MiB 2.01k 68 MiB 0 96 TiB pool 5 5 19 B 2 36 KiB 0 96 TiB
Am Di., 16. Juni 2020 um 14:13 Uhr schrieb Dan van der Ster <dan@vanderster.com>:
On Tue, Jun 16, 2020 at 2:00 PM Boris Behrens <bb@kervyn.de> wrote:
See inline comments Dan van der Ster <dan@vanderster.com> 于2020年6月16日周二 下午7:07写道：
Could you share the output of
ceph osd pool ls detail
pool 1 'pool 1' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 3 'pool 3' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change 2544040 lfor 0/0/1952329 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd pool 4 'pool 4' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 5 'pool 5' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd
OK now maybe share the output of `ceph df` so we can see how much data is in each pool?
Assuming that the majority of your data is in 'pool 3' with 16384 PGs, then your current PG values are just fine. (You should have around 110 PGs per OSD). The pg_autoscaler aims for 100 per OSD and doesn't make changes unless a pool has 4x too few or too many PGs.
Unless you are planning to put a large proportion of data into the other pools, I'd leave pg_autoscaler disabled and move on to the next task.
-- Dan
the mgr module is not enabled yet.
?
This way we can see how the pools are configured and help recommend if pg_autoscaler is worth enabling.
participants (3)
-
Boris Behrens
-
Dan van der Ster
-
Zhenshi Zhou