subtrees have overcommitted (target_size_bytes / target_size_ratio)
Hello everybody! What does this mean? health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away. $ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on $ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data'] Thanks Lars
My question requires too complex an answer. So let me ask a simple question: What does the SIZE of "osd pool autoscale-status" tell/mean/comes from? Thanks Lars Wed, 23 Oct 2019 14:28:10 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io :
Hello everybody!
What does this mean?
health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio
and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away.
$ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on
$ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data']
Thanks Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de
This question is answered here: https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ But it tells me that there is more data stored in the pool than the raw capacity provides (taking the replication factor RATE into account) hence the RATIO being above 1.0 . How comes this is the case? - Data is stored outside of the pool? How comes this is only the case when the autoscaler is active? Thanks Lars Thu, 24 Oct 2019 10:36:52 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io :
My question requires too complex an answer. So let me ask a simple question:
What does the SIZE of "osd pool autoscale-status" tell/mean/comes from?
Thanks Lars
Wed, 23 Oct 2019 14:28:10 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io :
Hello everybody!
What does this mean?
health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio
and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away.
$ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on
$ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data']
Thanks Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de
The formatting is mangled on my phone, but if I am reading it correctly, you have set Target Ratio to 4.0. This means you have told the balancer that this pool will occupy 4x the space of your whole cluster, and to optimize accordingly. This is naturally a problem. Setting it to 0 will clear the setting and allow the autobalancer to work. On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber@bbaw.de> wrote:
This question is answered here: https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/
But it tells me that there is more data stored in the pool than the raw capacity provides (taking the replication factor RATE into account) hence the RATIO being above 1.0 .
How comes this is the case? - Data is stored outside of the pool? How comes this is only the case when the autoscaler is active?
Thanks Lars
My question requires too complex an answer. So let me ask a simple question:
What does the SIZE of "osd pool autoscale-status" tell/mean/comes from?
Thanks Lars
Wed, 23 Oct 2019 14:28:10 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io :
Hello everybody!
What does this mean?
health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio
and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away.
$ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on
$ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted
Pools ['cephfs_data'] overcommit available storage by 1.031x due
to target_size_bytes 0 on pools []
POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted
Thu, 24 Oct 2019 10:36:52 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : pool target_size_bytes pool target_size_ratio
Pools ['cephfs_data'] overcommit available storage by 1.031x due
to target_size_ratio 0.900 on pools ['cephfs_data']
Thanks Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Nathan for your answer, but I set the the Target Ratio to 0.9. It is the cephfs_data pool that makes the troubles. The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is not explained on the page linked below. So I don't know its meaning. How can be a pool overcommited when it is the only pool on a set of OSDs? Best regards, Lars Thu, 24 Oct 2019 09:39:51 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
The formatting is mangled on my phone, but if I am reading it correctly, you have set Target Ratio to 4.0. This means you have told the balancer that this pool will occupy 4x the space of your whole cluster, and to optimize accordingly. This is naturally a problem. Setting it to 0 will clear the setting and allow the autobalancer to work.
On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber@bbaw.de> wrote:
This question is answered here: https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/
But it tells me that there is more data stored in the pool than the raw capacity provides (taking the replication factor RATE into account) hence the RATIO being above 1.0 .
How comes this is the case? - Data is stored outside of the pool? How comes this is only the case when the autoscaler is active?
Thanks Lars
My question requires too complex an answer. So let me ask a simple question:
What does the SIZE of "osd pool autoscale-status" tell/mean/comes from?
Thanks Lars
Wed, 23 Oct 2019 14:28:10 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io :
Hello everybody!
What does this mean?
health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio
and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away.
$ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on
$ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted
Pools ['cephfs_data'] overcommit available storage by 1.031x due
to target_size_bytes 0 on pools []
POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted
Thu, 24 Oct 2019 10:36:52 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : pool target_size_bytes pool target_size_ratio
Pools ['cephfs_data'] overcommit available storage by 1.031x due
to target_size_ratio 0.900 on pools ['cephfs_data']
Thanks Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ah, I see! The BIAS reflects the number of placement groups it should create. Since cephfs metadata pools are usually very small, but have many objects and high IO, the autoscaler gives them 4x the number of placement groups that it would normally give for that amount of data. So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to 0.3? Are the two pools using entirely different device classes, so they are not sharing space? Anyway, I see that your overcommit is only "1.031x". So if you set cephfs_data to 0.85, it should go away. On Thu, Oct 24, 2019 at 10:09 AM Lars Täuber <taeuber@bbaw.de> wrote:
Thanks Nathan for your answer,
but I set the the Target Ratio to 0.9. It is the cephfs_data pool that makes the troubles.
The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is not explained on the page linked below. So I don't know its meaning.
How can be a pool overcommited when it is the only pool on a set of OSDs?
Best regards, Lars
Thu, 24 Oct 2019 09:39:51 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
The formatting is mangled on my phone, but if I am reading it correctly, you have set Target Ratio to 4.0. This means you have told the balancer that this pool will occupy 4x the space of your whole cluster, and to optimize accordingly. This is naturally a problem. Setting it to 0 will clear the setting and allow the autobalancer to work.
On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber@bbaw.de> wrote:
This question is answered here: https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/
But it tells me that there is more data stored in the pool than the raw capacity provides (taking the replication factor RATE into account) hence the RATIO being above 1.0 .
How comes this is the case? - Data is stored outside of the pool? How comes this is only the case when the autoscaler is active?
Thanks Lars
My question requires too complex an answer. So let me ask a simple question:
What does the SIZE of "osd pool autoscale-status" tell/mean/comes from?
Thanks Lars
Wed, 23 Oct 2019 14:28:10 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io :
Hello everybody!
What does this mean?
health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio
and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away.
$ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on
$ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted
Pools ['cephfs_data'] overcommit available storage by 1.031x due
to target_size_bytes 0 on pools []
POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted
Thu, 24 Oct 2019 10:36:52 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : pool target_size_bytes pool target_size_ratio
Pools ['cephfs_data'] overcommit available storage by 1.031x due
to target_size_ratio 0.900 on pools ['cephfs_data']
Thanks Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Nathan, Thu, 24 Oct 2019 10:59:55 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
Ah, I see! The BIAS reflects the number of placement groups it should create. Since cephfs metadata pools are usually very small, but have many objects and high IO, the autoscaler gives them 4x the number of placement groups that it would normally give for that amount of data.
ah ok, I understand.
So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to 0.3? Are the two pools using entirely different device classes, so they are not sharing space?
Yes, the metadata is on SSDs and the data on HDDs.
Anyway, I see that your overcommit is only "1.031x". So if you set cephfs_data to 0.85, it should go away.
This is not the case. I set the target_ratio to 0.7 and get this: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15736M 3.0 2454G 0.0188 0.3000 4.0 256 on cephfs_data 122.2T 1.5 165.4T 1.1085 0.7000 1.0 1024 on The ratio seems to have nothing to do with the target_ratio but the SIZE and the RAW_CAPACITY. Because the pool is still getting more data the SIZE increases and therefore the RATIO increases. The RATIO seems to be calculated by this formula RATIO = SIZE * RATE / RAW_CAPACITY. This is what I don't understand. The data in the cephfs_data pool seems to need more space than the raw capacity of the cluster provides. Hence the situation is called "overcommitment". But why is this only the case when the autoscaler is active? Thanks Lars
On Thu, Oct 24, 2019 at 10:09 AM Lars Täuber <taeuber@bbaw.de> wrote:
Thanks Nathan for your answer,
but I set the the Target Ratio to 0.9. It is the cephfs_data pool that makes the troubles.
The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is not explained on the page linked below. So I don't know its meaning.
How can be a pool overcommited when it is the only pool on a set of OSDs?
Best regards, Lars
Thu, 24 Oct 2019 09:39:51 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
The formatting is mangled on my phone, but if I am reading it correctly, you have set Target Ratio to 4.0. This means you have told the balancer that this pool will occupy 4x the space of your whole cluster, and to optimize accordingly. This is naturally a problem. Setting it to 0 will clear the setting and allow the autobalancer to work.
On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber@bbaw.de> wrote:
This question is answered here: https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/
But it tells me that there is more data stored in the pool than the raw capacity provides (taking the replication factor RATE into account) hence the RATIO being above 1.0 .
How comes this is the case? - Data is stored outside of the pool? How comes this is only the case when the autoscaler is active?
Thanks Lars
My question requires too complex an answer. So let me ask a simple question:
What does the SIZE of "osd pool autoscale-status" tell/mean/comes from?
Thanks Lars
Wed, 23 Oct 2019 14:28:10 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io :
Hello everybody!
What does this mean?
health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio
and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away.
$ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on
$ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted
Pools ['cephfs_data'] overcommit available storage by 1.031x due
to target_size_bytes 0 on pools []
POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted
Thu, 24 Oct 2019 10:36:52 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : pool target_size_bytes pool target_size_ratio
Pools ['cephfs_data'] overcommit available storage by 1.031x due
to target_size_ratio 0.900 on pools ['cephfs_data']
Thanks Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de
Is there a way to get rid of this warnings with activated autoscaler besides adding new osds? Yet I couldn't get a satisfactory answer to the question why this all happens. ceph osd pool autoscale-status : POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_data 122.2T 1.5 165.4T 1.1085 0.8500 1.0 1024 on versus ceph df : RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 165 TiB 41 TiB 124 TiB 124 TiB 74.95 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 75 TiB 49.31M 122 TiB 87.16 12 TiB It seems that the overcommitment is wrongly calculated. Isn't the RATE already used to calculate the SIZE? It seems USED(df) = SIZE(autoscale-status) Isn't the RATE already taken into account here? Could someone please explain the numbers to me? Thanks! Lars Fri, 25 Oct 2019 07:42:58 +0200 Lars Täuber <taeuber@bbaw.de> ==> Nathan Fish <lordcirth@gmail.com> :
Hi Nathan,
Thu, 24 Oct 2019 10:59:55 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
Ah, I see! The BIAS reflects the number of placement groups it should create. Since cephfs metadata pools are usually very small, but have many objects and high IO, the autoscaler gives them 4x the number of placement groups that it would normally give for that amount of data.
ah ok, I understand.
So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to 0.3? Are the two pools using entirely different device classes, so they are not sharing space?
Yes, the metadata is on SSDs and the data on HDDs.
Anyway, I see that your overcommit is only "1.031x". So if you set cephfs_data to 0.85, it should go away.
This is not the case. I set the target_ratio to 0.7 and get this:
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15736M 3.0 2454G 0.0188 0.3000 4.0 256 on cephfs_data 122.2T 1.5 165.4T 1.1085 0.7000 1.0 1024 on
The ratio seems to have nothing to do with the target_ratio but the SIZE and the RAW_CAPACITY. Because the pool is still getting more data the SIZE increases and therefore the RATIO increases. The RATIO seems to be calculated by this formula RATIO = SIZE * RATE / RAW_CAPACITY.
This is what I don't understand. The data in the cephfs_data pool seems to need more space than the raw capacity of the cluster provides. Hence the situation is called "overcommitment".
But why is this only the case when the autoscaler is active?
Thanks Lars
On Thu, Oct 24, 2019 at 10:09 AM Lars Täuber <taeuber@bbaw.de> wrote:
Thanks Nathan for your answer,
but I set the the Target Ratio to 0.9. It is the cephfs_data pool that makes the troubles.
The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is not explained on the page linked below. So I don't know its meaning.
How can be a pool overcommited when it is the only pool on a set of OSDs?
Best regards, Lars
Thu, 24 Oct 2019 09:39:51 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
The formatting is mangled on my phone, but if I am reading it correctly, you have set Target Ratio to 4.0. This means you have told the balancer that this pool will occupy 4x the space of your whole cluster, and to optimize accordingly. This is naturally a problem. Setting it to 0 will clear the setting and allow the autobalancer to work.
On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber@bbaw.de> wrote:
This question is answered here: https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/
But it tells me that there is more data stored in the pool than the raw capacity provides (taking the replication factor RATE into account) hence the RATIO being above 1.0 .
How comes this is the case? - Data is stored outside of the pool? How comes this is only the case when the autoscaler is active?
Thanks Lars
My question requires too complex an answer. So let me ask a simple question:
What does the SIZE of "osd pool autoscale-status" tell/mean/comes from?
Thanks Lars
Wed, 23 Oct 2019 14:28:10 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : > Hello everybody! > > What does this mean? > > health: HEALTH_WARN > 1 subtrees have overcommitted pool target_size_bytes > 1 subtrees have overcommitted pool target_size_ratio > > and what does it have to do with the autoscaler? > When I deactivate the autoscaler the warning goes away. > > > $ ceph osd pool autoscale-status > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on > cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on > > > $ ceph health detail > HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted
> Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted
Thu, 24 Oct 2019 10:36:52 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : pool target_size_bytes pool target_size_ratio
> Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data'] > > > Thanks > Lars > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
Is there anybody who can explain the overcommitment calcuation? Thanks Mon, 28 Oct 2019 11:24:54 +0100 Lars Täuber <taeuber@bbaw.de> ==> ceph-users <ceph-users@ceph.io> :
Is there a way to get rid of this warnings with activated autoscaler besides adding new osds?
Yet I couldn't get a satisfactory answer to the question why this all happens.
ceph osd pool autoscale-status : POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_data 122.2T 1.5 165.4T 1.1085 0.8500 1.0 1024 on
versus
ceph df : RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 165 TiB 41 TiB 124 TiB 124 TiB 74.95
POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 75 TiB 49.31M 122 TiB 87.16 12 TiB
It seems that the overcommitment is wrongly calculated. Isn't the RATE already used to calculate the SIZE?
It seems USED(df) = SIZE(autoscale-status) Isn't the RATE already taken into account here?
Could someone please explain the numbers to me?
Thanks! Lars
Fri, 25 Oct 2019 07:42:58 +0200 Lars Täuber <taeuber@bbaw.de> ==> Nathan Fish <lordcirth@gmail.com> :
Hi Nathan,
Thu, 24 Oct 2019 10:59:55 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
Ah, I see! The BIAS reflects the number of placement groups it should create. Since cephfs metadata pools are usually very small, but have many objects and high IO, the autoscaler gives them 4x the number of placement groups that it would normally give for that amount of data.
ah ok, I understand.
So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to 0.3? Are the two pools using entirely different device classes, so they are not sharing space?
Yes, the metadata is on SSDs and the data on HDDs.
Anyway, I see that your overcommit is only "1.031x". So if you set cephfs_data to 0.85, it should go away.
This is not the case. I set the target_ratio to 0.7 and get this:
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15736M 3.0 2454G 0.0188 0.3000 4.0 256 on cephfs_data 122.2T 1.5 165.4T 1.1085 0.7000 1.0 1024 on
The ratio seems to have nothing to do with the target_ratio but the SIZE and the RAW_CAPACITY. Because the pool is still getting more data the SIZE increases and therefore the RATIO increases. The RATIO seems to be calculated by this formula RATIO = SIZE * RATE / RAW_CAPACITY.
This is what I don't understand. The data in the cephfs_data pool seems to need more space than the raw capacity of the cluster provides. Hence the situation is called "overcommitment".
But why is this only the case when the autoscaler is active?
Thanks Lars
On Thu, Oct 24, 2019 at 10:09 AM Lars Täuber <taeuber@bbaw.de> wrote:
Thanks Nathan for your answer,
but I set the the Target Ratio to 0.9. It is the cephfs_data pool that makes the troubles.
The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is not explained on the page linked below. So I don't know its meaning.
How can be a pool overcommited when it is the only pool on a set of OSDs?
Best regards, Lars
Thu, 24 Oct 2019 09:39:51 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
The formatting is mangled on my phone, but if I am reading it correctly, you have set Target Ratio to 4.0. This means you have told the balancer that this pool will occupy 4x the space of your whole cluster, and to optimize accordingly. This is naturally a problem. Setting it to 0 will clear the setting and allow the autobalancer to work.
On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber@bbaw.de> wrote:
This question is answered here: https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/
But it tells me that there is more data stored in the pool than the raw capacity provides (taking the replication factor RATE into account) hence the RATIO being above 1.0 .
How comes this is the case? - Data is stored outside of the pool? How comes this is only the case when the autoscaler is active?
Thanks Lars
Thu, 24 Oct 2019 10:36:52 +0200 Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : > My question requires too complex an answer. > So let me ask a simple question: > > What does the SIZE of "osd pool autoscale-status" tell/mean/comes from? > > Thanks > Lars > > Wed, 23 Oct 2019 14:28:10 +0200 > Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : > > Hello everybody! > > > > What does this mean? > > > > health: HEALTH_WARN > > 1 subtrees have overcommitted pool target_size_bytes > > 1 subtrees have overcommitted pool target_size_ratio > > > > and what does it have to do with the autoscaler? > > When I deactivate the autoscaler the warning goes away. > > > > > > $ ceph osd pool autoscale-status > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on > > cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on > > > > > > $ ceph health detail > > HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes > > Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio > > Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data'] > > > > > > Thanks > > Lars
This was fixed a few weeks back. It should be resolved in 14.2.5. https://tracker.ceph.com/issues/41567 https://github.com/ceph/ceph/pull/31100 sage On Fri, 1 Nov 2019, Lars Täuber wrote:
Is there anybody who can explain the overcommitment calcuation?
Thanks
Mon, 28 Oct 2019 11:24:54 +0100 Lars Täuber <taeuber@bbaw.de> ==> ceph-users <ceph-users@ceph.io> :
Is there a way to get rid of this warnings with activated autoscaler besides adding new osds?
Yet I couldn't get a satisfactory answer to the question why this all happens.
ceph osd pool autoscale-status : POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_data 122.2T 1.5 165.4T 1.1085 0.8500 1.0 1024 on
versus
ceph df : RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 165 TiB 41 TiB 124 TiB 124 TiB 74.95
POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 75 TiB 49.31M 122 TiB 87.16 12 TiB
It seems that the overcommitment is wrongly calculated. Isn't the RATE already used to calculate the SIZE?
It seems USED(df) = SIZE(autoscale-status) Isn't the RATE already taken into account here?
Could someone please explain the numbers to me?
Thanks! Lars
Fri, 25 Oct 2019 07:42:58 +0200 Lars Täuber <taeuber@bbaw.de> ==> Nathan Fish <lordcirth@gmail.com> :
Hi Nathan,
Thu, 24 Oct 2019 10:59:55 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
Ah, I see! The BIAS reflects the number of placement groups it should create. Since cephfs metadata pools are usually very small, but have many objects and high IO, the autoscaler gives them 4x the number of placement groups that it would normally give for that amount of data.
ah ok, I understand.
So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to 0.3? Are the two pools using entirely different device classes, so they are not sharing space?
Yes, the metadata is on SSDs and the data on HDDs.
Anyway, I see that your overcommit is only "1.031x". So if you set cephfs_data to 0.85, it should go away.
This is not the case. I set the target_ratio to 0.7 and get this:
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15736M 3.0 2454G 0.0188 0.3000 4.0 256 on cephfs_data 122.2T 1.5 165.4T 1.1085 0.7000 1.0 1024 on
The ratio seems to have nothing to do with the target_ratio but the SIZE and the RAW_CAPACITY. Because the pool is still getting more data the SIZE increases and therefore the RATIO increases. The RATIO seems to be calculated by this formula RATIO = SIZE * RATE / RAW_CAPACITY.
This is what I don't understand. The data in the cephfs_data pool seems to need more space than the raw capacity of the cluster provides. Hence the situation is called "overcommitment".
But why is this only the case when the autoscaler is active?
Thanks Lars
On Thu, Oct 24, 2019 at 10:09 AM Lars Täuber <taeuber@bbaw.de> wrote:
Thanks Nathan for your answer,
but I set the the Target Ratio to 0.9. It is the cephfs_data pool that makes the troubles.
The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is not explained on the page linked below. So I don't know its meaning.
How can be a pool overcommited when it is the only pool on a set of OSDs?
Best regards, Lars
Thu, 24 Oct 2019 09:39:51 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
The formatting is mangled on my phone, but if I am reading it correctly, you have set Target Ratio to 4.0. This means you have told the balancer that this pool will occupy 4x the space of your whole cluster, and to optimize accordingly. This is naturally a problem. Setting it to 0 will clear the setting and allow the autobalancer to work.
On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber@bbaw.de> wrote:
> This question is answered here: > https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ > > But it tells me that there is more data stored in the pool than the raw > capacity provides (taking the replication factor RATE into account) hence > the RATIO being above 1.0 . > > How comes this is the case? - Data is stored outside of the pool? > How comes this is only the case when the autoscaler is active? > > Thanks > Lars > > > Thu, 24 Oct 2019 10:36:52 +0200 > Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : > > My question requires too complex an answer. > > So let me ask a simple question: > > > > What does the SIZE of "osd pool autoscale-status" tell/mean/comes from? > > > > Thanks > > Lars > > > > Wed, 23 Oct 2019 14:28:10 +0200 > > Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : > > > Hello everybody! > > > > > > What does this mean? > > > > > > health: HEALTH_WARN > > > 1 subtrees have overcommitted pool target_size_bytes > > > 1 subtrees have overcommitted pool target_size_ratio > > > > > > and what does it have to do with the autoscaler? > > > When I deactivate the autoscaler the warning goes away. > > > > > > > > > $ ceph osd pool autoscale-status > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > > cephfs_metadata 15106M 3.0 2454G 0.0180 > 0.3000 4.0 256 on > > > cephfs_data 113.6T 1.5 165.4T 1.0306 > 0.9000 1.0 512 on > > > > > > > > > $ ceph health detail > > > HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 > subtrees have overcommitted pool target_size_ratio > > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted > pool target_size_bytes > > > Pools ['cephfs_data'] overcommit available storage by 1.031x due > to target_size_bytes 0 on pools [] > > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted > pool target_size_ratio > > > Pools ['cephfs_data'] overcommit available storage by 1.031x due > to target_size_ratio 0.900 on pools ['cephfs_data'] > > > > > > > > > Thanks > > > Lars
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks a lot! Lars Fri, 1 Nov 2019 13:03:25 +0000 (UTC) Sage Weil <sage@newdream.net> ==> Lars Täuber <taeuber@bbaw.de> :
This was fixed a few weeks back. It should be resolved in 14.2.5.
https://tracker.ceph.com/issues/41567 https://github.com/ceph/ceph/pull/31100
sage
On Fri, 1 Nov 2019, Lars Täuber wrote:
Is there anybody who can explain the overcommitment calcuation?
Thanks
Mon, 28 Oct 2019 11:24:54 +0100 Lars Täuber <taeuber@bbaw.de> ==> ceph-users <ceph-users@ceph.io> :
Is there a way to get rid of this warnings with activated autoscaler besides adding new osds?
Yet I couldn't get a satisfactory answer to the question why this all happens.
ceph osd pool autoscale-status : POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_data 122.2T 1.5 165.4T 1.1085 0.8500 1.0 1024 on
versus
ceph df : RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 165 TiB 41 TiB 124 TiB 124 TiB 74.95
POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 75 TiB 49.31M 122 TiB 87.16 12 TiB
It seems that the overcommitment is wrongly calculated. Isn't the RATE already used to calculate the SIZE?
It seems USED(df) = SIZE(autoscale-status) Isn't the RATE already taken into account here?
Could someone please explain the numbers to me?
Thanks! Lars
Fri, 25 Oct 2019 07:42:58 +0200 Lars Täuber <taeuber@bbaw.de> ==> Nathan Fish <lordcirth@gmail.com> :
Hi Nathan,
Thu, 24 Oct 2019 10:59:55 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> :
Ah, I see! The BIAS reflects the number of placement groups it should create. Since cephfs metadata pools are usually very small, but have many objects and high IO, the autoscaler gives them 4x the number of placement groups that it would normally give for that amount of data.
ah ok, I understand.
So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to 0.3? Are the two pools using entirely different device classes, so they are not sharing space?
Yes, the metadata is on SSDs and the data on HDDs.
Anyway, I see that your overcommit is only "1.031x". So if you set cephfs_data to 0.85, it should go away.
This is not the case. I set the target_ratio to 0.7 and get this:
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15736M 3.0 2454G 0.0188 0.3000 4.0 256 on cephfs_data 122.2T 1.5 165.4T 1.1085 0.7000 1.0 1024 on
The ratio seems to have nothing to do with the target_ratio but the SIZE and the RAW_CAPACITY. Because the pool is still getting more data the SIZE increases and therefore the RATIO increases. The RATIO seems to be calculated by this formula RATIO = SIZE * RATE / RAW_CAPACITY.
This is what I don't understand. The data in the cephfs_data pool seems to need more space than the raw capacity of the cluster provides. Hence the situation is called "overcommitment".
But why is this only the case when the autoscaler is active?
Thanks Lars
On Thu, Oct 24, 2019 at 10:09 AM Lars Täuber <taeuber@bbaw.de> wrote:
Thanks Nathan for your answer,
but I set the the Target Ratio to 0.9. It is the cephfs_data pool that makes the troubles.
The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is not explained on the page linked below. So I don't know its meaning.
How can be a pool overcommited when it is the only pool on a set of OSDs?
Best regards, Lars
Thu, 24 Oct 2019 09:39:51 -0400 Nathan Fish <lordcirth@gmail.com> ==> Lars Täuber <taeuber@bbaw.de> : > The formatting is mangled on my phone, but if I am reading it correctly, > you have set Target Ratio to 4.0. This means you have told the balancer > that this pool will occupy 4x the space of your whole cluster, and to > optimize accordingly. This is naturally a problem. Setting it to 0 will > clear the setting and allow the autobalancer to work. > > On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber@bbaw.de> wrote: > > > This question is answered here: > > https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ > > > > But it tells me that there is more data stored in the pool than the raw > > capacity provides (taking the replication factor RATE into account) hence > > the RATIO being above 1.0 . > > > > How comes this is the case? - Data is stored outside of the pool? > > How comes this is only the case when the autoscaler is active? > > > > Thanks > > Lars > > > > > > Thu, 24 Oct 2019 10:36:52 +0200 > > Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : > > > My question requires too complex an answer. > > > So let me ask a simple question: > > > > > > What does the SIZE of "osd pool autoscale-status" tell/mean/comes from? > > > > > > Thanks > > > Lars > > > > > > Wed, 23 Oct 2019 14:28:10 +0200 > > > Lars Täuber <taeuber@bbaw.de> ==> ceph-users@ceph.io : > > > > Hello everybody! > > > > > > > > What does this mean? > > > > > > > > health: HEALTH_WARN > > > > 1 subtrees have overcommitted pool target_size_bytes > > > > 1 subtrees have overcommitted pool target_size_ratio > > > > > > > > and what does it have to do with the autoscaler? > > > > When I deactivate the autoscaler the warning goes away. > > > > > > > > > > > > $ ceph osd pool autoscale-status > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO > > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > > > cephfs_metadata 15106M 3.0 2454G 0.0180 > > 0.3000 4.0 256 on > > > > cephfs_data 113.6T 1.5 165.4T 1.0306 > > 0.9000 1.0 512 on > > > > > > > > > > > > $ ceph health detail > > > > HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 > > subtrees have overcommitted pool target_size_ratio > > > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted > > pool target_size_bytes > > > > Pools ['cephfs_data'] overcommit available storage by 1.031x due > > to target_size_bytes 0 on pools [] > > > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted > > pool target_size_ratio > > > > Pools ['cephfs_data'] overcommit available storage by 1.031x due > > to target_size_ratio 0.900 on pools ['cephfs_data'] > > > > > > > > > > > > Thanks > > > > Lars
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de
participants (3)
-
Lars Täuber
-
Nathan Fish
-
Sage Weil