Hi Nathan, Attached crushmap output. let me know if you find any thing odd. On Sat, Oct 24, 2020 at 6:47 PM Nathan Fish <lordcirth(a)gmail.com> wrote:

Yes, There is a unbalance in PG's assigned to OSD's. `ceph osd df` output snip ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 5.45799 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 9.7 MiB 4.6 GiB 1.9 TiB 65.94 1.31 13 up 1 hdd 5.45799 1.00000 5.5 TiB 1.0 TiB 1.0 TiB 4.4 MiB 1.3 GiB 4.4 TiB 18.87 0.38 9 up 2 hdd 5.45799 1.00000 5.5 TiB 1.5 TiB 1.5 TiB 4.0 MiB 1.9 GiB 3.9 TiB 28.30 0.56 10 up 3 hdd 5.45799 1.00000 5.5 TiB 2.1 TiB 2.1 TiB 7.7 MiB 2.7 GiB 3.4 TiB 37.70 0.75 12 up 4 hdd 5.45799 1.00000 5.5 TiB 4.1 TiB 4.1 TiB 5.8 MiB 5.2 GiB 1.3 TiB 75.27 1.50 20 up 5 hdd 5.45799 1.00000 5.5 TiB 5.1 TiB 5.1 TiB 5.9 MiB 6.7 GiB 317 GiB 94.32 1.88 18 up 6 hdd 5.45799 1.00000 5.5 TiB 1.5 TiB 1.5 TiB 5.2 MiB 2.0 GiB 3.9 TiB 28.32 0.56 9 up MIN/MAX VAR: 0.19/1.88 STDDEV: 22.13 On Sun, Oct 25, 2020 at 12:08 AM Stefan Kooman <stefan(a)bit.nl> wrote:

Hi Stefan, I have started balancer but what I don't understand is there are enough free space in other disks. Why it's not showing those in available space? How to reclaim the free space? On Sun 25 Oct, 2020, 2:27 PM Stefan Kooman, <stefan(a)bit.nl> wrote:

Hi, For my quick understanding How PG's are responsible for allowing space allocation to a pool? My understanding that PG's basically helps in object placement when the number of PG's for a OSD's is high there is a high possibility that PG gets lot more data than other PG's. At this situation, we can use the balance between OSD's. But, I can't understand the logic of how does it restrict space to a pool? On Sun, Oct 25, 2020 at 5:55 PM 胡 玮文 <huww98(a)outlook.com> wrote:

Den sön 25 okt. 2020 kl 15:18 skrev Amudhan P <amudhan83(a)gmail.com>om>: An objects name will decide which PG (from the list of PGs in the pool) it will end up on, so if you have very few PGs, the hashed/pseudorandom placement will be unbalanced at times. As an example, if you have only 8 PGs and write 9 large objects, then at least one (but probably two or three) PGs will receive two or more of those 9, and some will receive none just on pure statistics. If you have 100 PGs, the chance of one getting two out of those nine objects is much smaller. Overall, with all pools accounted for, one should aim for something like 100 PGs per OSD, but you also need to count the replication factor for each pool so if you have replication = 3 and a pool gets 128 PGs, it will place 3*128 PGs out on various OSDs according to the crush rules. PGs don't have a size, but will grow as needed, and since the next object to be written can end up anywhere (depending on the hashed result) ceph df must always tell you the worst case when listing how much data this pool has "left". It will always be the OSD with least space left that limits the pool. This statement seems incorrect to me. -- May the most significant bit of your life be positive.

在 2020年10月26日，22:30，Amudhan P &lt;amudhan83(a)gmail.com&gt; 写道：  Hi Jane, I agree with you and I was trying to say disk which has more PG will fill up quick. But, My question even though RAW disk space is 262 TB, pool 2 replica max storage is showing only 132 TB in the dashboard and when mounting the pool using cephfs it's showing 62 TB, I could understand that due to replica it's showing half of the space. Your first mail shows 67T (instead of 62) why it's not showing the entire RAW disk space as available space? Number of PG per pool play any vital role in showing available space? I might be wrong, but I think the size of mounted cephfs is calculated by “used + available”. It is not directly related to raw disk space. You have unbalance issue, so you have less available space as explained previously. So the total size is less than expected. Maybe you should try to correct the unbalance first, and see if the available space and size go up. Increase pg_num, run balancer, etc. On Mon, Oct 26, 2020 at 12:37 PM Janne Johansson <icepic.dz@gmail.com<mailto:icepic.dz@gmail.com>> wrote: Den sön 25 okt. 2020 kl 15:18 skrev Amudhan P <amudhan83@gmail.com<mailto:amudhan83@gmail.com>>: Hi, For my quick understanding How PG's are responsible for allowing space allocation to a pool? An objects name will decide which PG (from the list of PGs in the pool) it will end up on, so if you have very few PGs, the hashed/pseudorandom placement will be unbalanced at times. As an example, if you have only 8 PGs and write 9 large objects, then at least one (but probably two or three) PGs will receive two or more of those 9, and some will receive none just on pure statistics. If you have 100 PGs, the chance of one getting two out of those nine objects is much smaller. Overall, with all pools accounted for, one should aim for something like 100 PGs per OSD, but you also need to count the replication factor for each pool so if you have replication = 3 and a pool gets 128 PGs, it will place 3*128 PGs out on various OSDs according to the crush rules. PGs don't have a size, but will grow as needed, and since the next object to be written can end up anywhere (depending on the hashed result) ceph df must always tell you the worst case when listing how much data this pool has "left". It will always be the OSD with least space left that limits the pool. My understanding that PG's basically helps in object placement when the number of PG's for a OSD's is high there is a high possibility that PG gets lot more data than other PG's. This statement seems incorrect to me. At this situation, we can use the balance between OSD's. But, I can't understand the logic of how does it restrict space to a pool? -- May the most significant bit of your life be positive.