HI all, Just looking for clarification around the relationship between PGs,OSDs and balancing on a ceph (octopus) cluster. We have pg autobalance on and balancing is set to upmap. There are 2 pools, one is the default metric pool with 1 pg, the other is the pool we are using for everything, it has 512 PG. There are 60 OSD's split across 4 hosts. The USD usage ranges between 39% and 68%: * current cluster score 0.048908 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.3 MiB 2.3 GiB 1.5 TiB 47.08 0.86 19 up 1 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 3.2 MiB 2.3 GiB 1.4 TiB 49.46 0.90 20 up 2 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 2.8 MiB 2.4 GiB 1.3 TiB 51.81 0.95 21 up 3 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 589 KiB 2.4 GiB 1.3 TiB 51.73 0.95 21 up 4 hdd 2.74550 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.2 MiB 2.7 GiB 1.1 TiB 59.05 1.08 24 up 5 hdd 9.00000 1.00000 9.1 TiB 5.2 TiB 5.2 TiB 6.9 MiB 8.2 GiB 3.9 TiB 56.75 1.04 77 up 6 hdd 2.74550 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 1.9 MiB 2.8 GiB 1.3 TiB 54.21 0.99 22 up 7 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.1 MiB 2.7 GiB 1.4 TiB 49.32 0.90 20 up 8 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.1 MiB 2.3 GiB 1.4 TiB 49.36 0.90 20 up 9 hdd 2.74550 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 3.3 MiB 3.0 GiB 1.1 TiB 59.21 1.08 24 up 10 hdd 2.74550 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 2.2 MiB 3.4 GiB 941 GiB 66.53 1.22 27 up 11 hdd 2.74550 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 3.9 MiB 2.0 GiB 1.7 TiB 39.69 0.73 16 up 12 hdd 2.74550 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 1.2 MiB 2.1 GiB 1.5 TiB 44.69 0.82 18 up 13 hdd 2.74550 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 2.4 MiB 1.9 GiB 1.7 TiB 39.59 0.72 16 up 14 hdd 2.74550 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 2.1 MiB 3.1 GiB 945 GiB 66.37 1.21 27 up 15 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 1.7 MiB 2.6 GiB 1.2 TiB 56.68 1.04 23 up 16 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 3.3 MiB 2.3 GiB 1.5 TiB 46.76 0.86 19 up 17 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.8 TiB 2.8 MiB 3.1 GiB 953 GiB 66.12 1.21 26 up 18 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.5 MiB 3.0 GiB 1010 GiB 64.08 1.17 26 up 19 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 3.0 MiB 2.9 GiB 1016 GiB 63.88 1.17 26 up 20 hdd 9.00000 1.00000 9.1 TiB 4.7 TiB 4.7 TiB 28 MiB 7.5 GiB 4.4 TiB 51.47 0.94 70 up 21 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 2.8 MiB 2.2 GiB 1.5 TiB 44.43 0.81 18 up 22 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 653 KiB 2.4 GiB 1.3 TiB 51.89 0.95 21 up 23 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.3 MiB 2.7 GiB 1.1 TiB 59.26 1.08 24 up 24 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 3.1 MiB 3.0 GiB 1.2 TiB 56.67 1.04 23 up 25 hdd 2.74599 0.90002 2.7 TiB 1.7 TiB 1.7 TiB 5.3 MiB 4.0 GiB 1.1 TiB 61.47 1.12 25 up 26 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 1.5 MiB 2.4 GiB 1.3 TiB 51.82 0.95 21 up 27 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 1.1 MiB 2.2 GiB 1.5 TiB 46.93 0.86 19 up 28 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.6 MiB 2.2 GiB 1.5 TiB 46.88 0.86 19 up 29 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 557 KiB 2.6 GiB 1.4 TiB 49.37 0.90 20 up 45 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.3 MiB 2.7 GiB 1.1 TiB 59.18 1.08 24 up 46 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 28 MiB 2.5 GiB 1.3 TiB 51.76 0.95 22 up 47 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 977 KiB 3.1 GiB 1.1 TiB 59.07 1.08 24 up 48 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 625 KiB 2.6 GiB 1.5 TiB 46.86 0.86 19 up 49 hdd 2.74599 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 1.9 MiB 3.2 GiB 1.1 TiB 61.68 1.13 25 up 50 hdd 2.74599 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 1.0 MiB 2.7 GiB 1.1 TiB 61.53 1.13 25 up 51 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.1 TiB 2.5 MiB 2.3 GiB 1.6 TiB 41.88 0.77 17 up 52 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.8 TiB 1.8 MiB 3.8 GiB 942 GiB 66.49 1.22 27 up 53 hdd 9.00000 1.00000 9.1 TiB 4.3 TiB 4.3 TiB 6.2 MiB 6.8 GiB 4.8 TiB 47.46 0.87 64 up 54 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.9 GiB 1008 GiB 64.13 1.17 26 up 55 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 1.5 MiB 3.0 GiB 1014 GiB 63.95 1.17 26 up 56 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.8 GiB 1013 GiB 63.99 1.17 26 up 57 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 2.9 MiB 2.6 GiB 1.2 TiB 56.67 1.04 23 up 58 hdd 2.74599 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 1.2 MiB 2.5 GiB 1.3 TiB 54.41 1.00 22 up 59 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.9 GiB 1015 GiB 63.92 1.17 26 up 30 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 1.6 MiB 3.1 GiB 878 GiB 68.79 1.26 28 up 31 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.3 MiB 2.2 GiB 1.5 TiB 47.03 0.86 19 up 32 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 3.2 MiB 3.3 GiB 944 GiB 66.45 1.22 27 up 33 hdd 2.74599 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 3.1 MiB 1.9 GiB 1.7 TiB 39.88 0.73 16 up 34 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 28 MiB 3.0 GiB 874 GiB 68.93 1.26 29 up 35 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 1.3 MiB 2.2 GiB 1.5 TiB 47.01 0.86 19 up 36 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 1.6 MiB 3.0 GiB 947 GiB 66.31 1.21 27 up 37 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 1.8 MiB 2.9 GiB 1.1 TiB 58.96 1.08 24 up 38 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 2.8 MiB 3.4 GiB 877 GiB 68.82 1.26 28 up 39 hdd 2.74599 0.85004 2.7 TiB 1.6 TiB 1.6 TiB 2.4 MiB 3.5 GiB 1.1 TiB 59.20 1.08 24 up 40 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.5 MiB 2.3 GiB 1.4 TiB 49.35 0.90 20 up 41 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 2.3 MiB 2.4 GiB 1.3 TiB 51.73 0.95 21 up 42 hdd 9.00000 1.00000 9.1 TiB 4.5 TiB 4.5 TiB 2.1 MiB 7.2 GiB 4.6 TiB 49.76 0.91 67 up 43 hdd 2.74599 0.90002 2.7 TiB 1.6 TiB 1.6 TiB 2.9 MiB 3.2 GiB 1.1 TiB 59.17 1.08 24 up 44 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 4.0 MiB 2.2 GiB 1.5 TiB 44.34 0.81 18 up 54 of the OSD are 3TiB and there are four 8TiB - can this cause imbalance? Or do I need to increase the PG from 512? or is this as good as it gets? many thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.