Hi all, I am jumping in with a similar (same?) issue. Running Mimic with a pool using EC 4+2, ~280 OSD all 8TB in size. We are far from an optimal distribution by any metrics (PGs per OSD or space used per OSD) but the balancer claims the distribution is optimal. I took a look at the ticket but did not want to muddle the water as it was opened against Nautilus. Dumping the osdmap and trying manually returns a laconic: # osdmaptool osdmap_upmapissue --upmap - --upmap-pool spindle osdmaptool: osdmap file 'osdmap_upmapissue' checking for upmap cleanups upmap, max-count 100, max deviation 0.01 limiting to pools spindle (11) no upmaps proposed Any idea? Best, Mattia This is the current distribution (problematic pool uses hdd) #ceph osd df | sort -nk11 ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS MIN/MAX VAR: 0.00/1.19 STDDEV: 20.80 TOTAL 2.0 PiB 1.4 PiB 524 TiB 73.86 174 nvme 0.05849 1.00000 1.1 TiB 1.9 GiB 1.1 TiB 0.17 0.00 1 96 nvme 0.05849 1.00000 60 GiB 1.3 GiB 59 GiB 2.23 0.03 0 109 nvme 0.05849 1.00000 60 GiB 2.6 GiB 57 GiB 4.31 0.06 2 70 nvme 0.05849 1.00000 60 GiB 2.7 GiB 57 GiB 4.44 0.06 2 200 nvme 0.02910 1.00000 30 GiB 1.5 GiB 28 GiB 5.18 0.07 1 247 nvme 0.02910 1.00000 30 GiB 1.6 GiB 28 GiB 5.38 0.07 1 213 nvme 0.02910 1.00000 30 GiB 1.6 GiB 28 GiB 5.39 0.07 1 10 nvme 0.05849 1.00000 60 GiB 3.6 GiB 56 GiB 6.08 0.08 3 161 nvme 0.05849 1.00000 60 GiB 3.9 GiB 56 GiB 6.43 0.09 3 135 nvme 0.05849 1.00000 60 GiB 4.6 GiB 55 GiB 7.60 0.10 4 187 nvme 0.05849 1.00000 60 GiB 4.6 GiB 55 GiB 7.71 0.10 4 83 nvme 0.05849 1.00000 60 GiB 4.7 GiB 55 GiB 7.84 0.11 4 57 nvme 0.05849 1.00000 60 GiB 4.8 GiB 55 GiB 7.99 0.11 4 273 nvme 0.02910 1.00000 30 GiB 2.6 GiB 27 GiB 8.56 0.12 2 122 nvme 0.05849 1.00000 60 GiB 6.5 GiB 53 GiB 10.84 0.15 6 312 nvme 0.02910 1.00000 30 GiB 3.6 GiB 26 GiB 12.03 0.16 3 260 nvme 0.02910 1.00000 30 GiB 3.7 GiB 26 GiB 12.39 0.17 3 324 nvme 0.02910 1.00000 30 GiB 4.5 GiB 25 GiB 15.06 0.20 4 148 nvme 0.05849 1.00000 60 GiB 10 GiB 49 GiB 17.42 0.24 0 284 nvme 0.02910 1.00000 30 GiB 5.4 GiB 24 GiB 18.23 0.25 0 298 nvme 0.02910 1.00000 30 GiB 5.4 GiB 24 GiB 18.24 0.25 0 28 ssd 1.45549 1.00000 1.5 TiB 545 GiB 946 GiB 36.54 0.49 75 23 ssd 1.45549 1.00000 1.5 TiB 554 GiB 937 GiB 37.14 0.50 76 27 ssd 1.45549 1.00000 1.5 TiB 560 GiB 931 GiB 37.55 0.51 76 26 ssd 1.45549 1.00000 1.5 TiB 562 GiB 929 GiB 37.68 0.51 77 16 ssd 1.45549 1.00000 1.5 TiB 573 GiB 917 GiB 38.46 0.52 78 216 ssd 1.45549 1.00000 1.5 TiB 573 GiB 917 GiB 38.46 0.52 78 31 ssd 1.45549 1.00000 1.5 TiB 574 GiB 917 GiB 38.48 0.52 78 20 ssd 1.45549 1.00000 1.5 TiB 574 GiB 917 GiB 38.50 0.52 79 36 ssd 1.45549 1.00000 1.5 TiB 575 GiB 916 GiB 38.57 0.52 78 14 ssd 1.45549 1.00000 1.5 TiB 575 GiB 915 GiB 38.59 0.52 78 17 ssd 1.45549 1.00000 1.5 TiB 575 GiB 915 GiB 38.59 0.52 78 37 ssd 1.45549 1.00000 1.5 TiB 575 GiB 915 GiB 38.61 0.52 78 219 ssd 1.45549 1.00000 1.5 TiB 576 GiB 915 GiB 38.63 0.52 79 218 ssd 1.45549 1.00000 1.5 TiB 576 GiB 914 GiB 38.65 0.52 78 221 ssd 1.45549 1.00000 1.5 TiB 576 GiB 914 GiB 38.67 0.52 78 13 ssd 1.45549 1.00000 1.5 TiB 577 GiB 914 GiB 38.68 0.52 78 41 ssd 1.45549 1.00000 1.5 TiB 577 GiB 914 GiB 38.68 0.52 79 43 ssd 1.45549 1.00000 1.5 TiB 577 GiB 913 GiB 38.72 0.52 81 18 ssd 1.45549 1.00000 1.5 TiB 578 GiB 913 GiB 38.77 0.52 78 32 ssd 1.45549 1.00000 1.5 TiB 578 GiB 912 GiB 38.78 0.52 79 42 ssd 1.45549 1.00000 1.5 TiB 578 GiB 912 GiB 38.79 0.53 80 40 ssd 1.45549 1.00000 1.5 TiB 579 GiB 912 GiB 38.82 0.53 79 22 ssd 1.45549 1.00000 1.5 TiB 582 GiB 908 GiB 39.06 0.53 79 24 ssd 1.45549 1.00000 1.5 TiB 583 GiB 908 GiB 39.09 0.53 78 217 ssd 1.45549 1.00000 1.5 TiB 583 GiB 908 GiB 39.10 0.53 80 21 ssd 1.45549 1.00000 1.5 TiB 583 GiB 908 GiB 39.10 0.53 80 15 ssd 1.45549 1.00000 1.5 TiB 583 GiB 907 GiB 39.12 0.53 80 30 ssd 1.45549 1.00000 1.5 TiB 583 GiB 907 GiB 39.13 0.53 78 214 ssd 1.45549 1.00000 1.5 TiB 584 GiB 907 GiB 39.16 0.53 80 25 ssd 1.45549 1.00000 1.5 TiB 584 GiB 906 GiB 39.21 0.53 80 29 ssd 1.45549 1.00000 1.5 TiB 584 GiB 906 GiB 39.21 0.53 78 215 ssd 1.45549 1.00000 1.5 TiB 588 GiB 902 GiB 39.48 0.53 80 39 ssd 1.45549 1.00000 1.5 TiB 591 GiB 900 GiB 39.63 0.54 80 19 ssd 1.45549 1.00000 1.5 TiB 596 GiB 895 GiB 39.96 0.54 80 38 ssd 1.45549 1.00000 1.5 TiB 600 GiB 891 GiB 40.23 0.54 83 220 ssd 1.45549 1.00000 1.5 TiB 600 GiB 890 GiB 40.26 0.55 81 44 ssd 1.45549 1.00000 1.5 TiB 606 GiB 885 GiB 40.64 0.55 83 35 ssd 1.45549 1.00000 1.5 TiB 613 GiB 877 GiB 41.13 0.56 83 33 ssd 1.45549 1.00000 1.5 TiB 616 GiB 874 GiB 41.33 0.56 84 34 ssd 1.45549 1.00000 1.5 TiB 618 GiB 872 GiB 41.47 0.56 83 261 hdd 7.33559 1.00000 7.3 TiB 4.7 TiB 2.6 TiB 64.46 0.87 80 265 hdd 7.33559 1.00000 7.3 TiB 4.8 TiB 2.5 TiB 66.09 0.89 82 282 hdd 7.33559 1.00000 7.3 TiB 5.1 TiB 2.3 TiB 69.21 0.94 86 283 hdd 7.33559 1.00000 7.3 TiB 5.1 TiB 2.3 TiB 69.25 0.94 86 272 hdd 7.33559 1.00000 7.3 TiB 5.1 TiB 2.3 TiB 69.26 0.94 86 285 hdd 7.33559 1.00000 7.3 TiB 5.1 TiB 2.3 TiB 69.30 0.94 86 276 hdd 7.33559 1.00000 7.3 TiB 5.1 TiB 2.2 TiB 69.99 0.95 87 281 hdd 7.33559 1.00000 7.3 TiB 5.1 TiB 2.2 TiB 70.04 0.95 87 279 hdd 7.33559 1.00000 7.3 TiB 5.1 TiB 2.2 TiB 70.08 0.95 87 269 hdd 7.33559 1.00000 7.3 TiB 5.2 TiB 2.1 TiB 70.83 0.96 88 268 hdd 7.33559 1.00000 7.3 TiB 5.2 TiB 2.1 TiB 70.86 0.96 88 144 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.03 0.96 88 8 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.04 0.96 88 92 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.04 0.96 88 241 hdd 7.33559 1.00000 7.3 TiB 5.3 TiB 2.1 TiB 71.58 0.97 89 243 hdd 7.33559 1.00000 7.3 TiB 5.3 TiB 2.1 TiB 71.61 0.97 89 262 hdd 7.33559 1.00000 7.3 TiB 5.3 TiB 2.1 TiB 71.63 0.97 89 3 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.79 0.97 89 2 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.80 0.97 89 140 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.81 0.97 89 139 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.83 0.97 89 6 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.83 0.97 89 142 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.86 0.97 89 1 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.86 0.97 89 138 hdd 7.39380 1.00000 7.4 TiB 5.3 TiB 2.1 TiB 71.91 0.97 89 258 hdd 7.33559 1.00000 7.3 TiB 5.3 TiB 2.0 TiB 72.42 0.98 90 257 hdd 7.33559 1.00000 7.3 TiB 5.3 TiB 2.0 TiB 72.47 0.98 90 236 hdd 7.33559 1.00000 7.3 TiB 5.3 TiB 2.0 TiB 72.48 0.98 90 9 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 72.62 0.98 90 128 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 72.64 0.98 90 132 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 72.64 0.98 90 11 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 72.66 0.98 90 89 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 72.66 0.98 90 124 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 72.67 0.98 90 137 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 72.68 0.98 90 237 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.17 0.99 91 196 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.19 0.99 91 248 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.20 0.99 91 249 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.20 0.99 91 198 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.21 0.99 91 267 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.21 0.99 91 238 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.22 0.99 91 244 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.22 0.99 91 266 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.22 0.99 91 195 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 2.0 TiB 73.23 0.99 91 88 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.40 0.99 91 94 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.41 0.99 91 125 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.42 0.99 91 136 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.42 0.99 91 147 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.42 0.99 91 185 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.43 0.99 91 130 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.44 0.99 91 85 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.45 0.99 91 93 hdd 7.39380 1.00000 7.4 TiB 5.4 TiB 2.0 TiB 73.45 0.99 91 163 hdd 7.30469 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.96 1.00 92 170 hdd 7.30469 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.96 1.00 92 168 hdd 7.30469 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.97 1.00 92 162 hdd 7.30469 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.98 1.00 92 263 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.98 1.00 92 212 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.99 1.00 92 242 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.99 1.00 92 251 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.99 1.00 92 316 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 73.99 1.00 92 164 hdd 7.30469 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.00 1.00 92 192 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.00 1.00 92 193 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.00 1.00 92 211 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.00 1.00 92 235 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.00 1.00 92 270 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.00 1.00 92 254 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.01 1.00 92 314 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.01 1.00 92 317 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.01 1.00 92 253 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.02 1.00 92 277 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.03 1.00 92 252 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.04 1.00 92 319 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.04 1.00 92 320 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.05 1.00 92 323 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.05 1.00 92 313 hdd 7.33559 1.00000 7.3 TiB 5.4 TiB 1.9 TiB 74.06 1.00 92 175 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.17 1.00 92 145 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.21 1.00 92 126 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.22 1.00 92 184 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.22 1.00 92 87 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.22 1.00 92 90 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.22 1.00 92 141 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.23 1.00 92 104 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.24 1.01 92 108 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.25 1.01 92 127 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.25 1.01 92 101 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.27 1.01 92 100 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.28 1.01 92 246 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.74 1.01 93 255 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.75 1.01 93 296 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.75 1.01 93 209 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.76 1.01 93 226 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.76 1.01 93 169 hdd 7.30469 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.77 1.01 93 171 hdd 7.30469 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.77 1.01 93 210 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.77 1.01 93 233 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.77 1.01 93 274 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.77 1.01 93 287 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.77 1.01 93 321 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.77 1.01 93 166 hdd 7.30469 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.78 1.01 93 191 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.78 1.01 93 194 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.78 1.01 93 205 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.78 1.01 93 275 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.78 1.01 93 301 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.78 1.01 93 308 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.9 TiB 74.78 1.01 93 201 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.79 1.01 93 207 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.79 1.01 93 208 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.79 1.01 93 302 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.79 1.01 93 304 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.79 1.01 93 318 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.79 1.01 93 172 hdd 7.30469 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.80 1.01 93 199 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.80 1.01 93 202 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.80 1.01 93 259 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.80 1.01 93 299 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.80 1.01 93 190 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 206 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 271 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 278 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 280 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 286 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 288 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 306 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 322 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 325 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.81 1.01 93 165 hdd 7.30469 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.82 1.01 93 225 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.82 1.01 93 239 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.83 1.01 93 290 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.83 1.01 93 292 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.83 1.01 93 305 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.83 1.01 93 188 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.84 1.01 93 197 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.84 1.01 93 203 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.84 1.01 93 222 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.84 1.01 93 250 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.84 1.01 93 291 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.84 1.01 93 231 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.85 1.01 93 240 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.85 1.01 93 294 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.85 1.01 93 307 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.85 1.01 93 315 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.85 1.01 93 189 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.86 1.01 93 204 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.86 1.01 93 245 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.86 1.01 93 264 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.86 1.01 93 303 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.87 1.01 93 256 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 74.90 1.01 93 150 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.92 1.01 93 63 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.93 1.01 93 123 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.95 1.01 93 60 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.95 1.01 93 118 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.96 1.01 93 179 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.96 1.01 93 98 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.96 1.01 93 103 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 74.98 1.02 93 154 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 74.98 1.02 93 67 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 74.98 1.02 93 86 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.9 TiB 74.98 1.02 93 114 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 74.99 1.02 93 120 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 74.99 1.02 93 181 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 74.99 1.02 93 4 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 74.99 1.02 93 107 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.00 1.02 93 111 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.00 1.02 93 77 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.00 1.02 93 81 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.00 1.02 93 113 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.01 1.02 93 143 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.01 1.02 93 48 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.01 1.02 93 84 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.01 1.02 93 95 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.01 1.02 93 115 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.02 1.02 93 121 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.02 1.02 93 97 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.02 1.02 93 182 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.03 1.02 93 68 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.03 1.02 93 116 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.04 1.02 93 119 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.04 1.02 93 158 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.04 1.02 93 73 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.04 1.02 93 52 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.05 1.02 93 59 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.05 1.02 93 99 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.05 1.02 93 177 hdd 7.39380 1.00000 7.4 TiB 5.5 TiB 1.8 TiB 75.06 1.02 93 180 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.06 1.02 93 72 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.07 1.02 93 173 hdd 7.30469 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 75.61 1.02 94 297 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 75.62 1.02 94 293 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 75.63 1.02 94 227 hdd 7.33559 1.00000 7.3 TiB 5.5 TiB 1.8 TiB 75.65 1.02 94 146 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.71 1.02 94 110 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.75 1.03 94 133 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.76 1.03 94 55 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.77 1.03 94 183 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.78 1.03 94 49 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.78 1.03 94 7 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.78 1.03 94 134 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.79 1.03 94 178 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.79 1.03 94 65 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.79 1.03 94 0 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.80 1.03 94 129 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.80 1.03 94 176 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.80 1.03 94 186 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.80 1.03 94 5 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.80 1.03 94 106 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.81 1.03 94 112 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.81 1.03 94 56 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.81 1.03 94 117 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.82 1.03 94 131 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.82 1.03 94 105 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.83 1.03 94 91 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.83 1.03 94 12 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.84 1.03 94 64 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.84 1.03 94 102 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.85 1.03 94 61 hdd 7.39380 1.00000 7.4 TiB 5.6 TiB 1.8 TiB 75.87 1.03 94 311 hdd 7.33559 1.00000 7.3 TiB 5.6 TiB 1.7 TiB 76.36 1.03 95 224 hdd 7.33559 1.00000 7.3 TiB 5.6 TiB 1.7 TiB 76.43 1.03 95 295 hdd 7.33559 1.00000 7.3 TiB 5.6 TiB 1.7 TiB 76.43 1.03 95 46 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 76.59 1.04 95 66 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 76.60 1.04 95 76 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 76.63 1.04 95 228 hdd 7.33559 1.00000 7.3 TiB 5.7 TiB 1.7 TiB 77.19 1.05 96 79 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.32 1.05 96 54 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.34 1.05 96 153 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.35 1.05 96 51 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.35 1.05 96 69 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.35 1.05 96 58 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.36 1.05 96 82 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.36 1.05 96 157 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.38 1.05 96 74 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.39 1.05 96 80 hdd 7.39380 1.00000 7.4 TiB 5.7 TiB 1.7 TiB 77.42 1.05 96 232 hdd 7.33559 1.00000 7.3 TiB 5.7 TiB 1.6 TiB 77.95 1.06 97 230 hdd 7.33559 1.00000 7.3 TiB 5.7 TiB 1.6 TiB 77.97 1.06 97 310 hdd 7.33559 1.00000 7.3 TiB 5.7 TiB 1.6 TiB 77.97 1.06 97 159 hdd 7.39380 1.00000 7.4 TiB 5.8 TiB 1.6 TiB 78.11 1.06 97 47 hdd 7.39380 1.00000 7.4 TiB 5.8 TiB 1.6 TiB 78.13 1.06 97 62 hdd 7.39380 1.00000 7.4 TiB 5.8 TiB 1.6 TiB 78.13 1.06 97 53 hdd 7.39380 1.00000 7.4 TiB 5.8 TiB 1.6 TiB 78.18 1.06 97 300 hdd 7.33559 1.00000 7.3 TiB 5.8 TiB 1.6 TiB 78.76 1.07 98 309 hdd 7.33559 1.00000 7.3 TiB 5.8 TiB 1.5 TiB 79.58 1.08 99 229 hdd 7.33559 1.00000 7.3 TiB 5.8 TiB 1.5 TiB 79.63 1.08 99 155 hdd 7.39380 1.00000 7.4 TiB 5.9 TiB 1.5 TiB 79.70 1.08 99 50 hdd 7.39380 1.00000 7.4 TiB 5.9 TiB 1.5 TiB 79.78 1.08 99 151 hdd 7.39380 1.00000 7.4 TiB 5.9 TiB 1.5 TiB 80.00 1.08 99 71 hdd 7.39380 1.00000 7.4 TiB 6.0 TiB 1.4 TiB 80.50 1.09 100 167 hdd 7.30469 1.00000 7.3 TiB 5.9 TiB 1.4 TiB 81.23 1.10 101 78 hdd 7.39380 1.00000 7.4 TiB 6.1 TiB 1.3 TiB 82.11 1.11 102 289 hdd 7.33559 1.00000 7.3 TiB 6.1 TiB 1.2 TiB 83.58 1.13 104 160 hdd 7.39380 1.00000 7.4 TiB 6.2 TiB 1.2 TiB 83.62 1.13 104 149 hdd 7.39380 1.00000 7.4 TiB 6.2 TiB 1.1 TiB 84.46 1.14 105 75 hdd 7.39380 1.00000 7.4 TiB 6.2 TiB 1.1 TiB 84.49 1.14 105 223 hdd 7.33559 1.00000 7.3 TiB 6.3 TiB 1.0 TiB 85.96 1.16 107 45 hdd 7.39380 1.00000 7.4 TiB 6.4 TiB 1.0 TiB 86.10 1.17 107 156 hdd 7.39380 1.00000 7.4 TiB 6.5 TiB 940 GiB 87.59 1.19 109 152 hdd 7.39380 1.00000 7.4 TiB 6.5 TiB 938 GiB 87.62 1.19 109 On 1/23/20 4:23 PM, Thomas Schneider wrote:

Hi Dan,

I have opened this this bug report for balancer not working as expected. https://tracker.ceph.com/issues/43586 Do you experience the same issue?

Regards Thomas

Am 23.01.2020 um 16:05 schrieb Dan van der Ster:

...

Hi Frank,

No, it is basically balancing the num_pgs per TB (per osd).

Cheers, Dan

On Thu, Jan 23, 2020 at 3:53 PM Frank R <frankaritchie@gmail.com <mailto:frankaritchie@gmail.com>> wrote:

Hi all,

Does using the Upmap balancer require that all OSDs be the same size (per device class)?

thx Frank _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Thomas, I lost track of your issue. Are you just trying to balance the PGs ? 14.2.8 has big improvements -- check the release notes / blog post about setting the upmap_max_deviations down to 2 or 1. -- Dan On Mon, Mar 16, 2020 at 4:00 PM Thomas Schneider <74cmonty@gmail.com> wrote:

Hi Dan,

I have opened this this bug report for balancer not working as expected. https://tracker.ceph.com/issues/43586

Then I thought it could make sense to balance the cluster manually by means of moving PGs from a heavily loaded OSD to another.

I found your slides "Luminous: pg upmap (dev) <https://indico.cern.ch/event/669931/contributions/2742401/attachments/1533434/2401109/upmap.pdf>", but I didn't fully understand.

Could you please advise how to move PGs manually?

Regards Thomas

Am 23.01.2020 um 16:05 schrieb Dan van der Ster:

...

Hi Frank,

No, it is basically balancing the num_pgs per TB (per osd).

Cheers, Dan

On Thu, Jan 23, 2020 at 3:53 PM Frank R <frankaritchie@gmail.com <mailto:frankaritchie@gmail.com>> wrote:

Hi all,

Does using the Upmap balancer require that all OSDs be the same size (per device class)?

thx Frank _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, I would upgrade, configure the balancer correctly, then wait a bit for it to smooth things out. Afterwards you can reweight back to 1.0. -- dan On Mon, Mar 16, 2020 at 4:19 PM Thomas Schneider <74cmonty@gmail.com> wrote:

Hi Dan,

indeed I'm trying to balance the PGs.

In order to ensure Ceph cluster operations I used OSD reweight, means some specific OSDs are not with reweight 0.8 and 0.9 respectively.

Question: Can I upgrade to Ceph 14.2.8 w/o resetting the weight to 1.0? Or should I cleanup this reweight first, the upgrade to 14.2.8 and enable balancer as last?

Regards Thomas

Am 16.03.2020 um 16:10 schrieb Dan van der Ster:

...

Hi Thomas, I lost track of your issue. Are you just trying to balance the PGs ? 14.2.8 has big improvements -- check the release notes / blog post about setting the upmap_max_deviations down to 2 or 1. -- Dan

On Mon, Mar 16, 2020 at 4:00 PM Thomas Schneider <74cmonty@gmail.com> wrote:

...

Hi Dan,

I have opened this this bug report for balancer not working as expected. https://tracker.ceph.com/issues/43586

Then I thought it could make sense to balance the cluster manually by means of moving PGs from a heavily loaded OSD to another.

I found your slides "Luminous: pg upmap (dev) <https://indico.cern.ch/event/669931/contributions/2742401/attachments/1533434/2401109/upmap.pdf>", but I didn't fully understand.

Could you please advise how to move PGs manually?

Regards Thomas

Am 23.01.2020 um 16:05 schrieb Dan van der Ster:

...

Hi Frank,

No, it is basically balancing the num_pgs per TB (per osd).

Cheers, Dan

On Thu, Jan 23, 2020 at 3:53 PM Frank R <frankaritchie@gmail.com <mailto:frankaritchie@gmail.com>> wrote:

Hi all,

Does using the Upmap balancer require that all OSDs be the same size (per device class)?

thx Frank _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io