Hi Thomas, For 100% byte distribution of data across OSD, you should setup ceph balancer in "byte" mode, not in PG mode. Change will distribute all osd with the same % of usage, but the objects will be NOT reduntant. After several weeks and months testing balancer the best profile is balance by PG with unmap. In PG mode you are going to get always "until balancer got a better algorithm" a not equially data distributed, an you sometime should manually redistribute weight by CLI. You can play with balancer directly from Dashboard from Nautilus. Balancer is not an "active" agent asked before storage data into disk, first ceph store data and them balancer move objects. Regards Manuel -----Mensaje original----- De: Thomas <74cmonty(a)gmail.com> Enviado el: lunes, 23 de septiembre de 2019 11:08 Para: ceph-users(a)ceph.io Asunto: [ceph-users] OSD rebalancing issue - should drives be distributed equally over all nodes Hi, I'm facing several issues with my ceph cluster (2x MDS, 6x ODS). Here I would like to focus on the issue with pgs backfill_toofull. I assume this is related to the fact that the data distribution on my OSDs is not balanced. This is the current ceph status: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 MDSs report slow metadata IOs 78 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 2 pgs inactive, 2 pgs peering Degraded data redundancy: 304136/153251211 objects degraded (0.198%), 57 pgs degraded, 57 pgs undersized Degraded data redundancy (low space): 265 pgs backfill_toofull 3 pools have too many placement groups 74 slow requests are blocked > 32 sec 80 stuck requests are blocked > 4096 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 98m) mgr: ld5505(active, since 3d), standbys: ld5506, ld5507 mds: pve_cephfs:1 {0=ld3976=up:active} 1 up:standby osd: 368 osds: 368 up, 367 in; 302 remapped pgs data: pools: 5 pools, 8868 pgs objects: 51.08M objects, 195 TiB usage: 590 TiB used, 563 TiB / 1.1 PiB avail pgs: 0.023% pgs not active 304136/153251211 objects degraded (0.198%) 1672190/153251211 objects misplaced (1.091%) 8564 active+clean 196 active+remapped+backfill_toofull 57 active+undersized+degraded+remapped+backfill_toofull 35 active+remapped+backfill_wait 12 active+remapped+backfill_wait+backfill_toofull 2 active+remapped+backfilling 2 peering io: recovery: 18 MiB/s, 4 objects/s Currently I'm using 6 OSD nodes. Node A 48x 1.6TB HDD Node B 48x 1.6TB HDD Node C 48x 1.6TB HDD Node D 48x 1.6TB HDD Node E 48x 7.2TB HDD Node F 48x 7.2TB HDD Question: Is it advisable to distribute the drives equally over all nodes? If yes, how should this be executed w/o ceph disruption? Regards Thomas _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Thomas , Check the documentation for CLI "ceph osd reweight-by-utilization" and run it. Another problem of unbalanced data is the pool global data available . Regards Manuel -----Mensaje original----- De: Thomas <74cmonty(a)gmail.com> Enviado el: lunes, 23 de septiembre de 2019 11:49 Para: EDH - Manuel Rios Fernandez <mriosfer(a)easydatahost.com>om>; ceph-users(a)ceph.io Asunto: Re: [ceph-users] OSD rebalancing issue - should drives be distributed equally over all nodes Hi, I have already balancer mode upmap enabled. root@ld3955:/mnt/pve/pve_cephfs/template/iso# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } However there are OSD with 60% and others with 90% usage belonging to the same pool with the same disk size. This looks to me like a big range. Regards Thomas Am 23.09.2019 um 11:42 schrieb EDH - Manuel Rios Fernandez:

Hi Reed, I'm not sure what is meant with the grouping / chassis and "set your failure domain to chassis" respectively. This is my current crush map: # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class nvme device 1 osd.1 class nvme device 2 osd.2 class nvme device 3 osd.3 class nvme device 4 osd.4 class nvme device 5 osd.5 class nvme device 6 osd.6 class nvme device 7 osd.7 class nvme device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd [...] device 365 osd.365 class hdd device 366 osd.366 class hdd device 367 osd.367 class hdd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 region type 10 root # buckets host ld5505 {     id -3        # do not change unnecessarily     id -4 class hdd        # do not change unnecessarily     id -5 class nvme        # do not change unnecessarily     # weight 33.720     alg straw2     hash 0    # rjenkins1     item osd.8 weight 1.640     item osd.9 weight 1.640     item osd.10 weight 1.640     item osd.11 weight 1.640     item osd.12 weight 1.640     item osd.13 weight 1.640     item osd.14 weight 1.640     item osd.15 weight 1.640     item osd.16 weight 1.640     item osd.17 weight 1.640     item osd.18 weight 1.640     item osd.19 weight 1.640     item osd.20 weight 1.640     item osd.21 weight 1.640     item osd.22 weight 1.640     item osd.23 weight 1.640     item osd.24 weight 1.640     item osd.0 weight 2.920     item osd.1 weight 2.920 } host ld5506 {     id -7        # do not change unnecessarily     id -8 class hdd        # do not change unnecessarily     id -9 class nvme        # do not change unnecessarily     # weight 33.720     alg straw2     hash 0    # rjenkins1     item osd.25 weight 1.640     item osd.26 weight 1.640     item osd.27 weight 1.640     item osd.28 weight 1.640     item osd.29 weight 1.640     item osd.30 weight 1.640     item osd.31 weight 1.640     item osd.32 weight 1.640     item osd.33 weight 1.640     item osd.34 weight 1.640     item osd.35 weight 1.640     item osd.36 weight 1.640     item osd.37 weight 1.640     item osd.38 weight 1.640     item osd.39 weight 1.640     item osd.40 weight 1.640     item osd.41 weight 1.640     item osd.2 weight 2.920     item osd.3 weight 2.920 } host ld5507 {     id -10        # do not change unnecessarily     id -11 class hdd        # do not change unnecessarily     id -12 class nvme        # do not change unnecessarily     # weight 33.720     alg straw2     hash 0    # rjenkins1     item osd.172 weight 1.640     item osd.173 weight 1.640     item osd.174 weight 1.640     item osd.175 weight 1.640     item osd.176 weight 1.640     item osd.177 weight 1.640     item osd.178 weight 1.640     item osd.179 weight 1.640     item osd.180 weight 1.640     item osd.181 weight 1.640     item osd.182 weight 1.640     item osd.183 weight 1.640     item osd.184 weight 1.640     item osd.185 weight 1.640     item osd.186 weight 1.640     item osd.187 weight 1.640     item osd.188 weight 1.640     item osd.4 weight 2.920     item osd.5 weight 2.920 } host ld5508 {     id -13        # do not change unnecessarily     id -14 class hdd        # do not change unnecessarily     id -15 class nvme        # do not change unnecessarily     # weight 33.720     alg straw2     hash 0    # rjenkins1     item osd.60 weight 1.640     item osd.61 weight 1.640     item osd.62 weight 1.640     item osd.63 weight 1.640     item osd.64 weight 1.640     item osd.65 weight 1.640     item osd.66 weight 1.640     item osd.69 weight 1.640     item osd.70 weight 1.640     item osd.71 weight 1.640     item osd.72 weight 1.640     item osd.73 weight 1.640     item osd.74 weight 1.640     item osd.75 weight 1.640     item osd.59 weight 1.640     item osd.67 weight 1.640     item osd.68 weight 1.640     item osd.6 weight 2.920     item osd.7 weight 2.920 } host ld4464 {     id -34        # do not change unnecessarily     id -36 class hdd        # do not change unnecessarily     id -35 class nvme        # do not change unnecessarily     # weight 2.000     alg straw2     hash 0    # rjenkins1     item osd.268 weight 1.000     item osd.269 weight 1.000 } host ld4465 {     id -37        # do not change unnecessarily     id -39 class hdd        # do not change unnecessarily     id -38 class nvme        # do not change unnecessarily     # weight 2.000     alg straw2     hash 0    # rjenkins1     item osd.270 weight 1.000     item osd.271 weight 1.000 } root default {     id -1        # do not change unnecessarily     id -2 class hdd        # do not change unnecessarily     id -6 class nvme        # do not change unnecessarily     # weight 138.880     alg straw2     hash 0    # rjenkins1     item ld5505 weight 33.720     item ld5506 weight 33.720     item ld5507 weight 33.720     item ld5508 weight 33.720     item ld4464 weight 2.000     item ld4465 weight 2.000 } host ld5505-hdd_strgbox {     id -16        # do not change unnecessarily     id -18 class hdd        # do not change unnecessarily     id -20 class nvme        # do not change unnecessarily     # weight 78.720     alg straw2     hash 0    # rjenkins1     item osd.76 weight 1.640     item osd.77 weight 1.640     item osd.78 weight 1.640     item osd.79 weight 1.640     item osd.80 weight 1.640     item osd.81 weight 1.640     item osd.82 weight 1.640     item osd.83 weight 1.640     item osd.84 weight 1.640     item osd.85 weight 1.640     item osd.86 weight 1.640     item osd.87 weight 1.640     item osd.88 weight 1.640     item osd.89 weight 1.640     item osd.90 weight 1.640     item osd.91 weight 1.640     item osd.92 weight 1.640     item osd.93 weight 1.640     item osd.94 weight 1.640     item osd.95 weight 1.640     item osd.96 weight 1.640     item osd.97 weight 1.640     item osd.98 weight 1.640     item osd.99 weight 1.640     item osd.100 weight 1.640     item osd.101 weight 1.640     item osd.102 weight 1.640     item osd.103 weight 1.640     item osd.104 weight 1.640     item osd.105 weight 1.640     item osd.106 weight 1.640     item osd.107 weight 1.640     item osd.108 weight 1.640     item osd.109 weight 1.640     item osd.110 weight 1.640     item osd.111 weight 1.640     item osd.112 weight 1.640     item osd.113 weight 1.640     item osd.114 weight 1.640     item osd.115 weight 1.640     item osd.116 weight 1.640     item osd.117 weight 1.640     item osd.118 weight 1.640     item osd.119 weight 1.640     item osd.120 weight 1.640     item osd.121 weight 1.640     item osd.122 weight 1.640     item osd.123 weight 1.640 } host ld5506-hdd_strgbox {     id -22        # do not change unnecessarily     id -23 class hdd        # do not change unnecessarily     id -24 class nvme        # do not change unnecessarily     # weight 78.720     alg straw2     hash 0    # rjenkins1     item osd.124 weight 1.640     item osd.125 weight 1.640     item osd.126 weight 1.640     item osd.127 weight 1.640     item osd.128 weight 1.640     item osd.129 weight 1.640     item osd.130 weight 1.640     item osd.131 weight 1.640     item osd.132 weight 1.640     item osd.133 weight 1.640     item osd.134 weight 1.640     item osd.135 weight 1.640     item osd.136 weight 1.640     item osd.137 weight 1.640     item osd.138 weight 1.640     item osd.139 weight 1.640     item osd.140 weight 1.640     item osd.141 weight 1.640     item osd.142 weight 1.640     item osd.143 weight 1.640     item osd.144 weight 1.640     item osd.145 weight 1.640     item osd.146 weight 1.640     item osd.147 weight 1.640     item osd.148 weight 1.640     item osd.149 weight 1.640     item osd.150 weight 1.640     item osd.151 weight 1.640     item osd.152 weight 1.640     item osd.153 weight 1.640     item osd.154 weight 1.640     item osd.155 weight 1.640     item osd.156 weight 1.640     item osd.157 weight 1.640     item osd.158 weight 1.640     item osd.159 weight 1.640     item osd.160 weight 1.640     item osd.161 weight 1.640     item osd.162 weight 1.640     item osd.163 weight 1.640     item osd.164 weight 1.640     item osd.165 weight 1.640     item osd.166 weight 1.640     item osd.167 weight 1.640     item osd.168 weight 1.640     item osd.169 weight 1.640     item osd.170 weight 1.640     item osd.171 weight 1.640 } host ld5507-hdd_strgbox {     id -25        # do not change unnecessarily     id -26 class hdd        # do not change unnecessarily     id -27 class nvme        # do not change unnecessarily     # weight 78.720     alg straw2     hash 0    # rjenkins1     item osd.215 weight 1.640     item osd.216 weight 1.640     item osd.217 weight 1.640     item osd.218 weight 1.640     item osd.219 weight 1.640     item osd.251 weight 1.640     item osd.252 weight 1.640     item osd.253 weight 1.640     item osd.254 weight 1.640     item osd.255 weight 1.640     item osd.256 weight 1.640     item osd.257 weight 1.640     item osd.258 weight 1.640     item osd.259 weight 1.640     item osd.260 weight 1.640     item osd.261 weight 1.640     item osd.262 weight 1.640     item osd.263 weight 1.640     item osd.264 weight 1.640     item osd.265 weight 1.640     item osd.266 weight 1.640     item osd.267 weight 1.640     item osd.189 weight 1.640     item osd.190 weight 1.640     item osd.191 weight 1.640     item osd.192 weight 1.640     item osd.193 weight 1.640     item osd.194 weight 1.640     item osd.195 weight 1.640     item osd.196 weight 1.640     item osd.197 weight 1.640     item osd.198 weight 1.640     item osd.199 weight 1.640     item osd.200 weight 1.640     item osd.201 weight 1.640     item osd.202 weight 1.640     item osd.203 weight 1.640     item osd.204 weight 1.640     item osd.205 weight 1.640     item osd.206 weight 1.640     item osd.207 weight 1.640     item osd.208 weight 1.640     item osd.209 weight 1.640     item osd.210 weight 1.640     item osd.211 weight 1.640     item osd.212 weight 1.640     item osd.213 weight 1.640     item osd.214 weight 1.640 } host ld5508-hdd_strgbox {     id -28        # do not change unnecessarily     id -29 class hdd        # do not change unnecessarily     id -30 class nvme        # do not change unnecessarily     # weight 78.720     alg straw2     hash 0    # rjenkins1     item osd.42 weight 1.640     item osd.43 weight 1.640     item osd.44 weight 1.640     item osd.45 weight 1.640     item osd.46 weight 1.640     item osd.47 weight 1.640     item osd.48 weight 1.640     item osd.49 weight 1.640     item osd.50 weight 1.640     item osd.51 weight 1.640     item osd.52 weight 1.640     item osd.53 weight 1.640     item osd.54 weight 1.640     item osd.55 weight 1.640     item osd.56 weight 1.640     item osd.57 weight 1.640     item osd.58 weight 1.640     item osd.220 weight 1.640     item osd.221 weight 1.640     item osd.222 weight 1.640     item osd.223 weight 1.640     item osd.224 weight 1.640     item osd.225 weight 1.640     item osd.226 weight 1.640     item osd.227 weight 1.640     item osd.228 weight 1.640     item osd.230 weight 1.640     item osd.231 weight 1.640     item osd.232 weight 1.640     item osd.233 weight 1.640     item osd.234 weight 1.640     item osd.235 weight 1.640     item osd.236 weight 1.640     item osd.237 weight 1.640     item osd.238 weight 1.640     item osd.239 weight 1.640     item osd.240 weight 1.640     item osd.241 weight 1.640     item osd.242 weight 1.640     item osd.243 weight 1.640     item osd.244 weight 1.640     item osd.245 weight 1.640     item osd.246 weight 1.640     item osd.247 weight 1.640     item osd.248 weight 1.640     item osd.249 weight 1.640     item osd.250 weight 1.640     item osd.229 weight 1.640 } host ld4464-hdd_strgbox {     id -31        # do not change unnecessarily     id -33 class hdd        # do not change unnecessarily     id -32 class nvme        # do not change unnecessarily     # weight 349.440     alg straw2     hash 0    # rjenkins1     item osd.272 weight 7.280     item osd.273 weight 7.280     item osd.274 weight 7.280     item osd.275 weight 7.280     item osd.276 weight 7.280     item osd.277 weight 7.280     item osd.278 weight 7.280     item osd.279 weight 7.280     item osd.280 weight 7.280     item osd.281 weight 7.280     item osd.282 weight 7.280     item osd.283 weight 7.280     item osd.284 weight 7.280     item osd.285 weight 7.280     item osd.286 weight 7.280     item osd.287 weight 7.280     item osd.288 weight 7.280     item osd.289 weight 7.280     item osd.290 weight 7.280     item osd.291 weight 7.280     item osd.292 weight 7.280     item osd.293 weight 7.280     item osd.294 weight 7.280     item osd.295 weight 7.280     item osd.296 weight 7.280     item osd.297 weight 7.280     item osd.298 weight 7.280     item osd.299 weight 7.280     item osd.300 weight 7.280     item osd.302 weight 7.280     item osd.303 weight 7.280     item osd.304 weight 7.280     item osd.305 weight 7.280     item osd.306 weight 7.280     item osd.307 weight 7.280     item osd.308 weight 7.280     item osd.309 weight 7.280     item osd.310 weight 7.280     item osd.311 weight 7.280     item osd.312 weight 7.280     item osd.313 weight 7.280     item osd.314 weight 7.280     item osd.315 weight 7.280     item osd.316 weight 7.280     item osd.317 weight 7.280     item osd.318 weight 7.280     item osd.301 weight 7.280     item osd.319 weight 7.280 } host ld4465-hdd_strgbox {     id -40        # do not change unnecessarily     id -42 class hdd        # do not change unnecessarily     id -41 class nvme        # do not change unnecessarily     # weight 349.440     alg straw2     hash 0    # rjenkins1     item osd.320 weight 7.280     item osd.321 weight 7.280     item osd.322 weight 7.280     item osd.323 weight 7.280     item osd.324 weight 7.280     item osd.325 weight 7.280     item osd.326 weight 7.280     item osd.327 weight 7.280     item osd.328 weight 7.280     item osd.329 weight 7.280     item osd.330 weight 7.280     item osd.331 weight 7.280     item osd.332 weight 7.280     item osd.333 weight 7.280     item osd.334 weight 7.280     item osd.335 weight 7.280     item osd.336 weight 7.280     item osd.337 weight 7.280     item osd.338 weight 7.280     item osd.339 weight 7.280     item osd.340 weight 7.280     item osd.341 weight 7.280     item osd.342 weight 7.280     item osd.343 weight 7.280     item osd.344 weight 7.280     item osd.345 weight 7.280     item osd.346 weight 7.280     item osd.347 weight 7.280     item osd.348 weight 7.280     item osd.349 weight 7.280     item osd.350 weight 7.280     item osd.351 weight 7.280     item osd.352 weight 7.280     item osd.353 weight 7.280     item osd.354 weight 7.280     item osd.355 weight 7.280     item osd.356 weight 7.280     item osd.357 weight 7.280     item osd.358 weight 7.280     item osd.360 weight 7.280     item osd.361 weight 7.280     item osd.362 weight 7.280     item osd.363 weight 7.280     item osd.364 weight 7.280     item osd.365 weight 7.280     item osd.366 weight 7.280     item osd.367 weight 7.280     item osd.359 weight 7.280 } root hdd_strgbox {     id -17        # do not change unnecessarily     id -19 class hdd        # do not change unnecessarily     id -21 class nvme        # do not change unnecessarily     # weight 1013.760     alg straw2     hash 0    # rjenkins1     item ld5505-hdd_strgbox weight 78.720     item ld5506-hdd_strgbox weight 78.720     item ld5507-hdd_strgbox weight 78.720     item ld5508-hdd_strgbox weight 78.720     item ld4464-hdd_strgbox weight 349.440     item ld4465-hdd_strgbox weight 349.440 } # rules rule replicated_rule {     id 0     type replicated     min_size 1     max_size 10     step take default     step chooseleaf firstn 0 type host     step emit } rule hdd_strgbox_rule {     id 1     type replicated     min_size 1     max_size 10     step take hdd_strgbox class hdd     step chooseleaf firstn 0 type host     step emit } rule nvme_rule {     id 2     type replicated     min_size 1     max_size 10     step take default class nvme     step chooseleaf firstn 0 type host     step emit } rule hdd_rule {     id 3     type replicated     min_size 1     max_size 10     step take default class hdd     step chooseleaf firstn 0 type host     step emit } # end crush map Am 24.09.2019 um 23:33 schrieb Reed Dier: