Without having looked too closely yet, you might need "chooseleaf" when it's supposed to select OSDs, this is just an example from one of my test nodes: step take default step choose indep 0 type host step chooseleaf indep 2 type osd step emit Zitat von Denis Polom <denispolom@gmail.com>:
Hi folks,
on our Ceph v19.2.1 distributed across 4 failure domains we have a EC pool using following crush rule: { "rule_id": 4, "rule_name": "ec53", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_indep", "num": 4, "type": "tor" }, { "op": "choose_indep", "num": 2, "type": "osd" }, { "op": "emit" } ] }
and following EC profile
crush-device-class= crush-failure-domain=tor crush-root=default jerasure-per-chunk-alignment=false k=5 m=3 plugin=jerasure technique=reed_sol_van w=8
here is our crush tree:
ID CLASS WEIGHT TYPE NAME -40 15.71931 root metadata -41 3.49318 tor s1-bos-e-10-ng.metadata -42 1.74660 host osd1-metadata 98 ssd 1.74660 osd.98 -49 1.74660 host osd2-metadata 99 ssd 1.74660 osd.99 -124 3.49318 tor s1-bos-f-7-ng.metadata -52 1.74660 host osd3-metadata 103 ssd 1.74660 osd.103 -76 1.74660 host osd9-metadata 106 ssd 1.74660 osd.106 -56 5.23978 tor s1-bos-f-8-ng.metadata -55 1.74660 host osd4-metadata 85 ssd 1.74660 osd.85 -61 1.74660 host osd5-metadata 93 ssd 1.74660 osd.93 -64 1.74660 host osd6-metadata 97 ssd 1.74660 osd.97 -67 3.49318 tor s1-bos-f-9-ng.metadata -70 1.74660 host osd7-metadata 104 ssd 1.74660 osd.104 -73 1.74660 host osd8-metadata 105 ssd 1.74660 osd.105 -1 2733.78516 root default -21 745.57776 tor s1-bos-e-10-ng -7 80.40544 host osd1 0 hdd 7.30959 osd.0 10 hdd 7.30959 osd.10 26 hdd 7.30959 osd.26 28 hdd 7.30959 osd.28 44 hdd 7.30959 osd.44 53 hdd 7.30959 osd.53 62 hdd 7.30959 osd.62 70 hdd 7.30959 osd.70 79 hdd 7.30959 osd.79 89 hdd 7.30959 osd.89 107 hdd 7.30959 osd.107 -88 116.95337 host osd12 128 hdd 7.30959 osd.128 139 hdd 7.30959 osd.139 142 hdd 7.30959 osd.142 146 hdd 7.30959 osd.146 150 hdd 7.30959 osd.150 152 hdd 7.30959 osd.152 153 hdd 7.30959 osd.153 154 hdd 7.30959 osd.154 191 hdd 7.30959 osd.191 192 hdd 7.30959 osd.192 193 hdd 7.30959 osd.193 194 hdd 7.30959 osd.194 195 hdd 7.30959 osd.195 196 hdd 7.30959 osd.196 199 hdd 7.30959 osd.199 200 hdd 7.30959 osd.200 -94 116.95337 host osd15 156 hdd 7.30959 osd.156 157 hdd 7.30959 osd.157 158 hdd 7.30959 osd.158 159 hdd 7.30959 osd.159 160 hdd 7.30959 osd.160 161 hdd 7.30959 osd.161 162 hdd 7.30959 osd.162 163 hdd 7.30959 osd.163 164 hdd 7.30959 osd.164 165 hdd 7.30959 osd.165 166 hdd 7.30959 osd.166 167 hdd 7.30959 osd.167 171 hdd 7.30959 osd.171 172 hdd 7.30959 osd.172 177 hdd 7.30959 osd.177 183 hdd 7.30959 osd.183 -17 80.40544 host osd2 4 hdd 7.30959 osd.4 15 hdd 7.30959 osd.15 19 hdd 7.30959 osd.19 33 hdd 7.30959 osd.33 38 hdd 7.30959 osd.38 52 hdd 7.30959 osd.52 58 hdd 7.30959 osd.58 69 hdd 7.30959 osd.69 73 hdd 7.30959 osd.73 81 hdd 7.30959 osd.81 95 hdd 7.30959 osd.95 -115 87.71503 host osd22 275 hdd 7.30959 osd.275 276 hdd 7.30959 osd.276 277 hdd 7.30959 osd.277 278 hdd 7.30959 osd.278 279 hdd 7.30959 osd.279 280 hdd 7.30959 osd.280 281 hdd 7.30959 osd.281 282 hdd 7.30959 osd.282 283 hdd 7.30959 osd.283 284 hdd 7.30959 osd.284 285 hdd 7.30959 osd.285 286 hdd 7.30959 osd.286 -118 87.71503 host osd23 287 hdd 7.30959 osd.287 288 hdd 7.30959 osd.288 289 hdd 7.30959 osd.289 290 hdd 7.30959 osd.290 291 hdd 7.30959 osd.291 292 hdd 7.30959 osd.292 293 hdd 7.30959 osd.293 294 hdd 7.30959 osd.294 295 hdd 7.30959 osd.295 296 hdd 7.30959 osd.296 297 hdd 7.30959 osd.297 298 hdd 7.30959 osd.298 -130 87.71503 host osd25 311 hdd 7.30959 osd.311 312 hdd 7.30959 osd.312 313 hdd 7.30959 osd.313 314 hdd 7.30959 osd.314 315 hdd 7.30959 osd.315 316 hdd 7.30959 osd.316 317 hdd 7.30959 osd.317 318 hdd 7.30959 osd.318 319 hdd 7.30959 osd.319 320 hdd 7.30959 osd.320 321 hdd 7.30959 osd.321 322 hdd 7.30959 osd.322 -134 87.71503 host osd29 359 hdd 7.30959 osd.359 360 hdd 7.30959 osd.360 361 hdd 7.30959 osd.361 362 hdd 7.30959 osd.362 363 hdd 7.30959 osd.363 364 hdd 7.30959 osd.364 365 hdd 7.30959 osd.365 366 hdd 7.30959 osd.366 367 hdd 7.30959 osd.367 368 hdd 7.30959 osd.368 369 hdd 7.30959 osd.369 370 hdd 7.30959 osd.370 -127 774.81604 tor s1-bos-f-7-ng -79 116.95337 host osd10 108 hdd 7.30959 osd.108 109 hdd 7.30959 osd.109 110 hdd 7.30959 osd.110 111 hdd 7.30959 osd.111 112 hdd 7.30959 osd.112 113 hdd 7.30959 osd.113 114 hdd 7.30959 osd.114 115 hdd 7.30959 osd.115 116 hdd 7.30959 osd.116 117 hdd 7.30959 osd.117 118 hdd 7.30959 osd.118 119 hdd 7.30959 osd.119 120 hdd 7.30959 osd.120 121 hdd 7.30959 osd.121 122 hdd 7.30959 osd.122 123 hdd 7.30959 osd.123 -85 116.95337 host osd11 125 hdd 7.30959 osd.125 126 hdd 7.30959 osd.126 127 hdd 7.30959 osd.127 129 hdd 7.30959 osd.129 138 hdd 7.30959 osd.138 144 hdd 7.30959 osd.144 149 hdd 7.30959 osd.149 151 hdd 7.30959 osd.151 188 hdd 7.30959 osd.188 189 hdd 7.30959 osd.189 190 hdd 7.30959 osd.190 197 hdd 7.30959 osd.197 198 hdd 7.30959 osd.198 201 hdd 7.30959 osd.201 202 hdd 7.30959 osd.202 203 hdd 7.30959 osd.203 -91 116.95337 host osd13 130 hdd 7.30959 osd.130 134 hdd 7.30959 osd.134 135 hdd 7.30959 osd.135 140 hdd 7.30959 osd.140 143 hdd 7.30959 osd.143 147 hdd 7.30959 osd.147 148 hdd 7.30959 osd.148 155 hdd 7.30959 osd.155 168 hdd 7.30959 osd.168 169 hdd 7.30959 osd.169 170 hdd 7.30959 osd.170 176 hdd 7.30959 osd.176 178 hdd 7.30959 osd.178 180 hdd 7.30959 osd.180 182 hdd 7.30959 osd.182 184 hdd 7.30959 osd.184 -121 87.71503 host osd24 299 hdd 7.30959 osd.299 300 hdd 7.30959 osd.300 301 hdd 7.30959 osd.301 302 hdd 7.30959 osd.302 303 hdd 7.30959 osd.303 304 hdd 7.30959 osd.304 305 hdd 7.30959 osd.305 306 hdd 7.30959 osd.306 307 hdd 7.30959 osd.307 308 hdd 7.30959 osd.308 309 hdd 7.30959 osd.309 310 hdd 7.30959 osd.310 -131 87.71503 host osd26 323 hdd 7.30959 osd.323 324 hdd 7.30959 osd.324 325 hdd 7.30959 osd.325 326 hdd 7.30959 osd.326 327 hdd 7.30959 osd.327 328 hdd 7.30959 osd.328 329 hdd 7.30959 osd.329 330 hdd 7.30959 osd.330 331 hdd 7.30959 osd.331 332 hdd 7.30959 osd.332 333 hdd 7.30959 osd.333 334 hdd 7.30959 osd.334 -5 80.40544 host osd3 2 hdd 7.30959 osd.2 12 hdd 7.30959 osd.12 21 hdd 7.30959 osd.21 34 hdd 7.30959 osd.34 42 hdd 7.30959 osd.42 50 hdd 7.30959 osd.50 54 hdd 7.30959 osd.54 63 hdd 7.30959 osd.63 100 hdd 7.30959 osd.100 101 hdd 7.30959 osd.101 102 hdd 7.30959 osd.102 -135 87.71503 host osd30 371 hdd 7.30959 osd.371 372 hdd 7.30959 osd.372 373 hdd 7.30959 osd.373 374 hdd 7.30959 osd.374 375 hdd 7.30959 osd.375 376 hdd 7.30959 osd.376 377 hdd 7.30959 osd.377 378 hdd 7.30959 osd.378 379 hdd 7.30959 osd.379 380 hdd 7.30959 osd.380 381 hdd 7.30959 osd.381 382 hdd 7.30959 osd.382 -13 80.40544 host osd9 8 hdd 7.30959 osd.8 13 hdd 7.30959 osd.13 24 hdd 7.30959 osd.24 32 hdd 7.30959 osd.32 39 hdd 7.30959 osd.39 48 hdd 7.30959 osd.48 56 hdd 7.30959 osd.56 64 hdd 7.30959 osd.64 75 hdd 7.30959 osd.75 84 hdd 7.30959 osd.84 94 hdd 7.30959 osd.94 -25 584.76685 tor s1-bos-f-8-ng -97 87.71503 host osd16 216 hdd 7.30959 osd.216 217 hdd 7.30959 osd.217 218 hdd 7.30959 osd.218 219 hdd 7.30959 osd.219 220 hdd 7.30959 osd.220 221 hdd 7.30959 osd.221 222 hdd 7.30959 osd.222 223 hdd 7.30959 osd.223 224 hdd 7.30959 osd.224 225 hdd 7.30959 osd.225 226 hdd 7.30959 osd.226 227 hdd 7.30959 osd.227 -100 87.71503 host osd17 204 hdd 7.30959 osd.204 205 hdd 7.30959 osd.205 206 hdd 7.30959 osd.206 207 hdd 7.30959 osd.207 208 hdd 7.30959 osd.208 209 hdd 7.30959 osd.209 210 hdd 7.30959 osd.210 211 hdd 7.30959 osd.211 212 hdd 7.30959 osd.212 213 hdd 7.30959 osd.213 214 hdd 7.30959 osd.214 215 hdd 7.30959 osd.215 -103 80.40544 host osd18 228 hdd 7.30959 osd.228 229 hdd 7.30959 osd.229 230 hdd 7.30959 osd.230 231 hdd 7.30959 osd.231 232 hdd 7.30959 osd.232 233 hdd 7.30959 osd.233 234 hdd 7.30959 osd.234 235 hdd 7.30959 osd.235 236 hdd 7.30959 osd.236 237 hdd 7.30959 osd.237 238 hdd 7.30959 osd.238 -132 87.71503 host osd27 335 hdd 7.30959 osd.335 336 hdd 7.30959 osd.336 337 hdd 7.30959 osd.337 338 hdd 7.30959 osd.338 339 hdd 7.30959 osd.339 340 hdd 7.30959 osd.340 341 hdd 7.30959 osd.341 342 hdd 7.30959 osd.342 343 hdd 7.30959 osd.343 344 hdd 7.30959 osd.344 345 hdd 7.30959 osd.345 346 hdd 7.30959 osd.346 -11 80.40544 host osd4 3 hdd 7.30959 osd.3 16 hdd 7.30959 osd.16 18 hdd 7.30959 osd.18 31 hdd 7.30959 osd.31 36 hdd 7.30959 osd.36 49 hdd 7.30959 osd.49 59 hdd 7.30959 osd.59 68 hdd 7.30959 osd.68 72 hdd 7.30959 osd.72 77 hdd 7.30959 osd.77 82 hdd 7.30959 osd.82 -19 80.40544 host osd5 6 hdd 7.30959 osd.6 11 hdd 7.30959 osd.11 25 hdd 7.30959 osd.25 27 hdd 7.30959 osd.27 41 hdd 7.30959 osd.41 46 hdd 7.30959 osd.46 60 hdd 7.30959 osd.60 66 hdd 7.30959 osd.66 80 hdd 7.30959 osd.80 88 hdd 7.30959 osd.88 91 hdd 7.30959 osd.91 -3 80.40544 host osd6 7 hdd 7.30959 osd.7 14 hdd 7.30959 osd.14 20 hdd 7.30959 osd.20 30 hdd 7.30959 osd.30 37 hdd 7.30959 osd.37 47 hdd 7.30959 osd.47 57 hdd 7.30959 osd.57 67 hdd 7.30959 osd.67 74 hdd 7.30959 osd.74 83 hdd 7.30959 osd.83 96 hdd 7.30959 osd.96 -23 628.62439 tor s1-bos-f-9-ng -82 116.95337 host osd14 124 hdd 7.30959 osd.124 131 hdd 7.30959 osd.131 132 hdd 7.30959 osd.132 133 hdd 7.30959 osd.133 136 hdd 7.30959 osd.136 137 hdd 7.30959 osd.137 141 hdd 7.30959 osd.141 145 hdd 7.30959 osd.145 173 hdd 7.30959 osd.173 174 hdd 7.30959 osd.174 175 hdd 7.30959 osd.175 179 hdd 7.30959 osd.179 181 hdd 7.30959 osd.181 185 hdd 7.30959 osd.185 186 hdd 7.30959 osd.186 187 hdd 7.30959 osd.187 -106 87.71503 host osd19 239 hdd 7.30959 osd.239 240 hdd 7.30959 osd.240 241 hdd 7.30959 osd.241 242 hdd 7.30959 osd.242 243 hdd 7.30959 osd.243 244 hdd 7.30959 osd.244 245 hdd 7.30959 osd.245 246 hdd 7.30959 osd.246 247 hdd 7.30959 osd.247 248 hdd 7.30959 osd.248 249 hdd 7.30959 osd.249 250 hdd 7.30959 osd.250 -109 87.71503 host osd20 251 hdd 7.30959 osd.251 252 hdd 7.30959 osd.252 253 hdd 7.30959 osd.253 254 hdd 7.30959 osd.254 255 hdd 7.30959 osd.255 256 hdd 7.30959 osd.256 257 hdd 7.30959 osd.257 258 hdd 7.30959 osd.258 259 hdd 7.30959 osd.259 260 hdd 7.30959 osd.260 261 hdd 7.30959 osd.261 262 hdd 7.30959 osd.262 -112 87.71503 host osd21 263 hdd 7.30959 osd.263 264 hdd 7.30959 osd.264 265 hdd 7.30959 osd.265 266 hdd 7.30959 osd.266 267 hdd 7.30959 osd.267 268 hdd 7.30959 osd.268 269 hdd 7.30959 osd.269 270 hdd 7.30959 osd.270 271 hdd 7.30959 osd.271 272 hdd 7.30959 osd.272 273 hdd 7.30959 osd.273 274 hdd 7.30959 osd.274 -133 87.71503 host osd28 347 hdd 7.30959 osd.347 348 hdd 7.30959 osd.348 349 hdd 7.30959 osd.349 350 hdd 7.30959 osd.350 351 hdd 7.30959 osd.351 352 hdd 7.30959 osd.352 353 hdd 7.30959 osd.353 354 hdd 7.30959 osd.354 355 hdd 7.30959 osd.355 356 hdd 7.30959 osd.356 357 hdd 7.30959 osd.357 358 hdd 7.30959 osd.358 -15 80.40544 host osd7 1 hdd 7.30959 osd.1 9 hdd 7.30959 osd.9 22 hdd 7.30959 osd.22 35 hdd 7.30959 osd.35 43 hdd 7.30959 osd.43 51 hdd 7.30959 osd.51 55 hdd 7.30959 osd.55 65 hdd 7.30959 osd.65 76 hdd 7.30959 osd.76 86 hdd 7.30959 osd.86 92 hdd 7.30959 osd.92 -9 80.40544 host osd8 5 hdd 7.30959 osd.5 17 hdd 7.30959 osd.17 23 hdd 7.30959 osd.23 29 hdd 7.30959 osd.29 40 hdd 7.30959 osd.40 45 hdd 7.30959 osd.45 61 hdd 7.30959 osd.61 71 hdd 7.30959 osd.71 78 hdd 7.30959 osd.78 87 hdd 7.30959 osd.87 90 hdd 7.30959 osd.90
Ceph MGR balancer is running in upmap mode, clients are "luminous".
During one FD s1-bos-f-9-ng outage there were many PGs in inactive state, for example:
pg 11.558 is stuck inactive for 102s, current state undersized+degraded+peered, last acting [176,169,NONE,NONE,199,NONE,227,221]
this clearly shows that PG was distributed unevenly through FDs as according crush rule it should be placed on 2 OSDs per FD.
# for i in 176 169 199 227 221;do echo -n "$i: "; ceph osd find $i | jq -r .crush_location.tor;done 176: s1-bos-f-7-ng 169: s1-bos-f-7-ng 199: s1-bos-e-10-ng 227: s1-bos-f-8-ng 221: s1-bos-f-8-ng
# ceph osd tree | grep tor -41 3.49318 tor s1-bos-e-10-ng.metadata -124 3.49318 tor s1-bos-f-7-ng.metadata -56 5.23978 tor s1-bos-f-8-ng.metadata -67 3.49318 tor s1-bos-f-9-ng.metadata -21 745.57776 tor s1-bos-e-10-ng -127 774.81604 tor s1-bos-f-7-ng -25 584.76685 tor s1-bos-f-8-ng -23 628.62439 tor s1-bos-f-9-ng
after our FD came up I did run pg-upmap-items manually to make placement correct.
But after enabling MGR balancer there were attempts to move PG placement again to different OSD under different FD that breaks crush rule.
Now I know this cluster FD weights are very different in CRUSH map. So I've tried to use osdmaptool to simulate balancer. Our balancer settings were:
mgr advanced mgr/balancer/mode upmap mgr advanced mgr/balancer/upmap_max_deviation 1 mgr advanced mgr/balancer/upmap_max_optimizations 100
So I tried to adjust max deviation to level, that balancer would not try to move PGs to OSDs in different FD that breaks crush rule. I ended up on value 50, and honestly don't know if at this value balancer will try to do anything anytime.
From my perspective MGR balancer is ignoring CRUSH rule - at least on EC pools. I observed this behaviour on our another Ceph cluster and also only on EC pool even if FD weights there were almost same.
I even tried TheJJ Ceph-balancer, and it breaks crush rule also.
Any ideas or similar experiences?
Thank you
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io