Hi Mohamed, Changing weights is no longer a good practice. The balancer is supposed to do the job. The number of pg per osd is really tight on your infrastructure. Can you display the ceph osd tree command? ________________________________________________________ Cordialement, *David CASIER* *Ligne directe: +33(0) 9 72 61 98 29* ________________________________________________________ Le lun. 11 déc. 2023 à 11:06, Mohamed LAMDAOUAR <mohamed.lamdaouar@enyx.fr> a écrit :
Hello the team,
We initially had a cluster of 3 machines with 4 osd on each machine, we added 4 machines in the cluster (each machine with 4 osd) We launched the balancing but it never finished, still in progress. But the big issue: we have an osd full and all the pools on this osd are read only.
*ceph osd df *:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 20 hdd 9.09569 1.00000 9.1 TiB 580 GiB 576 GiB 1.2 GiB 3.1 GiB 8.5 TiB 6.23 0.32 169 up 21 hdd 9.09569 1.00000 9.1 TiB 1.5 TiB 1.5 TiB 252 MiB 7.7 GiB 7.6 TiB 16.08 0.82 247 up 22 hdd 9.09569 1.00000 9.1 TiB 671 GiB 667 GiB 204 MiB 4.1 GiB 8.4 TiB 7.21 0.37 136 up 23 hdd 9.09569 1.00000 9.1 TiB 665 GiB 660 GiB 270 MiB 4.5 GiB 8.4 TiB 7.14 0.37 124 up 0 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 87 MiB 6.0 GiB 7.9 TiB 13.30 0.68 230 up 1 hdd 9.09569 1.00000 9.1 TiB 1.3 TiB 1.3 TiB 347 MiB 6.6 GiB 7.8 TiB 14.01 0.72 153 up 2 hdd 9.09569 0.65009 9.1 TiB 1.8 TiB 1.8 TiB 443 MiB 7.3 GiB 7.3 TiB 20.00 1.03 147 up 3 hdd 9.09569 1.00000 9.1 TiB 617 GiB 611 GiB 220 MiB 5.8 GiB 8.5 TiB 6.62 0.34 101 up 4 hdd 9.09569 0.80005 9.1 TiB 2.0 TiB 2.0 TiB 293 MiB 8.2 GiB 7.1 TiB 22.12 1.13 137 up 5 hdd 9.09569 1.00000 9.1 TiB 857 GiB 852 GiB 157 MiB 4.9 GiB 8.3 TiB 9.20 0.47 155 up 6 hdd 9.09569 1.00000 9.1 TiB 580 GiB 575 GiB 678 MiB 4.4 GiB 8.5 TiB 6.23 0.32 114 up 7 hdd 9.09569 0.50000 9.1 TiB 7.7 TiB 7.7 TiB 103 MiB 16 GiB 1.4 TiB 85.03 4.36 201 up 24 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 133 MiB 6.2 GiB 7.9 TiB 13.11 0.67 225 up 25 hdd 9.09569 0.34999 9.1 TiB 8.3 TiB 8.2 TiB 101 MiB 17 GiB 860 GiB 90.77 4.66 159 up 26 hdd 9.09569 1.00000 9.1 TiB 665 GiB 661 GiB 292 MiB 3.8 GiB 8.4 TiB 7.14 0.37 107 up 27 hdd 9.09569 1.00000 9.1 TiB 427 GiB 423 GiB 241 MiB 3.4 GiB 8.7 TiB 4.58 0.24 103 up 8 hdd 9.09569 1.00000 9.1 TiB 845 GiB 839 GiB 831 MiB 5.9 GiB 8.3 TiB 9.07 0.47 163 up 9 hdd 9.09569 1.00000 9.1 TiB 727 GiB 722 GiB 162 MiB 4.8 GiB 8.4 TiB 7.80 0.40 169 up 10 hdd 9.09569 0.80005 9.1 TiB 1.9 TiB 1.9 TiB 742 MiB 7.5 GiB 7.2 TiB 21.01 1.08 136 up 11 hdd 9.09569 1.00000 9.1 TiB 733 GiB 727 GiB 498 MiB 5.2 GiB 8.4 TiB 7.87 0.40 163 up 12 hdd 9.09569 1.00000 9.1 TiB 892 GiB 886 GiB 318 MiB 5.6 GiB 8.2 TiB 9.58 0.49 254 up 13 hdd 9.09569 1.00000 9.1 TiB 759 GiB 755 GiB 37 MiB 4.0 GiB 8.4 TiB 8.15 0.42 134 up 14 hdd 9.09569 0.85004 9.1 TiB 2.3 TiB 2.3 TiB 245 MiB 7.7 GiB 6.8 TiB 24.96 1.28 142 up 15 hdd 9.09569 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 435 MiB 16 GiB 1.8 TiB 80.17 4.11 213 up 16 hdd 9.09569 1.00000 9.1 TiB 784 GiB 781 GiB 104 MiB 3.6 GiB 8.3 TiB 8.42 0.43 247 up 17 hdd 9.09569 1.00000 9.1 TiB 861 GiB 856 GiB 269 MiB 5.1 GiB 8.3 TiB 9.25 0.47 102 up 18 hdd 9.09569 1.00000 9.1 TiB 1.9 TiB 1.9 TiB 962 MiB 8.2 GiB 7.2 TiB 21.15 1.09 283 up 19 hdd 9.09569 1.00000 9.1 TiB 893 GiB 888 GiB 291 MiB 4.6 GiB 8.2 TiB 9.59 0.49 148 up TOTAL 255 TiB 50 TiB 49 TiB 9.7 GiB 187 GiB 205 TiB 19.49 MIN/MAX VAR: 0.24/4.66 STDDEV: 19.63
*ceph health detail |grep -i wrn* [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set [WRN] OSD_NEARFULL: 2 nearfull osd(s) [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 16 pgs backfill_toofull [WRN] PG_NOT_DEEP_SCRUBBED: 1360 pgs not deep-scrubbed in time [WRN] PG_NOT_SCRUBBED: 53 pgs not scrubbed in time [WRN] POOL_NEARFULL: 36 pool(s) nearfull
Thanks the team ;) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io