Which Ceph release are you running? You mention the balancer, which would imply a certain lower bound. What does `ceph balancer status` show?

Does anyone know how I can rebalance my cluster to balance out the OSD usage?

I just added 12 more 14Tb HDDs to my cluster (cluster of made up of 12Tb and 14Tb disks) bringing my total to 48 OSDs. Ceph df reports my pool as 83% full (see below). I am aware this only reports the fullest drive. However, after backfilling, data distribution is still heavily skewed to a few OSDs. The oldest are at ~80% and the newest are at ~50. My issue is that data is currently being written to all drives equally and at this rate the cluster will start disabling writes despite still having significant raw capacity. PGs do not seem to be evenly distributed either. Weights are 12.73 for all 14Tb OSD and 10.9 for all 12Tb OSDs. I tried using the balancer to create a new plan but it had no real effect. Haven't seen much useful info online.

--- RAW STORAGE ---

CLASS SIZE AVAIL USED RAW USED %RAW USED

hdd 582 TiB 203 TiB 379 TiB 380 TiB 65.20

TOTAL 582 TiB 203 TiB 379 TiB 380 TiB 65.20

--- POOLS ---

POOL ID STORED OBJECTS USED %USED MAX AVAIL

cephfs_data 2 261 TiB 68.62M 379 TiB 83.64 56 TiB

cephfs_metadata 3 536 MiB 12.74k 1.6 GiB 0 25 TiB

`osd df`:

ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS

0 hdd 10.91409 1.00000 11 TiB 8.4 TiB 8.4 TiB 1.6 MiB 22 GiB 2.5 TiB 77.00 1.18 92 up

4 hdd 10.91409 1.00000 11 TiB 6.9 TiB 6.9 TiB 21 MiB 18 GiB 4.0 TiB 63.03 0.97 80 up

5 hdd 10.91409 1.00000 11 TiB 6.5 TiB 6.5 TiB 1.3 MiB 17 GiB 4.4 TiB 59.50 0.91 71 up

11 hdd 12.73340 1.00000 13 TiB 9.4 TiB 9.4 TiB 36 MiB 24 GiB 3.3 TiB 74.13 1.14 105 up

15 hdd 12.73340 1.00000 13 TiB 8.2 TiB 8.2 TiB 20 MiB 21 GiB 4.5 TiB 64.75 0.99 99 up

16 hdd 10.91409 1.00000 11 TiB 6.4 TiB 6.4 TiB 1.7 MiB 16 GiB 4.5 TiB 58.81 0.90 74 up

20 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 33 MiB 23 GiB 4.3 TiB 66.09 1.01 102 up

22 hdd 12.73340 1.00000 13 TiB 8.6 TiB 8.6 TiB 3.3 MiB 22 GiB 4.1 TiB 67.48 1.04 100 up

23 hdd 12.73340 1.00000 13 TiB 7.6 TiB 7.6 TiB 406 KiB 20 GiB 5.1 TiB 59.93 0.92 89 up

36 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.2 TiB 171 KiB 16 GiB 5.5 TiB 57.00 0.87 85 up

41 hdd 12.73340 1.00000 13 TiB 7.0 TiB 7.0 TiB 2.1 MiB 16 GiB 5.7 TiB 54.85 0.84 85 up

46 hdd 12.73340 1.00000 13 TiB 8.1 TiB 8.0 TiB 67 MiB 19 GiB 4.7 TiB 63.34 0.97 99 up

3 hdd 10.91409 1.00000 11 TiB 7.6 TiB 7.6 TiB 1.0 MiB 19 GiB 3.3 TiB 69.85 1.07 87 up

6 hdd 10.91409 1.00000 11 TiB 8.3 TiB 8.3 TiB 39 MiB 22 GiB 2.6 TiB 76.28 1.17 93 up

10 hdd 12.73340 1.00000 13 TiB 9.6 TiB 9.6 TiB 5.1 MiB 24 GiB 3.1 TiB 75.52 1.16 109 up

14 hdd 12.73340 1.00000 13 TiB 8.9 TiB 8.8 TiB 34 MiB 22 GiB 3.9 TiB 69.57 1.07 99 up

24 hdd 12.73340 1.00000 13 TiB 11 TiB 11 TiB 22 MiB 34 GiB 2.1 TiB 83.28 1.28 117 up

25 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 24 MiB 21 GiB 4.4 TiB 65.84 1.01 97 up

26 hdd 12.73340 1.00000 13 TiB 8.9 TiB 8.9 TiB 205 KiB 22 GiB 3.8 TiB 69.84 1.07 102 up

30 hdd 10.91409 1.00000 11 TiB 6.9 TiB 6.9 TiB 33 MiB 16 GiB 4.0 TiB 63.31 0.97 92 up

39 hdd 12.73340 1.00000 13 TiB 7.2 TiB 7.2 TiB 36 MiB 16 GiB 5.6 TiB 56.39 0.86 93 up

43 hdd 12.73340 1.00000 13 TiB 8.3 TiB 8.3 TiB 3.6 MiB 19 GiB 4.4 TiB 65.54 1.01 98 up

47 hdd 12.73340 1.00000 13 TiB 7.1 TiB 7.1 TiB 51 MiB 17 GiB 5.6 TiB 55.70 0.85 88 up

2 hdd 10.91409 1.00000 11 TiB 7.4 TiB 7.3 TiB 22 KiB 19 GiB 3.6 TiB 67.36 1.03 82 up

7 hdd 10.91409 1.00000 11 TiB 8.5 TiB 8.5 TiB 530 KiB 21 GiB 2.4 TiB 77.89 1.19 97 up

8 hdd 12.73340 1.00000 13 TiB 8.5 TiB 8.5 TiB 21 MiB 22 GiB 4.3 TiB 66.53 1.02 99 up

12 hdd 12.73340 1.00000 13 TiB 8.5 TiB 8.5 TiB 20 MiB 22 GiB 4.2 TiB 67.13 1.03 99 up

18 hdd 12.73340 1.00000 13 TiB 9.9 TiB 9.9 TiB 24 MiB 31 GiB 2.8 TiB 77.81 1.19 112 up

19 hdd 12.73340 1.00000 13 TiB 8.8 TiB 8.8 TiB 34 MiB 22 GiB 3.9 TiB 69.17 1.06 104 up

21 hdd 12.73340 1.00000 13 TiB 9.5 TiB 9.4 TiB 15 MiB 28 GiB 3.3 TiB 74.37 1.14 111 up

31 hdd 10.91409 1.00000 11 TiB 6.3 TiB 6.3 TiB 56 MiB 14 GiB 4.6 TiB 57.53 0.88 77 up

32 hdd 10.91409 1.00000 11 TiB 8.0 TiB 8.0 TiB 44 MiB 18 GiB 2.9 TiB 73.15 1.12 99 up

37 hdd 12.73340 1.00000 13 TiB 8.2 TiB 8.1 TiB 4.2 MiB 18 GiB 4.6 TiB 64.09 0.98 94 up

40 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.3 TiB 1.4 MiB 16 GiB 5.5 TiB 57.10 0.88 86 up

44 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 2.6 MiB 17 GiB 5.0 TiB 60.55 0.93 92 up

1 hdd 10.91409 1.00000 11 TiB 5.9 TiB 5.9 TiB 55 MiB 15 GiB 5.0 TiB 54.14 0.83 72 up

9 hdd 12.73340 1.00000 13 TiB 9.4 TiB 9.3 TiB 705 KiB 24 GiB 3.4 TiB 73.57 1.13 109 up

13 hdd 12.73340 1.00000 13 TiB 8.0 TiB 8.0 TiB 22 MiB 21 GiB 4.7 TiB 62.74 0.96 91 up

17 hdd 10.91409 1.00000 11 TiB 7.5 TiB 7.5 TiB 24 MiB 19 GiB 3.4 TiB 68.88 1.06 88 up

27 hdd 12.73340 1.00000 13 TiB 6.9 TiB 6.8 TiB 4.8 MiB 17 GiB 5.9 TiB 53.81 0.83 78 up

28 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 19 MiB 20 GiB 5.0 TiB 60.46 0.93 91 up

29 hdd 12.73340 1.00000 13 TiB 8.3 TiB 8.3 TiB 17 MiB 21 GiB 4.4 TiB 65.56 1.01 100 up

33 hdd 10.91409 1.00000 11 TiB 6.5 TiB 6.5 TiB 1.6 MiB 15 GiB 4.4 TiB 59.23 0.91 77 up

34 hdd 10.91409 1.00000 11 TiB 6.6 TiB 6.6 TiB 44 MiB 15 GiB 4.3 TiB 60.87 0.93 81 up

35 hdd 10.91409 1.00000 11 TiB 6.6 TiB 6.5 TiB 18 MiB 15 GiB 4.4 TiB 60.08 0.92 81 up

38 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 18 MiB 19 GiB 4.3 TiB 66.25 1.02 105 up

42 hdd 12.73340 1.00000 13 TiB 6.9 TiB 6.9 TiB 21 MiB 16 GiB 5.8 TiB 54.26 0.83 84 up

45 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 38 MiB 18 GiB 5.0 TiB 60.61 0.93 94 up

TOTAL 582 TiB 380 TiB 379 TiB 942 MiB 950 GiB 203 TiB 65.20

Thank you,

Ricardo

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io