13 Mar
2021
13 Mar
'21
2:48 p.m.
Hello people,
I am still struggeling with the balancer
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
Now I've read some more and might think that I do not have enough PGs.
Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I
have the autoscaler enabled, but I doesn't tell me to increase the
PGs.
What do you think?
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
13 Mar
13 Mar
2:56 p.m.
No, increasing num PGs won't help substantially.
Can you share the entire output of ceph osd df tree ?
Did you already set
ceph config set mgr mgr/balancer/upmap_max_deviation 1
??
And I recommend debug_mgr 4/5 so you can see some basic upmap balancer
logging.
.. Dan
On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <bb(a)kervyn.de> wrote:
Hello people,
I am still struggeling with the balancer
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
Now I've read some more and might think that I do not have enough PGs.
Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I
have the autoscaler enabled, but I doesn't tell me to increase the
PGs.
What do you think?
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:11 p.m.
Hi Dan,
upmap_max_deviation is default (5) in our cluster. Is 1 the recommended
deviation?
I added the whole ceph osd df tree, (I need to remove some OSDs and readd
them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I also
reweighted to try to get the %USE mitigated).
I will increase the mgr debugging to see what is the problem.
[root@s3db1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS TYPE NAME
-1 673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB 168
TiB 74.57 1.00 - root default
-2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB 22
TiB 49.82 0.67 - host s3db1
23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB 13
TiB 12.50 0.17 101 up osd.23
69 hdd 14.55269 0 0 B 0 B 0 B 0 B 0 B 0
B 0 0 11 up osd.69
73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB 4.2
TiB 71.15 0.95 107 up osd.73
79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B 747
GiB 79.94 1.07 52 up osd.79
80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B 1.0
TiB 71.61 0.96 58 up osd.80
81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B 1.5
TiB 60.07 0.81 55 up osd.81
82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B 1.7
TiB 52.68 0.71 30 up osd.82
-11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB 13
TiB 74.69 1.00 - host s3db10
63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB 1.7
TiB 76.04 1.02 92 up osd.63
64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB 1.8
TiB 75.54 1.01 101 up osd.64
65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB 2.0
TiB 73.17 0.98 105 up osd.65
66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB 1.4
TiB 80.38 1.08 98 up osd.66
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB 2.2
TiB 70.10 0.94 100 up osd.67
68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB 2.0
TiB 72.88 0.98 107 up osd.68
70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB 1.8
TiB 74.73 1.00 102 up osd.70
-12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB 12
TiB 77.24 1.04 - host s3db11
46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB 1.5
TiB 78.91 1.06 97 up osd.46
47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB 2.1
TiB 71.47 0.96 96 up osd.47
48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB 1.2
TiB 83.46 1.12 109 up osd.48
49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB 1.5
TiB 78.96 1.06 98 up osd.49
50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB 1.4
TiB 80.40 1.08 102 up osd.50
51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB 1.3
TiB 81.70 1.10 110 up osd.51
72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB 2.5
TiB 65.82 0.88 89 up osd.72
-37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB 12
TiB 79.04 1.06 - host s3db12
19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB 559
GiB 85.18 1.14 55 up osd.19
71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB 825
GiB 78.14 1.05 50 up osd.71
75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB 555
GiB 85.29 1.14 57 up osd.75
76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB 501
GiB 86.73 1.16 50 up osd.76
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB 3.6
TiB 75.57 1.01 202 up osd.77
78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB 3.4
TiB 76.65 1.03 196 up osd.78
83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB 2.9
TiB 80.04 1.07 223 up osd.83
-3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB 16
TiB 73.15 0.98 - host s3db2
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB 3.6
TiB 75.52 1.01 194 up osd.1
3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B 1.4
TiB 60.94 0.82 52 up osd.3
4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B 401
GiB 89.23 1.20 53 up osd.4
5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B 1.3
TiB 62.88 0.84 51 up osd.5
6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B 1.7
TiB 54.51 0.73 47 up osd.6
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26 GiB 3.8
TiB 73.90 0.99 185 up osd.7
74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 3.5
TiB 76.27 1.02 208 up osd.74
-4 58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB 86 GiB 15
TiB 74.05 0.99 - host s3db3
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26 GiB 4.0
TiB 72.78 0.98 203 up osd.2
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB 3.6
TiB 75.62 1.01 189 up osd.9
10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28 GiB 3.5
TiB 76.34 1.02 211 up osd.10
12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B 1.2
TiB 66.22 0.89 44 up osd.12
13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B 720
GiB 80.66 1.08 66 up osd.13
14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B 552
GiB 85.18 1.14 60 up osd.14
15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B 1.7
TiB 53.72 0.72 44 up osd.15
-5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB 14
TiB 76.37 1.02 - host s3db4
11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28 GiB 2.8
TiB 81.15 1.09 205 up osd.11
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB 3.6
TiB 75.38 1.01 211 up osd.17
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44 GiB 4.0
TiB 72.86 0.98 188 up osd.18
20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B 796
GiB 78.63 1.05 66 up osd.20
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B 1.1
TiB 70.32 0.94 47 up osd.21
22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B 802
GiB 78.47 1.05 58 up osd.22
24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B 856
GiB 77.01 1.03 47 up osd.24
-6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB 15
TiB 75.22 1.01 - host s3db5
0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB 1.3
TiB 65.00 0.87 48 up osd.0
25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB 1.4
TiB 63.86 0.86 41 up osd.25
26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB 862
GiB 77.47 1.04 48 up osd.26
27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB 1.5
TiB 61.00 0.82 48 up osd.27
28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30 GiB 2.8
TiB 81.19 1.09 203 up osd.28
29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26 GiB 3.8
TiB 73.95 0.99 200 up osd.29
30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28 GiB 3.1
TiB 79.07 1.06 207 up osd.30
-7 58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB 122 GiB 14
TiB 75.41 1.01 - host s3db6
32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB 622
GiB 83.74 1.12 65 up osd.32
33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB 740
GiB 80.67 1.08 52 up osd.33
34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB 877
GiB 77.08 1.03 52 up osd.34
35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB 1.4
TiB 62.18 0.83 42 up osd.35
36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26 GiB 4.0
TiB 72.98 0.98 220 up osd.36
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB 3.6
TiB 75.30 1.01 200 up osd.37
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB 3.3
TiB 77.43 1.04 217 up osd.38
-8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB 12
TiB 78.98 1.06 - host s3db7
39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB 499
GiB 86.96 1.17 43 up osd.39
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB 1.2
TiB 68.33 0.92 39 up osd.40
41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB 732
GiB 80.86 1.08 64 up osd.41
42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB 1.2
TiB 67.97 0.91 50 up osd.42
43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28 GiB 2.9
TiB 80.32 1.08 213 up osd.43
44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB 3.1
TiB 78.59 1.05 198 up osd.44
45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB 2.8
TiB 81.05 1.09 214 up osd.45
-9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB 10
TiB 79.75 1.07 - host s3db8
8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB 1.5
TiB 79.68 1.07 99 up osd.8
16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB 1.4
TiB 80.25 1.08 97 up osd.16
31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB 1.8
TiB 75.36 1.01 87 up osd.31
52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB 1.6
TiB 78.19 1.05 113 up osd.52
53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB 1.1
TiB 84.46 1.13 109 up osd.53
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB 1.7
TiB 76.73 1.03 115 up osd.54
55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB 1.2
TiB 83.56 1.12 122 up osd.55
-10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB 12
TiB 76.05 1.02 - host s3db9
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB 2.1
TiB 71.06 0.95 105 up osd.56
57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB 1.2
TiB 83.17 1.12 102 up osd.57
58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB 1.4
TiB 81.56 1.09 105 up osd.58
59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB 1.4
TiB 80.64 1.08 94 up osd.59
60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB 2.0
TiB 73.25 0.98 101 up osd.60
61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB 2.5
TiB 65.84 0.88 103 up osd.61
62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB 1.7
TiB 76.83 1.03 126 up osd.62
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB 173
TiB 74.57
MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
No, increasing num PGs won't help substantially.
Can you share the entire output of ceph osd df tree ?
Did you already set
ceph config set mgr mgr/balancer/upmap_max_deviation 1
??
And I recommend debug_mgr 4/5 so you can see some basic upmap balancer
logging.
.. Dan
On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hello people,
I am still struggeling with the balancer
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
Now I've read some more and might think that I do not have enough PGs.
Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I
have the autoscaler enabled, but I doesn't tell me to increase the
PGs.
What do you think?
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:20 p.m.
Thanks.
Decreasing the max deviation to 2 or 1 should help in your case. This
option controls when the balancer stops trying to move PGs around -- by
default it stops when the deviation from the mean is 5. Yes this is too
large IMO -- all of our clusters have this set to 1.
And given that you have some OSDs with more than 200 PGs, you definitely
shouldn't increase the num PGs.
But anyway with your mixed device sizes it might be challenging to make a
perfectly uniform distribution. Give it a try with 1 though, and let us
know how it goes.
.. Dan
On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <bb(a)kervyn.de> wrote:
Hi Dan,
upmap_max_deviation is default (5) in our cluster. Is 1 the recommended
deviation?
I added the whole ceph osd df tree, (I need to remove some OSDs and readd
them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I also
reweighted to try to get the %USE mitigated).
I will increase the mgr debugging to see what is the problem.
[root@s3db1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS TYPE NAME
-1 673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB 168
TiB 74.57 1.00 - root default
-2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB 22
TiB 49.82 0.67 - host s3db1
23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB 13
TiB 12.50 0.17 101 up osd.23
69 hdd 14.55269 0 0 B 0 B 0 B 0 B 0 B 0
B 0 0 11 up osd.69
73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB 4.2
TiB 71.15 0.95 107 up osd.73
79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B 747
GiB 79.94 1.07 52 up osd.79
80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B 1.0
TiB 71.61 0.96 58 up osd.80
81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B 1.5
TiB 60.07 0.81 55 up osd.81
82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B 1.7
TiB 52.68 0.71 30 up osd.82
-11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB 13
TiB 74.69 1.00 - host s3db10
63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB 1.7
TiB 76.04 1.02 92 up osd.63
64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB 1.8
TiB 75.54 1.01 101 up osd.64
65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB 2.0
TiB 73.17 0.98 105 up osd.65
66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB 1.4
TiB 80.38 1.08 98 up osd.66
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB 2.2
TiB 70.10 0.94 100 up osd.67
68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB 2.0
TiB 72.88 0.98 107 up osd.68
70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB 1.8
TiB 74.73 1.00 102 up osd.70
-12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB 12
TiB 77.24 1.04 - host s3db11
46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB 1.5
TiB 78.91 1.06 97 up osd.46
47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB 2.1
TiB 71.47 0.96 96 up osd.47
48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB 1.2
TiB 83.46 1.12 109 up osd.48
49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB 1.5
TiB 78.96 1.06 98 up osd.49
50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB 1.4
TiB 80.40 1.08 102 up osd.50
51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB 1.3
TiB 81.70 1.10 110 up osd.51
72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB 2.5
TiB 65.82 0.88 89 up osd.72
-37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB 12
TiB 79.04 1.06 - host s3db12
19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB 559
GiB 85.18 1.14 55 up osd.19
71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB 825
GiB 78.14 1.05 50 up osd.71
75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB 555
GiB 85.29 1.14 57 up osd.75
76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB 501
GiB 86.73 1.16 50 up osd.76
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB 3.6
TiB 75.57 1.01 202 up osd.77
78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB 3.4
TiB 76.65 1.03 196 up osd.78
83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB 2.9
TiB 80.04 1.07 223 up osd.83
-3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB 16
TiB 73.15 0.98 - host s3db2
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB 3.6
TiB 75.52 1.01 194 up osd.1
3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B 1.4
TiB 60.94 0.82 52 up osd.3
4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B 401
GiB 89.23 1.20 53 up osd.4
5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B 1.3
TiB 62.88 0.84 51 up osd.5
6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B 1.7
TiB 54.51 0.73 47 up osd.6
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26 GiB 3.8
TiB 73.90 0.99 185 up osd.7
74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 3.5
TiB 76.27 1.02 208 up osd.74
-4 58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB 86 GiB 15
TiB 74.05 0.99 - host s3db3
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26 GiB 4.0
TiB 72.78 0.98 203 up osd.2
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB 3.6
TiB 75.62 1.01 189 up osd.9
10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28 GiB 3.5
TiB 76.34 1.02 211 up osd.10
12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B 1.2
TiB 66.22 0.89 44 up osd.12
13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B 720
GiB 80.66 1.08 66 up osd.13
14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B 552
GiB 85.18 1.14 60 up osd.14
15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B 1.7
TiB 53.72 0.72 44 up osd.15
-5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB 14
TiB 76.37 1.02 - host s3db4
11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28 GiB 2.8
TiB 81.15 1.09 205 up osd.11
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB 3.6
TiB 75.38 1.01 211 up osd.17
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44 GiB 4.0
TiB 72.86 0.98 188 up osd.18
20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B 796
GiB 78.63 1.05 66 up osd.20
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B 1.1
TiB 70.32 0.94 47 up osd.21
22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B 802
GiB 78.47 1.05 58 up osd.22
24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B 856
GiB 77.01 1.03 47 up osd.24
-6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB 15
TiB 75.22 1.01 - host s3db5
0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB 1.3
TiB 65.00 0.87 48 up osd.0
25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB 1.4
TiB 63.86 0.86 41 up osd.25
26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB 862
GiB 77.47 1.04 48 up osd.26
27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB 1.5
TiB 61.00 0.82 48 up osd.27
28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30 GiB 2.8
TiB 81.19 1.09 203 up osd.28
29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26 GiB 3.8
TiB 73.95 0.99 200 up osd.29
30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28 GiB 3.1
TiB 79.07 1.06 207 up osd.30
-7 58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB 122 GiB 14
TiB 75.41 1.01 - host s3db6
32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB 622
GiB 83.74 1.12 65 up osd.32
33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB 740
GiB 80.67 1.08 52 up osd.33
34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB 877
GiB 77.08 1.03 52 up osd.34
35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB 1.4
TiB 62.18 0.83 42 up osd.35
36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26 GiB 4.0
TiB 72.98 0.98 220 up osd.36
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB 3.6
TiB 75.30 1.01 200 up osd.37
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB 3.3
TiB 77.43 1.04 217 up osd.38
-8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB 12
TiB 78.98 1.06 - host s3db7
39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB 499
GiB 86.96 1.17 43 up osd.39
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB 1.2
TiB 68.33 0.92 39 up osd.40
41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB 732
GiB 80.86 1.08 64 up osd.41
42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB 1.2
TiB 67.97 0.91 50 up osd.42
43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28 GiB 2.9
TiB 80.32 1.08 213 up osd.43
44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB 3.1
TiB 78.59 1.05 198 up osd.44
45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB 2.8
TiB 81.05 1.09 214 up osd.45
-9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB 10
TiB 79.75 1.07 - host s3db8
8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB 1.5
TiB 79.68 1.07 99 up osd.8
16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB 1.4
TiB 80.25 1.08 97 up osd.16
31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB 1.8
TiB 75.36 1.01 87 up osd.31
52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB 1.6
TiB 78.19 1.05 113 up osd.52
53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB 1.1
TiB 84.46 1.13 109 up osd.53
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB 1.7
TiB 76.73 1.03 115 up osd.54
55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB 1.2
TiB 83.56 1.12 122 up osd.55
-10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB 12
TiB 76.05 1.02 - host s3db9
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB 2.1
TiB 71.06 0.95 105 up osd.56
57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB 1.2
TiB 83.17 1.12 102 up osd.57
58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB 1.4
TiB 81.56 1.09 105 up osd.58
59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB 1.4
TiB 80.64 1.08 94 up osd.59
60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB 2.0
TiB 73.25 0.98 101 up osd.60
61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB 2.5
TiB 65.84 0.88 103 up osd.61
62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB 1.7
TiB 76.83 1.03 126 up osd.62
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB 173
TiB 74.57
MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
No, increasing num PGs won't help
substantially.
Can you share the entire output of ceph osd df tree ?
Did you already set
ceph config set mgr mgr/balancer/upmap_max_deviation 1
??
And I recommend debug_mgr 4/5 so you can see some basic upmap balancer
logging.
.. Dan
On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hello people,
I am still struggeling with the balancer
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
Now I've read some more and might think that I do not have enough PGs.
Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I
have the autoscaler enabled, but I doesn't tell me to increase the
PGs.
What do you think?
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:35 p.m.
Hi,
ok thanks. I just changed the value and rewighted everything back to 1. Now
I let it sync the weekend and check how it will be on monday.
We tried to have the systems total storage balanced as possible. New
systems will be with 8TB disks but for the exiting ones we added 16TB to
offset the 4TB disks and we needed a lot of storage fast, because of a DC
move. If you have any recommendations I would be happy to hear them.
Cheers
Boris
Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
Thanks.
Decreasing the max deviation to 2 or 1 should help in your case. This
option controls when the balancer stops trying to move PGs around -- by
default it stops when the deviation from the mean is 5. Yes this is too
large IMO -- all of our clusters have this set to 1.
And given that you have some OSDs with more than 200 PGs, you definitely
shouldn't increase the num PGs.
But anyway with your mixed device sizes it might be challenging to make a
perfectly uniform distribution. Give it a try with 1 though, and let us
know how it goes.
.. Dan
On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hi Dan,
upmap_max_deviation is default (5) in our cluster. Is 1 the recommended
deviation?
I added the whole ceph osd df tree, (I need to remove some OSDs and readd
them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I also
reweighted to try to get the %USE mitigated).
I will increase the mgr debugging to see what is the problem.
[root@s3db1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB 168
TiB 74.57 1.00 - root default
-2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB 22
TiB 49.82 0.67 - host s3db1
23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB 13
TiB 12.50 0.17 101 up osd.23
69 hdd 14.55269 0 0 B 0 B 0 B 0 B 0 B
0 B 0 0 11 up osd.69
73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB 4.2
TiB 71.15 0.95 107 up osd.73
79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B 747
GiB 79.94 1.07 52 up osd.79
80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B 1.0
TiB 71.61 0.96 58 up osd.80
81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B 1.5
TiB 60.07 0.81 55 up osd.81
82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B 1.7
TiB 52.68 0.71 30 up osd.82
-11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB 13
TiB 74.69 1.00 - host s3db10
63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB 1.7
TiB 76.04 1.02 92 up osd.63
64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB 1.8
TiB 75.54 1.01 101 up osd.64
65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB 2.0
TiB 73.17 0.98 105 up osd.65
66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB 1.4
TiB 80.38 1.08 98 up osd.66
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB 2.2
TiB 70.10 0.94 100 up osd.67
68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB 2.0
TiB 72.88 0.98 107 up osd.68
70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB 1.8
TiB 74.73 1.00 102 up osd.70
-12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB 12
TiB 77.24 1.04 - host s3db11
46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB 1.5
TiB 78.91 1.06 97 up osd.46
47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB 2.1
TiB 71.47 0.96 96 up osd.47
48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB 1.2
TiB 83.46 1.12 109 up osd.48
49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB 1.5
TiB 78.96 1.06 98 up osd.49
50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB 1.4
TiB 80.40 1.08 102 up osd.50
51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB 1.3
TiB 81.70 1.10 110 up osd.51
72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB 2.5
TiB 65.82 0.88 89 up osd.72
-37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB 12
TiB 79.04 1.06 - host s3db12
19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB 559
GiB 85.18 1.14 55 up osd.19
71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB 825
GiB 78.14 1.05 50 up osd.71
75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB 555
GiB 85.29 1.14 57 up osd.75
76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB 501
GiB 86.73 1.16 50 up osd.76
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB 3.6
TiB 75.57 1.01 202 up osd.77
78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB 3.4
TiB 76.65 1.03 196 up osd.78
83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB 2.9
TiB 80.04 1.07 223 up osd.83
-3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB 16
TiB 73.15 0.98 - host s3db2
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB 3.6
TiB 75.52 1.01 194 up osd.1
3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B 1.4
TiB 60.94 0.82 52 up osd.3
4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B 401
GiB 89.23 1.20 53 up osd.4
5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B 1.3
TiB 62.88 0.84 51 up osd.5
6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B 1.7
TiB 54.51 0.73 47 up osd.6
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26 GiB 3.8
TiB 73.90 0.99 185 up osd.7
74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 3.5
TiB 76.27 1.02 208 up osd.74
-4 58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB 86 GiB 15
TiB 74.05 0.99 - host s3db3
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26 GiB 4.0
TiB 72.78 0.98 203 up osd.2
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB 3.6
TiB 75.62 1.01 189 up osd.9
10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28 GiB 3.5
TiB 76.34 1.02 211 up osd.10
12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B 1.2
TiB 66.22 0.89 44 up osd.12
13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B 720
GiB 80.66 1.08 66 up osd.13
14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B 552
GiB 85.18 1.14 60 up osd.14
15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B 1.7
TiB 53.72 0.72 44 up osd.15
-5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB 14
TiB 76.37 1.02 - host s3db4
11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28 GiB 2.8
TiB 81.15 1.09 205 up osd.11
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB 3.6
TiB 75.38 1.01 211 up osd.17
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44 GiB 4.0
TiB 72.86 0.98 188 up osd.18
20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B 796
GiB 78.63 1.05 66 up osd.20
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B 1.1
TiB 70.32 0.94 47 up osd.21
22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B 802
GiB 78.47 1.05 58 up osd.22
24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B 856
GiB 77.01 1.03 47 up osd.24
-6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB 15
TiB 75.22 1.01 - host s3db5
0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB 1.3
TiB 65.00 0.87 48 up osd.0
25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB 1.4
TiB 63.86 0.86 41 up osd.25
26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB 862
GiB 77.47 1.04 48 up osd.26
27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB 1.5
TiB 61.00 0.82 48 up osd.27
28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30 GiB 2.8
TiB 81.19 1.09 203 up osd.28
29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26 GiB 3.8
TiB 73.95 0.99 200 up osd.29
30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28 GiB 3.1
TiB 79.07 1.06 207 up osd.30
-7 58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB 122 GiB 14
TiB 75.41 1.01 - host s3db6
32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB 622
GiB 83.74 1.12 65 up osd.32
33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB 740
GiB 80.67 1.08 52 up osd.33
34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB 877
GiB 77.08 1.03 52 up osd.34
35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB 1.4
TiB 62.18 0.83 42 up osd.35
36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26 GiB 4.0
TiB 72.98 0.98 220 up osd.36
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB 3.6
TiB 75.30 1.01 200 up osd.37
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB 3.3
TiB 77.43 1.04 217 up osd.38
-8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB 12
TiB 78.98 1.06 - host s3db7
39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB 499
GiB 86.96 1.17 43 up osd.39
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB 1.2
TiB 68.33 0.92 39 up osd.40
41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB 732
GiB 80.86 1.08 64 up osd.41
42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB 1.2
TiB 67.97 0.91 50 up osd.42
43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28 GiB 2.9
TiB 80.32 1.08 213 up osd.43
44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB 3.1
TiB 78.59 1.05 198 up osd.44
45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB 2.8
TiB 81.05 1.09 214 up osd.45
-9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB 10
TiB 79.75 1.07 - host s3db8
8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB 1.5
TiB 79.68 1.07 99 up osd.8
16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB 1.4
TiB 80.25 1.08 97 up osd.16
31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB 1.8
TiB 75.36 1.01 87 up osd.31
52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB 1.6
TiB 78.19 1.05 113 up osd.52
53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB 1.1
TiB 84.46 1.13 109 up osd.53
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB 1.7
TiB 76.73 1.03 115 up osd.54
55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB 1.2
TiB 83.56 1.12 122 up osd.55
-10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB 12
TiB 76.05 1.02 - host s3db9
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB 2.1
TiB 71.06 0.95 105 up osd.56
57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB 1.2
TiB 83.17 1.12 102 up osd.57
58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB 1.4
TiB 81.56 1.09 105 up osd.58
59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB 1.4
TiB 80.64 1.08 94 up osd.59
60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB 2.0
TiB 73.25 0.98 101 up osd.60
61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB 2.5
TiB 65.84 0.88 103 up osd.61
62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB 1.7
TiB 76.83 1.03 126 up osd.62
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB 173
TiB 74.57
MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
No, increasing num PGs won't help
substantially.
Can you share the entire output of ceph osd df tree ?
Did you already set
ceph config set mgr mgr/balancer/upmap_max_deviation 1
??
And I recommend debug_mgr 4/5 so you can see some basic upmap balancer
logging.
.. Dan
On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hello people,
I am still struggeling with the balancer
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
Now I've read some more and might think that I do not have enough PGs.
Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I
have the autoscaler enabled, but I doesn't tell me to increase the
PGs.
What do you think?
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:37 p.m.
OK
Btw, you might need to fail to a new mgr... I'm not sure if the current
active will read that new config.
.. dan
On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <bb(a)kervyn.de> wrote:
Hi,
ok thanks. I just changed the value and rewighted everything back to 1.
Now I let it sync the weekend and check how it will be on monday.
We tried to have the systems total storage balanced as possible. New
systems will be with 8TB disks but for the exiting ones we added 16TB to
offset the 4TB disks and we needed a lot of storage fast, because of a DC
move. If you have any recommendations I would be happy to hear them.
Cheers
Boris
Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
Thanks.
Decreasing the max deviation to 2 or 1 should help in your case. This
option controls when the balancer stops trying to move PGs around -- by
default it stops when the deviation from the mean is 5. Yes this is too
large IMO -- all of our clusters have this set to 1.
And given that you have some OSDs with more than 200 PGs, you definitely
shouldn't increase the num PGs.
But anyway with your mixed device sizes it might be challenging to make a
perfectly uniform distribution. Give it a try with 1 though, and let us
know how it goes.
.. Dan
On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hi Dan,
upmap_max_deviation is default (5) in our cluster. Is 1 the recommended
deviation?
I added the whole ceph osd df tree, (I need to remove some OSDs and
readd them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I
also reweighted to try to get the %USE mitigated).
I will increase the mgr debugging to see what is the problem.
[root@s3db1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB 168
TiB 74.57 1.00 - root default
-2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB 22
TiB 49.82 0.67 - host s3db1
23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB 13
TiB 12.50 0.17 101 up osd.23
69 hdd 14.55269 0 0 B 0 B 0 B 0 B 0 B
0 B 0 0 11 up osd.69
73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB 4.2
TiB 71.15 0.95 107 up osd.73
79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B 747
GiB 79.94 1.07 52 up osd.79
80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B 1.0
TiB 71.61 0.96 58 up osd.80
81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B 1.5
TiB 60.07 0.81 55 up osd.81
82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B 1.7
TiB 52.68 0.71 30 up osd.82
-11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB 13
TiB 74.69 1.00 - host s3db10
63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB 1.7
TiB 76.04 1.02 92 up osd.63
64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB 1.8
TiB 75.54 1.01 101 up osd.64
65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB 2.0
TiB 73.17 0.98 105 up osd.65
66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB 1.4
TiB 80.38 1.08 98 up osd.66
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB 2.2
TiB 70.10 0.94 100 up osd.67
68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB 2.0
TiB 72.88 0.98 107 up osd.68
70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB 1.8
TiB 74.73 1.00 102 up osd.70
-12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB 12
TiB 77.24 1.04 - host s3db11
46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB 1.5
TiB 78.91 1.06 97 up osd.46
47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB 2.1
TiB 71.47 0.96 96 up osd.47
48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB 1.2
TiB 83.46 1.12 109 up osd.48
49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB 1.5
TiB 78.96 1.06 98 up osd.49
50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB 1.4
TiB 80.40 1.08 102 up osd.50
51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB 1.3
TiB 81.70 1.10 110 up osd.51
72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB 2.5
TiB 65.82 0.88 89 up osd.72
-37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB 12
TiB 79.04 1.06 - host s3db12
19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB 559
GiB 85.18 1.14 55 up osd.19
71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB 825
GiB 78.14 1.05 50 up osd.71
75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB 555
GiB 85.29 1.14 57 up osd.75
76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB 501
GiB 86.73 1.16 50 up osd.76
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB 3.6
TiB 75.57 1.01 202 up osd.77
78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB 3.4
TiB 76.65 1.03 196 up osd.78
83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB 2.9
TiB 80.04 1.07 223 up osd.83
-3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB 16
TiB 73.15 0.98 - host s3db2
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB 3.6
TiB 75.52 1.01 194 up osd.1
3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B 1.4
TiB 60.94 0.82 52 up osd.3
4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B 401
GiB 89.23 1.20 53 up osd.4
5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B 1.3
TiB 62.88 0.84 51 up osd.5
6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B 1.7
TiB 54.51 0.73 47 up osd.6
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26 GiB 3.8
TiB 73.90 0.99 185 up osd.7
74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 3.5
TiB 76.27 1.02 208 up osd.74
-4 58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB 86 GiB 15
TiB 74.05 0.99 - host s3db3
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26 GiB 4.0
TiB 72.78 0.98 203 up osd.2
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB 3.6
TiB 75.62 1.01 189 up osd.9
10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28 GiB 3.5
TiB 76.34 1.02 211 up osd.10
12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B 1.2
TiB 66.22 0.89 44 up osd.12
13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B 720
GiB 80.66 1.08 66 up osd.13
14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B 552
GiB 85.18 1.14 60 up osd.14
15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B 1.7
TiB 53.72 0.72 44 up osd.15
-5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB 14
TiB 76.37 1.02 - host s3db4
11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28 GiB 2.8
TiB 81.15 1.09 205 up osd.11
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB 3.6
TiB 75.38 1.01 211 up osd.17
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44 GiB 4.0
TiB 72.86 0.98 188 up osd.18
20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B 796
GiB 78.63 1.05 66 up osd.20
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B 1.1
TiB 70.32 0.94 47 up osd.21
22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B 802
GiB 78.47 1.05 58 up osd.22
24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B 856
GiB 77.01 1.03 47 up osd.24
-6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB 15
TiB 75.22 1.01 - host s3db5
0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB 1.3
TiB 65.00 0.87 48 up osd.0
25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB 1.4
TiB 63.86 0.86 41 up osd.25
26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB 862
GiB 77.47 1.04 48 up osd.26
27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB 1.5
TiB 61.00 0.82 48 up osd.27
28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30 GiB 2.8
TiB 81.19 1.09 203 up osd.28
29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26 GiB 3.8
TiB 73.95 0.99 200 up osd.29
30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28 GiB 3.1
TiB 79.07 1.06 207 up osd.30
-7 58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB 122 GiB 14
TiB 75.41 1.01 - host s3db6
32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB 622
GiB 83.74 1.12 65 up osd.32
33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB 740
GiB 80.67 1.08 52 up osd.33
34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB 877
GiB 77.08 1.03 52 up osd.34
35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB 1.4
TiB 62.18 0.83 42 up osd.35
36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26 GiB 4.0
TiB 72.98 0.98 220 up osd.36
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB 3.6
TiB 75.30 1.01 200 up osd.37
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB 3.3
TiB 77.43 1.04 217 up osd.38
-8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB 12
TiB 78.98 1.06 - host s3db7
39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB 499
GiB 86.96 1.17 43 up osd.39
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB 1.2
TiB 68.33 0.92 39 up osd.40
41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB 732
GiB 80.86 1.08 64 up osd.41
42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB 1.2
TiB 67.97 0.91 50 up osd.42
43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28 GiB 2.9
TiB 80.32 1.08 213 up osd.43
44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB 3.1
TiB 78.59 1.05 198 up osd.44
45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB 2.8
TiB 81.05 1.09 214 up osd.45
-9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB 10
TiB 79.75 1.07 - host s3db8
8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB 1.5
TiB 79.68 1.07 99 up osd.8
16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB 1.4
TiB 80.25 1.08 97 up osd.16
31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB 1.8
TiB 75.36 1.01 87 up osd.31
52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB 1.6
TiB 78.19 1.05 113 up osd.52
53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB 1.1
TiB 84.46 1.13 109 up osd.53
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB 1.7
TiB 76.73 1.03 115 up osd.54
55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB 1.2
TiB 83.56 1.12 122 up osd.55
-10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB 12
TiB 76.05 1.02 - host s3db9
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB 2.1
TiB 71.06 0.95 105 up osd.56
57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB 1.2
TiB 83.17 1.12 102 up osd.57
58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB 1.4
TiB 81.56 1.09 105 up osd.58
59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB 1.4
TiB 80.64 1.08 94 up osd.59
60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB 2.0
TiB 73.25 0.98 101 up osd.60
61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB 2.5
TiB 65.84 0.88 103 up osd.61
62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB 1.7
TiB 76.83 1.03 126 up osd.62
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB 173
TiB 74.57
MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
No, increasing num PGs won't help
substantially.
Can you share the entire output of ceph osd df tree ?
Did you already set
ceph config set mgr mgr/balancer/upmap_max_deviation 1
??
And I recommend debug_mgr 4/5 so you can see some basic upmap balancer
logging.
.. Dan
On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <bb(a)kervyn.de> wrote:
> Hello people,
>
> I am still struggeling with the balancer
> (https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
> Now I've read some more and might think that I do not have enough PGs.
> Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I
> have the autoscaler enabled, but I doesn't tell me to increase the
> PGs.
>
> What do you think?
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
> im groüen Saal.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
15 Mar
15 Mar
12:47 p.m.
Hi,
this unfortunally did not solve my problem. I still have some OSDs that
fill up to 85%
According to the logging, the autoscaler might want to add more PGs to one
Bucken and reduce almost all other buckets to 32.
2021-03-15 12:19:58.825 7f307f601700 4 mgr[pg_autoscaler] Pool
'eu-central-1.rgw.buckets.data' root_id -1 using 0.705080476146 of space,
bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
Why the balancing does not happen is still nebulous to me.
Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
OK
Btw, you might need to fail to a new mgr... I'm not sure if the current
active will read that new config.
.. dan
On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hi,
ok thanks. I just changed the value and rewighted everything back to 1.
Now I let it sync the weekend and check how it will be on monday.
We tried to have the systems total storage balanced as possible. New
systems will be with 8TB disks but for the exiting ones we added 16TB to
offset the 4TB disks and we needed a lot of storage fast, because of a DC
move. If you have any recommendations I would be happy to hear them.
Cheers
Boris
Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
Thanks.
Decreasing the max deviation to 2 or 1 should help in your case. This
option controls when the balancer stops trying to move PGs around -- by
default it stops when the deviation from the mean is 5. Yes this is too
large IMO -- all of our clusters have this set to 1.
And given that you have some OSDs with more than 200 PGs, you definitely
shouldn't increase the num PGs.
But anyway with your mixed device sizes it might be challenging to make
a perfectly uniform distribution. Give it a try with 1 though, and let us
know how it goes.
.. Dan
On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hi Dan,
upmap_max_deviation is default (5) in our cluster. Is 1 the recommended
deviation?
I added the whole ceph osd df tree, (I need to remove some OSDs and
readd them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I
also reweighted to try to get the %USE mitigated).
I will increase the mgr debugging to see what is the problem.
[root@s3db1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB
168 TiB 74.57 1.00 - root default
-2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB
22 TiB 49.82 0.67 - host s3db1
23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB
13 TiB 12.50 0.17 101 up osd.23
69 hdd 14.55269 0 0 B 0 B 0 B 0 B 0 B
0 B 0 0 11 up osd.69
73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB
4.2 TiB 71.15 0.95 107 up osd.73
79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B
747 GiB 79.94 1.07 52 up osd.79
80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B
1.0 TiB 71.61 0.96 58 up osd.80
81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B
1.5 TiB 60.07 0.81 55 up osd.81
82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B
1.7 TiB 52.68 0.71 30 up osd.82
-11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB
13 TiB 74.69 1.00 - host s3db10
63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB
1.7 TiB 76.04 1.02 92 up osd.63
64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB
1.8 TiB 75.54 1.01 101 up osd.64
65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB
2.0 TiB 73.17 0.98 105 up osd.65
66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB
1.4 TiB 80.38 1.08 98 up osd.66
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB
2.2 TiB 70.10 0.94 100 up osd.67
68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB
2.0 TiB 72.88 0.98 107 up osd.68
70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB
1.8 TiB 74.73 1.00 102 up osd.70
-12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB
12 TiB 77.24 1.04 - host s3db11
46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB
1.5 TiB 78.91 1.06 97 up osd.46
47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB
2.1 TiB 71.47 0.96 96 up osd.47
48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB
1.2 TiB 83.46 1.12 109 up osd.48
49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB
1.5 TiB 78.96 1.06 98 up osd.49
50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB
1.4 TiB 80.40 1.08 102 up osd.50
51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB
1.3 TiB 81.70 1.10 110 up osd.51
72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB
2.5 TiB 65.82 0.88 89 up osd.72
-37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB
12 TiB 79.04 1.06 - host s3db12
19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB
559 GiB 85.18 1.14 55 up osd.19
71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB
825 GiB 78.14 1.05 50 up osd.71
75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB
555 GiB 85.29 1.14 57 up osd.75
76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB
501 GiB 86.73 1.16 50 up osd.76
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB
3.6 TiB 75.57 1.01 202 up osd.77
78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB
3.4 TiB 76.65 1.03 196 up osd.78
83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB
2.9 TiB 80.04 1.07 223 up osd.83
-3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB
16 TiB 73.15 0.98 - host s3db2
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB
3.6 TiB 75.52 1.01 194 up osd.1
3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B
1.4 TiB 60.94 0.82 52 up osd.3
4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B
401 GiB 89.23 1.20 53 up osd.4
5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B
1.3 TiB 62.88 0.84 51 up osd.5
6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B
1.7 TiB 54.51 0.73 47 up osd.6
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26 GiB
3.8 TiB 73.90 0.99 185 up osd.7
74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB
3.5 TiB 76.27 1.02 208 up osd.74
-4 58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB 86 GiB
15 TiB 74.05 0.99 - host s3db3
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26 GiB
4.0 TiB 72.78 0.98 203 up osd.2
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB
3.6 TiB 75.62 1.01 189 up osd.9
10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28 GiB
3.5 TiB 76.34 1.02 211 up osd.10
12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B
1.2 TiB 66.22 0.89 44 up osd.12
13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B
720 GiB 80.66 1.08 66 up osd.13
14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B
552 GiB 85.18 1.14 60 up osd.14
15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B
1.7 TiB 53.72 0.72 44 up osd.15
-5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB
14 TiB 76.37 1.02 - host s3db4
11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28 GiB
2.8 TiB 81.15 1.09 205 up osd.11
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB
3.6 TiB 75.38 1.01 211 up osd.17
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44 GiB
4.0 TiB 72.86 0.98 188 up osd.18
20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B
796 GiB 78.63 1.05 66 up osd.20
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B
1.1 TiB 70.32 0.94 47 up osd.21
22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B
802 GiB 78.47 1.05 58 up osd.22
24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B
856 GiB 77.01 1.03 47 up osd.24
-6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB
15 TiB 75.22 1.01 - host s3db5
0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB
1.3 TiB 65.00 0.87 48 up osd.0
25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB
1.4 TiB 63.86 0.86 41 up osd.25
26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB
862 GiB 77.47 1.04 48 up osd.26
27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB
1.5 TiB 61.00 0.82 48 up osd.27
28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30 GiB
2.8 TiB 81.19 1.09 203 up osd.28
29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26 GiB
3.8 TiB 73.95 0.99 200 up osd.29
30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28 GiB
3.1 TiB 79.07 1.06 207 up osd.30
-7 58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB 122 GiB
14 TiB 75.41 1.01 - host s3db6
32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB
622 GiB 83.74 1.12 65 up osd.32
33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB
740 GiB 80.67 1.08 52 up osd.33
34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB
877 GiB 77.08 1.03 52 up osd.34
35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB
1.4 TiB 62.18 0.83 42 up osd.35
36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26 GiB
4.0 TiB 72.98 0.98 220 up osd.36
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB
3.6 TiB 75.30 1.01 200 up osd.37
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB
3.3 TiB 77.43 1.04 217 up osd.38
-8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB
12 TiB 78.98 1.06 - host s3db7
39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB
499 GiB 86.96 1.17 43 up osd.39
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB
1.2 TiB 68.33 0.92 39 up osd.40
41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB
732 GiB 80.86 1.08 64 up osd.41
42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB
1.2 TiB 67.97 0.91 50 up osd.42
43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28 GiB
2.9 TiB 80.32 1.08 213 up osd.43
44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB
3.1 TiB 78.59 1.05 198 up osd.44
45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB
2.8 TiB 81.05 1.09 214 up osd.45
-9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB
10 TiB 79.75 1.07 - host s3db8
8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB
1.5 TiB 79.68 1.07 99 up osd.8
16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB
1.4 TiB 80.25 1.08 97 up osd.16
31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB
1.8 TiB 75.36 1.01 87 up osd.31
52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB
1.6 TiB 78.19 1.05 113 up osd.52
53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB
1.1 TiB 84.46 1.13 109 up osd.53
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB
1.7 TiB 76.73 1.03 115 up osd.54
55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB
1.2 TiB 83.56 1.12 122 up osd.55
-10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB
12 TiB 76.05 1.02 - host s3db9
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB
2.1 TiB 71.06 0.95 105 up osd.56
57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB
1.2 TiB 83.17 1.12 102 up osd.57
58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB
1.4 TiB 81.56 1.09 105 up osd.58
59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB
1.4 TiB 80.64 1.08 94 up osd.59
60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB
2.0 TiB 73.25 0.98 101 up osd.60
61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB
2.5 TiB 65.84 0.88 103 up osd.61
62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB
1.7 TiB 76.83 1.03 126 up osd.62
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB
173 TiB 74.57
MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> No, increasing num PGs won't help substantially.
>
> Can you share the entire output of ceph osd df tree ?
>
> Did you already set
>
> ceph config set mgr mgr/balancer/upmap_max_deviation 1
>
>
> ??
> And I recommend debug_mgr 4/5 so you can see some basic upmap balancer
> logging.
>
> .. Dan
>
>
>
>
>
>
> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <bb(a)kervyn.de> wrote:
>
>> Hello people,
>>
>> I am still struggeling with the balancer
>> (https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>> Now I've read some more and might think that I do not have enough PGs.
>> Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I
>> have the autoscaler enabled, but I doesn't tell me to increase the
>> PGs.
>>
>> What do you think?
>>
>> --
>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
>> im groüen Saal.
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
1:15 p.m.
I suggest to just disable the autoscaler until your balancing is understood.
What does your active mgr log say (with debug_mgr 4/5), grep balancer
/var/log/ceph/ceph-mgr.*.log
-- Dan
On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens <bb(a)kervyn.de> wrote:
>
> Hi,
> this unfortunally did not solve my problem. I still have some OSDs that fill up to
85%
>
> According to the logging, the autoscaler might want to add more PGs to one Bucken and
reduce almost all other buckets to 32.
> 2021-03-15 12:19:58.825 7f307f601700 4 mgr[pg_autoscaler] Pool
'eu-central-1.rgw.buckets.data' root_id -1 using 0.705080476146 of space, bias
1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
>
> Why the balancing does not happen is still nebulous to me.
>
>
>
> Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>
>> OK
>> Btw, you might need to fail to a new mgr... I'm not sure if the current
active will read that new config.
>>
>> .. dan
>>
>>
>> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <bb(a)kervyn.de> wrote:
>>>
>>> Hi,
>>>
>>> ok thanks. I just changed the value and rewighted everything back to 1. Now I
let it sync the weekend and check how it will be on monday.
>>> We tried to have the systems total storage balanced as possible. New systems
will be with 8TB disks but for the exiting ones we added 16TB to offset the 4TB disks and
we needed a lot of storage fast, because of a DC move. If you have any recommendations I
would be happy to hear them.
>>>
>>> Cheers
>>> Boris
>>>
>>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>>>
>>>> Thanks.
>>>>
>>>> Decreasing the max deviation to 2 or 1 should help in your case. This
option controls when the balancer stops trying to move PGs around -- by default it stops
when the deviation from the mean is 5. Yes this is too large IMO -- all of our clusters
have this set to 1.
>>>>
>>>> And given that you have some OSDs with more than 200 PGs, you definitely
shouldn't increase the num PGs.
>>>>
>>>> But anyway with your mixed device sizes it might be challenging to make a
perfectly uniform distribution. Give it a try with 1 though, and let us know how it goes.
>>>>
>>>> .. Dan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <bb(a)kervyn.de> wrote:
>>>>>
>>>>> Hi Dan,
>>>>>
>>>>> upmap_max_deviation is default (5) in our cluster. Is 1 the
recommended deviation?
>>>>>
>>>>> I added the whole ceph osd df tree, (I need to remove some OSDs and
readd them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I also reweighted to
try to get the %USE mitigated).
>>>>>
>>>>> I will increase the mgr debugging to see what is the problem.
>>>>>
>>>>> [root@s3db1 ~]# ceph osd df tree
>>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
>>>>> -1 673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB
168 TiB 74.57 1.00 - root default
>>>>> -2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB
22 TiB 49.82 0.67 - host s3db1
>>>>> 23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB
13 TiB 12.50 0.17 101 up osd.23
>>>>> 69 hdd 14.55269 0 0 B 0 B 0 B 0 B 0 B
0 B 0 0 11 up osd.69
>>>>> 73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB
4.2 TiB 71.15 0.95 107 up osd.73
>>>>> 79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B
747 GiB 79.94 1.07 52 up osd.79
>>>>> 80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B
1.0 TiB 71.61 0.96 58 up osd.80
>>>>> 81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B
1.5 TiB 60.07 0.81 55 up osd.81
>>>>> 82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B
1.7 TiB 52.68 0.71 30 up osd.82
>>>>> -11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB
13 TiB 74.69 1.00 - host s3db10
>>>>> 63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB
1.7 TiB 76.04 1.02 92 up osd.63
>>>>> 64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB
1.8 TiB 75.54 1.01 101 up osd.64
>>>>> 65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB
2.0 TiB 73.17 0.98 105 up osd.65
>>>>> 66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB
1.4 TiB 80.38 1.08 98 up osd.66
>>>>> 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB
2.2 TiB 70.10 0.94 100 up osd.67
>>>>> 68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB
2.0 TiB 72.88 0.98 107 up osd.68
>>>>> 70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB
1.8 TiB 74.73 1.00 102 up osd.70
>>>>> -12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB
12 TiB 77.24 1.04 - host s3db11
>>>>> 46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB
1.5 TiB 78.91 1.06 97 up osd.46
>>>>> 47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB
2.1 TiB 71.47 0.96 96 up osd.47
>>>>> 48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB
1.2 TiB 83.46 1.12 109 up osd.48
>>>>> 49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB
1.5 TiB 78.96 1.06 98 up osd.49
>>>>> 50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB
1.4 TiB 80.40 1.08 102 up osd.50
>>>>> 51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB
1.3 TiB 81.70 1.10 110 up osd.51
>>>>> 72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB
2.5 TiB 65.82 0.88 89 up osd.72
>>>>> -37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB
12 TiB 79.04 1.06 - host s3db12
>>>>> 19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB
559 GiB 85.18 1.14 55 up osd.19
>>>>> 71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB
825 GiB 78.14 1.05 50 up osd.71
>>>>> 75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB
555 GiB 85.29 1.14 57 up osd.75
>>>>> 76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB
501 GiB 86.73 1.16 50 up osd.76
>>>>> 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB
3.6 TiB 75.57 1.01 202 up osd.77
>>>>> 78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB
3.4 TiB 76.65 1.03 196 up osd.78
>>>>> 83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB
2.9 TiB 80.04 1.07 223 up osd.83
>>>>> -3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB
16 TiB 73.15 0.98 - host s3db2
>>>>> 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB
3.6 TiB 75.52 1.01 194 up osd.1
>>>>> 3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B
1.4 TiB 60.94 0.82 52 up osd.3
>>>>> 4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B
401 GiB 89.23 1.20 53 up osd.4
>>>>> 5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B
1.3 TiB 62.88 0.84 51 up osd.5
>>>>> 6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B
1.7 TiB 54.51 0.73 47 up osd.6
>>>>> 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26 GiB
3.8 TiB 73.90 0.99 185 up osd.7
>>>>> 74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB
3.5 TiB 76.27 1.02 208 up osd.74
>>>>> -4 58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB 86 GiB
15 TiB 74.05 0.99 - host s3db3
>>>>> 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26 GiB
4.0 TiB 72.78 0.98 203 up osd.2
>>>>> 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB
3.6 TiB 75.62 1.01 189 up osd.9
>>>>> 10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28 GiB
3.5 TiB 76.34 1.02 211 up osd.10
>>>>> 12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B
1.2 TiB 66.22 0.89 44 up osd.12
>>>>> 13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B
720 GiB 80.66 1.08 66 up osd.13
>>>>> 14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B
552 GiB 85.18 1.14 60 up osd.14
>>>>> 15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B
1.7 TiB 53.72 0.72 44 up osd.15
>>>>> -5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB
14 TiB 76.37 1.02 - host s3db4
>>>>> 11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28 GiB
2.8 TiB 81.15 1.09 205 up osd.11
>>>>> 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB
3.6 TiB 75.38 1.01 211 up osd.17
>>>>> 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44 GiB
4.0 TiB 72.86 0.98 188 up osd.18
>>>>> 20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B
796 GiB 78.63 1.05 66 up osd.20
>>>>> 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B
1.1 TiB 70.32 0.94 47 up osd.21
>>>>> 22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B
802 GiB 78.47 1.05 58 up osd.22
>>>>> 24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B
856 GiB 77.01 1.03 47 up osd.24
>>>>> -6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB
15 TiB 75.22 1.01 - host s3db5
>>>>> 0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB
1.3 TiB 65.00 0.87 48 up osd.0
>>>>> 25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB
1.4 TiB 63.86 0.86 41 up osd.25
>>>>> 26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB
862 GiB 77.47 1.04 48 up osd.26
>>>>> 27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB
1.5 TiB 61.00 0.82 48 up osd.27
>>>>> 28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30 GiB
2.8 TiB 81.19 1.09 203 up osd.28
>>>>> 29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26 GiB
3.8 TiB 73.95 0.99 200 up osd.29
>>>>> 30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28 GiB
3.1 TiB 79.07 1.06 207 up osd.30
>>>>> -7 58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB 122 GiB
14 TiB 75.41 1.01 - host s3db6
>>>>> 32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB
622 GiB 83.74 1.12 65 up osd.32
>>>>> 33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB
740 GiB 80.67 1.08 52 up osd.33
>>>>> 34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB
877 GiB 77.08 1.03 52 up osd.34
>>>>> 35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB
1.4 TiB 62.18 0.83 42 up osd.35
>>>>> 36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26 GiB
4.0 TiB 72.98 0.98 220 up osd.36
>>>>> 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB
3.6 TiB 75.30 1.01 200 up osd.37
>>>>> 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB
3.3 TiB 77.43 1.04 217 up osd.38
>>>>> -8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB
12 TiB 78.98 1.06 - host s3db7
>>>>> 39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB
499 GiB 86.96 1.17 43 up osd.39
>>>>> 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB
1.2 TiB 68.33 0.92 39 up osd.40
>>>>> 41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB
732 GiB 80.86 1.08 64 up osd.41
>>>>> 42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB
1.2 TiB 67.97 0.91 50 up osd.42
>>>>> 43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28 GiB
2.9 TiB 80.32 1.08 213 up osd.43
>>>>> 44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB
3.1 TiB 78.59 1.05 198 up osd.44
>>>>> 45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB
2.8 TiB 81.05 1.09 214 up osd.45
>>>>> -9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB
10 TiB 79.75 1.07 - host s3db8
>>>>> 8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB
1.5 TiB 79.68 1.07 99 up osd.8
>>>>> 16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB
1.4 TiB 80.25 1.08 97 up osd.16
>>>>> 31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB
1.8 TiB 75.36 1.01 87 up osd.31
>>>>> 52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB
1.6 TiB 78.19 1.05 113 up osd.52
>>>>> 53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB
1.1 TiB 84.46 1.13 109 up osd.53
>>>>> 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB
1.7 TiB 76.73 1.03 115 up osd.54
>>>>> 55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB
1.2 TiB 83.56 1.12 122 up osd.55
>>>>> -10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB
12 TiB 76.05 1.02 - host s3db9
>>>>> 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB
2.1 TiB 71.06 0.95 105 up osd.56
>>>>> 57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB
1.2 TiB 83.17 1.12 102 up osd.57
>>>>> 58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB
1.4 TiB 81.56 1.09 105 up osd.58
>>>>> 59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB
1.4 TiB 80.64 1.08 94 up osd.59
>>>>> 60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB
2.0 TiB 73.25 0.98 101 up osd.60
>>>>> 61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB
2.5 TiB 65.84 0.88 103 up osd.61
>>>>> 62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB
1.7 TiB 76.83 1.03 126 up osd.62
>>>>> TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB
173 TiB 74.57
>>>>> MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
>>>>>
>>>>>
>>>>>
>>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>>>>>
>>>>>> No, increasing num PGs won't help substantially.
>>>>>>
>>>>>> Can you share the entire output of ceph osd df tree ?
>>>>>>
>>>>>> Did you already set
>>>>>>
>>>>>> ceph config set mgr mgr/balancer/upmap_max_deviation 1
>>>>>>
>>>>>>
>>>>>> ??
>>>>>> And I recommend debug_mgr 4/5 so you can see some basic upmap
balancer logging.
>>>>>>
>>>>>> .. Dan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <bb(a)kervyn.de>
wrote:
>>>>>>>
>>>>>>> Hello people,
>>>>>>>
>>>>>>> I am still struggeling with the balancer
>>>>>>>
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>>>>>>> Now I've read some more and might think that I do not
have enough PGs.
>>>>>>> Currently I have 84OSDs and 1024PGs for the main pool (3008
total). I
>>>>>>> have the autoscaler enabled, but I doesn't tell me to
increase the
>>>>>>> PGs.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> --
>>>>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal abweichend
>>>>>>> im groüen Saal.
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>>>
>>>
>>>
>>> --
>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
1:54 p.m.
Hi Dan,
I've set the autoscaler to warn, but it actually does not warn for now. So
not touching it for now.
this is what the log says in minute intervals:
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/active
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/sleep_interval
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/begin_time
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/end_time
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/begin_weekday
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/end_weekday
2021-03-15 13:51:00.971 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/pool_ids
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Optimize plan
auto_2021-03-15_13:51:00
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/mode
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Mode upmap, max
misplaced 0.050000
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] do_upmap
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/upmap_max_iterations
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/upmap_max_deviation
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] pools
['eu-msg-1.rgw.data.root', 'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys',
'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log',
'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index',
'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control',
'.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control',
'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3',
'default.rgw.log', 'eu-central-1.rgw.meta',
'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
2021-03-15 13:51:01.224 7f307d5fd700 4 mgr[balancer] prepared 0/10 changes
Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
I suggest to just disable the autoscaler until your
balancing is
understood.
What does your active mgr log say (with debug_mgr 4/5), grep balancer
/var/log/ceph/ceph-mgr.*.log
-- Dan
On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hi,
this unfortunally did not solve my problem. I still have some OSDs that
fill up to
85%
According to the logging, the autoscaler might want to add more PGs to
one Bucken
and reduce almost all other buckets to 32.
2021-03-15 12:19:58.825 7f307f601700 4
mgr[pg_autoscaler] Pool
'eu-central-1.rgw.buckets.data' root_id -1 using
0.705080476146 of space,
bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
Why the balancing does not happen is still nebulous to me.
Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>
> OK
> Btw, you might need to fail to a new mgr... I'm not sure if the current
active will read that new config.
>
> .. dan
>
>
> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <bb(a)kervyn.de> wrote:
>>
>> Hi,
>>
>> ok thanks. I just changed the value and rewighted everything back to
1.
Now I let it sync the weekend and check how it will be on monday.
>> We tried to have the systems total
storage balanced as possible. New
systems will be with 8TB disks but for the
exiting ones we added 16TB to
offset the 4TB disks and we needed a lot of storage fast, because of a DC
move. If you have any recommendations I would be happy to hear them.
>>
>> Cheers
>> Boris
>>
>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>>>
>>> Thanks.
>>>
>>> Decreasing the max deviation to 2 or 1 should help in your case. This
option controls when the balancer stops trying to move PGs around -- by
default it stops when the deviation from the mean is 5. Yes this is too
large IMO -- all of our clusters have this set to 1.
>>>
>>> And given that you have some OSDs with more than 200 PGs, you
definitely shouldn't increase the num PGs.
>>>
>>> But anyway with your mixed device sizes it might be challenging to
make a perfectly uniform distribution. Give it a try with 1 though, and let
us know how it goes.
>>>
>>> .. Dan
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <bb(a)kervyn.de> wrote:
>>>>
>>>> Hi Dan,
>>>>
>>>> upmap_max_deviation is default (5) in our cluster. Is 1 the
recommended deviation?
>>>>
>>>> I added the whole ceph osd df tree, (I need to remove some OSDs and
readd them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I
also reweighted to try to get the %USE mitigated).
>>>>
>>>> I will increase the mgr debugging to see what is the problem.
>>>>
>>>> [root@s3db1 ~]# ceph osd df tree
>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
>>>> -1 673.54224 - 659
TiB 491 TiB 464 TiB 96 GiB 1.2 TiB
168 TiB 74.57 1.00 - root default
>>>> -2 58.30331 - 44
TiB 22 TiB 17 TiB 5.7 GiB 38
GiB 22 TiB 49.82 0.67 - host s3db1
>>>> 23 hdd 14.65039 1.00000 15
TiB 1.8 TiB 1.7 TiB 156 MiB 4.4
GiB 13 TiB 12.50 0.17 101 up osd.23
>>>> 69 hdd 14.55269 0
0 B 0 B 0 B 0 B 0
B 0 B 0 0 11 up osd.69
>>>> 73 hdd 14.55269 1.00000 15
TiB 10 TiB 10 TiB 6.1 MiB 33 GiB
4.2 TiB 71.15 0.95 107 up osd.73
>>>> 79 hdd 3.63689 1.00000 3.6
TiB 2.9 TiB 747 GiB 2.0 GiB 0 B
747 GiB 79.94 1.07 52 up osd.79
>>>> 80 hdd 3.63689 1.00000 3.6
TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B
1.0 TiB 71.61 0.96 58 up osd.80
>>>> 81 hdd 3.63689 1.00000 3.6
TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B
1.5 TiB 60.07 0.81 55 up osd.81
>>>> 82 hdd 3.63689 1.00000 3.6
TiB 1.9 TiB 1.7 TiB 536 MiB 0 B
1.7 TiB 52.68 0.71 30 up osd.82
>>>> -11 50.94173 - 51
TiB 38 TiB 38 TiB 3.7 GiB 100
GiB 13 TiB 74.69 1.00 - host s3db10
>>>> 63 hdd 7.27739 1.00000 7.3
TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB
1.7 TiB 76.04 1.02 92 up osd.63
>>>> 64 hdd 7.27739 1.00000 7.3
TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB
1.8 TiB 75.54 1.01 101 up osd.64
>>>> 65 hdd 7.27739 1.00000 7.3
TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB
2.0 TiB 73.17 0.98 105 up osd.65
>>>> 66 hdd 7.27739 1.00000 7.3
TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB
1.4 TiB 80.38 1.08 98 up osd.66
>>>> 67 hdd 7.27739 1.00000 7.3
TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB
2.2 TiB 70.10 0.94 100 up osd.67
>>>> 68 hdd 7.27739 1.00000 7.3
TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB
2.0 TiB 72.88 0.98 107 up osd.68
>>>> 70 hdd 7.27739 1.00000 7.3
TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB
1.8 TiB 74.73 1.00 102 up osd.70
>>>> -12 50.99052 - 51
TiB 39 TiB 39 TiB 2.9 GiB 99
GiB 12 TiB 77.24 1.04 - host s3db11
>>>> 46 hdd 7.27739 1.00000 7.3
TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB
1.5 TiB 78.91 1.06 97 up osd.46
>>>> 47 hdd 7.27739 1.00000 7.3
TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB
2.1 TiB 71.47 0.96 96 up osd.47
>>>> 48 hdd 7.27739 1.00000 7.3
TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB
1.2 TiB 83.46 1.12 109 up osd.48
>>>> 49 hdd 7.27739 1.00000 7.3
TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB
1.5 TiB 78.96 1.06 98 up osd.49
>>>> 50 hdd 7.27739 1.00000 7.3
TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB
1.4 TiB 80.40 1.08 102 up osd.50
>>>> 51 hdd 7.27739 1.00000 7.3
TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB
1.3 TiB 81.70 1.10 110 up osd.51
>>>> 72 hdd 7.32619 1.00000 7.3
TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB
2.5 TiB 65.82 0.88 89 up osd.72
>>>> -37 58.55478 - 59
TiB 46 TiB 46 TiB 5.0 GiB 124
GiB 12 TiB 79.04 1.06 - host s3db12
>>>> 19 hdd 3.68750 1.00000 3.7
TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB
559 GiB 85.18 1.14 55 up osd.19
>>>> 71 hdd 3.68750 1.00000 3.7
TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB
825 GiB 78.14 1.05 50 up osd.71
>>>> 75 hdd 3.68750 1.00000 3.7
TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB
555 GiB 85.29 1.14 57 up osd.75
>>>> 76 hdd 3.68750 1.00000 3.7
TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB
501 GiB 86.73 1.16 50 up osd.76
>>>> 77 hdd 14.60159 1.00000 15
TiB 11 TiB 11 TiB 880 MiB 30 GiB
3.6 TiB 75.57 1.01 202 up osd.77
>>>> 78 hdd 14.60159 1.00000 15
TiB 11 TiB 11 TiB 1.0 GiB 30 GiB
3.4 TiB 76.65 1.03 196 up osd.78
>>>> 83 hdd 14.60159 1.00000 15
TiB 12 TiB 12 TiB 1.8 GiB 31 GiB
2.9 TiB 80.04 1.07 223 up osd.83
>>>> -3 58.49872 - 58
TiB 43 TiB 38 TiB 8.1 GiB 91
GiB 16 TiB 73.15 0.98 - host s3db2
>>>> 1 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 3.1 GiB 38 GiB
3.6 TiB 75.52 1.01 194 up osd.1
>>>> 3 hdd 3.63689 1.00000 3.6
TiB 2.2 TiB 1.4 TiB 418 MiB 0 B
1.4 TiB 60.94 0.82 52 up osd.3
>>>> 4 hdd 3.63689 0.89999 3.6
TiB 3.2 TiB 401 GiB 845 MiB 0 B
401 GiB 89.23 1.20 53 up osd.4
>>>> 5 hdd 3.63689 1.00000 3.6
TiB 2.3 TiB 1.3 TiB 437 MiB 0 B
1.3 TiB 62.88 0.84 51 up osd.5
>>>> 6 hdd 3.63689 1.00000 3.6
TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B
1.7 TiB 54.51 0.73 47 up osd.6
>>>> 7 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 493 MiB 26 GiB
3.8 TiB 73.90 0.99 185 up osd.7
>>>> 74 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 1.1 GiB 27 GiB
3.5 TiB 76.27 1.02 208 up osd.74
>>>> -4 58.49872 - 58
TiB 43 TiB 37 TiB 33 GiB 86
GiB 15 TiB 74.05 0.99 - host s3db3
>>>> 2 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 850 MiB 26 GiB
4.0 TiB 72.78 0.98 203 up osd.2
>>>> 9 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 8.3 GiB 33 GiB
3.6 TiB 75.62 1.01 189 up osd.9
>>>> 10 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 663 MiB 28 GiB
3.5 TiB 76.34 1.02 211 up osd.10
>>>> 12 hdd 3.63689 1.00000 3.6
TiB 2.4 TiB 1.2 TiB 633 MiB 0 B
1.2 TiB 66.22 0.89 44 up osd.12
>>>> 13 hdd 3.63689 1.00000 3.6
TiB 2.9 TiB 720 GiB 2.3 GiB 0 B
720 GiB 80.66 1.08 66 up osd.13
>>>> 14 hdd 3.63689 1.00000 3.6
TiB 3.1 TiB 552 GiB 18 GiB 0 B
552 GiB 85.18 1.14 60 up osd.14
>>>> 15 hdd 3.63689 1.00000 3.6
TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B
1.7 TiB 53.72 0.72 44 up osd.15
>>>> -5 58.49872 - 58
TiB 45 TiB 37 TiB 7.2 GiB 99
GiB 14 TiB 76.37 1.02 - host s3db4
>>>> 11 hdd 14.65039 1.00000 15
TiB 12 TiB 12 TiB 897 MiB 28 GiB
2.8 TiB 81.15 1.09 205 up osd.11
>>>> 17 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 1.2 GiB 27 GiB
3.6 TiB 75.38 1.01 211 up osd.17
>>>> 18 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 965 MiB 44 GiB
4.0 TiB 72.86 0.98 188 up osd.18
>>>> 20 hdd 3.63689 1.00000 3.6
TiB 2.9 TiB 796 GiB 529 MiB 0 B
796 GiB 78.63 1.05 66 up osd.20
>>>> 21 hdd 3.63689 1.00000 3.6
TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B
1.1 TiB 70.32 0.94 47 up osd.21
>>>> 22 hdd 3.63689 1.00000 3.6
TiB 2.9 TiB 802 GiB 882 MiB 0 B
802 GiB 78.47 1.05 58 up osd.22
>>>> 24 hdd 3.63689 1.00000 3.6
TiB 2.8 TiB 856 GiB 645 MiB 0 B
856 GiB 77.01 1.03 47 up osd.24
>>>> -6 58.89636 - 59
TiB 44 TiB 44 TiB 2.4 GiB 111
GiB 15 TiB 75.22 1.01 - host s3db5
>>>> 0 hdd 3.73630 1.00000 3.7
TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB
1.3 TiB 65.00 0.87 48 up osd.0
>>>> 25 hdd 3.73630 1.00000 3.7
TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB
1.4 TiB 63.86 0.86 41 up osd.25
>>>> 26 hdd 3.73630 1.00000 3.7
TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB
862 GiB 77.47 1.04 48 up osd.26
>>>> 27 hdd 3.73630 1.00000 3.7
TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB
1.5 TiB 61.00 0.82 48 up osd.27
>>>> 28 hdd 14.65039 1.00000 15
TiB 12 TiB 12 TiB 937 MiB 30 GiB
2.8 TiB 81.19 1.09 203 up osd.28
>>>> 29 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 536 MiB 26 GiB
3.8 TiB 73.95 0.99 200 up osd.29
>>>> 30 hdd 14.65039 1.00000 15
TiB 12 TiB 11 TiB 744 MiB 28 GiB
3.1 TiB 79.07 1.06 207 up osd.30
>>>> -7 58.89636 - 59
TiB 44 TiB 44 TiB 14 GiB 122
GiB 14 TiB 75.41 1.01 - host s3db6
>>>> 32 hdd 3.73630 1.00000 3.7
TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB
622 GiB 83.74 1.12 65 up osd.32
>>>> 33 hdd 3.73630 0.79999 3.7
TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB
740 GiB 80.67 1.08 52 up osd.33
>>>> 34 hdd 3.73630 0.79999 3.7
TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB
877 GiB 77.08 1.03 52 up osd.34
>>>> 35 hdd 3.73630 0.79999 3.7
TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB
1.4 TiB 62.18 0.83 42 up osd.35
>>>> 36 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 544 MiB 26 GiB
4.0 TiB 72.98 0.98 220 up osd.36
>>>> 37 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 11 GiB 38 GiB
3.6 TiB 75.30 1.01 200 up osd.37
>>>> 38 hdd 14.65039 1.00000 15
TiB 11 TiB 11 TiB 1.2 GiB 28 GiB
3.3 TiB 77.43 1.04 217 up osd.38
>>>> -8 58.89636 - 59
TiB 47 TiB 46 TiB 3.9 GiB 116
GiB 12 TiB 78.98 1.06 - host s3db7
>>>> 39 hdd 3.73630 1.00000 3.7
TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB
499 GiB 86.96 1.17 43 up osd.39
>>>> 40 hdd 3.73630 1.00000 3.7
TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB
1.2 TiB 68.33 0.92 39 up osd.40
>>>> 41 hdd 3.73630 1.00000 3.7
TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB
732 GiB 80.86 1.08 64 up osd.41
>>>> 42 hdd 3.73630 1.00000 3.7
TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB
1.2 TiB 67.97 0.91 50 up osd.42
>>>> 43 hdd 14.65039 1.00000 15
TiB 12 TiB 12 TiB 564 MiB 28 GiB
2.9 TiB 80.32 1.08 213 up osd.43
>>>> 44 hdd 14.65039 1.00000 15
TiB 12 TiB 11 TiB 1.3 GiB 28 GiB
3.1 TiB 78.59 1.05 198 up osd.44
>>>> 45 hdd 14.65039 1.00000 15
TiB 12 TiB 12 TiB 1.2 GiB 30 GiB
2.8 TiB 81.05 1.09 214 up osd.45
>>>> -9 51.28331 - 51
TiB 41 TiB 41 TiB 4.9 GiB 108
GiB 10 TiB 79.75 1.07 - host s3db8
>>>> 8 hdd 7.32619 1.00000 7.3
TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB
1.5 TiB 79.68 1.07 99 up osd.8
>>>> 16 hdd 7.32619 1.00000 7.3
TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB
1.4 TiB 80.25 1.08 97 up osd.16
>>>> 31 hdd 7.32619 1.00000 7.3
TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB
1.8 TiB 75.36 1.01 87 up osd.31
>>>> 52 hdd 7.32619 1.00000 7.3
TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB
1.6 TiB 78.19 1.05 113 up osd.52
>>>> 53 hdd 7.32619 1.00000 7.3
TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB
1.1 TiB 84.46 1.13 109 up osd.53
>>>> 54 hdd 7.32619 1.00000 7.3
TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB
1.7 TiB 76.73 1.03 115 up osd.54
>>>> 55 hdd 7.32619 1.00000 7.3
TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB
1.2 TiB 83.56 1.12 122 up osd.55
>>>> -10 51.28331 - 51
TiB 39 TiB 39 TiB 4.7 GiB 100
GiB 12 TiB 76.05 1.02 - host s3db9
>>>> 56 hdd 7.32619 1.00000 7.3
TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB
2.1 TiB 71.06 0.95 105 up osd.56
>>>> 57 hdd 7.32619 1.00000 7.3
TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB
1.2 TiB 83.17 1.12 102 up osd.57
>>>> 58 hdd 7.32619 1.00000 7.3
TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB
1.4 TiB 81.56 1.09 105 up osd.58
>>>> 59 hdd 7.32619 1.00000 7.3
TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB
1.4 TiB 80.64 1.08 94 up osd.59
>>>> 60 hdd 7.32619 1.00000 7.3
TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB
2.0 TiB 73.25 0.98 101 up osd.60
>>>> 61 hdd 7.32619 1.00000 7.3
TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB
2.5 TiB 65.84 0.88 103 up osd.61
>>>> 62 hdd 7.32619 1.00000 7.3
TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB
1.7 TiB 76.83 1.03 126 up osd.62
>>>> TOTAL 674
TiB 501 TiB 473 TiB 96 GiB 1.2 TiB
173 TiB 74.57
>>>> MIN/MAX VAR: 0.17/1.20 STDDEV:
10.25
>>>>
>>>>
>>>>
>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>>>>>
>>>>> No, increasing num PGs won't help substantially.
>>>>>
>>>>> Can you share the entire output of ceph osd df tree ?
>>>>>
>>>>> Did you already set
>>>>>
>>>>> ceph config set mgr mgr/balancer/upmap_max_deviation 1
>>>>>
>>>>>
>>>>> ??
>>>>> And I recommend debug_mgr 4/5 so you can see some basic upmap
balancer logging.
>>>>>
>>>>> .. Dan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <bb(a)kervyn.de>
wrote:
>>>>>>
>>>>>> Hello people,
>>>>>>
>>>>>> I am still struggeling with the balancer
>>>>>> (https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>>>>>> Now I've read some more and might think that I do not have
enough
PGs.
>>>>>> Currently I have 84OSDs
and 1024PGs for the main pool (3008
total). I
>>>>>> have the autoscaler
enabled, but I doesn't tell me to increase the
>>>>>> PGs.
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>> --
>>>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal
abweichend
>>>>>> im groüen Saal.
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>
>>>>
>>>>
>>>> --
>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>>
>>
>>
>> --
>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
2:01 p.m.
OK thanks. Indeed "prepared 0/10 changes" means it thinks things are balanced.
Could you again share the full ceph osd df tree?
On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens <bb(a)kervyn.de> wrote:
>
> Hi Dan,
>
> I've set the autoscaler to warn, but it actually does not warn for now. So not
touching it for now.
>
> this is what the log says in minute intervals:
> 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/active
> 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/sleep_interval
> 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/begin_time
> 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/end_time
> 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/begin_weekday
> 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/end_weekday
> 2021-03-15 13:51:00.971 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/pool_ids
> 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Optimize plan
auto_2021-03-15_13:51:00
> 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/mode
> 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Mode upmap, max misplaced
0.050000
> 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] do_upmap
> 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/upmap_max_iterations
> 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/upmap_max_deviation
> 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] pools
['eu-msg-1.rgw.data.root', 'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys', 'eu-central-1.rgw.buckets.index',
'rbd', 'eu-msg-1.rgw.log', 'whitespace-again-2021-03-10_2',
'eu-msg-1.rgw.buckets.index', 'eu-msg-1.rgw.meta',
'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control', '.rgw.root', 'eu-msg-1.rgw.buckets.data',
'default.rgw.control', 'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3', 'default.rgw.log',
'eu-central-1.rgw.meta', 'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
> 2021-03-15 13:51:01.224 7f307d5fd700 4 mgr[balancer] prepared 0/10 changes
>
> Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>
>> I suggest to just disable the autoscaler until your balancing is understood.
>>
>> What does your active mgr log say (with debug_mgr 4/5), grep balancer
>> /var/log/ceph/ceph-mgr.*.log
>>
>> -- Dan
>>
>> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >
>> > Hi,
>> > this unfortunally did not solve my problem. I still have some OSDs that fill
up to 85%
>> >
>> > According to the logging, the autoscaler might want to add more PGs to one
Bucken and reduce almost all other buckets to 32.
>> > 2021-03-15 12:19:58.825 7f307f601700 4 mgr[pg_autoscaler] Pool
'eu-central-1.rgw.buckets.data' root_id -1 using 0.705080476146 of space, bias
1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
>> >
>> > Why the balancing does not happen is still nebulous to me.
>> >
>> >
>> >
>> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >>
>> >> OK
>> >> Btw, you might need to fail to a new mgr... I'm not sure if the
current active will read that new config.
>> >>
>> >> .. dan
>> >>
>> >>
>> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> ok thanks. I just changed the value and rewighted everything back to
1. Now I let it sync the weekend and check how it will be on monday.
>> >>> We tried to have the systems total storage balanced as possible. New
systems will be with 8TB disks but for the exiting ones we added 16TB to offset the 4TB
disks and we needed a lot of storage fast, because of a DC move. If you have any
recommendations I would be happy to hear them.
>> >>>
>> >>> Cheers
>> >>> Boris
>> >>>
>> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >>>>
>> >>>> Thanks.
>> >>>>
>> >>>> Decreasing the max deviation to 2 or 1 should help in your case.
This option controls when the balancer stops trying to move PGs around -- by default it
stops when the deviation from the mean is 5. Yes this is too large IMO -- all of our
clusters have this set to 1.
>> >>>>
>> >>>> And given that you have some OSDs with more than 200 PGs, you
definitely shouldn't increase the num PGs.
>> >>>>
>> >>>> But anyway with your mixed device sizes it might be challenging
to make a perfectly uniform distribution. Give it a try with 1 though, and let us know how
it goes.
>> >>>>
>> >>>> .. Dan
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <bb(a)kervyn.de>
wrote:
>> >>>>>
>> >>>>> Hi Dan,
>> >>>>>
>> >>>>> upmap_max_deviation is default (5) in our cluster. Is 1 the
recommended deviation?
>> >>>>>
>> >>>>> I added the whole ceph osd df tree, (I need to remove some
OSDs and readd them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I also
reweighted to try to get the %USE mitigated).
>> >>>>>
>> >>>>> I will increase the mgr debugging to see what is the
problem.
>> >>>>>
>> >>>>> [root@s3db1 ~]# ceph osd df tree
>> >>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
>> >>>>> -1 673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB
1.2 TiB 168 TiB 74.57 1.00 - root default
>> >>>>> -2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB
38 GiB 22 TiB 49.82 0.67 - host s3db1
>> >>>>> 23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB
4.4 GiB 13 TiB 12.50 0.17 101 up osd.23
>> >>>>> 69 hdd 14.55269 0 0 B 0 B 0 B 0 B
0 B 0 B 0 0 11 up osd.69
>> >>>>> 73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB
33 GiB 4.2 TiB 71.15 0.95 107 up osd.73
>> >>>>> 79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB
0 B 747 GiB 79.94 1.07 52 up osd.79
>> >>>>> 80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB
0 B 1.0 TiB 71.61 0.96 58 up osd.80
>> >>>>> 81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB
0 B 1.5 TiB 60.07 0.81 55 up osd.81
>> >>>>> 82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB
0 B 1.7 TiB 52.68 0.71 30 up osd.82
>> >>>>> -11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB
100 GiB 13 TiB 74.69 1.00 - host s3db10
>> >>>>> 63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB
14 GiB 1.7 TiB 76.04 1.02 92 up osd.63
>> >>>>> 64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB
15 GiB 1.8 TiB 75.54 1.01 101 up osd.64
>> >>>>> 65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB
14 GiB 2.0 TiB 73.17 0.98 105 up osd.65
>> >>>>> 66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB
15 GiB 1.4 TiB 80.38 1.08 98 up osd.66
>> >>>>> 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB
14 GiB 2.2 TiB 70.10 0.94 100 up osd.67
>> >>>>> 68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB
13 GiB 2.0 TiB 72.88 0.98 107 up osd.68
>> >>>>> 70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB
14 GiB 1.8 TiB 74.73 1.00 102 up osd.70
>> >>>>> -12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB
99 GiB 12 TiB 77.24 1.04 - host s3db11
>> >>>>> 46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB
15 GiB 1.5 TiB 78.91 1.06 97 up osd.46
>> >>>>> 47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB
13 GiB 2.1 TiB 71.47 0.96 96 up osd.47
>> >>>>> 48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB
15 GiB 1.2 TiB 83.46 1.12 109 up osd.48
>> >>>>> 49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB
15 GiB 1.5 TiB 78.96 1.06 98 up osd.49
>> >>>>> 50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB
15 GiB 1.4 TiB 80.40 1.08 102 up osd.50
>> >>>>> 51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB
15 GiB 1.3 TiB 81.70 1.10 110 up osd.51
>> >>>>> 72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB
12 GiB 2.5 TiB 65.82 0.88 89 up osd.72
>> >>>>> -37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB
124 GiB 12 TiB 79.04 1.06 - host s3db12
>> >>>>> 19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB
8.2 GiB 559 GiB 85.18 1.14 55 up osd.19
>> >>>>> 71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB
7.8 GiB 825 GiB 78.14 1.05 50 up osd.71
>> >>>>> 75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB
8.3 GiB 555 GiB 85.29 1.14 57 up osd.75
>> >>>>> 76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB
9.3 GiB 501 GiB 86.73 1.16 50 up osd.76
>> >>>>> 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB
30 GiB 3.6 TiB 75.57 1.01 202 up osd.77
>> >>>>> 78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB
30 GiB 3.4 TiB 76.65 1.03 196 up osd.78
>> >>>>> 83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB
31 GiB 2.9 TiB 80.04 1.07 223 up osd.83
>> >>>>> -3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB
91 GiB 16 TiB 73.15 0.98 - host s3db2
>> >>>>> 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB
38 GiB 3.6 TiB 75.52 1.01 194 up osd.1
>> >>>>> 3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB
0 B 1.4 TiB 60.94 0.82 52 up osd.3
>> >>>>> 4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB
0 B 401 GiB 89.23 1.20 53 up osd.4
>> >>>>> 5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB
0 B 1.3 TiB 62.88 0.84 51 up osd.5
>> >>>>> 6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB
0 B 1.7 TiB 54.51 0.73 47 up osd.6
>> >>>>> 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB
26 GiB 3.8 TiB 73.90 0.99 185 up osd.7
>> >>>>> 74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB
27 GiB 3.5 TiB 76.27 1.02 208 up osd.74
>> >>>>> -4 58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB
86 GiB 15 TiB 74.05 0.99 - host s3db3
>> >>>>> 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB
26 GiB 4.0 TiB 72.78 0.98 203 up osd.2
>> >>>>> 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB
33 GiB 3.6 TiB 75.62 1.01 189 up osd.9
>> >>>>> 10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB
28 GiB 3.5 TiB 76.34 1.02 211 up osd.10
>> >>>>> 12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB
0 B 1.2 TiB 66.22 0.89 44 up osd.12
>> >>>>> 13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB
0 B 720 GiB 80.66 1.08 66 up osd.13
>> >>>>> 14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB
0 B 552 GiB 85.18 1.14 60 up osd.14
>> >>>>> 15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB
0 B 1.7 TiB 53.72 0.72 44 up osd.15
>> >>>>> -5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB
99 GiB 14 TiB 76.37 1.02 - host s3db4
>> >>>>> 11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB
28 GiB 2.8 TiB 81.15 1.09 205 up osd.11
>> >>>>> 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB
27 GiB 3.6 TiB 75.38 1.01 211 up osd.17
>> >>>>> 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB
44 GiB 4.0 TiB 72.86 0.98 188 up osd.18
>> >>>>> 20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB
0 B 796 GiB 78.63 1.05 66 up osd.20
>> >>>>> 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB
0 B 1.1 TiB 70.32 0.94 47 up osd.21
>> >>>>> 22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB
0 B 802 GiB 78.47 1.05 58 up osd.22
>> >>>>> 24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB
0 B 856 GiB 77.01 1.03 47 up osd.24
>> >>>>> -6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB
111 GiB 15 TiB 75.22 1.01 - host s3db5
>> >>>>> 0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB
6.6 GiB 1.3 TiB 65.00 0.87 48 up osd.0
>> >>>>> 25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB
6.6 GiB 1.4 TiB 63.86 0.86 41 up osd.25
>> >>>>> 26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB
7.6 GiB 862 GiB 77.47 1.04 48 up osd.26
>> >>>>> 27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB
6.1 GiB 1.5 TiB 61.00 0.82 48 up osd.27
>> >>>>> 28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB
30 GiB 2.8 TiB 81.19 1.09 203 up osd.28
>> >>>>> 29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB
26 GiB 3.8 TiB 73.95 0.99 200 up osd.29
>> >>>>> 30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB
28 GiB 3.1 TiB 79.07 1.06 207 up osd.30
>> >>>>> -7 58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB
122 GiB 14 TiB 75.41 1.01 - host s3db6
>> >>>>> 32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB
8.2 GiB 622 GiB 83.74 1.12 65 up osd.32
>> >>>>> 33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB
8.1 GiB 740 GiB 80.67 1.08 52 up osd.33
>> >>>>> 34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB
7.7 GiB 877 GiB 77.08 1.03 52 up osd.34
>> >>>>> 35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB
7.0 GiB 1.4 TiB 62.18 0.83 42 up osd.35
>> >>>>> 36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB
26 GiB 4.0 TiB 72.98 0.98 220 up osd.36
>> >>>>> 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB
38 GiB 3.6 TiB 75.30 1.01 200 up osd.37
>> >>>>> 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB
28 GiB 3.3 TiB 77.43 1.04 217 up osd.38
>> >>>>> -8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB
116 GiB 12 TiB 78.98 1.06 - host s3db7
>> >>>>> 39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB
8.5 GiB 499 GiB 86.96 1.17 43 up osd.39
>> >>>>> 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB
7.0 GiB 1.2 TiB 68.33 0.92 39 up osd.40
>> >>>>> 41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB
7.9 GiB 732 GiB 80.86 1.08 64 up osd.41
>> >>>>> 42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB
7.0 GiB 1.2 TiB 67.97 0.91 50 up osd.42
>> >>>>> 43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB
28 GiB 2.9 TiB 80.32 1.08 213 up osd.43
>> >>>>> 44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB
28 GiB 3.1 TiB 78.59 1.05 198 up osd.44
>> >>>>> 45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB
30 GiB 2.8 TiB 81.05 1.09 214 up osd.45
>> >>>>> -9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB
108 GiB 10 TiB 79.75 1.07 - host s3db8
>> >>>>> 8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB
15 GiB 1.5 TiB 79.68 1.07 99 up osd.8
>> >>>>> 16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB
15 GiB 1.4 TiB 80.25 1.08 97 up osd.16
>> >>>>> 31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB
14 GiB 1.8 TiB 75.36 1.01 87 up osd.31
>> >>>>> 52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB
15 GiB 1.6 TiB 78.19 1.05 113 up osd.52
>> >>>>> 53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB
18 GiB 1.1 TiB 84.46 1.13 109 up osd.53
>> >>>>> 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB
15 GiB 1.7 TiB 76.73 1.03 115 up osd.54
>> >>>>> 55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB
16 GiB 1.2 TiB 83.56 1.12 122 up osd.55
>> >>>>> -10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB
100 GiB 12 TiB 76.05 1.02 - host s3db9
>> >>>>> 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB
13 GiB 2.1 TiB 71.06 0.95 105 up osd.56
>> >>>>> 57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB
16 GiB 1.2 TiB 83.17 1.12 102 up osd.57
>> >>>>> 58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB
15 GiB 1.4 TiB 81.56 1.09 105 up osd.58
>> >>>>> 59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB
15 GiB 1.4 TiB 80.64 1.08 94 up osd.59
>> >>>>> 60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB
14 GiB 2.0 TiB 73.25 0.98 101 up osd.60
>> >>>>> 61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB
12 GiB 2.5 TiB 65.84 0.88 103 up osd.61
>> >>>>> 62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB
15 GiB 1.7 TiB 76.83 1.03 126 up osd.62
>> >>>>> TOTAL 674 TiB 501 TiB 473 TiB 96 GiB
1.2 TiB 173 TiB 74.57
>> >>>>> MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >>>>>>
>> >>>>>> No, increasing num PGs won't help substantially.
>> >>>>>>
>> >>>>>> Can you share the entire output of ceph osd df tree ?
>> >>>>>>
>> >>>>>> Did you already set
>> >>>>>>
>> >>>>>> ceph config set mgr mgr/balancer/upmap_max_deviation
1
>> >>>>>>
>> >>>>>>
>> >>>>>> ??
>> >>>>>> And I recommend debug_mgr 4/5 so you can see some basic
upmap balancer logging.
>> >>>>>>
>> >>>>>> .. Dan
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>> >>>>>>>
>> >>>>>>> Hello people,
>> >>>>>>>
>> >>>>>>> I am still struggeling with the balancer
>> >>>>>>>
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>> >>>>>>> Now I've read some more and might think that I
do not have enough PGs.
>> >>>>>>> Currently I have 84OSDs and 1024PGs for the main
pool (3008 total). I
>> >>>>>>> have the autoscaler enabled, but I doesn't tell
me to increase the
>> >>>>>>> PGs.
>> >>>>>>>
>> >>>>>>> What do you think?
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Die Selbsthilfegruppe "UTF-8-Probleme"
trifft sich diesmal abweichend
>> >>>>>>> im groüen Saal.
>> >>>>>>> _______________________________________________
>> >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>> >>>>>>> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal abweichend im groüen Saal.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>> >
>> >
>> >
>> > --
>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
2:04 p.m.
Absolutly:
[root@s3db1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 673.54224 - 674 TiB 496 TiB 468 TiB 97 GiB 1.2 TiB 177
TiB 73.67 1.00 - root default
-2 58.30331 - 58 TiB 42 TiB 38 TiB 9.2 GiB 99 GiB 16
TiB 72.88 0.99 - host s3db1
23 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 714 MiB 25 GiB 3.7
TiB 74.87 1.02 194 up osd.23
69 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.6 GiB 40 GiB 3.4
TiB 76.32 1.04 199 up osd.69
73 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.3 GiB 34 GiB 3.8
TiB 74.15 1.01 203 up osd.73
79 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.8 GiB 0 B 1.3
TiB 65.44 0.89 47 up osd.79
80 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 2.2 GiB 0 B 1.3
TiB 65.34 0.89 48 up osd.80
81 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.1 GiB 0 B 1.3
TiB 65.38 0.89 47 up osd.81
82 hdd 3.63689 1.00000 3.6 TiB 2.5 TiB 1.1 TiB 619 MiB 0 B 1.1
TiB 68.46 0.93 41 up osd.82
-11 50.94173 - 51 TiB 37 TiB 37 TiB 3.5 GiB 98 GiB 14
TiB 71.90 0.98 - host s3db10
63 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 647 MiB 14 GiB 2.0
TiB 72.72 0.99 94 up osd.63
64 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.2 TiB 668 MiB 14 GiB 2.0
TiB 72.23 0.98 93 up osd.64
65 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 227 MiB 14 GiB 2.1
TiB 71.16 0.97 100 up osd.65
66 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 313 MiB 14 GiB 1.9
TiB 74.25 1.01 92 up osd.66
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 584 MiB 14 GiB 2.1
TiB 70.63 0.96 96 up osd.67
68 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 720 MiB 14 GiB 2.1
TiB 71.72 0.97 101 up osd.68
70 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 425 MiB 14 GiB 2.1
TiB 70.59 0.96 97 up osd.70
-12 50.99052 - 51 TiB 38 TiB 37 TiB 2.1 GiB 97 GiB 13
TiB 73.77 1.00 - host s3db11
46 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 229 MiB 14 GiB 1.7
TiB 77.05 1.05 97 up osd.46
47 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 159 MiB 13 GiB 2.2
TiB 70.00 0.95 89 up osd.47
48 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 279 MiB 14 GiB 2.1
TiB 71.82 0.97 98 up osd.48
49 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 276 MiB 14 GiB 1.8
TiB 74.90 1.02 95 up osd.49
50 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 336 MiB 14 GiB 2.0
TiB 72.13 0.98 93 up osd.50
51 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 728 MiB 15 GiB 1.6
TiB 77.76 1.06 98 up osd.51
72 hdd 7.32619 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 147 MiB 13 GiB 2.0
TiB 72.75 0.99 95 up osd.72
-37 58.55478 - 59 TiB 44 TiB 44 TiB 4.4 GiB 122 GiB 15
TiB 75.20 1.02 - host s3db12
19 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 454 MiB 8.2 GiB 780
GiB 79.35 1.08 53 up osd.19
71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 7.1 MiB 8.0 GiB 734
GiB 80.56 1.09 47 up osd.71
75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 439 MiB 8.2 GiB 777
GiB 79.43 1.08 48 up osd.75
76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 241 MiB 8.9 GiB 688
GiB 81.77 1.11 50 up osd.76
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB 3.6
TiB 75.44 1.02 201 up osd.77
78 hdd 14.60159 1.00000 15 TiB 10 TiB 10 TiB 1015 MiB 28 GiB 4.2
TiB 71.26 0.97 193 up osd.78
83 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.4 GiB 30 GiB 3.8
TiB 73.76 1.00 203 up osd.83
-3 58.49872 - 58 TiB 42 TiB 36 TiB 8.2 GiB 89 GiB 17
TiB 71.71 0.97 - host s3db2
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.2 GiB 37 GiB 3.7
TiB 74.58 1.01 196 up osd.1
3 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 566 MiB 0 B 1.3
TiB 64.11 0.87 50 up osd.3
4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 771 GiB 695 MiB 0 B 771
GiB 79.30 1.08 48 up osd.4
5 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 482 MiB 0 B 1.2
TiB 66.51 0.90 49 up osd.5
6 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 1.8 GiB 0 B 1.3
TiB 64.00 0.87 42 up osd.6
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 639 MiB 26 GiB 4.0
TiB 72.44 0.98 192 up osd.7
74 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 907 MiB 26 GiB 4.2
TiB 71.32 0.97 193 up osd.74
-4 58.49872 - 58 TiB 43 TiB 36 TiB 34 GiB 85 GiB 16
TiB 72.69 0.99 - host s3db3
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 980 MiB 26 GiB 3.8
TiB 74.36 1.01 203 up osd.2
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.4 GiB 33 GiB 3.9
TiB 73.51 1.00 186 up osd.9
10 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 650 MiB 26 GiB 4.2
TiB 71.64 0.97 201 up osd.10
12 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 754 MiB 0 B 1.3
TiB 64.17 0.87 44 up osd.12
13 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 813 GiB 2.4 GiB 0 B 813
GiB 78.17 1.06 58 up osd.13
14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 797 GiB 19 GiB 0 B 797
GiB 78.60 1.07 56 up osd.14
15 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 2.2 GiB 0 B 1.3
TiB 63.96 0.87 41 up osd.15
-5 58.49872 - 58 TiB 43 TiB 36 TiB 6.7 GiB 97 GiB 15
TiB 74.04 1.01 - host s3db4
11 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 940 MiB 26 GiB 4.0
TiB 72.49 0.98 196 up osd.11
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1022 MiB 26 GiB 3.6
TiB 75.23 1.02 204 up osd.17
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 945 MiB 45 GiB 3.8
TiB 74.16 1.01 193 up osd.18
20 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1020 GiB 596 MiB 0 B 1020
GiB 72.62 0.99 57 up osd.20
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 1.9 GiB 0 B 1023
GiB 72.54 0.98 41 up osd.21
22 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 797 MiB 0 B 1023
GiB 72.54 0.98 53 up osd.22
24 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 766 GiB 618 MiB 0 B 766
GiB 79.42 1.08 46 up osd.24
-6 58.89636 - 59 TiB 43 TiB 43 TiB 3.0 GiB 108 GiB 16
TiB 73.40 1.00 - host s3db5
0 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 92 MiB 7.2 GiB 1.1
TiB 71.16 0.97 45 up osd.0
25 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 2.4 MiB 7.3 GiB 1.1
TiB 71.23 0.97 41 up osd.25
26 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 181 MiB 7.6 GiB 935
GiB 75.57 1.03 45 up osd.26
27 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 5.1 MiB 7.0 GiB 1.1
TiB 71.20 0.97 47 up osd.27
28 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 977 MiB 26 GiB 3.8
TiB 73.85 1.00 197 up osd.28
29 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 872 MiB 26 GiB 4.1
TiB 71.98 0.98 196 up osd.29
30 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 943 MiB 27 GiB 3.6
TiB 75.51 1.03 202 up osd.30
-7 58.89636 - 59 TiB 44 TiB 43 TiB 13 GiB 122 GiB 15
TiB 74.97 1.02 - host s3db6
32 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 27 MiB 7.6 GiB 940
GiB 75.42 1.02 55 up osd.32
33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 376 MiB 8.2 GiB 691
GiB 81.94 1.11 55 up osd.33
34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 450 MiB 8.5 GiB 620
GiB 83.79 1.14 54 up osd.34
35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 316 MiB 8.4 GiB 690
GiB 81.98 1.11 50 up osd.35
36 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 489 MiB 25 GiB 4.1
TiB 71.69 0.97 208 up osd.36
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB 4.0
TiB 72.41 0.98 195 up osd.37
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 26 GiB 3.7
TiB 74.88 1.02 204 up osd.38
-8 58.89636 - 59 TiB 44 TiB 43 TiB 3.8 GiB 111 GiB 15
TiB 74.16 1.01 - host s3db7
39 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 19 MiB 7.5 GiB 936
GiB 75.54 1.03 39 up osd.39
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.1 GiB 1.1
TiB 69.87 0.95 39 up osd.40
41 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.7 TiB 219 MiB 7.6 GiB 1011
GiB 73.57 1.00 55 up osd.41
42 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 593 MiB 7.1 GiB 1.1
TiB 70.02 0.95 47 up osd.42
43 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 500 MiB 27 GiB 3.7
TiB 74.67 1.01 204 up osd.43
44 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 3.7
TiB 74.62 1.01 193 up osd.44
45 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 29 GiB 3.6
TiB 75.16 1.02 204 up osd.45
-9 51.28331 - 51 TiB 39 TiB 39 TiB 4.9 GiB 107 GiB 12
TiB 76.50 1.04 - host s3db8
8 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 474 MiB 14 GiB 1.7
TiB 76.37 1.04 98 up osd.8
16 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 783 MiB 15 GiB 1.6
TiB 78.39 1.06 100 up osd.16
31 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 441 MiB 14 GiB 1.6
TiB 77.70 1.05 91 up osd.31
52 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 939 MiB 14 GiB 1.7
TiB 76.29 1.04 102 up osd.52
53 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 848 MiB 18 GiB 1.9
TiB 74.30 1.01 98 up osd.53
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 16 GiB 1.7
TiB 76.99 1.05 106 up osd.54
55 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 460 MiB 15 GiB 1.8
TiB 75.46 1.02 105 up osd.55
-10 51.28331 - 51 TiB 37 TiB 37 TiB 3.8 GiB 96 GiB 14
TiB 72.77 0.99 - host s3db9
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 846 MiB 13 GiB 2.1
TiB 71.16 0.97 104 up osd.56
57 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 513 MiB 15 GiB 1.7
TiB 76.53 1.04 96 up osd.57
58 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 604 MiB 13 GiB 2.1
TiB 71.23 0.97 98 up osd.58
59 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 414 MiB 13 GiB 2.2
TiB 70.03 0.95 88 up osd.59
60 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 227 MiB 14 GiB 1.8
TiB 75.54 1.03 97 up osd.60
61 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 456 MiB 13 GiB 2.2
TiB 70.01 0.95 95 up osd.61
62 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 843 MiB 14 GiB 1.8
TiB 74.93 1.02 110 up osd.62
TOTAL 674 TiB 496 TiB 468 TiB 97 GiB 1.2 TiB 177
TiB 73.67
MIN/MAX VAR: 0.87/1.14 STDDEV: 4.22
Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
OK thanks. Indeed "prepared 0/10 changes"
means it thinks things are
balanced.
Could you again share the full ceph osd df tree?
On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hi Dan,
I've set the autoscaler to warn, but it actually does not warn for now.
So not
touching it for now.
this is what the log says in minute intervals:
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/active
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/sleep_interval
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/begin_time
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/end_time
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/begin_weekday
2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/end_weekday
2021-03-15 13:51:00.971 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/pool_ids
2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] Optimize plan
auto_2021-03-15_13:51:00
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/mode
2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] Mode upmap, max
misplaced 0.050000
> 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] do_upmap
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/upmap_max_iterations
2021-03-15 13:51:01.203 7f307d5fd700 4 mgr
get_config get_config key:
mgr/balancer/upmap_max_deviation
2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] pools
['eu-msg-1.rgw.data.root',
'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys',
'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log',
'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index',
'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control',
'.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control',
'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3',
'default.rgw.log', 'eu-central-1.rgw.meta',
'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
2021-03-15 13:51:01.224 7f307d5fd700 4
mgr[balancer] prepared 0/10
changes
Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>
> I suggest to just disable the autoscaler until your balancing is
understood.
>
> What does your active mgr log say (with debug_mgr 4/5), grep balancer
> /var/log/ceph/ceph-mgr.*.log
>
> -- Dan
>
> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens <bb(a)kervyn.de> wrote:
> >
> > Hi,
> > this unfortunally did not solve my problem. I still have some OSDs
that
fill up to 85%
> >
> > According to the logging, the autoscaler might want to add more PGs
to
one Bucken and reduce almost all other buckets to 32.
> > 2021-03-15 12:19:58.825 7f307f601700 4
mgr[pg_autoscaler] Pool
'eu-central-1.rgw.buckets.data' root_id -1 using
0.705080476146 of space,
bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
> >
> > Why the balancing does not happen is still nebulous to me.
> >
> >
> >
> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >>
> >> OK
> >> Btw, you might need to fail to a new mgr... I'm not sure if the
current active will read that new config.
> >>
> >> .. dan
> >>
> >>
> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <bb(a)kervyn.de> wrote:
> >>>
> >>> Hi,
> >>>
> >>> ok thanks. I just changed the value and rewighted everything back
to 1. Now I let it sync the weekend and check how it will be on monday.
> >>> We tried to have the systems
total storage balanced as possible.
New systems will be with 8TB disks but for the
exiting ones we added 16TB
to offset the 4TB disks and we needed a lot of storage fast, because of a
DC move. If you have any recommendations I would be happy to hear them.
> >>>
> >>> Cheers
> >>> Boris
> >>>
> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Decreasing the max deviation to 2 or 1 should help in your case.
This option controls when the balancer stops trying to move PGs around --
by default it stops when the deviation from the mean is 5. Yes this is too
large IMO -- all of our clusters have this set to 1.
> >>>>
> >>>> And given that you have some OSDs with more than 200 PGs, you
definitely shouldn't increase the num PGs.
> >>>>
> >>>> But anyway with your mixed device sizes it might be challenging to
make a perfectly uniform distribution. Give it a try with 1 though, and let
us know how it goes.
> >>>>
> >>>> .. Dan
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <bb(a)kervyn.de>
wrote:
> >>>>>
> >>>>> Hi Dan,
> >>>>>
> >>>>> upmap_max_deviation is default (5) in our cluster. Is 1 the
recommended deviation?
> >>>>>
> >>>>> I added the whole ceph osd df tree, (I need to remove some OSDs
and readd them as bluestore with SSD, so 69, 73 and 82 are a bit off now. I
also reweighted to try to get the %USE mitigated).
> >>>>>
> >>>>> I will increase the mgr debugging to see what is the problem.
> >>>>>
> >>>>> [root@s3db1 ~]# ceph osd df tree
> >>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
> >>>>> -1 673.54224
- 659 TiB 491 TiB 464 TiB 96 GiB 1.2
TiB 168 TiB 74.57 1.00 - root
default
> >>>>> -2 58.30331
- 44 TiB 22 TiB 17 TiB 5.7 GiB 38
GiB 22 TiB 49.82 0.67 - host
s3db1
> >>>>> 23 hdd 14.65039
1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4
GiB 13 TiB 12.50 0.17 101 up
osd.23
> >>>>> 69 hdd 14.55269
0 0 B 0 B 0 B 0 B
0 B 0 B 0 0 11 up
osd.69
> >>>>> 73 hdd 14.55269
1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33
GiB 4.2 TiB 71.15 0.95 107 up
osd.73
> >>>>> 79 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB
0 B 747 GiB 79.94 1.07 52 up
osd.79
> >>>>> 80 hdd 3.63689
1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB
0 B 1.0 TiB 71.61 0.96 58 up
osd.80
> >>>>> 81 hdd 3.63689
1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB
0 B 1.5 TiB 60.07 0.81 55 up
osd.81
> >>>>> 82 hdd 3.63689
1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB
0 B 1.7 TiB 52.68 0.71 30 up
osd.82
> >>>>> -11 50.94173
- 51 TiB 38 TiB 38 TiB 3.7 GiB 100
GiB 13 TiB 74.69 1.00 - host
s3db10
> >>>>> 63 hdd 7.27739
1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14
GiB 1.7 TiB 76.04 1.02 92 up
osd.63
> >>>>> 64 hdd 7.27739
1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15
GiB 1.8 TiB 75.54 1.01 101 up
osd.64
> >>>>> 65 hdd 7.27739
1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14
GiB 2.0 TiB 73.17 0.98 105 up
osd.65
> >>>>> 66 hdd 7.27739
1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15
GiB 1.4 TiB 80.38 1.08 98 up
osd.66
> >>>>> 67 hdd 7.27739
1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14
GiB 2.2 TiB 70.10 0.94 100 up
osd.67
> >>>>> 68 hdd 7.27739
1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13
GiB 2.0 TiB 72.88 0.98 107 up
osd.68
> >>>>> 70 hdd 7.27739
1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14
GiB 1.8 TiB 74.73 1.00 102 up
osd.70
> >>>>> -12 50.99052
- 51 TiB 39 TiB 39 TiB 2.9 GiB 99
GiB 12 TiB 77.24 1.04 - host
s3db11
> >>>>> 46 hdd 7.27739
1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15
GiB 1.5 TiB 78.91 1.06 97 up
osd.46
> >>>>> 47 hdd 7.27739
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13
GiB 2.1 TiB 71.47 0.96 96 up
osd.47
> >>>>> 48 hdd 7.27739
1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15
GiB 1.2 TiB 83.46 1.12 109 up
osd.48
> >>>>> 49 hdd 7.27739
1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15
GiB 1.5 TiB 78.96 1.06 98 up
osd.49
> >>>>> 50 hdd 7.27739
1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15
GiB 1.4 TiB 80.40 1.08 102 up
osd.50
> >>>>> 51 hdd 7.27739
1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15
GiB 1.3 TiB 81.70 1.10 110 up
osd.51
> >>>>> 72 hdd 7.32619
1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12
GiB 2.5 TiB 65.82 0.88 89 up
osd.72
> >>>>> -37 58.55478
- 59 TiB 46 TiB 46 TiB 5.0 GiB 124
GiB 12 TiB 79.04 1.06 - host
s3db12
> >>>>> 19 hdd 3.68750
1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2
GiB 559 GiB 85.18 1.14 55 up
osd.19
> >>>>> 71 hdd 3.68750
1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8
GiB 825 GiB 78.14 1.05 50 up
osd.71
> >>>>> 75 hdd 3.68750
1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3
GiB 555 GiB 85.29 1.14 57 up
osd.75
> >>>>> 76 hdd 3.68750
1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3
GiB 501 GiB 86.73 1.16 50 up
osd.76
> >>>>> 77 hdd 14.60159
1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30
GiB 3.6 TiB 75.57 1.01 202 up
osd.77
> >>>>> 78 hdd 14.60159
1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30
GiB 3.4 TiB 76.65 1.03 196 up
osd.78
> >>>>> 83 hdd 14.60159
1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31
GiB 2.9 TiB 80.04 1.07 223 up
osd.83
> >>>>> -3 58.49872
- 58 TiB 43 TiB 38 TiB 8.1 GiB 91
GiB 16 TiB 73.15 0.98 - host
s3db2
> >>>>> 1 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38
GiB 3.6 TiB 75.52 1.01 194 up
osd.1
> >>>>> 3 hdd 3.63689
1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB
0 B 1.4 TiB 60.94 0.82 52 up
osd.3
> >>>>> 4 hdd 3.63689
0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB
0 B 401 GiB 89.23 1.20 53 up
osd.4
> >>>>> 5 hdd 3.63689
1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB
0 B 1.3 TiB 62.88 0.84 51 up
osd.5
> >>>>> 6 hdd 3.63689
1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB
0 B 1.7 TiB 54.51 0.73 47 up
osd.6
> >>>>> 7 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26
GiB 3.8 TiB 73.90 0.99 185 up
osd.7
> >>>>> 74 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27
GiB 3.5 TiB 76.27 1.02 208 up
osd.74
> >>>>> -4 58.49872
- 58 TiB 43 TiB 37 TiB 33 GiB 86
GiB 15 TiB 74.05 0.99 - host
s3db3
> >>>>> 2 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26
GiB 4.0 TiB 72.78 0.98 203 up
osd.2
> >>>>> 9 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33
GiB 3.6 TiB 75.62 1.01 189 up
osd.9
> >>>>> 10 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28
GiB 3.5 TiB 76.34 1.02 211 up
osd.10
> >>>>> 12 hdd 3.63689
1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB
0 B 1.2 TiB 66.22 0.89 44 up
osd.12
> >>>>> 13 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB
0 B 720 GiB 80.66 1.08 66 up
osd.13
> >>>>> 14 hdd 3.63689
1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB
0 B 552 GiB 85.18 1.14 60 up
osd.14
> >>>>> 15 hdd 3.63689
1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB
0 B 1.7 TiB 53.72 0.72 44 up
osd.15
> >>>>> -5 58.49872
- 58 TiB 45 TiB 37 TiB 7.2 GiB 99
GiB 14 TiB 76.37 1.02 - host
s3db4
> >>>>> 11 hdd 14.65039
1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28
GiB 2.8 TiB 81.15 1.09 205 up
osd.11
> >>>>> 17 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27
GiB 3.6 TiB 75.38 1.01 211 up
osd.17
> >>>>> 18 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44
GiB 4.0 TiB 72.86 0.98 188 up
osd.18
> >>>>> 20 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB
0 B 796 GiB 78.63 1.05 66 up
osd.20
> >>>>> 21 hdd 3.63689
1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB
0 B 1.1 TiB 70.32 0.94 47 up
osd.21
> >>>>> 22 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB
0 B 802 GiB 78.47 1.05 58 up
osd.22
> >>>>> 24 hdd 3.63689
1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB
0 B 856 GiB 77.01 1.03 47 up
osd.24
> >>>>> -6 58.89636
- 59 TiB 44 TiB 44 TiB 2.4 GiB 111
GiB 15 TiB 75.22 1.01 - host
s3db5
> >>>>> 0 hdd 3.73630
1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6
GiB 1.3 TiB 65.00 0.87 48 up
osd.0
> >>>>> 25 hdd 3.73630
1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6
GiB 1.4 TiB 63.86 0.86 41 up
osd.25
> >>>>> 26 hdd 3.73630
1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6
GiB 862 GiB 77.47 1.04 48 up
osd.26
> >>>>> 27 hdd 3.73630
1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1
GiB 1.5 TiB 61.00 0.82 48 up
osd.27
> >>>>> 28 hdd 14.65039
1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30
GiB 2.8 TiB 81.19 1.09 203 up
osd.28
> >>>>> 29 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26
GiB 3.8 TiB 73.95 0.99 200 up
osd.29
> >>>>> 30 hdd 14.65039
1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28
GiB 3.1 TiB 79.07 1.06 207 up
osd.30
> >>>>> -7 58.89636
- 59 TiB 44 TiB 44 TiB 14 GiB 122
GiB 14 TiB 75.41 1.01 - host
s3db6
> >>>>> 32 hdd 3.73630
1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2
GiB 622 GiB 83.74 1.12 65 up
osd.32
> >>>>> 33 hdd 3.73630
0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1
GiB 740 GiB 80.67 1.08 52 up
osd.33
> >>>>> 34 hdd 3.73630
0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7
GiB 877 GiB 77.08 1.03 52 up
osd.34
> >>>>> 35 hdd 3.73630
0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0
GiB 1.4 TiB 62.18 0.83 42 up
osd.35
> >>>>> 36 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26
GiB 4.0 TiB 72.98 0.98 220 up
osd.36
> >>>>> 37 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38
GiB 3.6 TiB 75.30 1.01 200 up
osd.37
> >>>>> 38 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28
GiB 3.3 TiB 77.43 1.04 217 up
osd.38
> >>>>> -8 58.89636
- 59 TiB 47 TiB 46 TiB 3.9 GiB 116
GiB 12 TiB 78.98 1.06 - host
s3db7
> >>>>> 39 hdd 3.73630
1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5
GiB 499 GiB 86.96 1.17 43 up
osd.39
> >>>>> 40 hdd 3.73630
1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0
GiB 1.2 TiB 68.33 0.92 39 up
osd.40
> >>>>> 41 hdd 3.73630
1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9
GiB 732 GiB 80.86 1.08 64 up
osd.41
> >>>>> 42 hdd 3.73630
1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0
GiB 1.2 TiB 67.97 0.91 50 up
osd.42
> >>>>> 43 hdd 14.65039
1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28
GiB 2.9 TiB 80.32 1.08 213 up
osd.43
> >>>>> 44 hdd 14.65039
1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28
GiB 3.1 TiB 78.59 1.05 198 up
osd.44
> >>>>> 45 hdd 14.65039
1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30
GiB 2.8 TiB 81.05 1.09 214 up
osd.45
> >>>>> -9 51.28331
- 51 TiB 41 TiB 41 TiB 4.9 GiB 108
GiB 10 TiB 79.75 1.07 - host
s3db8
> >>>>> 8 hdd 7.32619
1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15
GiB 1.5 TiB 79.68 1.07 99 up
osd.8
> >>>>> 16 hdd 7.32619
1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15
GiB 1.4 TiB 80.25 1.08 97 up
osd.16
> >>>>> 31 hdd 7.32619
1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14
GiB 1.8 TiB 75.36 1.01 87 up
osd.31
> >>>>> 52 hdd 7.32619
1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15
GiB 1.6 TiB 78.19 1.05 113 up
osd.52
> >>>>> 53 hdd 7.32619
1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18
GiB 1.1 TiB 84.46 1.13 109 up
osd.53
> >>>>> 54 hdd 7.32619
1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15
GiB 1.7 TiB 76.73 1.03 115 up
osd.54
> >>>>> 55 hdd 7.32619
1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16
GiB 1.2 TiB 83.56 1.12 122 up
osd.55
> >>>>> -10 51.28331
- 51 TiB 39 TiB 39 TiB 4.7 GiB 100
GiB 12 TiB 76.05 1.02 - host
s3db9
> >>>>> 56 hdd 7.32619
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13
GiB 2.1 TiB 71.06 0.95 105 up
osd.56
> >>>>> 57 hdd 7.32619
1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16
GiB 1.2 TiB 83.17 1.12 102 up
osd.57
> >>>>> 58 hdd 7.32619
1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15
GiB 1.4 TiB 81.56 1.09 105 up
osd.58
> >>>>> 59 hdd 7.32619
1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15
GiB 1.4 TiB 80.64 1.08 94 up
osd.59
> >>>>> 60 hdd 7.32619
1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14
GiB 2.0 TiB 73.25 0.98 101 up
osd.60
> >>>>> 61 hdd 7.32619
1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12
GiB 2.5 TiB 65.84 0.88 103 up
osd.61
> >>>>> 62 hdd 7.32619
1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15
GiB 1.7 TiB 76.83 1.03 126 up
osd.62
> >>>>>
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2
TiB 173 TiB 74.57
> >>>>> MIN/MAX VAR: 0.17/1.20
STDDEV: 10.25
> >>>>>
> >>>>>
> >>>>>
> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >>>>>>
> >>>>>> No, increasing num PGs won't help substantially.
> >>>>>>
> >>>>>> Can you share the entire output of ceph osd df tree ?
> >>>>>>
> >>>>>> Did you already set
> >>>>>>
> >>>>>> ceph config set mgr mgr/balancer/upmap_max_deviation 1
> >>>>>>
> >>>>>>
> >>>>>> ??
> >>>>>> And I recommend debug_mgr 4/5 so you can see some basic
upmap
balancer logging.
> >>>>>>
> >>>>>> .. Dan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
> >>>>>>>
> >>>>>>> Hello people,
> >>>>>>>
> >>>>>>> I am still struggeling with the balancer
> >>>>>>>
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
> >>>>>>> Now I've read some more and might think that I do
not have
enough PGs.
> >>>>>>> Currently I have
84OSDs and 1024PGs for the main pool (3008
total). I
> >>>>>>> have the
autoscaler enabled, but I doesn't tell me to increase
the
> >>>>>>> PGs.
> >>>>>>>
> >>>>>>> What do you think?
> >>>>>>>
> >>>>>>> --
> >>>>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft
sich diesmal
abweichend
> >>>>>>> im groüen
Saal.
> >>>>>>> _______________________________________________
> >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
> >>>>>>> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal
abweichend im groüen Saal.
> >>>
> >>>
> >>>
> >>> --
> >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
> >
> >
> >
> > --
> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
4:23 p.m.
Hi,
How wide are your EC profiles? If they are really wide, you might be
reaching the limits of what is physically possible. Also, I'm not sure
that upmap in 14.2.11 is very smart about *improving* existing upmap
rules for a given PG, in the case that a PG already has an upmap-items
entry but it would help the distribution to add more mapping pairs to
that entry. What this means, is that it might sometimes be useful to
randomly remove some upmap entries and see if the balancer does a
better job when it replaces them.
But before you do that, I re-remembered that looking at the total PG
numbers is not useful -- you need to check the PGs per OSD for the
eu-central-1.rgw.buckets.data pool only.
We have a couple tools that can help with this:
1. To see the PGs per OSD for a given pool:
https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis…
E.g.: ./ceph-pool-pg-distribution 11 # to see the distribution of
your eu-central-1.rgw.buckets.data pool.
The output looks like this on my well balanced clusters:
# ceph-scripts/tools/ceph-pool-pg-distribution 15
Searching for PGs in pools: ['15']
Summary: 256 pgs on 56 osds
Num OSDs with X PGs:
13: 16
14: 40
You should expect a trimodal for your cluster.
2. You can also use another script from that repo to see the PGs per
OSD normalized to crush weight:
ceph-scripts/tools/ceph-pg-histogram --normalize --pool=15
This might explain what is going wrong.
Cheers, Dan
On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens <bb(a)kervyn.de> wrote:
>
> Absolutly:
> [root@s3db1 ~]# ceph osd df tree
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE
VAR PGS STATUS TYPE NAME
> -1 673.54224 - 674 TiB 496 TiB 468 TiB 97 GiB 1.2 TiB 177 TiB 73.67
1.00 - root default
> -2 58.30331 - 58 TiB 42 TiB 38 TiB 9.2 GiB 99 GiB 16 TiB 72.88
0.99 - host s3db1
> 23 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 714 MiB 25 GiB 3.7 TiB 74.87
1.02 194 up osd.23
> 69 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.6 GiB 40 GiB 3.4 TiB 76.32
1.04 199 up osd.69
> 73 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.3 GiB 34 GiB 3.8 TiB 74.15
1.01 203 up osd.73
> 79 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.8 GiB 0 B 1.3 TiB 65.44
0.89 47 up osd.79
> 80 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 2.2 GiB 0 B 1.3 TiB 65.34
0.89 48 up osd.80
> 81 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.1 GiB 0 B 1.3 TiB 65.38
0.89 47 up osd.81
> 82 hdd 3.63689 1.00000 3.6 TiB 2.5 TiB 1.1 TiB 619 MiB 0 B 1.1 TiB 68.46
0.93 41 up osd.82
> -11 50.94173 - 51 TiB 37 TiB 37 TiB 3.5 GiB 98 GiB 14 TiB 71.90
0.98 - host s3db10
> 63 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 647 MiB 14 GiB 2.0 TiB 72.72
0.99 94 up osd.63
> 64 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.2 TiB 668 MiB 14 GiB 2.0 TiB 72.23
0.98 93 up osd.64
> 65 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 227 MiB 14 GiB 2.1 TiB 71.16
0.97 100 up osd.65
> 66 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 313 MiB 14 GiB 1.9 TiB 74.25
1.01 92 up osd.66
> 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 584 MiB 14 GiB 2.1 TiB 70.63
0.96 96 up osd.67
> 68 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 720 MiB 14 GiB 2.1 TiB 71.72
0.97 101 up osd.68
> 70 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 425 MiB 14 GiB 2.1 TiB 70.59
0.96 97 up osd.70
> -12 50.99052 - 51 TiB 38 TiB 37 TiB 2.1 GiB 97 GiB 13 TiB 73.77
1.00 - host s3db11
> 46 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 229 MiB 14 GiB 1.7 TiB 77.05
1.05 97 up osd.46
> 47 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 159 MiB 13 GiB 2.2 TiB 70.00
0.95 89 up osd.47
> 48 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 279 MiB 14 GiB 2.1 TiB 71.82
0.97 98 up osd.48
> 49 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 276 MiB 14 GiB 1.8 TiB 74.90
1.02 95 up osd.49
> 50 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 336 MiB 14 GiB 2.0 TiB 72.13
0.98 93 up osd.50
> 51 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 728 MiB 15 GiB 1.6 TiB 77.76
1.06 98 up osd.51
> 72 hdd 7.32619 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 147 MiB 13 GiB 2.0 TiB 72.75
0.99 95 up osd.72
> -37 58.55478 - 59 TiB 44 TiB 44 TiB 4.4 GiB 122 GiB 15 TiB 75.20
1.02 - host s3db12
> 19 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 454 MiB 8.2 GiB 780 GiB 79.35
1.08 53 up osd.19
> 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 7.1 MiB 8.0 GiB 734 GiB 80.56
1.09 47 up osd.71
> 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 439 MiB 8.2 GiB 777 GiB 79.43
1.08 48 up osd.75
> 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 241 MiB 8.9 GiB 688 GiB 81.77
1.11 50 up osd.76
> 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB 3.6 TiB 75.44
1.02 201 up osd.77
> 78 hdd 14.60159 1.00000 15 TiB 10 TiB 10 TiB 1015 MiB 28 GiB 4.2 TiB 71.26
0.97 193 up osd.78
> 83 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.4 GiB 30 GiB 3.8 TiB 73.76
1.00 203 up osd.83
> -3 58.49872 - 58 TiB 42 TiB 36 TiB 8.2 GiB 89 GiB 17 TiB 71.71
0.97 - host s3db2
> 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.2 GiB 37 GiB 3.7 TiB 74.58
1.01 196 up osd.1
> 3 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 566 MiB 0 B 1.3 TiB 64.11
0.87 50 up osd.3
> 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 771 GiB 695 MiB 0 B 771 GiB 79.30
1.08 48 up osd.4
> 5 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 482 MiB 0 B 1.2 TiB 66.51
0.90 49 up osd.5
> 6 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 1.8 GiB 0 B 1.3 TiB 64.00
0.87 42 up osd.6
> 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 639 MiB 26 GiB 4.0 TiB 72.44
0.98 192 up osd.7
> 74 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 907 MiB 26 GiB 4.2 TiB 71.32
0.97 193 up osd.74
> -4 58.49872 - 58 TiB 43 TiB 36 TiB 34 GiB 85 GiB 16 TiB 72.69
0.99 - host s3db3
> 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 980 MiB 26 GiB 3.8 TiB 74.36
1.01 203 up osd.2
> 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.4 GiB 33 GiB 3.9 TiB 73.51
1.00 186 up osd.9
> 10 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 650 MiB 26 GiB 4.2 TiB 71.64
0.97 201 up osd.10
> 12 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 754 MiB 0 B 1.3 TiB 64.17
0.87 44 up osd.12
> 13 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 813 GiB 2.4 GiB 0 B 813 GiB 78.17
1.06 58 up osd.13
> 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 797 GiB 19 GiB 0 B 797 GiB 78.60
1.07 56 up osd.14
> 15 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 2.2 GiB 0 B 1.3 TiB 63.96
0.87 41 up osd.15
> -5 58.49872 - 58 TiB 43 TiB 36 TiB 6.7 GiB 97 GiB 15 TiB 74.04
1.01 - host s3db4
> 11 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 940 MiB 26 GiB 4.0 TiB 72.49
0.98 196 up osd.11
> 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1022 MiB 26 GiB 3.6 TiB 75.23
1.02 204 up osd.17
> 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 945 MiB 45 GiB 3.8 TiB 74.16
1.01 193 up osd.18
> 20 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1020 GiB 596 MiB 0 B 1020 GiB 72.62
0.99 57 up osd.20
> 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 1.9 GiB 0 B 1023 GiB 72.54
0.98 41 up osd.21
> 22 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 797 MiB 0 B 1023 GiB 72.54
0.98 53 up osd.22
> 24 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 766 GiB 618 MiB 0 B 766 GiB 79.42
1.08 46 up osd.24
> -6 58.89636 - 59 TiB 43 TiB 43 TiB 3.0 GiB 108 GiB 16 TiB 73.40
1.00 - host s3db5
> 0 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 92 MiB 7.2 GiB 1.1 TiB 71.16
0.97 45 up osd.0
> 25 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 2.4 MiB 7.3 GiB 1.1 TiB 71.23
0.97 41 up osd.25
> 26 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 181 MiB 7.6 GiB 935 GiB 75.57
1.03 45 up osd.26
> 27 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 5.1 MiB 7.0 GiB 1.1 TiB 71.20
0.97 47 up osd.27
> 28 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 977 MiB 26 GiB 3.8 TiB 73.85
1.00 197 up osd.28
> 29 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 872 MiB 26 GiB 4.1 TiB 71.98
0.98 196 up osd.29
> 30 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 943 MiB 27 GiB 3.6 TiB 75.51
1.03 202 up osd.30
> -7 58.89636 - 59 TiB 44 TiB 43 TiB 13 GiB 122 GiB 15 TiB 74.97
1.02 - host s3db6
> 32 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 27 MiB 7.6 GiB 940 GiB 75.42
1.02 55 up osd.32
> 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 376 MiB 8.2 GiB 691 GiB 81.94
1.11 55 up osd.33
> 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 450 MiB 8.5 GiB 620 GiB 83.79
1.14 54 up osd.34
> 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 316 MiB 8.4 GiB 690 GiB 81.98
1.11 50 up osd.35
> 36 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 489 MiB 25 GiB 4.1 TiB 71.69
0.97 208 up osd.36
> 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB 4.0 TiB 72.41
0.98 195 up osd.37
> 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 26 GiB 3.7 TiB 74.88
1.02 204 up osd.38
> -8 58.89636 - 59 TiB 44 TiB 43 TiB 3.8 GiB 111 GiB 15 TiB 74.16
1.01 - host s3db7
> 39 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 19 MiB 7.5 GiB 936 GiB 75.54
1.03 39 up osd.39
> 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.1 GiB 1.1 TiB 69.87
0.95 39 up osd.40
> 41 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.7 TiB 219 MiB 7.6 GiB 1011 GiB 73.57
1.00 55 up osd.41
> 42 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 593 MiB 7.1 GiB 1.1 TiB 70.02
0.95 47 up osd.42
> 43 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 500 MiB 27 GiB 3.7 TiB 74.67
1.01 204 up osd.43
> 44 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 3.7 TiB 74.62
1.01 193 up osd.44
> 45 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 29 GiB 3.6 TiB 75.16
1.02 204 up osd.45
> -9 51.28331 - 51 TiB 39 TiB 39 TiB 4.9 GiB 107 GiB 12 TiB 76.50
1.04 - host s3db8
> 8 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 474 MiB 14 GiB 1.7 TiB 76.37
1.04 98 up osd.8
> 16 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 783 MiB 15 GiB 1.6 TiB 78.39
1.06 100 up osd.16
> 31 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 441 MiB 14 GiB 1.6 TiB 77.70
1.05 91 up osd.31
> 52 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 939 MiB 14 GiB 1.7 TiB 76.29
1.04 102 up osd.52
> 53 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 848 MiB 18 GiB 1.9 TiB 74.30
1.01 98 up osd.53
> 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 16 GiB 1.7 TiB 76.99
1.05 106 up osd.54
> 55 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 460 MiB 15 GiB 1.8 TiB 75.46
1.02 105 up osd.55
> -10 51.28331 - 51 TiB 37 TiB 37 TiB 3.8 GiB 96 GiB 14 TiB 72.77
0.99 - host s3db9
> 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 846 MiB 13 GiB 2.1 TiB 71.16
0.97 104 up osd.56
> 57 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 513 MiB 15 GiB 1.7 TiB 76.53
1.04 96 up osd.57
> 58 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 604 MiB 13 GiB 2.1 TiB 71.23
0.97 98 up osd.58
> 59 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 414 MiB 13 GiB 2.2 TiB 70.03
0.95 88 up osd.59
> 60 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 227 MiB 14 GiB 1.8 TiB 75.54
1.03 97 up osd.60
> 61 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 456 MiB 13 GiB 2.2 TiB 70.01
0.95 95 up osd.61
> 62 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 843 MiB 14 GiB 1.8 TiB 74.93
1.02 110 up osd.62
> TOTAL 674 TiB 496 TiB 468 TiB 97 GiB 1.2 TiB 177 TiB
73.67
> MIN/MAX VAR: 0.87/1.14 STDDEV: 4.22
>
> Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>
>> OK thanks. Indeed "prepared 0/10 changes" means it thinks things are
balanced.
>> Could you again share the full ceph osd df tree?
>>
>> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >
>> > Hi Dan,
>> >
>> > I've set the autoscaler to warn, but it actually does not warn for now.
So not touching it for now.
>> >
>> > this is what the log says in minute intervals:
>> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/active
>> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/sleep_interval
>> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/begin_time
>> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/end_time
>> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/begin_weekday
>> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/end_weekday
>> > 2021-03-15 13:51:00.971 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/pool_ids
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Optimize plan
auto_2021-03-15_13:51:00
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/mode
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Mode upmap, max
misplaced 0.050000
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] do_upmap
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/upmap_max_iterations
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key:
mgr/balancer/upmap_max_deviation
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] pools
['eu-msg-1.rgw.data.root', 'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys', 'eu-central-1.rgw.buckets.index',
'rbd', 'eu-msg-1.rgw.log', 'whitespace-again-2021-03-10_2',
'eu-msg-1.rgw.buckets.index', 'eu-msg-1.rgw.meta',
'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control', '.rgw.root', 'eu-msg-1.rgw.buckets.data',
'default.rgw.control', 'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3', 'default.rgw.log',
'eu-central-1.rgw.meta', 'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
>> > 2021-03-15 13:51:01.224 7f307d5fd700 4 mgr[balancer] prepared 0/10 changes
>> >
>> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >>
>> >> I suggest to just disable the autoscaler until your balancing is
understood.
>> >>
>> >> What does your active mgr log say (with debug_mgr 4/5), grep balancer
>> >> /var/log/ceph/ceph-mgr.*.log
>> >>
>> >> -- Dan
>> >>
>> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens <bb(a)kervyn.de>
wrote:
>> >> >
>> >> > Hi,
>> >> > this unfortunally did not solve my problem. I still have some OSDs
that fill up to 85%
>> >> >
>> >> > According to the logging, the autoscaler might want to add more PGs
to one Bucken and reduce almost all other buckets to 32.
>> >> > 2021-03-15 12:19:58.825 7f307f601700 4 mgr[pg_autoscaler] Pool
'eu-central-1.rgw.buckets.data' root_id -1 using 0.705080476146 of space, bias
1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
>> >> >
>> >> > Why the balancing does not happen is still nebulous to me.
>> >> >
>> >> >
>> >> >
>> >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >> >>
>> >> >> OK
>> >> >> Btw, you might need to fail to a new mgr... I'm not sure if
the current active will read that new config.
>> >> >>
>> >> >> .. dan
>> >> >>
>> >> >>
>> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> ok thanks. I just changed the value and rewighted
everything back to 1. Now I let it sync the weekend and check how it will be on monday.
>> >> >>> We tried to have the systems total storage balanced as
possible. New systems will be with 8TB disks but for the exiting ones we added 16TB to
offset the 4TB disks and we needed a lot of storage fast, because of a DC move. If you
have any recommendations I would be happy to hear them.
>> >> >>>
>> >> >>> Cheers
>> >> >>> Boris
>> >> >>>
>> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >> >>>>
>> >> >>>> Thanks.
>> >> >>>>
>> >> >>>> Decreasing the max deviation to 2 or 1 should help in
your case. This option controls when the balancer stops trying to move PGs around -- by
default it stops when the deviation from the mean is 5. Yes this is too large IMO -- all
of our clusters have this set to 1.
>> >> >>>>
>> >> >>>> And given that you have some OSDs with more than 200
PGs, you definitely shouldn't increase the num PGs.
>> >> >>>>
>> >> >>>> But anyway with your mixed device sizes it might be
challenging to make a perfectly uniform distribution. Give it a try with 1 though, and let
us know how it goes.
>> >> >>>>
>> >> >>>> .. Dan
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>> >> >>>>>
>> >> >>>>> Hi Dan,
>> >> >>>>>
>> >> >>>>> upmap_max_deviation is default (5) in our cluster.
Is 1 the recommended deviation?
>> >> >>>>>
>> >> >>>>> I added the whole ceph osd df tree, (I need to
remove some OSDs and readd them as bluestore with SSD, so 69, 73 and 82 are a bit off now.
I also reweighted to try to get the %USE mitigated).
>> >> >>>>>
>> >> >>>>> I will increase the mgr debugging to see what is
the problem.
>> >> >>>>>
>> >> >>>>> [root@s3db1 ~]# ceph osd df tree
>> >> >>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA
OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
>> >> >>>>> -1 673.54224 - 659 TiB 491 TiB 464
TiB 96 GiB 1.2 TiB 168 TiB 74.57 1.00 - root default
>> >> >>>>> -2 58.30331 - 44 TiB 22 TiB 17
TiB 5.7 GiB 38 GiB 22 TiB 49.82 0.67 - host s3db1
>> >> >>>>> 23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7
TiB 156 MiB 4.4 GiB 13 TiB 12.50 0.17 101 up osd.23
>> >> >>>>> 69 hdd 14.55269 0 0 B 0 B 0
B 0 B 0 B 0 B 0 0 11 up osd.69
>> >> >>>>> 73 hdd 14.55269 1.00000 15 TiB 10 TiB 10
TiB 6.1 MiB 33 GiB 4.2 TiB 71.15 0.95 107 up osd.73
>> >> >>>>> 79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747
GiB 2.0 GiB 0 B 747 GiB 79.94 1.07 52 up osd.79
>> >> >>>>> 80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0
TiB 1.9 GiB 0 B 1.0 TiB 71.61 0.96 58 up osd.80
>> >> >>>>> 81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5
TiB 1.1 GiB 0 B 1.5 TiB 60.07 0.81 55 up osd.81
>> >> >>>>> 82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7
TiB 536 MiB 0 B 1.7 TiB 52.68 0.71 30 up osd.82
>> >> >>>>> -11 50.94173 - 51 TiB 38 TiB 38
TiB 3.7 GiB 100 GiB 13 TiB 74.69 1.00 - host s3db10
>> >> >>>>> 63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5
TiB 616 MiB 14 GiB 1.7 TiB 76.04 1.02 92 up osd.63
>> >> >>>>> 64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5
TiB 820 MiB 15 GiB 1.8 TiB 75.54 1.01 101 up osd.64
>> >> >>>>> 65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3
TiB 109 MiB 14 GiB 2.0 TiB 73.17 0.98 105 up osd.65
>> >> >>>>> 66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8
TiB 423 MiB 15 GiB 1.4 TiB 80.38 1.08 98 up osd.66
>> >> >>>>> 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 572 MiB 14 GiB 2.2 TiB 70.10 0.94 100 up osd.67
>> >> >>>>> 68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3
TiB 630 MiB 13 GiB 2.0 TiB 72.88 0.98 107 up osd.68
>> >> >>>>> 70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4
TiB 648 MiB 14 GiB 1.8 TiB 74.73 1.00 102 up osd.70
>> >> >>>>> -12 50.99052 - 51 TiB 39 TiB 39
TiB 2.9 GiB 99 GiB 12 TiB 77.24 1.04 - host s3db11
>> >> >>>>> 46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7
TiB 102 MiB 15 GiB 1.5 TiB 78.91 1.06 97 up osd.46
>> >> >>>>> 47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 61 MiB 13 GiB 2.1 TiB 71.47 0.96 96 up osd.47
>> >> >>>>> 48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1
TiB 853 MiB 15 GiB 1.2 TiB 83.46 1.12 109 up osd.48
>> >> >>>>> 49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7
TiB 708 MiB 15 GiB 1.5 TiB 78.96 1.06 98 up osd.49
>> >> >>>>> 50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8
TiB 472 MiB 15 GiB 1.4 TiB 80.40 1.08 102 up osd.50
>> >> >>>>> 51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9
TiB 729 MiB 15 GiB 1.3 TiB 81.70 1.10 110 up osd.51
>> >> >>>>> 72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8
TiB 91 MiB 12 GiB 2.5 TiB 65.82 0.88 89 up osd.72
>> >> >>>>> -37 58.55478 - 59 TiB 46 TiB 46
TiB 5.0 GiB 124 GiB 12 TiB 79.04 1.06 - host s3db12
>> >> >>>>> 19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1
TiB 462 MiB 8.2 GiB 559 GiB 85.18 1.14 55 up osd.19
>> >> >>>>> 71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8
TiB 3.9 MiB 7.8 GiB 825 GiB 78.14 1.05 50 up osd.71
>> >> >>>>> 75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1
TiB 576 MiB 8.3 GiB 555 GiB 85.29 1.14 57 up osd.75
>> >> >>>>> 76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1
TiB 239 MiB 9.3 GiB 501 GiB 86.73 1.16 50 up osd.76
>> >> >>>>> 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11
TiB 880 MiB 30 GiB 3.6 TiB 75.57 1.01 202 up osd.77
>> >> >>>>> 78 hdd 14.60159 1.00000 15 TiB 11 TiB 11
TiB 1.0 GiB 30 GiB 3.4 TiB 76.65 1.03 196 up osd.78
>> >> >>>>> 83 hdd 14.60159 1.00000 15 TiB 12 TiB 12
TiB 1.8 GiB 31 GiB 2.9 TiB 80.04 1.07 223 up osd.83
>> >> >>>>> -3 58.49872 - 58 TiB 43 TiB 38
TiB 8.1 GiB 91 GiB 16 TiB 73.15 0.98 - host s3db2
>> >> >>>>> 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 3.1 GiB 38 GiB 3.6 TiB 75.52 1.01 194 up osd.1
>> >> >>>>> 3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4
TiB 418 MiB 0 B 1.4 TiB 60.94 0.82 52 up osd.3
>> >> >>>>> 4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401
GiB 845 MiB 0 B 401 GiB 89.23 1.20 53 up osd.4
>> >> >>>>> 5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 437 MiB 0 B 1.3 TiB 62.88 0.84 51 up osd.5
>> >> >>>>> 6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7
TiB 1.8 GiB 0 B 1.7 TiB 54.51 0.73 47 up osd.6
>> >> >>>>> 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 493 MiB 26 GiB 3.8 TiB 73.90 0.99 185 up osd.7
>> >> >>>>> 74 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.1 GiB 27 GiB 3.5 TiB 76.27 1.02 208 up osd.74
>> >> >>>>> -4 58.49872 - 58 TiB 43 TiB 37
TiB 33 GiB 86 GiB 15 TiB 74.05 0.99 - host s3db3
>> >> >>>>> 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 850 MiB 26 GiB 4.0 TiB 72.78 0.98 203 up osd.2
>> >> >>>>> 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 8.3 GiB 33 GiB 3.6 TiB 75.62 1.01 189 up osd.9
>> >> >>>>> 10 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 663 MiB 28 GiB 3.5 TiB 76.34 1.02 211 up osd.10
>> >> >>>>> 12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2
TiB 633 MiB 0 B 1.2 TiB 66.22 0.89 44 up osd.12
>> >> >>>>> 13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720
GiB 2.3 GiB 0 B 720 GiB 80.66 1.08 66 up osd.13
>> >> >>>>> 14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552
GiB 18 GiB 0 B 552 GiB 85.18 1.14 60 up osd.14
>> >> >>>>> 15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7
TiB 2.1 GiB 0 B 1.7 TiB 53.72 0.72 44 up osd.15
>> >> >>>>> -5 58.49872 - 58 TiB 45 TiB 37
TiB 7.2 GiB 99 GiB 14 TiB 76.37 1.02 - host s3db4
>> >> >>>>> 11 hdd 14.65039 1.00000 15 TiB 12 TiB 12
TiB 897 MiB 28 GiB 2.8 TiB 81.15 1.09 205 up osd.11
>> >> >>>>> 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.2 GiB 27 GiB 3.6 TiB 75.38 1.01 211 up osd.17
>> >> >>>>> 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 965 MiB 44 GiB 4.0 TiB 72.86 0.98 188 up osd.18
>> >> >>>>> 20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796
GiB 529 MiB 0 B 796 GiB 78.63 1.05 66 up osd.20
>> >> >>>>> 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1
TiB 2.1 GiB 0 B 1.1 TiB 70.32 0.94 47 up osd.21
>> >> >>>>> 22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802
GiB 882 MiB 0 B 802 GiB 78.47 1.05 58 up osd.22
>> >> >>>>> 24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856
GiB 645 MiB 0 B 856 GiB 77.01 1.03 47 up osd.24
>> >> >>>>> -6 58.89636 - 59 TiB 44 TiB 44
TiB 2.4 GiB 111 GiB 15 TiB 75.22 1.01 - host s3db5
>> >> >>>>> 0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3
TiB 70 MiB 6.6 GiB 1.3 TiB 65.00 0.87 48 up osd.0
>> >> >>>>> 25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3
TiB 5.3 MiB 6.6 GiB 1.4 TiB 63.86 0.86 41 up osd.25
>> >> >>>>> 26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8
TiB 181 MiB 7.6 GiB 862 GiB 77.47 1.04 48 up osd.26
>> >> >>>>> 27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2
TiB 7.0 MiB 6.1 GiB 1.5 TiB 61.00 0.82 48 up osd.27
>> >> >>>>> 28 hdd 14.65039 1.00000 15 TiB 12 TiB 12
TiB 937 MiB 30 GiB 2.8 TiB 81.19 1.09 203 up osd.28
>> >> >>>>> 29 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 536 MiB 26 GiB 3.8 TiB 73.95 0.99 200 up osd.29
>> >> >>>>> 30 hdd 14.65039 1.00000 15 TiB 12 TiB 11
TiB 744 MiB 28 GiB 3.1 TiB 79.07 1.06 207 up osd.30
>> >> >>>>> -7 58.89636 - 59 TiB 44 TiB 44
TiB 14 GiB 122 GiB 14 TiB 75.41 1.01 - host s3db6
>> >> >>>>> 32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0
TiB 16 MiB 8.2 GiB 622 GiB 83.74 1.12 65 up osd.32
>> >> >>>>> 33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9
TiB 14 MiB 8.1 GiB 740 GiB 80.67 1.08 52 up osd.33
>> >> >>>>> 34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8
TiB 449 MiB 7.7 GiB 877 GiB 77.08 1.03 52 up osd.34
>> >> >>>>> 35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2
TiB 133 MiB 7.0 GiB 1.4 TiB 62.18 0.83 42 up osd.35
>> >> >>>>> 36 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 544 MiB 26 GiB 4.0 TiB 72.98 0.98 220 up osd.36
>> >> >>>>> 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 11 GiB 38 GiB 3.6 TiB 75.30 1.01 200 up osd.37
>> >> >>>>> 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.2 GiB 28 GiB 3.3 TiB 77.43 1.04 217 up osd.38
>> >> >>>>> -8 58.89636 - 59 TiB 47 TiB 46
TiB 3.9 GiB 116 GiB 12 TiB 78.98 1.06 - host s3db7
>> >> >>>>> 39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2
TiB 19 MiB 8.5 GiB 499 GiB 86.96 1.17 43 up osd.39
>> >> >>>>> 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5
TiB 144 MiB 7.0 GiB 1.2 TiB 68.33 0.92 39 up osd.40
>> >> >>>>> 41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9
TiB 218 MiB 7.9 GiB 732 GiB 80.86 1.08 64 up osd.41
>> >> >>>>> 42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4
TiB 594 MiB 7.0 GiB 1.2 TiB 67.97 0.91 50 up osd.42
>> >> >>>>> 43 hdd 14.65039 1.00000 15 TiB 12 TiB 12
TiB 564 MiB 28 GiB 2.9 TiB 80.32 1.08 213 up osd.43
>> >> >>>>> 44 hdd 14.65039 1.00000 15 TiB 12 TiB 11
TiB 1.3 GiB 28 GiB 3.1 TiB 78.59 1.05 198 up osd.44
>> >> >>>>> 45 hdd 14.65039 1.00000 15 TiB 12 TiB 12
TiB 1.2 GiB 30 GiB 2.8 TiB 81.05 1.09 214 up osd.45
>> >> >>>>> -9 51.28331 - 51 TiB 41 TiB 41
TiB 4.9 GiB 108 GiB 10 TiB 79.75 1.07 - host s3db8
>> >> >>>>> 8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8
TiB 472 MiB 15 GiB 1.5 TiB 79.68 1.07 99 up osd.8
>> >> >>>>> 16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8
TiB 785 MiB 15 GiB 1.4 TiB 80.25 1.08 97 up osd.16
>> >> >>>>> 31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5
TiB 438 MiB 14 GiB 1.8 TiB 75.36 1.01 87 up osd.31
>> >> >>>>> 52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7
TiB 844 MiB 15 GiB 1.6 TiB 78.19 1.05 113 up osd.52
>> >> >>>>> 53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1
TiB 792 MiB 18 GiB 1.1 TiB 84.46 1.13 109 up osd.53
>> >> >>>>> 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6
TiB 959 MiB 15 GiB 1.7 TiB 76.73 1.03 115 up osd.54
>> >> >>>>> 55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1
TiB 699 MiB 16 GiB 1.2 TiB 83.56 1.12 122 up osd.55
>> >> >>>>> -10 51.28331 - 51 TiB 39 TiB 39
TiB 4.7 GiB 100 GiB 12 TiB 76.05 1.02 - host s3db9
>> >> >>>>> 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 840 MiB 13 GiB 2.1 TiB 71.06 0.95 105 up osd.56
>> >> >>>>> 57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0
TiB 1.0 GiB 16 GiB 1.2 TiB 83.17 1.12 102 up osd.57
>> >> >>>>> 58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9
TiB 43 MiB 15 GiB 1.4 TiB 81.56 1.09 105 up osd.58
>> >> >>>>> 59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9
TiB 429 MiB 15 GiB 1.4 TiB 80.64 1.08 94 up osd.59
>> >> >>>>> 60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3
TiB 226 MiB 14 GiB 2.0 TiB 73.25 0.98 101 up osd.60
>> >> >>>>> 61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8
TiB 1.1 GiB 12 GiB 2.5 TiB 65.84 0.88 103 up osd.61
>> >> >>>>> 62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6
TiB 1.0 GiB 15 GiB 1.7 TiB 76.83 1.03 126 up osd.62
>> >> >>>>> TOTAL 674 TiB 501 TiB 473
TiB 96 GiB 1.2 TiB 173 TiB 74.57
>> >> >>>>> MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van
der Ster <dan(a)vanderster.com>om>:
>> >> >>>>>>
>> >> >>>>>> No, increasing num PGs won't help
substantially.
>> >> >>>>>>
>> >> >>>>>> Can you share the entire output of ceph osd df
tree ?
>> >> >>>>>>
>> >> >>>>>> Did you already set
>> >> >>>>>>
>> >> >>>>>> ceph config set mgr
mgr/balancer/upmap_max_deviation 1
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> ??
>> >> >>>>>> And I recommend debug_mgr 4/5 so you can see
some basic upmap balancer logging.
>> >> >>>>>>
>> >> >>>>>> .. Dan
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>> >> >>>>>>>
>> >> >>>>>>> Hello people,
>> >> >>>>>>>
>> >> >>>>>>> I am still struggeling with the balancer
>> >> >>>>>>>
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>> >> >>>>>>> Now I've read some more and might think
that I do not have enough PGs.
>> >> >>>>>>> Currently I have 84OSDs and 1024PGs for the
main pool (3008 total). I
>> >> >>>>>>> have the autoscaler enabled, but I
doesn't tell me to increase the
>> >> >>>>>>> PGs.
>> >> >>>>>>>
>> >> >>>>>>> What do you think?
>> >> >>>>>>>
>> >> >>>>>>> --
>> >> >>>>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend
>> >> >>>>>>> im groüen Saal.
>> >> >>>>>>>
_______________________________________________
>> >> >>>>>>> ceph-users mailing list --
ceph-users(a)ceph.io
>> >> >>>>>>> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> --
>> >> >>>>> Die Selbsthilfegruppe "UTF-8-Probleme"
trifft sich diesmal abweichend im groüen Saal.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft
sich diesmal abweichend im groüen Saal.
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal abweichend im groüen Saal.
>> >
>> >
>> >
>> > --
>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
16 Mar
16 Mar
8:34 a.m.
Hi Dan,
my EC profile look very "default" to me.
[root@s3db1 ~]# ceph osd erasure-code-profile ls
default
[root@s3db1 ~]# ceph osd erasure-code-profile get default
k=2
m=1
plugin=jerasure
technique=reed_sol_van
I don't understand the ouput, but the balancing get worse over night:
[root@s3db1 ~]# ceph-scripts/tools/ceph-pool-pg-distribution 11
Searching for PGs in pools: ['11']
Summary: 1024 PGs on 84 osds
Num OSDs with X PGs:
15: 8
16: 7
17: 6
18: 10
19: 1
32: 10
33: 4
34: 6
35: 8
65: 5
66: 5
67: 4
68: 10
[root@s3db1 ~]# ceph-scripts/tools/ceph-pg-histogram --normalize --pool=11
# NumSamples = 84; Min = 4.12; Max = 5.09
# Mean = 4.553355; Variance = 0.052415; SD = 0.228942; Median 4.561608
# each ∎ represents a count of 1
4.1244 - 4.2205 [ 8]: ∎∎∎∎∎∎∎∎
4.2205 - 4.3166 [ 6]: ∎∎∎∎∎∎
4.3166 - 4.4127 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
4.4127 - 4.5087 [ 10]: ∎∎∎∎∎∎∎∎∎∎
4.5087 - 4.6048 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
4.6048 - 4.7009 [ 19]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
4.7009 - 4.7970 [ 6]: ∎∎∎∎∎∎
4.7970 - 4.8931 [ 8]: ∎∎∎∎∎∎∎∎
4.8931 - 4.9892 [ 4]: ∎∎∎∎
4.9892 - 5.0852 [ 1]: ∎
[root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail
14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 724 GiB 19 GiB 0 B 724
GiB 80.56 1.07 56 up osd.14
19 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 466 MiB 7.9 GiB 708
GiB 81.25 1.08 53 up osd.19
4 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 698 GiB 703 MiB 0 B 698
GiB 81.27 1.08 48 up osd.4
24 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 695 GiB 640 MiB 0 B 695
GiB 81.34 1.08 46 up osd.24
75 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 440 MiB 8.1 GiB 704
GiB 81.35 1.08 48 up osd.75
71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 7.5 MiB 8.0 GiB 663
GiB 82.44 1.09 47 up osd.71
76 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 251 MiB 9.0 GiB 617
GiB 83.65 1.11 50 up osd.76
33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 399 MiB 8.1 GiB 618
GiB 83.85 1.11 55 up osd.33
35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 317 MiB 8.8 GiB 617
GiB 83.87 1.11 50 up osd.35
34 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 451 MiB 8.7 GiB 545
GiB 85.75 1.14 54 up osd.34
Am Mo., 15. März 2021 um 17:23 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
Hi,
How wide are your EC profiles? If they are really wide, you might be
reaching the limits of what is physically possible. Also, I'm not sure
that upmap in 14.2.11 is very smart about *improving* existing upmap
rules for a given PG, in the case that a PG already has an upmap-items
entry but it would help the distribution to add more mapping pairs to
that entry. What this means, is that it might sometimes be useful to
randomly remove some upmap entries and see if the balancer does a
better job when it replaces them.
But before you do that, I re-remembered that looking at the total PG
numbers is not useful -- you need to check the PGs per OSD for the
eu-central-1.rgw.buckets.data pool only.
We have a couple tools that can help with this:
1. To see the PGs per OSD for a given pool:
https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis…
E.g.: ./ceph-pool-pg-distribution 11 # to see the distribution of
your eu-central-1.rgw.buckets.data pool.
The output looks like this on my well balanced clusters:
# ceph-scripts/tools/ceph-pool-pg-distribution 15
Searching for PGs in pools: ['15']
Summary: 256 pgs on 56 osds
Num OSDs with X PGs:
13: 16
14: 40
You should expect a trimodal for your cluster.
2. You can also use another script from that repo to see the PGs per
OSD normalized to crush weight:
ceph-scripts/tools/ceph-pg-histogram --normalize --pool=15
This might explain what is going wrong.
Cheers, Dan
On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Absolutly:
[root@s3db1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE
VAR PGS STATUS TYPE NAME
-1 673.54224 - 674 TiB 496 TiB 468
TiB 97 GiB 1.2 TiB
177 TiB 73.67 1.00 - root default
-2 58.30331 - 58 TiB 42 TiB 38
TiB 9.2 GiB 99 GiB
16 TiB 72.88 0.99 - host s3db1
23 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 714 MiB 25 GiB
3.7 TiB 74.87 1.02 194 up osd.23
69 hdd 14.55269 1.00000 15 TiB 11 TiB 11
TiB 1.6 GiB 40 GiB
3.4 TiB 76.32 1.04 199 up osd.69
73 hdd 14.55269 1.00000 15 TiB 11 TiB 11
TiB 1.3 GiB 34 GiB
3.8 TiB 74.15 1.01 203 up osd.73
79 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3
TiB 1.8 GiB 0 B
1.3 TiB 65.44 0.89 47 up osd.79
80 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3
TiB 2.2 GiB 0 B
1.3 TiB 65.34 0.89 48 up osd.80
81 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3
TiB 1.1 GiB 0 B
1.3 TiB 65.38 0.89 47 up osd.81
82 hdd 3.63689 1.00000 3.6 TiB 2.5 TiB 1.1
TiB 619 MiB 0 B
1.1 TiB 68.46 0.93 41 up osd.82
-11 50.94173 - 51 TiB 37 TiB 37
TiB 3.5 GiB 98 GiB
14 TiB 71.90 0.98 - host s3db10
63 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3
TiB 647 MiB 14 GiB
2.0 TiB 72.72 0.99 94 up osd.63
64 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.2
TiB 668 MiB 14 GiB
2.0 TiB 72.23 0.98 93 up osd.64
65 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 227 MiB 14 GiB
2.1 TiB 71.16 0.97 100 up osd.65
66 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4
TiB 313 MiB 14 GiB
1.9 TiB 74.25 1.01 92 up osd.66
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 584 MiB 14 GiB
2.1 TiB 70.63 0.96 96 up osd.67
68 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 720 MiB 14 GiB
2.1 TiB 71.72 0.97 101 up osd.68
70 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 425 MiB 14 GiB
2.1 TiB 70.59 0.96 97 up osd.70
-12 50.99052 - 51 TiB 38 TiB 37
TiB 2.1 GiB 97 GiB
13 TiB 73.77 1.00 - host s3db11
46 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.6
TiB 229 MiB 14 GiB
1.7 TiB 77.05 1.05 97 up osd.46
47 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 159 MiB 13 GiB
2.2 TiB 70.00 0.95 89 up osd.47
48 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 279 MiB 14 GiB
2.1 TiB 71.82 0.97 98 up osd.48
49 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.4
TiB 276 MiB 14 GiB
1.8 TiB 74.90 1.02 95 up osd.49
50 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 336 MiB 14 GiB
2.0 TiB 72.13 0.98 93 up osd.50
51 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.6
TiB 728 MiB 15 GiB
1.6 TiB 77.76 1.06 98 up osd.51
72 hdd 7.32619 1.00000 7.3 TiB 5.3 TiB 5.3
TiB 147 MiB 13 GiB
2.0 TiB 72.75 0.99 95 up osd.72
-37 58.55478 - 59 TiB 44 TiB 44
TiB 4.4 GiB 122 GiB
15 TiB 75.20 1.02 - host s3db12
19 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9
TiB 454 MiB 8.2 GiB
780 GiB 79.35 1.08 53 up osd.19
71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9
TiB 7.1 MiB 8.0 GiB
734 GiB 80.56 1.09 47 up osd.71
75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9
TiB 439 MiB 8.2 GiB
777 GiB 79.43 1.08 48 up osd.75
76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0
TiB 241 MiB 8.9 GiB
688 GiB 81.77 1.11 50 up osd.76
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11
TiB 880 MiB 30 GiB
3.6 TiB 75.44 1.02 201 up osd.77
78 hdd 14.60159 1.00000 15 TiB 10 TiB 10
TiB 1015 MiB 28 GiB
4.2 TiB 71.26 0.97 193 up osd.78
83 hdd 14.60159 1.00000 15 TiB 11 TiB 11
TiB 1.4 GiB 30 GiB
3.8 TiB 73.76 1.00 203 up osd.83
-3 58.49872 - 58 TiB 42 TiB 36
TiB 8.2 GiB 89 GiB
17 TiB 71.71 0.97 - host s3db2
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 3.2 GiB 37 GiB
3.7 TiB 74.58 1.01 196 up osd.1
3 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 566 MiB 0 B
1.3 TiB 64.11 0.87 50 up osd.3
4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 771
GiB 695 MiB 0 B
771 GiB 79.30 1.08 48 up osd.4
5 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2
TiB 482 MiB 0 B
1.2 TiB 66.51 0.90 49 up osd.5
6 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 1.8 GiB 0 B
1.3 TiB 64.00 0.87 42 up osd.6
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 639 MiB 26 GiB
4.0 TiB 72.44 0.98 192 up osd.7
74 hdd 14.65039 1.00000 15 TiB 10 TiB 10
TiB 907 MiB 26 GiB
4.2 TiB 71.32 0.97 193 up osd.74
-4 58.49872 - 58 TiB 43 TiB 36
TiB 34 GiB 85 GiB
16 TiB 72.69 0.99 - host s3db3
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 980 MiB 26 GiB
3.8 TiB 74.36 1.01 203 up osd.2
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 8.4 GiB 33 GiB
3.9 TiB 73.51 1.00 186 up osd.9
10 hdd 14.65039 1.00000 15 TiB 10 TiB 10
TiB 650 MiB 26 GiB
4.2 TiB 71.64 0.97 201 up osd.10
12 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 754 MiB 0 B
1.3 TiB 64.17 0.87 44 up osd.12
13 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 813
GiB 2.4 GiB 0 B
813 GiB 78.17 1.06 58 up osd.13
14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 797
GiB 19 GiB 0 B
797 GiB 78.60 1.07 56 up osd.14
15 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 2.2 GiB 0 B
1.3 TiB 63.96 0.87 41 up osd.15
-5 58.49872 - 58 TiB 43 TiB 36
TiB 6.7 GiB 97 GiB
15 TiB 74.04 1.01 - host s3db4
11 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 940 MiB 26 GiB
4.0 TiB 72.49 0.98 196 up osd.11
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1022 MiB 26 GiB
3.6 TiB 75.23 1.02 204 up osd.17
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 945 MiB 45 GiB
3.8 TiB 74.16 1.01 193 up osd.18
20 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1020
GiB 596 MiB 0 B
1020 GiB 72.62 0.99 57 up osd.20
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023
GiB 1.9 GiB 0 B
1023 GiB 72.54 0.98 41 up osd.21
22 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023
GiB 797 MiB 0 B
1023 GiB 72.54 0.98 53 up osd.22
24 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 766
GiB 618 MiB 0 B
766 GiB 79.42 1.08 46 up osd.24
-6 58.89636 - 59 TiB 43 TiB 43
TiB 3.0 GiB 108 GiB
16 TiB 73.40 1.00 - host s3db5
0 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6
TiB 92 MiB 7.2 GiB
1.1 TiB 71.16 0.97 45 up osd.0
25 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6
TiB 2.4 MiB 7.3 GiB
1.1 TiB 71.23 0.97 41 up osd.25
26 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7
TiB 181 MiB 7.6 GiB
935 GiB 75.57 1.03 45 up osd.26
27 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6
TiB 5.1 MiB 7.0 GiB
1.1 TiB 71.20 0.97 47 up osd.27
28 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 977 MiB 26 GiB
3.8 TiB 73.85 1.00 197 up osd.28
29 hdd 14.65039 1.00000 15 TiB 11 TiB 10
TiB 872 MiB 26 GiB
4.1 TiB 71.98 0.98 196 up osd.29
30 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 943 MiB 27 GiB
3.6 TiB 75.51 1.03 202 up osd.30
-7 58.89636 - 59 TiB 44 TiB 43
TiB 13 GiB 122 GiB
15 TiB 74.97 1.02 - host s3db6
32 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7
TiB 27 MiB 7.6 GiB
940 GiB 75.42 1.02 55 up osd.32
33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0
TiB 376 MiB 8.2 GiB
691 GiB 81.94 1.11 55 up osd.33
34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0
TiB 450 MiB 8.5 GiB
620 GiB 83.79 1.14 54 up osd.34
35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0
TiB 316 MiB 8.4 GiB
690 GiB 81.98 1.11 50 up osd.35
36 hdd 14.65039 1.00000 15 TiB 11 TiB 10
TiB 489 MiB 25 GiB
4.1 TiB 71.69 0.97 208 up osd.36
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 11 GiB 38 GiB
4.0 TiB 72.41 0.98 195 up osd.37
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.1 GiB 26 GiB
3.7 TiB 74.88 1.02 204 up osd.38
-8 58.89636 - 59 TiB 44 TiB 43
TiB 3.8 GiB 111 GiB
15 TiB 74.16 1.01 - host s3db7
39 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7
TiB 19 MiB 7.5 GiB
936 GiB 75.54 1.03 39 up osd.39
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5
TiB 144 MiB 7.1 GiB
1.1 TiB 69.87 0.95 39 up osd.40
41 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.7
TiB 219 MiB 7.6 GiB
1011 GiB 73.57 1.00 55 up osd.41
42 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5
TiB 593 MiB 7.1 GiB
1.1 TiB 70.02 0.95 47 up osd.42
43 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 500 MiB 27 GiB
3.7 TiB 74.67 1.01 204 up osd.43
44 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.1 GiB 27 GiB
3.7 TiB 74.62 1.01 193 up osd.44
45 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.2 GiB 29 GiB
3.6 TiB 75.16 1.02 204 up osd.45
-9 51.28331 - 51 TiB 39 TiB 39
TiB 4.9 GiB 107 GiB
12 TiB 76.50 1.04 - host s3db8
8 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5
TiB 474 MiB 14 GiB
1.7 TiB 76.37 1.04 98 up osd.8
16 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7
TiB 783 MiB 15 GiB
1.6 TiB 78.39 1.06 100 up osd.16
31 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.6
TiB 441 MiB 14 GiB
1.6 TiB 77.70 1.05 91 up osd.31
52 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5
TiB 939 MiB 14 GiB
1.7 TiB 76.29 1.04 102 up osd.52
53 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.4
TiB 848 MiB 18 GiB
1.9 TiB 74.30 1.01 98 up osd.53
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6
TiB 1.0 GiB 16 GiB
1.7 TiB 76.99 1.05 106 up osd.54
55 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5
TiB 460 MiB 15 GiB
1.8 TiB 75.46 1.02 105 up osd.55
-10 51.28331 - 51 TiB 37 TiB 37
TiB 3.8 GiB 96 GiB
14 TiB 72.77 0.99 - host s3db9
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 846 MiB 13 GiB
2.1 TiB 71.16 0.97 104 up osd.56
57 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6
TiB 513 MiB 15 GiB
1.7 TiB 76.53 1.04 96 up osd.57
58 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 604 MiB 13 GiB
2.1 TiB 71.23 0.97 98 up osd.58
59 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 414 MiB 13 GiB
2.2 TiB 70.03 0.95 88 up osd.59
60 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5
TiB 227 MiB 14 GiB
1.8 TiB 75.54 1.03 97 up osd.60
61 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 456 MiB 13 GiB
2.2 TiB 70.01 0.95 95 up osd.61
62 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.4
TiB 843 MiB 14 GiB
1.8 TiB 74.93 1.02 110 up osd.62
TOTAL 674 TiB 496 TiB 468
TiB 97 GiB 1.2 TiB
177 TiB 73.67
MIN/MAX VAR: 0.87/1.14 STDDEV: 4.22
Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>
> OK thanks. Indeed "prepared 0/10 changes" means it thinks things are
balanced.
> Could you again share the full ceph osd df
tree?
>
> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens <bb(a)kervyn.de> wrote:
> >
> > Hi Dan,
> >
> > I've set the autoscaler to warn, but it actually does not warn for
now. So not touching it for now.
> >
> > this is what the log says in minute intervals:
> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config
key:
mgr/balancer/active
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/sleep_interval
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/begin_time
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/end_time
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/begin_weekday
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/end_weekday
> > 2021-03-15 13:51:00.971 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/pool_ids
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] Optimize plan
auto_2021-03-15_13:51:00
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/mode
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] Mode upmap, max
misplaced 0.050000
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] do_upmap
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/upmap_max_iterations
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/upmap_max_deviation
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] pools
['eu-msg-1.rgw.data.root',
'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys',
'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log',
'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index',
'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control',
'.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control',
'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3',
'default.rgw.log', 'eu-central-1.rgw.meta',
'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
> > 2021-03-15 13:51:01.224 7f307d5fd700 4
mgr[balancer] prepared 0/10
changes
> >
> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >>
> >> I suggest to just disable the autoscaler until your balancing is
understood.
> >>
> >> What does your active mgr log say (with debug_mgr 4/5), grep balancer
> >> /var/log/ceph/ceph-mgr.*.log
> >>
> >> -- Dan
> >>
> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens <bb(a)kervyn.de> wrote:
> >> >
> >> > Hi,
> >> > this unfortunally did not solve my problem. I still have some OSDs
that fill up to 85%
> >> >
> >> > According to the logging, the autoscaler might want to add more
PGs to one Bucken and reduce almost all other buckets to 32.
> >> > 2021-03-15 12:19:58.825
7f307f601700 4 mgr[pg_autoscaler] Pool
'eu-central-1.rgw.buckets.data'
root_id -1 using 0.705080476146 of space,
bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
> >> >
> >> > Why the balancing does not happen is still nebulous to me.
> >> >
> >> >
> >> >
> >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >> >>
> >> >> OK
> >> >> Btw, you might need to fail to a new mgr... I'm not sure if the
current active will read that new config.
> >> >>
> >> >> .. dan
> >> >>
> >> >>
> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <bb(a)kervyn.de>
wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> ok thanks. I just changed the value and rewighted everything
back to 1. Now I let it sync the weekend and check how it will be on monday.
> >> >>> We tried to have the
systems total storage balanced as possible.
New systems will be with 8TB disks but
for the exiting ones we added 16TB
to offset the 4TB disks and we needed a lot of storage fast, because of a
DC move. If you have any recommendations I would be happy to hear them.
> >> >>>
> >> >>> Cheers
> >> >>> Boris
> >> >>>
> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster
<
dan(a)vanderster.com>gt;:
> >> >>>>
> >> >>>> Thanks.
> >> >>>>
> >> >>>> Decreasing the max deviation to 2 or 1 should help in your
case. This option controls when the balancer stops trying to move PGs
around -- by default it stops when the deviation from the mean is 5. Yes
this is too large IMO -- all of our clusters have this set to 1.
> >> >>>>
> >> >>>> And given that you have some OSDs with more than 200 PGs,
you
definitely shouldn't increase the num PGs.
> >> >>>>
> >> >>>> But anyway with your mixed device sizes it might be
challenging
to make a perfectly uniform distribution. Give it a try with 1 though,
and
let us know how it goes.
> >> >>>>
> >> >>>> .. Dan
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
> >> >>>>>
> >> >>>>> Hi Dan,
> >> >>>>>
> >> >>>>> upmap_max_deviation is default (5) in our cluster. Is 1
the
recommended deviation?
> >> >>>>>
> >> >>>>> I added the whole ceph osd df tree, (I need to remove
some
OSDs and readd them as bluestore with SSD, so 69, 73 and 82 are a bit off
now. I also reweighted to try to get the %USE mitigated).
> >> >>>>>
> >> >>>>> I will increase the mgr debugging to see what is the
problem.
> >> >>>>>
> >> >>>>> [root@s3db1 ~]# ceph osd df tree
> >> >>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA
OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
> >> >>>>> -1
673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB
1.2 TiB 168 TiB 74.57 1.00 -
root default
> >> >>>>> -2
58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB
38 GiB 22 TiB 49.82 0.67 -
host s3db1
> >> >>>>> 23 hdd
14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB
4.4 GiB 13 TiB 12.50 0.17 101
up osd.23
> >> >>>>> 69 hdd
14.55269 0 0 B 0 B 0 B 0 B
0 B 0 B 0 0 11
up osd.69
> >> >>>>> 73 hdd
14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB
33 GiB 4.2 TiB 71.15 0.95 107
up osd.73
> >> >>>>> 79 hdd
3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB
0 B 747 GiB 79.94 1.07 52
up osd.79
> >> >>>>> 80 hdd
3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB
0 B 1.0 TiB 71.61 0.96 58
up osd.80
> >> >>>>> 81 hdd
3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB
0 B 1.5 TiB 60.07 0.81 55
up osd.81
> >> >>>>> 82 hdd
3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB
0 B 1.7 TiB 52.68 0.71 30
up osd.82
> >> >>>>> -11
50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB
100 GiB 13 TiB 74.69 1.00 -
host s3db10
> >> >>>>> 63 hdd
7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB
14 GiB 1.7 TiB 76.04 1.02 92
up osd.63
> >> >>>>> 64 hdd
7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB
15 GiB 1.8 TiB 75.54 1.01 101
up osd.64
> >> >>>>> 65 hdd
7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB
14 GiB 2.0 TiB 73.17 0.98 105
up osd.65
> >> >>>>> 66 hdd
7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB
15 GiB 1.4 TiB 80.38 1.08 98
up osd.66
> >> >>>>> 67 hdd
7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB
14 GiB 2.2 TiB 70.10 0.94 100
up osd.67
> >> >>>>> 68 hdd
7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB
13 GiB 2.0 TiB 72.88 0.98 107
up osd.68
> >> >>>>> 70 hdd
7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB
14 GiB 1.8 TiB 74.73 1.00 102
up osd.70
> >> >>>>> -12
50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB
99 GiB 12 TiB 77.24 1.04 -
host s3db11
> >> >>>>> 46 hdd
7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB
15 GiB 1.5 TiB 78.91 1.06 97
up osd.46
> >> >>>>> 47 hdd
7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB
13 GiB 2.1 TiB 71.47 0.96 96
up osd.47
> >> >>>>> 48 hdd
7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB
15 GiB 1.2 TiB 83.46 1.12 109
up osd.48
> >> >>>>> 49 hdd
7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB
15 GiB 1.5 TiB 78.96 1.06 98
up osd.49
> >> >>>>> 50 hdd
7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB
15 GiB 1.4 TiB 80.40 1.08 102
up osd.50
> >> >>>>> 51 hdd
7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB
15 GiB 1.3 TiB 81.70 1.10 110
up osd.51
> >> >>>>> 72 hdd
7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB
12 GiB 2.5 TiB 65.82 0.88 89
up osd.72
> >> >>>>> -37
58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB
124 GiB 12 TiB 79.04 1.06 -
host s3db12
> >> >>>>> 19 hdd
3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB
8.2 GiB 559 GiB 85.18 1.14 55
up osd.19
> >> >>>>> 71 hdd
3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB
7.8 GiB 825 GiB 78.14 1.05 50
up osd.71
> >> >>>>> 75 hdd
3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB
8.3 GiB 555 GiB 85.29 1.14 57
up osd.75
> >> >>>>> 76 hdd
3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB
9.3 GiB 501 GiB 86.73 1.16 50
up osd.76
> >> >>>>> 77 hdd
14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB
30 GiB 3.6 TiB 75.57 1.01 202
up osd.77
> >> >>>>> 78 hdd
14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB
30 GiB 3.4 TiB 76.65 1.03 196
up osd.78
> >> >>>>> 83 hdd
14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB
31 GiB 2.9 TiB 80.04 1.07 223
up osd.83
> >> >>>>> -3
58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB
91 GiB 16 TiB 73.15 0.98 -
host s3db2
> >> >>>>> 1 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB
38 GiB 3.6 TiB 75.52 1.01 194
up osd.1
> >> >>>>> 3 hdd
3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB
0 B 1.4 TiB 60.94 0.82 52
up osd.3
> >> >>>>> 4 hdd
3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB
0 B 401 GiB 89.23 1.20 53
up osd.4
> >> >>>>> 5 hdd
3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB
0 B 1.3 TiB 62.88 0.84 51
up osd.5
> >> >>>>> 6 hdd
3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB
0 B 1.7 TiB 54.51 0.73 47
up osd.6
> >> >>>>> 7 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB
26 GiB 3.8 TiB 73.90 0.99 185
up osd.7
> >> >>>>> 74 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB
27 GiB 3.5 TiB 76.27 1.02 208
up osd.74
> >> >>>>> -4
58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB
86 GiB 15 TiB 74.05 0.99 -
host s3db3
> >> >>>>> 2 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB
26 GiB 4.0 TiB 72.78 0.98 203
up osd.2
> >> >>>>> 9 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB
33 GiB 3.6 TiB 75.62 1.01 189
up osd.9
> >> >>>>> 10 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB
28 GiB 3.5 TiB 76.34 1.02 211
up osd.10
> >> >>>>> 12 hdd
3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB
0 B 1.2 TiB 66.22 0.89 44
up osd.12
> >> >>>>> 13 hdd
3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB
0 B 720 GiB 80.66 1.08 66
up osd.13
> >> >>>>> 14 hdd
3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB
0 B 552 GiB 85.18 1.14 60
up osd.14
> >> >>>>> 15 hdd
3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB
0 B 1.7 TiB 53.72 0.72 44
up osd.15
> >> >>>>> -5
58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB
99 GiB 14 TiB 76.37 1.02 -
host s3db4
> >> >>>>> 11 hdd
14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB
28 GiB 2.8 TiB 81.15 1.09 205
up osd.11
> >> >>>>> 17 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB
27 GiB 3.6 TiB 75.38 1.01 211
up osd.17
> >> >>>>> 18 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB
44 GiB 4.0 TiB 72.86 0.98 188
up osd.18
> >> >>>>> 20 hdd
3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB
0 B 796 GiB 78.63 1.05 66
up osd.20
> >> >>>>> 21 hdd
3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB
0 B 1.1 TiB 70.32 0.94 47
up osd.21
> >> >>>>> 22 hdd
3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB
0 B 802 GiB 78.47 1.05 58
up osd.22
> >> >>>>> 24 hdd
3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB
0 B 856 GiB 77.01 1.03 47
up osd.24
> >> >>>>> -6
58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB
111 GiB 15 TiB 75.22 1.01 -
host s3db5
> >> >>>>> 0 hdd
3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB
6.6 GiB 1.3 TiB 65.00 0.87 48
up osd.0
> >> >>>>> 25 hdd
3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB
6.6 GiB 1.4 TiB 63.86 0.86 41
up osd.25
> >> >>>>> 26 hdd
3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB
7.6 GiB 862 GiB 77.47 1.04 48
up osd.26
> >> >>>>> 27 hdd
3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB
6.1 GiB 1.5 TiB 61.00 0.82 48
up osd.27
> >> >>>>> 28 hdd
14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB
30 GiB 2.8 TiB 81.19 1.09 203
up osd.28
> >> >>>>> 29 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB
26 GiB 3.8 TiB 73.95 0.99 200
up osd.29
> >> >>>>> 30 hdd
14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB
28 GiB 3.1 TiB 79.07 1.06 207
up osd.30
> >> >>>>> -7
58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB
122 GiB 14 TiB 75.41 1.01 -
host s3db6
> >> >>>>> 32 hdd
3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB
8.2 GiB 622 GiB 83.74 1.12 65
up osd.32
> >> >>>>> 33 hdd
3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB
8.1 GiB 740 GiB 80.67 1.08 52
up osd.33
> >> >>>>> 34 hdd
3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB
7.7 GiB 877 GiB 77.08 1.03 52
up osd.34
> >> >>>>> 35 hdd
3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB
7.0 GiB 1.4 TiB 62.18 0.83 42
up osd.35
> >> >>>>> 36 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB
26 GiB 4.0 TiB 72.98 0.98 220
up osd.36
> >> >>>>> 37 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB
38 GiB 3.6 TiB 75.30 1.01 200
up osd.37
> >> >>>>> 38 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB
28 GiB 3.3 TiB 77.43 1.04 217
up osd.38
> >> >>>>> -8
58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB
116 GiB 12 TiB 78.98 1.06 -
host s3db7
> >> >>>>> 39 hdd
3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB
8.5 GiB 499 GiB 86.96 1.17 43
up osd.39
> >> >>>>> 40 hdd
3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB
7.0 GiB 1.2 TiB 68.33 0.92 39
up osd.40
> >> >>>>> 41 hdd
3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB
7.9 GiB 732 GiB 80.86 1.08 64
up osd.41
> >> >>>>> 42 hdd
3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB
7.0 GiB 1.2 TiB 67.97 0.91 50
up osd.42
> >> >>>>> 43 hdd
14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB
28 GiB 2.9 TiB 80.32 1.08 213
up osd.43
> >> >>>>> 44 hdd
14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB
28 GiB 3.1 TiB 78.59 1.05 198
up osd.44
> >> >>>>> 45 hdd
14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB
30 GiB 2.8 TiB 81.05 1.09 214
up osd.45
> >> >>>>> -9
51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB
108 GiB 10 TiB 79.75 1.07 -
host s3db8
> >> >>>>> 8 hdd
7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB
15 GiB 1.5 TiB 79.68 1.07 99
up osd.8
> >> >>>>> 16 hdd
7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB
15 GiB 1.4 TiB 80.25 1.08 97
up osd.16
> >> >>>>> 31 hdd
7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB
14 GiB 1.8 TiB 75.36 1.01 87
up osd.31
> >> >>>>> 52 hdd
7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB
15 GiB 1.6 TiB 78.19 1.05 113
up osd.52
> >> >>>>> 53 hdd
7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB
18 GiB 1.1 TiB 84.46 1.13 109
up osd.53
> >> >>>>> 54 hdd
7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB
15 GiB 1.7 TiB 76.73 1.03 115
up osd.54
> >> >>>>> 55 hdd
7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB
16 GiB 1.2 TiB 83.56 1.12 122
up osd.55
> >> >>>>> -10
51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB
100 GiB 12 TiB 76.05 1.02 -
host s3db9
> >> >>>>> 56 hdd
7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB
13 GiB 2.1 TiB 71.06 0.95 105
up osd.56
> >> >>>>> 57 hdd
7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB
16 GiB 1.2 TiB 83.17 1.12 102
up osd.57
> >> >>>>> 58 hdd
7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB
15 GiB 1.4 TiB 81.56 1.09 105
up osd.58
> >> >>>>> 59 hdd
7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB
15 GiB 1.4 TiB 80.64 1.08 94
up osd.59
> >> >>>>> 60 hdd
7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB
14 GiB 2.0 TiB 73.25 0.98 101
up osd.60
> >> >>>>> 61 hdd
7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB
12 GiB 2.5 TiB 65.84 0.88 103
up osd.61
> >> >>>>> 62 hdd
7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB
15 GiB 1.7 TiB 76.83 1.03 126
up osd.62
> >> >>>>>
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB
1.2 TiB 173 TiB 74.57
> >> >>>>> MIN/MAX VAR:
0.17/1.20 STDDEV: 10.25
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der
Ster <
dan(a)vanderster.com>gt;:
> >> >>>>>>
> >> >>>>>> No, increasing num PGs won't help
substantially.
> >> >>>>>>
> >> >>>>>> Can you share the entire output of ceph osd df tree
?
> >> >>>>>>
> >> >>>>>> Did you already set
> >> >>>>>>
> >> >>>>>> ceph config set mgr
mgr/balancer/upmap_max_deviation 1
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> ??
> >> >>>>>> And I recommend debug_mgr 4/5 so you can see some
basic upmap
balancer logging.
> >> >>>>>>
> >> >>>>>> .. Dan
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
> >> >>>>>>>
> >> >>>>>>> Hello people,
> >> >>>>>>>
> >> >>>>>>> I am still struggeling with the balancer
> >> >>>>>>> (
https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
> >> >>>>>>> Now
I've read some more and might think that I do not have
enough PGs.
> >> >>>>>>>
Currently I have 84OSDs and 1024PGs for the main pool (3008
total). I
> >> >>>>>>> have
the autoscaler enabled, but I doesn't tell me to
increase the
> >> >>>>>>> PGs.
> >> >>>>>>>
> >> >>>>>>> What do you think?
> >> >>>>>>>
> >> >>>>>>> --
> >> >>>>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal
abweichend
> >> >>>>>>> im
groüen Saal.
> >> >>>>>>>
_______________________________________________
> >> >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
> >> >>>>>>> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> --
> >> >>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft
sich diesmal
abweichend im groüen Saal.
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal
abweichend im groüen Saal.
> >> >
> >> >
> >> >
> >> > --
> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
> >
> >
> >
> > --
> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
23 Mar
23 Mar
5:07 p.m.
Ok, I should have listened to you :)
In the last week we added more storage but the issue got worse instead.
Today I realized that the PGs were up to 90GB (bytes column in ceph pg ls
said 95705749636), and the balance kept mentioning the 2048 PGs for this
pool. We were at 72% utilization (ceph osd df tree, first line) for our
cluster and I increased the PGs to 2048.
Now I am in a world of trouble.
The space in the cluster went down, I am at 45% misplaced objects, and we
already added 20x4TB disks just to not go down completly.
The utilization is still going up and the overall free space in the cluster
seems to go down. This is what my ceph status looks like and now I really
need help to get that thing back to normal:
[root@s3db1 ~]# ceph status
cluster:
id: dca79fff-ffd0-58f4-1cff-82a2feea05f4
health: HEALTH_WARN
4 backfillfull osd(s)
17 nearfull osd(s)
37 pool(s) backfillfull
13 large omap objects
Low space hindering backfill (add storage if this doesn't
resolve itself): 570 pgs backfill_toofull
services:
mon: 3 daemons, quorum ceph-s3-mon1,ceph-s3-mon2,ceph-s3-mon3 (age 44m)
mgr: ceph-mgr2(active, since 15m), standbys: ceph-mgr3, ceph-mgr1
mds: 3 up:standby
osd: 110 osds: 110 up (since 28m), 110 in (since 28m); 1535 remapped pgs
rgw: 3 daemons active (eu-central-1, eu-msg-1, eu-secure-1)
task status:
data:
pools: 37 pools, 4032 pgs
objects: 116.23M objects, 182 TiB
usage: 589 TiB used, 206 TiB / 795 TiB avail
pgs: 160918554/348689415 objects misplaced (46.150%)
2497 active+clean
779 active+remapped+backfill_wait
538 active+remapped+backfill_wait+backfill_toofull
186 active+remapped+backfilling
32 active+remapped+backfill_toofull
io:
client: 27 MiB/s rd, 69 MiB/s wr, 497 op/s rd, 153 op/s wr
recovery: 1.5 GiB/s, 922 objects/s
Am Di., 16. März 2021 um 09:34 Uhr schrieb Boris Behrens <bb(a)kervyn.de>de>:
Hi Dan,
my EC profile look very "default" to me.
[root@s3db1 ~]# ceph osd erasure-code-profile ls
default
[root@s3db1 ~]# ceph osd erasure-code-profile get default
k=2
m=1
plugin=jerasure
technique=reed_sol_van
I don't understand the ouput, but the balancing get worse over night:
[root@s3db1 ~]# ceph-scripts/tools/ceph-pool-pg-distribution 11
Searching for PGs in pools: ['11']
Summary: 1024 PGs on 84 osds
Num OSDs with X PGs:
15: 8
16: 7
17: 6
18: 10
19: 1
32: 10
33: 4
34: 6
35: 8
65: 5
66: 5
67: 4
68: 10
[root@s3db1 ~]# ceph-scripts/tools/ceph-pg-histogram --normalize --pool=11
# NumSamples = 84; Min = 4.12; Max = 5.09
# Mean = 4.553355; Variance = 0.052415; SD = 0.228942; Median 4.561608
# each ∎ represents a count of 1
4.1244 - 4.2205 [ 8]: ∎∎∎∎∎∎∎∎
4.2205 - 4.3166 [ 6]: ∎∎∎∎∎∎
4.3166 - 4.4127 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
4.4127 - 4.5087 [ 10]: ∎∎∎∎∎∎∎∎∎∎
4.5087 - 4.6048 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
4.6048 - 4.7009 [ 19]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
4.7009 - 4.7970 [ 6]: ∎∎∎∎∎∎
4.7970 - 4.8931 [ 8]: ∎∎∎∎∎∎∎∎
4.8931 - 4.9892 [ 4]: ∎∎∎∎
4.9892 - 5.0852 [ 1]: ∎
[root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail
14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 724 GiB 19 GiB 0 B 724
GiB 80.56 1.07 56 up osd.14
19 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 466 MiB 7.9 GiB 708
GiB 81.25 1.08 53 up osd.19
4 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 698 GiB 703 MiB 0 B 698
GiB 81.27 1.08 48 up osd.4
24 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 695 GiB 640 MiB 0 B 695
GiB 81.34 1.08 46 up osd.24
75 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 440 MiB 8.1 GiB 704
GiB 81.35 1.08 48 up osd.75
71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 7.5 MiB 8.0 GiB 663
GiB 82.44 1.09 47 up osd.71
76 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 251 MiB 9.0 GiB 617
GiB 83.65 1.11 50 up osd.76
33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 399 MiB 8.1 GiB 618
GiB 83.85 1.11 55 up osd.33
35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 317 MiB 8.8 GiB 617
GiB 83.87 1.11 50 up osd.35
34 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 451 MiB 8.7 GiB 545
GiB 85.75 1.14 54 up osd.34
Am Mo., 15. März 2021 um 17:23 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Hi,
How wide are your EC profiles? If they are really wide, you might be
reaching the limits of what is physically possible. Also, I'm not sure
that upmap in 14.2.11 is very smart about *improving* existing upmap
rules for a given PG, in the case that a PG already has an upmap-items
entry but it would help the distribution to add more mapping pairs to
that entry. What this means, is that it might sometimes be useful to
randomly remove some upmap entries and see if the balancer does a
better job when it replaces them.
But before you do that, I re-remembered that looking at the total PG
numbers is not useful -- you need to check the PGs per OSD for the
eu-central-1.rgw.buckets.data pool only.
We have a couple tools that can help with this:
1. To see the PGs per OSD for a given pool:
https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis…
E.g.: ./ceph-pool-pg-distribution 11 # to see the distribution of
your eu-central-1.rgw.buckets.data pool.
The output looks like this on my well balanced clusters:
# ceph-scripts/tools/ceph-pool-pg-distribution 15
Searching for PGs in pools: ['15']
Summary: 256 pgs on 56 osds
Num OSDs with X PGs:
13: 16
14: 40
You should expect a trimodal for your cluster.
2. You can also use another script from that repo to see the PGs per
OSD normalized to crush weight:
ceph-scripts/tools/ceph-pg-histogram --normalize --pool=15
This might explain what is going wrong.
Cheers, Dan
On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Absolutly:
[root@s3db1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE
VAR PGS STATUS TYPE NAME
-1 673.54224 - 674 TiB 496 TiB 468
TiB 97 GiB 1.2 TiB
177 TiB 73.67 1.00 - root default
-2 58.30331 - 58 TiB 42 TiB 38
TiB 9.2 GiB 99 GiB
16 TiB 72.88 0.99 - host s3db1
23 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 714 MiB 25 GiB
3.7 TiB 74.87 1.02 194 up osd.23
69 hdd 14.55269 1.00000 15 TiB 11 TiB 11
TiB 1.6 GiB 40 GiB
3.4 TiB 76.32 1.04 199 up osd.69
73 hdd 14.55269 1.00000 15 TiB 11 TiB 11
TiB 1.3 GiB 34 GiB
3.8 TiB 74.15 1.01 203 up osd.73
79 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3
TiB 1.8 GiB 0 B
1.3 TiB 65.44 0.89 47 up osd.79
80 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3
TiB 2.2 GiB 0 B
1.3 TiB 65.34 0.89 48 up osd.80
81 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3
TiB 1.1 GiB 0 B
1.3 TiB 65.38 0.89 47 up osd.81
82 hdd 3.63689 1.00000 3.6 TiB 2.5 TiB 1.1
TiB 619 MiB 0 B
1.1 TiB 68.46 0.93 41 up osd.82
-11 50.94173 - 51 TiB 37 TiB 37
TiB 3.5 GiB 98 GiB
14 TiB 71.90 0.98 - host s3db10
63 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3
TiB 647 MiB 14 GiB
2.0 TiB 72.72 0.99 94 up osd.63
64 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.2
TiB 668 MiB 14 GiB
2.0 TiB 72.23 0.98 93 up osd.64
65 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 227 MiB 14 GiB
2.1 TiB 71.16 0.97 100 up osd.65
66 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4
TiB 313 MiB 14 GiB
1.9 TiB 74.25 1.01 92 up osd.66
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 584 MiB 14 GiB
2.1 TiB 70.63 0.96 96 up osd.67
68 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 720 MiB 14 GiB
2.1 TiB 71.72 0.97 101 up osd.68
70 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 425 MiB 14 GiB
2.1 TiB 70.59 0.96 97 up osd.70
-12 50.99052 - 51 TiB 38 TiB 37
TiB 2.1 GiB 97 GiB
13 TiB 73.77 1.00 - host s3db11
46 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.6
TiB 229 MiB 14 GiB
1.7 TiB 77.05 1.05 97 up osd.46
47 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 159 MiB 13 GiB
2.2 TiB 70.00 0.95 89 up osd.47
48 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 279 MiB 14 GiB
2.1 TiB 71.82 0.97 98 up osd.48
49 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.4
TiB 276 MiB 14 GiB
1.8 TiB 74.90 1.02 95 up osd.49
50 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 336 MiB 14 GiB
2.0 TiB 72.13 0.98 93 up osd.50
51 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.6
TiB 728 MiB 15 GiB
1.6 TiB 77.76 1.06 98 up osd.51
72 hdd 7.32619 1.00000 7.3 TiB 5.3 TiB 5.3
TiB 147 MiB 13 GiB
2.0 TiB 72.75 0.99 95 up osd.72
-37 58.55478 - 59 TiB 44 TiB 44
TiB 4.4 GiB 122 GiB
15 TiB 75.20 1.02 - host s3db12
19 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9
TiB 454 MiB 8.2 GiB
780 GiB 79.35 1.08 53 up osd.19
71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9
TiB 7.1 MiB 8.0 GiB
734 GiB 80.56 1.09 47 up osd.71
75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9
TiB 439 MiB 8.2 GiB
777 GiB 79.43 1.08 48 up osd.75
76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0
TiB 241 MiB 8.9 GiB
688 GiB 81.77 1.11 50 up osd.76
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11
TiB 880 MiB 30 GiB
3.6 TiB 75.44 1.02 201 up osd.77
78 hdd 14.60159 1.00000 15 TiB 10 TiB 10
TiB 1015 MiB 28 GiB
4.2 TiB 71.26 0.97 193 up osd.78
83 hdd 14.60159 1.00000 15 TiB 11 TiB 11
TiB 1.4 GiB 30 GiB
3.8 TiB 73.76 1.00 203 up osd.83
-3 58.49872 - 58 TiB 42 TiB 36
TiB 8.2 GiB 89 GiB
17 TiB 71.71 0.97 - host s3db2
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 3.2 GiB 37 GiB
3.7 TiB 74.58 1.01 196 up osd.1
3 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 566 MiB 0 B
1.3 TiB 64.11 0.87 50 up osd.3
4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 771
GiB 695 MiB 0 B
771 GiB 79.30 1.08 48 up osd.4
5 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2
TiB 482 MiB 0 B
1.2 TiB 66.51 0.90 49 up osd.5
6 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 1.8 GiB 0 B
1.3 TiB 64.00 0.87 42 up osd.6
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 639 MiB 26 GiB
4.0 TiB 72.44 0.98 192 up osd.7
74 hdd 14.65039 1.00000 15 TiB 10 TiB 10
TiB 907 MiB 26 GiB
4.2 TiB 71.32 0.97 193 up osd.74
-4 58.49872 - 58 TiB 43 TiB 36
TiB 34 GiB 85 GiB
16 TiB 72.69 0.99 - host s3db3
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 980 MiB 26 GiB
3.8 TiB 74.36 1.01 203 up osd.2
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 8.4 GiB 33 GiB
3.9 TiB 73.51 1.00 186 up osd.9
10 hdd 14.65039 1.00000 15 TiB 10 TiB 10
TiB 650 MiB 26 GiB
4.2 TiB 71.64 0.97 201 up osd.10
12 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 754 MiB 0 B
1.3 TiB 64.17 0.87 44 up osd.12
13 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 813
GiB 2.4 GiB 0 B
813 GiB 78.17 1.06 58 up osd.13
14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 797
GiB 19 GiB 0 B
797 GiB 78.60 1.07 56 up osd.14
15 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3
TiB 2.2 GiB 0 B
1.3 TiB 63.96 0.87 41 up osd.15
-5 58.49872 - 58 TiB 43 TiB 36
TiB 6.7 GiB 97 GiB
15 TiB 74.04 1.01 - host s3db4
11 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 940 MiB 26 GiB
4.0 TiB 72.49 0.98 196 up osd.11
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1022 MiB 26 GiB
3.6 TiB 75.23 1.02 204 up osd.17
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 945 MiB 45 GiB
3.8 TiB 74.16 1.01 193 up osd.18
20 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1020
GiB 596 MiB 0 B
1020 GiB 72.62 0.99 57 up osd.20
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023
GiB 1.9 GiB 0 B
1023 GiB 72.54 0.98 41 up osd.21
22 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023
GiB 797 MiB 0 B
1023 GiB 72.54 0.98 53 up osd.22
24 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 766
GiB 618 MiB 0 B
766 GiB 79.42 1.08 46 up osd.24
-6 58.89636 - 59 TiB 43 TiB 43
TiB 3.0 GiB 108 GiB
16 TiB 73.40 1.00 - host s3db5
0 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6
TiB 92 MiB 7.2 GiB
1.1 TiB 71.16 0.97 45 up osd.0
25 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6
TiB 2.4 MiB 7.3 GiB
1.1 TiB 71.23 0.97 41 up osd.25
26 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7
TiB 181 MiB 7.6 GiB
935 GiB 75.57 1.03 45 up osd.26
27 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6
TiB 5.1 MiB 7.0 GiB
1.1 TiB 71.20 0.97 47 up osd.27
28 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 977 MiB 26 GiB
3.8 TiB 73.85 1.00 197 up osd.28
29 hdd 14.65039 1.00000 15 TiB 11 TiB 10
TiB 872 MiB 26 GiB
4.1 TiB 71.98 0.98 196 up osd.29
30 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 943 MiB 27 GiB
3.6 TiB 75.51 1.03 202 up osd.30
-7 58.89636 - 59 TiB 44 TiB 43
TiB 13 GiB 122 GiB
15 TiB 74.97 1.02 - host s3db6
32 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7
TiB 27 MiB 7.6 GiB
940 GiB 75.42 1.02 55 up osd.32
33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0
TiB 376 MiB 8.2 GiB
691 GiB 81.94 1.11 55 up osd.33
34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0
TiB 450 MiB 8.5 GiB
620 GiB 83.79 1.14 54 up osd.34
35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0
TiB 316 MiB 8.4 GiB
690 GiB 81.98 1.11 50 up osd.35
36 hdd 14.65039 1.00000 15 TiB 11 TiB 10
TiB 489 MiB 25 GiB
4.1 TiB 71.69 0.97 208 up osd.36
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 11 GiB 38 GiB
4.0 TiB 72.41 0.98 195 up osd.37
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.1 GiB 26 GiB
3.7 TiB 74.88 1.02 204 up osd.38
-8 58.89636 - 59 TiB 44 TiB 43
TiB 3.8 GiB 111 GiB
15 TiB 74.16 1.01 - host s3db7
39 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7
TiB 19 MiB 7.5 GiB
936 GiB 75.54 1.03 39 up osd.39
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5
TiB 144 MiB 7.1 GiB
1.1 TiB 69.87 0.95 39 up osd.40
41 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.7
TiB 219 MiB 7.6 GiB
1011 GiB 73.57 1.00 55 up osd.41
42 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5
TiB 593 MiB 7.1 GiB
1.1 TiB 70.02 0.95 47 up osd.42
43 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 500 MiB 27 GiB
3.7 TiB 74.67 1.01 204 up osd.43
44 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.1 GiB 27 GiB
3.7 TiB 74.62 1.01 193 up osd.44
45 hdd 14.65039 1.00000 15 TiB 11 TiB 11
TiB 1.2 GiB 29 GiB
3.6 TiB 75.16 1.02 204 up osd.45
-9 51.28331 - 51 TiB 39 TiB 39
TiB 4.9 GiB 107 GiB
12 TiB 76.50 1.04 - host s3db8
8 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5
TiB 474 MiB 14 GiB
1.7 TiB 76.37 1.04 98 up osd.8
16 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7
TiB 783 MiB 15 GiB
1.6 TiB 78.39 1.06 100 up osd.16
31 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.6
TiB 441 MiB 14 GiB
1.6 TiB 77.70 1.05 91 up osd.31
52 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5
TiB 939 MiB 14 GiB
1.7 TiB 76.29 1.04 102 up osd.52
53 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.4
TiB 848 MiB 18 GiB
1.9 TiB 74.30 1.01 98 up osd.53
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6
TiB 1.0 GiB 16 GiB
1.7 TiB 76.99 1.05 106 up osd.54
55 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5
TiB 460 MiB 15 GiB
1.8 TiB 75.46 1.02 105 up osd.55
-10 51.28331 - 51 TiB 37 TiB 37
TiB 3.8 GiB 96 GiB
14 TiB 72.77 0.99 - host s3db9
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 846 MiB 13 GiB
2.1 TiB 71.16 0.97 104 up osd.56
57 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6
TiB 513 MiB 15 GiB
1.7 TiB 76.53 1.04 96 up osd.57
58 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2
TiB 604 MiB 13 GiB
2.1 TiB 71.23 0.97 98 up osd.58
59 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 414 MiB 13 GiB
2.2 TiB 70.03 0.95 88 up osd.59
60 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5
TiB 227 MiB 14 GiB
1.8 TiB 75.54 1.03 97 up osd.60
61 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1
TiB 456 MiB 13 GiB
2.2 TiB 70.01 0.95 95 up osd.61
62 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.4
TiB 843 MiB 14 GiB
1.8 TiB 74.93 1.02 110 up osd.62
TOTAL 674 TiB 496 TiB 468
TiB 97 GiB 1.2 TiB
177 TiB 73.67
MIN/MAX VAR: 0.87/1.14 STDDEV: 4.22
Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>
> OK thanks. Indeed "prepared 0/10 changes" means it thinks things are
balanced.
> Could you again share the full ceph osd df
tree?
>
> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens <bb(a)kervyn.de> wrote:
> >
> > Hi Dan,
> >
> > I've set the autoscaler to warn, but it actually does not warn for
now. So not touching it for now.
> >
> > this is what the log says in minute intervals:
> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config
key:
mgr/balancer/active
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/sleep_interval
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/begin_time
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/end_time
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/begin_weekday
> > 2021-03-15 13:51:00.970 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/end_weekday
> > 2021-03-15 13:51:00.971 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/pool_ids
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] Optimize plan
auto_2021-03-15_13:51:00
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/mode
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] Mode upmap,
max misplaced 0.050000
>> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] do_upmap
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/upmap_max_iterations
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr get_config get_config
key: mgr/balancer/upmap_max_deviation
> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] pools
['eu-msg-1.rgw.data.root',
'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys',
'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log',
'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index',
'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control',
'.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control',
'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3',
'default.rgw.log', 'eu-central-1.rgw.meta',
'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
> > 2021-03-15 13:51:01.224 7f307d5fd700 4
mgr[balancer] prepared 0/10
changes
> >
> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >>
> >> I suggest to just disable the autoscaler until your balancing is
understood.
> >>
> >> What does your active mgr log say (with debug_mgr 4/5), grep
balancer
> >> /var/log/ceph/ceph-mgr.*.log
> >>
> >> -- Dan
> >>
> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens <bb(a)kervyn.de> wrote:
> >> >
> >> > Hi,
> >> > this unfortunally did not solve my problem. I still have some
OSDs that fill up to 85%
> >> >
> >> > According to the logging, the autoscaler might want to add more
PGs to one Bucken and reduce almost all other buckets to 32.
> >> > 2021-03-15 12:19:58.825
7f307f601700 4 mgr[pg_autoscaler] Pool
'eu-central-1.rgw.buckets.data'
root_id -1 using 0.705080476146 of space,
bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
> >> >
> >> > Why the balancing does not happen is still nebulous to me.
> >> >
> >> >
> >> >
> >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >> >>
> >> >> OK
> >> >> Btw, you might need to fail to a new mgr... I'm not sure if the
current active will read that new config.
> >> >>
> >> >> .. dan
> >> >>
> >> >>
> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <bb(a)kervyn.de>
wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> ok thanks. I just changed the value and rewighted everything
back to 1. Now I let it sync the weekend and check how it will be on monday.
> >> >>> We tried to have the
systems total storage balanced as
possible. New systems will be with 8TB disks but
for the exiting ones we
added 16TB to offset the 4TB disks and we needed a lot of storage fast,
because of a DC move. If you have any recommendations I would be happy to
hear them.
> >> >>>
> >> >>> Cheers
> >> >>> Boris
> >> >>>
> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster
<
dan(a)vanderster.com>gt;:
> >> >>>>
> >> >>>> Thanks.
> >> >>>>
> >> >>>> Decreasing the max deviation to 2 or 1 should help in your
case. This option controls when the balancer stops trying to move PGs
around -- by default it stops when the deviation from the mean is 5. Yes
this is too large IMO -- all of our clusters have this set to 1.
> >> >>>>
> >> >>>> And given that you have some OSDs with more than 200 PGs,
you
definitely shouldn't increase the num PGs.
> >> >>>>
> >> >>>> But anyway with your mixed device sizes it might be
challenging to make a perfectly uniform distribution. Give it a try with 1
though, and let us know how it goes.
> >> >>>>
> >> >>>> .. Dan
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
> >> >>>>>
> >> >>>>> Hi Dan,
> >> >>>>>
> >> >>>>> upmap_max_deviation is default (5) in our cluster. Is 1
the
recommended deviation?
> >> >>>>>
> >> >>>>> I added the whole ceph osd df tree, (I need to remove
some
OSDs and readd them as bluestore with SSD, so 69, 73 and 82 are a bit off
now. I also reweighted to try to get the %USE mitigated).
> >> >>>>>
> >> >>>>> I will increase the mgr debugging to see what is the
problem.
> >> >>>>>
> >> >>>>> [root@s3db1 ~]# ceph osd df tree
> >> >>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA
OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
> >> >>>>> -1
673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB
1.2 TiB 168 TiB 74.57 1.00 -
root default
> >> >>>>> -2
58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB
38 GiB 22 TiB 49.82 0.67 -
host s3db1
> >> >>>>> 23 hdd
14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB
4.4 GiB 13 TiB 12.50 0.17 101
up osd.23
> >> >>>>> 69 hdd
14.55269 0 0 B 0 B 0 B 0 B
0 B 0 B 0 0 11
up osd.69
> >> >>>>> 73 hdd
14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB
33 GiB 4.2 TiB 71.15 0.95 107
up osd.73
> >> >>>>> 79 hdd
3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB
0 B 747 GiB 79.94 1.07 52
up osd.79
> >> >>>>> 80 hdd
3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB
0 B 1.0 TiB 71.61 0.96 58
up osd.80
> >> >>>>> 81 hdd
3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB
0 B 1.5 TiB 60.07 0.81 55
up osd.81
> >> >>>>> 82 hdd
3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB
0 B 1.7 TiB 52.68 0.71 30
up osd.82
> >> >>>>> -11
50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB
100 GiB 13 TiB 74.69 1.00 -
host s3db10
> >> >>>>> 63 hdd
7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB
14 GiB 1.7 TiB 76.04 1.02 92
up osd.63
> >> >>>>> 64 hdd
7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB
15 GiB 1.8 TiB 75.54 1.01 101
up osd.64
> >> >>>>> 65 hdd
7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB
14 GiB 2.0 TiB 73.17 0.98 105
up osd.65
> >> >>>>> 66 hdd
7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB
15 GiB 1.4 TiB 80.38 1.08 98
up osd.66
> >> >>>>> 67 hdd
7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB
14 GiB 2.2 TiB 70.10 0.94 100
up osd.67
> >> >>>>> 68 hdd
7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB
13 GiB 2.0 TiB 72.88 0.98 107
up osd.68
> >> >>>>> 70 hdd
7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB
14 GiB 1.8 TiB 74.73 1.00 102
up osd.70
> >> >>>>> -12
50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB
99 GiB 12 TiB 77.24 1.04 -
host s3db11
> >> >>>>> 46 hdd
7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB
15 GiB 1.5 TiB 78.91 1.06 97
up osd.46
> >> >>>>> 47 hdd
7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB
13 GiB 2.1 TiB 71.47 0.96 96
up osd.47
> >> >>>>> 48 hdd
7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB
15 GiB 1.2 TiB 83.46 1.12 109
up osd.48
> >> >>>>> 49 hdd
7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB
15 GiB 1.5 TiB 78.96 1.06 98
up osd.49
> >> >>>>> 50 hdd
7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB
15 GiB 1.4 TiB 80.40 1.08 102
up osd.50
> >> >>>>> 51 hdd
7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB
15 GiB 1.3 TiB 81.70 1.10 110
up osd.51
> >> >>>>> 72 hdd
7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB
12 GiB 2.5 TiB 65.82 0.88 89
up osd.72
> >> >>>>> -37
58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB
124 GiB 12 TiB 79.04 1.06 -
host s3db12
> >> >>>>> 19 hdd
3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB
8.2 GiB 559 GiB 85.18 1.14 55
up osd.19
> >> >>>>> 71 hdd
3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB
7.8 GiB 825 GiB 78.14 1.05 50
up osd.71
> >> >>>>> 75 hdd
3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB
8.3 GiB 555 GiB 85.29 1.14 57
up osd.75
> >> >>>>> 76 hdd
3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB
9.3 GiB 501 GiB 86.73 1.16 50
up osd.76
> >> >>>>> 77 hdd
14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB
30 GiB 3.6 TiB 75.57 1.01 202
up osd.77
> >> >>>>> 78 hdd
14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB
30 GiB 3.4 TiB 76.65 1.03 196
up osd.78
> >> >>>>> 83 hdd
14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB
31 GiB 2.9 TiB 80.04 1.07 223
up osd.83
> >> >>>>> -3
58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB
91 GiB 16 TiB 73.15 0.98 -
host s3db2
> >> >>>>> 1 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB
38 GiB 3.6 TiB 75.52 1.01 194
up osd.1
> >> >>>>> 3 hdd
3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB
0 B 1.4 TiB 60.94 0.82 52
up osd.3
> >> >>>>> 4 hdd
3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB
0 B 401 GiB 89.23 1.20 53
up osd.4
> >> >>>>> 5 hdd
3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB
0 B 1.3 TiB 62.88 0.84 51
up osd.5
> >> >>>>> 6 hdd
3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB
0 B 1.7 TiB 54.51 0.73 47
up osd.6
> >> >>>>> 7 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB
26 GiB 3.8 TiB 73.90 0.99 185
up osd.7
> >> >>>>> 74 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB
27 GiB 3.5 TiB 76.27 1.02 208
up osd.74
> >> >>>>> -4
58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB
86 GiB 15 TiB 74.05 0.99 -
host s3db3
> >> >>>>> 2 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB
26 GiB 4.0 TiB 72.78 0.98 203
up osd.2
> >> >>>>> 9 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB
33 GiB 3.6 TiB 75.62 1.01 189
up osd.9
> >> >>>>> 10 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB
28 GiB 3.5 TiB 76.34 1.02 211
up osd.10
> >> >>>>> 12 hdd
3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB
0 B 1.2 TiB 66.22 0.89 44
up osd.12
> >> >>>>> 13 hdd
3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB
0 B 720 GiB 80.66 1.08 66
up osd.13
> >> >>>>> 14 hdd
3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB
0 B 552 GiB 85.18 1.14 60
up osd.14
> >> >>>>> 15 hdd
3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB
0 B 1.7 TiB 53.72 0.72 44
up osd.15
> >> >>>>> -5
58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB
99 GiB 14 TiB 76.37 1.02 -
host s3db4
> >> >>>>> 11 hdd
14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB
28 GiB 2.8 TiB 81.15 1.09 205
up osd.11
> >> >>>>> 17 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB
27 GiB 3.6 TiB 75.38 1.01 211
up osd.17
> >> >>>>> 18 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB
44 GiB 4.0 TiB 72.86 0.98 188
up osd.18
> >> >>>>> 20 hdd
3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB
0 B 796 GiB 78.63 1.05 66
up osd.20
> >> >>>>> 21 hdd
3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB
0 B 1.1 TiB 70.32 0.94 47
up osd.21
> >> >>>>> 22 hdd
3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB
0 B 802 GiB 78.47 1.05 58
up osd.22
> >> >>>>> 24 hdd
3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB
0 B 856 GiB 77.01 1.03 47
up osd.24
> >> >>>>> -6
58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB
111 GiB 15 TiB 75.22 1.01 -
host s3db5
> >> >>>>> 0 hdd
3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB
6.6 GiB 1.3 TiB 65.00 0.87 48
up osd.0
> >> >>>>> 25 hdd
3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB
6.6 GiB 1.4 TiB 63.86 0.86 41
up osd.25
> >> >>>>> 26 hdd
3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB
7.6 GiB 862 GiB 77.47 1.04 48
up osd.26
> >> >>>>> 27 hdd
3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB
6.1 GiB 1.5 TiB 61.00 0.82 48
up osd.27
> >> >>>>> 28 hdd
14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB
30 GiB 2.8 TiB 81.19 1.09 203
up osd.28
> >> >>>>> 29 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB
26 GiB 3.8 TiB 73.95 0.99 200
up osd.29
> >> >>>>> 30 hdd
14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB
28 GiB 3.1 TiB 79.07 1.06 207
up osd.30
> >> >>>>> -7
58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB
122 GiB 14 TiB 75.41 1.01 -
host s3db6
> >> >>>>> 32 hdd
3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB
8.2 GiB 622 GiB 83.74 1.12 65
up osd.32
> >> >>>>> 33 hdd
3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB
8.1 GiB 740 GiB 80.67 1.08 52
up osd.33
> >> >>>>> 34 hdd
3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB
7.7 GiB 877 GiB 77.08 1.03 52
up osd.34
> >> >>>>> 35 hdd
3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB
7.0 GiB 1.4 TiB 62.18 0.83 42
up osd.35
> >> >>>>> 36 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB
26 GiB 4.0 TiB 72.98 0.98 220
up osd.36
> >> >>>>> 37 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB
38 GiB 3.6 TiB 75.30 1.01 200
up osd.37
> >> >>>>> 38 hdd
14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB
28 GiB 3.3 TiB 77.43 1.04 217
up osd.38
> >> >>>>> -8
58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB
116 GiB 12 TiB 78.98 1.06 -
host s3db7
> >> >>>>> 39 hdd
3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB
8.5 GiB 499 GiB 86.96 1.17 43
up osd.39
> >> >>>>> 40 hdd
3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB
7.0 GiB 1.2 TiB 68.33 0.92 39
up osd.40
> >> >>>>> 41 hdd
3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB
7.9 GiB 732 GiB 80.86 1.08 64
up osd.41
> >> >>>>> 42 hdd
3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB
7.0 GiB 1.2 TiB 67.97 0.91 50
up osd.42
> >> >>>>> 43 hdd
14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB
28 GiB 2.9 TiB 80.32 1.08 213
up osd.43
> >> >>>>> 44 hdd
14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB
28 GiB 3.1 TiB 78.59 1.05 198
up osd.44
> >> >>>>> 45 hdd
14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB
30 GiB 2.8 TiB 81.05 1.09 214
up osd.45
> >> >>>>> -9
51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB
108 GiB 10 TiB 79.75 1.07 -
host s3db8
> >> >>>>> 8 hdd
7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB
15 GiB 1.5 TiB 79.68 1.07 99
up osd.8
> >> >>>>> 16 hdd
7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB
15 GiB 1.4 TiB 80.25 1.08 97
up osd.16
> >> >>>>> 31 hdd
7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB
14 GiB 1.8 TiB 75.36 1.01 87
up osd.31
> >> >>>>> 52 hdd
7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB
15 GiB 1.6 TiB 78.19 1.05 113
up osd.52
> >> >>>>> 53 hdd
7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB
18 GiB 1.1 TiB 84.46 1.13 109
up osd.53
> >> >>>>> 54 hdd
7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB
15 GiB 1.7 TiB 76.73 1.03 115
up osd.54
> >> >>>>> 55 hdd
7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB
16 GiB 1.2 TiB 83.56 1.12 122
up osd.55
> >> >>>>> -10
51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB
100 GiB 12 TiB 76.05 1.02 -
host s3db9
> >> >>>>> 56 hdd
7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB
13 GiB 2.1 TiB 71.06 0.95 105
up osd.56
> >> >>>>> 57 hdd
7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB
16 GiB 1.2 TiB 83.17 1.12 102
up osd.57
> >> >>>>> 58 hdd
7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB
15 GiB 1.4 TiB 81.56 1.09 105
up osd.58
> >> >>>>> 59 hdd
7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB
15 GiB 1.4 TiB 80.64 1.08 94
up osd.59
> >> >>>>> 60 hdd
7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB
14 GiB 2.0 TiB 73.25 0.98 101
up osd.60
> >> >>>>> 61 hdd
7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB
12 GiB 2.5 TiB 65.84 0.88 103
up osd.61
> >> >>>>> 62 hdd
7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB
15 GiB 1.7 TiB 76.83 1.03 126
up osd.62
> >> >>>>>
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB
1.2 TiB 173 TiB 74.57
> >> >>>>> MIN/MAX VAR:
0.17/1.20 STDDEV: 10.25
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der
Ster <
dan(a)vanderster.com>gt;:
> >> >>>>>>
> >> >>>>>> No, increasing num PGs won't help
substantially.
> >> >>>>>>
> >> >>>>>> Can you share the entire output of ceph osd df tree
?
> >> >>>>>>
> >> >>>>>> Did you already set
> >> >>>>>>
> >> >>>>>> ceph config set mgr
mgr/balancer/upmap_max_deviation 1
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> ??
> >> >>>>>> And I recommend debug_mgr 4/5 so you can see some
basic
upmap balancer logging.
> >> >>>>>>
> >> >>>>>> .. Dan
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
> >> >>>>>>>
> >> >>>>>>> Hello people,
> >> >>>>>>>
> >> >>>>>>> I am still struggeling with the balancer
> >> >>>>>>> (
https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
> >> >>>>>>> Now
I've read some more and might think that I do not have
enough PGs.
> >> >>>>>>>
Currently I have 84OSDs and 1024PGs for the main pool (3008
total). I
> >> >>>>>>> have
the autoscaler enabled, but I doesn't tell me to
increase the
> >> >>>>>>> PGs.
> >> >>>>>>>
> >> >>>>>>> What do you think?
> >> >>>>>>>
> >> >>>>>>> --
> >> >>>>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal
abweichend
> >> >>>>>>> im
groüen Saal.
> >> >>>>>>>
_______________________________________________
> >> >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
> >> >>>>>>> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> --
> >> >>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft
sich diesmal
abweichend im groüen Saal.
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal
abweichend im groüen Saal.
> >> >
> >> >
> >> >
> >> > --
> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
> >
> >
> >
> > --
> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im
groüen Saal.
5:57 p.m.
Hi,
backfill_toofull is not a bad thing when the cluster is really full
like yours. You should expect some of the most full OSDs to eventually
start decreasing in usage, as the PGs are moved to the new OSDs. Those
backfill_toofull states should then resolve themselves as the OSD
usage flattens out.
Keep an eye on the usage of the backfill_full and nearfull OSDs though
-- if they do eventually go above the full_ratio (95% by default),
then writes to those OSDs would stop.
But if on the other hand you're suffering from lots of slow ops or
anything else visible to your users, then you could try to take some
actions to slow down the rebalancing. Just let us know if that's the
case and we can see about changing osd_max_backfills, some weights or
maybe using the upmap-remapped tool.
-- Dan
On Tue, Mar 23, 2021 at 6:07 PM Boris Behrens <bb(a)kervyn.de> wrote:
>
> Ok, I should have listened to you :)
>
> In the last week we added more storage but the issue got worse instead.
> Today I realized that the PGs were up to 90GB (bytes column in ceph pg ls said
95705749636), and the balance kept mentioning the 2048 PGs for this pool. We were at 72%
utilization (ceph osd df tree, first line) for our cluster and I increased the PGs to
2048.
>
> Now I am in a world of trouble.
> The space in the cluster went down, I am at 45% misplaced objects, and we already
added 20x4TB disks just to not go down completly.
>
> The utilization is still going up and the overall free space in the cluster seems to
go down. This is what my ceph status looks like and now I really need help to get that
thing back to normal:
> [root@s3db1 ~]# ceph status
> cluster:
> id: dca79fff-ffd0-58f4-1cff-82a2feea05f4
> health: HEALTH_WARN
> 4 backfillfull osd(s)
> 17 nearfull osd(s)
> 37 pool(s) backfillfull
> 13 large omap objects
> Low space hindering backfill (add storage if this doesn't resolve
itself): 570 pgs backfill_toofull
>
> services:
> mon: 3 daemons, quorum ceph-s3-mon1,ceph-s3-mon2,ceph-s3-mon3 (age 44m)
> mgr: ceph-mgr2(active, since 15m), standbys: ceph-mgr3, ceph-mgr1
> mds: 3 up:standby
> osd: 110 osds: 110 up (since 28m), 110 in (since 28m); 1535 remapped pgs
> rgw: 3 daemons active (eu-central-1, eu-msg-1, eu-secure-1)
>
> task status:
>
> data:
> pools: 37 pools, 4032 pgs
> objects: 116.23M objects, 182 TiB
> usage: 589 TiB used, 206 TiB / 795 TiB avail
> pgs: 160918554/348689415 objects misplaced (46.150%)
> 2497 active+clean
> 779 active+remapped+backfill_wait
> 538 active+remapped+backfill_wait+backfill_toofull
> 186 active+remapped+backfilling
> 32 active+remapped+backfill_toofull
>
> io:
> client: 27 MiB/s rd, 69 MiB/s wr, 497 op/s rd, 153 op/s wr
> recovery: 1.5 GiB/s, 922 objects/s
>
> Am Di., 16. März 2021 um 09:34 Uhr schrieb Boris Behrens <bb(a)kervyn.de>de>:
>>
>> Hi Dan,
>>
>> my EC profile look very "default" to me.
>> [root@s3db1 ~]# ceph osd erasure-code-profile ls
>> default
>> [root@s3db1 ~]# ceph osd erasure-code-profile get default
>> k=2
>> m=1
>> plugin=jerasure
>> technique=reed_sol_van
>>
>> I don't understand the ouput, but the balancing get worse over night:
>>
>> [root@s3db1 ~]# ceph-scripts/tools/ceph-pool-pg-distribution 11
>> Searching for PGs in pools: ['11']
>> Summary: 1024 PGs on 84 osds
>>
>> Num OSDs with X PGs:
>> 15: 8
>> 16: 7
>> 17: 6
>> 18: 10
>> 19: 1
>> 32: 10
>> 33: 4
>> 34: 6
>> 35: 8
>> 65: 5
>> 66: 5
>> 67: 4
>> 68: 10
>> [root@s3db1 ~]# ceph-scripts/tools/ceph-pg-histogram --normalize --pool=11
>> # NumSamples = 84; Min = 4.12; Max = 5.09
>> # Mean = 4.553355; Variance = 0.052415; SD = 0.228942; Median 4.561608
>> # each ∎ represents a count of 1
>> 4.1244 - 4.2205 [ 8]: ∎∎∎∎∎∎∎∎
>> 4.2205 - 4.3166 [ 6]: ∎∎∎∎∎∎
>> 4.3166 - 4.4127 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
>> 4.4127 - 4.5087 [ 10]: ∎∎∎∎∎∎∎∎∎∎
>> 4.5087 - 4.6048 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
>> 4.6048 - 4.7009 [ 19]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
>> 4.7009 - 4.7970 [ 6]: ∎∎∎∎∎∎
>> 4.7970 - 4.8931 [ 8]: ∎∎∎∎∎∎∎∎
>> 4.8931 - 4.9892 [ 4]: ∎∎∎∎
>> 4.9892 - 5.0852 [ 1]: ∎
>> [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail
>> 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 724 GiB 19 GiB 0 B 724 GiB
80.56 1.07 56 up osd.14
>> 19 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 466 MiB 7.9 GiB 708 GiB
81.25 1.08 53 up osd.19
>> 4 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 698 GiB 703 MiB 0 B 698 GiB
81.27 1.08 48 up osd.4
>> 24 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 695 GiB 640 MiB 0 B 695 GiB
81.34 1.08 46 up osd.24
>> 75 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 440 MiB 8.1 GiB 704 GiB
81.35 1.08 48 up osd.75
>> 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 7.5 MiB 8.0 GiB 663 GiB
82.44 1.09 47 up osd.71
>> 76 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 251 MiB 9.0 GiB 617 GiB
83.65 1.11 50 up osd.76
>> 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 399 MiB 8.1 GiB 618 GiB
83.85 1.11 55 up osd.33
>> 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 317 MiB 8.8 GiB 617 GiB
83.87 1.11 50 up osd.35
>> 34 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 451 MiB 8.7 GiB 545 GiB
85.75 1.14 54 up osd.34
>>
>> Am Mo., 15. März 2021 um 17:23 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>>
>>> Hi,
>>>
>>> How wide are your EC profiles? If they are really wide, you might be
>>> reaching the limits of what is physically possible. Also, I'm not sure
>>> that upmap in 14.2.11 is very smart about *improving* existing upmap
>>> rules for a given PG, in the case that a PG already has an upmap-items
>>> entry but it would help the distribution to add more mapping pairs to
>>> that entry. What this means, is that it might sometimes be useful to
>>> randomly remove some upmap entries and see if the balancer does a
>>> better job when it replaces them.
>>>
>>> But before you do that, I re-remembered that looking at the total PG
>>> numbers is not useful -- you need to check the PGs per OSD for the
>>> eu-central-1.rgw.buckets.data pool only.
>>>
>>> We have a couple tools that can help with this:
>>>
>>> 1. To see the PGs per OSD for a given pool:
>>>
https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis…
>>>
>>> E.g.: ./ceph-pool-pg-distribution 11 # to see the distribution of
>>> your eu-central-1.rgw.buckets.data pool.
>>>
>>> The output looks like this on my well balanced clusters:
>>>
>>> # ceph-scripts/tools/ceph-pool-pg-distribution 15
>>> Searching for PGs in pools: ['15']
>>> Summary: 256 pgs on 56 osds
>>>
>>> Num OSDs with X PGs:
>>> 13: 16
>>> 14: 40
>>>
>>> You should expect a trimodal for your cluster.
>>>
>>> 2. You can also use another script from that repo to see the PGs per
>>> OSD normalized to crush weight:
>>> ceph-scripts/tools/ceph-pg-histogram --normalize --pool=15
>>>
>>> This might explain what is going wrong.
>>>
>>> Cheers, Dan
>>>
>>>
>>> On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens <bb(a)kervyn.de> wrote:
>>> >
>>> > Absolutly:
>>> > [root@s3db1 ~]# ceph osd df tree
>>> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
>>> > -1 673.54224 - 674 TiB 496 TiB 468 TiB 97 GiB 1.2 TiB
177 TiB 73.67 1.00 - root default
>>> > -2 58.30331 - 58 TiB 42 TiB 38 TiB 9.2 GiB 99 GiB
16 TiB 72.88 0.99 - host s3db1
>>> > 23 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 714 MiB 25 GiB
3.7 TiB 74.87 1.02 194 up osd.23
>>> > 69 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.6 GiB 40 GiB
3.4 TiB 76.32 1.04 199 up osd.69
>>> > 73 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.3 GiB 34 GiB
3.8 TiB 74.15 1.01 203 up osd.73
>>> > 79 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.8 GiB 0 B
1.3 TiB 65.44 0.89 47 up osd.79
>>> > 80 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 2.2 GiB 0 B
1.3 TiB 65.34 0.89 48 up osd.80
>>> > 81 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.1 GiB 0 B
1.3 TiB 65.38 0.89 47 up osd.81
>>> > 82 hdd 3.63689 1.00000 3.6 TiB 2.5 TiB 1.1 TiB 619 MiB 0 B
1.1 TiB 68.46 0.93 41 up osd.82
>>> > -11 50.94173 - 51 TiB 37 TiB 37 TiB 3.5 GiB 98 GiB
14 TiB 71.90 0.98 - host s3db10
>>> > 63 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 647 MiB 14 GiB
2.0 TiB 72.72 0.99 94 up osd.63
>>> > 64 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.2 TiB 668 MiB 14 GiB
2.0 TiB 72.23 0.98 93 up osd.64
>>> > 65 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 227 MiB 14 GiB
2.1 TiB 71.16 0.97 100 up osd.65
>>> > 66 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 313 MiB 14 GiB
1.9 TiB 74.25 1.01 92 up osd.66
>>> > 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 584 MiB 14 GiB
2.1 TiB 70.63 0.96 96 up osd.67
>>> > 68 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 720 MiB 14 GiB
2.1 TiB 71.72 0.97 101 up osd.68
>>> > 70 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 425 MiB 14 GiB
2.1 TiB 70.59 0.96 97 up osd.70
>>> > -12 50.99052 - 51 TiB 38 TiB 37 TiB 2.1 GiB 97 GiB
13 TiB 73.77 1.00 - host s3db11
>>> > 46 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 229 MiB 14 GiB
1.7 TiB 77.05 1.05 97 up osd.46
>>> > 47 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 159 MiB 13 GiB
2.2 TiB 70.00 0.95 89 up osd.47
>>> > 48 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 279 MiB 14 GiB
2.1 TiB 71.82 0.97 98 up osd.48
>>> > 49 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 276 MiB 14 GiB
1.8 TiB 74.90 1.02 95 up osd.49
>>> > 50 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 336 MiB 14 GiB
2.0 TiB 72.13 0.98 93 up osd.50
>>> > 51 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 728 MiB 15 GiB
1.6 TiB 77.76 1.06 98 up osd.51
>>> > 72 hdd 7.32619 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 147 MiB 13 GiB
2.0 TiB 72.75 0.99 95 up osd.72
>>> > -37 58.55478 - 59 TiB 44 TiB 44 TiB 4.4 GiB 122 GiB
15 TiB 75.20 1.02 - host s3db12
>>> > 19 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 454 MiB 8.2 GiB
780 GiB 79.35 1.08 53 up osd.19
>>> > 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 7.1 MiB 8.0 GiB
734 GiB 80.56 1.09 47 up osd.71
>>> > 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 439 MiB 8.2 GiB
777 GiB 79.43 1.08 48 up osd.75
>>> > 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 241 MiB 8.9 GiB
688 GiB 81.77 1.11 50 up osd.76
>>> > 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB
3.6 TiB 75.44 1.02 201 up osd.77
>>> > 78 hdd 14.60159 1.00000 15 TiB 10 TiB 10 TiB 1015 MiB 28 GiB
4.2 TiB 71.26 0.97 193 up osd.78
>>> > 83 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.4 GiB 30 GiB
3.8 TiB 73.76 1.00 203 up osd.83
>>> > -3 58.49872 - 58 TiB 42 TiB 36 TiB 8.2 GiB 89 GiB
17 TiB 71.71 0.97 - host s3db2
>>> > 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.2 GiB 37 GiB
3.7 TiB 74.58 1.01 196 up osd.1
>>> > 3 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 566 MiB 0 B
1.3 TiB 64.11 0.87 50 up osd.3
>>> > 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 771 GiB 695 MiB 0 B
771 GiB 79.30 1.08 48 up osd.4
>>> > 5 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 482 MiB 0 B
1.2 TiB 66.51 0.90 49 up osd.5
>>> > 6 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 1.8 GiB 0 B
1.3 TiB 64.00 0.87 42 up osd.6
>>> > 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 639 MiB 26 GiB
4.0 TiB 72.44 0.98 192 up osd.7
>>> > 74 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 907 MiB 26 GiB
4.2 TiB 71.32 0.97 193 up osd.74
>>> > -4 58.49872 - 58 TiB 43 TiB 36 TiB 34 GiB 85 GiB
16 TiB 72.69 0.99 - host s3db3
>>> > 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 980 MiB 26 GiB
3.8 TiB 74.36 1.01 203 up osd.2
>>> > 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.4 GiB 33 GiB
3.9 TiB 73.51 1.00 186 up osd.9
>>> > 10 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 650 MiB 26 GiB
4.2 TiB 71.64 0.97 201 up osd.10
>>> > 12 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 754 MiB 0 B
1.3 TiB 64.17 0.87 44 up osd.12
>>> > 13 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 813 GiB 2.4 GiB 0 B
813 GiB 78.17 1.06 58 up osd.13
>>> > 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 797 GiB 19 GiB 0 B
797 GiB 78.60 1.07 56 up osd.14
>>> > 15 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 2.2 GiB 0 B
1.3 TiB 63.96 0.87 41 up osd.15
>>> > -5 58.49872 - 58 TiB 43 TiB 36 TiB 6.7 GiB 97 GiB
15 TiB 74.04 1.01 - host s3db4
>>> > 11 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 940 MiB 26 GiB
4.0 TiB 72.49 0.98 196 up osd.11
>>> > 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1022 MiB 26 GiB
3.6 TiB 75.23 1.02 204 up osd.17
>>> > 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 945 MiB 45 GiB
3.8 TiB 74.16 1.01 193 up osd.18
>>> > 20 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1020 GiB 596 MiB 0 B
1020 GiB 72.62 0.99 57 up osd.20
>>> > 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 1.9 GiB 0 B
1023 GiB 72.54 0.98 41 up osd.21
>>> > 22 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 797 MiB 0 B
1023 GiB 72.54 0.98 53 up osd.22
>>> > 24 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 766 GiB 618 MiB 0 B
766 GiB 79.42 1.08 46 up osd.24
>>> > -6 58.89636 - 59 TiB 43 TiB 43 TiB 3.0 GiB 108 GiB
16 TiB 73.40 1.00 - host s3db5
>>> > 0 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 92 MiB 7.2 GiB
1.1 TiB 71.16 0.97 45 up osd.0
>>> > 25 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 2.4 MiB 7.3 GiB
1.1 TiB 71.23 0.97 41 up osd.25
>>> > 26 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 181 MiB 7.6 GiB
935 GiB 75.57 1.03 45 up osd.26
>>> > 27 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 5.1 MiB 7.0 GiB
1.1 TiB 71.20 0.97 47 up osd.27
>>> > 28 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 977 MiB 26 GiB
3.8 TiB 73.85 1.00 197 up osd.28
>>> > 29 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 872 MiB 26 GiB
4.1 TiB 71.98 0.98 196 up osd.29
>>> > 30 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 943 MiB 27 GiB
3.6 TiB 75.51 1.03 202 up osd.30
>>> > -7 58.89636 - 59 TiB 44 TiB 43 TiB 13 GiB 122 GiB
15 TiB 74.97 1.02 - host s3db6
>>> > 32 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 27 MiB 7.6 GiB
940 GiB 75.42 1.02 55 up osd.32
>>> > 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 376 MiB 8.2 GiB
691 GiB 81.94 1.11 55 up osd.33
>>> > 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 450 MiB 8.5 GiB
620 GiB 83.79 1.14 54 up osd.34
>>> > 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 316 MiB 8.4 GiB
690 GiB 81.98 1.11 50 up osd.35
>>> > 36 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 489 MiB 25 GiB
4.1 TiB 71.69 0.97 208 up osd.36
>>> > 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB
4.0 TiB 72.41 0.98 195 up osd.37
>>> > 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 26 GiB
3.7 TiB 74.88 1.02 204 up osd.38
>>> > -8 58.89636 - 59 TiB 44 TiB 43 TiB 3.8 GiB 111 GiB
15 TiB 74.16 1.01 - host s3db7
>>> > 39 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 19 MiB 7.5 GiB
936 GiB 75.54 1.03 39 up osd.39
>>> > 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.1 GiB
1.1 TiB 69.87 0.95 39 up osd.40
>>> > 41 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.7 TiB 219 MiB 7.6 GiB
1011 GiB 73.57 1.00 55 up osd.41
>>> > 42 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 593 MiB 7.1 GiB
1.1 TiB 70.02 0.95 47 up osd.42
>>> > 43 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 500 MiB 27 GiB
3.7 TiB 74.67 1.01 204 up osd.43
>>> > 44 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB
3.7 TiB 74.62 1.01 193 up osd.44
>>> > 45 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 29 GiB
3.6 TiB 75.16 1.02 204 up osd.45
>>> > -9 51.28331 - 51 TiB 39 TiB 39 TiB 4.9 GiB 107 GiB
12 TiB 76.50 1.04 - host s3db8
>>> > 8 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 474 MiB 14 GiB
1.7 TiB 76.37 1.04 98 up osd.8
>>> > 16 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 783 MiB 15 GiB
1.6 TiB 78.39 1.06 100 up osd.16
>>> > 31 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 441 MiB 14 GiB
1.6 TiB 77.70 1.05 91 up osd.31
>>> > 52 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 939 MiB 14 GiB
1.7 TiB 76.29 1.04 102 up osd.52
>>> > 53 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 848 MiB 18 GiB
1.9 TiB 74.30 1.01 98 up osd.53
>>> > 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 16 GiB
1.7 TiB 76.99 1.05 106 up osd.54
>>> > 55 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 460 MiB 15 GiB
1.8 TiB 75.46 1.02 105 up osd.55
>>> > -10 51.28331 - 51 TiB 37 TiB 37 TiB 3.8 GiB 96 GiB
14 TiB 72.77 0.99 - host s3db9
>>> > 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 846 MiB 13 GiB
2.1 TiB 71.16 0.97 104 up osd.56
>>> > 57 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 513 MiB 15 GiB
1.7 TiB 76.53 1.04 96 up osd.57
>>> > 58 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 604 MiB 13 GiB
2.1 TiB 71.23 0.97 98 up osd.58
>>> > 59 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 414 MiB 13 GiB
2.2 TiB 70.03 0.95 88 up osd.59
>>> > 60 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 227 MiB 14 GiB
1.8 TiB 75.54 1.03 97 up osd.60
>>> > 61 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 456 MiB 13 GiB
2.2 TiB 70.01 0.95 95 up osd.61
>>> > 62 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 843 MiB 14 GiB
1.8 TiB 74.93 1.02 110 up osd.62
>>> > TOTAL 674 TiB 496 TiB 468 TiB 97 GiB 1.2 TiB
177 TiB 73.67
>>> > MIN/MAX VAR: 0.87/1.14 STDDEV: 4.22
>>> >
>>> > Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>> >>
>>> >> OK thanks. Indeed "prepared 0/10 changes" means it thinks
things are balanced.
>>> >> Could you again share the full ceph osd df tree?
>>> >>
>>> >> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens <bb(a)kervyn.de>
wrote:
>>> >> >
>>> >> > Hi Dan,
>>> >> >
>>> >> > I've set the autoscaler to warn, but it actually does not
warn for now. So not touching it for now.
>>> >> >
>>> >> > this is what the log says in minute intervals:
>>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/active
>>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/sleep_interval
>>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/begin_time
>>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/end_time
>>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/begin_weekday
>>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/end_weekday
>>> >> > 2021-03-15 13:51:00.971 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/pool_ids
>>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Optimize
plan auto_2021-03-15_13:51:00
>>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/mode
>>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Mode
upmap, max misplaced 0.050000
>>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] do_upmap
>>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/upmap_max_iterations
>>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/upmap_max_deviation
>>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] pools
['eu-msg-1.rgw.data.root', 'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys', 'eu-central-1.rgw.buckets.index',
'rbd', 'eu-msg-1.rgw.log', 'whitespace-again-2021-03-10_2',
'eu-msg-1.rgw.buckets.index', 'eu-msg-1.rgw.meta',
'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control', '.rgw.root', 'eu-msg-1.rgw.buckets.data',
'default.rgw.control', 'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3', 'default.rgw.log',
'eu-central-1.rgw.meta', 'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
>>> >> > 2021-03-15 13:51:01.224 7f307d5fd700 4 mgr[balancer] prepared
0/10 changes
>>> >> >
>>> >> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>> >> >>
>>> >> >> I suggest to just disable the autoscaler until your
balancing is understood.
>>> >> >>
>>> >> >> What does your active mgr log say (with debug_mgr 4/5),
grep balancer
>>> >> >> /var/log/ceph/ceph-mgr.*.log
>>> >> >>
>>> >> >> -- Dan
>>> >> >>
>>> >> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>>> >> >> >
>>> >> >> > Hi,
>>> >> >> > this unfortunally did not solve my problem. I still
have some OSDs that fill up to 85%
>>> >> >> >
>>> >> >> > According to the logging, the autoscaler might want to
add more PGs to one Bucken and reduce almost all other buckets to 32.
>>> >> >> > 2021-03-15 12:19:58.825 7f307f601700 4
mgr[pg_autoscaler] Pool 'eu-central-1.rgw.buckets.data' root_id -1 using
0.705080476146 of space, bias 1.0, pg target 1974.22533321 quantized to 2048 (current
1024)
>>> >> >> >
>>> >> >> > Why the balancing does not happen is still nebulous to
me.
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der
Ster <dan(a)vanderster.com>om>:
>>> >> >> >>
>>> >> >> >> OK
>>> >> >> >> Btw, you might need to fail to a new mgr...
I'm not sure if the current active will read that new config.
>>> >> >> >>
>>> >> >> >> .. dan
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>>> >> >> >>>
>>> >> >> >>> Hi,
>>> >> >> >>>
>>> >> >> >>> ok thanks. I just changed the value and
rewighted everything back to 1. Now I let it sync the weekend and check how it will be on
monday.
>>> >> >> >>> We tried to have the systems total storage
balanced as possible. New systems will be with 8TB disks but for the exiting ones we added
16TB to offset the 4TB disks and we needed a lot of storage fast, because of a DC move. If
you have any recommendations I would be happy to hear them.
>>> >> >> >>>
>>> >> >> >>> Cheers
>>> >> >> >>> Boris
>>> >> >> >>>
>>> >> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan
van der Ster <dan(a)vanderster.com>om>:
>>> >> >> >>>>
>>> >> >> >>>> Thanks.
>>> >> >> >>>>
>>> >> >> >>>> Decreasing the max deviation to 2 or 1
should help in your case. This option controls when the balancer stops trying to move PGs
around -- by default it stops when the deviation from the mean is 5. Yes this is too large
IMO -- all of our clusters have this set to 1.
>>> >> >> >>>>
>>> >> >> >>>> And given that you have some OSDs with
more than 200 PGs, you definitely shouldn't increase the num PGs.
>>> >> >> >>>>
>>> >> >> >>>> But anyway with your mixed device sizes it
might be challenging to make a perfectly uniform distribution. Give it a try with 1
though, and let us know how it goes.
>>> >> >> >>>>
>>> >> >> >>>> .. Dan
>>> >> >> >>>>
>>> >> >> >>>>
>>> >> >> >>>>
>>> >> >> >>>>
>>> >> >> >>>>
>>> >> >> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris
Behrens <bb(a)kervyn.de> wrote:
>>> >> >> >>>>>
>>> >> >> >>>>> Hi Dan,
>>> >> >> >>>>>
>>> >> >> >>>>> upmap_max_deviation is default (5) in
our cluster. Is 1 the recommended deviation?
>>> >> >> >>>>>
>>> >> >> >>>>> I added the whole ceph osd df tree, (I
need to remove some OSDs and readd them as bluestore with SSD, so 69, 73 and 82 are a bit
off now. I also reweighted to try to get the %USE mitigated).
>>> >> >> >>>>>
>>> >> >> >>>>> I will increase the mgr debugging to
see what is the problem.
>>> >> >> >>>>>
>>> >> >> >>>>> [root@s3db1 ~]# ceph osd df tree
>>> >> >> >>>>> ID CLASS WEIGHT REWEIGHT SIZE
RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
>>> >> >> >>>>> -1 673.54224 - 659 TiB
491 TiB 464 TiB 96 GiB 1.2 TiB 168 TiB 74.57 1.00 - root default
>>> >> >> >>>>> -2 58.30331 - 44 TiB
22 TiB 17 TiB 5.7 GiB 38 GiB 22 TiB 49.82 0.67 - host s3db1
>>> >> >> >>>>> 23 hdd 14.65039 1.00000 15 TiB
1.8 TiB 1.7 TiB 156 MiB 4.4 GiB 13 TiB 12.50 0.17 101 up osd.23
>>> >> >> >>>>> 69 hdd 14.55269 0 0 B
0 B 0 B 0 B 0 B 0 B 0 0 11 up osd.69
>>> >> >> >>>>> 73 hdd 14.55269 1.00000 15 TiB
10 TiB 10 TiB 6.1 MiB 33 GiB 4.2 TiB 71.15 0.95 107 up osd.73
>>> >> >> >>>>> 79 hdd 3.63689 1.00000 3.6 TiB
2.9 TiB 747 GiB 2.0 GiB 0 B 747 GiB 79.94 1.07 52 up osd.79
>>> >> >> >>>>> 80 hdd 3.63689 1.00000 3.6 TiB
2.6 TiB 1.0 TiB 1.9 GiB 0 B 1.0 TiB 71.61 0.96 58 up osd.80
>>> >> >> >>>>> 81 hdd 3.63689 1.00000 3.6 TiB
2.2 TiB 1.5 TiB 1.1 GiB 0 B 1.5 TiB 60.07 0.81 55 up osd.81
>>> >> >> >>>>> 82 hdd 3.63689 1.00000 3.6 TiB
1.9 TiB 1.7 TiB 536 MiB 0 B 1.7 TiB 52.68 0.71 30 up osd.82
>>> >> >> >>>>> -11 50.94173 - 51 TiB
38 TiB 38 TiB 3.7 GiB 100 GiB 13 TiB 74.69 1.00 - host s3db10
>>> >> >> >>>>> 63 hdd 7.27739 1.00000 7.3 TiB
5.5 TiB 5.5 TiB 616 MiB 14 GiB 1.7 TiB 76.04 1.02 92 up osd.63
>>> >> >> >>>>> 64 hdd 7.27739 1.00000 7.3 TiB
5.5 TiB 5.5 TiB 820 MiB 15 GiB 1.8 TiB 75.54 1.01 101 up osd.64
>>> >> >> >>>>> 65 hdd 7.27739 1.00000 7.3 TiB
5.3 TiB 5.3 TiB 109 MiB 14 GiB 2.0 TiB 73.17 0.98 105 up osd.65
>>> >> >> >>>>> 66 hdd 7.27739 1.00000 7.3 TiB
5.8 TiB 5.8 TiB 423 MiB 15 GiB 1.4 TiB 80.38 1.08 98 up osd.66
>>> >> >> >>>>> 67 hdd 7.27739 1.00000 7.3 TiB
5.1 TiB 5.1 TiB 572 MiB 14 GiB 2.2 TiB 70.10 0.94 100 up osd.67
>>> >> >> >>>>> 68 hdd 7.27739 1.00000 7.3 TiB
5.3 TiB 5.3 TiB 630 MiB 13 GiB 2.0 TiB 72.88 0.98 107 up osd.68
>>> >> >> >>>>> 70 hdd 7.27739 1.00000 7.3 TiB
5.4 TiB 5.4 TiB 648 MiB 14 GiB 1.8 TiB 74.73 1.00 102 up osd.70
>>> >> >> >>>>> -12 50.99052 - 51 TiB
39 TiB 39 TiB 2.9 GiB 99 GiB 12 TiB 77.24 1.04 - host s3db11
>>> >> >> >>>>> 46 hdd 7.27739 1.00000 7.3 TiB
5.7 TiB 5.7 TiB 102 MiB 15 GiB 1.5 TiB 78.91 1.06 97 up osd.46
>>> >> >> >>>>> 47 hdd 7.27739 1.00000 7.3 TiB
5.2 TiB 5.2 TiB 61 MiB 13 GiB 2.1 TiB 71.47 0.96 96 up osd.47
>>> >> >> >>>>> 48 hdd 7.27739 1.00000 7.3 TiB
6.1 TiB 6.1 TiB 853 MiB 15 GiB 1.2 TiB 83.46 1.12 109 up osd.48
>>> >> >> >>>>> 49 hdd 7.27739 1.00000 7.3 TiB
5.7 TiB 5.7 TiB 708 MiB 15 GiB 1.5 TiB 78.96 1.06 98 up osd.49
>>> >> >> >>>>> 50 hdd 7.27739 1.00000 7.3 TiB
5.9 TiB 5.8 TiB 472 MiB 15 GiB 1.4 TiB 80.40 1.08 102 up osd.50
>>> >> >> >>>>> 51 hdd 7.27739 1.00000 7.3 TiB
5.9 TiB 5.9 TiB 729 MiB 15 GiB 1.3 TiB 81.70 1.10 110 up osd.51
>>> >> >> >>>>> 72 hdd 7.32619 1.00000 7.3 TiB
4.8 TiB 4.8 TiB 91 MiB 12 GiB 2.5 TiB 65.82 0.88 89 up osd.72
>>> >> >> >>>>> -37 58.55478 - 59 TiB
46 TiB 46 TiB 5.0 GiB 124 GiB 12 TiB 79.04 1.06 - host s3db12
>>> >> >> >>>>> 19 hdd 3.68750 1.00000 3.7 TiB
3.1 TiB 3.1 TiB 462 MiB 8.2 GiB 559 GiB 85.18 1.14 55 up osd.19
>>> >> >> >>>>> 71 hdd 3.68750 1.00000 3.7 TiB
2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB 825 GiB 78.14 1.05 50 up osd.71
>>> >> >> >>>>> 75 hdd 3.68750 1.00000 3.7 TiB
3.1 TiB 3.1 TiB 576 MiB 8.3 GiB 555 GiB 85.29 1.14 57 up osd.75
>>> >> >> >>>>> 76 hdd 3.68750 1.00000 3.7 TiB
3.2 TiB 3.1 TiB 239 MiB 9.3 GiB 501 GiB 86.73 1.16 50 up osd.76
>>> >> >> >>>>> 77 hdd 14.60159 1.00000 15 TiB
11 TiB 11 TiB 880 MiB 30 GiB 3.6 TiB 75.57 1.01 202 up osd.77
>>> >> >> >>>>> 78 hdd 14.60159 1.00000 15 TiB
11 TiB 11 TiB 1.0 GiB 30 GiB 3.4 TiB 76.65 1.03 196 up osd.78
>>> >> >> >>>>> 83 hdd 14.60159 1.00000 15 TiB
12 TiB 12 TiB 1.8 GiB 31 GiB 2.9 TiB 80.04 1.07 223 up osd.83
>>> >> >> >>>>> -3 58.49872 - 58 TiB
43 TiB 38 TiB 8.1 GiB 91 GiB 16 TiB 73.15 0.98 - host s3db2
>>> >> >> >>>>> 1 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 3.1 GiB 38 GiB 3.6 TiB 75.52 1.01 194 up osd.1
>>> >> >> >>>>> 3 hdd 3.63689 1.00000 3.6 TiB
2.2 TiB 1.4 TiB 418 MiB 0 B 1.4 TiB 60.94 0.82 52 up osd.3
>>> >> >> >>>>> 4 hdd 3.63689 0.89999 3.6 TiB
3.2 TiB 401 GiB 845 MiB 0 B 401 GiB 89.23 1.20 53 up osd.4
>>> >> >> >>>>> 5 hdd 3.63689 1.00000 3.6 TiB
2.3 TiB 1.3 TiB 437 MiB 0 B 1.3 TiB 62.88 0.84 51 up osd.5
>>> >> >> >>>>> 6 hdd 3.63689 1.00000 3.6 TiB
2.0 TiB 1.7 TiB 1.8 GiB 0 B 1.7 TiB 54.51 0.73 47 up osd.6
>>> >> >> >>>>> 7 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 493 MiB 26 GiB 3.8 TiB 73.90 0.99 185 up osd.7
>>> >> >> >>>>> 74 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 1.1 GiB 27 GiB 3.5 TiB 76.27 1.02 208 up osd.74
>>> >> >> >>>>> -4 58.49872 - 58 TiB
43 TiB 37 TiB 33 GiB 86 GiB 15 TiB 74.05 0.99 - host s3db3
>>> >> >> >>>>> 2 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 850 MiB 26 GiB 4.0 TiB 72.78 0.98 203 up osd.2
>>> >> >> >>>>> 9 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 8.3 GiB 33 GiB 3.6 TiB 75.62 1.01 189 up osd.9
>>> >> >> >>>>> 10 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 663 MiB 28 GiB 3.5 TiB 76.34 1.02 211 up osd.10
>>> >> >> >>>>> 12 hdd 3.63689 1.00000 3.6 TiB
2.4 TiB 1.2 TiB 633 MiB 0 B 1.2 TiB 66.22 0.89 44 up osd.12
>>> >> >> >>>>> 13 hdd 3.63689 1.00000 3.6 TiB
2.9 TiB 720 GiB 2.3 GiB 0 B 720 GiB 80.66 1.08 66 up osd.13
>>> >> >> >>>>> 14 hdd 3.63689 1.00000 3.6 TiB
3.1 TiB 552 GiB 18 GiB 0 B 552 GiB 85.18 1.14 60 up osd.14
>>> >> >> >>>>> 15 hdd 3.63689 1.00000 3.6 TiB
2.0 TiB 1.7 TiB 2.1 GiB 0 B 1.7 TiB 53.72 0.72 44 up osd.15
>>> >> >> >>>>> -5 58.49872 - 58 TiB
45 TiB 37 TiB 7.2 GiB 99 GiB 14 TiB 76.37 1.02 - host s3db4
>>> >> >> >>>>> 11 hdd 14.65039 1.00000 15 TiB
12 TiB 12 TiB 897 MiB 28 GiB 2.8 TiB 81.15 1.09 205 up osd.11
>>> >> >> >>>>> 17 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 1.2 GiB 27 GiB 3.6 TiB 75.38 1.01 211 up osd.17
>>> >> >> >>>>> 18 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 965 MiB 44 GiB 4.0 TiB 72.86 0.98 188 up osd.18
>>> >> >> >>>>> 20 hdd 3.63689 1.00000 3.6 TiB
2.9 TiB 796 GiB 529 MiB 0 B 796 GiB 78.63 1.05 66 up osd.20
>>> >> >> >>>>> 21 hdd 3.63689 1.00000 3.6 TiB
2.6 TiB 1.1 TiB 2.1 GiB 0 B 1.1 TiB 70.32 0.94 47 up osd.21
>>> >> >> >>>>> 22 hdd 3.63689 1.00000 3.6 TiB
2.9 TiB 802 GiB 882 MiB 0 B 802 GiB 78.47 1.05 58 up osd.22
>>> >> >> >>>>> 24 hdd 3.63689 1.00000 3.6 TiB
2.8 TiB 856 GiB 645 MiB 0 B 856 GiB 77.01 1.03 47 up osd.24
>>> >> >> >>>>> -6 58.89636 - 59 TiB
44 TiB 44 TiB 2.4 GiB 111 GiB 15 TiB 75.22 1.01 - host s3db5
>>> >> >> >>>>> 0 hdd 3.73630 1.00000 3.7 TiB
2.4 TiB 2.3 TiB 70 MiB 6.6 GiB 1.3 TiB 65.00 0.87 48 up osd.0
>>> >> >> >>>>> 25 hdd 3.73630 1.00000 3.7 TiB
2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB 1.4 TiB 63.86 0.86 41 up osd.25
>>> >> >> >>>>> 26 hdd 3.73630 1.00000 3.7 TiB
2.9 TiB 2.8 TiB 181 MiB 7.6 GiB 862 GiB 77.47 1.04 48 up osd.26
>>> >> >> >>>>> 27 hdd 3.73630 1.00000 3.7 TiB
2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB 1.5 TiB 61.00 0.82 48 up osd.27
>>> >> >> >>>>> 28 hdd 14.65039 1.00000 15 TiB
12 TiB 12 TiB 937 MiB 30 GiB 2.8 TiB 81.19 1.09 203 up osd.28
>>> >> >> >>>>> 29 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 536 MiB 26 GiB 3.8 TiB 73.95 0.99 200 up osd.29
>>> >> >> >>>>> 30 hdd 14.65039 1.00000 15 TiB
12 TiB 11 TiB 744 MiB 28 GiB 3.1 TiB 79.07 1.06 207 up osd.30
>>> >> >> >>>>> -7 58.89636 - 59 TiB
44 TiB 44 TiB 14 GiB 122 GiB 14 TiB 75.41 1.01 - host s3db6
>>> >> >> >>>>> 32 hdd 3.73630 1.00000 3.7 TiB
3.1 TiB 3.0 TiB 16 MiB 8.2 GiB 622 GiB 83.74 1.12 65 up osd.32
>>> >> >> >>>>> 33 hdd 3.73630 0.79999 3.7 TiB
3.0 TiB 2.9 TiB 14 MiB 8.1 GiB 740 GiB 80.67 1.08 52 up osd.33
>>> >> >> >>>>> 34 hdd 3.73630 0.79999 3.7 TiB
2.9 TiB 2.8 TiB 449 MiB 7.7 GiB 877 GiB 77.08 1.03 52 up osd.34
>>> >> >> >>>>> 35 hdd 3.73630 0.79999 3.7 TiB
2.3 TiB 2.2 TiB 133 MiB 7.0 GiB 1.4 TiB 62.18 0.83 42 up osd.35
>>> >> >> >>>>> 36 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 544 MiB 26 GiB 4.0 TiB 72.98 0.98 220 up osd.36
>>> >> >> >>>>> 37 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 11 GiB 38 GiB 3.6 TiB 75.30 1.01 200 up osd.37
>>> >> >> >>>>> 38 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 1.2 GiB 28 GiB 3.3 TiB 77.43 1.04 217 up osd.38
>>> >> >> >>>>> -8 58.89636 - 59 TiB
47 TiB 46 TiB 3.9 GiB 116 GiB 12 TiB 78.98 1.06 - host s3db7
>>> >> >> >>>>> 39 hdd 3.73630 1.00000 3.7 TiB
3.2 TiB 3.2 TiB 19 MiB 8.5 GiB 499 GiB 86.96 1.17 43 up osd.39
>>> >> >> >>>>> 40 hdd 3.73630 1.00000 3.7 TiB
2.6 TiB 2.5 TiB 144 MiB 7.0 GiB 1.2 TiB 68.33 0.92 39 up osd.40
>>> >> >> >>>>> 41 hdd 3.73630 1.00000 3.7 TiB
3.0 TiB 2.9 TiB 218 MiB 7.9 GiB 732 GiB 80.86 1.08 64 up osd.41
>>> >> >> >>>>> 42 hdd 3.73630 1.00000 3.7 TiB
2.5 TiB 2.4 TiB 594 MiB 7.0 GiB 1.2 TiB 67.97 0.91 50 up osd.42
>>> >> >> >>>>> 43 hdd 14.65039 1.00000 15 TiB
12 TiB 12 TiB 564 MiB 28 GiB 2.9 TiB 80.32 1.08 213 up osd.43
>>> >> >> >>>>> 44 hdd 14.65039 1.00000 15 TiB
12 TiB 11 TiB 1.3 GiB 28 GiB 3.1 TiB 78.59 1.05 198 up osd.44
>>> >> >> >>>>> 45 hdd 14.65039 1.00000 15 TiB
12 TiB 12 TiB 1.2 GiB 30 GiB 2.8 TiB 81.05 1.09 214 up osd.45
>>> >> >> >>>>> -9 51.28331 - 51 TiB
41 TiB 41 TiB 4.9 GiB 108 GiB 10 TiB 79.75 1.07 - host s3db8
>>> >> >> >>>>> 8 hdd 7.32619 1.00000 7.3 TiB
5.8 TiB 5.8 TiB 472 MiB 15 GiB 1.5 TiB 79.68 1.07 99 up osd.8
>>> >> >> >>>>> 16 hdd 7.32619 1.00000 7.3 TiB
5.9 TiB 5.8 TiB 785 MiB 15 GiB 1.4 TiB 80.25 1.08 97 up osd.16
>>> >> >> >>>>> 31 hdd 7.32619 1.00000 7.3 TiB
5.5 TiB 5.5 TiB 438 MiB 14 GiB 1.8 TiB 75.36 1.01 87 up osd.31
>>> >> >> >>>>> 52 hdd 7.32619 1.00000 7.3 TiB
5.7 TiB 5.7 TiB 844 MiB 15 GiB 1.6 TiB 78.19 1.05 113 up osd.52
>>> >> >> >>>>> 53 hdd 7.32619 1.00000 7.3 TiB
6.2 TiB 6.1 TiB 792 MiB 18 GiB 1.1 TiB 84.46 1.13 109 up osd.53
>>> >> >> >>>>> 54 hdd 7.32619 1.00000 7.3 TiB
5.6 TiB 5.6 TiB 959 MiB 15 GiB 1.7 TiB 76.73 1.03 115 up osd.54
>>> >> >> >>>>> 55 hdd 7.32619 1.00000 7.3 TiB
6.1 TiB 6.1 TiB 699 MiB 16 GiB 1.2 TiB 83.56 1.12 122 up osd.55
>>> >> >> >>>>> -10 51.28331 - 51 TiB
39 TiB 39 TiB 4.7 GiB 100 GiB 12 TiB 76.05 1.02 - host s3db9
>>> >> >> >>>>> 56 hdd 7.32619 1.00000 7.3 TiB
5.2 TiB 5.2 TiB 840 MiB 13 GiB 2.1 TiB 71.06 0.95 105 up osd.56
>>> >> >> >>>>> 57 hdd 7.32619 1.00000 7.3 TiB
6.1 TiB 6.0 TiB 1.0 GiB 16 GiB 1.2 TiB 83.17 1.12 102 up osd.57
>>> >> >> >>>>> 58 hdd 7.32619 1.00000 7.3 TiB
6.0 TiB 5.9 TiB 43 MiB 15 GiB 1.4 TiB 81.56 1.09 105 up osd.58
>>> >> >> >>>>> 59 hdd 7.32619 1.00000 7.3 TiB
5.9 TiB 5.9 TiB 429 MiB 15 GiB 1.4 TiB 80.64 1.08 94 up osd.59
>>> >> >> >>>>> 60 hdd 7.32619 1.00000 7.3 TiB
5.4 TiB 5.3 TiB 226 MiB 14 GiB 2.0 TiB 73.25 0.98 101 up osd.60
>>> >> >> >>>>> 61 hdd 7.32619 1.00000 7.3 TiB
4.8 TiB 4.8 TiB 1.1 GiB 12 GiB 2.5 TiB 65.84 0.88 103 up osd.61
>>> >> >> >>>>> 62 hdd 7.32619 1.00000 7.3 TiB
5.6 TiB 5.6 TiB 1.0 GiB 15 GiB 1.7 TiB 76.83 1.03 126 up osd.62
>>> >> >> >>>>> TOTAL 674 TiB
501 TiB 473 TiB 96 GiB 1.2 TiB 173 TiB 74.57
>>> >> >> >>>>> MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr
schrieb Dan van der Ster <dan(a)vanderster.com>om>:
>>> >> >> >>>>>>
>>> >> >> >>>>>> No, increasing num PGs won't
help substantially.
>>> >> >> >>>>>>
>>> >> >> >>>>>> Can you share the entire output of
ceph osd df tree ?
>>> >> >> >>>>>>
>>> >> >> >>>>>> Did you already set
>>> >> >> >>>>>>
>>> >> >> >>>>>> ceph config set mgr
mgr/balancer/upmap_max_deviation 1
>>> >> >> >>>>>>
>>> >> >> >>>>>>
>>> >> >> >>>>>> ??
>>> >> >> >>>>>> And I recommend debug_mgr 4/5 so
you can see some basic upmap balancer logging.
>>> >> >> >>>>>>
>>> >> >> >>>>>> .. Dan
>>> >> >> >>>>>>
>>> >> >> >>>>>>
>>> >> >> >>>>>>
>>> >> >> >>>>>>
>>> >> >> >>>>>>
>>> >> >> >>>>>>
>>> >> >> >>>>>> On Sat, Mar 13, 2021, 3:49 PM
Boris Behrens <bb(a)kervyn.de> wrote:
>>> >> >> >>>>>>>
>>> >> >> >>>>>>> Hello people,
>>> >> >> >>>>>>>
>>> >> >> >>>>>>> I am still struggeling with
the balancer
>>> >> >> >>>>>>>
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>>> >> >> >>>>>>> Now I've read some more
and might think that I do not have enough PGs.
>>> >> >> >>>>>>> Currently I have 84OSDs and
1024PGs for the main pool (3008 total). I
>>> >> >> >>>>>>> have the autoscaler enabled,
but I doesn't tell me to increase the
>>> >> >> >>>>>>> PGs.
>>> >> >> >>>>>>>
>>> >> >> >>>>>>> What do you think?
>>> >> >> >>>>>>>
>>> >> >> >>>>>>> --
>>> >> >> >>>>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend
>>> >> >> >>>>>>> im groüen Saal.
>>> >> >> >>>>>>>
_______________________________________________
>>> >> >> >>>>>>> ceph-users mailing list --
ceph-users(a)ceph.io
>>> >> >> >>>>>>> To unsubscribe send an email
to ceph-users-leave(a)ceph.io
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>>
>>> >> >> >>>>> --
>>> >> >> >>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
>>> >> >> >>>
>>> >> >> >>>
>>> >> >> >>>
>>> >> >> >>> --
>>> >> >> >>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Die Selbsthilfegruppe "UTF-8-Probleme"
trifft sich diesmal abweichend im groüen Saal.
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal abweichend im groüen Saal.
>>> >
>>> >
>>> >
>>> > --
>>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>>
>>
>>
>> --
>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
6:05 p.m.
So,
doing nothing and wait for the ceph to recover?
In theory there should be enough disk space (more disks arriving tomorrow),
but I fear that there might be an issue, when the backups get exported over
night to this s3. Currently the max_avail lingers around 13TB and I hope,
that the data will go to other PGs than the ones that are currently on
filled OSDs.
Am Di., 23. März 2021 um 18:58 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
Hi,
backfill_toofull is not a bad thing when the cluster is really full
like yours. You should expect some of the most full OSDs to eventually
start decreasing in usage, as the PGs are moved to the new OSDs. Those
backfill_toofull states should then resolve themselves as the OSD
usage flattens out.
Keep an eye on the usage of the backfill_full and nearfull OSDs though
-- if they do eventually go above the full_ratio (95% by default),
then writes to those OSDs would stop.
But if on the other hand you're suffering from lots of slow ops or
anything else visible to your users, then you could try to take some
actions to slow down the rebalancing. Just let us know if that's the
case and we can see about changing osd_max_backfills, some weights or
maybe using the upmap-remapped tool.
-- Dan
On Tue, Mar 23, 2021 at 6:07 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
Ok, I should have listened to you :)
In the last week we added more storage but the issue got worse instead.
Today I realized that the PGs were up to 90GB (bytes column in ceph pg
ls said
95705749636), and the balance kept mentioning the 2048 PGs for this
pool. We were at 72% utilization (ceph osd df tree, first line) for our
cluster and I increased the PGs to 2048.
Now I am in a world of trouble.
The space in the cluster went down, I am at 45% misplaced objects, and
we already
added 20x4TB disks just to not go down completly.
The utilization is still going up and the overall free space in the
cluster seems
to go down. This is what my ceph status looks like and now I
really need help to get that thing back to normal:
[root@s3db1 ~]# ceph status
cluster:
id: dca79fff-ffd0-58f4-1cff-82a2feea05f4
health: HEALTH_WARN
4 backfillfull osd(s)
17 nearfull osd(s)
37 pool(s) backfillfull
13 large omap objects
Low space hindering backfill (add storage if this doesn't
resolve
itself): 570 pgs backfill_toofull
services:
mon: 3 daemons, quorum ceph-s3-mon1,ceph-s3-mon2,ceph-s3-mon3 (age
44m)
mgr: ceph-mgr2(active, since 15m), standbys:
ceph-mgr3, ceph-mgr1
mds: 3 up:standby
osd: 110 osds: 110 up (since 28m), 110 in (since 28m); 1535 remapped
pgs
rgw: 3 daemons active (eu-central-1,
eu-msg-1, eu-secure-1)
task status:
data:
pools: 37 pools, 4032 pgs
objects: 116.23M objects, 182 TiB
usage: 589 TiB used, 206 TiB / 795 TiB avail
pgs: 160918554/348689415 objects misplaced (46.150%)
2497 active+clean
779 active+remapped+backfill_wait
538 active+remapped+backfill_wait+backfill_toofull
186 active+remapped+backfilling
32 active+remapped+backfill_toofull
io:
client: 27 MiB/s rd, 69 MiB/s wr, 497 op/s rd, 153 op/s wr
recovery: 1.5 GiB/s, 922 objects/s
Am Di., 16. März 2021 um 09:34 Uhr schrieb Boris Behrens <bb(a)kervyn.de>de>:
>
> Hi Dan,
>
> my EC profile look very "default" to me.
> [root@s3db1 ~]# ceph osd erasure-code-profile ls
> default
> [root@s3db1 ~]# ceph osd erasure-code-profile get default
> k=2
> m=1
> plugin=jerasure
> technique=reed_sol_van
>
> I don't understand the ouput, but the balancing get worse over night:
>
> [root@s3db1 ~]# ceph-scripts/tools/ceph-pool-pg-distribution 11
> Searching for PGs in pools: ['11']
> Summary: 1024 PGs on 84 osds
>
> Num OSDs with X PGs:
> 15: 8
> 16: 7
> 17: 6
> 18: 10
> 19: 1
> 32: 10
> 33: 4
> 34: 6
> 35: 8
> 65: 5
> 66: 5
> 67: 4
> 68: 10
> [root@s3db1 ~]# ceph-scripts/tools/ceph-pg-histogram --normalize
--pool=11
> # NumSamples = 84; Min = 4.12; Max = 5.09
> # Mean = 4.553355; Variance = 0.052415; SD = 0.228942; Median 4.561608
> # each ∎ represents a count of 1
> 4.1244 - 4.2205 [ 8]: ∎∎∎∎∎∎∎∎
> 4.2205 - 4.3166 [ 6]: ∎∎∎∎∎∎
> 4.3166 - 4.4127 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
> 4.4127 - 4.5087 [ 10]: ∎∎∎∎∎∎∎∎∎∎
> 4.5087 - 4.6048 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
> 4.6048 - 4.7009 [ 19]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
> 4.7009 - 4.7970 [ 6]: ∎∎∎∎∎∎
> 4.7970 - 4.8931 [ 8]: ∎∎∎∎∎∎∎∎
> 4.8931 - 4.9892 [ 4]: ∎∎∎∎
> 4.9892 - 5.0852 [ 1]: ∎
> [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail
> 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 724 GiB 19 GiB 0 B
724 GiB
80.56 1.07 56 up osd.14
> 19 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB
2.9 TiB 466 MiB 7.9 GiB
708 GiB 81.25 1.08 53 up osd.19
> 4 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB
698 GiB 703 MiB 0 B
698 GiB 81.27 1.08 48 up osd.4
> 24 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB
695 GiB 640 MiB 0 B
695 GiB 81.34 1.08 46 up osd.24
> 75 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB
2.9 TiB 440 MiB 8.1 GiB
704 GiB 81.35 1.08 48 up osd.75
> 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB
3.0 TiB 7.5 MiB 8.0 GiB
663 GiB 82.44 1.09 47 up osd.71
> 76 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB
3.0 TiB 251 MiB 9.0 GiB
617 GiB 83.65 1.11 50 up osd.76
> 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB
3.0 TiB 399 MiB 8.1 GiB
618 GiB 83.85 1.11 55 up osd.33
> 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB
3.0 TiB 317 MiB 8.8 GiB
617 GiB 83.87 1.11 50 up osd.35
> 34 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB
3.1 TiB 451 MiB 8.7 GiB
545 GiB 85.75 1.14 54 up osd.34
>
> Am Mo., 15. März 2021 um 17:23 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>>
>> Hi,
>>
>> How wide are your EC profiles? If they are really wide, you might be
>> reaching the limits of what is physically possible. Also, I'm not sure
>> that upmap in 14.2.11 is very smart about *improving* existing upmap
>> rules for a given PG, in the case that a PG already has an upmap-items
>> entry but it would help the distribution to add more mapping pairs to
>> that entry. What this means, is that it might sometimes be useful to
>> randomly remove some upmap entries and see if the balancer does a
>> better job when it replaces them.
>>
>> But before you do that, I re-remembered that looking at the total PG
>> numbers is not useful -- you need to check the PGs per OSD for the
>> eu-central-1.rgw.buckets.data pool only.
>>
>> We have a couple tools that can help with this:
>>
>> 1. To see the PGs per OSD for a given pool:
>>
https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis…
>>
>> E.g.: ./ceph-pool-pg-distribution 11 # to see the distribution of
>> your eu-central-1.rgw.buckets.data pool.
>>
>> The output looks like this on my well balanced clusters:
>>
>> # ceph-scripts/tools/ceph-pool-pg-distribution 15
>> Searching for PGs in pools: ['15']
>> Summary: 256 pgs on 56 osds
>>
>> Num OSDs with X PGs:
>> 13: 16
>> 14: 40
>>
>> You should expect a trimodal for your cluster.
>>
>> 2. You can also use another script from that repo to see the PGs per
>> OSD normalized to crush weight:
>> ceph-scripts/tools/ceph-pg-histogram --normalize --pool=15
>>
>> This might explain what is going wrong.
>>
>> Cheers, Dan
>>
>>
>> On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >
>> > Absolutly:
>> > [root@s3db1 ~]# ceph osd df tree
>> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
>> > -1 673.54224 - 674 TiB
496 TiB 468 TiB 97 GiB 1.2
TiB 177 TiB 73.67 1.00 - root default
>> > -2 58.30331 - 58 TiB
42 TiB 38 TiB 9.2 GiB 99
GiB 16 TiB 72.88 0.99 - host s3db1
>> > 23 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 714 MiB 25
GiB 3.7 TiB 74.87 1.02 194 up osd.23
>> > 69 hdd 14.55269 1.00000 15 TiB
11 TiB 11 TiB 1.6 GiB 40
GiB 3.4 TiB 76.32 1.04 199 up osd.69
>> > 73 hdd 14.55269 1.00000 15 TiB
11 TiB 11 TiB 1.3 GiB 34
GiB 3.8 TiB 74.15 1.01 203 up osd.73
>> > 79 hdd 3.63689 1.00000 3.6 TiB
2.4 TiB 1.3 TiB 1.8 GiB 0
B 1.3 TiB 65.44 0.89 47 up osd.79
>> > 80 hdd 3.63689 1.00000 3.6 TiB
2.4 TiB 1.3 TiB 2.2 GiB 0
B 1.3 TiB 65.34 0.89 48 up osd.80
>> > 81 hdd 3.63689 1.00000 3.6 TiB
2.4 TiB 1.3 TiB 1.1 GiB 0
B 1.3 TiB 65.38 0.89 47 up osd.81
>> > 82 hdd 3.63689 1.00000 3.6 TiB
2.5 TiB 1.1 TiB 619 MiB 0
B 1.1 TiB 68.46 0.93 41 up osd.82
>> > -11 50.94173 - 51 TiB
37 TiB 37 TiB 3.5 GiB 98
GiB 14 TiB 71.90 0.98 - host s3db10
>> > 63 hdd 7.27739 1.00000 7.3 TiB
5.3 TiB 5.3 TiB 647 MiB 14
GiB 2.0 TiB 72.72 0.99 94 up osd.63
>> > 64 hdd 7.27739 1.00000 7.3 TiB
5.3 TiB 5.2 TiB 668 MiB 14
GiB 2.0 TiB 72.23 0.98 93 up osd.64
>> > 65 hdd 7.27739 1.00000 7.3 TiB
5.2 TiB 5.2 TiB 227 MiB 14
GiB 2.1 TiB 71.16 0.97 100 up osd.65
>> > 66 hdd 7.27739 1.00000 7.3 TiB
5.4 TiB 5.4 TiB 313 MiB 14
GiB 1.9 TiB 74.25 1.01 92 up osd.66
>> > 67 hdd 7.27739 1.00000 7.3 TiB
5.1 TiB 5.1 TiB 584 MiB 14
GiB 2.1 TiB 70.63 0.96 96 up osd.67
>> > 68 hdd 7.27739 1.00000 7.3 TiB
5.2 TiB 5.2 TiB 720 MiB 14
GiB 2.1 TiB 71.72 0.97 101 up osd.68
>> > 70 hdd 7.27739 1.00000 7.3 TiB
5.1 TiB 5.1 TiB 425 MiB 14
GiB 2.1 TiB 70.59 0.96 97 up osd.70
>> > -12 50.99052 - 51 TiB
38 TiB 37 TiB 2.1 GiB 97
GiB 13 TiB 73.77 1.00 - host s3db11
>> > 46 hdd 7.27739 1.00000 7.3 TiB
5.6 TiB 5.6 TiB 229 MiB 14
GiB 1.7 TiB 77.05 1.05 97 up osd.46
>> > 47 hdd 7.27739 1.00000 7.3 TiB
5.1 TiB 5.1 TiB 159 MiB 13
GiB 2.2 TiB 70.00 0.95 89 up osd.47
>> > 48 hdd 7.27739 1.00000 7.3 TiB
5.2 TiB 5.2 TiB 279 MiB 14
GiB 2.1 TiB 71.82 0.97 98 up osd.48
>> > 49 hdd 7.27739 1.00000 7.3 TiB
5.5 TiB 5.4 TiB 276 MiB 14
GiB 1.8 TiB 74.90 1.02 95 up osd.49
>> > 50 hdd 7.27739 1.00000 7.3 TiB
5.2 TiB 5.2 TiB 336 MiB 14
GiB 2.0 TiB 72.13 0.98 93 up osd.50
>> > 51 hdd 7.27739 1.00000 7.3 TiB
5.7 TiB 5.6 TiB 728 MiB 15
GiB 1.6 TiB 77.76 1.06 98 up osd.51
>> > 72 hdd 7.32619 1.00000 7.3 TiB
5.3 TiB 5.3 TiB 147 MiB 13
GiB 2.0 TiB 72.75 0.99 95 up osd.72
>> > -37 58.55478 - 59 TiB
44 TiB 44 TiB 4.4 GiB 122
GiB 15 TiB 75.20 1.02 - host s3db12
>> > 19 hdd 3.68750 1.00000 3.7 TiB
2.9 TiB 2.9 TiB 454 MiB 8.2
GiB 780 GiB 79.35 1.08 53 up osd.19
>> > 71 hdd 3.68750 1.00000 3.7 TiB
3.0 TiB 2.9 TiB 7.1 MiB 8.0
GiB 734 GiB 80.56 1.09 47 up osd.71
>> > 75 hdd 3.68750 1.00000 3.7 TiB
2.9 TiB 2.9 TiB 439 MiB 8.2
GiB 777 GiB 79.43 1.08 48 up osd.75
>> > 76 hdd 3.68750 1.00000 3.7 TiB
3.0 TiB 3.0 TiB 241 MiB 8.9
GiB 688 GiB 81.77 1.11 50 up osd.76
>> > 77 hdd 14.60159 1.00000 15 TiB
11 TiB 11 TiB 880 MiB 30
GiB 3.6 TiB 75.44 1.02 201 up osd.77
>> > 78 hdd 14.60159 1.00000 15 TiB
10 TiB 10 TiB 1015 MiB 28
GiB 4.2 TiB 71.26 0.97 193 up osd.78
>> > 83 hdd 14.60159 1.00000 15 TiB
11 TiB 11 TiB 1.4 GiB 30
GiB 3.8 TiB 73.76 1.00 203 up osd.83
>> > -3 58.49872 - 58 TiB
42 TiB 36 TiB 8.2 GiB 89
GiB 17 TiB 71.71 0.97 - host s3db2
>> > 1 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 3.2 GiB 37
GiB 3.7 TiB 74.58 1.01 196 up osd.1
>> > 3 hdd 3.63689 1.00000 3.6 TiB
2.3 TiB 1.3 TiB 566 MiB 0
B 1.3 TiB 64.11 0.87 50 up osd.3
>> > 4 hdd 3.63689 1.00000 3.6 TiB
2.9 TiB 771 GiB 695 MiB 0
B 771 GiB 79.30 1.08 48 up osd.4
>> > 5 hdd 3.63689 1.00000 3.6 TiB
2.4 TiB 1.2 TiB 482 MiB 0
B 1.2 TiB 66.51 0.90 49 up osd.5
>> > 6 hdd 3.63689 1.00000 3.6 TiB
2.3 TiB 1.3 TiB 1.8 GiB 0
B 1.3 TiB 64.00 0.87 42 up osd.6
>> > 7 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 639 MiB 26
GiB 4.0 TiB 72.44 0.98 192 up osd.7
>> > 74 hdd 14.65039 1.00000 15 TiB
10 TiB 10 TiB 907 MiB 26
GiB 4.2 TiB 71.32 0.97 193 up osd.74
>> > -4 58.49872 - 58 TiB
43 TiB 36 TiB 34 GiB 85
GiB 16 TiB 72.69 0.99 - host s3db3
>> > 2 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 980 MiB 26
GiB 3.8 TiB 74.36 1.01 203 up osd.2
>> > 9 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 8.4 GiB 33
GiB 3.9 TiB 73.51 1.00 186 up osd.9
>> > 10 hdd 14.65039 1.00000 15 TiB
10 TiB 10 TiB 650 MiB 26
GiB 4.2 TiB 71.64 0.97 201 up osd.10
>> > 12 hdd 3.63689 1.00000 3.6 TiB
2.3 TiB 1.3 TiB 754 MiB 0
B 1.3 TiB 64.17 0.87 44 up osd.12
>> > 13 hdd 3.63689 1.00000 3.6 TiB
2.8 TiB 813 GiB 2.4 GiB 0
B 813 GiB 78.17 1.06 58 up osd.13
>> > 14 hdd 3.63689 1.00000 3.6 TiB
2.9 TiB 797 GiB 19 GiB 0
B 797 GiB 78.60 1.07 56 up osd.14
>> > 15 hdd 3.63689 1.00000 3.6 TiB
2.3 TiB 1.3 TiB 2.2 GiB 0
B 1.3 TiB 63.96 0.87 41 up osd.15
>> > -5 58.49872 - 58 TiB
43 TiB 36 TiB 6.7 GiB 97
GiB 15 TiB 74.04 1.01 - host s3db4
>> > 11 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 940 MiB 26
GiB 4.0 TiB 72.49 0.98 196 up osd.11
>> > 17 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 1022 MiB 26
GiB 3.6 TiB 75.23 1.02 204 up osd.17
>> > 18 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 945 MiB 45
GiB 3.8 TiB 74.16 1.01 193 up osd.18
>> > 20 hdd 3.63689 1.00000 3.6 TiB
2.6 TiB 1020 GiB 596 MiB 0
B 1020 GiB 72.62 0.99 57 up osd.20
>> > 21 hdd 3.63689 1.00000 3.6 TiB
2.6 TiB 1023 GiB 1.9 GiB 0
B 1023 GiB 72.54 0.98 41 up osd.21
>> > 22 hdd 3.63689 1.00000 3.6 TiB
2.6 TiB 1023 GiB 797 MiB 0
B 1023 GiB 72.54 0.98 53 up osd.22
>> > 24 hdd 3.63689 1.00000 3.6 TiB
2.9 TiB 766 GiB 618 MiB 0
B 766 GiB 79.42 1.08 46 up osd.24
>> > -6 58.89636 - 59 TiB
43 TiB 43 TiB 3.0 GiB 108
GiB 16 TiB 73.40 1.00 - host s3db5
>> > 0 hdd 3.73630 1.00000 3.7 TiB
2.7 TiB 2.6 TiB 92 MiB 7.2
GiB 1.1 TiB 71.16 0.97 45 up osd.0
>> > 25 hdd 3.73630 1.00000 3.7 TiB
2.7 TiB 2.6 TiB 2.4 MiB 7.3
GiB 1.1 TiB 71.23 0.97 41 up osd.25
>> > 26 hdd 3.73630 1.00000 3.7 TiB
2.8 TiB 2.7 TiB 181 MiB 7.6
GiB 935 GiB 75.57 1.03 45 up osd.26
>> > 27 hdd 3.73630 1.00000 3.7 TiB
2.7 TiB 2.6 TiB 5.1 MiB 7.0
GiB 1.1 TiB 71.20 0.97 47 up osd.27
>> > 28 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 977 MiB 26
GiB 3.8 TiB 73.85 1.00 197 up osd.28
>> > 29 hdd 14.65039 1.00000 15 TiB
11 TiB 10 TiB 872 MiB 26
GiB 4.1 TiB 71.98 0.98 196 up osd.29
>> > 30 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 943 MiB 27
GiB 3.6 TiB 75.51 1.03 202 up osd.30
>> > -7 58.89636 - 59 TiB
44 TiB 43 TiB 13 GiB 122
GiB 15 TiB 74.97 1.02 - host s3db6
>> > 32 hdd 3.73630 1.00000 3.7 TiB
2.8 TiB 2.7 TiB 27 MiB 7.6
GiB 940 GiB 75.42 1.02 55 up osd.32
>> > 33 hdd 3.73630 1.00000 3.7 TiB
3.1 TiB 3.0 TiB 376 MiB 8.2
GiB 691 GiB 81.94 1.11 55 up osd.33
>> > 34 hdd 3.73630 1.00000 3.7 TiB
3.1 TiB 3.0 TiB 450 MiB 8.5
GiB 620 GiB 83.79 1.14 54 up osd.34
>> > 35 hdd 3.73630 1.00000 3.7 TiB
3.1 TiB 3.0 TiB 316 MiB 8.4
GiB 690 GiB 81.98 1.11 50 up osd.35
>> > 36 hdd 14.65039 1.00000 15 TiB
11 TiB 10 TiB 489 MiB 25
GiB 4.1 TiB 71.69 0.97 208 up osd.36
>> > 37 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 11 GiB 38
GiB 4.0 TiB 72.41 0.98 195 up osd.37
>> > 38 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 1.1 GiB 26
GiB 3.7 TiB 74.88 1.02 204 up osd.38
>> > -8 58.89636 - 59 TiB
44 TiB 43 TiB 3.8 GiB 111
GiB 15 TiB 74.16 1.01 - host s3db7
>> > 39 hdd 3.73630 1.00000 3.7 TiB
2.8 TiB 2.7 TiB 19 MiB 7.5
GiB 936 GiB 75.54 1.03 39 up osd.39
>> > 40 hdd 3.73630 1.00000 3.7 TiB
2.6 TiB 2.5 TiB 144 MiB 7.1
GiB 1.1 TiB 69.87 0.95 39 up osd.40
>> > 41 hdd 3.73630 1.00000 3.7 TiB
2.7 TiB 2.7 TiB 219 MiB 7.6
GiB 1011 GiB 73.57 1.00 55 up osd.41
>> > 42 hdd 3.73630 1.00000 3.7 TiB
2.6 TiB 2.5 TiB 593 MiB 7.1
GiB 1.1 TiB 70.02 0.95 47 up osd.42
>> > 43 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 500 MiB 27
GiB 3.7 TiB 74.67 1.01 204 up osd.43
>> > 44 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 1.1 GiB 27
GiB 3.7 TiB 74.62 1.01 193 up osd.44
>> > 45 hdd 14.65039 1.00000 15 TiB
11 TiB 11 TiB 1.2 GiB 29
GiB 3.6 TiB 75.16 1.02 204 up osd.45
>> > -9 51.28331 - 51 TiB
39 TiB 39 TiB 4.9 GiB 107
GiB 12 TiB 76.50 1.04 - host s3db8
>> > 8 hdd 7.32619 1.00000 7.3 TiB
5.6 TiB 5.5 TiB 474 MiB 14
GiB 1.7 TiB 76.37 1.04 98 up osd.8
>> > 16 hdd 7.32619 1.00000 7.3 TiB
5.7 TiB 5.7 TiB 783 MiB 15
GiB 1.6 TiB 78.39 1.06 100 up osd.16
>> > 31 hdd 7.32619 1.00000 7.3 TiB
5.7 TiB 5.6 TiB 441 MiB 14
GiB 1.6 TiB 77.70 1.05 91 up osd.31
>> > 52 hdd 7.32619 1.00000 7.3 TiB
5.6 TiB 5.5 TiB 939 MiB 14
GiB 1.7 TiB 76.29 1.04 102 up osd.52
>> > 53 hdd 7.32619 1.00000 7.3 TiB
5.4 TiB 5.4 TiB 848 MiB 18
GiB 1.9 TiB 74.30 1.01 98 up osd.53
>> > 54 hdd 7.32619 1.00000 7.3 TiB
5.6 TiB 5.6 TiB 1.0 GiB 16
GiB 1.7 TiB 76.99 1.05 106 up osd.54
>> > 55 hdd 7.32619 1.00000 7.3 TiB
5.5 TiB 5.5 TiB 460 MiB 15
GiB 1.8 TiB 75.46 1.02 105 up osd.55
>> > -10 51.28331 - 51 TiB
37 TiB 37 TiB 3.8 GiB 96
GiB 14 TiB 72.77 0.99 - host s3db9
>> > 56 hdd 7.32619 1.00000 7.3 TiB
5.2 TiB 5.2 TiB 846 MiB 13
GiB 2.1 TiB 71.16 0.97 104 up osd.56
>> > 57 hdd 7.32619 1.00000 7.3 TiB
5.6 TiB 5.6 TiB 513 MiB 15
GiB 1.7 TiB 76.53 1.04 96 up osd.57
>> > 58 hdd 7.32619 1.00000 7.3 TiB
5.2 TiB 5.2 TiB 604 MiB 13
GiB 2.1 TiB 71.23 0.97 98 up osd.58
>> > 59 hdd 7.32619 1.00000 7.3 TiB
5.1 TiB 5.1 TiB 414 MiB 13
GiB 2.2 TiB 70.03 0.95 88 up osd.59
>> > 60 hdd 7.32619 1.00000 7.3 TiB
5.5 TiB 5.5 TiB 227 MiB 14
GiB 1.8 TiB 75.54 1.03 97 up osd.60
>> > 61 hdd 7.32619 1.00000 7.3 TiB
5.1 TiB 5.1 TiB 456 MiB 13
GiB 2.2 TiB 70.01 0.95 95 up osd.61
>> > 62 hdd 7.32619 1.00000 7.3 TiB
5.5 TiB 5.4 TiB 843 MiB 14
GiB 1.8 TiB 74.93 1.02 110 up osd.62
>> > TOTAL 674 TiB
496 TiB 468 TiB 97 GiB 1.2
TiB 177 TiB 73.67
>> > MIN/MAX VAR: 0.87/1.14 STDDEV:
4.22
>> >
>> > Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>> >>
>> >> OK thanks. Indeed "prepared 0/10 changes" means it thinks
things
are balanced.
>> >> Could you again share the full
ceph osd df tree?
>> >>
>> >> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens <bb(a)kervyn.de>
wrote:
>> >> >
>> >> > Hi Dan,
>> >> >
>> >> > I've set the autoscaler to warn, but it actually does not warn
for now. So not touching it for now.
>> >> >
>> >> > this is what the log says in minute intervals:
>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/active
>> >> > 2021-03-15 13:51:00.970
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/sleep_interval
>> >> > 2021-03-15 13:51:00.970
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/begin_time
>> >> > 2021-03-15 13:51:00.970
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/end_time
>> >> > 2021-03-15 13:51:00.970
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/begin_weekday
>> >> > 2021-03-15 13:51:00.970
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/end_weekday
>> >> > 2021-03-15 13:51:00.971
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/pool_ids
>> >> > 2021-03-15 13:51:01.203
7f307d5fd700 4 mgr[balancer] Optimize
plan auto_2021-03-15_13:51:00
>> >> > 2021-03-15 13:51:01.203
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/mode
>> >> > 2021-03-15 13:51:01.203
7f307d5fd700 4 mgr[balancer] Mode upmap,
max misplaced 0.050000
>>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer]
do_upmap
>> >> > 2021-03-15 13:51:01.203
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/upmap_max_iterations
>> >> > 2021-03-15 13:51:01.203
7f307d5fd700 4 mgr get_config get_config
key: mgr/balancer/upmap_max_deviation
>> >> > 2021-03-15 13:51:01.203
7f307d5fd700 4 mgr[balancer] pools
['eu-msg-1.rgw.data.root',
'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys',
'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log',
'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index',
'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control',
'.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control',
'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3',
'default.rgw.log', 'eu-central-1.rgw.meta',
'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
>> >> > 2021-03-15 13:51:01.224
7f307d5fd700 4 mgr[balancer] prepared
0/10 changes
>> >> >
>> >> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>> >> >>
>> >> >> I suggest to just disable the autoscaler until your balancing
is
understood.
>> >> >>
>> >> >> What does your active mgr log say (with debug_mgr 4/5), grep
balancer
>> >> >>
/var/log/ceph/ceph-mgr.*.log
>> >> >>
>> >> >> -- Dan
>> >> >>
>> >> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
>> >> >> >
>> >> >> > Hi,
>> >> >> > this unfortunally did not solve my problem. I still have
some
OSDs that fill up to 85%
>> >> >> >
>> >> >> > According to the logging, the autoscaler might want to add
more PGs to one Bucken and reduce almost all other buckets to 32.
>> >> >> > 2021-03-15
12:19:58.825 7f307f601700 4 mgr[pg_autoscaler]
Pool
'eu-central-1.rgw.buckets.data' root_id -1 using 0.705080476146 of
space, bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
>> >> >> >
>> >> >> > Why the balancing does not happen is still nebulous to
me.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der
Ster <
dan(a)vanderster.com>gt;:
>> >> >> >>
>> >> >> >> OK
>> >> >> >> Btw, you might need to fail to a new mgr... I'm
not sure if
the current active will read that new config.
>> >> >> >>
>> >> >> >> .. dan
>> >> >> >>
>> >> >> >>
>> >> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
>> >> >> >>>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> ok thanks. I just changed the value and rewighted
everything
back to 1. Now I let it sync the weekend and check how it will be on
monday.
>> >> >> >>> We tried
to have the systems total storage balanced as
possible. New systems will be with
8TB disks but for the exiting ones we
added 16TB to offset the 4TB disks and we needed a lot of storage fast,
because of a DC move. If you have any recommendations I would be happy to
hear them.
>> >> >> >>>
>> >> >> >>> Cheers
>> >> >> >>> Boris
>> >> >> >>>
>> >> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van
der Ster <
dan(a)vanderster.com>gt;:
>> >> >> >>>>
>> >> >> >>>> Thanks.
>> >> >> >>>>
>> >> >> >>>> Decreasing the max deviation to 2 or 1 should
help in your
case. This option controls when the balancer stops trying to move PGs
around -- by default it stops when the deviation from the mean is 5. Yes
this is too large IMO -- all of our clusters have this set to 1.
>> >> >> >>>>
>> >> >> >>>> And given that you have some OSDs with more
than 200 PGs,
you definitely shouldn't increase the num PGs.
>> >> >> >>>>
>> >> >> >>>> But anyway with your mixed device sizes it
might be
challenging to make a perfectly uniform distribution. Give it a try with
1
though, and let us know how it goes.
>> >> >> >>>>
>> >> >> >>>> .. Dan
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
>> >> >> >>>>>
>> >> >> >>>>> Hi Dan,
>> >> >> >>>>>
>> >> >> >>>>> upmap_max_deviation is default (5) in our
cluster. Is 1
the recommended deviation?
>> >> >> >>>>>
>> >> >> >>>>> I added the whole ceph osd df tree, (I
need to remove some
OSDs and readd them as bluestore with SSD, so 69, 73 and 82 are
a bit off
now. I also reweighted to try to get the %USE mitigated).
>> >> >> >>>>>
>> >> >> >>>>> I will increase the mgr debugging to see
what is the
problem.
>> >> >> >>>>>
>> >> >> >>>>> [root@s3db1 ~]# ceph osd df tree
>> >> >> >>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW
USE DATA OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
>> >> >> >>>>>
-1 673.54224 - 659 TiB 491 TiB 464 TiB 96
GiB 1.2 TiB 168 TiB 74.57
1.00 - root default
>> >> >> >>>>>
-2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7
GiB 38 GiB 22 TiB 49.82
0.67 - host s3db1
>> >> >> >>>>>
23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156
MiB 4.4 GiB 13 TiB 12.50
0.17 101 up osd.23
>> >> >> >>>>>
69 hdd 14.55269 0 0 B 0 B 0 B 0
B 0 B 0 B 0
0 11 up osd.69
>> >> >> >>>>>
73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1
MiB 33 GiB 4.2 TiB 71.15
0.95 107 up osd.73
>> >> >> >>>>>
79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0
GiB 0 B 747 GiB 79.94
1.07 52 up osd.79
>> >> >> >>>>>
80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9
GiB 0 B 1.0 TiB 71.61
0.96 58 up osd.80
>> >> >> >>>>>
81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1
GiB 0 B 1.5 TiB 60.07
0.81 55 up osd.81
>> >> >> >>>>>
82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536
MiB 0 B 1.7 TiB 52.68
0.71 30 up osd.82
>> >> >> >>>>>
-11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7
GiB 100 GiB 13 TiB 74.69
1.00 - host s3db10
>> >> >> >>>>>
63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616
MiB 14 GiB 1.7 TiB 76.04
1.02 92 up osd.63
>> >> >> >>>>>
64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820
MiB 15 GiB 1.8 TiB 75.54
1.01 101 up osd.64
>> >> >> >>>>>
65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109
MiB 14 GiB 2.0 TiB 73.17
0.98 105 up osd.65
>> >> >> >>>>>
66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423
MiB 15 GiB 1.4 TiB 80.38
1.08 98 up osd.66
>> >> >> >>>>>
67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572
MiB 14 GiB 2.2 TiB 70.10
0.94 100 up osd.67
>> >> >> >>>>>
68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630
MiB 13 GiB 2.0 TiB 72.88
0.98 107 up osd.68
>> >> >> >>>>>
70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648
MiB 14 GiB 1.8 TiB 74.73
1.00 102 up osd.70
>> >> >> >>>>>
-12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9
GiB 99 GiB 12 TiB 77.24
1.04 - host s3db11
>> >> >> >>>>>
46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102
MiB 15 GiB 1.5 TiB 78.91
1.06 97 up osd.46
>> >> >> >>>>>
47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61
MiB 13 GiB 2.1 TiB 71.47
0.96 96 up osd.47
>> >> >> >>>>>
48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853
MiB 15 GiB 1.2 TiB 83.46
1.12 109 up osd.48
>> >> >> >>>>>
49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708
MiB 15 GiB 1.5 TiB 78.96
1.06 98 up osd.49
>> >> >> >>>>>
50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472
MiB 15 GiB 1.4 TiB 80.40
1.08 102 up osd.50
>> >> >> >>>>>
51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729
MiB 15 GiB 1.3 TiB 81.70
1.10 110 up osd.51
>> >> >> >>>>>
72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91
MiB 12 GiB 2.5 TiB 65.82
0.88 89 up osd.72
>> >> >> >>>>>
-37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0
GiB 124 GiB 12 TiB 79.04
1.06 - host s3db12
>> >> >> >>>>>
19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462
MiB 8.2 GiB 559 GiB 85.18
1.14 55 up osd.19
>> >> >> >>>>>
71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9
MiB 7.8 GiB 825 GiB 78.14
1.05 50 up osd.71
>> >> >> >>>>>
75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576
MiB 8.3 GiB 555 GiB 85.29
1.14 57 up osd.75
>> >> >> >>>>>
76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239
MiB 9.3 GiB 501 GiB 86.73
1.16 50 up osd.76
>> >> >> >>>>>
77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880
MiB 30 GiB 3.6 TiB 75.57
1.01 202 up osd.77
>> >> >> >>>>>
78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0
GiB 30 GiB 3.4 TiB 76.65
1.03 196 up osd.78
>> >> >> >>>>>
83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8
GiB 31 GiB 2.9 TiB 80.04
1.07 223 up osd.83
>> >> >> >>>>>
-3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1
GiB 91 GiB 16 TiB 73.15
0.98 - host s3db2
>> >> >> >>>>>
1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1
GiB 38 GiB 3.6 TiB 75.52
1.01 194 up osd.1
>> >> >> >>>>>
3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418
MiB 0 B 1.4 TiB 60.94
0.82 52 up osd.3
>> >> >> >>>>>
4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845
MiB 0 B 401 GiB 89.23
1.20 53 up osd.4
>> >> >> >>>>>
5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437
MiB 0 B 1.3 TiB 62.88
0.84 51 up osd.5
>> >> >> >>>>>
6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8
GiB 0 B 1.7 TiB 54.51
0.73 47 up osd.6
>> >> >> >>>>>
7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493
MiB 26 GiB 3.8 TiB 73.90
0.99 185 up osd.7
>> >> >> >>>>>
74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1
GiB 27 GiB 3.5 TiB 76.27
1.02 208 up osd.74
>> >> >> >>>>>
-4 58.49872 - 58 TiB 43 TiB 37 TiB 33
GiB 86 GiB 15 TiB 74.05
0.99 - host s3db3
>> >> >> >>>>>
2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850
MiB 26 GiB 4.0 TiB 72.78
0.98 203 up osd.2
>> >> >> >>>>>
9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3
GiB 33 GiB 3.6 TiB 75.62
1.01 189 up osd.9
>> >> >> >>>>>
10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663
MiB 28 GiB 3.5 TiB 76.34
1.02 211 up osd.10
>> >> >> >>>>>
12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633
MiB 0 B 1.2 TiB 66.22
0.89 44 up osd.12
>> >> >> >>>>>
13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3
GiB 0 B 720 GiB 80.66
1.08 66 up osd.13
>> >> >> >>>>>
14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18
GiB 0 B 552 GiB 85.18
1.14 60 up osd.14
>> >> >> >>>>>
15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1
GiB 0 B 1.7 TiB 53.72
0.72 44 up osd.15
>> >> >> >>>>>
-5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2
GiB 99 GiB 14 TiB 76.37
1.02 - host s3db4
>> >> >> >>>>>
11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897
MiB 28 GiB 2.8 TiB 81.15
1.09 205 up osd.11
>> >> >> >>>>>
17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2
GiB 27 GiB 3.6 TiB 75.38
1.01 211 up osd.17
>> >> >> >>>>>
18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965
MiB 44 GiB 4.0 TiB 72.86
0.98 188 up osd.18
>> >> >> >>>>>
20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529
MiB 0 B 796 GiB 78.63
1.05 66 up osd.20
>> >> >> >>>>>
21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1
GiB 0 B 1.1 TiB 70.32
0.94 47 up osd.21
>> >> >> >>>>>
22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882
MiB 0 B 802 GiB 78.47
1.05 58 up osd.22
>> >> >> >>>>>
24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645
MiB 0 B 856 GiB 77.01
1.03 47 up osd.24
>> >> >> >>>>>
-6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4
GiB 111 GiB 15 TiB 75.22
1.01 - host s3db5
>> >> >> >>>>>
0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70
MiB 6.6 GiB 1.3 TiB 65.00
0.87 48 up osd.0
>> >> >> >>>>>
25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3
MiB 6.6 GiB 1.4 TiB 63.86
0.86 41 up osd.25
>> >> >> >>>>>
26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181
MiB 7.6 GiB 862 GiB 77.47
1.04 48 up osd.26
>> >> >> >>>>>
27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0
MiB 6.1 GiB 1.5 TiB 61.00
0.82 48 up osd.27
>> >> >> >>>>>
28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937
MiB 30 GiB 2.8 TiB 81.19
1.09 203 up osd.28
>> >> >> >>>>>
29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536
MiB 26 GiB 3.8 TiB 73.95
0.99 200 up osd.29
>> >> >> >>>>>
30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744
MiB 28 GiB 3.1 TiB 79.07
1.06 207 up osd.30
>> >> >> >>>>>
-7 58.89636 - 59 TiB 44 TiB 44 TiB 14
GiB 122 GiB 14 TiB 75.41
1.01 - host s3db6
>> >> >> >>>>>
32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16
MiB 8.2 GiB 622 GiB 83.74
1.12 65 up osd.32
>> >> >> >>>>>
33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14
MiB 8.1 GiB 740 GiB 80.67
1.08 52 up osd.33
>> >> >> >>>>>
34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449
MiB 7.7 GiB 877 GiB 77.08
1.03 52 up osd.34
>> >> >> >>>>>
35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133
MiB 7.0 GiB 1.4 TiB 62.18
0.83 42 up osd.35
>> >> >> >>>>>
36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544
MiB 26 GiB 4.0 TiB 72.98
0.98 220 up osd.36
>> >> >> >>>>>
37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11
GiB 38 GiB 3.6 TiB 75.30
1.01 200 up osd.37
>> >> >> >>>>>
38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2
GiB 28 GiB 3.3 TiB 77.43
1.04 217 up osd.38
>> >> >> >>>>>
-8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9
GiB 116 GiB 12 TiB 78.98
1.06 - host s3db7
>> >> >> >>>>>
39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19
MiB 8.5 GiB 499 GiB 86.96
1.17 43 up osd.39
>> >> >> >>>>>
40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144
MiB 7.0 GiB 1.2 TiB 68.33
0.92 39 up osd.40
>> >> >> >>>>>
41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218
MiB 7.9 GiB 732 GiB 80.86
1.08 64 up osd.41
>> >> >> >>>>>
42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594
MiB 7.0 GiB 1.2 TiB 67.97
0.91 50 up osd.42
>> >> >> >>>>>
43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564
MiB 28 GiB 2.9 TiB 80.32
1.08 213 up osd.43
>> >> >> >>>>>
44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3
GiB 28 GiB 3.1 TiB 78.59
1.05 198 up osd.44
>> >> >> >>>>>
45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2
GiB 30 GiB 2.8 TiB 81.05
1.09 214 up osd.45
>> >> >> >>>>>
-9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9
GiB 108 GiB 10 TiB 79.75
1.07 - host s3db8
>> >> >> >>>>>
8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472
MiB 15 GiB 1.5 TiB 79.68
1.07 99 up osd.8
>> >> >> >>>>>
16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785
MiB 15 GiB 1.4 TiB 80.25
1.08 97 up osd.16
>> >> >> >>>>>
31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438
MiB 14 GiB 1.8 TiB 75.36
1.01 87 up osd.31
>> >> >> >>>>>
52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844
MiB 15 GiB 1.6 TiB 78.19
1.05 113 up osd.52
>> >> >> >>>>>
53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792
MiB 18 GiB 1.1 TiB 84.46
1.13 109 up osd.53
>> >> >> >>>>>
54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959
MiB 15 GiB 1.7 TiB 76.73
1.03 115 up osd.54
>> >> >> >>>>>
55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699
MiB 16 GiB 1.2 TiB 83.56
1.12 122 up osd.55
>> >> >> >>>>>
-10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7
GiB 100 GiB 12 TiB 76.05
1.02 - host s3db9
>> >> >> >>>>>
56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840
MiB 13 GiB 2.1 TiB 71.06
0.95 105 up osd.56
>> >> >> >>>>>
57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0
GiB 16 GiB 1.2 TiB 83.17
1.12 102 up osd.57
>> >> >> >>>>>
58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43
MiB 15 GiB 1.4 TiB 81.56
1.09 105 up osd.58
>> >> >> >>>>>
59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429
MiB 15 GiB 1.4 TiB 80.64
1.08 94 up osd.59
>> >> >> >>>>>
60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226
MiB 14 GiB 2.0 TiB 73.25
0.98 101 up osd.60
>> >> >> >>>>>
61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1
GiB 12 GiB 2.5 TiB 65.84
0.88 103 up osd.61
>> >> >> >>>>>
62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0
GiB 15 GiB 1.7 TiB 76.83
1.03 126 up osd.62
>> >> >> >>>>>
TOTAL 674 TiB 501 TiB 473 TiB 96
GiB 1.2 TiB 173 TiB 74.57
>> >> >> >>>>>
MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb
Dan van der
Ster <dan(a)vanderster.com>om>:
>> >> >>
>>>>>>
>> >> >> >>>>>> No, increasing num PGs won't help
substantially.
>> >> >> >>>>>>
>> >> >> >>>>>> Can you share the entire output of
ceph osd df tree ?
>> >> >> >>>>>>
>> >> >> >>>>>> Did you already set
>> >> >> >>>>>>
>> >> >> >>>>>> ceph config set mgr
mgr/balancer/upmap_max_deviation 1
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>> ??
>> >> >> >>>>>> And I recommend debug_mgr 4/5 so you
can see some basic
upmap balancer logging.
>> >> >>
>>>>>>
>> >> >> >>>>>> .. Dan
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris
Behrens <bb(a)kervyn.de>
wrote:
>> >> >>
>>>>>>>
>> >> >> >>>>>>> Hello people,
>> >> >> >>>>>>>
>> >> >> >>>>>>> I am still struggeling with the
balancer
>> >> >> >>>>>>> (
https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>> >> >>
>>>>>>> Now I've read some more and might think that I do not
have enough PGs.
>> >> >>
>>>>>>> Currently I have 84OSDs and 1024PGs for the main pool
(3008 total). I
>> >> >>
>>>>>>> have the autoscaler enabled, but I doesn't tell me to
increase the
>> >> >>
>>>>>>> PGs.
>> >> >> >>>>>>>
>> >> >> >>>>>>> What do you think?
>> >> >> >>>>>>>
>> >> >> >>>>>>> --
>> >> >> >>>>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich
diesmal abweichend
>> >> >>
>>>>>>> im groüen Saal.
>> >> >> >>>>>>>
_______________________________________________
>> >> >> >>>>>>> ceph-users mailing list --
ceph-users(a)ceph.io
>> >> >> >>>>>>> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> --
>> >> >> >>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> Die Selbsthilfegruppe "UTF-8-Probleme"
trifft sich diesmal
abweichend im groüen Saal.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft
sich diesmal
abweichend im groüen Saal.
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal
abweichend im groüen Saal.
>> >
>> >
>> >
>> > --
>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
6:12 p.m.
Data goes to *all* PGs uniformly.
Max_avail is limited by the available space on the most full OSD --
you should pay close attention to those and make sure they are moving
in the right direction (decreasing!)
Another point -- IMHO you should aim to get all PGs active+clean
before you add yet another batch of new disks. While there are PGs
backfilling, your osdmaps are accumulating on the mons and osds --
this itself will start to use a lot of space, and active+clean is the
only way to trim the old maps.
-- dan
On Tue, Mar 23, 2021 at 7:05 PM Boris Behrens <bb(a)kervyn.de> wrote:
>
> So,
> doing nothing and wait for the ceph to recover?
>
> In theory there should be enough disk space (more disks arriving tomorrow), but I
fear that there might be an issue, when the backups get exported over night to this s3.
Currently the max_avail lingers around 13TB and I hope, that the data will go to other PGs
than the ones that are currently on filled OSDs.
>
>
>
> Am Di., 23. März 2021 um 18:58 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>
>> Hi,
>>
>> backfill_toofull is not a bad thing when the cluster is really full
>> like yours. You should expect some of the most full OSDs to eventually
>> start decreasing in usage, as the PGs are moved to the new OSDs. Those
>> backfill_toofull states should then resolve themselves as the OSD
>> usage flattens out.
>> Keep an eye on the usage of the backfill_full and nearfull OSDs though
>> -- if they do eventually go above the full_ratio (95% by default),
>> then writes to those OSDs would stop.
>>
>> But if on the other hand you're suffering from lots of slow ops or
>> anything else visible to your users, then you could try to take some
>> actions to slow down the rebalancing. Just let us know if that's the
>> case and we can see about changing osd_max_backfills, some weights or
>> maybe using the upmap-remapped tool.
>>
>> -- Dan
>>
>> On Tue, Mar 23, 2021 at 6:07 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >
>> > Ok, I should have listened to you :)
>> >
>> > In the last week we added more storage but the issue got worse instead.
>> > Today I realized that the PGs were up to 90GB (bytes column in ceph pg ls
said 95705749636), and the balance kept mentioning the 2048 PGs for this pool. We were at
72% utilization (ceph osd df tree, first line) for our cluster and I increased the PGs to
2048.
>> >
>> > Now I am in a world of trouble.
>> > The space in the cluster went down, I am at 45% misplaced objects, and we
already added 20x4TB disks just to not go down completly.
>> >
>> > The utilization is still going up and the overall free space in the cluster
seems to go down. This is what my ceph status looks like and now I really need help to get
that thing back to normal:
>> > [root@s3db1 ~]# ceph status
>> > cluster:
>> > id: dca79fff-ffd0-58f4-1cff-82a2feea05f4
>> > health: HEALTH_WARN
>> > 4 backfillfull osd(s)
>> > 17 nearfull osd(s)
>> > 37 pool(s) backfillfull
>> > 13 large omap objects
>> > Low space hindering backfill (add storage if this doesn't
resolve itself): 570 pgs backfill_toofull
>> >
>> > services:
>> > mon: 3 daemons, quorum ceph-s3-mon1,ceph-s3-mon2,ceph-s3-mon3 (age 44m)
>> > mgr: ceph-mgr2(active, since 15m), standbys: ceph-mgr3, ceph-mgr1
>> > mds: 3 up:standby
>> > osd: 110 osds: 110 up (since 28m), 110 in (since 28m); 1535 remapped
pgs
>> > rgw: 3 daemons active (eu-central-1, eu-msg-1, eu-secure-1)
>> >
>> > task status:
>> >
>> > data:
>> > pools: 37 pools, 4032 pgs
>> > objects: 116.23M objects, 182 TiB
>> > usage: 589 TiB used, 206 TiB / 795 TiB avail
>> > pgs: 160918554/348689415 objects misplaced (46.150%)
>> > 2497 active+clean
>> > 779 active+remapped+backfill_wait
>> > 538 active+remapped+backfill_wait+backfill_toofull
>> > 186 active+remapped+backfilling
>> > 32 active+remapped+backfill_toofull
>> >
>> > io:
>> > client: 27 MiB/s rd, 69 MiB/s wr, 497 op/s rd, 153 op/s wr
>> > recovery: 1.5 GiB/s, 922 objects/s
>> >
>> > Am Di., 16. März 2021 um 09:34 Uhr schrieb Boris Behrens
<bb(a)kervyn.de>de>:
>> >>
>> >> Hi Dan,
>> >>
>> >> my EC profile look very "default" to me.
>> >> [root@s3db1 ~]# ceph osd erasure-code-profile ls
>> >> default
>> >> [root@s3db1 ~]# ceph osd erasure-code-profile get default
>> >> k=2
>> >> m=1
>> >> plugin=jerasure
>> >> technique=reed_sol_van
>> >>
>> >> I don't understand the ouput, but the balancing get worse over
night:
>> >>
>> >> [root@s3db1 ~]# ceph-scripts/tools/ceph-pool-pg-distribution 11
>> >> Searching for PGs in pools: ['11']
>> >> Summary: 1024 PGs on 84 osds
>> >>
>> >> Num OSDs with X PGs:
>> >> 15: 8
>> >> 16: 7
>> >> 17: 6
>> >> 18: 10
>> >> 19: 1
>> >> 32: 10
>> >> 33: 4
>> >> 34: 6
>> >> 35: 8
>> >> 65: 5
>> >> 66: 5
>> >> 67: 4
>> >> 68: 10
>> >> [root@s3db1 ~]# ceph-scripts/tools/ceph-pg-histogram --normalize
--pool=11
>> >> # NumSamples = 84; Min = 4.12; Max = 5.09
>> >> # Mean = 4.553355; Variance = 0.052415; SD = 0.228942; Median 4.561608
>> >> # each ∎ represents a count of 1
>> >> 4.1244 - 4.2205 [ 8]: ∎∎∎∎∎∎∎∎
>> >> 4.2205 - 4.3166 [ 6]: ∎∎∎∎∎∎
>> >> 4.3166 - 4.4127 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
>> >> 4.4127 - 4.5087 [ 10]: ∎∎∎∎∎∎∎∎∎∎
>> >> 4.5087 - 4.6048 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
>> >> 4.6048 - 4.7009 [ 19]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
>> >> 4.7009 - 4.7970 [ 6]: ∎∎∎∎∎∎
>> >> 4.7970 - 4.8931 [ 8]: ∎∎∎∎∎∎∎∎
>> >> 4.8931 - 4.9892 [ 4]: ∎∎∎∎
>> >> 4.9892 - 5.0852 [ 1]: ∎
>> >> [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail
>> >> 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 724 GiB 19 GiB 0 B
724 GiB 80.56 1.07 56 up osd.14
>> >> 19 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 466 MiB 7.9 GiB
708 GiB 81.25 1.08 53 up osd.19
>> >> 4 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 698 GiB 703 MiB 0 B
698 GiB 81.27 1.08 48 up osd.4
>> >> 24 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 695 GiB 640 MiB 0 B
695 GiB 81.34 1.08 46 up osd.24
>> >> 75 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 440 MiB 8.1 GiB
704 GiB 81.35 1.08 48 up osd.75
>> >> 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 7.5 MiB 8.0 GiB
663 GiB 82.44 1.09 47 up osd.71
>> >> 76 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 251 MiB 9.0 GiB
617 GiB 83.65 1.11 50 up osd.76
>> >> 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 399 MiB 8.1 GiB
618 GiB 83.85 1.11 55 up osd.33
>> >> 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 317 MiB 8.8 GiB
617 GiB 83.87 1.11 50 up osd.35
>> >> 34 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 451 MiB 8.7 GiB
545 GiB 85.75 1.14 54 up osd.34
>> >>
>> >> Am Mo., 15. März 2021 um 17:23 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >>>
>> >>> Hi,
>> >>>
>> >>> How wide are your EC profiles? If they are really wide, you might
be
>> >>> reaching the limits of what is physically possible. Also, I'm
not sure
>> >>> that upmap in 14.2.11 is very smart about *improving* existing
upmap
>> >>> rules for a given PG, in the case that a PG already has an
upmap-items
>> >>> entry but it would help the distribution to add more mapping pairs
to
>> >>> that entry. What this means, is that it might sometimes be useful
to
>> >>> randomly remove some upmap entries and see if the balancer does a
>> >>> better job when it replaces them.
>> >>>
>> >>> But before you do that, I re-remembered that looking at the total
PG
>> >>> numbers is not useful -- you need to check the PGs per OSD for the
>> >>> eu-central-1.rgw.buckets.data pool only.
>> >>>
>> >>> We have a couple tools that can help with this:
>> >>>
>> >>> 1. To see the PGs per OSD for a given pool:
>> >>>
https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis…
>> >>>
>> >>> E.g.: ./ceph-pool-pg-distribution 11 # to see the distribution
of
>> >>> your eu-central-1.rgw.buckets.data pool.
>> >>>
>> >>> The output looks like this on my well balanced clusters:
>> >>>
>> >>> # ceph-scripts/tools/ceph-pool-pg-distribution 15
>> >>> Searching for PGs in pools: ['15']
>> >>> Summary: 256 pgs on 56 osds
>> >>>
>> >>> Num OSDs with X PGs:
>> >>> 13: 16
>> >>> 14: 40
>> >>>
>> >>> You should expect a trimodal for your cluster.
>> >>>
>> >>> 2. You can also use another script from that repo to see the PGs
per
>> >>> OSD normalized to crush weight:
>> >>> ceph-scripts/tools/ceph-pg-histogram --normalize --pool=15
>> >>>
>> >>> This might explain what is going wrong.
>> >>>
>> >>> Cheers, Dan
>> >>>
>> >>>
>> >>> On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens <bb(a)kervyn.de>
wrote:
>> >>> >
>> >>> > Absolutly:
>> >>> > [root@s3db1 ~]# ceph osd df tree
>> >>> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
>> >>> > -1 673.54224 - 674 TiB 496 TiB 468 TiB 97 GiB
1.2 TiB 177 TiB 73.67 1.00 - root default
>> >>> > -2 58.30331 - 58 TiB 42 TiB 38 TiB 9.2 GiB
99 GiB 16 TiB 72.88 0.99 - host s3db1
>> >>> > 23 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 714 MiB
25 GiB 3.7 TiB 74.87 1.02 194 up osd.23
>> >>> > 69 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.6 GiB
40 GiB 3.4 TiB 76.32 1.04 199 up osd.69
>> >>> > 73 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.3 GiB
34 GiB 3.8 TiB 74.15 1.01 203 up osd.73
>> >>> > 79 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.8 GiB
0 B 1.3 TiB 65.44 0.89 47 up osd.79
>> >>> > 80 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 2.2 GiB
0 B 1.3 TiB 65.34 0.89 48 up osd.80
>> >>> > 81 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.1 GiB
0 B 1.3 TiB 65.38 0.89 47 up osd.81
>> >>> > 82 hdd 3.63689 1.00000 3.6 TiB 2.5 TiB 1.1 TiB 619 MiB
0 B 1.1 TiB 68.46 0.93 41 up osd.82
>> >>> > -11 50.94173 - 51 TiB 37 TiB 37 TiB 3.5 GiB
98 GiB 14 TiB 71.90 0.98 - host s3db10
>> >>> > 63 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 647 MiB
14 GiB 2.0 TiB 72.72 0.99 94 up osd.63
>> >>> > 64 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.2 TiB 668 MiB
14 GiB 2.0 TiB 72.23 0.98 93 up osd.64
>> >>> > 65 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 227 MiB
14 GiB 2.1 TiB 71.16 0.97 100 up osd.65
>> >>> > 66 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 313 MiB
14 GiB 1.9 TiB 74.25 1.01 92 up osd.66
>> >>> > 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 584 MiB
14 GiB 2.1 TiB 70.63 0.96 96 up osd.67
>> >>> > 68 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 720 MiB
14 GiB 2.1 TiB 71.72 0.97 101 up osd.68
>> >>> > 70 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 425 MiB
14 GiB 2.1 TiB 70.59 0.96 97 up osd.70
>> >>> > -12 50.99052 - 51 TiB 38 TiB 37 TiB 2.1 GiB
97 GiB 13 TiB 73.77 1.00 - host s3db11
>> >>> > 46 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 229 MiB
14 GiB 1.7 TiB 77.05 1.05 97 up osd.46
>> >>> > 47 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 159 MiB
13 GiB 2.2 TiB 70.00 0.95 89 up osd.47
>> >>> > 48 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 279 MiB
14 GiB 2.1 TiB 71.82 0.97 98 up osd.48
>> >>> > 49 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 276 MiB
14 GiB 1.8 TiB 74.90 1.02 95 up osd.49
>> >>> > 50 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 336 MiB
14 GiB 2.0 TiB 72.13 0.98 93 up osd.50
>> >>> > 51 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 728 MiB
15 GiB 1.6 TiB 77.76 1.06 98 up osd.51
>> >>> > 72 hdd 7.32619 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 147 MiB
13 GiB 2.0 TiB 72.75 0.99 95 up osd.72
>> >>> > -37 58.55478 - 59 TiB 44 TiB 44 TiB 4.4 GiB
122 GiB 15 TiB 75.20 1.02 - host s3db12
>> >>> > 19 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 454 MiB
8.2 GiB 780 GiB 79.35 1.08 53 up osd.19
>> >>> > 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 7.1 MiB
8.0 GiB 734 GiB 80.56 1.09 47 up osd.71
>> >>> > 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 439 MiB
8.2 GiB 777 GiB 79.43 1.08 48 up osd.75
>> >>> > 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 241 MiB
8.9 GiB 688 GiB 81.77 1.11 50 up osd.76
>> >>> > 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB
30 GiB 3.6 TiB 75.44 1.02 201 up osd.77
>> >>> > 78 hdd 14.60159 1.00000 15 TiB 10 TiB 10 TiB 1015 MiB
28 GiB 4.2 TiB 71.26 0.97 193 up osd.78
>> >>> > 83 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.4 GiB
30 GiB 3.8 TiB 73.76 1.00 203 up osd.83
>> >>> > -3 58.49872 - 58 TiB 42 TiB 36 TiB 8.2 GiB
89 GiB 17 TiB 71.71 0.97 - host s3db2
>> >>> > 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.2 GiB
37 GiB 3.7 TiB 74.58 1.01 196 up osd.1
>> >>> > 3 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 566 MiB
0 B 1.3 TiB 64.11 0.87 50 up osd.3
>> >>> > 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 771 GiB 695 MiB
0 B 771 GiB 79.30 1.08 48 up osd.4
>> >>> > 5 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 482 MiB
0 B 1.2 TiB 66.51 0.90 49 up osd.5
>> >>> > 6 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 1.8 GiB
0 B 1.3 TiB 64.00 0.87 42 up osd.6
>> >>> > 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 639 MiB
26 GiB 4.0 TiB 72.44 0.98 192 up osd.7
>> >>> > 74 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 907 MiB
26 GiB 4.2 TiB 71.32 0.97 193 up osd.74
>> >>> > -4 58.49872 - 58 TiB 43 TiB 36 TiB 34 GiB
85 GiB 16 TiB 72.69 0.99 - host s3db3
>> >>> > 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 980 MiB
26 GiB 3.8 TiB 74.36 1.01 203 up osd.2
>> >>> > 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.4 GiB
33 GiB 3.9 TiB 73.51 1.00 186 up osd.9
>> >>> > 10 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 650 MiB
26 GiB 4.2 TiB 71.64 0.97 201 up osd.10
>> >>> > 12 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 754 MiB
0 B 1.3 TiB 64.17 0.87 44 up osd.12
>> >>> > 13 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 813 GiB 2.4 GiB
0 B 813 GiB 78.17 1.06 58 up osd.13
>> >>> > 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 797 GiB 19 GiB
0 B 797 GiB 78.60 1.07 56 up osd.14
>> >>> > 15 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 2.2 GiB
0 B 1.3 TiB 63.96 0.87 41 up osd.15
>> >>> > -5 58.49872 - 58 TiB 43 TiB 36 TiB 6.7 GiB
97 GiB 15 TiB 74.04 1.01 - host s3db4
>> >>> > 11 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 940 MiB
26 GiB 4.0 TiB 72.49 0.98 196 up osd.11
>> >>> > 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1022 MiB
26 GiB 3.6 TiB 75.23 1.02 204 up osd.17
>> >>> > 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 945 MiB
45 GiB 3.8 TiB 74.16 1.01 193 up osd.18
>> >>> > 20 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1020 GiB 596 MiB
0 B 1020 GiB 72.62 0.99 57 up osd.20
>> >>> > 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 1.9 GiB
0 B 1023 GiB 72.54 0.98 41 up osd.21
>> >>> > 22 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 797 MiB
0 B 1023 GiB 72.54 0.98 53 up osd.22
>> >>> > 24 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 766 GiB 618 MiB
0 B 766 GiB 79.42 1.08 46 up osd.24
>> >>> > -6 58.89636 - 59 TiB 43 TiB 43 TiB 3.0 GiB
108 GiB 16 TiB 73.40 1.00 - host s3db5
>> >>> > 0 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 92 MiB
7.2 GiB 1.1 TiB 71.16 0.97 45 up osd.0
>> >>> > 25 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 2.4 MiB
7.3 GiB 1.1 TiB 71.23 0.97 41 up osd.25
>> >>> > 26 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 181 MiB
7.6 GiB 935 GiB 75.57 1.03 45 up osd.26
>> >>> > 27 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 5.1 MiB
7.0 GiB 1.1 TiB 71.20 0.97 47 up osd.27
>> >>> > 28 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 977 MiB
26 GiB 3.8 TiB 73.85 1.00 197 up osd.28
>> >>> > 29 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 872 MiB
26 GiB 4.1 TiB 71.98 0.98 196 up osd.29
>> >>> > 30 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 943 MiB
27 GiB 3.6 TiB 75.51 1.03 202 up osd.30
>> >>> > -7 58.89636 - 59 TiB 44 TiB 43 TiB 13 GiB
122 GiB 15 TiB 74.97 1.02 - host s3db6
>> >>> > 32 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 27 MiB
7.6 GiB 940 GiB 75.42 1.02 55 up osd.32
>> >>> > 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 376 MiB
8.2 GiB 691 GiB 81.94 1.11 55 up osd.33
>> >>> > 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 450 MiB
8.5 GiB 620 GiB 83.79 1.14 54 up osd.34
>> >>> > 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 316 MiB
8.4 GiB 690 GiB 81.98 1.11 50 up osd.35
>> >>> > 36 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 489 MiB
25 GiB 4.1 TiB 71.69 0.97 208 up osd.36
>> >>> > 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB
38 GiB 4.0 TiB 72.41 0.98 195 up osd.37
>> >>> > 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB
26 GiB 3.7 TiB 74.88 1.02 204 up osd.38
>> >>> > -8 58.89636 - 59 TiB 44 TiB 43 TiB 3.8 GiB
111 GiB 15 TiB 74.16 1.01 - host s3db7
>> >>> > 39 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 19 MiB
7.5 GiB 936 GiB 75.54 1.03 39 up osd.39
>> >>> > 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB
7.1 GiB 1.1 TiB 69.87 0.95 39 up osd.40
>> >>> > 41 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.7 TiB 219 MiB
7.6 GiB 1011 GiB 73.57 1.00 55 up osd.41
>> >>> > 42 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 593 MiB
7.1 GiB 1.1 TiB 70.02 0.95 47 up osd.42
>> >>> > 43 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 500 MiB
27 GiB 3.7 TiB 74.67 1.01 204 up osd.43
>> >>> > 44 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB
27 GiB 3.7 TiB 74.62 1.01 193 up osd.44
>> >>> > 45 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB
29 GiB 3.6 TiB 75.16 1.02 204 up osd.45
>> >>> > -9 51.28331 - 51 TiB 39 TiB 39 TiB 4.9 GiB
107 GiB 12 TiB 76.50 1.04 - host s3db8
>> >>> > 8 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 474 MiB
14 GiB 1.7 TiB 76.37 1.04 98 up osd.8
>> >>> > 16 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 783 MiB
15 GiB 1.6 TiB 78.39 1.06 100 up osd.16
>> >>> > 31 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 441 MiB
14 GiB 1.6 TiB 77.70 1.05 91 up osd.31
>> >>> > 52 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 939 MiB
14 GiB 1.7 TiB 76.29 1.04 102 up osd.52
>> >>> > 53 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 848 MiB
18 GiB 1.9 TiB 74.30 1.01 98 up osd.53
>> >>> > 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB
16 GiB 1.7 TiB 76.99 1.05 106 up osd.54
>> >>> > 55 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 460 MiB
15 GiB 1.8 TiB 75.46 1.02 105 up osd.55
>> >>> > -10 51.28331 - 51 TiB 37 TiB 37 TiB 3.8 GiB
96 GiB 14 TiB 72.77 0.99 - host s3db9
>> >>> > 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 846 MiB
13 GiB 2.1 TiB 71.16 0.97 104 up osd.56
>> >>> > 57 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 513 MiB
15 GiB 1.7 TiB 76.53 1.04 96 up osd.57
>> >>> > 58 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 604 MiB
13 GiB 2.1 TiB 71.23 0.97 98 up osd.58
>> >>> > 59 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 414 MiB
13 GiB 2.2 TiB 70.03 0.95 88 up osd.59
>> >>> > 60 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 227 MiB
14 GiB 1.8 TiB 75.54 1.03 97 up osd.60
>> >>> > 61 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 456 MiB
13 GiB 2.2 TiB 70.01 0.95 95 up osd.61
>> >>> > 62 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 843 MiB
14 GiB 1.8 TiB 74.93 1.02 110 up osd.62
>> >>> > TOTAL 674 TiB 496 TiB 468 TiB 97 GiB
1.2 TiB 177 TiB 73.67
>> >>> > MIN/MAX VAR: 0.87/1.14 STDDEV: 4.22
>> >>> >
>> >>> > Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >>> >>
>> >>> >> OK thanks. Indeed "prepared 0/10 changes" means
it thinks things are balanced.
>> >>> >> Could you again share the full ceph osd df tree?
>> >>> >>
>> >>> >> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>> >>> >> >
>> >>> >> > Hi Dan,
>> >>> >> >
>> >>> >> > I've set the autoscaler to warn, but it actually
does not warn for now. So not touching it for now.
>> >>> >> >
>> >>> >> > this is what the log says in minute intervals:
>> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/active
>> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/sleep_interval
>> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/begin_time
>> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/end_time
>> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/begin_weekday
>> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/end_weekday
>> >>> >> > 2021-03-15 13:51:00.971 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/pool_ids
>> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer]
Optimize plan auto_2021-03-15_13:51:00
>> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/mode
>> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer]
Mode upmap, max misplaced 0.050000
>> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer]
do_upmap
>> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/upmap_max_iterations
>> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/upmap_max_deviation
>> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer]
pools ['eu-msg-1.rgw.data.root', 'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys', 'eu-central-1.rgw.buckets.index',
'rbd', 'eu-msg-1.rgw.log', 'whitespace-again-2021-03-10_2',
'eu-msg-1.rgw.buckets.index', 'eu-msg-1.rgw.meta',
'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control', '.rgw.root', 'eu-msg-1.rgw.buckets.data',
'default.rgw.control', 'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3', 'default.rgw.log',
'eu-central-1.rgw.meta', 'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
>> >>> >> > 2021-03-15 13:51:01.224 7f307d5fd700 4 mgr[balancer]
prepared 0/10 changes
>> >>> >> >
>> >>> >> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der
Ster <dan(a)vanderster.com>om>:
>> >>> >> >>
>> >>> >> >> I suggest to just disable the autoscaler until
your balancing is understood.
>> >>> >> >>
>> >>> >> >> What does your active mgr log say (with debug_mgr
4/5), grep balancer
>> >>> >> >> /var/log/ceph/ceph-mgr.*.log
>> >>> >> >>
>> >>> >> >> -- Dan
>> >>> >> >>
>> >>> >> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>> >>> >> >> >
>> >>> >> >> > Hi,
>> >>> >> >> > this unfortunally did not solve my problem. I
still have some OSDs that fill up to 85%
>> >>> >> >> >
>> >>> >> >> > According to the logging, the autoscaler
might want to add more PGs to one Bucken and reduce almost all other buckets to 32.
>> >>> >> >> > 2021-03-15 12:19:58.825 7f307f601700 4
mgr[pg_autoscaler] Pool 'eu-central-1.rgw.buckets.data' root_id -1 using
0.705080476146 of space, bias 1.0, pg target 1974.22533321 quantized to 2048 (current
1024)
>> >>> >> >> >
>> >>> >> >> > Why the balancing does not happen is still
nebulous to me.
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb
Dan van der Ster <dan(a)vanderster.com>om>:
>> >>> >> >> >>
>> >>> >> >> >> OK
>> >>> >> >> >> Btw, you might need to fail to a new
mgr... I'm not sure if the current active will read that new config.
>> >>> >> >> >>
>> >>> >> >> >> .. dan
>> >>> >> >> >>
>> >>> >> >> >>
>> >>> >> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris
Behrens <bb(a)kervyn.de> wrote:
>> >>> >> >> >>>
>> >>> >> >> >>> Hi,
>> >>> >> >> >>>
>> >>> >> >> >>> ok thanks. I just changed the value
and rewighted everything back to 1. Now I let it sync the weekend and check how it will be
on monday.
>> >>> >> >> >>> We tried to have the systems total
storage balanced as possible. New systems will be with 8TB disks but for the exiting ones
we added 16TB to offset the 4TB disks and we needed a lot of storage fast, because of a DC
move. If you have any recommendations I would be happy to hear them.
>> >>> >> >> >>>
>> >>> >> >> >>> Cheers
>> >>> >> >> >>> Boris
>> >>> >> >> >>>
>> >>> >> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr
schrieb Dan van der Ster <dan(a)vanderster.com>om>:
>> >>> >> >> >>>>
>> >>> >> >> >>>> Thanks.
>> >>> >> >> >>>>
>> >>> >> >> >>>> Decreasing the max deviation to 2
or 1 should help in your case. This option controls when the balancer stops trying to move
PGs around -- by default it stops when the deviation from the mean is 5. Yes this is too
large IMO -- all of our clusters have this set to 1.
>> >>> >> >> >>>>
>> >>> >> >> >>>> And given that you have some OSDs
with more than 200 PGs, you definitely shouldn't increase the num PGs.
>> >>> >> >> >>>>
>> >>> >> >> >>>> But anyway with your mixed device
sizes it might be challenging to make a perfectly uniform distribution. Give it a try with
1 though, and let us know how it goes.
>> >>> >> >> >>>>
>> >>> >> >> >>>> .. Dan
>> >>> >> >> >>>>
>> >>> >> >> >>>>
>> >>> >> >> >>>>
>> >>> >> >> >>>>
>> >>> >> >> >>>>
>> >>> >> >> >>>> On Sat, Mar 13, 2021, 4:11 PM
Boris Behrens <bb(a)kervyn.de> wrote:
>> >>> >> >> >>>>>
>> >>> >> >> >>>>> Hi Dan,
>> >>> >> >> >>>>>
>> >>> >> >> >>>>> upmap_max_deviation is
default (5) in our cluster. Is 1 the recommended deviation?
>> >>> >> >> >>>>>
>> >>> >> >> >>>>> I added the whole ceph osd df
tree, (I need to remove some OSDs and readd them as bluestore with SSD, so 69, 73 and 82
are a bit off now. I also reweighted to try to get the %USE mitigated).
>> >>> >> >> >>>>>
>> >>> >> >> >>>>> I will increase the mgr
debugging to see what is the problem.
>> >>> >> >> >>>>>
>> >>> >> >> >>>>> [root@s3db1 ~]# ceph osd df
tree
>> >>> >> >> >>>>> ID CLASS WEIGHT REWEIGHT
SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
>> >>> >> >> >>>>> -1 673.54224 -
659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB 168 TiB 74.57 1.00 - root default
>> >>> >> >> >>>>> -2 58.30331 -
44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB 22 TiB 49.82 0.67 - host s3db1
>> >>> >> >> >>>>> 23 hdd 14.65039 1.00000
15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB 13 TiB 12.50 0.17 101 up osd.23
>> >>> >> >> >>>>> 69 hdd 14.55269 0
0 B 0 B 0 B 0 B 0 B 0 B 0 0 11 up osd.69
>> >>> >> >> >>>>> 73 hdd 14.55269 1.00000
15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB 4.2 TiB 71.15 0.95 107 up osd.73
>> >>> >> >> >>>>> 79 hdd 3.63689 1.00000
3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B 747 GiB 79.94 1.07 52 up osd.79
>> >>> >> >> >>>>> 80 hdd 3.63689 1.00000
3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B 1.0 TiB 71.61 0.96 58 up osd.80
>> >>> >> >> >>>>> 81 hdd 3.63689 1.00000
3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B 1.5 TiB 60.07 0.81 55 up osd.81
>> >>> >> >> >>>>> 82 hdd 3.63689 1.00000
3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B 1.7 TiB 52.68 0.71 30 up osd.82
>> >>> >> >> >>>>> -11 50.94173 -
51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB 13 TiB 74.69 1.00 - host s3db10
>> >>> >> >> >>>>> 63 hdd 7.27739 1.00000
7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB 1.7 TiB 76.04 1.02 92 up osd.63
>> >>> >> >> >>>>> 64 hdd 7.27739 1.00000
7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB 1.8 TiB 75.54 1.01 101 up osd.64
>> >>> >> >> >>>>> 65 hdd 7.27739 1.00000
7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB 2.0 TiB 73.17 0.98 105 up osd.65
>> >>> >> >> >>>>> 66 hdd 7.27739 1.00000
7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB 1.4 TiB 80.38 1.08 98 up osd.66
>> >>> >> >> >>>>> 67 hdd 7.27739 1.00000
7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB 2.2 TiB 70.10 0.94 100 up osd.67
>> >>> >> >> >>>>> 68 hdd 7.27739 1.00000
7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB 2.0 TiB 72.88 0.98 107 up osd.68
>> >>> >> >> >>>>> 70 hdd 7.27739 1.00000
7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB 1.8 TiB 74.73 1.00 102 up osd.70
>> >>> >> >> >>>>> -12 50.99052 -
51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB 12 TiB 77.24 1.04 - host s3db11
>> >>> >> >> >>>>> 46 hdd 7.27739 1.00000
7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB 1.5 TiB 78.91 1.06 97 up osd.46
>> >>> >> >> >>>>> 47 hdd 7.27739 1.00000
7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB 2.1 TiB 71.47 0.96 96 up osd.47
>> >>> >> >> >>>>> 48 hdd 7.27739 1.00000
7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB 1.2 TiB 83.46 1.12 109 up osd.48
>> >>> >> >> >>>>> 49 hdd 7.27739 1.00000
7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB 1.5 TiB 78.96 1.06 98 up osd.49
>> >>> >> >> >>>>> 50 hdd 7.27739 1.00000
7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB 1.4 TiB 80.40 1.08 102 up osd.50
>> >>> >> >> >>>>> 51 hdd 7.27739 1.00000
7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB 1.3 TiB 81.70 1.10 110 up osd.51
>> >>> >> >> >>>>> 72 hdd 7.32619 1.00000
7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB 2.5 TiB 65.82 0.88 89 up osd.72
>> >>> >> >> >>>>> -37 58.55478 -
59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB 12 TiB 79.04 1.06 - host s3db12
>> >>> >> >> >>>>> 19 hdd 3.68750 1.00000
3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB 559 GiB 85.18 1.14 55 up osd.19
>> >>> >> >> >>>>> 71 hdd 3.68750 1.00000
3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB 825 GiB 78.14 1.05 50 up osd.71
>> >>> >> >> >>>>> 75 hdd 3.68750 1.00000
3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB 555 GiB 85.29 1.14 57 up osd.75
>> >>> >> >> >>>>> 76 hdd 3.68750 1.00000
3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB 501 GiB 86.73 1.16 50 up osd.76
>> >>> >> >> >>>>> 77 hdd 14.60159 1.00000
15 TiB 11 TiB 11 TiB 880 MiB 30 GiB 3.6 TiB 75.57 1.01 202 up osd.77
>> >>> >> >> >>>>> 78 hdd 14.60159 1.00000
15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB 3.4 TiB 76.65 1.03 196 up osd.78
>> >>> >> >> >>>>> 83 hdd 14.60159 1.00000
15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB 2.9 TiB 80.04 1.07 223 up osd.83
>> >>> >> >> >>>>> -3 58.49872 -
58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB 16 TiB 73.15 0.98 - host s3db2
>> >>> >> >> >>>>> 1 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB 3.6 TiB 75.52 1.01 194 up osd.1
>> >>> >> >> >>>>> 3 hdd 3.63689 1.00000
3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B 1.4 TiB 60.94 0.82 52 up osd.3
>> >>> >> >> >>>>> 4 hdd 3.63689 0.89999
3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B 401 GiB 89.23 1.20 53 up osd.4
>> >>> >> >> >>>>> 5 hdd 3.63689 1.00000
3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B 1.3 TiB 62.88 0.84 51 up osd.5
>> >>> >> >> >>>>> 6 hdd 3.63689 1.00000
3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B 1.7 TiB 54.51 0.73 47 up osd.6
>> >>> >> >> >>>>> 7 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 493 MiB 26 GiB 3.8 TiB 73.90 0.99 185 up osd.7
>> >>> >> >> >>>>> 74 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 3.5 TiB 76.27 1.02 208 up osd.74
>> >>> >> >> >>>>> -4 58.49872 -
58 TiB 43 TiB 37 TiB 33 GiB 86 GiB 15 TiB 74.05 0.99 - host s3db3
>> >>> >> >> >>>>> 2 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 850 MiB 26 GiB 4.0 TiB 72.78 0.98 203 up osd.2
>> >>> >> >> >>>>> 9 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB 3.6 TiB 75.62 1.01 189 up osd.9
>> >>> >> >> >>>>> 10 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 663 MiB 28 GiB 3.5 TiB 76.34 1.02 211 up osd.10
>> >>> >> >> >>>>> 12 hdd 3.63689 1.00000
3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B 1.2 TiB 66.22 0.89 44 up osd.12
>> >>> >> >> >>>>> 13 hdd 3.63689 1.00000
3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B 720 GiB 80.66 1.08 66 up osd.13
>> >>> >> >> >>>>> 14 hdd 3.63689 1.00000
3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B 552 GiB 85.18 1.14 60 up osd.14
>> >>> >> >> >>>>> 15 hdd 3.63689 1.00000
3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B 1.7 TiB 53.72 0.72 44 up osd.15
>> >>> >> >> >>>>> -5 58.49872 -
58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB 14 TiB 76.37 1.02 - host s3db4
>> >>> >> >> >>>>> 11 hdd 14.65039 1.00000
15 TiB 12 TiB 12 TiB 897 MiB 28 GiB 2.8 TiB 81.15 1.09 205 up osd.11
>> >>> >> >> >>>>> 17 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB 3.6 TiB 75.38 1.01 211 up osd.17
>> >>> >> >> >>>>> 18 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 965 MiB 44 GiB 4.0 TiB 72.86 0.98 188 up osd.18
>> >>> >> >> >>>>> 20 hdd 3.63689 1.00000
3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B 796 GiB 78.63 1.05 66 up osd.20
>> >>> >> >> >>>>> 21 hdd 3.63689 1.00000
3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B 1.1 TiB 70.32 0.94 47 up osd.21
>> >>> >> >> >>>>> 22 hdd 3.63689 1.00000
3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B 802 GiB 78.47 1.05 58 up osd.22
>> >>> >> >> >>>>> 24 hdd 3.63689 1.00000
3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B 856 GiB 77.01 1.03 47 up osd.24
>> >>> >> >> >>>>> -6 58.89636 -
59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB 15 TiB 75.22 1.01 - host s3db5
>> >>> >> >> >>>>> 0 hdd 3.73630 1.00000
3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB 1.3 TiB 65.00 0.87 48 up osd.0
>> >>> >> >> >>>>> 25 hdd 3.73630 1.00000
3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB 1.4 TiB 63.86 0.86 41 up osd.25
>> >>> >> >> >>>>> 26 hdd 3.73630 1.00000
3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB 862 GiB 77.47 1.04 48 up osd.26
>> >>> >> >> >>>>> 27 hdd 3.73630 1.00000
3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB 1.5 TiB 61.00 0.82 48 up osd.27
>> >>> >> >> >>>>> 28 hdd 14.65039 1.00000
15 TiB 12 TiB 12 TiB 937 MiB 30 GiB 2.8 TiB 81.19 1.09 203 up osd.28
>> >>> >> >> >>>>> 29 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 536 MiB 26 GiB 3.8 TiB 73.95 0.99 200 up osd.29
>> >>> >> >> >>>>> 30 hdd 14.65039 1.00000
15 TiB 12 TiB 11 TiB 744 MiB 28 GiB 3.1 TiB 79.07 1.06 207 up osd.30
>> >>> >> >> >>>>> -7 58.89636 -
59 TiB 44 TiB 44 TiB 14 GiB 122 GiB 14 TiB 75.41 1.01 - host s3db6
>> >>> >> >> >>>>> 32 hdd 3.73630 1.00000
3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB 622 GiB 83.74 1.12 65 up osd.32
>> >>> >> >> >>>>> 33 hdd 3.73630 0.79999
3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB 740 GiB 80.67 1.08 52 up osd.33
>> >>> >> >> >>>>> 34 hdd 3.73630 0.79999
3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB 877 GiB 77.08 1.03 52 up osd.34
>> >>> >> >> >>>>> 35 hdd 3.73630 0.79999
3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB 1.4 TiB 62.18 0.83 42 up osd.35
>> >>> >> >> >>>>> 36 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 544 MiB 26 GiB 4.0 TiB 72.98 0.98 220 up osd.36
>> >>> >> >> >>>>> 37 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 11 GiB 38 GiB 3.6 TiB 75.30 1.01 200 up osd.37
>> >>> >> >> >>>>> 38 hdd 14.65039 1.00000
15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB 3.3 TiB 77.43 1.04 217 up osd.38
>> >>> >> >> >>>>> -8 58.89636 -
59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB 12 TiB 78.98 1.06 - host s3db7
>> >>> >> >> >>>>> 39 hdd 3.73630 1.00000
3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB 499 GiB 86.96 1.17 43 up osd.39
>> >>> >> >> >>>>> 40 hdd 3.73630 1.00000
3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB 1.2 TiB 68.33 0.92 39 up osd.40
>> >>> >> >> >>>>> 41 hdd 3.73630 1.00000
3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB 732 GiB 80.86 1.08 64 up osd.41
>> >>> >> >> >>>>> 42 hdd 3.73630 1.00000
3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB 1.2 TiB 67.97 0.91 50 up osd.42
>> >>> >> >> >>>>> 43 hdd 14.65039 1.00000
15 TiB 12 TiB 12 TiB 564 MiB 28 GiB 2.9 TiB 80.32 1.08 213 up osd.43
>> >>> >> >> >>>>> 44 hdd 14.65039 1.00000
15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB 3.1 TiB 78.59 1.05 198 up osd.44
>> >>> >> >> >>>>> 45 hdd 14.65039 1.00000
15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB 2.8 TiB 81.05 1.09 214 up osd.45
>> >>> >> >> >>>>> -9 51.28331 -
51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB 10 TiB 79.75 1.07 - host s3db8
>> >>> >> >> >>>>> 8 hdd 7.32619 1.00000
7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB 1.5 TiB 79.68 1.07 99 up osd.8
>> >>> >> >> >>>>> 16 hdd 7.32619 1.00000
7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB 1.4 TiB 80.25 1.08 97 up osd.16
>> >>> >> >> >>>>> 31 hdd 7.32619 1.00000
7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB 1.8 TiB 75.36 1.01 87 up osd.31
>> >>> >> >> >>>>> 52 hdd 7.32619 1.00000
7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB 1.6 TiB 78.19 1.05 113 up osd.52
>> >>> >> >> >>>>> 53 hdd 7.32619 1.00000
7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB 1.1 TiB 84.46 1.13 109 up osd.53
>> >>> >> >> >>>>> 54 hdd 7.32619 1.00000
7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB 1.7 TiB 76.73 1.03 115 up osd.54
>> >>> >> >> >>>>> 55 hdd 7.32619 1.00000
7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB 1.2 TiB 83.56 1.12 122 up osd.55
>> >>> >> >> >>>>> -10 51.28331 -
51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB 12 TiB 76.05 1.02 - host s3db9
>> >>> >> >> >>>>> 56 hdd 7.32619 1.00000
7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB 2.1 TiB 71.06 0.95 105 up osd.56
>> >>> >> >> >>>>> 57 hdd 7.32619 1.00000
7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB 1.2 TiB 83.17 1.12 102 up osd.57
>> >>> >> >> >>>>> 58 hdd 7.32619 1.00000
7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB 1.4 TiB 81.56 1.09 105 up osd.58
>> >>> >> >> >>>>> 59 hdd 7.32619 1.00000
7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB 1.4 TiB 80.64 1.08 94 up osd.59
>> >>> >> >> >>>>> 60 hdd 7.32619 1.00000
7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB 2.0 TiB 73.25 0.98 101 up osd.60
>> >>> >> >> >>>>> 61 hdd 7.32619 1.00000
7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB 2.5 TiB 65.84 0.88 103 up osd.61
>> >>> >> >> >>>>> 62 hdd 7.32619 1.00000
7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB 1.7 TiB 76.83 1.03 126 up osd.62
>> >>> >> >> >>>>> TOTAL
674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB 173 TiB 74.57
>> >>> >> >> >>>>> MIN/MAX VAR: 0.17/1.20
STDDEV: 10.25
>> >>> >> >> >>>>>
>> >>> >> >> >>>>>
>> >>> >> >> >>>>>
>> >>> >> >> >>>>> Am Sa., 13. März 2021 um
15:57 Uhr schrieb Dan van der Ster <dan(a)vanderster.com>om>:
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>> No, increasing num PGs
won't help substantially.
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>> Can you share the entire
output of ceph osd df tree ?
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>> Did you already set
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>> ceph config set mgr
mgr/balancer/upmap_max_deviation 1
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>> ??
>> >>> >> >> >>>>>> And I recommend debug_mgr
4/5 so you can see some basic upmap balancer logging.
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>> .. Dan
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>>
>> >>> >> >> >>>>>> On Sat, Mar 13, 2021,
3:49 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >>> >> >> >>>>>>>
>> >>> >> >> >>>>>>> Hello people,
>> >>> >> >> >>>>>>>
>> >>> >> >> >>>>>>> I am still
struggeling with the balancer
>> >>> >> >> >>>>>>>
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>> >>> >> >> >>>>>>> Now I've read
some more and might think that I do not have enough PGs.
>> >>> >> >> >>>>>>> Currently I have
84OSDs and 1024PGs for the main pool (3008 total). I
>> >>> >> >> >>>>>>> have the autoscaler
enabled, but I doesn't tell me to increase the
>> >>> >> >> >>>>>>> PGs.
>> >>> >> >> >>>>>>>
>> >>> >> >> >>>>>>> What do you think?
>> >>> >> >> >>>>>>>
>> >>> >> >> >>>>>>> --
>> >>> >> >> >>>>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend
>> >>> >> >> >>>>>>> im groüen Saal.
>> >>> >> >> >>>>>>>
_______________________________________________
>> >>> >> >> >>>>>>> ceph-users mailing
list -- ceph-users(a)ceph.io
>> >>> >> >> >>>>>>> To unsubscribe send
an email to ceph-users-leave(a)ceph.io
>> >>> >> >> >>>>>
>> >>> >> >> >>>>>
>> >>> >> >> >>>>>
>> >>> >> >> >>>>> --
>> >>> >> >> >>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
>> >>> >> >> >>>
>> >>> >> >> >>>
>> >>> >> >> >>>
>> >>> >> >> >>> --
>> >>> >> >> >>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> > --
>> >>> >> >> > Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > --
>> >>> >> > Die Selbsthilfegruppe "UTF-8-Probleme"
trifft sich diesmal abweichend im groüen Saal.
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal abweichend im groüen Saal.
>> >>
>> >>
>> >>
>> >> --
>> >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>> >
>> >
>> >
>> > --
>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
6:16 p.m.
Ok, then I will try to reweight the most filled OSDs to .95 and see if this
helps.
Am Di., 23. März 2021 um 19:13 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
Data goes to *all* PGs uniformly.
Max_avail is limited by the available space on the most full OSD --
you should pay close attention to those and make sure they are moving
in the right direction (decreasing!)
Another point -- IMHO you should aim to get all PGs active+clean
before you add yet another batch of new disks. While there are PGs
backfilling, your osdmaps are accumulating on the mons and osds --
this itself will start to use a lot of space, and active+clean is the
only way to trim the old maps.
-- dan
On Tue, Mar 23, 2021 at 7:05 PM Boris Behrens <bb(a)kervyn.de> wrote:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
So,
doing nothing and wait for the ceph to recover?
In theory there should be enough disk space (more disks arriving
tomorrow), but I
fear that there might be an issue, when the backups get
exported over night to this s3. Currently the max_avail lingers around 13TB
and I hope, that the data will go to other PGs than the ones that are
currently on filled OSDs.
Am Di., 23. März 2021 um 18:58 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
>
> Hi,
>
> backfill_toofull is not a bad thing when the cluster is really full
> like yours. You should expect some of the most full OSDs to eventually
> start decreasing in usage, as the PGs are moved to the new OSDs. Those
> backfill_toofull states should then resolve themselves as the OSD
> usage flattens out.
> Keep an eye on the usage of the backfill_full and nearfull OSDs though
> -- if they do eventually go above the full_ratio (95% by default),
> then writes to those OSDs would stop.
>
> But if on the other hand you're suffering from lots of slow ops or
> anything else visible to your users, then you could try to take some
> actions to slow down the rebalancing. Just let us know if that's the
> case and we can see about changing osd_max_backfills, some weights or
> maybe using the upmap-remapped tool.
>
> -- Dan
>
> On Tue, Mar 23, 2021 at 6:07 PM Boris Behrens <bb(a)kervyn.de> wrote:
> >
> > Ok, I should have listened to you :)
> >
> > In the last week we added more storage but the issue got worse
instead.
> > Today I realized that the PGs were up to
90GB (bytes column in ceph
pg ls said 95705749636), and the balance kept mentioning
the 2048 PGs for
this pool. We were at 72% utilization (ceph osd df tree, first line) for
our cluster and I increased the PGs to 2048.
> >
> > Now I am in a world of trouble.
> > The space in the cluster went down, I am at 45% misplaced objects,
and we
already added 20x4TB disks just to not go down completly.
> >
> > The utilization is still going up and the overall free space in the
cluster seems to go down. This is what my ceph status looks like and now I
really need help to get that thing back to normal:
> > [root@s3db1 ~]# ceph status
> > cluster:
> > id: dca79fff-ffd0-58f4-1cff-82a2feea05f4
> > health: HEALTH_WARN
> > 4 backfillfull osd(s)
> > 17 nearfull osd(s)
> > 37 pool(s) backfillfull
> > 13 large omap objects
> > Low space hindering backfill (add storage if this doesn't
resolve itself): 570 pgs backfill_toofull
> >
> > services:
> > mon: 3 daemons, quorum ceph-s3-mon1,ceph-s3-mon2,ceph-s3-mon3
(age
44m)
> > mgr: ceph-mgr2(active, since 15m),
standbys: ceph-mgr3, ceph-mgr1
> > mds: 3 up:standby
> > osd: 110 osds: 110 up (since 28m), 110 in (since 28m); 1535
remapped
pgs
> > rgw: 3 daemons active (eu-central-1,
eu-msg-1, eu-secure-1)
> >
> > task status:
> >
> > data:
> > pools: 37 pools, 4032 pgs
> > objects: 116.23M objects, 182 TiB
> > usage: 589 TiB used, 206 TiB / 795 TiB avail
> > pgs: 160918554/348689415 objects misplaced (46.150%)
> > 2497 active+clean
> > 779 active+remapped+backfill_wait
> > 538 active+remapped+backfill_wait+backfill_toofull
> > 186 active+remapped+backfilling
> > 32 active+remapped+backfill_toofull
> >
> > io:
> > client: 27 MiB/s rd, 69 MiB/s wr, 497 op/s rd, 153 op/s wr
> > recovery: 1.5 GiB/s, 922 objects/s
> >
> > Am Di., 16. März 2021 um 09:34 Uhr schrieb Boris Behrens <
bb(a)kervyn.de>gt;:
> >>
> >> Hi Dan,
> >>
> >> my EC profile look very "default" to me.
> >> [root@s3db1 ~]# ceph osd erasure-code-profile ls
> >> default
> >> [root@s3db1 ~]# ceph osd erasure-code-profile get default
> >> k=2
> >> m=1
> >> plugin=jerasure
> >> technique=reed_sol_van
> >>
> >> I don't understand the ouput, but the balancing get worse over night:
> >>
> >> [root@s3db1 ~]# ceph-scripts/tools/ceph-pool-pg-distribution 11
> >> Searching for PGs in pools: ['11']
> >> Summary: 1024 PGs on 84 osds
> >>
> >> Num OSDs with X PGs:
> >> 15: 8
> >> 16: 7
> >> 17: 6
> >> 18: 10
> >> 19: 1
> >> 32: 10
> >> 33: 4
> >> 34: 6
> >> 35: 8
> >> 65: 5
> >> 66: 5
> >> 67: 4
> >> 68: 10
> >> [root@s3db1 ~]# ceph-scripts/tools/ceph-pg-histogram --normalize
--pool=11
> >> # NumSamples = 84; Min = 4.12; Max =
5.09
> >> # Mean = 4.553355; Variance = 0.052415; SD = 0.228942; Median
4.561608
> >> # each ∎ represents a count of 1
> >> 4.1244 - 4.2205 [ 8]: ∎∎∎∎∎∎∎∎
> >> 4.2205 - 4.3166 [ 6]: ∎∎∎∎∎∎
> >> 4.3166 - 4.4127 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
> >> 4.4127 - 4.5087 [ 10]: ∎∎∎∎∎∎∎∎∎∎
> >> 4.5087 - 4.6048 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
> >> 4.6048 - 4.7009 [ 19]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
> >> 4.7009 - 4.7970 [ 6]: ∎∎∎∎∎∎
> >> 4.7970 - 4.8931 [ 8]: ∎∎∎∎∎∎∎∎
> >> 4.8931 - 4.9892 [ 4]: ∎∎∎∎
> >> 4.9892 - 5.0852 [ 1]: ∎
> >> [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail
> >> 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 724 GiB 19 GiB 0
B
724 GiB 80.56 1.07 56 up osd.14
> >> 19 hdd 3.68750 1.00000 3.7 TiB
3.0 TiB 2.9 TiB 466 MiB 7.9
GiB 708 GiB 81.25 1.08 53 up osd.19
> >> 4 hdd 3.63689 1.00000 3.6 TiB
3.0 TiB 698 GiB 703 MiB 0
B 698 GiB 81.27 1.08 48 up osd.4
> >> 24 hdd 3.63689 1.00000 3.6 TiB
3.0 TiB 695 GiB 640 MiB 0
B 695 GiB 81.34 1.08 46 up osd.24
> >> 75 hdd 3.68750 1.00000 3.7 TiB
3.0 TiB 2.9 TiB 440 MiB 8.1
GiB 704 GiB 81.35 1.08 48 up osd.75
> >> 71 hdd 3.68750 1.00000 3.7 TiB
3.0 TiB 3.0 TiB 7.5 MiB 8.0
GiB 663 GiB 82.44 1.09 47 up osd.71
> >> 76 hdd 3.68750 1.00000 3.7 TiB
3.1 TiB 3.0 TiB 251 MiB 9.0
GiB 617 GiB 83.65 1.11 50 up osd.76
> >> 33 hdd 3.73630 1.00000 3.7 TiB
3.1 TiB 3.0 TiB 399 MiB 8.1
GiB 618 GiB 83.85 1.11 55 up osd.33
> >> 35 hdd 3.73630 1.00000 3.7 TiB
3.1 TiB 3.0 TiB 317 MiB 8.8
GiB 617 GiB 83.87 1.11 50 up osd.35
> >> 34 hdd 3.73630 1.00000 3.7 TiB
3.2 TiB 3.1 TiB 451 MiB 8.7
GiB 545 GiB 85.75 1.14 54 up osd.34
> >>
> >> Am Mo., 15. März 2021 um 17:23 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >>>
> >>> Hi,
> >>>
> >>> How wide are your EC profiles? If they are really wide, you might be
> >>> reaching the limits of what is physically possible. Also, I'm not
sure
> >>> that upmap in 14.2.11 is very
smart about *improving* existing upmap
> >>> rules for a given PG, in the case that a PG already has an
upmap-items
> >>> entry but it would help the
distribution to add more mapping pairs
to
> >>> that entry. What this means, is
that it might sometimes be useful to
> >>> randomly remove some upmap entries and see if the balancer does a
> >>> better job when it replaces them.
> >>>
> >>> But before you do that, I re-remembered that looking at the total PG
> >>> numbers is not useful -- you need to check the PGs per OSD for the
> >>> eu-central-1.rgw.buckets.data pool only.
> >>>
> >>> We have a couple tools that can help with this:
> >>>
> >>> 1. To see the PGs per OSD for a given pool:
> >>>
https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis…
> >>>
> >>> E.g.: ./ceph-pool-pg-distribution 11 # to see the distribution
of
> >>> your
eu-central-1.rgw.buckets.data pool.
> >>>
> >>> The output looks like this on my well balanced clusters:
> >>>
> >>> # ceph-scripts/tools/ceph-pool-pg-distribution 15
> >>> Searching for PGs in pools: ['15']
> >>> Summary: 256 pgs on 56 osds
> >>>
> >>> Num OSDs with X PGs:
> >>> 13: 16
> >>> 14: 40
> >>>
> >>> You should expect a trimodal for your cluster.
> >>>
> >>> 2. You can also use another script from that repo to see the PGs per
> >>> OSD normalized to crush weight:
> >>> ceph-scripts/tools/ceph-pg-histogram --normalize --pool=15
> >>>
> >>> This might explain what is going wrong.
> >>>
> >>> Cheers, Dan
> >>>
> >>>
> >>> On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens <bb(a)kervyn.de>
wrote:
> >>> >
> >>> > Absolutly:
> >>> > [root@s3db1 ~]# ceph osd df tree
> >>> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
> >>> > -1 673.54224
- 674 TiB 496 TiB 468 TiB 97 GiB
1.2 TiB 177 TiB 73.67 1.00 - root
default
> >>> > -2 58.30331
- 58 TiB 42 TiB 38 TiB 9.2 GiB
99 GiB 16 TiB 72.88 0.99 - host
s3db1
> >>> > 23 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 714 MiB
25 GiB 3.7 TiB 74.87 1.02 194 up
osd.23
> >>> > 69 hdd 14.55269
1.00000 15 TiB 11 TiB 11 TiB 1.6 GiB
40 GiB 3.4 TiB 76.32 1.04 199 up
osd.69
> >>> > 73 hdd 14.55269
1.00000 15 TiB 11 TiB 11 TiB 1.3 GiB
34 GiB 3.8 TiB 74.15 1.01 203 up
osd.73
> >>> > 79 hdd 3.63689
1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.8 GiB
0 B 1.3 TiB 65.44 0.89 47 up
osd.79
> >>> > 80 hdd 3.63689
1.00000 3.6 TiB 2.4 TiB 1.3 TiB 2.2 GiB
0 B 1.3 TiB 65.34 0.89 48 up
osd.80
> >>> > 81 hdd 3.63689
1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.1 GiB
0 B 1.3 TiB 65.38 0.89 47 up
osd.81
> >>> > 82 hdd 3.63689
1.00000 3.6 TiB 2.5 TiB 1.1 TiB 619 MiB
0 B 1.1 TiB 68.46 0.93 41 up
osd.82
> >>> > -11 50.94173
- 51 TiB 37 TiB 37 TiB 3.5 GiB
98 GiB 14 TiB 71.90 0.98 - host
s3db10
> >>> > 63 hdd 7.27739
1.00000 7.3 TiB 5.3 TiB 5.3 TiB 647 MiB
14 GiB 2.0 TiB 72.72 0.99 94 up
osd.63
> >>> > 64 hdd 7.27739
1.00000 7.3 TiB 5.3 TiB 5.2 TiB 668 MiB
14 GiB 2.0 TiB 72.23 0.98 93 up
osd.64
> >>> > 65 hdd 7.27739
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 227 MiB
14 GiB 2.1 TiB 71.16 0.97 100 up
osd.65
> >>> > 66 hdd 7.27739
1.00000 7.3 TiB 5.4 TiB 5.4 TiB 313 MiB
14 GiB 1.9 TiB 74.25 1.01 92 up
osd.66
> >>> > 67 hdd 7.27739
1.00000 7.3 TiB 5.1 TiB 5.1 TiB 584 MiB
14 GiB 2.1 TiB 70.63 0.96 96 up
osd.67
> >>> > 68 hdd 7.27739
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 720 MiB
14 GiB 2.1 TiB 71.72 0.97 101 up
osd.68
> >>> > 70 hdd 7.27739
1.00000 7.3 TiB 5.1 TiB 5.1 TiB 425 MiB
14 GiB 2.1 TiB 70.59 0.96 97 up
osd.70
> >>> > -12 50.99052
- 51 TiB 38 TiB 37 TiB 2.1 GiB
97 GiB 13 TiB 73.77 1.00 - host
s3db11
> >>> > 46 hdd 7.27739
1.00000 7.3 TiB 5.6 TiB 5.6 TiB 229 MiB
14 GiB 1.7 TiB 77.05 1.05 97 up
osd.46
> >>> > 47 hdd 7.27739
1.00000 7.3 TiB 5.1 TiB 5.1 TiB 159 MiB
13 GiB 2.2 TiB 70.00 0.95 89 up
osd.47
> >>> > 48 hdd 7.27739
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 279 MiB
14 GiB 2.1 TiB 71.82 0.97 98 up
osd.48
> >>> > 49 hdd 7.27739
1.00000 7.3 TiB 5.5 TiB 5.4 TiB 276 MiB
14 GiB 1.8 TiB 74.90 1.02 95 up
osd.49
> >>> > 50 hdd 7.27739
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 336 MiB
14 GiB 2.0 TiB 72.13 0.98 93 up
osd.50
> >>> > 51 hdd 7.27739
1.00000 7.3 TiB 5.7 TiB 5.6 TiB 728 MiB
15 GiB 1.6 TiB 77.76 1.06 98 up
osd.51
> >>> > 72 hdd 7.32619
1.00000 7.3 TiB 5.3 TiB 5.3 TiB 147 MiB
13 GiB 2.0 TiB 72.75 0.99 95 up
osd.72
> >>> > -37 58.55478
- 59 TiB 44 TiB 44 TiB 4.4 GiB
122 GiB 15 TiB 75.20 1.02 - host
s3db12
> >>> > 19 hdd 3.68750
1.00000 3.7 TiB 2.9 TiB 2.9 TiB 454 MiB
8.2 GiB 780 GiB 79.35 1.08 53 up
osd.19
> >>> > 71 hdd 3.68750
1.00000 3.7 TiB 3.0 TiB 2.9 TiB 7.1 MiB
8.0 GiB 734 GiB 80.56 1.09 47 up
osd.71
> >>> > 75 hdd 3.68750
1.00000 3.7 TiB 2.9 TiB 2.9 TiB 439 MiB
8.2 GiB 777 GiB 79.43 1.08 48 up
osd.75
> >>> > 76 hdd 3.68750
1.00000 3.7 TiB 3.0 TiB 3.0 TiB 241 MiB
8.9 GiB 688 GiB 81.77 1.11 50 up
osd.76
> >>> > 77 hdd 14.60159
1.00000 15 TiB 11 TiB 11 TiB 880 MiB
30 GiB 3.6 TiB 75.44 1.02 201 up
osd.77
> >>> > 78 hdd 14.60159
1.00000 15 TiB 10 TiB 10 TiB 1015 MiB
28 GiB 4.2 TiB 71.26 0.97 193 up
osd.78
> >>> > 83 hdd 14.60159
1.00000 15 TiB 11 TiB 11 TiB 1.4 GiB
30 GiB 3.8 TiB 73.76 1.00 203 up
osd.83
> >>> > -3 58.49872
- 58 TiB 42 TiB 36 TiB 8.2 GiB
89 GiB 17 TiB 71.71 0.97 - host
s3db2
> >>> > 1 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 3.2 GiB
37 GiB 3.7 TiB 74.58 1.01 196 up
osd.1
> >>> > 3 hdd 3.63689
1.00000 3.6 TiB 2.3 TiB 1.3 TiB 566 MiB
0 B 1.3 TiB 64.11 0.87 50 up
osd.3
> >>> > 4 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 771 GiB 695 MiB
0 B 771 GiB 79.30 1.08 48 up
osd.4
> >>> > 5 hdd 3.63689
1.00000 3.6 TiB 2.4 TiB 1.2 TiB 482 MiB
0 B 1.2 TiB 66.51 0.90 49 up
osd.5
> >>> > 6 hdd 3.63689
1.00000 3.6 TiB 2.3 TiB 1.3 TiB 1.8 GiB
0 B 1.3 TiB 64.00 0.87 42 up
osd.6
> >>> > 7 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 639 MiB
26 GiB 4.0 TiB 72.44 0.98 192 up
osd.7
> >>> > 74 hdd 14.65039
1.00000 15 TiB 10 TiB 10 TiB 907 MiB
26 GiB 4.2 TiB 71.32 0.97 193 up
osd.74
> >>> > -4 58.49872
- 58 TiB 43 TiB 36 TiB 34 GiB
85 GiB 16 TiB 72.69 0.99 - host
s3db3
> >>> > 2 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 980 MiB
26 GiB 3.8 TiB 74.36 1.01 203 up
osd.2
> >>> > 9 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 8.4 GiB
33 GiB 3.9 TiB 73.51 1.00 186 up
osd.9
> >>> > 10 hdd 14.65039
1.00000 15 TiB 10 TiB 10 TiB 650 MiB
26 GiB 4.2 TiB 71.64 0.97 201 up
osd.10
> >>> > 12 hdd 3.63689
1.00000 3.6 TiB 2.3 TiB 1.3 TiB 754 MiB
0 B 1.3 TiB 64.17 0.87 44 up
osd.12
> >>> > 13 hdd 3.63689
1.00000 3.6 TiB 2.8 TiB 813 GiB 2.4 GiB
0 B 813 GiB 78.17 1.06 58 up
osd.13
> >>> > 14 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 797 GiB 19 GiB
0 B 797 GiB 78.60 1.07 56 up
osd.14
> >>> > 15 hdd 3.63689
1.00000 3.6 TiB 2.3 TiB 1.3 TiB 2.2 GiB
0 B 1.3 TiB 63.96 0.87 41 up
osd.15
> >>> > -5 58.49872
- 58 TiB 43 TiB 36 TiB 6.7 GiB
97 GiB 15 TiB 74.04 1.01 - host
s3db4
> >>> > 11 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 940 MiB
26 GiB 4.0 TiB 72.49 0.98 196 up
osd.11
> >>> > 17 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1022 MiB
26 GiB 3.6 TiB 75.23 1.02 204 up
osd.17
> >>> > 18 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 945 MiB
45 GiB 3.8 TiB 74.16 1.01 193 up
osd.18
> >>> > 20 hdd 3.63689
1.00000 3.6 TiB 2.6 TiB 1020 GiB 596 MiB
0 B 1020 GiB 72.62 0.99 57 up
osd.20
> >>> > 21 hdd 3.63689
1.00000 3.6 TiB 2.6 TiB 1023 GiB 1.9 GiB
0 B 1023 GiB 72.54 0.98 41 up
osd.21
> >>> > 22 hdd 3.63689
1.00000 3.6 TiB 2.6 TiB 1023 GiB 797 MiB
0 B 1023 GiB 72.54 0.98 53 up
osd.22
> >>> > 24 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 766 GiB 618 MiB
0 B 766 GiB 79.42 1.08 46 up
osd.24
> >>> > -6 58.89636
- 59 TiB 43 TiB 43 TiB 3.0 GiB
108 GiB 16 TiB 73.40 1.00 - host
s3db5
> >>> > 0 hdd 3.73630
1.00000 3.7 TiB 2.7 TiB 2.6 TiB 92 MiB
7.2 GiB 1.1 TiB 71.16 0.97 45 up
osd.0
> >>> > 25 hdd 3.73630
1.00000 3.7 TiB 2.7 TiB 2.6 TiB 2.4 MiB
7.3 GiB 1.1 TiB 71.23 0.97 41 up
osd.25
> >>> > 26 hdd 3.73630
1.00000 3.7 TiB 2.8 TiB 2.7 TiB 181 MiB
7.6 GiB 935 GiB 75.57 1.03 45 up
osd.26
> >>> > 27 hdd 3.73630
1.00000 3.7 TiB 2.7 TiB 2.6 TiB 5.1 MiB
7.0 GiB 1.1 TiB 71.20 0.97 47 up
osd.27
> >>> > 28 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 977 MiB
26 GiB 3.8 TiB 73.85 1.00 197 up
osd.28
> >>> > 29 hdd 14.65039
1.00000 15 TiB 11 TiB 10 TiB 872 MiB
26 GiB 4.1 TiB 71.98 0.98 196 up
osd.29
> >>> > 30 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 943 MiB
27 GiB 3.6 TiB 75.51 1.03 202 up
osd.30
> >>> > -7 58.89636
- 59 TiB 44 TiB 43 TiB 13 GiB
122 GiB 15 TiB 74.97 1.02 - host
s3db6
> >>> > 32 hdd 3.73630
1.00000 3.7 TiB 2.8 TiB 2.7 TiB 27 MiB
7.6 GiB 940 GiB 75.42 1.02 55 up
osd.32
> >>> > 33 hdd 3.73630
1.00000 3.7 TiB 3.1 TiB 3.0 TiB 376 MiB
8.2 GiB 691 GiB 81.94 1.11 55 up
osd.33
> >>> > 34 hdd 3.73630
1.00000 3.7 TiB 3.1 TiB 3.0 TiB 450 MiB
8.5 GiB 620 GiB 83.79 1.14 54 up
osd.34
> >>> > 35 hdd 3.73630
1.00000 3.7 TiB 3.1 TiB 3.0 TiB 316 MiB
8.4 GiB 690 GiB 81.98 1.11 50 up
osd.35
> >>> > 36 hdd 14.65039
1.00000 15 TiB 11 TiB 10 TiB 489 MiB
25 GiB 4.1 TiB 71.69 0.97 208 up
osd.36
> >>> > 37 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 11 GiB
38 GiB 4.0 TiB 72.41 0.98 195 up
osd.37
> >>> > 38 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB
26 GiB 3.7 TiB 74.88 1.02 204 up
osd.38
> >>> > -8 58.89636
- 59 TiB 44 TiB 43 TiB 3.8 GiB
111 GiB 15 TiB 74.16 1.01 - host
s3db7
> >>> > 39 hdd 3.73630
1.00000 3.7 TiB 2.8 TiB 2.7 TiB 19 MiB
7.5 GiB 936 GiB 75.54 1.03 39 up
osd.39
> >>> > 40 hdd 3.73630
1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB
7.1 GiB 1.1 TiB 69.87 0.95 39 up
osd.40
> >>> > 41 hdd 3.73630
1.00000 3.7 TiB 2.7 TiB 2.7 TiB 219 MiB
7.6 GiB 1011 GiB 73.57 1.00 55 up
osd.41
> >>> > 42 hdd 3.73630
1.00000 3.7 TiB 2.6 TiB 2.5 TiB 593 MiB
7.1 GiB 1.1 TiB 70.02 0.95 47 up
osd.42
> >>> > 43 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 500 MiB
27 GiB 3.7 TiB 74.67 1.01 204 up
osd.43
> >>> > 44 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB
27 GiB 3.7 TiB 74.62 1.01 193 up
osd.44
> >>> > 45 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB
29 GiB 3.6 TiB 75.16 1.02 204 up
osd.45
> >>> > -9 51.28331
- 51 TiB 39 TiB 39 TiB 4.9 GiB
107 GiB 12 TiB 76.50 1.04 - host
s3db8
> >>> > 8 hdd 7.32619
1.00000 7.3 TiB 5.6 TiB 5.5 TiB 474 MiB
14 GiB 1.7 TiB 76.37 1.04 98 up
osd.8
> >>> > 16 hdd 7.32619
1.00000 7.3 TiB 5.7 TiB 5.7 TiB 783 MiB
15 GiB 1.6 TiB 78.39 1.06 100 up
osd.16
> >>> > 31 hdd 7.32619
1.00000 7.3 TiB 5.7 TiB 5.6 TiB 441 MiB
14 GiB 1.6 TiB 77.70 1.05 91 up
osd.31
> >>> > 52 hdd 7.32619
1.00000 7.3 TiB 5.6 TiB 5.5 TiB 939 MiB
14 GiB 1.7 TiB 76.29 1.04 102 up
osd.52
> >>> > 53 hdd 7.32619
1.00000 7.3 TiB 5.4 TiB 5.4 TiB 848 MiB
18 GiB 1.9 TiB 74.30 1.01 98 up
osd.53
> >>> > 54 hdd 7.32619
1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB
16 GiB 1.7 TiB 76.99 1.05 106 up
osd.54
> >>> > 55 hdd 7.32619
1.00000 7.3 TiB 5.5 TiB 5.5 TiB 460 MiB
15 GiB 1.8 TiB 75.46 1.02 105 up
osd.55
> >>> > -10 51.28331
- 51 TiB 37 TiB 37 TiB 3.8 GiB
96 GiB 14 TiB 72.77 0.99 - host
s3db9
> >>> > 56 hdd 7.32619
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 846 MiB
13 GiB 2.1 TiB 71.16 0.97 104 up
osd.56
> >>> > 57 hdd 7.32619
1.00000 7.3 TiB 5.6 TiB 5.6 TiB 513 MiB
15 GiB 1.7 TiB 76.53 1.04 96 up
osd.57
> >>> > 58 hdd 7.32619
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 604 MiB
13 GiB 2.1 TiB 71.23 0.97 98 up
osd.58
> >>> > 59 hdd 7.32619
1.00000 7.3 TiB 5.1 TiB 5.1 TiB 414 MiB
13 GiB 2.2 TiB 70.03 0.95 88 up
osd.59
> >>> > 60 hdd 7.32619
1.00000 7.3 TiB 5.5 TiB 5.5 TiB 227 MiB
14 GiB 1.8 TiB 75.54 1.03 97 up
osd.60
> >>> > 61 hdd 7.32619
1.00000 7.3 TiB 5.1 TiB 5.1 TiB 456 MiB
13 GiB 2.2 TiB 70.01 0.95 95 up
osd.61
> >>> > 62 hdd 7.32619
1.00000 7.3 TiB 5.5 TiB 5.4 TiB 843 MiB
14 GiB 1.8 TiB 74.93 1.02 110 up
osd.62
> >>> >
TOTAL 674 TiB 496 TiB 468 TiB 97 GiB
1.2 TiB 177 TiB 73.67
> >>> > MIN/MAX VAR: 0.87/1.14
STDDEV: 4.22
> >>> >
> >>> > Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster <
dan(a)vanderster.com>gt;:
> >>> >>
> >>> >> OK thanks. Indeed "prepared 0/10 changes" means it
thinks things
are balanced.
> >>> >> Could you again share
the full ceph osd df tree?
> >>> >>
> >>> >> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
> >>> >> >
> >>> >> > Hi Dan,
> >>> >> >
> >>> >> > I've set the autoscaler to warn, but it actually does
not warn
for now. So not touching it for now.
> >>> >> >
> >>> >> > this is what the log says in minute intervals:
> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/active
> >>> >> > 2021-03-15
13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key:
mgr/balancer/sleep_interval
> >>> >> > 2021-03-15
13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key:
mgr/balancer/begin_time
> >>> >> > 2021-03-15
13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/end_time
> >>> >> > 2021-03-15
13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key:
mgr/balancer/begin_weekday
> >>> >> > 2021-03-15
13:51:00.970 7f307d5fd700 4 mgr get_config
get_config key:
mgr/balancer/end_weekday
> >>> >> > 2021-03-15
13:51:00.971 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/pool_ids
> >>> >> > 2021-03-15
13:51:01.203 7f307d5fd700 4 mgr[balancer] Optimize
plan auto_2021-03-15_13:51:00
> >>> >> > 2021-03-15
13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key: mgr/balancer/mode
> >>> >> > 2021-03-15
13:51:01.203 7f307d5fd700 4 mgr[balancer] Mode
upmap, max misplaced 0.050000
>> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer]
do_upmap
> >>> >> > 2021-03-15
13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key:
mgr/balancer/upmap_max_iterations
> >>> >> > 2021-03-15
13:51:01.203 7f307d5fd700 4 mgr get_config
get_config key:
mgr/balancer/upmap_max_deviation
> >>> >> > 2021-03-15
13:51:01.203 7f307d5fd700 4 mgr[balancer] pools
['eu-msg-1.rgw.data.root',
'eu-msg-1.rgw.buckets.non-ec',
'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc',
'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email',
'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage',
'eu-msg-1.rgw.users.keys',
'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log',
'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index',
'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control',
'.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control',
'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3',
'default.rgw.log', 'eu-central-1.rgw.meta',
'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
> >>> >> > 2021-03-15
13:51:01.224 7f307d5fd700 4 mgr[balancer] prepared
0/10 changes
> >>> >> >
> >>> >> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der
Ster <
dan(a)vanderster.com>gt;:
> >>> >> >>
> >>> >> >> I suggest to just disable the autoscaler until your
balancing
is understood.
> >>> >> >>
> >>> >> >> What does your active mgr log say (with debug_mgr
4/5), grep
balancer
> >>> >> >>
/var/log/ceph/ceph-mgr.*.log
> >>> >> >>
> >>> >> >> -- Dan
> >>> >> >>
> >>> >> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
> >>> >> >> >
> >>> >> >> > Hi,
> >>> >> >> > this unfortunally did not solve my problem. I
still have
some OSDs that fill up to 85%
> >>> >> >> >
> >>> >> >> > According to the logging, the autoscaler might
want to add
more PGs to one Bucken and reduce almost all other buckets to 32.
> >>> >> >> >
2021-03-15 12:19:58.825 7f307f601700 4 mgr[pg_autoscaler]
Pool
'eu-central-1.rgw.buckets.data' root_id -1 using 0.705080476146 of
space, bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024)
> >>> >> >> >
> >>> >> >> > Why the balancing does not happen is still
nebulous to me.
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan
van der Ster
<dan(a)vanderster.com>om>:
> >>> >> >> >>
> >>> >> >> >> OK
> >>> >> >> >> Btw, you might need to fail to a new mgr...
I'm not sure
if the current active will read that new config.
> >>> >> >> >>
> >>> >> >> >> .. dan
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens
<bb(a)kervyn.de>
wrote:
> >>> >> >> >>>
> >>> >> >> >>> Hi,
> >>> >> >> >>>
> >>> >> >> >>> ok thanks. I just changed the value and
rewighted
everything back to 1. Now I let it sync the weekend and check how it
will
be on monday.
> >>> >> >> >>>
We tried to have the systems total storage balanced as
possible. New systems will
be with 8TB disks but for the exiting ones we
added 16TB to offset the 4TB disks and we needed a lot of storage fast,
because of a DC move. If you have any recommendations I would be happy to
hear them.
> >>> >> >> >>>
> >>> >> >> >>> Cheers
> >>> >> >> >>> Boris
> >>> >> >> >>>
> >>> >> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr
schrieb Dan van der
Ster <dan(a)vanderster.com>om>:
> >>> >> >>
>>>>
> >>> >> >> >>>> Thanks.
> >>> >> >> >>>>
> >>> >> >> >>>> Decreasing the max deviation to 2 or
1 should help in
your case. This option controls when the balancer stops trying to
move PGs
around -- by default it stops when the deviation from the mean is 5. Yes
this is too large IMO -- all of our clusters have this set to 1.
> >>> >> >>
>>>>
> >>> >> >> >>>> And given that you have some OSDs
with more than 200
PGs, you definitely shouldn't increase the num PGs.
> >>> >> >>
>>>>
> >>> >> >> >>>> But anyway with your mixed device
sizes it might be
challenging to make a perfectly uniform distribution. Give it a
try with 1
though, and let us know how it goes.
> >>> >> >>
>>>>
> >>> >> >> >>>> .. Dan
> >>> >> >> >>>>
> >>> >> >> >>>>
> >>> >> >> >>>>
> >>> >> >> >>>>
> >>> >> >> >>>>
> >>> >> >> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris
Behrens <
bb(a)kervyn.de> wrote:
> >>> >> >>
>>>>>
> >>> >> >> >>>>> Hi Dan,
> >>> >> >> >>>>>
> >>> >> >> >>>>> upmap_max_deviation is default
(5) in our cluster. Is 1
the recommended deviation?
> >>> >> >>
>>>>>
> >>> >> >> >>>>> I added the whole ceph osd df
tree, (I need to remove
some OSDs and readd them as bluestore with SSD, so 69, 73
and 82 are a bit
off now. I also reweighted to try to get the %USE mitigated).
> >>> >> >>
>>>>>
> >>> >> >> >>>>> I will increase the mgr debugging
to see what is the
problem.
> >>> >> >>
>>>>>
> >>> >> >> >>>>> [root@s3db1 ~]# ceph osd df tree
> >>> >> >> >>>>> ID CLASS WEIGHT REWEIGHT SIZE
RAW USE DATA
OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
> >>> >> >>
>>>>> -1 673.54224 - 659 TiB 491 TiB 464 TiB
96 GiB
1.2 TiB 168 TiB 74.57 1.00 - root default
> >>> >> >>
>>>>> -2 58.30331 - 44 TiB 22 TiB 17 TiB
5.7 GiB
38 GiB 22 TiB 49.82 0.67 - host s3db1
> >>> >> >>
>>>>> 23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB
156 MiB
4.4 GiB 13 TiB 12.50 0.17 101 up osd.23
> >>> >> >>
>>>>> 69 hdd 14.55269 0 0 B 0 B 0 B
0 B
0 B 0 B 0 0 11 up osd.69
> >>> >> >>
>>>>> 73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB
6.1 MiB
33 GiB 4.2 TiB 71.15 0.95 107 up osd.73
> >>> >> >>
>>>>> 79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB
2.0 GiB
0 B 747 GiB 79.94 1.07 52 up osd.79
> >>> >> >>
>>>>> 80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB
1.9 GiB
0 B 1.0 TiB 71.61 0.96 58 up osd.80
> >>> >> >>
>>>>> 81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB
1.1 GiB
0 B 1.5 TiB 60.07 0.81 55 up osd.81
> >>> >> >>
>>>>> 82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB
536 MiB
0 B 1.7 TiB 52.68 0.71 30 up osd.82
> >>> >> >>
>>>>> -11 50.94173 - 51 TiB 38 TiB 38 TiB
3.7 GiB
100 GiB 13 TiB 74.69 1.00 - host s3db10
> >>> >> >>
>>>>> 63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB
616 MiB
14 GiB 1.7 TiB 76.04 1.02 92 up osd.63
> >>> >> >>
>>>>> 64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB
820 MiB
15 GiB 1.8 TiB 75.54 1.01 101 up osd.64
> >>> >> >>
>>>>> 65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB
109 MiB
14 GiB 2.0 TiB 73.17 0.98 105 up osd.65
> >>> >> >>
>>>>> 66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB
423 MiB
15 GiB 1.4 TiB 80.38 1.08 98 up osd.66
> >>> >> >>
>>>>> 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB
572 MiB
14 GiB 2.2 TiB 70.10 0.94 100 up osd.67
> >>> >> >>
>>>>> 68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB
630 MiB
13 GiB 2.0 TiB 72.88 0.98 107 up osd.68
> >>> >> >>
>>>>> 70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB
648 MiB
14 GiB 1.8 TiB 74.73 1.00 102 up osd.70
> >>> >> >>
>>>>> -12 50.99052 - 51 TiB 39 TiB 39 TiB
2.9 GiB
99 GiB 12 TiB 77.24 1.04 - host s3db11
> >>> >> >>
>>>>> 46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB
102 MiB
15 GiB 1.5 TiB 78.91 1.06 97 up osd.46
> >>> >> >>
>>>>> 47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB
61 MiB
13 GiB 2.1 TiB 71.47 0.96 96 up osd.47
> >>> >> >>
>>>>> 48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB
853 MiB
15 GiB 1.2 TiB 83.46 1.12 109 up osd.48
> >>> >> >>
>>>>> 49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB
708 MiB
15 GiB 1.5 TiB 78.96 1.06 98 up osd.49
> >>> >> >>
>>>>> 50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB
472 MiB
15 GiB 1.4 TiB 80.40 1.08 102 up osd.50
> >>> >> >>
>>>>> 51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB
729 MiB
15 GiB 1.3 TiB 81.70 1.10 110 up osd.51
> >>> >> >>
>>>>> 72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB
91 MiB
12 GiB 2.5 TiB 65.82 0.88 89 up osd.72
> >>> >> >>
>>>>> -37 58.55478 - 59 TiB 46 TiB 46 TiB
5.0 GiB
124 GiB 12 TiB 79.04 1.06 - host s3db12
> >>> >> >>
>>>>> 19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB
462 MiB
8.2 GiB 559 GiB 85.18 1.14 55 up osd.19
> >>> >> >>
>>>>> 71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB
3.9 MiB
7.8 GiB 825 GiB 78.14 1.05 50 up osd.71
> >>> >> >>
>>>>> 75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB
576 MiB
8.3 GiB 555 GiB 85.29 1.14 57 up osd.75
> >>> >> >>
>>>>> 76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB
239 MiB
9.3 GiB 501 GiB 86.73 1.16 50 up osd.76
> >>> >> >>
>>>>> 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB
880 MiB
30 GiB 3.6 TiB 75.57 1.01 202 up osd.77
> >>> >> >>
>>>>> 78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB
1.0 GiB
30 GiB 3.4 TiB 76.65 1.03 196 up osd.78
> >>> >> >>
>>>>> 83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB
1.8 GiB
31 GiB 2.9 TiB 80.04 1.07 223 up osd.83
> >>> >> >>
>>>>> -3 58.49872 - 58 TiB 43 TiB 38 TiB
8.1 GiB
91 GiB 16 TiB 73.15 0.98 - host s3db2
> >>> >> >>
>>>>> 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
3.1 GiB
38 GiB 3.6 TiB 75.52 1.01 194 up osd.1
> >>> >> >>
>>>>> 3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB
418 MiB
0 B 1.4 TiB 60.94 0.82 52 up osd.3
> >>> >> >>
>>>>> 4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB
845 MiB
0 B 401 GiB 89.23 1.20 53 up osd.4
> >>> >> >>
>>>>> 5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB
437 MiB
0 B 1.3 TiB 62.88 0.84 51 up osd.5
> >>> >> >>
>>>>> 6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB
1.8 GiB
0 B 1.7 TiB 54.51 0.73 47 up osd.6
> >>> >> >>
>>>>> 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
493 MiB
26 GiB 3.8 TiB 73.90 0.99 185 up osd.7
> >>> >> >>
>>>>> 74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
1.1 GiB
27 GiB 3.5 TiB 76.27 1.02 208 up osd.74
> >>> >> >>
>>>>> -4 58.49872 - 58 TiB 43 TiB 37 TiB
33 GiB
86 GiB 15 TiB 74.05 0.99 - host s3db3
> >>> >> >>
>>>>> 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
850 MiB
26 GiB 4.0 TiB 72.78 0.98 203 up osd.2
> >>> >> >>
>>>>> 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
8.3 GiB
33 GiB 3.6 TiB 75.62 1.01 189 up osd.9
> >>> >> >>
>>>>> 10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
663 MiB
28 GiB 3.5 TiB 76.34 1.02 211 up osd.10
> >>> >> >>
>>>>> 12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB
633 MiB
0 B 1.2 TiB 66.22 0.89 44 up osd.12
> >>> >> >>
>>>>> 13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB
2.3 GiB
0 B 720 GiB 80.66 1.08 66 up osd.13
> >>> >> >>
>>>>> 14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB
18 GiB
0 B 552 GiB 85.18 1.14 60 up osd.14
> >>> >> >>
>>>>> 15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB
2.1 GiB
0 B 1.7 TiB 53.72 0.72 44 up osd.15
> >>> >> >>
>>>>> -5 58.49872 - 58 TiB 45 TiB 37 TiB
7.2 GiB
99 GiB 14 TiB 76.37 1.02 - host s3db4
> >>> >> >>
>>>>> 11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB
897 MiB
28 GiB 2.8 TiB 81.15 1.09 205 up osd.11
> >>> >> >>
>>>>> 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
1.2 GiB
27 GiB 3.6 TiB 75.38 1.01 211 up osd.17
> >>> >> >>
>>>>> 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
965 MiB
44 GiB 4.0 TiB 72.86 0.98 188 up osd.18
> >>> >> >>
>>>>> 20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB
529 MiB
0 B 796 GiB 78.63 1.05 66 up osd.20
> >>> >> >>
>>>>> 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB
2.1 GiB
0 B 1.1 TiB 70.32 0.94 47 up osd.21
> >>> >> >>
>>>>> 22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB
882 MiB
0 B 802 GiB 78.47 1.05 58 up osd.22
> >>> >> >>
>>>>> 24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB
645 MiB
0 B 856 GiB 77.01 1.03 47 up osd.24
> >>> >> >>
>>>>> -6 58.89636 - 59 TiB 44 TiB 44 TiB
2.4 GiB
111 GiB 15 TiB 75.22 1.01 - host s3db5
> >>> >> >>
>>>>> 0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB
70 MiB
6.6 GiB 1.3 TiB 65.00 0.87 48 up osd.0
> >>> >> >>
>>>>> 25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB
5.3 MiB
6.6 GiB 1.4 TiB 63.86 0.86 41 up osd.25
> >>> >> >>
>>>>> 26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB
181 MiB
7.6 GiB 862 GiB 77.47 1.04 48 up osd.26
> >>> >> >>
>>>>> 27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB
7.0 MiB
6.1 GiB 1.5 TiB 61.00 0.82 48 up osd.27
> >>> >> >>
>>>>> 28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB
937 MiB
30 GiB 2.8 TiB 81.19 1.09 203 up osd.28
> >>> >> >>
>>>>> 29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
536 MiB
26 GiB 3.8 TiB 73.95 0.99 200 up osd.29
> >>> >> >>
>>>>> 30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB
744 MiB
28 GiB 3.1 TiB 79.07 1.06 207 up osd.30
> >>> >> >>
>>>>> -7 58.89636 - 59 TiB 44 TiB 44 TiB
14 GiB
122 GiB 14 TiB 75.41 1.01 - host s3db6
> >>> >> >>
>>>>> 32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB
16 MiB
8.2 GiB 622 GiB 83.74 1.12 65 up osd.32
> >>> >> >>
>>>>> 33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB
14 MiB
8.1 GiB 740 GiB 80.67 1.08 52 up osd.33
> >>> >> >>
>>>>> 34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB
449 MiB
7.7 GiB 877 GiB 77.08 1.03 52 up osd.34
> >>> >> >>
>>>>> 35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB
133 MiB
7.0 GiB 1.4 TiB 62.18 0.83 42 up osd.35
> >>> >> >>
>>>>> 36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
544 MiB
26 GiB 4.0 TiB 72.98 0.98 220 up osd.36
> >>> >> >>
>>>>> 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
11 GiB
38 GiB 3.6 TiB 75.30 1.01 200 up osd.37
> >>> >> >>
>>>>> 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
1.2 GiB
28 GiB 3.3 TiB 77.43 1.04 217 up osd.38
> >>> >> >>
>>>>> -8 58.89636 - 59 TiB 47 TiB 46 TiB
3.9 GiB
116 GiB 12 TiB 78.98 1.06 - host s3db7
> >>> >> >>
>>>>> 39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB
19 MiB
8.5 GiB 499 GiB 86.96 1.17 43 up osd.39
> >>> >> >>
>>>>> 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB
144 MiB
7.0 GiB 1.2 TiB 68.33 0.92 39 up osd.40
> >>> >> >>
>>>>> 41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB
218 MiB
7.9 GiB 732 GiB 80.86 1.08 64 up osd.41
> >>> >> >>
>>>>> 42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB
594 MiB
7.0 GiB 1.2 TiB 67.97 0.91 50 up osd.42
> >>> >> >>
>>>>> 43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB
564 MiB
28 GiB 2.9 TiB 80.32 1.08 213 up osd.43
> >>> >> >>
>>>>> 44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB
1.3 GiB
28 GiB 3.1 TiB 78.59 1.05 198 up osd.44
> >>> >> >>
>>>>> 45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB
1.2 GiB
30 GiB 2.8 TiB 81.05 1.09 214 up osd.45
> >>> >> >>
>>>>> -9 51.28331 - 51 TiB 41 TiB 41 TiB
4.9 GiB
108 GiB 10 TiB 79.75 1.07 - host s3db8
> >>> >> >>
>>>>> 8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB
472 MiB
15 GiB 1.5 TiB 79.68 1.07 99 up osd.8
> >>> >> >>
>>>>> 16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB
785 MiB
15 GiB 1.4 TiB 80.25 1.08 97 up osd.16
> >>> >> >>
>>>>> 31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB
438 MiB
14 GiB 1.8 TiB 75.36 1.01 87 up osd.31
> >>> >> >>
>>>>> 52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB
844 MiB
15 GiB 1.6 TiB 78.19 1.05 113 up osd.52
> >>> >> >>
>>>>> 53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB
792 MiB
18 GiB 1.1 TiB 84.46 1.13 109 up osd.53
> >>> >> >>
>>>>> 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB
959 MiB
15 GiB 1.7 TiB 76.73 1.03 115 up osd.54
> >>> >> >>
>>>>> 55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB
699 MiB
16 GiB 1.2 TiB 83.56 1.12 122 up osd.55
> >>> >> >>
>>>>> -10 51.28331 - 51 TiB 39 TiB 39 TiB
4.7 GiB
100 GiB 12 TiB 76.05 1.02 - host s3db9
> >>> >> >>
>>>>> 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB
840 MiB
13 GiB 2.1 TiB 71.06 0.95 105 up osd.56
> >>> >> >>
>>>>> 57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB
1.0 GiB
16 GiB 1.2 TiB 83.17 1.12 102 up osd.57
> >>> >> >>
>>>>> 58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB
43 MiB
15 GiB 1.4 TiB 81.56 1.09 105 up osd.58
> >>> >> >>
>>>>> 59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB
429 MiB
15 GiB 1.4 TiB 80.64 1.08 94 up osd.59
> >>> >> >>
>>>>> 60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB
226 MiB
14 GiB 2.0 TiB 73.25 0.98 101 up osd.60
> >>> >> >>
>>>>> 61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB
1.1 GiB
12 GiB 2.5 TiB 65.84 0.88 103 up osd.61
> >>> >> >>
>>>>> 62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB
1.0 GiB
15 GiB 1.7 TiB 76.83 1.03 126 up osd.62
> >>> >> >>
>>>>> TOTAL 674 TiB 501 TiB 473 TiB
96 GiB
1.2 TiB 173 TiB 74.57
> >>> >> >>
>>>>> MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25
> >>> >> >> >>>>>
> >>> >> >> >>>>>
> >>> >> >> >>>>>
> >>> >> >> >>>>> Am Sa., 13. März 2021 um 15:57
Uhr schrieb Dan van der
Ster <dan(a)vanderster.com>om>:
> >>> >> >>
>>>>>>
> >>> >> >> >>>>>> No, increasing num PGs
won't help substantially.
> >>> >> >> >>>>>>
> >>> >> >> >>>>>> Can you share the entire
output of ceph osd df tree ?
> >>> >> >> >>>>>>
> >>> >> >> >>>>>> Did you already set
> >>> >> >> >>>>>>
> >>> >> >> >>>>>> ceph config set mgr
mgr/balancer/upmap_max_deviation
1
> >>> >> >>
>>>>>>
> >>> >> >> >>>>>>
> >>> >> >> >>>>>> ??
> >>> >> >> >>>>>> And I recommend debug_mgr 4/5
so you can see some
basic upmap balancer logging.
> >>> >> >>
>>>>>>
> >>> >> >> >>>>>> .. Dan
> >>> >> >> >>>>>>
> >>> >> >> >>>>>>
> >>> >> >> >>>>>>
> >>> >> >> >>>>>>
> >>> >> >> >>>>>>
> >>> >> >> >>>>>>
> >>> >> >> >>>>>> On Sat, Mar 13, 2021, 3:49 PM
Boris Behrens <
bb(a)kervyn.de> wrote:
> >>> >> >>
>>>>>>>
> >>> >> >> >>>>>>> Hello people,
> >>> >> >> >>>>>>>
> >>> >> >> >>>>>>> I am still struggeling
with the balancer
> >>> >> >> >>>>>>> (
https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
> >>> >> >>
>>>>>>> Now I've read some more and might think that I do not
have enough PGs.
> >>> >> >>
>>>>>>> Currently I have 84OSDs and 1024PGs for the main pool
(3008 total). I
> >>> >> >>
>>>>>>> have the autoscaler enabled, but I doesn't tell me to
increase the
> >>> >> >>
>>>>>>> PGs.
> >>> >> >> >>>>>>>
> >>> >> >> >>>>>>> What do you think?
> >>> >> >> >>>>>>>
> >>> >> >> >>>>>>> --
> >>> >> >> >>>>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich
diesmal abweichend
> >>> >> >>
>>>>>>> im groüen Saal.
> >>> >> >> >>>>>>>
_______________________________________________
> >>> >> >> >>>>>>> ceph-users mailing list
-- ceph-users(a)ceph.io
> >>> >> >> >>>>>>> To unsubscribe send an
email to
ceph-users-leave(a)ceph.io
> >>> >> >>
>>>>>
> >>> >> >> >>>>>
> >>> >> >> >>>>>
> >>> >> >> >>>>> --
> >>> >> >> >>>>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich
diesmal abweichend im groüen Saal.
> >>> >> >> >>>
> >>> >> >> >>>
> >>> >> >> >>>
> >>> >> >> >>> --
> >>> >> >> >>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich
diesmal abweichend im groüen Saal.
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > --
> >>> >> >> > Die Selbsthilfegruppe "UTF-8-Probleme"
trifft sich diesmal
abweichend im groüen Saal.
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft
sich diesmal
abweichend im groüen Saal.
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal
abweichend im groüen Saal.
> >>
> >>
> >>
> >> --
> >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
> >
> >
> >
> > --
> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
6:21 p.m.
While you're watching things, if an OSD is getting too close for
comfort to the full ratio, you can temporarily increase it, e.g.
ceph osd set-full-ratio 0.96
But don't set that too high -- you can really break an OSD if it gets
100% full (and then can't delete objects or whatever...)
-- dan
On Tue, Mar 23, 2021 at 7:17 PM Boris Behrens <bb(a)kervyn.de> wrote:
>
> Ok, then I will try to reweight the most filled OSDs to .95 and see if this helps.
>
> Am Di., 23. März 2021 um 19:13 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>>
>> Data goes to *all* PGs uniformly.
>> Max_avail is limited by the available space on the most full OSD --
>> you should pay close attention to those and make sure they are moving
>> in the right direction (decreasing!)
>>
>> Another point -- IMHO you should aim to get all PGs active+clean
>> before you add yet another batch of new disks. While there are PGs
>> backfilling, your osdmaps are accumulating on the mons and osds --
>> this itself will start to use a lot of space, and active+clean is the
>> only way to trim the old maps.
>>
>> -- dan
>>
>> On Tue, Mar 23, 2021 at 7:05 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >
>> > So,
>> > doing nothing and wait for the ceph to recover?
>> >
>> > In theory there should be enough disk space (more disks arriving tomorrow),
but I fear that there might be an issue, when the backups get exported over night to this
s3. Currently the max_avail lingers around 13TB and I hope, that the data will go to other
PGs than the ones that are currently on filled OSDs.
>> >
>> >
>> >
>> > Am Di., 23. März 2021 um 18:58 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >>
>> >> Hi,
>> >>
>> >> backfill_toofull is not a bad thing when the cluster is really full
>> >> like yours. You should expect some of the most full OSDs to eventually
>> >> start decreasing in usage, as the PGs are moved to the new OSDs. Those
>> >> backfill_toofull states should then resolve themselves as the OSD
>> >> usage flattens out.
>> >> Keep an eye on the usage of the backfill_full and nearfull OSDs though
>> >> -- if they do eventually go above the full_ratio (95% by default),
>> >> then writes to those OSDs would stop.
>> >>
>> >> But if on the other hand you're suffering from lots of slow ops or
>> >> anything else visible to your users, then you could try to take some
>> >> actions to slow down the rebalancing. Just let us know if that's
the
>> >> case and we can see about changing osd_max_backfills, some weights or
>> >> maybe using the upmap-remapped tool.
>> >>
>> >> -- Dan
>> >>
>> >> On Tue, Mar 23, 2021 at 6:07 PM Boris Behrens <bb(a)kervyn.de>
wrote:
>> >> >
>> >> > Ok, I should have listened to you :)
>> >> >
>> >> > In the last week we added more storage but the issue got worse
instead.
>> >> > Today I realized that the PGs were up to 90GB (bytes column in ceph
pg ls said 95705749636), and the balance kept mentioning the 2048 PGs for this pool. We
were at 72% utilization (ceph osd df tree, first line) for our cluster and I increased the
PGs to 2048.
>> >> >
>> >> > Now I am in a world of trouble.
>> >> > The space in the cluster went down, I am at 45% misplaced objects,
and we already added 20x4TB disks just to not go down completly.
>> >> >
>> >> > The utilization is still going up and the overall free space in the
cluster seems to go down. This is what my ceph status looks like and now I really need
help to get that thing back to normal:
>> >> > [root@s3db1 ~]# ceph status
>> >> > cluster:
>> >> > id: dca79fff-ffd0-58f4-1cff-82a2feea05f4
>> >> > health: HEALTH_WARN
>> >> > 4 backfillfull osd(s)
>> >> > 17 nearfull osd(s)
>> >> > 37 pool(s) backfillfull
>> >> > 13 large omap objects
>> >> > Low space hindering backfill (add storage if this
doesn't resolve itself): 570 pgs backfill_toofull
>> >> >
>> >> > services:
>> >> > mon: 3 daemons, quorum ceph-s3-mon1,ceph-s3-mon2,ceph-s3-mon3
(age 44m)
>> >> > mgr: ceph-mgr2(active, since 15m), standbys: ceph-mgr3,
ceph-mgr1
>> >> > mds: 3 up:standby
>> >> > osd: 110 osds: 110 up (since 28m), 110 in (since 28m); 1535
remapped pgs
>> >> > rgw: 3 daemons active (eu-central-1, eu-msg-1, eu-secure-1)
>> >> >
>> >> > task status:
>> >> >
>> >> > data:
>> >> > pools: 37 pools, 4032 pgs
>> >> > objects: 116.23M objects, 182 TiB
>> >> > usage: 589 TiB used, 206 TiB / 795 TiB avail
>> >> > pgs: 160918554/348689415 objects misplaced (46.150%)
>> >> > 2497 active+clean
>> >> > 779 active+remapped+backfill_wait
>> >> > 538 active+remapped+backfill_wait+backfill_toofull
>> >> > 186 active+remapped+backfilling
>> >> > 32 active+remapped+backfill_toofull
>> >> >
>> >> > io:
>> >> > client: 27 MiB/s rd, 69 MiB/s wr, 497 op/s rd, 153 op/s wr
>> >> > recovery: 1.5 GiB/s, 922 objects/s
>> >> >
>> >> > Am Di., 16. März 2021 um 09:34 Uhr schrieb Boris Behrens
<bb(a)kervyn.de>de>:
>> >> >>
>> >> >> Hi Dan,
>> >> >>
>> >> >> my EC profile look very "default" to me.
>> >> >> [root@s3db1 ~]# ceph osd erasure-code-profile ls
>> >> >> default
>> >> >> [root@s3db1 ~]# ceph osd erasure-code-profile get default
>> >> >> k=2
>> >> >> m=1
>> >> >> plugin=jerasure
>> >> >> technique=reed_sol_van
>> >> >>
>> >> >> I don't understand the ouput, but the balancing get worse
over night:
>> >> >>
>> >> >> [root@s3db1 ~]# ceph-scripts/tools/ceph-pool-pg-distribution
11
>> >> >> Searching for PGs in pools: ['11']
>> >> >> Summary: 1024 PGs on 84 osds
>> >> >>
>> >> >> Num OSDs with X PGs:
>> >> >> 15: 8
>> >> >> 16: 7
>> >> >> 17: 6
>> >> >> 18: 10
>> >> >> 19: 1
>> >> >> 32: 10
>> >> >> 33: 4
>> >> >> 34: 6
>> >> >> 35: 8
>> >> >> 65: 5
>> >> >> 66: 5
>> >> >> 67: 4
>> >> >> 68: 10
>> >> >> [root@s3db1 ~]# ceph-scripts/tools/ceph-pg-histogram
--normalize --pool=11
>> >> >> # NumSamples = 84; Min = 4.12; Max = 5.09
>> >> >> # Mean = 4.553355; Variance = 0.052415; SD = 0.228942; Median
4.561608
>> >> >> # each ∎ represents a count of 1
>> >> >> 4.1244 - 4.2205 [ 8]: ∎∎∎∎∎∎∎∎
>> >> >> 4.2205 - 4.3166 [ 6]: ∎∎∎∎∎∎
>> >> >> 4.3166 - 4.4127 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
>> >> >> 4.4127 - 4.5087 [ 10]: ∎∎∎∎∎∎∎∎∎∎
>> >> >> 4.5087 - 4.6048 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎
>> >> >> 4.6048 - 4.7009 [ 19]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
>> >> >> 4.7009 - 4.7970 [ 6]: ∎∎∎∎∎∎
>> >> >> 4.7970 - 4.8931 [ 8]: ∎∎∎∎∎∎∎∎
>> >> >> 4.8931 - 4.9892 [ 4]: ∎∎∎∎
>> >> >> 4.9892 - 5.0852 [ 1]: ∎
>> >> >> [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail
>> >> >> 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 724 GiB 19 GiB
0 B 724 GiB 80.56 1.07 56 up osd.14
>> >> >> 19 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 466 MiB
7.9 GiB 708 GiB 81.25 1.08 53 up osd.19
>> >> >> 4 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 698 GiB 703 MiB
0 B 698 GiB 81.27 1.08 48 up osd.4
>> >> >> 24 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 695 GiB 640 MiB
0 B 695 GiB 81.34 1.08 46 up osd.24
>> >> >> 75 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 440 MiB
8.1 GiB 704 GiB 81.35 1.08 48 up osd.75
>> >> >> 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 7.5 MiB
8.0 GiB 663 GiB 82.44 1.09 47 up osd.71
>> >> >> 76 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 251 MiB
9.0 GiB 617 GiB 83.65 1.11 50 up osd.76
>> >> >> 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 399 MiB
8.1 GiB 618 GiB 83.85 1.11 55 up osd.33
>> >> >> 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 317 MiB
8.8 GiB 617 GiB 83.87 1.11 50 up osd.35
>> >> >> 34 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 451 MiB
8.7 GiB 545 GiB 85.75 1.14 54 up osd.34
>> >> >>
>> >> >> Am Mo., 15. März 2021 um 17:23 Uhr schrieb Dan van der Ster
<dan(a)vanderster.com>om>:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> How wide are your EC profiles? If they are really wide, you
might be
>> >> >>> reaching the limits of what is physically possible. Also,
I'm not sure
>> >> >>> that upmap in 14.2.11 is very smart about *improving*
existing upmap
>> >> >>> rules for a given PG, in the case that a PG already has an
upmap-items
>> >> >>> entry but it would help the distribution to add more
mapping pairs to
>> >> >>> that entry. What this means, is that it might sometimes be
useful to
>> >> >>> randomly remove some upmap entries and see if the balancer
does a
>> >> >>> better job when it replaces them.
>> >> >>>
>> >> >>> But before you do that, I re-remembered that looking at the
total PG
>> >> >>> numbers is not useful -- you need to check the PGs per OSD
for the
>> >> >>> eu-central-1.rgw.buckets.data pool only.
>> >> >>>
>> >> >>> We have a couple tools that can help with this:
>> >> >>>
>> >> >>> 1. To see the PGs per OSD for a given pool:
>> >> >>>
https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis…
>> >> >>>
>> >> >>> E.g.: ./ceph-pool-pg-distribution 11 # to see the
distribution of
>> >> >>> your eu-central-1.rgw.buckets.data pool.
>> >> >>>
>> >> >>> The output looks like this on my well balanced
clusters:
>> >> >>>
>> >> >>> # ceph-scripts/tools/ceph-pool-pg-distribution 15
>> >> >>> Searching for PGs in pools: ['15']
>> >> >>> Summary: 256 pgs on 56 osds
>> >> >>>
>> >> >>> Num OSDs with X PGs:
>> >> >>> 13: 16
>> >> >>> 14: 40
>> >> >>>
>> >> >>> You should expect a trimodal for your cluster.
>> >> >>>
>> >> >>> 2. You can also use another script from that repo to see
the PGs per
>> >> >>> OSD normalized to crush weight:
>> >> >>> ceph-scripts/tools/ceph-pg-histogram --normalize
--pool=15
>> >> >>>
>> >> >>> This might explain what is going wrong.
>> >> >>>
>> >> >>> Cheers, Dan
>> >> >>>
>> >> >>>
>> >> >>> On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>> >> >>> >
>> >> >>> > Absolutly:
>> >> >>> > [root@s3db1 ~]# ceph osd df tree
>> >> >>> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA
OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
>> >> >>> > -1 673.54224 - 674 TiB 496 TiB 468 TiB
97 GiB 1.2 TiB 177 TiB 73.67 1.00 - root default
>> >> >>> > -2 58.30331 - 58 TiB 42 TiB 38 TiB
9.2 GiB 99 GiB 16 TiB 72.88 0.99 - host s3db1
>> >> >>> > 23 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
714 MiB 25 GiB 3.7 TiB 74.87 1.02 194 up osd.23
>> >> >>> > 69 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB
1.6 GiB 40 GiB 3.4 TiB 76.32 1.04 199 up osd.69
>> >> >>> > 73 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB
1.3 GiB 34 GiB 3.8 TiB 74.15 1.01 203 up osd.73
>> >> >>> > 79 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB
1.8 GiB 0 B 1.3 TiB 65.44 0.89 47 up osd.79
>> >> >>> > 80 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB
2.2 GiB 0 B 1.3 TiB 65.34 0.89 48 up osd.80
>> >> >>> > 81 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB
1.1 GiB 0 B 1.3 TiB 65.38 0.89 47 up osd.81
>> >> >>> > 82 hdd 3.63689 1.00000 3.6 TiB 2.5 TiB 1.1 TiB
619 MiB 0 B 1.1 TiB 68.46 0.93 41 up osd.82
>> >> >>> > -11 50.94173 - 51 TiB 37 TiB 37 TiB
3.5 GiB 98 GiB 14 TiB 71.90 0.98 - host s3db10
>> >> >>> > 63 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB
647 MiB 14 GiB 2.0 TiB 72.72 0.99 94 up osd.63
>> >> >>> > 64 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.2 TiB
668 MiB 14 GiB 2.0 TiB 72.23 0.98 93 up osd.64
>> >> >>> > 65 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB
227 MiB 14 GiB 2.1 TiB 71.16 0.97 100 up osd.65
>> >> >>> > 66 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB
313 MiB 14 GiB 1.9 TiB 74.25 1.01 92 up osd.66
>> >> >>> > 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB
584 MiB 14 GiB 2.1 TiB 70.63 0.96 96 up osd.67
>> >> >>> > 68 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB
720 MiB 14 GiB 2.1 TiB 71.72 0.97 101 up osd.68
>> >> >>> > 70 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB
425 MiB 14 GiB 2.1 TiB 70.59 0.96 97 up osd.70
>> >> >>> > -12 50.99052 - 51 TiB 38 TiB 37 TiB
2.1 GiB 97 GiB 13 TiB 73.77 1.00 - host s3db11
>> >> >>> > 46 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.6 TiB
229 MiB 14 GiB 1.7 TiB 77.05 1.05 97 up osd.46
>> >> >>> > 47 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB
159 MiB 13 GiB 2.2 TiB 70.00 0.95 89 up osd.47
>> >> >>> > 48 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB
279 MiB 14 GiB 2.1 TiB 71.82 0.97 98 up osd.48
>> >> >>> > 49 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.4 TiB
276 MiB 14 GiB 1.8 TiB 74.90 1.02 95 up osd.49
>> >> >>> > 50 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB
336 MiB 14 GiB 2.0 TiB 72.13 0.98 93 up osd.50
>> >> >>> > 51 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.6 TiB
728 MiB 15 GiB 1.6 TiB 77.76 1.06 98 up osd.51
>> >> >>> > 72 hdd 7.32619 1.00000 7.3 TiB 5.3 TiB 5.3 TiB
147 MiB 13 GiB 2.0 TiB 72.75 0.99 95 up osd.72
>> >> >>> > -37 58.55478 - 59 TiB 44 TiB 44 TiB
4.4 GiB 122 GiB 15 TiB 75.20 1.02 - host s3db12
>> >> >>> > 19 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB
454 MiB 8.2 GiB 780 GiB 79.35 1.08 53 up osd.19
>> >> >>> > 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB
7.1 MiB 8.0 GiB 734 GiB 80.56 1.09 47 up osd.71
>> >> >>> > 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB
439 MiB 8.2 GiB 777 GiB 79.43 1.08 48 up osd.75
>> >> >>> > 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB
241 MiB 8.9 GiB 688 GiB 81.77 1.11 50 up osd.76
>> >> >>> > 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB
880 MiB 30 GiB 3.6 TiB 75.44 1.02 201 up osd.77
>> >> >>> > 78 hdd 14.60159 1.00000 15 TiB 10 TiB 10 TiB
1015 MiB 28 GiB 4.2 TiB 71.26 0.97 193 up osd.78
>> >> >>> > 83 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB
1.4 GiB 30 GiB 3.8 TiB 73.76 1.00 203 up osd.83
>> >> >>> > -3 58.49872 - 58 TiB 42 TiB 36 TiB
8.2 GiB 89 GiB 17 TiB 71.71 0.97 - host s3db2
>> >> >>> > 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
3.2 GiB 37 GiB 3.7 TiB 74.58 1.01 196 up osd.1
>> >> >>> > 3 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB
566 MiB 0 B 1.3 TiB 64.11 0.87 50 up osd.3
>> >> >>> > 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 771 GiB
695 MiB 0 B 771 GiB 79.30 1.08 48 up osd.4
>> >> >>> > 5 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB
482 MiB 0 B 1.2 TiB 66.51 0.90 49 up osd.5
>> >> >>> > 6 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB
1.8 GiB 0 B 1.3 TiB 64.00 0.87 42 up osd.6
>> >> >>> > 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
639 MiB 26 GiB 4.0 TiB 72.44 0.98 192 up osd.7
>> >> >>> > 74 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB
907 MiB 26 GiB 4.2 TiB 71.32 0.97 193 up osd.74
>> >> >>> > -4 58.49872 - 58 TiB 43 TiB 36 TiB
34 GiB 85 GiB 16 TiB 72.69 0.99 - host s3db3
>> >> >>> > 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
980 MiB 26 GiB 3.8 TiB 74.36 1.01 203 up osd.2
>> >> >>> > 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
8.4 GiB 33 GiB 3.9 TiB 73.51 1.00 186 up osd.9
>> >> >>> > 10 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB
650 MiB 26 GiB 4.2 TiB 71.64 0.97 201 up osd.10
>> >> >>> > 12 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB
754 MiB 0 B 1.3 TiB 64.17 0.87 44 up osd.12
>> >> >>> > 13 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 813 GiB
2.4 GiB 0 B 813 GiB 78.17 1.06 58 up osd.13
>> >> >>> > 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 797 GiB
19 GiB 0 B 797 GiB 78.60 1.07 56 up osd.14
>> >> >>> > 15 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB
2.2 GiB 0 B 1.3 TiB 63.96 0.87 41 up osd.15
>> >> >>> > -5 58.49872 - 58 TiB 43 TiB 36 TiB
6.7 GiB 97 GiB 15 TiB 74.04 1.01 - host s3db4
>> >> >>> > 11 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
940 MiB 26 GiB 4.0 TiB 72.49 0.98 196 up osd.11
>> >> >>> > 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
1022 MiB 26 GiB 3.6 TiB 75.23 1.02 204 up osd.17
>> >> >>> > 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
945 MiB 45 GiB 3.8 TiB 74.16 1.01 193 up osd.18
>> >> >>> > 20 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1020 GiB
596 MiB 0 B 1020 GiB 72.62 0.99 57 up osd.20
>> >> >>> > 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB
1.9 GiB 0 B 1023 GiB 72.54 0.98 41 up osd.21
>> >> >>> > 22 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB
797 MiB 0 B 1023 GiB 72.54 0.98 53 up osd.22
>> >> >>> > 24 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 766 GiB
618 MiB 0 B 766 GiB 79.42 1.08 46 up osd.24
>> >> >>> > -6 58.89636 - 59 TiB 43 TiB 43 TiB
3.0 GiB 108 GiB 16 TiB 73.40 1.00 - host s3db5
>> >> >>> > 0 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB
92 MiB 7.2 GiB 1.1 TiB 71.16 0.97 45 up osd.0
>> >> >>> > 25 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB
2.4 MiB 7.3 GiB 1.1 TiB 71.23 0.97 41 up osd.25
>> >> >>> > 26 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB
181 MiB 7.6 GiB 935 GiB 75.57 1.03 45 up osd.26
>> >> >>> > 27 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB
5.1 MiB 7.0 GiB 1.1 TiB 71.20 0.97 47 up osd.27
>> >> >>> > 28 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
977 MiB 26 GiB 3.8 TiB 73.85 1.00 197 up osd.28
>> >> >>> > 29 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB
872 MiB 26 GiB 4.1 TiB 71.98 0.98 196 up osd.29
>> >> >>> > 30 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
943 MiB 27 GiB 3.6 TiB 75.51 1.03 202 up osd.30
>> >> >>> > -7 58.89636 - 59 TiB 44 TiB 43 TiB
13 GiB 122 GiB 15 TiB 74.97 1.02 - host s3db6
>> >> >>> > 32 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB
27 MiB 7.6 GiB 940 GiB 75.42 1.02 55 up osd.32
>> >> >>> > 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB
376 MiB 8.2 GiB 691 GiB 81.94 1.11 55 up osd.33
>> >> >>> > 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB
450 MiB 8.5 GiB 620 GiB 83.79 1.14 54 up osd.34
>> >> >>> > 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB
316 MiB 8.4 GiB 690 GiB 81.98 1.11 50 up osd.35
>> >> >>> > 36 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB
489 MiB 25 GiB 4.1 TiB 71.69 0.97 208 up osd.36
>> >> >>> > 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
11 GiB 38 GiB 4.0 TiB 72.41 0.98 195 up osd.37
>> >> >>> > 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
1.1 GiB 26 GiB 3.7 TiB 74.88 1.02 204 up osd.38
>> >> >>> > -8 58.89636 - 59 TiB 44 TiB 43 TiB
3.8 GiB 111 GiB 15 TiB 74.16 1.01 - host s3db7
>> >> >>> > 39 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB
19 MiB 7.5 GiB 936 GiB 75.54 1.03 39 up osd.39
>> >> >>> > 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB
144 MiB 7.1 GiB 1.1 TiB 69.87 0.95 39 up osd.40
>> >> >>> > 41 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.7 TiB
219 MiB 7.6 GiB 1011 GiB 73.57 1.00 55 up osd.41
>> >> >>> > 42 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB
593 MiB 7.1 GiB 1.1 TiB 70.02 0.95 47 up osd.42
>> >> >>> > 43 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
500 MiB 27 GiB 3.7 TiB 74.67 1.01 204 up osd.43
>> >> >>> > 44 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
1.1 GiB 27 GiB 3.7 TiB 74.62 1.01 193 up osd.44
>> >> >>> > 45 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB
1.2 GiB 29 GiB 3.6 TiB 75.16 1.02 204 up osd.45
>> >> >>> > -9 51.28331 - 51 TiB 39 TiB 39 TiB
4.9 GiB 107 GiB 12 TiB 76.50 1.04 - host s3db8
>> >> >>> > 8 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB
474 MiB 14 GiB 1.7 TiB 76.37 1.04 98 up osd.8
>> >> >>> > 16 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB
783 MiB 15 GiB 1.6 TiB 78.39 1.06 100 up osd.16
>> >> >>> > 31 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.6 TiB
441 MiB 14 GiB 1.6 TiB 77.70 1.05 91 up osd.31
>> >> >>> > 52 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB
939 MiB 14 GiB 1.7 TiB 76.29 1.04 102 up osd.52
>> >> >>> > 53 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.4 TiB
848 MiB 18 GiB 1.9 TiB 74.30 1.01 98 up osd.53
>> >> >>> > 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB
1.0 GiB 16 GiB 1.7 TiB 76.99 1.05 106 up osd.54
>> >> >>> > 55 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB
460 MiB 15 GiB 1.8 TiB 75.46 1.02 105 up osd.55
>> >> >>> > -10 51.28331 - 51 TiB 37 TiB 37 TiB
3.8 GiB 96 GiB 14 TiB 72.77 0.99 - host s3db9
>> >> >>> > 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB
846 MiB 13 GiB 2.1 TiB 71.16 0.97 104 up osd.56
>> >> >>> > 57 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB
513 MiB 15 GiB 1.7 TiB 76.53 1.04 96 up osd.57
>> >> >>> > 58 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB
604 MiB 13 GiB 2.1 TiB 71.23 0.97 98 up osd.58
>> >> >>> > 59 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB
414 MiB 13 GiB 2.2 TiB 70.03 0.95 88 up osd.59
>> >> >>> > 60 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB
227 MiB 14 GiB 1.8 TiB 75.54 1.03 97 up osd.60
>> >> >>> > 61 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB
456 MiB 13 GiB 2.2 TiB 70.01 0.95 95 up osd.61
>> >> >>> > 62 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.4 TiB
843 MiB 14 GiB 1.8 TiB 74.93 1.02 110 up osd.62
>> >> >>> > TOTAL 674 TiB 496 TiB 468 TiB
97 GiB 1.2 TiB 177 TiB 73.67
>> >> >>> > MIN/MAX VAR: 0.87/1.14 STDDEV: 4.22
>> >> >>> >
>> >> >>> > Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der
Ster <dan(a)vanderster.com>om>:
>> >> >>> >>
>> >> >>> >> OK thanks. Indeed "prepared 0/10
changes" means it thinks things are balanced.
>> >> >>> >> Could you again share the full ceph osd df tree?
>> >> >>> >>
>> >> >>> >> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens
<bb(a)kervyn.de> wrote:
>> >> >>> >> >
>> >> >>> >> > Hi Dan,
>> >> >>> >> >
>> >> >>> >> > I've set the autoscaler to warn, but it
actually does not warn for now. So not touching it for now.
>> >> >>> >> >
>> >> >>> >> > this is what the log says in minute
intervals:
>> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/active
>> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/sleep_interval
>> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/begin_time
>> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/end_time
>> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/begin_weekday
>> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/end_weekday
>> >> >>> >> > 2021-03-15 13:51:00.971 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/pool_ids
>> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] Optimize plan auto_2021-03-15_13:51:00
>> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/mode
>> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] Mode upmap, max misplaced 0.050000
>> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] do_upmap
>> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/upmap_max_iterations
>> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr
get_config get_config key: mgr/balancer/upmap_max_deviation
>> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4
mgr[balancer] pools ['eu-msg-1.rgw.data.root',
'eu-msg-1.rgw.buckets.non-ec', 'eu-central-1.rgw.users.keys',
'eu-central-1.rgw.gc', 'eu-central-1.rgw.buckets.data',
'eu-central-1.rgw.users.email', 'eu-msg-1.rgw.gc',
'eu-central-1.rgw.usage', 'eu-msg-1.rgw.users.keys',
'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log',
'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index',
'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc',
'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage',
'whitespace-again-2021-03-10', 'fra-1.rgw.meta',
'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email',
'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid',
'eu-msg-1.rgw.control', '.rgw.root', 'eu-msg-1.rgw.buckets.data',
'default.rgw.control', 'fra-1.rgw.log', 'default.rgw.data.root',
'whitespace-again-2021-03-10_3', 'default.rgw.log',
'eu-central-1.rgw.meta', 'eu-central-1.rgw.data.root',
'default.rgw.users.uid', 'eu-central-1.rgw.control']
>> >> >>> >> > 2021-03-15 13:51:01.224 7f307d5fd700 4
mgr[balancer] prepared 0/10 changes
>> >> >>> >> >
>> >> >>> >> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb
Dan van der Ster <dan(a)vanderster.com>om>:
>> >> >>> >> >>
>> >> >>> >> >> I suggest to just disable the autoscaler
until your balancing is understood.
>> >> >>> >> >>
>> >> >>> >> >> What does your active mgr log say (with
debug_mgr 4/5), grep balancer
>> >> >>> >> >> /var/log/ceph/ceph-mgr.*.log
>> >> >>> >> >>
>> >> >>> >> >> -- Dan
>> >> >>> >> >>
>> >> >>> >> >> On Mon, Mar 15, 2021 at 1:47 PM Boris
Behrens <bb(a)kervyn.de> wrote:
>> >> >>> >> >> >
>> >> >>> >> >> > Hi,
>> >> >>> >> >> > this unfortunally did not solve my
problem. I still have some OSDs that fill up to 85%
>> >> >>> >> >> >
>> >> >>> >> >> > According to the logging, the
autoscaler might want to add more PGs to one Bucken and reduce almost all other buckets to
32.
>> >> >>> >> >> > 2021-03-15 12:19:58.825 7f307f601700
4 mgr[pg_autoscaler] Pool 'eu-central-1.rgw.buckets.data' root_id -1 using
0.705080476146 of space, bias 1.0, pg target 1974.22533321 quantized to 2048 (current
1024)
>> >> >>> >> >> >
>> >> >>> >> >> > Why the balancing does not happen is
still nebulous to me.
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> > Am Sa., 13. März 2021 um 16:37 Uhr
schrieb Dan van der Ster <dan(a)vanderster.com>om>:
>> >> >>> >> >> >>
>> >> >>> >> >> >> OK
>> >> >>> >> >> >> Btw, you might need to fail to a
new mgr... I'm not sure if the current active will read that new config.
>> >> >>> >> >> >>
>> >> >>> >> >> >> .. dan
>> >> >>> >> >> >>
>> >> >>> >> >> >>
>> >> >>> >> >> >> On Sat, Mar 13, 2021, 4:36 PM
Boris Behrens <bb(a)kervyn.de> wrote:
>> >> >>> >> >> >>>
>> >> >>> >> >> >>> Hi,
>> >> >>> >> >> >>>
>> >> >>> >> >> >>> ok thanks. I just changed
the value and rewighted everything back to 1. Now I let it sync the weekend and check how
it will be on monday.
>> >> >>> >> >> >>> We tried to have the systems
total storage balanced as possible. New systems will be with 8TB disks but for the exiting
ones we added 16TB to offset the 4TB disks and we needed a lot of storage fast, because of
a DC move. If you have any recommendations I would be happy to hear them.
>> >> >>> >> >> >>>
>> >> >>> >> >> >>> Cheers
>> >> >>> >> >> >>> Boris
>> >> >>> >> >> >>>
>> >> >>> >> >> >>> Am Sa., 13. März 2021 um
16:20 Uhr schrieb Dan van der Ster <dan(a)vanderster.com>om>:
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>> Thanks.
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>> Decreasing the max
deviation to 2 or 1 should help in your case. This option controls when the balancer stops
trying to move PGs around -- by default it stops when the deviation from the mean is 5.
Yes this is too large IMO -- all of our clusters have this set to 1.
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>> And given that you have
some OSDs with more than 200 PGs, you definitely shouldn't increase the num PGs.
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>> But anyway with your
mixed device sizes it might be challenging to make a perfectly uniform distribution. Give
it a try with 1 though, and let us know how it goes.
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>> .. Dan
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>>
>> >> >>> >> >> >>>> On Sat, Mar 13, 2021,
4:11 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>> Hi Dan,
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>> upmap_max_deviation
is default (5) in our cluster. Is 1 the recommended deviation?
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>> I added the whole
ceph osd df tree, (I need to remove some OSDs and readd them as bluestore with SSD, so 69,
73 and 82 are a bit off now. I also reweighted to try to get the %USE mitigated).
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>> I will increase the
mgr debugging to see what is the problem.
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>> [root@s3db1 ~]# ceph
osd df tree
>> >> >>> >> >> >>>>> ID CLASS WEIGHT
REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
>> >> >>> >> >> >>>>> -1 673.54224
- 659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB 168 TiB 74.57 1.00 - root
default
>> >> >>> >> >> >>>>> -2 58.30331
- 44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB 22 TiB 49.82 0.67 - host
s3db1
>> >> >>> >> >> >>>>> 23 hdd 14.65039
1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB 13 TiB 12.50 0.17 101 up
osd.23
>> >> >>> >> >> >>>>> 69 hdd 14.55269
0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 11 up
osd.69
>> >> >>> >> >> >>>>> 73 hdd 14.55269
1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB 4.2 TiB 71.15 0.95 107 up
osd.73
>> >> >>> >> >> >>>>> 79 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B 747 GiB 79.94 1.07 52 up
osd.79
>> >> >>> >> >> >>>>> 80 hdd 3.63689
1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B 1.0 TiB 71.61 0.96 58 up
osd.80
>> >> >>> >> >> >>>>> 81 hdd 3.63689
1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B 1.5 TiB 60.07 0.81 55 up
osd.81
>> >> >>> >> >> >>>>> 82 hdd 3.63689
1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B 1.7 TiB 52.68 0.71 30 up
osd.82
>> >> >>> >> >> >>>>> -11 50.94173
- 51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB 13 TiB 74.69 1.00 - host
s3db10
>> >> >>> >> >> >>>>> 63 hdd 7.27739
1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB 1.7 TiB 76.04 1.02 92 up
osd.63
>> >> >>> >> >> >>>>> 64 hdd 7.27739
1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB 1.8 TiB 75.54 1.01 101 up
osd.64
>> >> >>> >> >> >>>>> 65 hdd 7.27739
1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB 2.0 TiB 73.17 0.98 105 up
osd.65
>> >> >>> >> >> >>>>> 66 hdd 7.27739
1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB 1.4 TiB 80.38 1.08 98 up
osd.66
>> >> >>> >> >> >>>>> 67 hdd 7.27739
1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB 2.2 TiB 70.10 0.94 100 up
osd.67
>> >> >>> >> >> >>>>> 68 hdd 7.27739
1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB 2.0 TiB 72.88 0.98 107 up
osd.68
>> >> >>> >> >> >>>>> 70 hdd 7.27739
1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB 1.8 TiB 74.73 1.00 102 up
osd.70
>> >> >>> >> >> >>>>> -12 50.99052
- 51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB 12 TiB 77.24 1.04 - host
s3db11
>> >> >>> >> >> >>>>> 46 hdd 7.27739
1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB 1.5 TiB 78.91 1.06 97 up
osd.46
>> >> >>> >> >> >>>>> 47 hdd 7.27739
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB 2.1 TiB 71.47 0.96 96 up
osd.47
>> >> >>> >> >> >>>>> 48 hdd 7.27739
1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB 1.2 TiB 83.46 1.12 109 up
osd.48
>> >> >>> >> >> >>>>> 49 hdd 7.27739
1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB 1.5 TiB 78.96 1.06 98 up
osd.49
>> >> >>> >> >> >>>>> 50 hdd 7.27739
1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB 1.4 TiB 80.40 1.08 102 up
osd.50
>> >> >>> >> >> >>>>> 51 hdd 7.27739
1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB 1.3 TiB 81.70 1.10 110 up
osd.51
>> >> >>> >> >> >>>>> 72 hdd 7.32619
1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB 2.5 TiB 65.82 0.88 89 up
osd.72
>> >> >>> >> >> >>>>> -37 58.55478
- 59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB 12 TiB 79.04 1.06 - host
s3db12
>> >> >>> >> >> >>>>> 19 hdd 3.68750
1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB 559 GiB 85.18 1.14 55 up
osd.19
>> >> >>> >> >> >>>>> 71 hdd 3.68750
1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB 825 GiB 78.14 1.05 50 up
osd.71
>> >> >>> >> >> >>>>> 75 hdd 3.68750
1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB 555 GiB 85.29 1.14 57 up
osd.75
>> >> >>> >> >> >>>>> 76 hdd 3.68750
1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB 501 GiB 86.73 1.16 50 up
osd.76
>> >> >>> >> >> >>>>> 77 hdd 14.60159
1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB 3.6 TiB 75.57 1.01 202 up
osd.77
>> >> >>> >> >> >>>>> 78 hdd 14.60159
1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB 3.4 TiB 76.65 1.03 196 up
osd.78
>> >> >>> >> >> >>>>> 83 hdd 14.60159
1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB 2.9 TiB 80.04 1.07 223 up
osd.83
>> >> >>> >> >> >>>>> -3 58.49872
- 58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB 16 TiB 73.15 0.98 - host
s3db2
>> >> >>> >> >> >>>>> 1 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB 3.6 TiB 75.52 1.01 194 up
osd.1
>> >> >>> >> >> >>>>> 3 hdd 3.63689
1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B 1.4 TiB 60.94 0.82 52 up
osd.3
>> >> >>> >> >> >>>>> 4 hdd 3.63689
0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B 401 GiB 89.23 1.20 53 up
osd.4
>> >> >>> >> >> >>>>> 5 hdd 3.63689
1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B 1.3 TiB 62.88 0.84 51 up
osd.5
>> >> >>> >> >> >>>>> 6 hdd 3.63689
1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B 1.7 TiB 54.51 0.73 47 up
osd.6
>> >> >>> >> >> >>>>> 7 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26 GiB 3.8 TiB 73.90 0.99 185 up
osd.7
>> >> >>> >> >> >>>>> 74 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 3.5 TiB 76.27 1.02 208 up
osd.74
>> >> >>> >> >> >>>>> -4 58.49872
- 58 TiB 43 TiB 37 TiB 33 GiB 86 GiB 15 TiB 74.05 0.99 - host
s3db3
>> >> >>> >> >> >>>>> 2 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26 GiB 4.0 TiB 72.78 0.98 203 up
osd.2
>> >> >>> >> >> >>>>> 9 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB 3.6 TiB 75.62 1.01 189 up
osd.9
>> >> >>> >> >> >>>>> 10 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28 GiB 3.5 TiB 76.34 1.02 211 up
osd.10
>> >> >>> >> >> >>>>> 12 hdd 3.63689
1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B 1.2 TiB 66.22 0.89 44 up
osd.12
>> >> >>> >> >> >>>>> 13 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B 720 GiB 80.66 1.08 66 up
osd.13
>> >> >>> >> >> >>>>> 14 hdd 3.63689
1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B 552 GiB 85.18 1.14 60 up
osd.14
>> >> >>> >> >> >>>>> 15 hdd 3.63689
1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B 1.7 TiB 53.72 0.72 44 up
osd.15
>> >> >>> >> >> >>>>> -5 58.49872
- 58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB 14 TiB 76.37 1.02 - host
s3db4
>> >> >>> >> >> >>>>> 11 hdd 14.65039
1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28 GiB 2.8 TiB 81.15 1.09 205 up
osd.11
>> >> >>> >> >> >>>>> 17 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB 3.6 TiB 75.38 1.01 211 up
osd.17
>> >> >>> >> >> >>>>> 18 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44 GiB 4.0 TiB 72.86 0.98 188 up
osd.18
>> >> >>> >> >> >>>>> 20 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B 796 GiB 78.63 1.05 66 up
osd.20
>> >> >>> >> >> >>>>> 21 hdd 3.63689
1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B 1.1 TiB 70.32 0.94 47 up
osd.21
>> >> >>> >> >> >>>>> 22 hdd 3.63689
1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B 802 GiB 78.47 1.05 58 up
osd.22
>> >> >>> >> >> >>>>> 24 hdd 3.63689
1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B 856 GiB 77.01 1.03 47 up
osd.24
>> >> >>> >> >> >>>>> -6 58.89636
- 59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB 15 TiB 75.22 1.01 - host
s3db5
>> >> >>> >> >> >>>>> 0 hdd 3.73630
1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB 1.3 TiB 65.00 0.87 48 up
osd.0
>> >> >>> >> >> >>>>> 25 hdd 3.73630
1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB 1.4 TiB 63.86 0.86 41 up
osd.25
>> >> >>> >> >> >>>>> 26 hdd 3.73630
1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB 862 GiB 77.47 1.04 48 up
osd.26
>> >> >>> >> >> >>>>> 27 hdd 3.73630
1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB 1.5 TiB 61.00 0.82 48 up
osd.27
>> >> >>> >> >> >>>>> 28 hdd 14.65039
1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30 GiB 2.8 TiB 81.19 1.09 203 up
osd.28
>> >> >>> >> >> >>>>> 29 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26 GiB 3.8 TiB 73.95 0.99 200 up
osd.29
>> >> >>> >> >> >>>>> 30 hdd 14.65039
1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28 GiB 3.1 TiB 79.07 1.06 207 up
osd.30
>> >> >>> >> >> >>>>> -7 58.89636
- 59 TiB 44 TiB 44 TiB 14 GiB 122 GiB 14 TiB 75.41 1.01 - host
s3db6
>> >> >>> >> >> >>>>> 32 hdd 3.73630
1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB 622 GiB 83.74 1.12 65 up
osd.32
>> >> >>> >> >> >>>>> 33 hdd 3.73630
0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB 740 GiB 80.67 1.08 52 up
osd.33
>> >> >>> >> >> >>>>> 34 hdd 3.73630
0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB 877 GiB 77.08 1.03 52 up
osd.34
>> >> >>> >> >> >>>>> 35 hdd 3.73630
0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB 1.4 TiB 62.18 0.83 42 up
osd.35
>> >> >>> >> >> >>>>> 36 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26 GiB 4.0 TiB 72.98 0.98 220 up
osd.36
>> >> >>> >> >> >>>>> 37 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB 3.6 TiB 75.30 1.01 200 up
osd.37
>> >> >>> >> >> >>>>> 38 hdd 14.65039
1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB 3.3 TiB 77.43 1.04 217 up
osd.38
>> >> >>> >> >> >>>>> -8 58.89636
- 59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB 12 TiB 78.98 1.06 - host
s3db7
>> >> >>> >> >> >>>>> 39 hdd 3.73630
1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB 499 GiB 86.96 1.17 43 up
osd.39
>> >> >>> >> >> >>>>> 40 hdd 3.73630
1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB 1.2 TiB 68.33 0.92 39 up
osd.40
>> >> >>> >> >> >>>>> 41 hdd 3.73630
1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB 732 GiB 80.86 1.08 64 up
osd.41
>> >> >>> >> >> >>>>> 42 hdd 3.73630
1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB 1.2 TiB 67.97 0.91 50 up
osd.42
>> >> >>> >> >> >>>>> 43 hdd 14.65039
1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28 GiB 2.9 TiB 80.32 1.08 213 up
osd.43
>> >> >>> >> >> >>>>> 44 hdd 14.65039
1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB 3.1 TiB 78.59 1.05 198 up
osd.44
>> >> >>> >> >> >>>>> 45 hdd 14.65039
1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB 2.8 TiB 81.05 1.09 214 up
osd.45
>> >> >>> >> >> >>>>> -9 51.28331
- 51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB 10 TiB 79.75 1.07 - host
s3db8
>> >> >>> >> >> >>>>> 8 hdd 7.32619
1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB 1.5 TiB 79.68 1.07 99 up
osd.8
>> >> >>> >> >> >>>>> 16 hdd 7.32619
1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB 1.4 TiB 80.25 1.08 97 up
osd.16
>> >> >>> >> >> >>>>> 31 hdd 7.32619
1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB 1.8 TiB 75.36 1.01 87 up
osd.31
>> >> >>> >> >> >>>>> 52 hdd 7.32619
1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB 1.6 TiB 78.19 1.05 113 up
osd.52
>> >> >>> >> >> >>>>> 53 hdd 7.32619
1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB 1.1 TiB 84.46 1.13 109 up
osd.53
>> >> >>> >> >> >>>>> 54 hdd 7.32619
1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB 1.7 TiB 76.73 1.03 115 up
osd.54
>> >> >>> >> >> >>>>> 55 hdd 7.32619
1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB 1.2 TiB 83.56 1.12 122 up
osd.55
>> >> >>> >> >> >>>>> -10 51.28331
- 51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB 12 TiB 76.05 1.02 - host
s3db9
>> >> >>> >> >> >>>>> 56 hdd 7.32619
1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB 2.1 TiB 71.06 0.95 105 up
osd.56
>> >> >>> >> >> >>>>> 57 hdd 7.32619
1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB 1.2 TiB 83.17 1.12 102 up
osd.57
>> >> >>> >> >> >>>>> 58 hdd 7.32619
1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB 1.4 TiB 81.56 1.09 105 up
osd.58
>> >> >>> >> >> >>>>> 59 hdd 7.32619
1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB 1.4 TiB 80.64 1.08 94 up
osd.59
>> >> >>> >> >> >>>>> 60 hdd 7.32619
1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB 2.0 TiB 73.25 0.98 101 up
osd.60
>> >> >>> >> >> >>>>> 61 hdd 7.32619
1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB 2.5 TiB 65.84 0.88 103 up
osd.61
>> >> >>> >> >> >>>>> 62 hdd 7.32619
1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB 1.7 TiB 76.83 1.03 126 up
osd.62
>> >> >>> >> >> >>>>>
TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB 173 TiB 74.57
>> >> >>> >> >> >>>>> MIN/MAX VAR:
0.17/1.20 STDDEV: 10.25
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>> Am Sa., 13. März
2021 um 15:57 Uhr schrieb Dan van der Ster <dan(a)vanderster.com>om>:
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>> No, increasing
num PGs won't help substantially.
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>> Can you share
the entire output of ceph osd df tree ?
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>> Did you already
set
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>> ceph config
set mgr mgr/balancer/upmap_max_deviation 1
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>> ??
>> >> >>> >> >> >>>>>> And I recommend
debug_mgr 4/5 so you can see some basic upmap balancer logging.
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>> .. Dan
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>>
>> >> >>> >> >> >>>>>> On Sat, Mar 13,
2021, 3:49 PM Boris Behrens <bb(a)kervyn.de> wrote:
>> >> >>> >> >> >>>>>>>
>> >> >>> >> >> >>>>>>> Hello
people,
>> >> >>> >> >> >>>>>>>
>> >> >>> >> >> >>>>>>> I am still
struggeling with the balancer
>> >> >>> >> >> >>>>>>>
(https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html)
>> >> >>> >> >> >>>>>>> Now I've
read some more and might think that I do not have enough PGs.
>> >> >>> >> >> >>>>>>> Currently I
have 84OSDs and 1024PGs for the main pool (3008 total). I
>> >> >>> >> >> >>>>>>> have the
autoscaler enabled, but I doesn't tell me to increase the
>> >> >>> >> >> >>>>>>> PGs.
>> >> >>> >> >> >>>>>>>
>> >> >>> >> >> >>>>>>> What do you
think?
>> >> >>> >> >> >>>>>>>
>> >> >>> >> >> >>>>>>> --
>> >> >>> >> >> >>>>>>> Die
Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
>> >> >>> >> >> >>>>>>> im groüen
Saal.
>> >> >>> >> >> >>>>>>>
_______________________________________________
>> >> >>> >> >> >>>>>>> ceph-users
mailing list -- ceph-users(a)ceph.io
>> >> >>> >> >> >>>>>>> To
unsubscribe send an email to ceph-users-leave(a)ceph.io
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>>
>> >> >>> >> >> >>>>> --
>> >> >>> >> >> >>>>> Die
Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen
Saal.
>> >> >>> >> >> >>>
>> >> >>> >> >> >>>
>> >> >>> >> >> >>>
>> >> >>> >> >> >>> --
>> >> >>> >> >> >>> Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> > --
>> >> >>> >> >> > Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > --
>> >> >>> >> > Die Selbsthilfegruppe
"UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > --
>> >> >>> > Die Selbsthilfegruppe "UTF-8-Probleme"
trifft sich diesmal abweichend im groüen Saal.
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal abweichend im groüen Saal.
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich
diesmal abweichend im groüen Saal.
>> >
>> >
>> >
>> > --
>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
abweichend im groüen Saal.
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
24 Mar
24 Mar
7:57 a.m.
New subject: RadosGW multiple crash
Hello!
On a two nodes ceph cluster, use mainly as s3 provider, i have manyRadosGW crashs which
probably explain why i have fail of large (multipart) files uploads.Indeed, i havent no
issue with small file (<10 Gb).
If someone here can help me dig to this issue, i'll be great!
Thanks a lot for your work!
Best regards,Kwame
cat /etc/redhat-release
Red Hat Enterprise Linux release 8.2 (Ootpa)
ceph -v
ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
# same issue with 15.2.8 and 15.2.9
ceph -s
cluster:
id: 33ccf2a3-2efc-4def-a857-ade988228e80
health: HEALTH_WARN
965 daemons have recently crashed
services:
mon: 2 daemons, quorum domaster3,domaster1 (age 10h)
mgr: domaster1(active, since 10h), standbys: domaster3
mds: 2 up:standby
osd: 4 osds: 4 up (since 10h), 4 in (since 8d)
rgw: 2 daemons active (domaster1, domaster3)
task status:
data:
pools: 8 pools, 273 pgs
objects: 34.98k objects, 93 GiB
usage: 285 GiB used, 315 GiB / 600 GiB avail
pgs: 273 active+clean
io:
client: 20 MiB/s rd, 13 op/s rd, 0 op/s wr
ceph crash ls-new<snip>
2021-03-24T07:05:03.257297Z_76400e5d-7334-4628-af67-5f5069a27f4e client.rgw.domaster1
*
2021-03-24T07:24:50.300691Z_a015c60f-c5e6-4be1-841e-e6c8720e2fb7 client.rgw.domaster3
*
2021-03-24T07:25:19.027295Z_0a1998fe-87d1-4a38-99e6-618ed36bcc24 client.rgw.domaster1
*
ceph crash info 2021-03-24T07:25:19.027295Z_0a1998fe-87d1-4a38-99e6-618ed36bcc24
{
"backtrace": [
"(()+0x12dd0) [0x7f8de7e6ddd0]",
"(gsignal()+0x10f) [0x7f8de64b270f]",
"(abort()+0x127) [0x7f8de649cb25]",
"(()+0x364928) [0x7f8df2a05928]",
"(RGWRESTStreamRWRequest::do_send_prepare(RGWAccessKey*,
std::map<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > >&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
ceph::buffer::v15_2_0::list*)+0x74d) [0x7f8df2e7f60d]",
"(RGWRESTStreamRWRequest::send_prepare(RGWAccessKey*,
std::map<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > >&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
ceph::buffer::v15_2_0::list*)+0x66) [0x7f8df2e7f7a6]",
"(RGWRESTStreamRWRequest::send_request(RGWAccessKey*,
std::map<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > >&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, RGWHTTPManager*,
ceph::buffer::v15_2_0::list*)+0x18) [0x7f8df2e7f808]",
"(RGWRESTReadResource::aio_read()+0x50) [0x7f8df2e87980]",
"(RGWReadRawRESTResourceCR::send_request()+0x75) [0x7f8df2cabfc5]",
"(RGWSimpleCoroutine::state_send_request()+0x13) [0x7f8df2d8f103]",
"(RGWSimpleCoroutine::operate()+0xac) [0x7f8df2d94cdc]",
"(RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x67)
[0x7f8df2d925c7]",
"(RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*,
std::allocator<RGWCoroutinesStack*> >&)+0x271) [0x7f8df2d933f1]",
"(RGWSyncLogTrimThread::process()+0x200) [0x7f8df2e5d2b0]",
"(RGWRadosThread::Worker::entry()+0x176) [0x7f8df2e25936]",
"(()+0x82de) [0x7f8de7e632de]",
"(clone()+0x43) [0x7f8de6576e83]"
],
"ceph_version": "15.2.4",
"crash_id":
"2021-03-24T07:25:19.027295Z_0a1998fe-87d1-4a38-99e6-618ed36bcc24",
"entity_name": "client.rgw.domaster1",
"os_id": "rhel",
"os_name": "Red Hat Enterprise Linux",
"os_version": "8.2 (Ootpa)",
"os_version_id": "8.2",
"process_name": "radosgw",
"stack_sig":
"39a3d8cc71898dd0f486ea1cace46babf5c4cdbf18d64adaf1505e7d447ee44d",
"timestamp": "2021-03-24T07:25:19.027295Z",
"utsname_hostname": "domaster1",
"utsname_machine": "x86_64",
"utsname_release": "4.18.0-193.6.3.el8_2.x86_64",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP Mon Jun 1 20:24:55 UTC 2020"
}
1122
days inactive
1133
days old
19 comments
3 participants
participants (3)
-
Boris Behrens -
Dan van der Ster -
Kwame Amedodji