26 Feb
2021
26 Feb
'21
1:33 a.m.
Hi,
We've got an interesting issue we're running into on Ceph 15.2.9. We're
experiencing VERY slow performance from the cluster, and extremely slow
misplaced object correction, with very little cpu/disk/network utilization
(almost idle) across all nodes in the cluster.
We have 7 servers in this cluster, 24 rotational OSDs, and two NVMEs with
12 OSD's worth of DB/WAL files on them. The OSDs are all equal weighted, so
the tree is pretty straightforward:
root@ceph01:~# ceph osd tree
Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 2149.39062 root default
-2 2149.39062 rack rack1
-5 307.05579 host ceph01
0 hdd 12.79399 osd.0 up 1.00000 1.00000
1 hdd 12.79399 osd.1 up 1.00000 1.00000
2 hdd 12.79399 osd.2 up 1.00000 1.00000
3 hdd 12.79399 osd.3 up 1.00000 1.00000
4 hdd 12.79399 osd.4 up 1.00000 1.00000
5 hdd 12.79399 osd.5 up 1.00000 1.00000
6 hdd 12.79399 osd.6 up 1.00000 1.00000
7 hdd 12.79399 osd.7 up 1.00000 1.00000
8 hdd 12.79399 osd.8 up 1.00000 1.00000
9 hdd 12.79399 osd.9 up 1.00000 1.00000
10 hdd 12.79399 osd.10 up 1.00000 1.00000
11 hdd 12.79399 osd.11 up 1.00000 1.00000
12 hdd 12.79399 osd.12 up 1.00000 1.00000
13 hdd 12.79399 osd.13 up 1.00000 1.00000
14 hdd 12.79399 osd.14 up 1.00000 1.00000
15 hdd 12.79399 osd.15 up 1.00000 1.00000
16 hdd 12.79399 osd.16 up 1.00000 1.00000
17 hdd 12.79399 osd.17 up 1.00000 1.00000
18 hdd 12.79399 osd.18 up 1.00000 1.00000
19 hdd 12.79399 osd.19 up 1.00000 1.00000
20 hdd 12.79399 osd.20 up 1.00000 1.00000
21 hdd 12.79399 osd.21 up 1.00000 1.00000
22 hdd 12.79399 osd.22 up 1.00000 1.00000
23 hdd 12.79399 osd.23 up 1.00000 1.00000
-7 307.05579 host ceph02
24 hdd 12.79399 osd.24 up 1.00000 1.00000
25 hdd 12.79399 osd.25 up 1.00000 1.00000
26 hdd 12.79399 osd.26 up 1.00000 1.00000
27 hdd 12.79399 osd.27 up 1.00000 1.00000
28 hdd 12.79399 osd.28 up 1.00000 1.00000
29 hdd 12.79399 osd.29 up 1.00000 1.00000
30 hdd 12.79399 osd.30 up 1.00000 1.00000
31 hdd 12.79399 osd.31 up 1.00000 1.00000
32 hdd 12.79399 osd.32 up 1.00000 1.00000
33 hdd 12.79399 osd.33 up 1.00000 1.00000
34 hdd 12.79399 osd.34 up 1.00000 1.00000
35 hdd 12.79399 osd.35 up 1.00000 1.00000
36 hdd 12.79399 osd.36 up 1.00000 1.00000
37 hdd 12.79399 osd.37 up 1.00000 1.00000
38 hdd 12.79399 osd.38 up 1.00000 1.00000
39 hdd 12.79399 osd.39 up 1.00000 1.00000
40 hdd 12.79399 osd.40 up 1.00000 1.00000
41 hdd 12.79399 osd.41 up 1.00000 1.00000
42 hdd 12.79399 osd.42 up 1.00000 1.00000
43 hdd 12.79399 osd.43 up 1.00000 1.00000
44 hdd 12.79399 osd.44 up 1.00000 1.00000
45 hdd 12.79399 osd.45 up 1.00000 1.00000
46 hdd 12.79399 osd.46 up 1.00000 1.00000
47 hdd 12.79399 osd.47 up 1.00000 1.00000
-9 307.05579 host ceph03
48 hdd 12.79399 osd.48 up 1.00000 1.00000
49 hdd 12.79399 osd.49 up 1.00000 1.00000
50 hdd 12.79399 osd.50 up 1.00000 1.00000
51 hdd 12.79399 osd.51 up 1.00000 1.00000
52 hdd 12.79399 osd.52 up 1.00000 1.00000
53 hdd 12.79399 osd.53 up 1.00000 1.00000
54 hdd 12.79399 osd.54 up 1.00000 1.00000
55 hdd 12.79399 osd.55 up 1.00000 1.00000
56 hdd 12.79399 osd.56 up 1.00000 1.00000
57 hdd 12.79399 osd.57 up 1.00000 1.00000
58 hdd 12.79399 osd.58 up 1.00000 1.00000
59 hdd 12.79399 osd.59 up 1.00000 1.00000
60 hdd 12.79399 osd.60 up 1.00000 1.00000
61 hdd 12.79399 osd.61 up 1.00000 1.00000
62 hdd 12.79399 osd.62 up 1.00000 1.00000
63 hdd 12.79399 osd.63 up 1.00000 1.00000
64 hdd 12.79399 osd.64 up 1.00000 1.00000
65 hdd 12.79399 osd.65 up 1.00000 1.00000
66 hdd 12.79399 osd.66 up 1.00000 1.00000
67 hdd 12.79399 osd.67 up 1.00000 1.00000
68 hdd 12.79399 osd.68 up 1.00000 1.00000
69 hdd 12.79399 osd.69 up 1.00000 1.00000
70 hdd 12.79399 osd.70 up 1.00000 1.00000
71 hdd 12.79399 osd.71 up 1.00000 1.00000
-11 307.05579 host ceph04
72 hdd 12.79399 osd.72 up 1.00000 1.00000
73 hdd 12.79399 osd.73 up 1.00000 1.00000
74 hdd 12.79399 osd.74 up 1.00000 1.00000
75 hdd 12.79399 osd.75 up 1.00000 1.00000
76 hdd 12.79399 osd.76 up 1.00000 1.00000
77 hdd 12.79399 osd.77 up 1.00000 1.00000
78 hdd 12.79399 osd.78 up 1.00000 1.00000
79 hdd 12.79399 osd.79 up 1.00000 1.00000
80 hdd 12.79399 osd.80 up 1.00000 1.00000
81 hdd 12.79399 osd.81 up 1.00000 1.00000
82 hdd 12.79399 osd.82 up 1.00000 1.00000
83 hdd 12.79399 osd.83 up 1.00000 1.00000
84 hdd 12.79399 osd.84 up 1.00000 1.00000
85 hdd 12.79399 osd.85 up 1.00000 1.00000
86 hdd 12.79399 osd.86 up 1.00000 1.00000
87 hdd 12.79399 osd.87 up 1.00000 1.00000
88 hdd 12.79399 osd.88 up 1.00000 1.00000
89 hdd 12.79399 osd.89 up 1.00000 1.00000
90 hdd 12.79399 osd.90 up 1.00000 1.00000
91 hdd 12.79399 osd.91 up 1.00000 1.00000
92 hdd 12.79399 osd.92 up 1.00000 1.00000
93 hdd 12.79399 osd.93 up 1.00000 1.00000
94 hdd 12.79399 osd.94 up 1.00000 1.00000
95 hdd 12.79399 osd.95 up 1.00000 1.00000
-13 307.05579 host ceph05
96 hdd 12.79399 osd.96 up 1.00000 1.00000
97 hdd 12.79399 osd.97 up 1.00000 1.00000
98 hdd 12.79399 osd.98 up 1.00000 1.00000
99 hdd 12.79399 osd.99 up 1.00000 1.00000
100 hdd 12.79399 osd.100 up 1.00000 1.00000
101 hdd 12.79399 osd.101 up 1.00000 1.00000
102 hdd 12.79399 osd.102 up 1.00000 1.00000
103 hdd 12.79399 osd.103 up 1.00000 1.00000
104 hdd 12.79399 osd.104 up 1.00000 1.00000
105 hdd 12.79399 osd.105 up 1.00000 1.00000
106 hdd 12.79399 osd.106 up 1.00000 1.00000
107 hdd 12.79399 osd.107 up 1.00000 1.00000
108 hdd 12.79399 osd.108 up 1.00000 1.00000
109 hdd 12.79399 osd.109 up 1.00000 1.00000
110 hdd 12.79399 osd.110 up 1.00000 1.00000
111 hdd 12.79399 osd.111 up 1.00000 1.00000
112 hdd 12.79399 osd.112 up 1.00000 1.00000
113 hdd 12.79399 osd.113 up 1.00000 1.00000
114 hdd 12.79399 osd.114 up 1.00000 1.00000
115 hdd 12.79399 osd.115 up 1.00000 1.00000
116 hdd 12.79399 osd.116 up 1.00000 1.00000
117 hdd 12.79399 osd.117 up 1.00000 1.00000
118 hdd 12.79399 osd.118 up 1.00000 1.00000
119 hdd 12.79399 osd.119 up 1.00000 1.00000
-15 307.05579 host ceph06
120 hdd 12.79399 osd.120 up 1.00000 1.00000
121 hdd 12.79399 osd.121 up 1.00000 1.00000
122 hdd 12.79399 osd.122 up 1.00000 1.00000
123 hdd 12.79399 osd.123 up 1.00000 1.00000
124 hdd 12.79399 osd.124 up 1.00000 1.00000
125 hdd 12.79399 osd.125 up 1.00000 1.00000
126 hdd 12.79399 osd.126 up 1.00000 1.00000
127 hdd 12.79399 osd.127 up 1.00000 1.00000
128 hdd 12.79399 osd.128 up 1.00000 1.00000
129 hdd 12.79399 osd.129 up 1.00000 1.00000
130 hdd 12.79399 osd.130 up 1.00000 1.00000
131 hdd 12.79399 osd.131 up 1.00000 1.00000
132 hdd 12.79399 osd.132 up 1.00000 1.00000
133 hdd 12.79399 osd.133 up 1.00000 1.00000
134 hdd 12.79399 osd.134 up 1.00000 1.00000
135 hdd 12.79399 osd.135 up 1.00000 1.00000
136 hdd 12.79399 osd.136 up 1.00000 1.00000
137 hdd 12.79399 osd.137 up 1.00000 1.00000
138 hdd 12.79399 osd.138 up 1.00000 1.00000
139 hdd 12.79399 osd.139 up 1.00000 1.00000
140 hdd 12.79399 osd.140 up 1.00000 1.00000
141 hdd 12.79399 osd.141 up 1.00000 1.00000
142 hdd 12.79399 osd.142 up 1.00000 1.00000
143 hdd 12.79399 osd.143 up 1.00000 1.00000
-17 307.05579 host ceph07
144 hdd 12.79399 osd.144 up 1.00000 1.00000
145 hdd 12.79399 osd.145 up 1.00000 1.00000
146 hdd 12.79399 osd.146 up 1.00000 1.00000
147 hdd 12.79399 osd.147 up 1.00000 1.00000
148 hdd 12.79399 osd.148 up 1.00000 1.00000
149 hdd 12.79399 osd.149 up 1.00000 1.00000
150 hdd 12.79399 osd.150 up 1.00000 1.00000
151 hdd 12.79399 osd.151 up 1.00000 1.00000
152 hdd 12.79399 osd.152 up 1.00000 1.00000
153 hdd 12.79399 osd.153 up 1.00000 1.00000
154 hdd 12.79399 osd.154 up 1.00000 1.00000
155 hdd 12.79399 osd.155 up 1.00000 1.00000
156 hdd 12.79399 osd.156 up 1.00000 1.00000
157 hdd 12.79399 osd.157 up 1.00000 1.00000
158 hdd 12.79399 osd.158 up 1.00000 1.00000
159 hdd 12.79399 osd.159 up 1.00000 1.00000
160 hdd 12.79399 osd.160 up 1.00000 1.00000
161 hdd 12.79399 osd.161 up 1.00000 1.00000
162 hdd 12.79399 osd.162 up 1.00000 1.00000
163 hdd 12.79399 osd.163 up 1.00000 1.00000
164 hdd 12.79399 osd.164 up 1.00000 1.00000
165 hdd 12.79399 osd.165 up 1.00000 1.00000
166 hdd 12.79399 osd.166 up 1.00000 1.00000
167 hdd 12.79399 osd.167 up 1.00000 1.00000
root@ceph01:~#
The data distribution looks relatively even (we use upmap as the balancer):
root@ceph01:~# ceph osd df
Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS
0 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.0 MiB 14
GiB 7.1 TiB 44.73 0.93 35 up
1 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.8 MiB 13
GiB 7.2 TiB 43.46 0.90 33 up
2 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.4 MiB 16
GiB 6.2 TiB 51.27 1.07 39 up
3 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 4.9 MiB 15
GiB 6.7 TiB 47.38 0.99 36 up
4 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 812 KiB 13
GiB 7.2 TiB 43.37 0.90 35 up
5 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 31 KiB 16
GiB 6.2 TiB 51.27 1.07 40 up
6 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 506 KiB 13
GiB 7.3 TiB 43.32 0.90 34 up
7 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 904 KiB 15
GiB 6.4 TiB 49.93 1.04 38 up
8 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.6 MiB 14
GiB 6.7 TiB 47.35 0.98 37 up
9 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.5 MiB 16
GiB 6.1 TiB 52.58 1.09 40 up
10 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.6 TiB 75 MiB 16
GiB 6.1 TiB 52.42 1.09 41 up
11 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.4 MiB 13
GiB 7.2 TiB 43.36 0.90 34 up
12 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 586 KiB 13
GiB 7.3 TiB 43.31 0.90 33 up
13 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 222 KiB 14
GiB 7.1 TiB 44.71 0.93 35 up
14 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16
GiB 6.1 TiB 52.58 1.09 41 up
15 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.2 MiB 14
GiB 7.1 TiB 44.73 0.93 34 up
16 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16
GiB 6.1 TiB 52.53 1.09 40 up
17 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.3 MiB 15
GiB 6.4 TiB 49.92 1.04 38 up
18 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16
GiB 6.1 TiB 52.61 1.09 45 up
19 hdd 12.79399 1.00000 13 TiB 5.4 TiB 5.3 TiB 4.0 MiB 13
GiB 7.4 TiB 42.15 0.88 34 up
20 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 705 KiB 16
GiB 6.1 TiB 52.53 1.09 40 up
21 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.5 MiB 16
GiB 6.1 TiB 52.58 1.09 40 up
22 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.7 MiB 16
GiB 6.1 TiB 52.59 1.09 40 up
23 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.9 MiB 16
GiB 6.1 TiB 52.58 1.09 41 up
24 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14
GiB 6.7 TiB 47.35 0.98 36 up
25 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.1 MiB 16
GiB 6.1 TiB 52.56 1.09 41 up
26 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 592 KiB 14
GiB 7.1 TiB 44.75 0.93 34 up
27 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.8 MiB 14
GiB 6.9 TiB 46.07 0.96 36 up
28 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.4 MiB 16
GiB 6.1 TiB 52.56 1.09 40 up
29 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 340 KiB 15
GiB 6.4 TiB 49.92 1.04 39 up
30 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 93 KiB 14
GiB 6.9 TiB 46.03 0.96 35 up
31 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.1 MiB 13
GiB 7.2 TiB 43.57 0.91 35 up
32 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 5.0 MiB 15
GiB 6.6 TiB 48.68 1.01 38 up
33 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 175 KiB 13
GiB 7.2 TiB 43.44 0.90 33 up
34 hdd 12.79399 1.00000 13 TiB 6.9 TiB 6.8 TiB 77 MiB 16
GiB 5.9 TiB 53.86 1.12 43 up
35 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 2.3 MiB 15
GiB 6.6 TiB 48.68 1.01 37 up
36 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.2 MiB 16
GiB 6.2 TiB 51.25 1.07 41 up
37 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 5.0 MiB 14
GiB 6.7 TiB 47.37 0.99 36 up
38 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.7 MiB 13
GiB 7.2 TiB 43.44 0.90 35 up
39 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 341 KiB 15
GiB 6.7 TiB 47.38 0.99 37 up
40 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.1 MiB 14
GiB 7.1 TiB 44.73 0.93 34 up
41 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 22 KiB 16
GiB 6.1 TiB 52.59 1.09 40 up
42 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 1.9 MiB 14
GiB 6.7 TiB 47.33 0.98 36 up
43 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 485 KiB 16
GiB 6.1 TiB 52.59 1.09 40 up
44 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.1 MiB 16
GiB 6.1 TiB 52.56 1.09 40 up
45 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.3 MiB 16
GiB 6.1 TiB 52.53 1.09 40 up
46 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 124 KiB 14
GiB 6.9 TiB 46.08 0.96 37 up
47 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.2 MiB 13
GiB 7.2 TiB 43.37 0.90 33 up
48 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 141 KiB 16
GiB 6.1 TiB 52.58 1.09 40 up
49 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.9 MiB 13
GiB 7.2 TiB 43.64 0.91 33 up
50 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 745 KiB 16
GiB 6.1 TiB 52.52 1.09 41 up
51 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 186 KiB 16
GiB 6.1 TiB 52.55 1.09 41 up
52 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.0 MiB 16
GiB 6.2 TiB 51.24 1.07 41 up
53 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 737 KiB 14
GiB 6.7 TiB 47.33 0.98 38 up
54 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.3 MiB 14
GiB 7.1 TiB 44.66 0.93 34 up
55 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 97 KiB 14
GiB 7.1 TiB 44.73 0.93 34 up
56 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.2 MiB 14
GiB 7.1 TiB 44.76 0.93 35 up
57 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.5 MiB 14
GiB 7.1 TiB 44.78 0.93 35 up
58 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 80 MiB 16
GiB 6.1 TiB 52.56 1.09 41 up
59 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 4.5 MiB 14
GiB 6.9 TiB 46.08 0.96 36 up
60 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 903 KiB 14
GiB 7.1 TiB 44.73 0.93 34 up
61 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1 KiB 14
GiB 7.1 TiB 44.65 0.93 34 up
62 hdd 12.79399 1.00000 13 TiB 6.9 TiB 6.8 TiB 425 KiB 16
GiB 5.9 TiB 53.87 1.12 41 up
63 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.1 MiB 14
GiB 7.2 TiB 43.56 0.91 34 up
64 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 578 KiB 16
GiB 6.2 TiB 51.27 1.07 39 up
65 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.6 MiB 13
GiB 7.2 TiB 43.44 0.90 33 up
66 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 288 KiB 13
GiB 7.2 TiB 43.40 0.90 34 up
67 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 413 KiB 14
GiB 6.7 TiB 47.34 0.98 37 up
68 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.3 MiB 16
GiB 6.1 TiB 52.56 1.09 41 up
69 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.0 MiB 13
GiB 7.2 TiB 43.44 0.90 33 up
70 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.2 MiB 16
GiB 6.1 TiB 52.56 1.09 40 up
71 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.3 MiB 14
GiB 7.2 TiB 43.36 0.90 35 up
72 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.7 MiB 16
GiB 6.2 TiB 51.29 1.07 39 up
73 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14
GiB 7.1 TiB 44.73 0.93 34 up
74 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.6 TiB 1.2 MiB 14
GiB 7.1 TiB 44.64 0.93 34 up
75 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 1.5 MiB 15
GiB 6.4 TiB 49.95 1.04 38 up
76 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 3.8 MiB 15
GiB 6.6 TiB 48.68 1.01 37 up
77 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.0 MiB 14
GiB 7.1 TiB 44.65 0.93 34 up
78 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14
GiB 7.1 TiB 44.68 0.93 35 up
79 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 984 KiB 14
GiB 6.7 TiB 47.36 0.98 36 up
80 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.6 MiB 13
GiB 7.2 TiB 43.45 0.90 34 up
81 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.2 MiB 16
GiB 6.2 TiB 51.26 1.07 41 up
82 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 4.9 MiB 15
GiB 6.6 TiB 48.61 1.01 37 up
83 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 827 KiB 14
GiB 6.7 TiB 47.33 0.98 36 up
84 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 2.7 MiB 16
GiB 6.2 TiB 51.22 1.07 39 up
85 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 868 KiB 15
GiB 6.4 TiB 49.94 1.04 40 up
86 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.1 MiB 16
GiB 6.1 TiB 52.57 1.09 42 up
87 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 702 KiB 16
GiB 6.1 TiB 52.54 1.09 41 up
88 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 12 KiB 14
GiB 7.1 TiB 44.76 0.93 35 up
89 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.9 MiB 14
GiB 6.9 TiB 46.07 0.96 35 up
90 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.8 MiB 13
GiB 7.2 TiB 43.48 0.90 34 up
91 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 836 KiB 16
GiB 6.1 TiB 52.55 1.09 40 up
92 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.0 MiB 13
GiB 7.2 TiB 43.43 0.90 33 up
93 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.5 MiB 16
GiB 6.1 TiB 52.57 1.09 41 up
94 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 459 KiB 13
GiB 7.2 TiB 43.44 0.90 34 up
95 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.1 MiB 16
GiB 6.2 TiB 51.28 1.07 39 up
96 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.1 MiB 14
GiB 7.1 TiB 44.75 0.93 35 up
97 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 662 KiB 14
GiB 7.1 TiB 44.75 0.93 34 up
98 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.9 MiB 14
GiB 6.9 TiB 45.97 0.96 35 up
99 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.8 MiB 16
GiB 6.1 TiB 52.56 1.09 40 up
100 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14
GiB 7.1 TiB 44.76 0.93 36 up
101 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.6 MiB 15
GiB 6.4 TiB 49.95 1.04 40 up
102 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.9 MiB 14
GiB 7.1 TiB 44.65 0.93 34 up
103 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 643 KiB 16
GiB 6.1 TiB 52.57 1.09 41 up
104 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 5.5 MiB 15
GiB 6.4 TiB 49.98 1.04 38 up
105 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 830 KiB 13
GiB 7.2 TiB 43.46 0.90 33 up
106 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 1.0 MiB 14
GiB 6.7 TiB 47.34 0.98 36 up
107 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 286 KiB 16
GiB 6.2 TiB 51.23 1.07 39 up
108 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 961 KiB 15
GiB 6.6 TiB 48.68 1.01 37 up
109 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 291 KiB 16
GiB 6.1 TiB 52.56 1.09 42 up
110 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1.0 MiB 14
GiB 6.9 TiB 46.02 0.96 36 up
111 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 2.7 MiB 16
GiB 6.2 TiB 51.22 1.07 39 up
112 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 631 KiB 15
GiB 6.6 TiB 48.67 1.01 37 up
113 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.5 MiB 16
GiB 6.1 TiB 52.56 1.09 41 up
114 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.0 MiB 15
GiB 6.4 TiB 49.94 1.04 39 up
115 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1.5 MiB 14
GiB 6.9 TiB 46.05 0.96 35 up
116 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.1 MiB 15
GiB 6.4 TiB 49.93 1.04 38 up
117 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 194 KiB 13
GiB 7.2 TiB 43.44 0.90 35 up
118 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.1 MiB 15
GiB 6.4 TiB 49.96 1.04 38 up
119 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.9 MiB 14
GiB 7.2 TiB 43.46 0.90 33 up
120 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.8 MiB 16
GiB 6.1 TiB 52.55 1.09 40 up
121 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.7 MiB 16
GiB 6.1 TiB 52.54 1.09 40 up
122 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 2.4 MiB 15
GiB 6.4 TiB 49.97 1.04 38 up
123 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1003 KiB 14
GiB 6.9 TiB 46.02 0.96 35 up
124 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 526 KiB 14
GiB 6.7 TiB 47.35 0.98 36 up
125 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 92 KiB 15
GiB 6.7 TiB 47.37 0.98 37 up
126 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.9 MiB 14
GiB 7.1 TiB 44.74 0.93 34 up
127 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.5 MiB 16
GiB 6.2 TiB 51.26 1.07 39 up
128 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.4 MiB 16
GiB 6.1 TiB 52.56 1.09 40 up
129 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 3.6 MiB 15
GiB 6.6 TiB 48.66 1.01 38 up
130 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.8 MiB 16
GiB 6.1 TiB 52.56 1.09 41 up
131 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.1 MiB 16
GiB 6.2 TiB 51.28 1.07 40 up
132 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.7 MiB 16
GiB 6.2 TiB 51.25 1.07 39 up
133 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 230 KiB 16
GiB 6.2 TiB 51.25 1.07 39 up
134 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.0 MiB 14
GiB 7.2 TiB 43.45 0.90 34 up
135 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.4 MiB 13
GiB 7.2 TiB 43.46 0.90 33 up
136 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.6 MiB 16
GiB 6.2 TiB 51.29 1.07 39 up
137 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.6 MiB 14
GiB 7.1 TiB 44.75 0.93 36 up
138 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.9 MiB 15
GiB 6.4 TiB 49.98 1.04 40 up
139 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 21 KiB 16
GiB 6.2 TiB 51.23 1.07 39 up
140 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.4 MiB 14
GiB 7.1 TiB 44.74 0.93 36 up
141 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 359 KiB 13
GiB 7.2 TiB 43.42 0.90 34 up
142 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 27 KiB 13
GiB 7.2 TiB 43.46 0.90 33 up
143 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 395 KiB 14
GiB 6.9 TiB 46.07 0.96 35 up
144 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.2 MiB 14
GiB 7.1 TiB 44.75 0.93 34 up
145 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.3 MiB 14
GiB 7.1 TiB 44.66 0.93 35 up
146 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 4.0 MiB 14
GiB 6.9 TiB 46.06 0.96 35 up
147 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 46 KiB 16
GiB 6.1 TiB 52.55 1.09 40 up
148 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.8 MiB 16
GiB 6.2 TiB 51.23 1.07 39 up
149 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.4 MiB 14
GiB 7.1 TiB 44.70 0.93 35 up
150 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.4 MiB 14
GiB 7.1 TiB 44.65 0.93 35 up
151 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.7 MiB 14
GiB 7.1 TiB 44.65 0.93 35 up
152 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.3 MiB 15
GiB 6.2 TiB 51.26 1.07 39 up
153 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 2.3 MiB 15
GiB 6.4 TiB 49.94 1.04 39 up
154 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14
GiB 6.7 TiB 47.37 0.99 36 up
155 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14
GiB 6.7 TiB 47.35 0.98 38 up
156 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.0 MiB 16
GiB 6.2 TiB 51.26 1.07 41 up
157 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.8 MiB 14
GiB 6.9 TiB 46.07 0.96 36 up
158 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 3.0 MiB 16
GiB 6.2 TiB 51.25 1.07 40 up
159 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.9 MiB 15
GiB 6.4 TiB 49.93 1.04 38 up
160 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.4 MiB 14
GiB 6.9 TiB 46.03 0.96 35 up
161 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.1 MiB 16
GiB 6.2 TiB 51.25 1.07 39 up
162 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.6 TiB 4.1 MiB 13
GiB 7.1 TiB 44.64 0.93 34 up
163 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 445 KiB 14
GiB 7.1 TiB 44.69 0.93 34 up
164 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.9 MiB 15
GiB 6.4 TiB 49.97 1.04 38 up
165 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.5 MiB 16
GiB 6.2 TiB 51.25 1.07 40 up
166 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.5 MiB 13
GiB 7.2 TiB 43.45 0.90 33 up
167 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.0 MiB 15
GiB 6.4 TiB 49.98 1.04 38 up
TOTAL 2.1 PiB 1.0 PiB 1023 TiB 630 MiB 2.4
TiB 1.1 PiB 48.09
MIN/MAX VAR: 0.88/1.12 STDDEV: 3.50
root@ceph01:~#
The cluster status, however, shows we have many misplaced objects and a few
remapped PGs. We had been performing maintenance (upgrade to Ceph 15.2.9),
and initially the misplaced object count was dropping rapidly at a rate of
about 400-500MB/s and 100-200 objects/s. However, once it hit a certain
point, it dropped to 0-4MB/s and 0 objects/s (it's not 0, it is
progressing, but it's below 1 object/s):
root@ceph01:~# ceph -s
Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
cluster:
id: 41bb9256-c3bf-11ea-85b9-9e07b0435492
health: HEALTH_WARN
5 slow ops, oldest one blocked for 84 sec, daemons
[osd.141,osd.46] have slow ops.
services:
mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 25h)
mgr: ceph05.yropto(active, since 11h), standbys: ceph01.aqkgbl,
ceph03.ytkuyr, ceph04.smbdew, ceph02.ndynmo
osd: 168 osds: 168 up (since 9h), 168 in (since 6M); 10 remapped pgs
task status:
data:
pools: 3 pools, 1057 pgs
objects: 166.90M objects, 632 TiB
usage: 1.0 PiB used, 1.1 PiB / 2.1 PiB avail
pgs: 583177/1001370417 objects misplaced (0.058%)
1039 active+clean
11 active+clean+scrubbing
5 active+remapped+backfilling
1 active+clean+scrubbing+deep
1 active+clean+laggy
io:
client: 4.5 MiB/s rd, 31 MiB/s wr, 75 op/s rd, 59 op/s wr
recovery: 2.7 MiB/s, 0 objects/s
root@ceph01:~#
root@ceph01:~# ceph pg dump |grep remapped
Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
2.3d1 163805 0 0 0 0
682273001472 0 0 7462 7462
active+*remapped*+backfilling
2021-02-26T01:28:06.212904+0000 53782'2773494 53782:5593409
[48,122,99,92,158,31] 48 [48,122,99,92,158,34] 48
50907'2762333 2021-02-23T20:13:50.501757+0000 50907'2762333
2021-02-23T20:13:50.501757+0000 0
2.341 163180 0 0 136844 0
679698268160 0 0 8569 8569
active+*remapped*+backfilling
2021-02-26T01:27:30.693520+0000 53782'2694466 53782:5497907
[65,125,79,147,45,100] 65 [65,18,79,147,45,100] 65
51073'2688628 2021-02-24T09:38:41.588350+0000 51073'2688628
2021-02-24T09:38:41.588350+0000 0
2.1e1 162402 0 0 150624 0
676505321472 0 0 7893 7893
active+*remapped*+backfilling
2021-02-25T15:36:29.121352+0000 53782'3105605 53782:5885140
[112,63,126,0,147,25] 112 [112,50,126,0,147,25] 112
50907'3086340 2021-02-22T23:14:49.510535+0000 47263'2883486
2021-02-01T15:43:32.089129+0000 0
2.268 163459 0 0 136459 0
680743612416 0 0 7337 7337
active+*remapped*+backfilling
2021-02-25T15:27:58.957497+0000 53782'2713107 53782:5536140
[93,98,150,49,26,135] 93 [93,98,150,62,26,135] 93
50907'2698319 2021-02-23T14:16:11.171370+0000 50907'2698319
2021-02-23T14:16:11.171370+0000 0
2.285 163380 0 0 158866 0
680507670528 0 0 8148 8148
active+*remapped*+backfilling
2021-02-26T01:26:31.442018+0000 53782'3191803 53782:5986426
[67,166,37,90,114,138] 67 [67,147,37,90,114,138] 67
51073'3177251 2021-02-23T23:51:51.751625+0000 47597'2917541
2021-02-03T20:14:38.762384+0000 0
dumped all
root@ceph01:~#
Even if we stop the scrubs/deep scrubs, things don't improve re: recovery
rate or cluster performance. We've monitored the HDDs/NVME backing these
OSDs and with the scrubs stopped, they are nearly idle. CPU is almost idle.
There's nothing to indicate any problem at all with the underlying hardware
that we can find, at least from a load/failure perspective.
Also, interestingly, rados bench on a newly created **replicated** pool is
also extremely slow, as in < 50MB/s.
Crush rule for our EC (4+2) pool, but we see the performance issues on even
a newly created triple-replicated pool:
root@ceph01:~# ceph osd crush rule dump ecpool
Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
{
"rule_id": 1,
"rule_name": "ecpool",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 6,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
Cluster/pool space:
root@ceph01:~# ceph df
Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492
Inferring config
/var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config
Using recent ceph image
docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 2.1 PiB 1.1 PiB 1.0 PiB 1.0 PiB 48.09
TOTAL 2.1 PiB 1.1 PiB 1.0 PiB 1.0 PiB 48.09
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX
AVAIL
device_health_metrics 1 1 78 MiB 182 233 MiB 0 295
TiB
ecpool 2 1024 632 TiB 166.90M 1023 TiB 53.65 589
TiB
rbd 3 32 59 MiB 5 178 MiB 0 295
TiB
root@ceph01:~#
We're at a loss for why this cluster has suddenly started behaving this
way. With the disks (all of them, on all nodes) effectively being idle with
all the scrubs disabled, plenty of CPU available, and no network issues
(iperf on all paths is full line rate - ~10GB/s - we have a dedicated
public and cluster network), it's befuddling.
We appreciate any assistance you can provide, and are happy to provide any
information that might be useful.