16 Mar
2023
16 Mar
'23
12:34 p.m.
Hi all,
I have a 9 node cluster running *Pacific 16.2.10*. OSDs live on 9 of the
nodes with each one having 4 x 1.8T ssd and 8 x 10.9T hdd for a total of
108 OSDs. We create three crush roots as belows.
1. The hdds (8x9=72) of all nodes form a large crush root, which is used as
a data pool, and object storage and cephfs share this crush root.
2. Take 3 ssds from the 4 ssds of each node as rbd block storage.
3. An ssd on each remaining node is used as an index pool for cephfs and
object storage.
[root@node01 smd]# ceph osd treeID CLASS WEIGHT TYPE NAME
STATUS REWEIGHT PRI-AFF
-92 15.71910 root
root-1c31624a-ad18-445e-8e42-86b71c1fd76f
-112 1.74657
host node01-fa2cdf3e-7212-4b5f-b62a-3ab1e803547f
13 ssd 1.74657
osd.13 up 1.00000
1.00000 -103 1.74657
host node02-4e232f27-fe4b-4d0e-bd2a-67d5006a0cdd
34 ssd 1.74657
osd.34 up 1.00000
1.00000 -109 1.74657
host node03-3ae63d7a-9f65-4bea-b2ba-ff3fe342753d
28 ssd 1.74657
osd.28 up 1.00000
1.00000 -118 1.74657
host node04-37a3f92a-f6d8-41f9-a774-3069fc2f50b8
54 ssd 1.74657
osd.54 up 1.00000
1.00000 -106 1.74657
host node05-f667fa27-cc13-4b93-ad56-5dc4c31ffd77
53 ssd 1.74657
osd.53 up 1.00000
1.00000 -91 1.74657
host node06-3808c8f6-8e10-47c7-8456-62c1e0e800ed
61 ssd 1.74657
osd.61 up 1.00000
1.00000 -97 1.74657
host node07-78216b0d-0999-44e8-905d-8737a5f6f51f
50 ssd 1.74657
osd.50 up 1.00000
1.00000 -115 1.74657
host node08-947bd556-fb06-497d-8f2c-c4a679d2b06f
86 ssd 1.74657
osd.86 up 1.00000
1.00000 -100 1.74657
host node09-d9ae9046-0716-454f-ba0c-b03cf9986ba8
85 ssd 1.74657 osd.85
up 1.00000 1.00000
-38 785.80701 root root-6041a4dc-7c9a-44ed-999c-a847cca81012
-85 87.31189 host
node01-e5646053-2cf8-4ba5-90d5-bb1a63b1234c
1 hdd 10.91399 osd.1
up 1.00000 1.00000 22 hdd 10.91399 osd.22
up 0.90002 1.00000 31
hdd 10.91399 osd.31
up 1.00000 1.00000 51 hdd 10.91399 osd.51
up 1.00000 1.00000 60 hdd
10.91399 osd.60 up
1.00000 1.00000 70 hdd 10.91399 osd.70
up 1.00000 1.00000 78 hdd
10.91399 osd.78 up
1.00000 1.00000 96 hdd
10.91399 osd.96 up
1.00000 1.00000 -37
87.31189 host node02-be9925fd-60de-4147-81eb-720d7145715f
9 hdd
10.91399 osd.9 up
1.00000 1.00000 19 hdd
10.91399 osd.19 up
1.00000 1.00000 29 hdd
10.91399 osd.29 up
1.00000 1.00000 47 hdd
10.91399 osd.47 up
1.00000 1.00000 56 hdd
10.91399 osd.56 up
1.00000 1.00000 65 hdd
10.91399 osd.65 up
1.00000 1.00000 88 hdd
10.91399 osd.88 up
1.00000 1.00000 98 hdd
10.91399 osd.98 up
1.00000 1.00000 -52
87.31189 host node03-7828653d-6033-4e88-92b0-d8709b0ab218
2 hdd
10.91399 osd.2 up
1.00000 1.00000 30 hdd
10.91399 osd.30 up
1.00000 1.00000 40 hdd
10.91399 osd.40 up
1.00000 1.00000 48 hdd
10.91399 osd.48 up
1.00000 1.00000 58 hdd
10.91399 osd.58 up
1.00000 1.00000 74 hdd
10.91399 osd.74 up
1.00000 1.00000 83 hdd 10.91399 osd.83
up 1.00000 1.00000 92 hdd
10.91399 osd.92 up
1.00000 1.00000 -46 87.31189 host
node04-e986c3fc-a21b-44ff-9b02-b60b82ee63d7
12 hdd 10.91399 osd.12
up 1.00000 1.00000 23 hdd 10.91399
osd.23 up 1.00000
1.00000 32 hdd 10.91399 osd.32
up 1.00000 1.00000 43 hdd 10.91399
osd.43 up 1.00000
1.00000 52 hdd 10.91399 osd.52
up 1.00000 1.00000 71 hdd 10.91399
osd.71 up 1.00000
1.00000 95 hdd 10.91399
osd.95 up 1.00000
1.00000 104 hdd 10.91399 osd.104
up 1.00000 1.00000 -88 87.31189
host node05-fe31d85f-b3b9-4393-b24b-030dbcdfacea
3 hdd 10.91399 osd.3
up 1.00000 1.00000 24 hdd 10.91399
osd.24 up 1.00000
1.00000 33 hdd 10.91399 osd.33
up 1.00000 1.00000 45 hdd 10.91399
osd.45 up 1.00000
1.00000 69 hdd 10.91399 osd.69
up 1.00000 1.00000 79 hdd 10.91399
osd.79 up 1.00000
1.00000 89 hdd 10.91399
osd.89 up 1.00000
1.00000 99 hdd 10.91399
osd.99 up 1.00000
1.00000 -55 87.31189
host node06-6f16ba4b-0082-472a-b243-b1a058070918
5 hdd 10.91399
osd.5 up 1.00000
1.00000 15 hdd 10.91399
osd.15 up 1.00000
1.00000 25 hdd 10.91399
osd.25 up 1.00000
1.00000 44 hdd 10.91399
osd.44 up 1.00000
1.00000 63 hdd 10.91399
osd.63 up 1.00000
1.00000 72 hdd 10.91399
osd.72 up 1.00000
1.00000 81 hdd 10.91399
osd.81 up 1.00000
1.00000 91 hdd 10.91399
osd.91 up 1.00000
1.00000 -43 87.31189
host node07-5dee846a-2814-4e04-bcfd-ff689d49795c
7 hdd 10.91399
osd.7 up 1.00000
1.00000 37 hdd 10.91399
osd.37 up 1.00000
1.00000 46 hdd 10.91399
osd.46 up 1.00000
1.00000 55 hdd 10.91399
osd.55 up 1.00000
1.00000 64 hdd 10.91399
osd.64 up 1.00000
1.00000 84 hdd 10.91399 osd.84
up 1.00000 1.00000 94 hdd 10.91399
osd.94 up 1.00000
1.00000 102 hdd 10.91399 osd.102
up 1.00000 1.00000 -58 87.31189
host node08-2d6b7ab3-2067-4e94-b77c-24d6e626e396
6 hdd 10.91399 osd.6
up 1.00000 1.00000 18 hdd 10.91399
osd.18 up 1.00000
1.00000 41 hdd 10.91399 osd.41
up 1.00000 1.00000 49 hdd 10.91399
osd.49 up 1.00000
1.00000 59 hdd 10.91399 osd.59
up 1.00000 1.00000
68 hdd 10.91399 osd.68
up 1.00000 1.00000 97 hdd 10.91399
osd.97 up 1.00000
1.00000 105 hdd 10.91399 osd.105
up 1.00000 1.00000 -49 87.31189
host node09-e6bf0642-e3b4-48f0-9d0e-3d87ceacced8
8 hdd 10.91399 osd.8
up 1.00000 1.00000 17 hdd 10.91399
osd.17 up 1.00000
1.00000 27 hdd 10.91399 osd.27
up 1.00000 1.00000 38 hdd 10.91399
osd.38 up 1.00000
1.00000 57 hdd 10.91399 osd.57
up 1.00000 1.00000 67 hdd 10.91399
osd.67 up 1.00000
1.00000 93 hdd 10.91399 osd.93
up 1.00000 1.00000 103 hdd 10.91399
osd.103 up 1.00000
1.00000 -8 47.16115 root
root-ea7a1878-722e-49d5-8a91-c618a6aefe29
-13 5.24013 host
node01-4c465825-6bd1-42a5-b087-51a82cb2865c
0 ssd 1.74657 osd.0
up 0.95001 1.00000 11 ssd 1.74699 osd.11
up 1.00000 1.00000 16
ssd 1.74657 osd.16
up 1.00000 1.00000 -22 5.24013 host
node02-0e3e418d-9129-44ee-8453-53680171270e
4 ssd 1.74657 osd.4
up 1.00000 1.00000 14 ssd 1.74657 osd.14
up 0.95001 1.00000 82
ssd 1.74699 osd.82
up 1.00000 1.00000 -28 5.24013 host
node03-274397b4-8559-4a17-a9eb-1c16920ad432
10 ssd 1.74657 osd.10
up 1.00000 1.00000 76 ssd 1.74657 osd.76
up 1.00000 1.00000 100
ssd 1.74699 osd.100
up 0.95001 1.00000 -31 5.24013 host
node04-df05c3cc-08ef-426e-85b8-088cb8c1b4e2
20 ssd 1.74657 osd.20
up 1.00000 1.00000 39 ssd 1.74657 osd.39
up 0.95001 1.00000 90
ssd 1.74699 osd.90
up 0.95001 1.00000 -25 5.24013 host
node05-d07e7e92-8290-49ba-b09c-ad34bddb1eae
26 ssd 1.74657 osd.26
up 0.95001 1.00000 62 ssd 1.74699 osd.62
up 1.00000 1.00000 73
ssd 1.74657 osd.73
up 1.00000 1.00000 -16 5.24013 host
node06-a7a40371-a128-4ab1-90a2-62c99e040036
42 ssd 1.74657 osd.42
up 0.95001 1.00000 75 ssd 1.74657 osd.75
up 0.95001 1.00000 107
ssd 1.74699 osd.107
up 1.00000 1.00000 -7 5.24013 host
node07-971dee6e-dec4-4c0a-86d1-54d0b23832bd
21 ssd 1.74699 osd.21
up 1.00000 1.00000 66 ssd 1.74657 osd.66
up 0.95001 1.00000 77
ssd 1.74657 osd.77
up 1.00000 1.00000 -19 5.24013 host
node08-9a47282b-9530-4cc7-9e29-7f6c0b2f5184
35 ssd 1.74699 osd.35
up 1.00000 1.00000 80 ssd 1.74657 osd.80
up 1.00000 1.00000 101
ssd 1.74657 osd.101
up 1.00000 1.00000 -34 5.24013 host
node09-f09a8013-2426-4ced-b9c7-02c06ca9d6fc
36 ssd 1.74699 osd.36
up 0.95001 1.00000 87 ssd 1.74657 osd.87
up 1.00000 1.00000 106
ssd 1.74657 osd.106
up 1.00000 1.00000
After ceph has been running normally for a period of time, I found that the
distribution of osd data is seriously uneven, and autoscale PGs were turned
on in my envs. My hdds are identical, but utilization ranges from 6.14% to
74.35%
root@node09:/# ceph osd df | grep hdd ID CLASS WEIGHT REWEIGHT
SIZE RAW USE DATA OMAP META AVAIL %USE VAR
PGS STATUS 1 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.2 TiB
15 KiB 53 GiB 5.7 TiB 48.15 1.38 174 up 22 hdd
10.91399 0.90002 11 TiB 8.1 TiB 8.0 TiB 12 KiB 71 GiB 2.8
TiB 74.35 2.13 160 up 31 hdd 10.91399 1.00000 11 TiB
3.0 TiB 2.9 TiB 13 KiB 27 GiB 8.0 TiB 27.14 0.78 179 up
51 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 9 KiB
36 GiB 7.2 TiB 34.12 0.98 173 up 60 hdd 10.91399
1.00000 11 TiB 973 GiB 962 GiB 14 KiB 11 GiB 10 TiB 8.71
0.25 167 up 70 hdd 10.91399 1.00000 11 TiB 2.0 TiB
2.0 TiB 11 KiB 3.9 GiB 8.9 TiB 18.42 0.53 173 up 78 hdd
10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 12 KiB 28 GiB 7.9
TiB 27.18 0.78 173 up 96 hdd 10.91399 1.00000 11 TiB
3.7 TiB 3.7 TiB 10 KiB 36 GiB 7.2 TiB 34.15 0.98 176 up
9 hdd 10.91399 1.00000 11 TiB 2.8 TiB 2.8 TiB 16 KiB
13 GiB 8.1 TiB 25.38 0.73 178 up 19 hdd 10.91399
1.00000 11 TiB 3.8 TiB 3.8 TiB 13 KiB 43 GiB 7.1 TiB 35.04
1.00 177 up 29 hdd 10.91399 1.00000 11 TiB 5.1 TiB
5.0 TiB 14 KiB 38 GiB 5.9 TiB 46.38 1.33 182 up 47 hdd
10.91399 1.00000 11 TiB 686 GiB 684 GiB 9 KiB 1.4 GiB 10
TiB 6.14 0.18 172 up 56 hdd 10.91399 1.00000 11 TiB
3.7 TiB 3.7 TiB 12 KiB 36 GiB 7.2 TiB 34.16 0.98 174 up
65 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 7 KiB
54 GiB 5.0 TiB 54.27 1.55 183 up 88 hdd 10.91399
1.00000 11 TiB 4.5 TiB 4.4 TiB 12 KiB 44 GiB 6.4 TiB 41.17
1.18 173 up 98 hdd 10.91399 1.00000 11 TiB 7.3 TiB
7.2 TiB 13 KiB 56 GiB 3.7 TiB 66.49 1.90 179 up 2 hdd
10.91399 1.00000 11 TiB 1.5 TiB 1.5 TiB 12 KiB 18 GiB 9.4
TiB 14.02 0.40 177 up 30 hdd 10.91399 1.00000 11 TiB
4.4 TiB 4.4 TiB 10 KiB 37 GiB 6.5 TiB 40.29 1.15 180 up
40 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 10 KiB
11 GiB 9.5 TiB 13.14 0.38 172 up 48 hdd 10.91399
1.00000 11 TiB 1.4 TiB 1.4 TiB 12 KiB 11 GiB 9.5 TiB 13.17
0.38 168 up 58 hdd 10.91399 1.00000 11 TiB 7.5 TiB
7.4 TiB 14 KiB 70 GiB 3.5 TiB 68.31 1.96 182 up 74 hdd
10.91399 1.00000 11 TiB 6.0 TiB 6.0 TiB 10 KiB 61 GiB 4.9
TiB 55.15 1.58 181 up 83 hdd 10.91399 1.00000 11 TiB
2.2 TiB 2.2 TiB 22 KiB 19 GiB 8.7 TiB 20.13 0.58 174 up
92 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 11 KiB
27 GiB 8.0 TiB 27.15 0.78 171 up 12 hdd 10.91399
1.00000 11 TiB 5.1 TiB 5.0 TiB 9 KiB 38 GiB 5.9 TiB 46.37
1.33 176 up 23 hdd 10.91399 1.00000 11 TiB 4.4 TiB
4.4 TiB 22 KiB 37 GiB 6.5 TiB 40.24 1.15 173 up 32 hdd
10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 10 KiB 19 GiB 8.7
TiB 20.13 0.58 176 up 43 hdd 10.91399 1.00000 11 TiB
2.0 TiB 2.0 TiB 10 KiB 4.4 GiB 8.9 TiB 18.41 0.53 176 up
52 hdd 10.91399 1.00000 11 TiB 3.6 TiB 3.6 TiB 12 KiB
29 GiB 7.3 TiB 33.27 0.95 171 up 71 hdd 10.91399
1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.13
0.78 170 up 95 hdd 10.91399 1.00000 11 TiB 5.4 TiB
5.3 TiB 10 KiB 60 GiB 5.6 TiB 49.04 1.40 173 up104 hdd
10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 8 KiB 36 GiB 7.2
TiB 34.13 0.98 171 up 3 hdd 10.91399 1.00000 11 TiB
2.2 TiB 2.2 TiB 9 KiB 19 GiB 8.7 TiB 20.14 0.58 173 up
24 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 9 KiB
19 GiB 8.7 TiB 20.12 0.58 170 up 33 hdd 10.91399
1.00000 11 TiB 6.8 TiB 6.7 TiB 14 KiB 69 GiB 4.1 TiB 62.15
1.78 181 up 45 hdd 10.91399 1.00000 11 TiB 2.9 TiB
2.8 TiB 10 KiB 20 GiB 8.0 TiB 26.27 0.75 178 up 69 hdd
10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 15 KiB 20 GiB 8.0
TiB 26.25 0.75 173 up 79 hdd 10.91399 1.00000 11 TiB
5.9 TiB 5.9 TiB 28 KiB 54 GiB 5.0 TiB 54.25 1.55 178 up
89 hdd 10.91399 1.00000 11 TiB 2.1 TiB 2.1 TiB 16 KiB
11 GiB 8.8 TiB 19.25 0.55 173 up 99 hdd 10.91399
1.00000 11 TiB 4.5 TiB 4.4 TiB 13 KiB 44 GiB 6.4 TiB 41.14
1.18 174 up 5 hdd 10.91399 1.00000 11 TiB 2.2 TiB
2.2 TiB 14 KiB 19 GiB 8.7 TiB 20.12 0.58 172 up 15 hdd
10.91399 1.00000 11 TiB 7.3 TiB 7.2 TiB 7 KiB 56 GiB 3.7
TiB 66.48 1.90 177 up 25 hdd 10.91399 1.00000 11 TiB
1.4 TiB 1.4 TiB 11 KiB 10 GiB 9.5 TiB 13.13 0.38 176 up
44 hdd 10.91399 1.00000 11 TiB 3.1 TiB 3.0 TiB 9 KiB
34 GiB 7.9 TiB 28.03 0.80 179 up 63 hdd 10.91399
1.00000 11 TiB 2.9 TiB 2.8 TiB 10 KiB 20 GiB 8.0 TiB 26.26
0.75 178 up 72 hdd 10.91399 1.00000 11 TiB 3.7 TiB
3.7 TiB 12 KiB 36 GiB 7.2 TiB 34.13 0.98 180 up 81 hdd
10.91399 1.00000 11 TiB 6.7 TiB 6.6 TiB 11 KiB 62 GiB 4.2
TiB 61.30 1.76 181 up 91 hdd 10.91399 1.00000 11 TiB
2.2 TiB 2.2 TiB 16 KiB 19 GiB 8.7 TiB 20.11 0.58 176 up
7 hdd 10.91399 1.00000 11 TiB 4.6 TiB 4.5 TiB 13 KiB
52 GiB 6.3 TiB 42.03 1.20 180 up 37 hdd 10.91399
1.00000 11 TiB 5.3 TiB 5.2 TiB 14 KiB 52 GiB 5.7 TiB 48.16
1.38 182 up 46 hdd 10.91399 1.00000 11 TiB 784 GiB
775 GiB 14 KiB 8.8 GiB 10 TiB 7.01 0.20 173 up 55 hdd
10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 20 KiB 54 GiB 5.0
TiB 54.25 1.55 177 up 64 hdd 10.91399 1.00000 11 TiB
3.6 TiB 3.6 TiB 18 KiB 28 GiB 7.3 TiB 33.23 0.95 176 up
84 hdd 10.91399 1.00000 11 TiB 784 GiB 775 GiB 10 KiB
9.0 GiB 10 TiB 7.01 0.20 176 up 94 hdd 10.91399
1.00000 11 TiB 2.1 TiB 2.1 TiB 8 KiB 12 GiB 8.8 TiB 19.28
0.55 169 up102 hdd 10.91399 1.00000 11 TiB 3.0 TiB
2.9 TiB 10 KiB 28 GiB 8.0 TiB 27.13 0.78 170 up 6 hdd
10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 23 KiB 44 GiB 6.4
TiB 41.14 1.18 181 up 18 hdd 10.91399 1.00000 11 TiB
6.0 TiB 6.0 TiB 9 KiB 61 GiB 4.9 TiB 55.15 1.58 183 up
41 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB
27 GiB 8.0 TiB 27.12 0.78 179 up 49 hdd 10.91399
1.00000 11 TiB 785 GiB 775 GiB 19 KiB 9.5 GiB 10 TiB 7.02
0.20 176 up 59 hdd 10.91399 1.00000 11 TiB 2.2 TiB
2.2 TiB 8 KiB 19 GiB 8.7 TiB 20.12 0.58 178 up 68 hdd
10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 13 KiB 36 GiB 7.2
TiB 34.15 0.98 172 up 97 hdd 10.91399 1.00000 11 TiB
3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.13 0.78 178
up105 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 13 KiB
36 GiB 7.2 TiB 34.14 0.98 179 up 8 hdd 10.91399
1.00000 11 TiB 6.6 TiB 6.5 TiB 13 KiB 55 GiB 4.3 TiB 60.40
1.73 181 up 17 hdd 10.91399 1.00000 11 TiB 4.6 TiB
4.5 TiB 14 KiB 52 GiB 6.3 TiB 42.03 1.20 177 up 27 hdd
10.91399 1.00000 11 TiB 783 GiB 774 GiB 8 KiB 9.4 GiB 10
TiB 7.01 0.20 176 up 38 hdd 10.91399 1.00000 11 TiB
2.2 TiB 2.2 TiB 15 KiB 19 GiB 8.7 TiB 20.11 0.58 174 up
57 hdd 10.91399 1.00000 11 TiB 2.3 TiB 2.3 TiB 12 KiB
26 GiB 8.6 TiB 21.01 0.60 177 up 67 hdd 10.91399
1.00000 11 TiB 4.5 TiB 4.4 TiB 9 KiB 44 GiB 6.4 TiB 41.14
1.18 176 up 93 hdd 10.91399 1.00000 11 TiB 3.1 TiB
3.0 TiB 7 KiB 35 GiB 7.9 TiB 28.01 0.80 171 up103 hdd
10.91399 1.00000 11 TiB 4.2 TiB 4.2 TiB 10 KiB 22 GiB 6.7
TiB 38.52 1.10 176 up
three pools (deeproute-replica-hdd-pool,
os-dsglczutvqsgowpz.rgw.buckets.data, and cephfs-replicated-pool ) are set
to use the same device class hdd. I noticed that the *effective ratio* of
the three pools is very different, not sure if it will affect the data
rebalancing.
[root@node01 smd]# ceph osd pool autoscale-statusPOOL
SIZE TARGET SIZE RATE RAW
CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW
PG_NUM AUTOSCALE BULK device_health_metrics
278.5M 3.0 48289G 0.0000
1.0 1 on False
deeproute-replica-hdd-pool 12231M
3.0 785.8T 0.9901 50.0000 0.9901 1.0
4096 on False deeproute-replica-ssd-pool
12333G 3.0 48289G 0.9901
50.0000 0.9901 1.0 1024 on False
.rgw.root 5831
3.0 48289G 0.0099 0.5000 0.0099 1.0
8 on False default.rgw.log
182 3.0 48289G 0.0000
1.0 32 on
False default.rgw.control 0
3.0 48289G 0.0000
1.0 32 on False default.rgw.meta
0 3.0 48289G
0.0000 4.0 8 on
False os-dsglczutvqsgowpz.rgw.control 0
3.0 16096G 0.1667 0.5000
0.1667 1.0 64 on False
os-dsglczutvqsgowpz.rgw.meta 99.81k
3.0 16096G 0.1667 0.5000 0.1667 1.0
64 on False
os-dsglczutvqsgowpz.rgw.buckets.index 59437M
3.0 16096G 0.1667 0.5000 0.1667 1.0
32 on False
os-dsglczutvqsgowpz.rgw.buckets.non-ec 498.7M
3.0 16096G 0.1667 0.5000 0.1667 1.0
32 on False os-dsglczutvqsgowpz.rgw.log
1331M 3.0 16096G 0.1667
0.5000 0.1667 1.0 32 on False
os-dsglczutvqsgowpz.rgw.buckets.data 143.1T
1.3333333730697632 785.8T 0.2429 0.5000
0.0099 1.0 32 on False cephfs-metadata
3161M 3.0
16096G 0.0006 4.0 32
on False cephfs-replicated-pool 21862G
3.0 785.8T 0.0815
1.0 32 on False .nfs
84599 3.0
48289G 0.0000 1.0 32
on False os-dsglczutvqsgowpz.rgw.otp 0
3.0 16096G 0.1667 0.5000
[root@node01 smd]# ceph df--- RAW STORAGE ---CLASS SIZE AVAIL
USED RAW USED %RAW USEDhdd 786 TiB 545 TiB 240 TiB 240 TiB
30.60ssd 63 TiB 25 TiB 38 TiB 38 TiB 59.95TOTAL
849 TiB 571 TiB 278 TiB 278 TiB 32.77 --- POOLS ---POOL
ID PGS STORED OBJECTS USED
%USED MAX AVAILdevice_health_metrics 1 1 280
MiB 135 841 MiB 0.02 1.8 TiBdeeproute-replica-hdd-pool
11 4096 12 GiB 9.44k 36 GiB 0.02 69
TiBdeeproute-replica-ssd-pool 12 1024 12 TiB 3.43M
37 TiB 87.62 1.8 TiB.rgw.root 25
8 5.7 KiB 20 228 KiB 0 1.8 TiBdefault.rgw.log
26 32 182 B 2 24 KiB 0 1.8
TiBdefault.rgw.control 27 32 0 B 8
0 B 0 1.8 TiBdefault.rgw.meta 28
8 0 B 0 0 B 0 1.8
TiBos-dsglczutvqsgowpz.rgw.control 29 64 0 B 8
0 B 0 4.9 TiBos-dsglczutvqsgowpz.rgw.meta 30
64 100 KiB 456 4.3 MiB 0 4.9
TiBos-dsglczutvqsgowpz.rgw.buckets.index 31 32 58 GiB 10.40k
174 GiB 1.15 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.non-ec 32
32 500 MiB 247.78k 4.3 GiB 0.03 4.9
TiBos-dsglczutvqsgowpz.rgw.log 33 32 689 MiB 432
2.0 GiB 0.01 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.data 34
32 130 TiB 134.96M 174 TiB 45.69 155 TiBcephfs-metadata
35 32 3.1 GiB 1.67M 9.3 GiB 0.06 4.9
TiBcephfs-replicated-pool 36 32 21 TiB 8.25M
64 TiB 23.79 69 TiB.nfs 37
32 89 KiB 9 341 KiB 0 1.8
TiBos-dsglczutvqsgowpz.rgw.otp 38 8 0 B 0
0 B 0 4.9 TiB
[root@node01 smd]# ceph -s
cluster:
id: f7f1c8ba-f793-436b-bb73-0964108a30c1
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 7w)
mgr: a(active, since 2w), standbys: b
mds: 1/1 daemons up, 1 hot standby
osd: 108 osds: 108 up (since 5h), 108 in (since 4M); 1 remapped pgs
rgw: 9 daemons active (9 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 17 pools, 5561 pgs
objects: 148.37M objects, 162 TiB
usage: 276 TiB used, 572 TiB / 849 TiB avail
pgs: 2786556/1119434807 objects misplaced (0.249%)
5559 active+clean
1 active+remapped+backfilling
1 active+clean+scrubbing+deep
io:
client: 212 MiB/s rd, 27 MiB/s wr, 4.04k op/s rd, 2.01k op/s wr
recovery: 4.0 MiB/s, 4 objects/s
progress:
Global Recovery Event (2d)
[===========================.] (remaining: 39m)
So my question is, how can I adjust to make the osd data more evenly
distributed? Thanks!
Best Regards,
Liang Zheng