18 Nov
2019
18 Nov
'19
5:32 p.m.
Hi,
in this <https://ceph.io/community/the-first-telemetry-results-are-in/>
blog post I find this statement:
"So, in our ideal world so far (assuming equal size OSDs), every OSD now
has the same number of PGs assigned."
My issue is that accross all pools the number of PGs per OSD is not equal.
And I conclude that this is causing very unbalanced data placement.
As a matter of fact the data stored on my 1.6TB HDD in specific pool
"hdb_backup" is in a range starting with
osd.228 size: 1.6 usage: 52.61 reweight: 1.00000
and ending with
osd.145 size: 1.6 usage: 81.11 reweight: 1.00000
This impacts the amount of data that can be stored in the cluster heavily.
Ceph balancer is enabled, but this is not solving this issue.
root@ld3955:~# ceph balancer status
{
"active": true,
"plans": [],
"mode": "upmap"
}
Therefore I would ask you for suggestions how to work on this unbalanced
data distribution.
I have attached pastebin for
- ceph osd df sorted by usage <https://pastebin.com/QLQHjA9g>
- ceph osd df tree <https://pastebin.com/SvhP2hp5>
My cluster has multiple crush roots respresenting different disks.
In addition I have defined multiple pools, one pool for each disk type:
hdd, ssd, nvme.
THX
18 Nov
18 Nov
10:48 p.m.
You have way too few PGs in one of the roots. Many OSDs have so few
PGs that you should see a lot of health warnings because of it.
The other root has a factor 5 difference in disk size which isn't ideal either.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Mon, Nov 18, 2019 at 3:03 PM Thomas Schneider <74cmonty(a)gmail.com> wrote:
>
> Hi,
>
> in this <https://ceph.io/community/the-first-telemetry-results-are-in/>
> blog post I find this statement:
> "So, in our ideal world so far (assuming equal size OSDs), every OSD now
> has the same number of PGs assigned."
>
> My issue is that accross all pools the number of PGs per OSD is not equal.
> And I conclude that this is causing very unbalanced data placement.
> As a matter of fact the data stored on my 1.6TB HDD in specific pool
> "hdb_backup" is in a range starting with
> osd.228 size: 1.6 usage: 52.61 reweight: 1.00000
> and ending with
> osd.145 size: 1.6 usage: 81.11 reweight: 1.00000
>
> This impacts the amount of data that can be stored in the cluster heavily.
>
> Ceph balancer is enabled, but this is not solving this issue.
> root@ld3955:~# ceph balancer status
> {
> "active": true,
> "plans": [],
> "mode": "upmap"
> }
>
> Therefore I would ask you for suggestions how to work on this unbalanced
> data distribution.
>
> I have attached pastebin for
> - ceph osd df sorted by usage <https://pastebin.com/QLQHjA9g>
> - ceph osd df tree <https://pastebin.com/SvhP2hp5>
>
> My cluster has multiple crush roots respresenting different disks.
> In addition I have defined multiple pools, one pool for each disk type:
> hdd, ssd, nvme.
>
> THX
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
19 Nov
19 Nov
12:31 p.m.
Hello Paul,
thanks for your analysis.
I want to share more statistics of my cluster to follow-up on your
response "You have way too few PGs in one of the roots".
Here are the pool details:
root@ld3955:~# ceph osd pool ls detail
pool 11 'hdb_backup' replicated size 3 min_size 2 crush_rule 1
object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode warn
last_change 294572 flags hashpspool,selfmanaged_snaps stripe_width 0
application rbd
removed_snaps [1~3]
pool 59 'hdd' replicated size 2 min_size 2 crush_rule 3 object_hash
rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 267271
flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps [1~3]
pool 60 'ssd' replicated size 2 min_size 2 crush_rule 4 object_hash
rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 299719
lfor 299717/299717/299717 flags hashpspool,selfmanaged_snaps
stripe_width 0 application rbd
removed_snaps [1~3]
pool 61 'nvme' replicated size 2 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 267125 flags
hashpspool stripe_width 0 application rbd
pool 62 'cephfs_data' replicated size 3 min_size 2 crush_rule 3
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn
last_change 300312 lfor 300310/300310/300310 flags hashpspool
stripe_width 0 application cephfs
pool 63 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 3
object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change
267069 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16
recovery_priority 5 application cephfs
Any pg_num / pgp_num is monitored by Ceph, means I get a warning in the
log / health status if a pool is undersized.
I didn't enable PG auto-scaler for any pool, though.
The calculation of PGs per Pool is done with pgcalc
<https://ceph.io/pgcalc/>.
Here's a screenshot <https://ibb.co/VjR6X3x> of this calculation.
My focus is on pool hdb_backup.
Based on these statistics
root@ld3955:~# ceph df detail
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 1.4 PiB 744 TiB 729 TiB 730 TiB 49.53
nvme 23 TiB 23 TiB 43 GiB 51 GiB 0.22
ssd 27 TiB 25 TiB 1.9 TiB 1.9 TiB 7.15
TOTAL 1.5 PiB 792 TiB 731 TiB 732 TiB 48.02
POOLS:
POOL ID STORED OBJECTS USED
%USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY
USED COMPR UNDER COMPR
hdb_backup 11 241 TiB 63.29M 241 TiB
57.03 61 TiB N/A N/A 63.29M
0 B 0 B
hdd 59 553 GiB 142.16k 553 GiB
0.50 54 TiB N/A N/A 142.16k
0 B 0 B
ssd 60 2.0 TiB 530.75k 2.0 TiB
8.72 10 TiB N/A N/A 530.75k
0 B 0 B
nvme 61 0 B 0 0 B
0 11 TiB N/A N/A 0
0 B 0 B
cephfs_data 62 356 GiB 102.29k 356 GiB
0.32 36 TiB N/A N/A 102.29k
0 B 0 B
cephfs_metadata 63 117 MiB 52 117 MiB
0 36 TiB N/A N/A 52
0 B 0 B
there's only 57% used, but effectively I cannot store much more data
because some OSDs are filling up by +80%.
It is true that the disks that are used for this pool exclusively are
different in size, means
3x 48 disks à 7.2TB
4x 48 disks à 1.6TB
and the disk usage is for 7.2TB disk from 41% to 54% and for 1.6TB disks
from 52% to 81%.
If Ceph is not cabable to manage rebalancing automatically, how can I
proceed to rebalance the data manually?
OSD reweight is not an option in my opinion because it starts filling
OSDs that are not with lowest usage rate.
Can I move PGs to specific OSDs?
THX
Am 18.11.2019 um 20:18 schrieb Paul Emmerich:
You have way too few PGs in one of the roots. Many
OSDs have so few
PGs that you should see a lot of health warnings because of it.
The other root has a factor 5 difference in disk size which isn't ideal either.
Paul
12:52 p.m.
On 11/19/19 4:01 PM, Thomas Schneider wrote:
If Ceph is not cabable to manage rebalancing
automatically, how can I
proceed to rebalance the data manually?
Use offline upmap for your target pool:
ceph osd getmap -o om; osdmaptool om --upmap upmap.sh
--upmap-pool=hdd_backup --upmap-deviation 0; bash upmap.sh; rm -f
upmap.sh om
k
2 Dec
2 Dec
10:38 a.m.
Hi,
the folline upmap reports "no upmaps proposed".
root@ld3955:/home# osdmaptool om --upmap hdd-upmap.sh
--upmap-pool=hdb_backup --upmap-deviation 0
osdmaptool: osdmap file 'om'
writing upmap command output to: hdd-upmap.sh
checking for upmap cleanups
upmap, max-count 100, max deviation 0
limiting to pools hdb_backup (11)
nno upmaps proposed
root@ld3955:/home# ls -ltr | tail
-rw-r--r-- 1 root root 60758 Nov 5 08:27 osd_tree
-rw-r--r-- 1 root root 21686 Nov 15 14:12
compiled-crushmap-15-11-2019_14-12
-rw-r--r-- 1 root root 29996 Nov 15 14:13
decompiled-crushmap-15-11-2019_14-12
-rw-r--r-- 1 root root 30426 Nov 15 14:55
new-decompiled-crushmap-15-11-2019_14-12
-rw-r--r-- 1 root root 20381 Nov 15 14:55
new-compiled-crushmap-15-11-2019_14-12
-rw-r--r-- 1 root root 12507 Nov 18 14:57 disk-usage
-rw-r--r-- 1 root root 60626 Nov 18 14:58 osd-tree
-rw-r--r-- 1 root root 7380 Nov 22 16:46 ceph.tar.gz
-rw-r--r-- 1 root root 166421 Nov 30 22:53 om
-rw-r--r-- 1 root root 0 Nov 30 23:11 hdd-upmap.sh
In my opinion this shows that the evalutation step is not working
because my OSDs are not balanced.
Therefore I would conclude to open a bug report for this issue.
Thomas
Am 19.11.2019 um 10:22 schrieb Konstantin Shalygin:
On 11/19/19 4:01 PM, Thomas Schneider wrote:
If Ceph is not cabable to manage rebalancing
automatically, how can I
proceed to rebalance the data manually?
Use offline upmap for your target pool:
ceph osd getmap -o om; osdmaptool om --upmap upmap.sh
--upmap-pool=hdd_backup --upmap-deviation 0; bash upmap.sh; rm -f
upmap.sh om
k
2:25 p.m.
Hi there!
Here we have a similar situation.
After adding some OSDs to the cluster the PGs are not equally distributed over the OSDs.
The balancing mode is set to upmap.
The docs https://docs.ceph.com/docs/master/rados/operations/balancer/#modes say:
"This CRUSH mode will optimize the placement of individual PGs in order to achieve a
balanced distribution. In most cases, this distribution is “perfect,” which an equal
number of PGs on each OSD (+/-1 PG, since they might not divide evenly)."
This is not the case here with our cluster. The number of PGs reaches from 157 up to 214.
And so the resulting usage of the HDDs varies from 60% to 82%.
$ ceph osd df class hdd
[…]
MIN/MAX VAR: 0.89/1.22 STDDEV: 7.47
In the meantime I had to use reweight-by-utilization to get the cluster in healthy state
again, because it had a near full PG and near full OSDs.
The cluster is nautilus 14.2.4 on debian 10 with packages from croit.io.
I think about switching back to crush-compat.
Is there a bug known regarding this?
Thanks
Lars
3 Dec
3 Dec
6:31 a.m.
On 12/2/19 5:55 PM, Lars Täuber wrote:
Here we have a similar situation.
After adding some OSDs to the cluster the PGs are not equally distributed over the OSDs.
The balancing mode is set to upmap.
The docshttps://docs.ceph.com/docs/master/rados/operations/balancer/#modes say:
"This CRUSH mode will optimize the placement of individual PGs in order to achieve a
balanced distribution. In most cases, this distribution is “perfect,” which an equal
number of PGs on each OSD (+/-1 PG, since they might not divide evenly)."
This is not the case here with our cluster. The number of PGs reaches from 157 up to 214.
And so the resulting usage of the HDDs varies from 60% to 82%.
$ ceph osd df class hdd
[…]
MIN/MAX VAR: 0.89/1.22 STDDEV: 7.47
In the meantime I had to use reweight-by-utilization to get the cluster in healthy state
again, because it had a near full PG and near full OSDs.
The cluster is nautilus 14.2.4 on debian 10 with packages from croit.io.
I think about switching back to crush-compat.
Is there a bug known regarding this?
Please paste your `ceph osd df tree`, `ceph osd pool ls detail`, `ceph
osd crush rule dump`.
k
10 a.m.
Hi Konstantin,
Tue, 3 Dec 2019 10:01:34 +0700
Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber <taeuber(a)bbaw.de>de>,
ceph-users(a)ceph.io :
Please paste your `ceph osd df tree`, `ceph osd pool
ls detail`, `ceph
osd crush rule dump`.
here it comes:
$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR
PGS STATUS TYPE NAME
-1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- root default
-25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- room PRZ
-26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- row rechts
-27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB 67.51 1.02
- rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB 62.80 0.95
- host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.94
163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.94
163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.94
163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.95
164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.01
176 up osd.37
4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB 3.01 0.05
110 up osd.4
-13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB 62.64 0.94
- host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.94
163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.94
163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.94
163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.94
162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.01
176 up osd.40
33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB 3.02 0.05
110 up osd.33
-22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB 77.10 1.16
- host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.17
203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.18
205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.14
198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.15
199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.24
215 up osd.30
36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB 2.92 0.04
110 up osd.36
-28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB 62.36 0.94
- rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB 62.27 0.94
- host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.93
162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.93
162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.93
161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.93
161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.01
176 up osd.38
31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB 3.04 0.05
110 up osd.31
-16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB 62.44 0.94
- host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.94
162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.94
162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.94
162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.93
162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.01
176 up osd.41
34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB 3.04 0.05
109 up osd.34
-29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB 68.95 1.04
- rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB 61.02 0.92
- host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.91
158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.91
157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.91
157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.91
157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.01
176 up osd.39
32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB 2.88 0.04
109 up osd.32
-19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB 76.89 1.16
- host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.10
190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.23
214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.21
210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.16
202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.16
201 up osd.29
35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB 2.89 0.04
110 up osd.35
TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45
MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR
PGS STATUS TYPE NAME
-1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- root default
-25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- room PRZ
-26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- row rechts
-27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB 68.31 1.02
- rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB 63.54 0.94
- host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.93
163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.93
163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.93
163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.94
164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.00
176 up osd.37
-13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB 63.38 0.94
- host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.93
163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.93
163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.93
163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.92
162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.00
176 up osd.40
-22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB 78.02 1.16
- host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.15
203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.17
205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.13
198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.13
199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.22
215 up osd.30
-28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB 63.09 0.94
- rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB 63.01 0.94
- host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.92
162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.92
162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.92
161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.92
161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.00
176 up osd.38
-16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB 63.18 0.94
- host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.92
162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.92
162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.92
162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.92
162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.00
176 up osd.41
-29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB 69.77 1.04
- rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB 61.74 0.92
- host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.90
158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.90
157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.90
157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.90
157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.00
176 up osd.39
-19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB 77.81 1.16
- host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.08
190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.22
214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.19
210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.15
202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.14
201 up osd.29
TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24
MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail
pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash rjenkins
pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags
hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384
target_size_ratio 0.85 application cephfs
removed_snaps
[2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1]
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259 flags
hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5
target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "cephfs_data",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 6,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -2,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "rep_3_ssd",
"ruleset": 2,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -6,
"item_name": "default~ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
Tanks,
Lars
10:32 a.m.
BTW: The osdmaptool doesn't see anything to do either:
$ ceph osd getmap -o om
$ osdmaptool om --upmap /tmp/upmap.sh --upmap-pool cephfs_data
osdmaptool: osdmap file 'om'
writing upmap command output to: /tmp/upmap.sh
checking for upmap cleanups
upmap, max-count 100, max deviation 0.01
limiting to pools cephfs_data (1)
no upmaps proposed
Tue, 3 Dec 2019 07:30:24 +0100
Lars Täuber <taeuber(a)bbaw.de> ==> Konstantin Shalygin <k0ste(a)k0ste.ru> :
Hi Konstantin,
Tue, 3 Dec 2019 10:01:34 +0700
Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber <taeuber(a)bbaw.de>de>,
ceph-users(a)ceph.io :
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
Please paste your `ceph osd df tree`, `ceph osd
pool ls detail`, `ceph
osd crush rule dump`.
here it comes:
$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR
PGS STATUS TYPE NAME
-1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- root default
-25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- room PRZ
-26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- row rechts
-27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB 67.51 1.02
- rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB 62.80 0.95
- host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.94
163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.94
163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.94
163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.95
164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.01
176 up osd.37
4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB 3.01 0.05
110 up osd.4
-13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB 62.64 0.94
- host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.94
163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.94
163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.94
163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.94
162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.01
176 up osd.40
33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB 3.02 0.05
110 up osd.33
-22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB 77.10 1.16
- host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.17
203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.18
205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.14
198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.15
199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.24
215 up osd.30
36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB 2.92 0.04
110 up osd.36
-28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB 62.36 0.94
- rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB 62.27 0.94
- host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.93
162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.93
162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.93
161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.93
161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.01
176 up osd.38
31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB 3.04 0.05
110 up osd.31
-16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB 62.44 0.94
- host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.94
162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.94
162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.94
162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.93
162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.01
176 up osd.41
34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB 3.04 0.05
109 up osd.34
-29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB 68.95 1.04
- rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB 61.02 0.92
- host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.91
158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.91
157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.91
157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.91
157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.01
176 up osd.39
32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB 2.88 0.04
109 up osd.32
-19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB 76.89 1.16
- host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.10
190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.23
214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.21
210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.16
202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.16
201 up osd.29
35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB 2.89 0.04
110 up osd.35
TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45
MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR
PGS STATUS TYPE NAME
-1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- root default
-25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- room PRZ
-26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- row rechts
-27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB 68.31 1.02
- rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB 63.54 0.94
- host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.93
163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.93
163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.93
163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.94
164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.00
176 up osd.37
-13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB 63.38 0.94
- host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.93
163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.93
163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.93
163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.92
162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.00
176 up osd.40
-22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB 78.02 1.16
- host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.15
203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.17
205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.13
198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.13
199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.22
215 up osd.30
-28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB 63.09 0.94
- rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB 63.01 0.94
- host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.92
162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.92
162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.92
161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.92
161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.00
176 up osd.38
-16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB 63.18 0.94
- host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.92
162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.92
162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.92
162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.92
162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.00
176 up osd.41
-29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB 69.77 1.04
- rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB 61.74 0.92
- host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.90
158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.90
157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.90
157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.90
157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.00
176 up osd.39
-19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB 77.81 1.16
- host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.08
190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.22
214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.19
210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.15
202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.14
201 up osd.29
TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24
MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail
pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash rjenkins
pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags
hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384
target_size_ratio 0.85 application cephfs
removed_snaps
[2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1]
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259 flags
hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5
target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "cephfs_data",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 6,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -2,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "rep_3_ssd",
"ruleset": 2,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -6,
"item_name": "default~ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
Tanks,
Lars
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
10:56 a.m.
Hi all
Something to try:
ceph config set mgr mgr/balancer/upmap_max_iterations 20
(Default is 100.)
Cheers
Harry
On 03.12.19 08:02, Lars Täuber wrote:
> BTW: The osdmaptool doesn't see anything to do either:
>
> $ ceph osd getmap -o om
> $ osdmaptool om --upmap /tmp/upmap.sh --upmap-pool cephfs_data
> osdmaptool: osdmap file 'om'
> writing upmap command output to: /tmp/upmap.sh
> checking for upmap cleanups
> upmap, max-count 100, max deviation 0.01
> limiting to pools cephfs_data (1)
> no upmaps proposed
>
>
>
>
> Tue, 3 Dec 2019 07:30:24 +0100
> Lars Täuber <taeuber(a)bbaw.de> ==> Konstantin Shalygin <k0ste(a)k0ste.ru>
:
>> Hi Konstantin,
>>
>>
>> Tue, 3 Dec 2019 10:01:34 +0700
>> Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber
<taeuber(a)bbaw.de>de>, ceph-users(a)ceph.io :
>>> Please paste your `ceph osd df tree`, `ceph osd pool ls detail`, `ceph
>>> osd crush rule dump`.
>>
>> here it comes:
>>
>> $ ceph osd df tree
>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS TYPE NAME
>> -1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB
66.45 1.00 - root default
>> -25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB
66.45 1.00 - room PRZ
>> -26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB
66.45 1.00 - row rechts
>> -27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB
67.51 1.02 - rack 1-eins
>> -3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB
62.80 0.95 - host onode1
>> 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB
62.48 0.94 163 up osd.0
>> 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB
62.47 0.94 163 up osd.1
>> 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB
62.53 0.94 163 up osd.2
>> 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB
62.90 0.95 164 up osd.3
>> 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB
67.32 1.01 176 up osd.37
>> 4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB
3.01 0.05 110 up osd.4
>> -13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB
62.64 0.94 - host onode4
>> 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB
62.49 0.94 163 up osd.13
>> 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB
62.49 0.94 163 up osd.14
>> 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB
62.43 0.94 163 up osd.15
>> 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB
62.13 0.94 162 up osd.16
>> 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB
67.36 1.01 176 up osd.40
>> 33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB
3.02 0.05 110 up osd.33
>> -22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB
77.10 1.16 - host onode7
>> 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB
77.59 1.17 203 up osd.25
>> 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB
78.40 1.18 205 up osd.26
>> 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB
75.80 1.14 198 up osd.27
>> 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB
76.13 1.15 199 up osd.28
>> 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB
82.18 1.24 215 up osd.30
>> 36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB
2.92 0.04 110 up osd.36
>> -28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB
62.36 0.94 - rack 2-zwei
>> -7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB
62.27 0.94 - host onode2
>> 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB
62.08 0.93 162 up osd.5
>> 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB
62.13 0.93 162 up osd.6
>> 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB
61.77 0.93 161 up osd.7
>> 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB
61.75 0.93 161 up osd.8
>> 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB
67.31 1.01 176 up osd.38
>> 31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB
3.04 0.05 110 up osd.31
>> -16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB
62.44 0.94 - host onode5
>> 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB
62.15 0.94 162 up osd.17
>> 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB
62.16 0.94 162 up osd.18
>> 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB
62.14 0.94 162 up osd.19
>> 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB
62.12 0.93 162 up osd.20
>> 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB
67.31 1.01 176 up osd.41
>> 34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB
3.04 0.05 109 up osd.34
>> -29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB
68.95 1.04 - rack 3-drei
>> -10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB
61.02 0.92 - host onode3
>> 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB
60.63 0.91 158 up osd.9
>> 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB
60.19 0.91 157 up osd.10
>> 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB
60.27 0.91 157 up osd.11
>> 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB
60.28 0.91 157 up osd.12
>> 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB
67.34 1.01 176 up osd.39
>> 32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB
2.88 0.04 109 up osd.32
>> -19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB
76.89 1.16 - host onode6
>> 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB
72.79 1.10 190 up osd.21
>> 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB
81.79 1.23 214 up osd.22
>> 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB
80.29 1.21 210 up osd.23
>> 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB
77.31 1.16 202 up osd.24
>> 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB
76.86 1.16 201 up osd.29
>> 35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB
2.89 0.04 110 up osd.35
>> TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB
66.45
>> MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
>>
>>
>> better only for the class hdd
>>
>> $ ceph osd df tree class hdd
>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS TYPE NAME
>> -1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB
67.24 1.00 - root default
>> -25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB
67.24 1.00 - room PRZ
>> -26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB
67.24 1.00 - row rechts
>> -27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB
68.31 1.02 - rack 1-eins
>> -3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB
63.54 0.94 - host onode1
>> 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB
62.48 0.93 163 up osd.0
>> 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB
62.47 0.93 163 up osd.1
>> 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB
62.53 0.93 163 up osd.2
>> 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB
62.90 0.94 164 up osd.3
>> 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB
67.32 1.00 176 up osd.37
>> -13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB
63.38 0.94 - host onode4
>> 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB
62.49 0.93 163 up osd.13
>> 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB
62.49 0.93 163 up osd.14
>> 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB
62.43 0.93 163 up osd.15
>> 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB
62.13 0.92 162 up osd.16
>> 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB
67.36 1.00 176 up osd.40
>> -22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB
78.02 1.16 - host onode7
>> 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB
77.59 1.15 203 up osd.25
>> 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB
78.40 1.17 205 up osd.26
>> 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB
75.80 1.13 198 up osd.27
>> 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB
76.13 1.13 199 up osd.28
>> 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB
82.18 1.22 215 up osd.30
>> -28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB
63.09 0.94 - rack 2-zwei
>> -7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB
63.01 0.94 - host onode2
>> 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB
62.08 0.92 162 up osd.5
>> 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB
62.13 0.92 162 up osd.6
>> 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB
61.77 0.92 161 up osd.7
>> 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB
61.75 0.92 161 up osd.8
>> 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB
67.31 1.00 176 up osd.38
>> -16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB
63.18 0.94 - host onode5
>> 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB
62.15 0.92 162 up osd.17
>> 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB
62.16 0.92 162 up osd.18
>> 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB
62.14 0.92 162 up osd.19
>> 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB
62.12 0.92 162 up osd.20
>> 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB
67.31 1.00 176 up osd.41
>> -29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB
69.77 1.04 - rack 3-drei
>> -10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB
61.74 0.92 - host onode3
>> 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB
60.63 0.90 158 up osd.9
>> 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB
60.19 0.90 157 up osd.10
>> 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB
60.27 0.90 157 up osd.11
>> 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB
60.28 0.90 157 up osd.12
>> 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB
67.34 1.00 176 up osd.39
>> -19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB
77.81 1.16 - host onode6
>> 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB
72.79 1.08 190 up osd.21
>> 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB
81.79 1.22 214 up osd.22
>> 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB
80.29 1.19 210 up osd.23
>> 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB
77.31 1.15 202 up osd.24
>> 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB
76.86 1.14 201 up osd.29
>> TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB
67.24
>> MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
>>
>>
>>
>>
>> ceph osd pool ls detail
>> pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash
rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags
hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384
target_size_ratio 0.85 application cephfs
>> removed_snaps
[2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1]
>> pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2
object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259
flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5
target_size_ratio 0.3 application cephfs
>>
>>
>> $ ceph osd crush rule dump
>> [
>> {
>> "rule_id": 0,
>> "rule_name": "replicated_rule",
>> "ruleset": 0,
>> "type": 1,
>> "min_size": 1,
>> "max_size": 10,
>> "steps": [
>> {
>> "op": "take",
>> "item": -1,
>> "item_name": "default"
>> },
>> {
>> "op": "chooseleaf_firstn",
>> "num": 0,
>> "type": "host"
>> },
>> {
>> "op": "emit"
>> }
>> ]
>> },
>> {
>> "rule_id": 1,
>> "rule_name": "cephfs_data",
>> "ruleset": 1,
>> "type": 3,
>> "min_size": 3,
>> "max_size": 6,
>> "steps": [
>> {
>> "op": "set_chooseleaf_tries",
>> "num": 5
>> },
>> {
>> "op": "set_choose_tries",
>> "num": 100
>> },
>> {
>> "op": "take",
>> "item": -2,
>> "item_name": "default~hdd"
>> },
>> {
>> "op": "chooseleaf_indep",
>> "num": 0,
>> "type": "host"
>> },
>> {
>> "op": "emit"
>> }
>> ]
>> },
>> {
>> "rule_id": 2,
>> "rule_name": "rep_3_ssd",
>> "ruleset": 2,
>> "type": 1,
>> "min_size": 1,
>> "max_size": 10,
>> "steps": [
>> {
>> "op": "take",
>> "item": -6,
>> "item_name": "default~ssd"
>> },
>> {
>> "op": "chooseleaf_firstn",
>> "num": 0,
>> "type": "host"
>> },
>> {
>> "op": "emit"
>> }
>> ]
>> }
>> ]
>>
>>
>> Tanks,
>> Lars
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
12:14 p.m.
Hi,
I have set
upmap_max_iterations 2
w/o any impact.
In my opinion the issue is that the evaluation of OSDs data load is not
working.
Or can you explain why osdmaptool does not report anything to do?
Regards
Thomas
Am 03.12.2019 um 08:26 schrieb Harald Staub:
Hi all
Something to try:
ceph config set mgr mgr/balancer/upmap_max_iterations 20
(Default is 100.)
Cheers
Harry
On 03.12.19 08:02, Lars Täuber wrote:
BTW: The osdmaptool doesn't see anything to
do either:
$ ceph osd getmap -o om
$ osdmaptool om --upmap /tmp/upmap.sh --upmap-pool cephfs_data
osdmaptool: osdmap file 'om'
writing upmap command output to: /tmp/upmap.sh
checking for upmap cleanups
upmap, max-count 100, max deviation 0.01
limiting to pools cephfs_data (1)
no upmaps proposed
Tue, 3 Dec 2019 07:30:24 +0100
Lars Täuber <taeuber(a)bbaw.de> ==> Konstantin Shalygin <k0ste(a)k0ste.ru> :
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Hi Konstantin,
Tue, 3 Dec 2019 10:01:34 +0700
Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber
<taeuber(a)bbaw.de>de>, ceph-users(a)ceph.io :
Please paste your `ceph osd df tree`, `ceph osd
pool ls detail`, `ceph
osd crush rule dump`.
here it comes:
$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476
GiB 66 TiB 66.45 1.00 - root default
-25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476
GiB 66 TiB 66.45 1.00 - room PRZ
-26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476
GiB 66 TiB 66.45 1.00 - row rechts
-27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206
GiB 27 TiB 67.51 1.02 - rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66
GiB 10 TiB 62.80 0.95 - host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14
GiB 2.1 TiB 62.48 0.94 163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12
GiB 2.1 TiB 62.47 0.94 163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12
GiB 2.1 TiB 62.53 0.94 163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12
GiB 2.0 TiB 62.90 0.95 164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13
GiB 1.8 TiB 67.32 1.01 176 up osd.37
4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0
GiB 340 GiB 3.01 0.05 110 up osd.4
-13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66
GiB 10 TiB 62.64 0.94 - host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13
GiB 2.1 TiB 62.49 0.94 163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13
GiB 2.1 TiB 62.49 0.94 163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12
GiB 2.1 TiB 62.43 0.94 163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12
GiB 2.1 TiB 62.13 0.94 162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13
GiB 1.8 TiB 67.36 1.01 176 up osd.40
33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2
GiB 340 GiB 3.02 0.05 110 up osd.33
-22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74
GiB 6.4 TiB 77.10 1.16 - host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14
GiB 1.2 TiB 77.59 1.17 203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14
GiB 1.2 TiB 78.40 1.18 205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14
GiB 1.3 TiB 75.80 1.14 198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14
GiB 1.3 TiB 76.13 1.15 199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15
GiB 1006 GiB 82.18 1.24 215 up osd.30
36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0
GiB 340 GiB 2.92 0.04 110 up osd.36
-28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132
GiB 21 TiB 62.36 0.94 - rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66
GiB 11 TiB 62.27 0.94 - host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12
GiB 2.1 TiB 62.08 0.93 162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13
GiB 2.1 TiB 62.13 0.93 162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12
GiB 2.1 TiB 61.77 0.93 161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12
GiB 2.1 TiB 61.75 0.93 161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14
GiB 1.8 TiB 67.31 1.01 176 up osd.38
31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4
GiB 340 GiB 3.04 0.05 110 up osd.31
-16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66
GiB 10 TiB 62.44 0.94 - host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12
GiB 2.1 TiB 62.15 0.94 162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13
GiB 2.1 TiB 62.16 0.94 162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13
GiB 2.1 TiB 62.14 0.94 162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13
GiB 2.1 TiB 62.12 0.93 162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14
GiB 1.8 TiB 67.31 1.01 176 up osd.41
34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8
GiB 340 GiB 3.04 0.05 109 up osd.34
-29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138
GiB 17 TiB 68.95 1.04 - rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63
GiB 11 TiB 61.02 0.92 - host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12
GiB 2.2 TiB 60.63 0.91 158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12
GiB 2.2 TiB 60.19 0.91 157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12
GiB 2.2 TiB 60.27 0.91 157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12
GiB 2.2 TiB 60.28 0.91 157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13
GiB 1.8 TiB 67.34 1.01 176 up osd.39
32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8
GiB 341 GiB 2.88 0.04 109 up osd.32
-19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74
GiB 6.5 TiB 76.89 1.16 - host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13
GiB 1.5 TiB 72.79 1.10 190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16
GiB 1.0 TiB 81.79 1.23 214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16
GiB 1.1 TiB 80.29 1.21 210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14
GiB 1.3 TiB 77.31 1.16 202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14
GiB 1.3 TiB 76.86 1.16 201 up osd.29
35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9
GiB 340 GiB 2.89 0.04 110 up osd.35
TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476
GiB 66 TiB 66.45
MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462
GiB 63 TiB 67.24 1.00 - root default
-25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462
GiB 63 TiB 67.24 1.00 - room PRZ
-26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462
GiB 63 TiB 67.24 1.00 - row rechts
-27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200
GiB 26 TiB 68.31 1.02 - rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64
GiB 10 TiB 63.54 0.94 - host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14
GiB 2.1 TiB 62.48 0.93 163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12
GiB 2.1 TiB 62.47 0.93 163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12
GiB 2.1 TiB 62.53 0.93 163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12
GiB 2.0 TiB 62.90 0.94 164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13
GiB 1.8 TiB 67.32 1.00 176 up osd.37
-13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64
GiB 10 TiB 63.38 0.94 - host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13
GiB 2.1 TiB 62.49 0.93 163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13
GiB 2.1 TiB 62.49 0.93 163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12
GiB 2.1 TiB 62.43 0.93 163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12
GiB 2.1 TiB 62.13 0.92 162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13
GiB 1.8 TiB 67.36 1.00 176 up osd.40
-22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72
GiB 6.1 TiB 78.02 1.16 - host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14
GiB 1.2 TiB 77.59 1.15 203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14
GiB 1.2 TiB 78.40 1.17 205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14
GiB 1.3 TiB 75.80 1.13 198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14
GiB 1.3 TiB 76.13 1.13 199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15
GiB 1006 GiB 82.18 1.22 215 up osd.30
-28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128
GiB 20 TiB 63.09 0.94 - rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63
GiB 10 TiB 63.01 0.94 - host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12
GiB 2.1 TiB 62.08 0.92 162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13
GiB 2.1 TiB 62.13 0.92 162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12
GiB 2.1 TiB 61.77 0.92 161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12
GiB 2.1 TiB 61.75 0.92 161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14
GiB 1.8 TiB 67.31 1.00 176 up osd.38
-16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65
GiB 10 TiB 63.18 0.94 - host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12
GiB 2.1 TiB 62.15 0.92 162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13
GiB 2.1 TiB 62.16 0.92 162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13
GiB 2.1 TiB 62.14 0.92 162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13
GiB 2.1 TiB 62.12 0.92 162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14
GiB 1.8 TiB 67.31 1.00 176 up osd.41
-29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134
GiB 17 TiB 69.77 1.04 - rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62
GiB 11 TiB 61.74 0.92 - host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12
GiB 2.2 TiB 60.63 0.90 158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12
GiB 2.2 TiB 60.19 0.90 157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12
GiB 2.2 TiB 60.27 0.90 157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12
GiB 2.2 TiB 60.28 0.90 157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13
GiB 1.8 TiB 67.34 1.00 176 up osd.39
-19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72
GiB 6.1 TiB 77.81 1.16 - host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13
GiB 1.5 TiB 72.79 1.08 190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16
GiB 1.0 TiB 81.79 1.22 214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16
GiB 1.1 TiB 80.29 1.19 210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14
GiB 1.3 TiB 77.31 1.15 202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14
GiB 1.3 TiB 76.86 1.14 201 up osd.29
TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462
GiB 63 TiB 67.24
MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail
pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1
object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on
last_change 20353 lfor 0/0/2366 flags
hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410
stripe_width 16384 target_size_ratio 0.85 application cephfs
removed_snaps
[2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1]
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2
object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on
last_change 261 lfor 0/0/259 flags hashpspool stripe_width 0
pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5
target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "cephfs_data",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 6,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -2,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "rep_3_ssd",
"ruleset": 2,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -6,
"item_name": "default~ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
Tanks,
Lars
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4 Dec
4 Dec
12:16 p.m.
On 12/3/19 1:30 PM, Lars Täuber wrote:
here it comes:
$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR
PGS STATUS TYPE NAME
-1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- root default
-25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- room PRZ
-26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00
- row rechts
-27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB 67.51 1.02
- rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB 62.80 0.95
- host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.94
163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.94
163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.94
163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.95
164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.01
176 up osd.37
4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB 3.01 0.05
110 up osd.4
-13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB 62.64 0.94
- host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.94
163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.94
163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.94
163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.94
162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.01
176 up osd.40
33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB 3.02 0.05
110 up osd.33
-22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB 77.10 1.16
- host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.17
203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.18
205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.14
198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.15
199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.24
215 up osd.30
36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB 2.92 0.04
110 up osd.36
-28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB 62.36 0.94
- rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB 62.27 0.94
- host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.93
162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.93
162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.93
161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.93
161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.01
176 up osd.38
31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB 3.04 0.05
110 up osd.31
-16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB 62.44 0.94
- host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.94
162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.94
162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.94
162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.93
162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.01
176 up osd.41
34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB 3.04 0.05
109 up osd.34
-29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB 68.95 1.04
- rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB 61.02 0.92
- host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.91
158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.91
157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.91
157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.91
157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.01
176 up osd.39
32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB 2.88 0.04
109 up osd.32
-19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB 76.89 1.16
- host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.10
190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.23
214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.21
210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.16
202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.16
201 up osd.29
35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB 2.89 0.04
110 up osd.35
TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45
MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR
PGS STATUS TYPE NAME
-1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- root default
-25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- room PRZ
-26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00
- row rechts
-27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB 68.31 1.02
- rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB 63.54 0.94
- host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.93
163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.93
163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.93
163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.94
164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.00
176 up osd.37
-13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB 63.38 0.94
- host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.93
163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.93
163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.93
163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.92
162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.00
176 up osd.40
-22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB 78.02 1.16
- host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.15
203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.17
205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.13
198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.13
199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.22
215 up osd.30
-28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB 63.09 0.94
- rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB 63.01 0.94
- host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.92
162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.92
162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.92
161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.92
161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.00
176 up osd.38
-16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB 63.18 0.94
- host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.92
162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.92
162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.92
162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.92
162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.00
176 up osd.41
-29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB 69.77 1.04
- rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB 61.74 0.92
- host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.90
158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.90
157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.90
157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.90
157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.00
176 up osd.39
-19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB 77.81 1.16
- host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.08
190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.22
214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.19
210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.15
202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.14
201 up osd.29
TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24
MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail
pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash rjenkins
pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags
hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384
target_size_ratio 0.85 application cephfs
removed_snaps
[2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1]
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259 flags
hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5
target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "cephfs_data",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 6,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -2,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "rep_3_ssd",
"ruleset": 2,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -6,
"item_name": "default~ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
Lars, you have too much PG's for this OSD's. I suggest to disable PG
autoscaler and:
- reduce number of PG's for cephfs_metada pool to something like 16 PG's.
- reduce number of PG's for cephfs_data to something like 512.
- update crush rule for cephfs_metadata pool - set domain to 'rack'
instead 'host'.
Also please paste your `ceph osd tree`.
k
12:34 p.m.
Hi Konstantin,
thanks for your suggestions.
Lars, you have too much PG's for this OSD's. I
suggest to disable PG
autoscaler and:
- reduce number of PG's for cephfs_metada pool to something like 16 PG's.
Done.
- reduce number of PG's for cephfs_data to something like 512.
Done.
- update crush rule for cephfs_metadata pool - set
domain to 'rack'
instead 'host'.
Done.
Also please paste your `ceph osd tree`.
$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 195.40730 root default
-25 195.40730 room PRZ
-26 195.40730 row rechts
-27 83.74599 rack 1-eins
-3 27.91533 host onode1
0 hdd 5.51459 osd.0 up 1.00000 1.00000
1 hdd 5.51459 osd.1 up 1.00000 1.00000
2 hdd 5.51459 osd.2 up 1.00000 1.00000
3 hdd 5.51459 osd.3 up 1.00000 1.00000
37 hdd 5.51459 osd.37 up 1.00000 1.00000
4 ssd 0.34239 osd.4 up 1.00000 1.00000
-13 27.91533 host onode4
13 hdd 5.51459 osd.13 up 1.00000 1.00000
14 hdd 5.51459 osd.14 up 1.00000 1.00000
15 hdd 5.51459 osd.15 up 1.00000 1.00000
16 hdd 5.51459 osd.16 up 1.00000 1.00000
40 hdd 5.51459 osd.40 up 1.00000 1.00000
33 ssd 0.34239 osd.33 up 1.00000 1.00000
-22 27.91533 host onode7
25 hdd 5.51459 osd.25 up 1.00000 1.00000
26 hdd 5.51459 osd.26 up 1.00000 1.00000
27 hdd 5.51459 osd.27 up 1.00000 1.00000
28 hdd 5.51459 osd.28 up 1.00000 1.00000
30 hdd 5.51459 osd.30 up 1.00000 1.00000
36 ssd 0.34239 osd.36 up 1.00000 1.00000
-28 55.83066 rack 2-zwei
-7 27.91533 host onode2
5 hdd 5.51459 osd.5 up 1.00000 1.00000
6 hdd 5.51459 osd.6 up 1.00000 1.00000
7 hdd 5.51459 osd.7 up 1.00000 1.00000
8 hdd 5.51459 osd.8 up 1.00000 1.00000
38 hdd 5.51459 osd.38 up 1.00000 1.00000
31 ssd 0.34239 osd.31 up 1.00000 1.00000
-16 27.91533 host onode5
17 hdd 5.51459 osd.17 up 1.00000 1.00000
18 hdd 5.51459 osd.18 up 1.00000 1.00000
19 hdd 5.51459 osd.19 up 1.00000 1.00000
20 hdd 5.51459 osd.20 up 1.00000 1.00000
41 hdd 5.51459 osd.41 up 1.00000 1.00000
34 ssd 0.34239 osd.34 up 1.00000 1.00000
-29 55.83066 rack 3-drei
-10 27.91533 host onode3
9 hdd 5.51459 osd.9 up 1.00000 1.00000
10 hdd 5.51459 osd.10 up 1.00000 1.00000
11 hdd 5.51459 osd.11 up 1.00000 1.00000
12 hdd 5.51459 osd.12 up 1.00000 1.00000
39 hdd 5.51459 osd.39 up 1.00000 1.00000
32 ssd 0.34239 osd.32 up 1.00000 1.00000
-19 27.91533 host onode6
21 hdd 5.51459 osd.21 up 1.00000 1.00000
22 hdd 5.51459 osd.22 up 1.00000 1.00000
23 hdd 5.51459 osd.23 up 1.00000 1.00000
24 hdd 5.51459 osd.24 up 1.00000 1.00000
29 hdd 5.51459 osd.29 up 1.00000 1.00000
35 ssd 0.34239 osd.35 up 1.00000 1.00000
So I just wait for the remapping and merging being done and see what happens.
Thanks so far!
Best regards,
Lars
12:51 p.m.
On 12/4/19 4:04 PM, Lars Täuber wrote:
So I just wait for the remapping and merging being
done and see what happens.
Thanks so far!
Also don't forget to call `ceph osd crush weight-set rm-compat`.
And stop mgr balancer `ceph balancer off`.
After your rebalance is complete you can try:
`ceph osd getmap -o om ; osdmaptool om --upmap upmap.sh
--upmap-pool=cephfs_data --upmap-deviation=0`
k
16 Dec
16 Dec
11:12 a.m.
Hi Konstantin,
the number PG's for the metadata pool now is 16.
The number PG's for the data pool now is 512.
I removed the backward-compatible weight-set.
But now:
$ ceph osd getmap -o om ; osdmaptool om --upmap upmap.sh --upmap-pool=cephfs_data
--upmap-deviation=0
got osdmap epoch 94013
osdmaptool: osdmap file 'om'
writing upmap command output to: upmap.sh
checking for upmap cleanups
upmap, max-count 100, max deviation 0
limiting to pools cephfs_data (1)
no upmaps proposed
and
$ ceph osd df class hdd
[…]
MIN/MAX VAR: 0.80/1.20 STDDEV: 6.45
There seems to be a bug in nautilus.
I think about increasing the number of PG's for the data pool again, because the
average number of PG's per OSD now is 76.8.
What do you say?
Thanks,
Lars
Wed, 4 Dec 2019 16:21:33 +0700
Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber <taeuber(a)bbaw.de>de>,
ceph-users(a)ceph.io :
On 12/4/19 4:04 PM, Lars Täuber wrote:
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
So I just wait for the remapping and merging
being done and see what happens.
Thanks so far!
Also don't forget to call `ceph osd crush weight-set rm-compat`.
And stop mgr balancer `ceph balancer off`.
After your rebalance is complete you can try:
`ceph osd getmap -o om ; osdmaptool om --upmap upmap.sh
--upmap-pool=cephfs_data --upmap-deviation=0`
k
11:47 a.m.
On 12/16/19 2:42 PM, Lars Täuber wrote:
There seems to be a bug in nautilus.
I think about increasing the number of PG's for the data pool again, because the
average number of PG's per OSD now is 76.8.
What do you say?
May be bug in Nautilus, may be in osdmaptool.
Please, upload your binary crush map (`osd getmap`).
k
11:55 a.m.
Mon, 16 Dec 2019 15:17:37 +0700
Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber <taeuber(a)bbaw.de> :
On 12/16/19 2:42 PM, Lars Täuber wrote:
Here it comes.
Thanks,
Lars
There seems to be a bug in nautilus.
I think about increasing the number of PG's for the data pool again, because the
average number of PG's per OSD now is 76.8.
What do you say?
May be bug in Nautilus, may be in osdmaptool.
Please, upload your binary crush map (`osd getmap`).
12:08 p.m.
On 12/16/19 3:25 PM, Lars Täuber wrote:
Here it comes.
Maybe some bug in osdmaptool, when defined pools is less than one no
actually do_upmap is executed.
Try like this:
`osdmaptool osdmap.om --upmap upmap.sh --upmap-pool=cephfs_data
--upmap-pool=cephfs_metadata --upmap-deviation=0 --upmap-max=1000`
In my env actually upmaps purposed only for pool_id 1.
CC'ed David.
k
12:20 p.m.
Mon, 16 Dec 2019 15:38:30 +0700
Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber <taeuber(a)bbaw.de> :
On 12/16/19 3:25 PM, Lars Täuber wrote:
Oh yes. There are 240 lines in the upmap.sh.
I started the balancer again in the meanwhile.
I'll watch the cluster and report.
Thanks!
Lars
Here it comes.
Maybe some bug in osdmaptool, when defined pools is less than one no
actually do_upmap is executed.
Try like this:
`osdmaptool osdmap.om --upmap upmap.sh --upmap-pool=cephfs_data
--upmap-pool=cephfs_metadata --upmap-deviation=0 --upmap-max=1000`
In my env actually upmaps purposed only for pool_id 1.
CC'ed David.
k
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
17 Dec
17 Dec
10:15 a.m.
Hi Konstantin,
the cluster has finished it's backfilling.
I got this situation:
$ ceph osd df class hdd
[…]
MIN/MAX VAR: 0.86/1.08 STDDEV: 2.05
Now I created a new upmap.sh and sourced it. The cluster is busy again with 3% of its
objects.
I'll report the result.
Thanks for all your hints.
Regards,
Lars
Mon, 16 Dec 2019 15:38:30 +0700
Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber <taeuber(a)bbaw.de> :
> On 12/16/19 3:25 PM, Lars Täuber wrote:
> > Here it comes.
>
> Maybe some bug in osdmaptool, when defined pools is less than one no
> actually do_upmap is executed.
>
> Try like this:
>
> `osdmaptool osdmap.om --upmap upmap.sh --upmap-pool=cephfs_data
> --upmap-pool=cephfs_metadata --upmap-deviation=0 --upmap-max=1000`
>
> In my env actually upmaps purposed only for pool_id 1.
>
> CC'ed David.
>
>
>
> k
>
18 Dec
18 Dec
10:46 a.m.
Hi Konstantin,
the situation after moving the PGs with osdmaptool is not really better than without:
$ ceph osd df class hdd
[…]
MIN/MAX VAR: 0.86/1.08 STDDEV: 2.04
The OSD with the fewest PGs has 66 of them, the one with the most has 83.
Is this the expected result? I'm unsure how much unusable space this means but I'm
sure there is a relevant amount of it.
Thanks for all your patience and support
Lars
Tue, 17 Dec 2019 07:45:24 +0100
Lars Täuber <taeuber(a)bbaw.de> ==> Konstantin Shalygin <k0ste(a)k0ste.ru> :
> Hi Konstantin,
>
> the cluster has finished it's backfilling.
> I got this situation:
>
> $ ceph osd df class hdd
> […]
> MIN/MAX VAR: 0.86/1.08 STDDEV: 2.05
>
> Now I created a new upmap.sh and sourced it. The cluster is busy again with 3% of its
objects.
> I'll report the result.
>
> Thanks for all your hints.
>
> Regards,
> Lars
>
>
>
> Mon, 16 Dec 2019 15:38:30 +0700
> Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber <taeuber(a)bbaw.de>
:
> > On 12/16/19 3:25 PM, Lars Täuber wrote:
> > > Here it comes.
> >
> > Maybe some bug in osdmaptool, when defined pools is less than one no
> > actually do_upmap is executed.
> >
> > Try like this:
> >
> > `osdmaptool osdmap.om --upmap upmap.sh --upmap-pool=cephfs_data
> > --upmap-pool=cephfs_metadata --upmap-deviation=0 --upmap-max=1000`
> >
> > In my env actually upmaps purposed only for pool_id 1.
> >
> > CC'ed David.
> >
> >
> >
> > k
11:44 a.m.
New subject: Nautilus RadosGW "One Zone" like AWS
Hello,
is it already possible or planned to enable RadosGW to have a storage
class like AWS does with "one zone" where the objects only exists in one
zone and don't get mirrored to any other zone? eg. by creating the
placement pool only in one zone?
If this feature isn't planned is it worth creating a feature request?
What do you think about that feature?
All the best,
Florian
23 Dec
23 Dec
10:17 a.m.
On 12/18/19 2:16 PM, Lars Täuber wrote:
the situation after moving the PGs with osdmaptool is
not really better than without:
$ ceph osd df class hdd
[…]
MIN/MAX VAR: 0.86/1.08 STDDEV: 2.04
The OSD with the fewest PGs has 66 of them, the one with the most has 83.
Is this the expected result? I'm unsure how much unusable space this means but
I'm sure there is a relevant amount of it.
You lower your STDDEV from 7.47 to 2.04. You should run script until "no
upmap's purposed".
k
6 Jan
6 Jan
3:48 p.m.
Hi Konstantin,
Mon, 23 Dec 2019 13:47:55 +0700
Konstantin Shalygin <k0ste(a)k0ste.ru> ==> Lars Täuber <taeuber(a)bbaw.de> :
On 12/18/19 2:16 PM, Lars Täuber wrote:
this is already the case.
Can I do anything about it?
Thanks,
Lars
the situation after moving the PGs with
osdmaptool is not really better than without:
$ ceph osd df class hdd
[…]
MIN/MAX VAR: 0.86/1.08 STDDEV: 2.04
The OSD with the fewest PGs has 66 of them, the one with the most has 83.
Is this the expected result? I'm unsure how much unusable space this means but
I'm sure there is a relevant amount of it.
You lower your STDDEV from 7.47 to 2.04. You should run script until "no
upmap's purposed".
1566
days inactive
1615
days old
23 comments
6 participants
participants (6)
-
Florian Engelmann -
Harald Staub -
Konstantin Shalygin -
Lars Täuber -
Paul Emmerich -
Thomas Schneider