Hi, in this <https://ceph.io/community/the-first-telemetry-results-are-in/> blog post I find this statement: "So, in our ideal world so far (assuming equal size OSDs), every OSD now has the same number of PGs assigned." My issue is that accross all pools the number of PGs per OSD is not equal. And I conclude that this is causing very unbalanced data placement. As a matter of fact the data stored on my 1.6TB HDD in specific pool "hdb_backup" is in a range starting with osd.228 size: 1.6 usage: 52.61 reweight: 1.00000 and ending with osd.145 size: 1.6 usage: 81.11 reweight: 1.00000 This impacts the amount of data that can be stored in the cluster heavily. Ceph balancer is enabled, but this is not solving this issue. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Therefore I would ask you for suggestions how to work on this unbalanced data distribution. I have attached pastebin for - ceph osd df sorted by usage <https://pastebin.com/QLQHjA9g> - ceph osd df tree <https://pastebin.com/SvhP2hp5> My cluster has multiple crush roots respresenting different disks. In addition I have defined multiple pools, one pool for each disk type: hdd, ssd, nvme. THX
You have way too few PGs in one of the roots. Many OSDs have so few PGs that you should see a lot of health warnings because of it. The other root has a factor 5 difference in disk size which isn't ideal either. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Nov 18, 2019 at 3:03 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
in this <https://ceph.io/community/the-first-telemetry-results-are-in/> blog post I find this statement: "So, in our ideal world so far (assuming equal size OSDs), every OSD now has the same number of PGs assigned."
My issue is that accross all pools the number of PGs per OSD is not equal. And I conclude that this is causing very unbalanced data placement. As a matter of fact the data stored on my 1.6TB HDD in specific pool "hdb_backup" is in a range starting with osd.228 size: 1.6 usage: 52.61 reweight: 1.00000 and ending with osd.145 size: 1.6 usage: 81.11 reweight: 1.00000
This impacts the amount of data that can be stored in the cluster heavily.
Ceph balancer is enabled, but this is not solving this issue. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
Therefore I would ask you for suggestions how to work on this unbalanced data distribution.
I have attached pastebin for - ceph osd df sorted by usage <https://pastebin.com/QLQHjA9g> - ceph osd df tree <https://pastebin.com/SvhP2hp5>
My cluster has multiple crush roots respresenting different disks. In addition I have defined multiple pools, one pool for each disk type: hdd, ssd, nvme.
THX _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Paul, thanks for your analysis. I want to share more statistics of my cluster to follow-up on your response "You have way too few PGs in one of the roots". Here are the pool details: root@ld3955:~# ceph osd pool ls detail pool 11 'hdb_backup' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode warn last_change 294572 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~3] pool 59 'hdd' replicated size 2 min_size 2 crush_rule 3 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 267271 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~3] pool 60 'ssd' replicated size 2 min_size 2 crush_rule 4 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 299719 lfor 299717/299717/299717 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~3] pool 61 'nvme' replicated size 2 min_size 2 crush_rule 2 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 267125 flags hashpspool stripe_width 0 application rbd pool 62 'cephfs_data' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 300312 lfor 300310/300310/300310 flags hashpspool stripe_width 0 application cephfs pool 63 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 267069 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs Any pg_num / pgp_num is monitored by Ceph, means I get a warning in the log / health status if a pool is undersized. I didn't enable PG auto-scaler for any pool, though. The calculation of PGs per Pool is done with pgcalc <https://ceph.io/pgcalc/>. Here's a screenshot <https://ibb.co/VjR6X3x> of this calculation. My focus is on pool hdb_backup. Based on these statistics root@ld3955:~# ceph df detail RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.4 PiB 744 TiB 729 TiB 730 TiB 49.53 nvme 23 TiB 23 TiB 43 GiB 51 GiB 0.22 ssd 27 TiB 25 TiB 1.9 TiB 1.9 TiB 7.15 TOTAL 1.5 PiB 792 TiB 731 TiB 732 TiB 48.02 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR hdb_backup 11 241 TiB 63.29M 241 TiB 57.03 61 TiB N/A N/A 63.29M 0 B 0 B hdd 59 553 GiB 142.16k 553 GiB 0.50 54 TiB N/A N/A 142.16k 0 B 0 B ssd 60 2.0 TiB 530.75k 2.0 TiB 8.72 10 TiB N/A N/A 530.75k 0 B 0 B nvme 61 0 B 0 0 B 0 11 TiB N/A N/A 0 0 B 0 B cephfs_data 62 356 GiB 102.29k 356 GiB 0.32 36 TiB N/A N/A 102.29k 0 B 0 B cephfs_metadata 63 117 MiB 52 117 MiB 0 36 TiB N/A N/A 52 0 B 0 B there's only 57% used, but effectively I cannot store much more data because some OSDs are filling up by +80%. It is true that the disks that are used for this pool exclusively are different in size, means 3x 48 disks à 7.2TB 4x 48 disks à 1.6TB and the disk usage is for 7.2TB disk from 41% to 54% and for 1.6TB disks from 52% to 81%. If Ceph is not cabable to manage rebalancing automatically, how can I proceed to rebalance the data manually? OSD reweight is not an option in my opinion because it starts filling OSDs that are not with lowest usage rate. Can I move PGs to specific OSDs? THX Am 18.11.2019 um 20:18 schrieb Paul Emmerich:
You have way too few PGs in one of the roots. Many OSDs have so few PGs that you should see a lot of health warnings because of it. The other root has a factor 5 difference in disk size which isn't ideal either.
Paul
On 11/19/19 4:01 PM, Thomas Schneider wrote:
If Ceph is not cabable to manage rebalancing automatically, how can I proceed to rebalance the data manually?
Use offline upmap for your target pool: ceph osd getmap -o om; osdmaptool om --upmap upmap.sh --upmap-pool=hdd_backup --upmap-deviation 0; bash upmap.sh; rm -f upmap.sh om k
Hi, the folline upmap reports "no upmaps proposed". root@ld3955:/home# osdmaptool om --upmap hdd-upmap.sh --upmap-pool=hdb_backup --upmap-deviation 0 osdmaptool: osdmap file 'om' writing upmap command output to: hdd-upmap.sh checking for upmap cleanups upmap, max-count 100, max deviation 0 limiting to pools hdb_backup (11) nno upmaps proposed root@ld3955:/home# ls -ltr | tail -rw-r--r-- 1 root root 60758 Nov 5 08:27 osd_tree -rw-r--r-- 1 root root 21686 Nov 15 14:12 compiled-crushmap-15-11-2019_14-12 -rw-r--r-- 1 root root 29996 Nov 15 14:13 decompiled-crushmap-15-11-2019_14-12 -rw-r--r-- 1 root root 30426 Nov 15 14:55 new-decompiled-crushmap-15-11-2019_14-12 -rw-r--r-- 1 root root 20381 Nov 15 14:55 new-compiled-crushmap-15-11-2019_14-12 -rw-r--r-- 1 root root 12507 Nov 18 14:57 disk-usage -rw-r--r-- 1 root root 60626 Nov 18 14:58 osd-tree -rw-r--r-- 1 root root 7380 Nov 22 16:46 ceph.tar.gz -rw-r--r-- 1 root root 166421 Nov 30 22:53 om -rw-r--r-- 1 root root 0 Nov 30 23:11 hdd-upmap.sh In my opinion this shows that the evalutation step is not working because my OSDs are not balanced. Therefore I would conclude to open a bug report for this issue. Thomas Am 19.11.2019 um 10:22 schrieb Konstantin Shalygin:
On 11/19/19 4:01 PM, Thomas Schneider wrote:
If Ceph is not cabable to manage rebalancing automatically, how can I proceed to rebalance the data manually?
Use offline upmap for your target pool:
ceph osd getmap -o om; osdmaptool om --upmap upmap.sh --upmap-pool=hdd_backup --upmap-deviation 0; bash upmap.sh; rm -f upmap.sh om
k
Hi there! Here we have a similar situation. After adding some OSDs to the cluster the PGs are not equally distributed over the OSDs. The balancing mode is set to upmap. The docs https://docs.ceph.com/docs/master/rados/operations/balancer/#modes say: "This CRUSH mode will optimize the placement of individual PGs in order to achieve a balanced distribution. In most cases, this distribution is “perfect,” which an equal number of PGs on each OSD (+/-1 PG, since they might not divide evenly)." This is not the case here with our cluster. The number of PGs reaches from 157 up to 214. And so the resulting usage of the HDDs varies from 60% to 82%. $ ceph osd df class hdd […] MIN/MAX VAR: 0.89/1.22 STDDEV: 7.47 In the meantime I had to use reweight-by-utilization to get the cluster in healthy state again, because it had a near full PG and near full OSDs. The cluster is nautilus 14.2.4 on debian 10 with packages from croit.io. I think about switching back to crush-compat. Is there a bug known regarding this? Thanks Lars
On 12/2/19 5:55 PM, Lars Täuber wrote:
Here we have a similar situation. After adding some OSDs to the cluster the PGs are not equally distributed over the OSDs.
The balancing mode is set to upmap. The docshttps://docs.ceph.com/docs/master/rados/operations/balancer/#modes say: "This CRUSH mode will optimize the placement of individual PGs in order to achieve a balanced distribution. In most cases, this distribution is “perfect,” which an equal number of PGs on each OSD (+/-1 PG, since they might not divide evenly)."
This is not the case here with our cluster. The number of PGs reaches from 157 up to 214. And so the resulting usage of the HDDs varies from 60% to 82%. $ ceph osd df class hdd […] MIN/MAX VAR: 0.89/1.22 STDDEV: 7.47
In the meantime I had to use reweight-by-utilization to get the cluster in healthy state again, because it had a near full PG and near full OSDs.
The cluster is nautilus 14.2.4 on debian 10 with packages from croit.io.
I think about switching back to crush-compat.
Is there a bug known regarding this?
Please paste your `ceph osd df tree`, `ceph osd pool ls detail`, `ceph osd crush rule dump`. k
Hi Konstantin, Tue, 3 Dec 2019 10:01:34 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de>, ceph-users@ceph.io :
Please paste your `ceph osd df tree`, `ceph osd pool ls detail`, `ceph osd crush rule dump`.
here it comes: $ ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - root default -25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - room PRZ -26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - row rechts -27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB 67.51 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB 62.80 0.95 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.94 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.94 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.94 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.95 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.01 176 up osd.37 4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB 3.01 0.05 110 up osd.4 -13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB 62.64 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.94 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.94 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.01 176 up osd.40 33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB 3.02 0.05 110 up osd.33 -22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB 77.10 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.17 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.18 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.14 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.15 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.24 215 up osd.30 36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB 2.92 0.04 110 up osd.36 -28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB 62.36 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB 62.27 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.93 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.93 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.93 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.93 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.38 31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB 3.04 0.05 110 up osd.31 -16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB 62.44 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.94 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.94 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.94 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.93 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.41 34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB 3.04 0.05 109 up osd.34 -29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB 68.95 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB 61.02 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.91 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.91 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.91 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.91 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.01 176 up osd.39 32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB 2.88 0.04 109 up osd.32 -19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB 76.89 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.10 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.23 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.21 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.16 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.16 201 up osd.29 35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB 2.89 0.04 110 up osd.35 TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74 better only for the class hdd $ ceph osd df tree class hdd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - root default -25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - room PRZ -26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - row rechts -27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB 68.31 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB 63.54 0.94 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.93 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.93 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.93 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.94 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.00 176 up osd.37 -13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB 63.38 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.93 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.92 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.00 176 up osd.40 -22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB 78.02 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.15 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.17 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.13 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.13 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.22 215 up osd.30 -28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB 63.09 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB 63.01 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.92 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.92 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.92 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.92 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.38 -16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB 63.18 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.92 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.92 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.92 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.92 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.41 -29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB 69.77 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB 61.74 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.90 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.90 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.90 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.90 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.00 176 up osd.39 -19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB 77.81 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.08 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.22 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.19 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.15 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.14 201 up osd.29 TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17 ceph osd pool ls detail pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384 target_size_ratio 0.85 application cephfs removed_snaps [2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1] pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 target_size_ratio 0.3 application cephfs $ ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "cephfs_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "rep_3_ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -6, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] Tanks, Lars
BTW: The osdmaptool doesn't see anything to do either: $ ceph osd getmap -o om $ osdmaptool om --upmap /tmp/upmap.sh --upmap-pool cephfs_data osdmaptool: osdmap file 'om' writing upmap command output to: /tmp/upmap.sh checking for upmap cleanups upmap, max-count 100, max deviation 0.01 limiting to pools cephfs_data (1) no upmaps proposed Tue, 3 Dec 2019 07:30:24 +0100 Lars Täuber <taeuber@bbaw.de> ==> Konstantin Shalygin <k0ste@k0ste.ru> :
Hi Konstantin,
Tue, 3 Dec 2019 10:01:34 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de>, ceph-users@ceph.io :
Please paste your `ceph osd df tree`, `ceph osd pool ls detail`, `ceph osd crush rule dump`.
here it comes:
$ ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - root default -25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - room PRZ -26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - row rechts -27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB 67.51 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB 62.80 0.95 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.94 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.94 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.94 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.95 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.01 176 up osd.37 4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB 3.01 0.05 110 up osd.4 -13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB 62.64 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.94 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.94 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.01 176 up osd.40 33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB 3.02 0.05 110 up osd.33 -22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB 77.10 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.17 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.18 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.14 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.15 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.24 215 up osd.30 36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB 2.92 0.04 110 up osd.36 -28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB 62.36 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB 62.27 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.93 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.93 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.93 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.93 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.38 31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB 3.04 0.05 110 up osd.31 -16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB 62.44 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.94 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.94 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.94 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.93 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.41 34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB 3.04 0.05 109 up osd.34 -29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB 68.95 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB 61.02 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.91 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.91 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.91 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.91 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.01 176 up osd.39 32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB 2.88 0.04 109 up osd.32 -19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB 76.89 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.10 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.23 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.21 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.16 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.16 201 up osd.29 35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB 2.89 0.04 110 up osd.35 TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - root default -25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - room PRZ -26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - row rechts -27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB 68.31 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB 63.54 0.94 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.93 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.93 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.93 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.94 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.00 176 up osd.37 -13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB 63.38 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.93 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.92 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.00 176 up osd.40 -22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB 78.02 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.15 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.17 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.13 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.13 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.22 215 up osd.30 -28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB 63.09 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB 63.01 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.92 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.92 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.92 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.92 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.38 -16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB 63.18 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.92 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.92 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.92 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.92 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.41 -29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB 69.77 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB 61.74 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.90 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.90 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.90 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.90 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.00 176 up osd.39 -19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB 77.81 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.08 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.22 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.19 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.15 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.14 201 up osd.29 TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384 target_size_ratio 0.85 application cephfs removed_snaps [2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1] pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "cephfs_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "rep_3_ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -6, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
Tanks, Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de
Hi all Something to try: ceph config set mgr mgr/balancer/upmap_max_iterations 20 (Default is 100.) Cheers Harry On 03.12.19 08:02, Lars Täuber wrote:
BTW: The osdmaptool doesn't see anything to do either:
$ ceph osd getmap -o om $ osdmaptool om --upmap /tmp/upmap.sh --upmap-pool cephfs_data osdmaptool: osdmap file 'om' writing upmap command output to: /tmp/upmap.sh checking for upmap cleanups upmap, max-count 100, max deviation 0.01 limiting to pools cephfs_data (1) no upmaps proposed
Tue, 3 Dec 2019 07:30:24 +0100 Lars Täuber <taeuber@bbaw.de> ==> Konstantin Shalygin <k0ste@k0ste.ru> :
Hi Konstantin,
Tue, 3 Dec 2019 10:01:34 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de>, ceph-users@ceph.io :
Please paste your `ceph osd df tree`, `ceph osd pool ls detail`, `ceph osd crush rule dump`.
here it comes:
$ ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - root default -25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - room PRZ -26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - row rechts -27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB 67.51 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB 62.80 0.95 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.94 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.94 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.94 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.95 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.01 176 up osd.37 4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB 3.01 0.05 110 up osd.4 -13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB 62.64 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.94 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.94 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.01 176 up osd.40 33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB 3.02 0.05 110 up osd.33 -22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB 77.10 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.17 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.18 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.14 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.15 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.24 215 up osd.30 36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB 2.92 0.04 110 up osd.36 -28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB 62.36 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB 62.27 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.93 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.93 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.93 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.93 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.38 31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB 3.04 0.05 110 up osd.31 -16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB 62.44 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.94 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.94 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.94 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.93 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.41 34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB 3.04 0.05 109 up osd.34 -29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB 68.95 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB 61.02 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.91 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.91 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.91 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.91 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.01 176 up osd.39 32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB 2.88 0.04 109 up osd.32 -19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB 76.89 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.10 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.23 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.21 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.16 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.16 201 up osd.29 35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB 2.89 0.04 110 up osd.35 TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - root default -25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - room PRZ -26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - row rechts -27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB 68.31 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB 63.54 0.94 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.93 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.93 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.93 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.94 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.00 176 up osd.37 -13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB 63.38 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.93 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.92 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.00 176 up osd.40 -22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB 78.02 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.15 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.17 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.13 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.13 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.22 215 up osd.30 -28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB 63.09 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB 63.01 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.92 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.92 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.92 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.92 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.38 -16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB 63.18 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.92 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.92 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.92 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.92 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.41 -29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB 69.77 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB 61.74 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.90 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.90 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.90 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.90 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.00 176 up osd.39 -19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB 77.81 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.08 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.22 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.19 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.15 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.14 201 up osd.29 TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384 target_size_ratio 0.85 application cephfs removed_snaps [2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1] pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "cephfs_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "rep_3_ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -6, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
Tanks, Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I have set upmap_max_iterations 2 w/o any impact. In my opinion the issue is that the evaluation of OSDs data load is not working. Or can you explain why osdmaptool does not report anything to do? Regards Thomas Am 03.12.2019 um 08:26 schrieb Harald Staub:
Hi all
Something to try: ceph config set mgr mgr/balancer/upmap_max_iterations 20
(Default is 100.)
Cheers Harry
On 03.12.19 08:02, Lars Täuber wrote:
BTW: The osdmaptool doesn't see anything to do either:
$ ceph osd getmap -o om $ osdmaptool om --upmap /tmp/upmap.sh --upmap-pool cephfs_data osdmaptool: osdmap file 'om' writing upmap command output to: /tmp/upmap.sh checking for upmap cleanups upmap, max-count 100, max deviation 0.01 limiting to pools cephfs_data (1) no upmaps proposed
Tue, 3 Dec 2019 07:30:24 +0100 Lars Täuber <taeuber@bbaw.de> ==> Konstantin Shalygin <k0ste@k0ste.ru> :
Hi Konstantin,
Tue, 3 Dec 2019 10:01:34 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de>, ceph-users@ceph.io :
Please paste your `ceph osd df tree`, `ceph osd pool ls detail`, `ceph osd crush rule dump`.
here it comes:
$ ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - root default -25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - room PRZ -26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - row rechts -27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB 67.51 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB 62.80 0.95 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.94 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.94 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.94 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.95 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.01 176 up osd.37 4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB 3.01 0.05 110 up osd.4 -13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB 62.64 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.94 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.94 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.01 176 up osd.40 33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB 3.02 0.05 110 up osd.33 -22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB 77.10 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.17 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.18 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.14 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.15 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.24 215 up osd.30 36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB 2.92 0.04 110 up osd.36 -28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB 62.36 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB 62.27 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.93 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.93 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.93 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.93 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.38 31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB 3.04 0.05 110 up osd.31 -16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB 62.44 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.94 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.94 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.94 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.93 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.41 34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB 3.04 0.05 109 up osd.34 -29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB 68.95 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB 61.02 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.91 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.91 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.91 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.91 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.01 176 up osd.39 32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB 2.88 0.04 109 up osd.32 -19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB 76.89 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.10 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.23 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.21 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.16 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.16 201 up osd.29 35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB 2.89 0.04 110 up osd.35 TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - root default -25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - room PRZ -26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - row rechts -27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB 68.31 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB 63.54 0.94 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.93 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.93 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.93 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.94 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.00 176 up osd.37 -13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB 63.38 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.93 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.92 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.00 176 up osd.40 -22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB 78.02 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.15 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.17 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.13 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.13 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.22 215 up osd.30 -28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB 63.09 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB 63.01 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.92 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.92 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.92 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.92 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.38 -16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB 63.18 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.92 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.92 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.92 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.92 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.41 -29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB 69.77 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB 61.74 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.90 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.90 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.90 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.90 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.00 176 up osd.39 -19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB 77.81 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.08 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.22 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.19 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.15 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.14 201 up osd.29 TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384 target_size_ratio 0.85 application cephfs removed_snaps [2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1] pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "cephfs_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "rep_3_ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -6, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
Tanks, Lars _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 12/3/19 1:30 PM, Lars Täuber wrote:
here it comes:
$ ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - root default -25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - room PRZ -26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 1.00 - row rechts -27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB 67.51 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB 62.80 0.95 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.94 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.94 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.94 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.95 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.01 176 up osd.37 4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB 3.01 0.05 110 up osd.4 -13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB 62.64 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.94 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.94 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.94 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.01 176 up osd.40 33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB 3.02 0.05 110 up osd.33 -22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB 77.10 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.17 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.18 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.14 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.15 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.24 215 up osd.30 36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB 2.92 0.04 110 up osd.36 -28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB 62.36 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB 62.27 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.93 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.93 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.93 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.93 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.38 31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB 3.04 0.05 110 up osd.31 -16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB 62.44 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.94 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.94 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.94 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.93 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.01 176 up osd.41 34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB 3.04 0.05 109 up osd.34 -29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB 68.95 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB 61.02 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.91 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.91 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.91 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.91 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.01 176 up osd.39 32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB 2.88 0.04 109 up osd.32 -19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB 76.89 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.10 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.23 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.21 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.16 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.16 201 up osd.29 35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB 2.89 0.04 110 up osd.35 TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB 66.45 MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - root default -25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - room PRZ -26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 1.00 - row rechts -27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB 68.31 1.02 - rack 1-eins -3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB 63.54 0.94 - host onode1 0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB 62.48 0.93 163 up osd.0 1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB 62.47 0.93 163 up osd.1 2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB 62.53 0.93 163 up osd.2 3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB 62.90 0.94 164 up osd.3 37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB 67.32 1.00 176 up osd.37 -13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB 63.38 0.94 - host onode4 13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.13 14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB 62.49 0.93 163 up osd.14 15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB 62.43 0.93 163 up osd.15 16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB 62.13 0.92 162 up osd.16 40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB 67.36 1.00 176 up osd.40 -22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB 78.02 1.16 - host onode7 25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB 77.59 1.15 203 up osd.25 26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB 78.40 1.17 205 up osd.26 27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB 75.80 1.13 198 up osd.27 28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB 76.13 1.13 199 up osd.28 30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB 82.18 1.22 215 up osd.30 -28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB 63.09 0.94 - rack 2-zwei -7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB 63.01 0.94 - host onode2 5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB 62.08 0.92 162 up osd.5 6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB 62.13 0.92 162 up osd.6 7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB 61.77 0.92 161 up osd.7 8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB 61.75 0.92 161 up osd.8 38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.38 -16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB 63.18 0.94 - host onode5 17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB 62.15 0.92 162 up osd.17 18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB 62.16 0.92 162 up osd.18 19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB 62.14 0.92 162 up osd.19 20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB 62.12 0.92 162 up osd.20 41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB 67.31 1.00 176 up osd.41 -29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB 69.77 1.04 - rack 3-drei -10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB 61.74 0.92 - host onode3 9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB 60.63 0.90 158 up osd.9 10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB 60.19 0.90 157 up osd.10 11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB 60.27 0.90 157 up osd.11 12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB 60.28 0.90 157 up osd.12 39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB 67.34 1.00 176 up osd.39 -19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB 77.81 1.16 - host onode6 21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB 72.79 1.08 190 up osd.21 22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB 81.79 1.22 214 up osd.22 23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB 80.29 1.19 210 up osd.23 24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB 77.31 1.15 202 up osd.24 29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB 76.86 1.14 201 up osd.29 TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB 67.24 MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor 0/0/2366 flags hashpspool,ec_overwrites,selfmanaged_snaps max_bytes 119457034600410 stripe_width 16384 target_size_ratio 0.85 application cephfs removed_snaps [2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1] pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "cephfs_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "rep_3_ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -6, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
Lars, you have too much PG's for this OSD's. I suggest to disable PG autoscaler and: - reduce number of PG's for cephfs_metada pool to something like 16 PG's. - reduce number of PG's for cephfs_data to something like 512. - update crush rule for cephfs_metadata pool - set domain to 'rack' instead 'host'. Also please paste your `ceph osd tree`. k
Hi Konstantin, thanks for your suggestions.
Lars, you have too much PG's for this OSD's. I suggest to disable PG autoscaler and:
- reduce number of PG's for cephfs_metada pool to something like 16 PG's.
Done.
- reduce number of PG's for cephfs_data to something like 512.
Done.
- update crush rule for cephfs_metadata pool - set domain to 'rack' instead 'host'.
Done.
Also please paste your `ceph osd tree`.
$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 195.40730 root default -25 195.40730 room PRZ -26 195.40730 row rechts -27 83.74599 rack 1-eins -3 27.91533 host onode1 0 hdd 5.51459 osd.0 up 1.00000 1.00000 1 hdd 5.51459 osd.1 up 1.00000 1.00000 2 hdd 5.51459 osd.2 up 1.00000 1.00000 3 hdd 5.51459 osd.3 up 1.00000 1.00000 37 hdd 5.51459 osd.37 up 1.00000 1.00000 4 ssd 0.34239 osd.4 up 1.00000 1.00000 -13 27.91533 host onode4 13 hdd 5.51459 osd.13 up 1.00000 1.00000 14 hdd 5.51459 osd.14 up 1.00000 1.00000 15 hdd 5.51459 osd.15 up 1.00000 1.00000 16 hdd 5.51459 osd.16 up 1.00000 1.00000 40 hdd 5.51459 osd.40 up 1.00000 1.00000 33 ssd 0.34239 osd.33 up 1.00000 1.00000 -22 27.91533 host onode7 25 hdd 5.51459 osd.25 up 1.00000 1.00000 26 hdd 5.51459 osd.26 up 1.00000 1.00000 27 hdd 5.51459 osd.27 up 1.00000 1.00000 28 hdd 5.51459 osd.28 up 1.00000 1.00000 30 hdd 5.51459 osd.30 up 1.00000 1.00000 36 ssd 0.34239 osd.36 up 1.00000 1.00000 -28 55.83066 rack 2-zwei -7 27.91533 host onode2 5 hdd 5.51459 osd.5 up 1.00000 1.00000 6 hdd 5.51459 osd.6 up 1.00000 1.00000 7 hdd 5.51459 osd.7 up 1.00000 1.00000 8 hdd 5.51459 osd.8 up 1.00000 1.00000 38 hdd 5.51459 osd.38 up 1.00000 1.00000 31 ssd 0.34239 osd.31 up 1.00000 1.00000 -16 27.91533 host onode5 17 hdd 5.51459 osd.17 up 1.00000 1.00000 18 hdd 5.51459 osd.18 up 1.00000 1.00000 19 hdd 5.51459 osd.19 up 1.00000 1.00000 20 hdd 5.51459 osd.20 up 1.00000 1.00000 41 hdd 5.51459 osd.41 up 1.00000 1.00000 34 ssd 0.34239 osd.34 up 1.00000 1.00000 -29 55.83066 rack 3-drei -10 27.91533 host onode3 9 hdd 5.51459 osd.9 up 1.00000 1.00000 10 hdd 5.51459 osd.10 up 1.00000 1.00000 11 hdd 5.51459 osd.11 up 1.00000 1.00000 12 hdd 5.51459 osd.12 up 1.00000 1.00000 39 hdd 5.51459 osd.39 up 1.00000 1.00000 32 ssd 0.34239 osd.32 up 1.00000 1.00000 -19 27.91533 host onode6 21 hdd 5.51459 osd.21 up 1.00000 1.00000 22 hdd 5.51459 osd.22 up 1.00000 1.00000 23 hdd 5.51459 osd.23 up 1.00000 1.00000 24 hdd 5.51459 osd.24 up 1.00000 1.00000 29 hdd 5.51459 osd.29 up 1.00000 1.00000 35 ssd 0.34239 osd.35 up 1.00000 1.00000 So I just wait for the remapping and merging being done and see what happens. Thanks so far! Best regards, Lars
On 12/4/19 4:04 PM, Lars Täuber wrote:
So I just wait for the remapping and merging being done and see what happens. Thanks so far!
Also don't forget to call `ceph osd crush weight-set rm-compat`. And stop mgr balancer `ceph balancer off`. After your rebalance is complete you can try: `ceph osd getmap -o om ; osdmaptool om --upmap upmap.sh --upmap-pool=cephfs_data --upmap-deviation=0` k
Hi Konstantin, the number PG's for the metadata pool now is 16. The number PG's for the data pool now is 512. I removed the backward-compatible weight-set. But now: $ ceph osd getmap -o om ; osdmaptool om --upmap upmap.sh --upmap-pool=cephfs_data --upmap-deviation=0 got osdmap epoch 94013 osdmaptool: osdmap file 'om' writing upmap command output to: upmap.sh checking for upmap cleanups upmap, max-count 100, max deviation 0 limiting to pools cephfs_data (1) no upmaps proposed and $ ceph osd df class hdd […] MIN/MAX VAR: 0.80/1.20 STDDEV: 6.45 There seems to be a bug in nautilus. I think about increasing the number of PG's for the data pool again, because the average number of PG's per OSD now is 76.8. What do you say? Thanks, Lars Wed, 4 Dec 2019 16:21:33 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de>, ceph-users@ceph.io :
On 12/4/19 4:04 PM, Lars Täuber wrote:
So I just wait for the remapping and merging being done and see what happens. Thanks so far!
Also don't forget to call `ceph osd crush weight-set rm-compat`.
And stop mgr balancer `ceph balancer off`.
After your rebalance is complete you can try:
`ceph osd getmap -o om ; osdmaptool om --upmap upmap.sh --upmap-pool=cephfs_data --upmap-deviation=0`
k
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de
On 12/16/19 2:42 PM, Lars Täuber wrote:
There seems to be a bug in nautilus.
I think about increasing the number of PG's for the data pool again, because the average number of PG's per OSD now is 76.8. What do you say?
May be bug in Nautilus, may be in osdmaptool. Please, upload your binary crush map (`osd getmap`). k
Mon, 16 Dec 2019 15:17:37 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de> :
On 12/16/19 2:42 PM, Lars Täuber wrote:
There seems to be a bug in nautilus.
I think about increasing the number of PG's for the data pool again, because the average number of PG's per OSD now is 76.8. What do you say?
May be bug in Nautilus, may be in osdmaptool.
Please, upload your binary crush map (`osd getmap`).
Here it comes. Thanks, Lars
On 12/16/19 3:25 PM, Lars Täuber wrote:
Here it comes.
Maybe some bug in osdmaptool, when defined pools is less than one no actually do_upmap is executed. Try like this: `osdmaptool osdmap.om --upmap upmap.sh --upmap-pool=cephfs_data --upmap-pool=cephfs_metadata --upmap-deviation=0 --upmap-max=1000` In my env actually upmaps purposed only for pool_id 1. CC'ed David. k
Mon, 16 Dec 2019 15:38:30 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de> :
On 12/16/19 3:25 PM, Lars Täuber wrote:
Here it comes.
Maybe some bug in osdmaptool, when defined pools is less than one no actually do_upmap is executed.
Try like this:
`osdmaptool osdmap.om --upmap upmap.sh --upmap-pool=cephfs_data --upmap-pool=cephfs_metadata --upmap-deviation=0 --upmap-max=1000`
In my env actually upmaps purposed only for pool_id 1.
Oh yes. There are 240 lines in the upmap.sh. I started the balancer again in the meanwhile. I'll watch the cluster and report. Thanks! Lars
CC'ed David.
k
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de
Hi Konstantin, the cluster has finished it's backfilling. I got this situation: $ ceph osd df class hdd […] MIN/MAX VAR: 0.86/1.08 STDDEV: 2.05 Now I created a new upmap.sh and sourced it. The cluster is busy again with 3% of its objects. I'll report the result. Thanks for all your hints. Regards, Lars Mon, 16 Dec 2019 15:38:30 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de> :
On 12/16/19 3:25 PM, Lars Täuber wrote:
Here it comes.
Maybe some bug in osdmaptool, when defined pools is less than one no actually do_upmap is executed.
Try like this:
`osdmaptool osdmap.om --upmap upmap.sh --upmap-pool=cephfs_data --upmap-pool=cephfs_metadata --upmap-deviation=0 --upmap-max=1000`
In my env actually upmaps purposed only for pool_id 1.
CC'ed David.
k
Hi Konstantin, the situation after moving the PGs with osdmaptool is not really better than without: $ ceph osd df class hdd […] MIN/MAX VAR: 0.86/1.08 STDDEV: 2.04 The OSD with the fewest PGs has 66 of them, the one with the most has 83. Is this the expected result? I'm unsure how much unusable space this means but I'm sure there is a relevant amount of it. Thanks for all your patience and support Lars Tue, 17 Dec 2019 07:45:24 +0100 Lars Täuber <taeuber@bbaw.de> ==> Konstantin Shalygin <k0ste@k0ste.ru> :
Hi Konstantin,
the cluster has finished it's backfilling. I got this situation:
$ ceph osd df class hdd […] MIN/MAX VAR: 0.86/1.08 STDDEV: 2.05
Now I created a new upmap.sh and sourced it. The cluster is busy again with 3% of its objects. I'll report the result.
Thanks for all your hints.
Regards, Lars
Mon, 16 Dec 2019 15:38:30 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de> :
On 12/16/19 3:25 PM, Lars Täuber wrote:
Here it comes.
Maybe some bug in osdmaptool, when defined pools is less than one no actually do_upmap is executed.
Try like this:
`osdmaptool osdmap.om --upmap upmap.sh --upmap-pool=cephfs_data --upmap-pool=cephfs_metadata --upmap-deviation=0 --upmap-max=1000`
In my env actually upmaps purposed only for pool_id 1.
CC'ed David.
k
Hello, is it already possible or planned to enable RadosGW to have a storage class like AWS does with "one zone" where the objects only exists in one zone and don't get mirrored to any other zone? eg. by creating the placement pool only in one zone? If this feature isn't planned is it worth creating a feature request? What do you think about that feature? All the best, Florian
On 12/18/19 2:16 PM, Lars Täuber wrote:
the situation after moving the PGs with osdmaptool is not really better than without:
$ ceph osd df class hdd […] MIN/MAX VAR: 0.86/1.08 STDDEV: 2.04
The OSD with the fewest PGs has 66 of them, the one with the most has 83.
Is this the expected result? I'm unsure how much unusable space this means but I'm sure there is a relevant amount of it.
You lower your STDDEV from 7.47 to 2.04. You should run script until "no upmap's purposed". k
Hi Konstantin, Mon, 23 Dec 2019 13:47:55 +0700 Konstantin Shalygin <k0ste@k0ste.ru> ==> Lars Täuber <taeuber@bbaw.de> :
On 12/18/19 2:16 PM, Lars Täuber wrote:
the situation after moving the PGs with osdmaptool is not really better than without:
$ ceph osd df class hdd […] MIN/MAX VAR: 0.86/1.08 STDDEV: 2.04
The OSD with the fewest PGs has 66 of them, the one with the most has 83.
Is this the expected result? I'm unsure how much unusable space this means but I'm sure there is a relevant amount of it.
You lower your STDDEV from 7.47 to 2.04. You should run script until "no upmap's purposed".
this is already the case. Can I do anything about it? Thanks, Lars
participants (6)
-
Florian Engelmann
-
Harald Staub
-
Konstantin Shalygin
-
Lars Täuber
-
Paul Emmerich
-
Thomas Schneider