Hi, in my 7 OSD node cluster I have the following disks: Node 1 48x 1.6TB Node 2 48x 1.6TB Node 3 48x 1.6TB Node 4 48x 1.6TB Node 5 48x 7.2TB Node 6 48x 7.2TB Node 7 48x 7.2TB The disk sizes are represented in CRUSH map accordingly. For these disks only I created a pool "hdb_backup" with size 3. Based on output of rados df this pool is using 247TB currently. root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR cephfs_data 345 GiB 99092 0 297276 0 0 0 1638 811 MiB 109235 365 GiB 0 B 0 B cephfs_metadata 102 MiB 48 0 144 0 0 0 8 8 KiB 8588 106 MiB 0 B 0 B hdb_backup 247 TiB 64671398 0 194014194 0 0 0 12902005 4.3 TiB 323647757 601 TiB 0 B 0 B hdd 2.4 TiB 635457 0 1270914 0 0 0 13237278 321 GiB 21526953 3.0 TiB 0 B 0 B nvme 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B ssd 251 GiB 64307 0 128614 0 0 0 615475 29 GiB 885085 55 GiB 0 B 0 B total_objects 65470302 total_used 747 TiB total_avail 784 TiB total_space 1.5 PiB In order to rebalance the data in this pool I have configured balance mode upmap: root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Unfortunately the data distribution is not balanced at all on 1.6TB disks, means the range is between 53.37% and 83.04%. root@ld3955:~# ceph osd df | awk '{ print "osd."$1, "size: "$5, "usage: " $17, "reweight: "$4 }' | sort -nk5 osd.ID size: SIZE usage: reweight: REWEIGHT osd.MIN/MAX size: 26.45 usage: reweight: STDDEV: osd.TOTAL size: TiB usage: reweight: 747 osd.265 size: 1.6 usage: 53.37 reweight: 1.00000 osd.248 size: 1.6 usage: 53.41 reweight: 1.00000 osd.111 size: 1.6 usage: 53.43 reweight: 1.00000 osd.161 size: 1.6 usage: 53.46 reweight: 1.00000 osd.85 size: 1.6 usage: 53.46 reweight: 1.00000 osd.241 size: 1.6 usage: 53.49 reweight: 1.00000 osd.238 size: 1.6 usage: 53.51 reweight: 1.00000 osd.259 size: 1.6 usage: 53.56 reweight: 1.00000 osd.88 size: 1.6 usage: 53.57 reweight: 1.00000 osd.204 size: 1.6 usage: 53.58 reweight: 1.00000 osd.159 size: 1.6 usage: 55.16 reweight: 1.00000 osd.81 size: 1.6 usage: 55.16 reweight: 1.00000 osd.116 size: 1.6 usage: 55.20 reweight: 1.00000 osd.195 size: 1.6 usage: 55.25 reweight: 1.00000 osd.169 size: 1.6 usage: 55.33 reweight: 1.00000 osd.158 size: 1.6 usage: 55.34 reweight: 1.00000 [...] osd.146 size: 1.6 usage: 79.31 reweight: 1.00000 osd.140 size: 1.6 usage: 79.34 reweight: 0.89999 osd.262 size: 1.6 usage: 79.38 reweight: 0.89999 osd.217 size: 1.6 usage: 79.48 reweight: 1.00000 osd.83 size: 1.6 usage: 79.50 reweight: 1.00000 osd.239 size: 1.6 usage: 79.52 reweight: 0.79999 osd.190 size: 1.6 usage: 80.87 reweight: 1.00000 osd.97 size: 1.6 usage: 80.95 reweight: 1.00000 osd.216 size: 1.6 usage: 80.97 reweight: 1.00000 osd.160 size: 1.6 usage: 81.03 reweight: 1.00000 osd.145 size: 1.6 usage: 81.19 reweight: 1.00000 osd.137 size: 1.6 usage: 81.20 reweight: 0.89999 osd.136 size: 1.6 usage: 81.21 reweight: 0.89999 osd.54 size: 1.6 usage: 82.88 reweight: 1.00000 osd.252 size: 1.6 usage: 83.04 reweight: 0.89999 Question: Why is the data distribution on the 1.6TB disks unequal? How can I correct this? THX
On 10/22/19 7:52 PM, Thomas wrote:
Node 1 48x 1.6TB Node 2 48x 1.6TB Node 3 48x 1.6TB Node 4 48x 1.6TB Node 5 48x 7.2TB Node 6 48x 7.2TB Node 7 48x 7.2TB
I suggest to balance disks in hosts, e.g. ~ 28x1.6TB + 20x7.2TB per host.
Why is the data distribution on the 1.6TB disks unequal? How can I correct this? Balancer in upmap mode works with pools. I guess some of your 1.6TB OSD's not serve some pools.
k
I agree wrt making the nodes weights uniform. When mixing drive sizes, be careful that the larger ones don’t run afoul of the pg max — they will receive more pgs than the smaller ones, and if you lose a node that might be enough to send some over the max. ‘ceph OSD df’ and look at the PG counts. This can also degrade performance since IO is not spread uniformly. Primary affinity hops can mitigate somewhat.
On Oct 22, 2019, at 8:26 PM, Konstantin Shalygin <k0ste@k0ste.ru> wrote:
On 10/22/19 7:52 PM, Thomas wrote:
Node 1 48x 1.6TB Node 2 48x 1.6TB Node 3 48x 1.6TB Node 4 48x 1.6TB Node 5 48x 7.2TB Node 6 48x 7.2TB Node 7 48x 7.2TB
I suggest to balance disks in hosts, e.g. ~ 28x1.6TB + 20x7.2TB per host.
Why is the data distribution on the 1.6TB disks unequal? How can I correct this? Balancer in upmap mode works with pools. I guess some of your 1.6TB OSD's not serve some pools.
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The number of PGs on the 7.2TB disks is 120 in avg., and the number of PGs on the 1.6TB disks is 35 in avg. This means a difference by factor 3-4. However I don't understand why this should explain the unbalanced data distribution on the 1.6TB disks only (the 7.2 TB disks are balanced) And all the disks are defined to serve the same pool only by a suitable Crush Map configuration. This means any other pool is served by different disks. Here's an example for one node, all other 6 nodes are similar: host ld5505-hdd_strgbox { id -16 # do not change unnecessarily id -18 class hdd # do not change unnecessarily id -20 class nvme # do not change unnecessarily id -49 class ssd # do not change unnecessarily # weight 78.720 alg straw2 hash 0 # rjenkins1 item osd.76 weight 1.640 item osd.77 weight 1.640 item osd.78 weight 1.640 [...] item osd.97 weight 1.640 item osd.102 weight 1.640 item osd.110 weight 1.640 } In addition I don't understand why distributing the disks equally over all nodes should solve the issue? My understanding is that Ceph's algorithm should be smart enough to determine which object should be placed where and ensure balanced utilisation. I agree that I have a major impact if a node with 7.2TB disks go down, though. Am 23.10.2019 um 06:59 schrieb Anthony D'Atri:
I agree wrt making the nodes weights uniform.
When mixing drive sizes, be careful that the larger ones don’t run afoul of the pg max — they will receive more pgs than the smaller ones, and if you lose a node that might be enough to send some over the max. ‘ceph OSD df’ and look at the PG counts.
This can also degrade performance since IO is not spread uniformly. Primary affinity hops can mitigate somewhat.
On Oct 22, 2019, at 8:26 PM, Konstantin Shalygin <k0ste@k0ste.ru> wrote:
On 10/22/19 7:52 PM, Thomas wrote:
Node 1 48x 1.6TB Node 2 48x 1.6TB Node 3 48x 1.6TB Node 4 48x 1.6TB Node 5 48x 7.2TB Node 6 48x 7.2TB Node 7 48x 7.2TB I suggest to balance disks in hosts, e.g. ~ 28x1.6TB + 20x7.2TB per host.
Why is the data distribution on the 1.6TB disks unequal? How can I correct this? Balancer in upmap mode works with pools. I guess some of your 1.6TB OSD's not serve some pools.
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 10/23/19 1:14 PM, Thomas Schneider wrote:
My understanding is that Ceph's algorithm should be smart enough to determine which object should be placed where and ensure balanced utilisation. I agree that I have a major impact if a node with 7.2TB disks go down, though.
Ceph is don't care about disk utilization, Ceph is don't care (mostly) what is your OSD. This is basic for usage of generic purpose hardware. Please pastebin your `ceph osd tree`, `ceph osd df tree` & `ceph osd pool ls detail`. k
Sure, here's the pastebin <https://pastebin.com/isqbm88b>. Am 23.10.2019 um 08:31 schrieb Konstantin Shalygin:
On 10/23/19 1:14 PM, Thomas Schneider wrote:
My understanding is that Ceph's algorithm should be smart enough to determine which object should be placed where and ensure balanced utilisation. I agree that I have a major impact if a node with 7.2TB disks go down, though.
Ceph is don't care about disk utilization, Ceph is don't care (mostly) what is your OSD.
This is basic for usage of generic purpose hardware. Please pastebin your
`ceph osd tree`, `ceph osd df tree` & `ceph osd pool ls detail`.
k
On 10/23/19 2:46 PM, Thomas Schneider wrote:
Sure, here's the pastebin<https://pastebin.com/isqbm88b>.
Since you have several rules, please also provide`ceph osd crush rule dump`. k
OK. Here's my new pastebin <https://pastebin.com/rF9g2ztC>. Am 23.10.2019 um 09:50 schrieb Konstantin Shalygin:
ceph osd crush rule dump
On 10/23/19 2:46 PM, Thomas Schneider wrote:
Sure, here's the pastebin<https://pastebin.com/isqbm88b>.
Some of your 1.6Tb OSD's is reweighted, like osd.89 is 0.89999, osd.100 is 0.79999, etc... By this reason this OSD's get less PG's then other. k
Hello, this is understood. I needed to start reweighting specific OSD because rebalancing was not working and I got a warning in Ceph that some OSDs are running out of space. KR Am 24.10.2019 um 05:58 schrieb Konstantin Shalygin:
On 10/23/19 2:46 PM, Thomas Schneider wrote:
Sure, here's the pastebin<https://pastebin.com/isqbm88b>.
Some of your 1.6Tb OSD's is reweighted, like osd.89 is 0.89999, osd.100 is 0.79999, etc...
By this reason this OSD's get less PG's then other.
k
On 10/24/19 6:54 PM, Thomas Schneider wrote:
this is understood.
I needed to start reweighting specific OSD because rebalancing was not working and I got a warning in Ceph that some OSDs are running out of space.
Still, the main your issue is that your buckets is uneven, 350TB vs 79TB, more that 4 times. I suggest to you disable multiroot (use only default), use your 1.6Tb drives from current default root (I count ~48 1.6Tb OSD's). And mix your OSD's in hosts to be more evenly distributed in cluster - this is one of basic Ceph best practices. Also you can try to use offline upmap method, some folks get better results with this (don't forget to disable balancer): `ceph osd getmap -o om; osdmaptool om --upmap upmap.sh --upmap-deviation 0; bash upmap.sh; rm -f upmap.sh om` k
participants (4)
-
Anthony D'Atri
-
Konstantin Shalygin
-
Thomas
-
Thomas Schneider