22 Oct
2019
22 Oct
'19
9:52 p.m.
Hi,
in my 7 OSD node cluster I have the following disks:
Node 1
48x 1.6TB
Node 2
48x 1.6TB
Node 3
48x 1.6TB
Node 4
48x 1.6TB
Node 5
48x 7.2TB
Node 6
48x 7.2TB
Node 7
48x 7.2TB
The disk sizes are represented in CRUSH map accordingly.
For these disks only I created a pool "hdb_backup" with size 3.
Based on output of rados df this pool is using 247TB currently.
root@ld3955:~# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY
UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
cephfs_data 345 GiB 99092 0 297276
0 0 0 1638 811 MiB 109235 365 GiB 0 B 0 B
cephfs_metadata 102 MiB 48 0 144
0 0 0 8 8 KiB 8588 106 MiB 0 B 0 B
hdb_backup 247 TiB 64671398 0 194014194
0 0 0 12902005 4.3 TiB 323647757 601 TiB 0 B 0 B
hdd 2.4 TiB 635457 0 1270914
0 0 0 13237278 321 GiB 21526953 3.0 TiB 0 B 0 B
nvme 0 B 0 0 0
0 0 0 0 0 B 0 0 B 0 B 0 B
ssd 251 GiB 64307 0 128614
0 0 0 615475 29 GiB 885085 55 GiB 0 B 0 B
total_objects 65470302
total_used 747 TiB
total_avail 784 TiB
total_space 1.5 PiB
In order to rebalance the data in this pool I have configured balance
mode upmap:
root@ld3955:~# ceph balancer status
{
"active": true,
"plans": [],
"mode": "upmap"
}
Unfortunately the data distribution is not balanced at all on 1.6TB
disks, means the range is between 53.37% and 83.04%.
root@ld3955:~# ceph osd df | awk '{ print "osd."$1, "size: "$5,
"usage:
" $17, "reweight: "$4 }' | sort -nk5
osd.ID size: SIZE usage: reweight: REWEIGHT
osd.MIN/MAX size: 26.45 usage: reweight: STDDEV:
osd.TOTAL size: TiB usage: reweight: 747
osd.265 size: 1.6 usage: 53.37 reweight:
1.00000
osd.248 size: 1.6 usage: 53.41 reweight: 1.00000
osd.111 size: 1.6 usage: 53.43 reweight: 1.00000
osd.161 size: 1.6 usage: 53.46 reweight: 1.00000
osd.85 size: 1.6 usage: 53.46 reweight: 1.00000
osd.241 size: 1.6 usage: 53.49 reweight: 1.00000
osd.238 size: 1.6 usage: 53.51 reweight: 1.00000
osd.259 size: 1.6 usage: 53.56 reweight: 1.00000
osd.88 size: 1.6 usage: 53.57 reweight: 1.00000
osd.204 size: 1.6 usage: 53.58 reweight: 1.00000
osd.159 size: 1.6 usage: 55.16 reweight: 1.00000
osd.81 size: 1.6 usage: 55.16 reweight: 1.00000
osd.116 size: 1.6 usage: 55.20 reweight: 1.00000
osd.195 size: 1.6 usage: 55.25 reweight: 1.00000
osd.169 size: 1.6 usage: 55.33 reweight: 1.00000
osd.158 size: 1.6 usage: 55.34 reweight: 1.00000
[...]
osd.146 size: 1.6 usage: 79.31 reweight: 1.00000
osd.140 size: 1.6 usage: 79.34 reweight: 0.89999
osd.262 size: 1.6 usage: 79.38 reweight: 0.89999
osd.217 size: 1.6 usage: 79.48 reweight: 1.00000
osd.83 size: 1.6 usage: 79.50 reweight: 1.00000
osd.239 size: 1.6 usage: 79.52 reweight: 0.79999
osd.190 size: 1.6 usage: 80.87 reweight: 1.00000
osd.97 size: 1.6 usage: 80.95 reweight: 1.00000
osd.216 size: 1.6 usage: 80.97 reweight: 1.00000
osd.160 size: 1.6 usage: 81.03 reweight: 1.00000
osd.145 size: 1.6 usage: 81.19 reweight: 1.00000
osd.137 size: 1.6 usage: 81.20 reweight: 0.89999
osd.136 size: 1.6 usage: 81.21 reweight: 0.89999
osd.54 size: 1.6 usage: 82.88 reweight: 1.00000
osd.252 size: 1.6 usage: 83.04 reweight: 0.89999
Question:
Why is the data distribution on the 1.6TB disks unequal?
How can I correct this?
THX
23 Oct
23 Oct
12:25 p.m.
On 10/22/19 7:52 PM, Thomas wrote:
Node 1
48x 1.6TB
Node 2
48x 1.6TB
Node 3
48x 1.6TB
Node 4
48x 1.6TB
Node 5
48x 7.2TB
Node 6
48x 7.2TB
Node 7
48x 7.2TB
I suggest to balance disks in hosts, e.g. ~ 28x1.6TB + 20x7.2TB per host.
Why is the data distribution on the 1.6TB disks
unequal?
How can I correct this?
Balancer in upmap mode works with pools. I guess some of
your 1.6TB
OSD's not serve some pools.
k
1:59 p.m.
I agree wrt making the nodes weights uniform.
When mixing drive sizes, be careful that the larger ones don’t run afoul of the pg max —
they will receive more pgs than the smaller ones, and if you lose a node that might be
enough to send some over the max. ‘ceph OSD df’ and look at the PG counts.
This can also degrade performance since IO is not spread uniformly. Primary affinity
hops can mitigate somewhat.
> On Oct 22, 2019, at 8:26 PM, Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
>
> On 10/22/19 7:52 PM, Thomas wrote:
>> Node 1
>> 48x 1.6TB
>> Node 2
>> 48x 1.6TB
>> Node 3
>> 48x 1.6TB
>> Node 4
>> 48x 1.6TB
>> Node 5
>> 48x 7.2TB
>> Node 6
>> 48x 7.2TB
>> Node 7
>> 48x 7.2TB
>
> I suggest to balance disks in hosts, e.g. ~ 28x1.6TB + 20x7.2TB per host.
>
>> Why is the data distribution on the 1.6TB disks unequal?
>> How can I correct this?
> Balancer in upmap mode works with pools. I guess some of your 1.6TB OSD's not
serve some pools.
>
>
>
> k
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:14 p.m.
The number of PGs on the 7.2TB disks is 120 in avg., and the number of
PGs on the 1.6TB disks is 35 in avg.
This means a difference by factor 3-4.
However I don't understand why this should explain the unbalanced data
distribution on the 1.6TB disks only (the 7.2 TB disks are balanced)
And all the disks are defined to serve the same pool only by a suitable
Crush Map configuration. This means any other pool is served by
different disks.
Here's an example for one node, all other 6 nodes are similar:
host ld5505-hdd_strgbox { id -16 # do not change unnecessarily id -18
class hdd # do not change unnecessarily id -20 class nvme # do not
change unnecessarily id -49 class ssd # do not change unnecessarily #
weight 78.720 alg straw2 hash 0 # rjenkins1 item osd.76 weight 1.640
item osd.77 weight 1.640 item osd.78 weight 1.640 [...] item osd.97
weight 1.640 item osd.102 weight 1.640 item osd.110 weight 1.640 }
In addition I don't understand why distributing the disks equally over
all nodes should solve the issue?
My understanding is that Ceph's algorithm should be smart enough to
determine which object should be placed where and ensure balanced
utilisation.
I agree that I have a major impact if a node with 7.2TB disks go down,
though.
Am 23.10.2019 um 06:59 schrieb Anthony D'Atri:
I agree wrt making the nodes weights uniform.
When mixing drive sizes, be careful that the larger ones don’t run afoul of the pg max —
they will receive more pgs than the smaller ones, and if you lose a node that might be
enough to send some over the max. ‘ceph OSD df’ and look at the PG counts.
This can also degrade performance since IO is not spread uniformly. Primary affinity
hops can mitigate somewhat.
> On Oct 22, 2019, at 8:26 PM, Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
>
> On 10/22/19 7:52 PM, Thomas wrote:
>> Node 1
>> 48x 1.6TB
>> Node 2
>> 48x 1.6TB
>> Node 3
>> 48x 1.6TB
>> Node 4
>> 48x 1.6TB
>> Node 5
>> 48x 7.2TB
>> Node 6
>> 48x 7.2TB
>> Node 7
>> 48x 7.2TB
> I suggest to balance disks in hosts, e.g. ~ 28x1.6TB + 20x7.2TB per host.
>
>> Why is the data distribution on the 1.6TB disks unequal?
>> How can I correct this?
> Balancer in upmap mode works with pools. I guess some of your 1.6TB OSD's not
serve some pools.
>
>
>
> k
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:31 p.m.
On 10/23/19 1:14 PM, Thomas Schneider wrote:
My understanding is that Ceph's algorithm should
be smart enough to
determine which object should be placed where and ensure balanced
utilisation.
I agree that I have a major impact if a node with 7.2TB disks go down,
though.
Ceph is don't care about disk utilization, Ceph is don't care (mostly)
what is your OSD.
This is basic for usage of generic purpose hardware. Please pastebin your
`ceph osd tree`, `ceph osd df tree` & `ceph osd pool ls detail`.
k
4:46 p.m.
Sure, here's the pastebin <https://pastebin.com/isqbm88b>.
Am 23.10.2019 um 08:31 schrieb Konstantin Shalygin:
On 10/23/19 1:14 PM, Thomas Schneider wrote:
My understanding is that Ceph's algorithm
should be smart enough to
determine which object should be placed where and ensure balanced
utilisation.
I agree that I have a major impact if a node with 7.2TB disks go down,
though.
Ceph is don't care about disk utilization, Ceph is don't care (mostly)
what is your OSD.
This is basic for usage of generic purpose hardware. Please pastebin your
`ceph osd tree`, `ceph osd df tree` & `ceph osd pool ls detail`.
k
4:50 p.m.
On 10/23/19 2:46 PM, Thomas Schneider wrote:
Sure, here's the
pastebin<https://pastebin.com/isqbm88b>8b>.
Since you have several rules, please also provide`ceph osd crush rule dump`.
k
24 Oct
24 Oct
12:58 p.m.
On 10/23/19 2:46 PM, Thomas Schneider wrote:
Sure, here's the
pastebin<https://pastebin.com/isqbm88b>8b>.
Some of your 1.6Tb OSD's is reweighted, like osd.89 is 0.89999, osd.100
is 0.79999, etc...
By this reason this OSD's get less PG's then other.
k
8:54 p.m.
Hello,
this is understood.
I needed to start reweighting specific OSD because rebalancing was not
working and I got a warning in Ceph that some OSDs are running out of space.
KR
Am 24.10.2019 um 05:58 schrieb Konstantin Shalygin:
On 10/23/19 2:46 PM, Thomas Schneider wrote:
Sure, here's the
pastebin<https://pastebin.com/isqbm88b>8b>.
Some of your 1.6Tb OSD's is reweighted, like osd.89 is 0.89999,
osd.100 is 0.79999, etc...
By this reason this OSD's get less PG's then other.
k
25 Oct
25 Oct
2:06 p.m.
On 10/24/19 6:54 PM, Thomas Schneider wrote:
this is understood.
I needed to start reweighting specific OSD because rebalancing was not
working and I got a warning in Ceph that some OSDs are running out of space.
Still, the main your issue is that your buckets is uneven, 350TB vs
79TB, more that 4 times.
I suggest to you disable multiroot (use only default), use your 1.6Tb
drives from current default root (I count ~48 1.6Tb OSD's).
And mix your OSD's in hosts to be more evenly distributed in cluster -
this is one of basic Ceph best practices.
Also you can try to use offline upmap method, some folks get better
results with this (don't forget to disable balancer):
`ceph osd getmap -o om; osdmaptool om --upmap upmap.sh --upmap-deviation
0; bash upmap.sh; rm -f upmap.sh om`
k
1638
days inactive
1641
days old
10 comments
4 participants
participants (4)
-
Anthony D'Atri -
Konstantin Shalygin -
Thomas -
Thomas Schneider