24 Feb
2020
24 Feb
'20
7:01 p.m.
Hi,
we currently creating a new cluster. This cluster is (as far as we can
tell) an config-copy (ansible) of our existing cluster, just 5 years later
- with new hardware (nvme instead of ssd, bigger disks, ...)
The setup:
* NVMe for Journals and "Cache"-Pool
* HDD with NVMe Journals for "Data"-Pool
* Cache-Pool as writeback-Tier on Data-Pool
* We are using 12.2.13 without bluestore.
If we run an rados benchmark against this pool, everything seems fine, but
as soon as we start a fio-benchmark
-<-
[global]
ioengine=rbd
clientname=cinder
pool=cinder
rbdname=fio_test
rw=write
bs=4M
[rbd_iodepth32]
iodepth=32
->-
after some seconds the bandwidth drops to <15 MB/s and our hdd-disks are
doing more IOs than our Journal-Disks.
We also unconfigured the caching completely, but the issue remains.
The output of "ceph osd pool stats" shows ~100 op/s, but our disks are
doing:
-<-
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 0.00 0.00 278.50 0.00 34.07 250.51
0.14 0.50 0.00 0.50 0.03 0.80
nvme1n1 0.00 0.00 0.00 64.00 0.00 7.77 248.50
0.01 0.22 0.00 0.22 0.03 0.20
sda 0.00 1.50 0.00 557.00 0.00 29.49 108.45
180.57 160.59 0.00 160.59 1.80 100.00
sdb 0.00 42.00 0.00 592.00 0.00 28.21 97.60
176.51 1105.79 0.00 1105.79 1.69 100.00
sdc 0.00 14.50 0.00 528.50 0.00 27.95 108.31
183.02 179.47 0.00 179.47 1.89 100.00
sde 0.00 134.50 0.00 223.50 0.00 14.05 128.72
17.38 60.05 0.00 60.05 0.89 20.00
sdg 0.00 76.00 0.00 492.00 0.00 26.32 109.54
191.81 1474.96 0.00 1474.96 2.03 100.00
sdf 0.00 0.00 0.00 491.50 0.00 26.76 111.49
176.55 326.05 0.00 326.05 2.03 100.00
sdh 0.00 0.00 0.00 548.50 0.00 26.71 99.75
204.39 327.57 0.00 327.57 1.82 100.00
sdi 0.00 112.00 0.00 526.00 0.00 23.15 90.14
158.32 1325.61 0.00 1325.61 1.90 100.00
sdj 0.00 12.00 0.00 641.00 0.00 34.78 111.13
185.51 278.29 0.00 278.29 1.56 100.00
sdk 0.00 23.50 0.00 399.50 0.00 20.38 104.46
166.77 461.67 0.00 461.67 2.50 100.00
sdl 0.00 267.00 0.00 498.50 0.00 34.46 141.58
200.37 490.80 0.00 490.80 2.01 100.00
->-
Any hints how to debug the issue?
Thanks a lot,
Fabian
25 Feb
25 Feb
9:32 a.m.
Hi Fabian,
El 24/2/20 a las 19:01, Fabian Zimmermann escribió:
we currently creating a new cluster. This cluster is
(as far as we can
tell) an config-copy (ansible) of our existing cluster, just 5 years later
- with new hardware (nvme instead of ssd, bigger disks, ...)
The setup:
* NVMe for Journals and "Cache"-Pool
* HDD with NVMe Journals for "Data"-Pool
* Cache-Pool as writeback-Tier on Data-Pool
* We are using 12.2.13 without bluestore.
What model of disks?
If we run an rados benchmark against this pool,
everything seems fine, but
as soon as we start a fio-benchmark
-<-
[global]
ioengine=rbd
clientname=cinder
pool=cinder
rbdname=fio_test
rw=write
bs=4M
[rbd_iodepth32]
iodepth=32
->-
after some seconds the bandwidth drops to <15 MB/s and our hdd-disks are
doing more IOs than our Journal-Disks.
We also unconfigured the caching completely, but the issue remains.
What is the
peak performance before that <15MB/s?
Journal is a "write buffer", so after few secs has to ensure data goes
to its final destination in HDD AFAIK. Have you keep the fio test
running enough time to see if it regains performance?
The output of "ceph osd pool stats" shows ~100 op/s, but our disks are
doing:
-<-
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 0.00 0.00 278.50 0.00 34.07 250.51
0.14 0.50 0.00 0.50 0.03 0.80
nvme1n1 0.00 0.00 0.00 64.00 0.00 7.77 248.50
0.01 0.22 0.00 0.22 0.03 0.20
sda 0.00 1.50 0.00 557.00 0.00 29.49 108.45
180.57 160.59 0.00 160.59 1.80 100.00
sdb 0.00 42.00 0.00 592.00 0.00 28.21 97.60
176.51 1105.79 0.00 1105.79 1.69 100.00
sdc 0.00 14.50 0.00 528.50 0.00 27.95 108.31
183.02 179.47 0.00 179.47 1.89 100.00
sde 0.00 134.50 0.00 223.50 0.00 14.05 128.72
17.38 60.05 0.00 60.05 0.89 20.00
sdg 0.00 76.00 0.00 492.00 0.00 26.32 109.54
191.81 1474.96 0.00 1474.96 2.03 100.00
sdf 0.00 0.00 0.00 491.50 0.00 26.76 111.49
176.55 326.05 0.00 326.05 2.03 100.00
sdh 0.00 0.00 0.00 548.50 0.00 26.71 99.75
204.39 327.57 0.00 327.57 1.82 100.00
sdi 0.00 112.00 0.00 526.00 0.00 23.15 90.14
158.32 1325.61 0.00 1325.61 1.90 100.00
sdj 0.00 12.00 0.00 641.00 0.00 34.78 111.13
185.51 278.29 0.00 278.29 1.56 100.00
sdk 0.00 23.50 0.00 399.50 0.00 20.38 104.46
166.77 461.67 0.00 461.67 2.50 100.00
sdl 0.00 267.00 0.00 498.50 0.00 34.46 141.58
200.37 490.80 0.00 490.80 2.01 100.00
->-
Any hints how to debug the issue?
Not an expert to that regard, but have you considered trying bluestore?
You should get more consistent performance...
Cheers
Eneko
--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943569206
Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
www.binovo.es
4:43 p.m.
Fabian said:
The output of "ceph osd pool stats" shows
~100 op/s, but our disks are doing:
What does the iostat output look like on the old cluster?
Thanks,
Mark
On Mon, Feb 24, 2020 at 11:02 AM Fabian Zimmermann <dev.faz(a)gmail.com> wrote:
>
> Hi,
>
> we currently creating a new cluster. This cluster is (as far as we can
> tell) an config-copy (ansible) of our existing cluster, just 5 years later
> - with new hardware (nvme instead of ssd, bigger disks, ...)
>
> The setup:
>
> * NVMe for Journals and "Cache"-Pool
> * HDD with NVMe Journals for "Data"-Pool
> * Cache-Pool as writeback-Tier on Data-Pool
> * We are using 12.2.13 without bluestore.
>
> If we run an rados benchmark against this pool, everything seems fine, but
> as soon as we start a fio-benchmark
>
> -<-
> [global]
> ioengine=rbd
> clientname=cinder
> pool=cinder
> rbdname=fio_test
> rw=write
> bs=4M
>
> [rbd_iodepth32]
> iodepth=32
> ->-
>
> after some seconds the bandwidth drops to <15 MB/s and our hdd-disks are
> doing more IOs than our Journal-Disks.
> We also unconfigured the caching completely, but the issue remains.
>
> The output of "ceph osd pool stats" shows ~100 op/s, but our disks are
> doing:
> -<-
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> nvme0n1 0.00 0.00 0.00 278.50 0.00 34.07 250.51
> 0.14 0.50 0.00 0.50 0.03 0.80
> nvme1n1 0.00 0.00 0.00 64.00 0.00 7.77 248.50
> 0.01 0.22 0.00 0.22 0.03 0.20
> sda 0.00 1.50 0.00 557.00 0.00 29.49 108.45
> 180.57 160.59 0.00 160.59 1.80 100.00
> sdb 0.00 42.00 0.00 592.00 0.00 28.21 97.60
> 176.51 1105.79 0.00 1105.79 1.69 100.00
> sdc 0.00 14.50 0.00 528.50 0.00 27.95 108.31
> 183.02 179.47 0.00 179.47 1.89 100.00
> sde 0.00 134.50 0.00 223.50 0.00 14.05 128.72
> 17.38 60.05 0.00 60.05 0.89 20.00
> sdg 0.00 76.00 0.00 492.00 0.00 26.32 109.54
> 191.81 1474.96 0.00 1474.96 2.03 100.00
> sdf 0.00 0.00 0.00 491.50 0.00 26.76 111.49
> 176.55 326.05 0.00 326.05 2.03 100.00
> sdh 0.00 0.00 0.00 548.50 0.00 26.71 99.75
> 204.39 327.57 0.00 327.57 1.82 100.00
> sdi 0.00 112.00 0.00 526.00 0.00 23.15 90.14
> 158.32 1325.61 0.00 1325.61 1.90 100.00
> sdj 0.00 12.00 0.00 641.00 0.00 34.78 111.13
> 185.51 278.29 0.00 278.29 1.56 100.00
> sdk 0.00 23.50 0.00 399.50 0.00 20.38 104.46
> 166.77 461.67 0.00 461.67 2.50 100.00
> sdl 0.00 267.00 0.00 498.50 0.00 34.46 141.58
> 200.37 490.80 0.00 490.80 2.01 100.00
> ->-
>
> Any hints how to debug the issue?
>
> Thanks a lot,
>
> Fabian
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1531
days inactive
1532
days old
2 comments
3 participants
participants (3)
-
Eneko Lacunza -
Fabian Zimmermann -
Mark Lehrer