24 Oct
2019
24 Oct
'19
5:22 p.m.
I'm in the process of testing the iscsi target feature of ceph. The cluster
is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts with 12
SSD OSDs per host. Some basic testing moving VMs to a ceph backed datastore
is only showing 60MB/s transfers. However moving these back off the
datastore is fast at 200-300MB/s.
What should I be looking at to track down the write performance issue? In
comparison with the Nimble Storage arrays I can see 200-300MB/s in both
directions.
Thanks,
Ryan
24 Oct
24 Oct
5:26 p.m.
Are you using Erasure Coding or replication? What is your crush rule?
What SSDs and CPUs? Does each OSD use 100% of a core or more when
writing?
On Thu, Oct 24, 2019 at 1:22 PM Ryan <rswagoner(a)gmail.com> wrote:
>
> I'm in the process of testing the iscsi target feature of ceph. The cluster is
running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts with 12 SSD OSDs per host.
Some basic testing moving VMs to a ceph backed datastore is only showing 60MB/s transfers.
However moving these back off the datastore is fast at 200-300MB/s.
>
> What should I be looking at to track down the write performance issue? In comparison
with the Nimble Storage arrays I can see 200-300MB/s in both directions.
>
> Thanks,
> Ryan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:37 p.m.
I was told by someone at Red Hat that ISCSI performance is still several magnitudes behind
using the client / driver.
Thanks,
-Drew
-----Original Message-----
From: Nathan Fish <lordcirth(a)gmail.com>
Sent: Thursday, October 24, 2019 1:27 PM
To: Ryan <rswagoner(a)gmail.com>
Cc: ceph-users <ceph-users(a)ceph.com>
Subject: [ceph-users] Re: iSCSI write performance
Are you using Erasure Coding or replication? What is your crush rule?
What SSDs and CPUs? Does each OSD use 100% of a core or more when writing?
On Thu, Oct 24, 2019 at 1:22 PM Ryan <rswagoner(a)gmail.com> wrote:
I'm in the process of testing the iscsi target feature of ceph. The cluster is
running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts with 12 SSD OSDs per host.
Some basic testing moving VMs to a ceph backed datastore is only showing 60MB/s transfers.
However moving these back off the datastore is fast at 200-300MB/s.
What should I be looking at to track down the write performance issue? In comparison with
the Nimble Storage arrays I can see 200-300MB/s in both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
6:33 p.m.
Hello,
we did some local testing a few days ago on a new installation of a small
cluster.
Performance of our iSCSI implementation showed a performance drop of 20-30%
against krbd.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Okt. 2019 um 19:37 Uhr schrieb Drew Weaver <
drew.weaver(a)thenap.com>gt;:
I was told by someone at Red Hat that ISCSI
performance is still several
magnitudes behind using the client / driver.
Thanks,
-Drew
-----Original Message-----
From: Nathan Fish <lordcirth(a)gmail.com>
Sent: Thursday, October 24, 2019 1:27 PM
To: Ryan <rswagoner(a)gmail.com>
Cc: ceph-users <ceph-users(a)ceph.com>
Subject: [ceph-users] Re: iSCSI write performance
Are you using Erasure Coding or replication? What is your crush rule?
What SSDs and CPUs? Does each OSD use 100% of a core or more when writing?
On Thu, Oct 24, 2019 at 1:22 PM Ryan <rswagoner(a)gmail.com> wrote:
I'm in the process of testing the iscsi target feature of ceph. The
cluster is
running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts
with 12 SSD OSDs per host. Some basic testing moving VMs to a ceph backed
datastore is only showing 60MB/s transfers. However moving these back off
the datastore is fast at 200-300MB/s.
What should I be looking at to track down the write performance issue?
In
comparison with the Nimble Storage arrays I can see 200-300MB/s in both
directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 Oct
25 Oct
4:22 a.m.
They are Samsung 860 EVO 2TB SSDs. The Dell R740xd servers have dual Intel
Gold 6130 CPUs and dual SAS controllers with 6 SSDs each. Top shows around
20-25% of a core being used by each OSD daemon. I am using erasure coding
with crush-failure-domain=host k=3 m=2.
On Thu, Oct 24, 2019 at 1:37 PM Drew Weaver <drew.weaver(a)thenap.com> wrote:
I was told by someone at Red Hat that ISCSI
performance is still several
magnitudes behind using the client / driver.
Thanks,
-Drew
-----Original Message-----
From: Nathan Fish <lordcirth(a)gmail.com>
Sent: Thursday, October 24, 2019 1:27 PM
To: Ryan <rswagoner(a)gmail.com>
Cc: ceph-users <ceph-users(a)ceph.com>
Subject: [ceph-users] Re: iSCSI write performance
Are you using Erasure Coding or replication? What is your crush rule?
What SSDs and CPUs? Does each OSD use 100% of a core or more when writing?
On Thu, Oct 24, 2019 at 1:22 PM Ryan <rswagoner(a)gmail.com> wrote:
I'm in the process of testing the iscsi target feature of ceph. The
cluster is
running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts
with 12 SSD OSDs per host. Some basic testing moving VMs to a ceph backed
datastore is only showing 60MB/s transfers. However moving these back off
the datastore is fast at 200-300MB/s.
What should I be looking at to track down the write performance issue?
In
comparison with the Nimble Storage arrays I can see 200-300MB/s in both
directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
24 Oct
24 Oct
11:15 p.m.
On 10/24/2019 12:22 PM, Ryan wrote:
I'm in the process of testing the iscsi target
feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic testing
moving VMs to a ceph
backed datastore is only showing 60MB/s transfers. However moving these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the throughput?
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
pass data between the kernel and tcmu-runner. The default is only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi session by
default. We increase that to 128, but you can increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue depths so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write performance issue?
In comparison with the Nimble Storage arrays I can see 200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 Oct
25 Oct
4:47 a.m.
I'm using CentOS 7.7.1908 with kernel 3.10.0-1062.1.2.el7.x86_64. The
workload was a VMware Storage Motion from a local SSD backed datastore to
the ceph backed datastore. Performance was measured using dstat on the
iscsi gateway for network traffic and ceph status as this cluster is
basically idle. I changed max_data_area_mb to 256 and cmdsn_depth to 128.
This appears to have given a slight improvement of maybe 10MB/s.
Moving VM to the ceph backed datastore
io:
client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
Moving VM off the ceph backed datastore
io:
client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
I'm going to test bonnie++ with an rbd volume mounted directly on the iscsi
gateway. Also will test bonnie++ inside a VM on a ceph backed datastore.
On Thu, Oct 24, 2019 at 7:15 PM Mike Christie <mchristi(a)redhat.com> wrote:
On 10/24/2019 12:22 PM, Ryan wrote:
I'm in the process of testing the iscsi
target feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving VMs to a ceph
backed datastore is only showing 60MB/s transfers. However moving these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the throughput?
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
pass data between the kernel and tcmu-runner. The default is only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi session by
default. We increase that to 128, but you can increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue depths so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write performance issue?
In comparison with the Nimble Storage arrays I can see 200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:28 a.m.
For vmotion speed, check "emulate_3pc" attribute on the LIO target. If 0
(default), VMWare will issue io in 64KB blocks which gives low speed. if
set to 1 this will trigger VMWare to use vaai extended copy, which
activates LIO's xcopy functionality which uses 512KB block sizes by
default. We also bumped the xcopy block size to 4M (rbd object size)
which gives around 400 MB/s vmotion speed, the same speed can also be
achieved via Veeam backups.
/Maged
On 25/10/2019 06:47, Ryan wrote:
I'm using CentOS 7.7.1908 with kernel
3.10.0-1062.1.2.el7.x86_64. The
workload was a VMware Storage Motion from a local SSD backed datastore
to the ceph backed datastore. Performance was measured using dstat on
the iscsi gateway for network traffic and ceph status as this cluster
is basically idle. I changed max_data_area_mb to 256 and cmdsn_depth
to 128. This appears to have given a slight improvement of maybe 10MB/s.
Moving VM to the ceph backed datastore
io:
client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
Moving VM off the ceph backed datastore
io:
client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
I'm going to test bonnie++ with an rbd volume mounted directly on the
iscsi gateway. Also will test bonnie++ inside a VM on a ceph backed
datastore.
On Thu, Oct 24, 2019 at 7:15 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
On 10/24/2019 12:22 PM, Ryan wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I'm in the process of testing the iscsi
target feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving VMs
to a ceph
backed datastore is only showing 60MB/s
transfers. However
moving these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the throughput?
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target
system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
pass data between the kernel and tcmu-runner. The default is only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi
session by
default. We increase that to 128, but you can increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue depths
so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write performance
issue?
In comparison with the Nimble Storage arrays I
can see
200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
8:46 a.m.
Actually this may not work if moving from a local datastore to Ceph. For
iSCSI xcopy, both the source and destination need to be accessible by
the target such as in moving vms across Ceph datastores. So in your
case, vmotion will be handled by VMWare data mover which uses 64K block
sizes.
On 25/10/2019 10:28, Maged Mokhtar wrote:
For vmotion speed, check "emulate_3pc" attribute on the LIO target. If
0 (default), VMWare will issue io in 64KB blocks which gives low
speed. if set to 1 this will trigger VMWare to use vaai extended
copy, which activates LIO's xcopy functionality which uses 512KB block
sizes by default. We also bumped the xcopy block size to 4M (rbd
object size) which gives around 400 MB/s vmotion speed, the same speed
can also be achieved via Veeam backups.
/Maged
On 25/10/2019 06:47, Ryan wrote:
I'm using CentOS 7.7.1908 with kernel
3.10.0-1062.1.2.el7.x86_64. The
workload was a VMware Storage Motion from a local SSD backed
datastore to the ceph backed datastore. Performance was measured
using dstat on the iscsi gateway for network traffic and ceph status
as this cluster is basically idle. I changed max_data_area_mb to 256
and cmdsn_depth to 128. This appears to have given a slight
improvement of maybe 10MB/s.
Moving VM to the ceph backed datastore
io:
client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
Moving VM off the ceph backed datastore
io:
client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
I'm going to test bonnie++ with an rbd volume mounted directly on the
iscsi gateway. Also will test bonnie++ inside a VM on a ceph backed
datastore.
On Thu, Oct 24, 2019 at 7:15 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
On 10/24/2019 12:22 PM, Ryan wrote:
_______________________________________________
ceph-users mailing list --ceph-users(a)ceph.io
To unsubscribe send an email toceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io I'm in the process of testing the iscsi
target feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving VMs
to a ceph
backed datastore is only showing 60MB/s
transfers. However
moving these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the
throughput?
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target
system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu
uses to
pass data between the kernel and tcmu-runner. The default is only
8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi
session by
default. We increase that to 128, but you can increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue
depths so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write performance
issue?
In comparison with the Nimble Storage arrays I
can see
200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
2:31 p.m.
I'm not seeing the emulate_3pc setting under disks/rbd/diskname when
calling info. A google search shows that SUSE Enterprise Storage has it
available. I thought I had the latest packages, but maybe not. I'm using
tcmu-runner 1.5.2 and ceph-iscsi 3.3. Almost all of my VMs are currently on
Nimble iSCSI storage. I've actually tested from both and performance is the
same. Doing the math off the ceph status does show it using 64K blocks in
both cases.
Control Values
- hw_max_sectors .. 1024
- max_data_area_mb .. 256 (override)
- osd_op_timeout .. 30
- qfull_timeout .. 5
On Fri, Oct 25, 2019 at 4:46 AM Maged Mokhtar <mmokhtar(a)petasan.org> wrote:
Actually this may not work if moving from a local
datastore to Ceph. For
iSCSI xcopy, both the source and destination need to be accessible by the
target such as in moving vms across Ceph datastores. So in your case,
vmotion will be handled by VMWare data mover which uses 64K block sizes.
On 25/10/2019 10:28, Maged Mokhtar wrote:
For vmotion speed, check "emulate_3pc" attribute on the LIO target. If 0
(default), VMWare will issue io in 64KB blocks which gives low speed. if
set to 1 this will trigger VMWare to use vaai extended copy, which
activates LIO's xcopy functionality which uses 512KB block sizes by
default. We also bumped the xcopy block size to 4M (rbd object size) which
gives around 400 MB/s vmotion speed, the same speed can also be achieved
via Veeam backups.
/Maged
On 25/10/2019 06:47, Ryan wrote:
I'm using CentOS 7.7.1908 with kernel 3.10.0-1062.1.2.el7.x86_64. The
workload was a VMware Storage Motion from a local SSD backed datastore to
the ceph backed datastore. Performance was measured using dstat on the
iscsi gateway for network traffic and ceph status as this cluster is
basically idle. I changed max_data_area_mb to 256 and cmdsn_depth to 128.
This appears to have given a slight improvement of maybe 10MB/s.
Moving VM to the ceph backed datastore
io:
client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
Moving VM off the ceph backed datastore
io:
client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
I'm going to test bonnie++ with an rbd volume mounted directly on the
iscsi gateway. Also will test bonnie++ inside a VM on a ceph backed
datastore.
On Thu, Oct 24, 2019 at 7:15 PM Mike Christie <mchristi(a)redhat.com> wrote:
On 10/24/2019 12:22 PM, Ryan wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I'm in the process of testing the iscsi
target feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving VMs to a ceph
backed datastore is only showing 60MB/s transfers. However moving these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the throughput?
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
pass data between the kernel and tcmu-runner. The default is only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi session by
default. We increase that to 128, but you can increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue depths so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write performance issue?
In comparison with the Nimble Storage arrays I can see 200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:29 p.m.
On 10/25/2019 09:31 AM, Ryan wrote:
I'm not seeing the emulate_3pc setting under
disks/rbd/diskname when
emulate_3pc is only for kernel based backends. tcmu-runner always has
xcopy on.
calling info. A google search shows that SUSE
Enterprise Storage has it
available. I thought I had the latest packages, but maybe not. I'm using
tcmu-runner 1.5.2 and ceph-iscsi 3.3. Almost all of my VMs are currently
on Nimble iSCSI storage. I've actually tested from both and performance
is the same. Doing the math off the ceph status does show it using 64K
blocks in both cases.
Control Values
- hw_max_sectors .. 1024
- max_data_area_mb .. 256 (override)
- osd_op_timeout .. 30
- qfull_timeout .. 5
On Fri, Oct 25, 2019 at 4:46 AM Maged Mokhtar <mmokhtar(a)petasan.org
<mailto:mmokhtar@petasan.org>> wrote:
Actually this may not work if moving from a local datastore to Ceph.
For iSCSI xcopy, both the source and destination need to be
accessible by the target such as in moving vms across Ceph
datastores. So in your case, vmotion will be handled by VMWare data
mover which uses 64K block sizes.
On 25/10/2019 10:28, Maged Mokhtar wrote:
For vmotion speed, check "emulate_3pc" attribute on the LIO
target. If 0 (default), VMWare will issue io in 64KB blocks which
gives low speed. if set to 1 this will trigger VMWare to use vaai
extended copy, which activates LIO's xcopy functionality which
uses 512KB block sizes by default. We also bumped the xcopy block
size to 4M (rbd object size) which gives around 400 MB/s vmotion
speed, the same speed can also be achieved via Veeam backups.
/Maged
On 25/10/2019 06:47, Ryan wrote:
I'm using CentOS 7.7.1908 with kernel
3.10.0-1062.1.2.el7.x86_64.
The workload was a VMware Storage Motion from a local SSD backed
datastore to the ceph backed datastore. Performance was measured
using dstat on the iscsi gateway for network traffic and ceph
status as this cluster is basically idle. I changed
max_data_area_mb to 256 and cmdsn_depth to 128. This appears to
have given a slight improvement of maybe 10MB/s.
Moving VM to the ceph backed datastore
io:
client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
Moving VM off the ceph backed datastore
io:
client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
I'm going to test bonnie++ with an rbd volume mounted directly on
the iscsi gateway. Also will test bonnie++ inside a VM on a ceph
backed datastore.
On Thu, Oct 24, 2019 at 7:15 PM Mike Christie
<mchristi(a)redhat.com <mailto:mchristi@redhat.com>> wrote:
On 10/24/2019 12:22 PM, Ryan wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io> I'm in the process of testing the iscsi
target feature of
ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi
3.3. It
consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving
VMs to a ceph
backed datastore is only showing 60MB/s
transfers. However
moving these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the
throughput?
If you are using fio, what arguments are you using? And,
could you
change the ioengine to rbd and re-run the test from the
target system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s
for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer
lio/tcmu uses to
pass data between the kernel and tcmu-runner. The default is
only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi
session by
default. We increase that to 128, but you can increase this
to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue
depths so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write
performance issue?
In comparison with the Nimble Storage arrays I
can see
200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
8:25 p.m.
Can you point me to the directions for the kernel mode iscsi backend. I was
following these directions
https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/
Thanks,
Ryan
On Fri, Oct 25, 2019 at 11:29 AM Mike Christie <mchristi(a)redhat.com> wrote:
On 10/25/2019 09:31 AM, Ryan wrote:
>
_______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:
ceph-users(a)ceph.io>
I'm not seeing the emulate_3pc setting under
disks/rbd/diskname when
emulate_3pc is only for kernel based backends. tcmu-runner always has
xcopy on.
calling info. A google search shows that SUSE
Enterprise Storage has it
available. I thought I had the latest packages, but maybe not. I'm using
tcmu-runner 1.5.2 and ceph-iscsi 3.3. Almost all of my VMs are currently
on Nimble iSCSI storage. I've actually tested from both and performance
is the same. Doing the math off the ceph status does show it using 64K
blocks in both cases.
Control Values
- hw_max_sectors .. 1024
- max_data_area_mb .. 256 (override)
- osd_op_timeout .. 30
- qfull_timeout .. 5
On Fri, Oct 25, 2019 at 4:46 AM Maged Mokhtar <mmokhtar(a)petasan.org
<mailto:mmokhtar@petasan.org>> wrote:
Actually this may not work if moving from a local datastore to Ceph.
For iSCSI xcopy, both the source and destination need to be
accessible by the target such as in moving vms across Ceph
datastores. So in your case, vmotion will be handled by VMWare data
mover which uses 64K block sizes.
On 25/10/2019 10:28, Maged Mokhtar wrote:
>
> For vmotion speed, check "emulate_3pc" attribute on the LIO
> target. If 0 (default), VMWare will issue io in 64KB blocks which
> gives low speed. if set to 1 this will trigger VMWare to use vaai
> extended copy, which activates LIO's xcopy functionality which
> uses 512KB block sizes by default. We also bumped the xcopy block
> size to 4M (rbd object size) which gives around 400 MB/s vmotion
> speed, the same speed can also be achieved via Veeam backups.
>
> /Maged
>
> On 25/10/2019 06:47, Ryan wrote:
>> I'm using CentOS 7.7.1908 with kernel 3.10.0-1062.1.2.el7.x86_64.
>> The workload was a VMware Storage Motion from a local SSD backed
>> datastore to the ceph backed datastore. Performance was measured
>> using dstat on the iscsi gateway for network traffic and ceph
>> status as this cluster is basically idle. I changed
>> max_data_area_mb to 256 and cmdsn_depth to 128. This appears to
>> have given a slight improvement of maybe 10MB/s.
>>
>> Moving VM to the ceph backed datastore
>> io:
>> client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
>>
>> Moving VM off the ceph backed datastore
>> io:
>> client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
>>
>> I'm going to test bonnie++ with an rbd volume mounted directly on
>> the iscsi gateway. Also will test bonnie++ inside a VM on a ceph
>> backed datastore.
>>
>> On Thu, Oct 24, 2019 at 7:15 PM Mike Christie
>> <mchristi(a)redhat.com <mailto:mchristi@redhat.com>> wrote:
>>
>> On 10/24/2019 12:22 PM, Ryan wrote:
>> > I'm in the process of testing the iscsi target feature of
>> ceph. The
>> > cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It
>> consists of 5
>>
>> What kernel are you using?
>>
>> > hosts with 12 SSD OSDs per host. Some basic testing moving
>> VMs to a ceph
>> > backed datastore is only showing 60MB/s transfers. However
>> moving these
>> > back off the datastore is fast at 200-300MB/s.
>>
>> What is the workload and what are you using to measure the
>> throughput?
>>
>> If you are using fio, what arguments are you using? And,
>> could you
>> change the ioengine to rbd and re-run the test from the
>> target system so
>> we can check if rbd is slow or iscsi?
>>
>> For small IOs, 60 is about right.
>>
>> For 128-512K IOs you should be able to get around 300 MB/s
>> for writes
>> and 600 for reads.
>>
>> 1. Increase max_data_area_mb. This is a kernel buffer
>> lio/tcmu uses to
>> pass data between the kernel and tcmu-runner. The default is
>> only 8MB.
>>
>> In gwcli cd to your disk and do:
>>
>> # reconfigure max_data_area_mb %N
>>
>> where N is between 8 and 2048 MBs.
>>
>> 2. The Linux kernel target only allows 64 commands per iscsi
>> session by
>> default. We increase that to 128, but you can increase this
>> to 512.
>>
>> In gwcli cd to the target dir and do
>>
>> reconfigure cmdsn_depth 512
>>
>> 3. I think ceph-iscsi and lio work better with higher queue
>> depths so if
>> you are using fio you want higher numjobs and/or iodepths.
>>
>> >> > What
should I be looking at to track down the write
>> performance issue?
>> > In comparison with the Nimble Storage arrays I can see
>> 200-300MB/s in
>> > both directions.
>> >> >
Thanks,
>> > Ryan
>> >> >> >
_______________________________________________
>> > ceph-users mailing list -- ceph-users(a)ceph.io
>> <mailto:ceph-users@ceph.io>
>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> <mailto:ceph-users-leave@ceph.io>
>> >>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:
ceph-users(a)ceph.io>
>> To unsubscribe send an email to
ceph-users-leave(a)ceph.io <mailto:
ceph-users-leave(a)ceph.io>
> To unsubscribe send an email to
ceph-users-leave(a)ceph.io <mailto:
ceph-users-leave(a)ceph.io>
28 Oct
28 Oct
7:06 p.m.
On 10/25/2019 03:25 PM, Ryan wrote:
Can you point me to the directions for the kernel mode
iscsi backend. I
was following these directions
https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/
If you just wanted to use the krbd device /dev/rbd$N and export it with
iscsi from a single iscsi target, then you can use targetcli like
described here:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/ht…
Instead of /dev/sdb in the example you would just use /dev/rbd$N in the
section "BLOCK (Linux BLOCK devices)".
Maged and SUSE also have a special module but I am not sure where to get
their kernels and manuals. Check out suse and petasan's sites for that.
Thanks,
Ryan
On Fri, Oct 25, 2019 at 11:29 AM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
On 10/25/2019 09:31 AM, Ryan wrote:
I'm not seeing the emulate_3pc setting under
disks/rbd/diskname when
emulate_3pc is only for kernel based backends. tcmu-runner always has
xcopy on.
calling info. A google search shows that SUSE
Enterprise Storage
has it
available. I thought I had the latest packages,
but maybe not. I'm
using
tcmu-runner 1.5.2 and ceph-iscsi 3.3. Almost all
of my VMs are
currently
on Nimble iSCSI storage. I've actually tested
from both and
performance
is the same. Doing the math off the ceph status
does show it using 64K
blocks in both cases.
Control Values
- hw_max_sectors .. 1024
- max_data_area_mb .. 256 (override)
- osd_op_timeout .. 30
- qfull_timeout .. 5
On Fri, Oct 25, 2019 at 4:46 AM Maged Mokhtar
<mmokhtar(a)petasan.org
<mailto:mmokhtar@petasan.org>
<mailto:mmokhtar@petasan.org
<mailto:mmokhtar@petasan.org>>> wrote:
Actually this may not work if moving from a local datastore to
Ceph.
For iSCSI xcopy, both the source and
destination need to be
accessible by the target such as in moving vms across Ceph
datastores. So in your case, vmotion will be handled by VMWare
data
mover which uses 64K block sizes.
On 25/10/2019 10:28, Maged Mokhtar wrote:
>
> For vmotion speed, check "emulate_3pc" attribute on the LIO
> target. If 0 (default), VMWare will issue io in 64KB blocks which
> gives low speed. if set to 1 this will trigger VMWare to use
vaai
> extended copy, which activates LIO's
xcopy functionality which
> uses 512KB block sizes by default. We also bumped the xcopy block
> size to 4M (rbd object size) which gives around 400 MB/s vmotion
> speed, the same speed can also be achieved via Veeam backups.
>
> /Maged
>
> On 25/10/2019 06:47, Ryan wrote:
>> I'm using CentOS 7.7.1908 with kernel
3.10.0-1062.1.2.el7.x86_64.
>> The workload was a VMware Storage
Motion from a local SSD backed
>> datastore to the ceph backed datastore. Performance was measured
>> using dstat on the iscsi gateway for network traffic and ceph
>> status as this cluster is basically idle. I changed
>> max_data_area_mb to 256 and cmdsn_depth to 128. This appears to
>> have given a slight improvement of maybe 10MB/s.
>>
>> Moving VM to the ceph backed datastore
>> io:
>> client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k
op/s
wr
>>
>> Moving VM off the ceph backed datastore
>> io:
>> client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62
op/s
wr
>>
>> I'm going to test bonnie++ with an rbd volume mounted
directly
on
>> the iscsi gateway. Also will test
bonnie++ inside a VM on a ceph
>> backed datastore.
>>
>> On Thu, Oct 24, 2019 at 7:15 PM Mike Christie
>> <mchristi(a)redhat.com <mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>> wrote:
>>
>> On 10/24/2019 12:22 PM, Ryan wrote:
>> > I'm in the process of testing the iscsi target feature of
>> ceph. The
>> > cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It
>> consists of 5
>>
>> What kernel are you using?
>>
>> > hosts with 12 SSD OSDs per host. Some basic testing moving
>> VMs to a ceph
>> > backed datastore is only showing 60MB/s transfers. However
>> moving these
>> > back off the datastore is fast at 200-300MB/s.
>>
>> What is the workload and what are you using to measure the
>> throughput?
>>
>> If you are using fio, what arguments are you using? And,
>> could you
>> change the ioengine to rbd and re-run the test from the
>> target system so
>> we can check if rbd is slow or iscsi?
>>
>> For small IOs, 60 is about right.
>>
>> For 128-512K IOs you should be able to get around 300 MB/s
>> for writes
>> and 600 for reads.
>>
>> 1. Increase max_data_area_mb. This is a kernel buffer
>> lio/tcmu uses to
>> pass data between the kernel and tcmu-runner. The default is
>> only 8MB.
>>
>> In gwcli cd to your disk and do:
>>
>> # reconfigure max_data_area_mb %N
>>
>> where N is between 8 and 2048 MBs.
>>
>> 2. The Linux kernel target only allows 64 commands per iscsi
>> session by
>> default. We increase that to 128, but you can increase this
>> to 512.
>>
>> In gwcli cd to the target dir and do
>>
>> reconfigure cmdsn_depth 512
>>
>> 3. I think ceph-iscsi and lio work better with higher queue
>> depths so if
>> you are using fio you want higher numjobs and/or iodepths.
>>
>> >> > What
should I be looking at to track down the write
>> performance issue?
>> > In comparison with the Nimble Storage arrays I can see
>> 200-300MB/s in
>> > both directions.
>> >> > Thanks,
>> > Ryan
>> >> >> >
_______________________________________________
>> > ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
>> <mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
>> > To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
>>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
>> >>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
>> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
31 Oct
31 Oct
4:40 p.m.
On Mon, Oct 28, 2019 at 8:07 PM Mike Christie <mchristi(a)redhat.com> wrote:
On 10/25/2019 03:25 PM, Ryan wrote:
this did not work very well in the past (even without multi-pathing)
Paul
>
> Maged and SUSE also have a special module but I am not sure where to get
> their kernels and manuals. Check out suse and petasan's sites for that.
>
>
> > Thanks,
> > Ryan
> >
> >
> > On Fri, Oct 25, 2019 at 11:29 AM Mike Christie <mchristi(a)redhat.com
> > <mailto:mchristi@redhat.com>> wrote:
> >
> > On 10/25/2019 09:31 AM, Ryan wrote:
> > > I'm not seeing the emulate_3pc setting under disks/rbd/diskname
when
> >
> > emulate_3pc is only for kernel based backends. tcmu-runner always has
> > xcopy on.
> >
> > > calling info. A google search shows that SUSE Enterprise Storage
> > has it
> > > available. I thought I had the latest packages, but maybe not. I'm
> > using
> > > tcmu-runner 1.5.2 and ceph-iscsi 3.3. Almost all of my VMs are
> > currently
> > > on Nimble iSCSI storage. I've actually tested from both and
> > performance
> > > is the same. Doing the math off the ceph status does show it using 64K
> > > blocks in both cases.
> > >
> > > Control Values
> > > - hw_max_sectors .. 1024
> > > - max_data_area_mb .. 256 (override)
> > > - osd_op_timeout .. 30
> > > - qfull_timeout .. 5
> > >
> > > On Fri, Oct 25, 2019 at 4:46 AM Maged Mokhtar
> > <mmokhtar(a)petasan.org <mailto:mmokhtar@petasan.org>
> > > <mailto:mmokhtar@petasan.org
<mailto:mmokhtar@petasan.org>>> wrote:
> > >
> > > Actually this may not work if moving from a local datastore to
> > Ceph.
> > > For iSCSI xcopy, both the source and destination need to be
> > > accessible by the target such as in moving vms across Ceph
> > > datastores. So in your case, vmotion will be handled by VMWare
> > data
> > > mover which uses 64K block sizes.
> > >
> > > On 25/10/2019 10:28, Maged Mokhtar wrote:
> > >>
> > >> For vmotion speed, check "emulate_3pc" attribute on
the LIO
> > >> target. If 0 (default), VMWare will issue io in 64KB blocks
which
> > >> gives low speed. if set to 1 this will trigger VMWare to use
> > vaai
> > >> extended copy, which activates LIO's xcopy functionality
which
> > >> uses 512KB block sizes by default. We also bumped the xcopy
block
> > >> size to 4M (rbd object size) which gives around 400 MB/s
vmotion
> > >> speed, the same speed can also be achieved via Veeam backups.
> > >>
> > >> /Maged
> > >>
> > >> On 25/10/2019 06:47, Ryan wrote:
> > >>> I'm using CentOS 7.7.1908 with kernel
> > 3.10.0-1062.1.2.el7.x86_64.
> > >>> The workload was a VMware Storage Motion from a local SSD
backed
> > >>> datastore to the ceph backed datastore. Performance was
measured
> > >>> using dstat on the iscsi gateway for network traffic and
ceph
> > >>> status as this cluster is basically idle. I changed
> > >>> max_data_area_mb to 256 and cmdsn_depth to 128. This
appears to
> > >>> have given a slight improvement of maybe 10MB/s.
> > >>>
> > >>> Moving VM to the ceph backed datastore
> > >>> io:
> > >>> client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k
> > op/s wr
> > >>>
> > >>> Moving VM off the ceph backed datastore
> > >>> io:
> > >>> client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd,
62
> > op/s wr
> > >>>
> > >>> I'm going to test bonnie++ with an rbd volume mounted
> > directly on
> > >>> the iscsi gateway. Also will test bonnie++ inside a VM on a
ceph
> > >>> backed datastore.
> > >>>
> > >>> On Thu, Oct 24, 2019 at 7:15 PM Mike Christie
> > >>> <mchristi(a)redhat.com <mailto:mchristi@redhat.com>
> > <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>>
wrote:
> > >>>
> > >>> On 10/24/2019 12:22 PM, Ryan wrote:
> > >>> > I'm in the process of testing the iscsi target
feature of
> > >>> ceph. The
> > >>> > cluster is running ceph 14.2.4 and ceph-iscsi 3.3.
It
> > >>> consists of 5
> > >>>
> > >>> What kernel are you using?
> > >>>
> > >>> > hosts with 12 SSD OSDs per host. Some basic
testing moving
> > >>> VMs to a ceph
> > >>> > backed datastore is only showing 60MB/s transfers.
However
> > >>> moving these
> > >>> > back off the datastore is fast at 200-300MB/s.
> > >>>
> > >>> What is the workload and what are you using to measure
the
> > >>> throughput?
> > >>>
> > >>> If you are using fio, what arguments are you using?
And,
> > >>> could you
> > >>> change the ioengine to rbd and re-run the test from
the
> > >>> target system so
> > >>> we can check if rbd is slow or iscsi?
> > >>>
> > >>> For small IOs, 60 is about right.
> > >>>
> > >>> For 128-512K IOs you should be able to get around 300
MB/s
> > >>> for writes
> > >>> and 600 for reads.
> > >>>
> > >>> 1. Increase max_data_area_mb. This is a kernel buffer
> > >>> lio/tcmu uses to
> > >>> pass data between the kernel and tcmu-runner. The
default is
> > >>> only 8MB.
> > >>>
> > >>> In gwcli cd to your disk and do:
> > >>>
> > >>> # reconfigure max_data_area_mb %N
> > >>>
> > >>> where N is between 8 and 2048 MBs.
> > >>>
> > >>> 2. The Linux kernel target only allows 64 commands per
iscsi
> > >>> session by
> > >>> default. We increase that to 128, but you can increase
this
> > >>> to 512.
> > >>>
> > >>> In gwcli cd to the target dir and do
> > >>>
> > >>> reconfigure cmdsn_depth 512
> > >>>
> > >>> 3. I think ceph-iscsi and lio work better with higher
queue
> > >>> depths so if
> > >>> you are using fio you want higher numjobs and/or
iodepths.
> > >>>
> > >>> >
> > >>> > What should I be looking at to track down the
write
> > >>> performance issue?
> > >>> > In comparison with the Nimble Storage arrays I can
see
> > >>> 200-300MB/s in
> > >>> > both directions.
> > >>> >
> > >>> > Thanks,
> > >>> > Ryan
> > >>> >
> > >>> >
> > >>> > _______________________________________________
> > >>> > ceph-users mailing list -- ceph-users(a)ceph.io
> > <mailto:ceph-users@ceph.io>
> > >>> <mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
> > >>> > To unsubscribe send an email to
> > ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>
> > >>> <mailto:ceph-users-leave@ceph.io
> > <mailto:ceph-users-leave@ceph.io>>
> > >>> >
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> ceph-users mailing list -- ceph-users(a)ceph.io
> > <mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io
> > <mailto:ceph-users@ceph.io>>
> > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users-leave@ceph.io
> > <mailto:ceph-users-leave@ceph.io>>
> > >>
> > >> _______________________________________________
> > >> ceph-users mailing list -- ceph-users(a)ceph.io
> > <mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io
> > <mailto:ceph-users@ceph.io>>
> > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users-leave@ceph.io
> > <mailto:ceph-users-leave@ceph.io>>
> > >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
Can you point me to the directions for the kernel
mode iscsi backend. I
was following these directions
https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/
If you just wanted to use the krbd device /dev/rbd$N and export it with
iscsi from a single iscsi target, then you can use targetcli like
described here:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/ht…
Instead of /dev/sdb in the example you would just use /dev/rbd$N in the
section "BLOCK (Linux BLOCK devices)".
25 Oct
25 Oct
3:54 p.m.
On 10/24/2019 11:47 PM, Ryan wrote:
I'm using CentOS 7.7.1908 with kernel
3.10.0-1062.1.2.el7.x86_64. The
workload was a VMware Storage Motion from a local SSD backed datastore
Ignore my comments. I thought you were just doing fio like tests in the vm.
to the ceph backed datastore. Performance was measured
using dstat on
the iscsi gateway for network traffic and ceph status as this cluster is
basically idle. I changed max_data_area_mb to 256 and cmdsn_depth to
128. This appears to have given a slight improvement of maybe 10MB/s.
Moving VM to the ceph backed datastore
io:
client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
Moving VM off the ceph backed datastore
io:
client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
If you run esxtop while running your test what do you see for the number
of commands in the iscsi LUN's queue?
I'm going to test bonnie++ with an rbd volume
mounted directly on the
To try and isolate if its the iscsi or rbd, you need to run fio with the
librbd io engine. We know krbd is going to be the fastest. ceph-iscsi
uses librbd so it is a better baseline. If you are not familiar with fio
you can just do something like:
fio --group_reporting --ioengine=rbd --direct=1 --name=librbdtest
--numjobs=32 --bs=512k --iodepth=128 --size=10G --rw=write
--rbd=name_of_your_image -pool=name_of_pool
iscsi gateway. Also will test bonnie++ inside a VM on
a ceph backed
datastore.
On Thu, Oct 24, 2019 at 7:15 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
On 10/24/2019 12:22 PM, Ryan wrote:
I'm in the process of testing the iscsi
target feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving VMs to
a ceph
backed datastore is only showing 60MB/s
transfers. However moving
these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the throughput?
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
pass data between the kernel and tcmu-runner. The default is only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi session by
default. We increase that to 128, but you can increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue depths so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write performance issue?
In comparison with the Nimble Storage arrays I can see 200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
8:01 p.m.
esxtop is showing a queue length of 0
Storage motion to ceph
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV
QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd
KAVG/cmd GAVG/cmd QAVG/cmd
naa.6001405ec60d8b82342404d929fbbd03 - 128 - 32
0 25 0.25 1442.32 0.18 1440.50 0.00 89.78 21.32
0.01 21.34 0.01
Storage motion from ceph
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV
QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd
KAVG/cmd GAVG/cmd QAVG/cmd
naa.6001405ec60d8b82342404d929fbbd03 - 128 - 32
0 25 0.25 4065.38 4064.83 0.36 253.52 0.00 7.57
0.01 7.58 0.00
I tried using fio like you mentioned but it was hanging with
[r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS] and the ETA kept climbing. I ended up
using rbd bench on the ceph iscsi gateway. With a 64K write workload I'm
seeing 400MB/s transfers.
rbd create test --size 100G --image-feature layering
rbd map test
mkfs.ext4 /dev/rbd/rbd/test
mount /dev/rbd/rbd/test test
rbd create testec --size 100G --image-feature layering --data-pool rbd_ec
rbd map testec
mkfs.ext4 /dev/rbd/rbd/testec
mount /dev/rbd/rbd/testec testec
[root@ceph-iscsi1 mnt]# rbd bench --image test --io-size 64K --io-type
write --io-total 10G
bench type write io_size 65536 io_threads 16 bytes 10737418240 pattern
sequential
SEC OPS OPS/SEC BYTES/SEC
1 6368 6377.59 417961796.64
2 12928 6462.27 423511630.71
3 19296 6420.18 420752986.78
4 26320 6585.61 431594792.67
5 33296 6662.37 436624891.04
6 40128 6754.67 442673957.25
7 46784 6765.75 443400452.26
8 53280 6809.02 446236110.93
9 60032 6739.67 441691068.73
10 66784 6698.91 439019550.77
11 73616 6690.88 438493253.66
12 80016 6654.35 436099640.00
13 85712 6485.07 425005611.11
14 91088 6202.49 406486113.46
15 96896 6021.17 394603137.62
16 102368 5741.19 376254347.24
17 107568 5501.57 360550910.38
18 113728 5603.17 367209502.58
19 120144 5820.48 381451245.32
20 126496 5917.60 387816078.53
21 132768 6089.71 399095466.00
22 139040 6306.98 413334431.09
23 145104 6276.42 411331743.63
24 151440 6256.67 410036891.68
25 157808 6261.12 410328554.98
26 163456 6140.03 402392725.36
elapsed: 26 ops: 163840 ops/sec: 6271.36 bytes/sec: 410999626.38
[root@ceph-iscsi1 mnt]# rbd bench --image testec --io-size 64K --io-type
write --io-total 10G
bench type write io_size 65536 io_threads 16 bytes 10737418240 pattern
sequential
SEC OPS OPS/SEC BYTES/SEC
1 7392 7415.38 485974266.41
2 14464 7243.59 474715656.29
3 22000 7341.08 481104853.50
4 29408 7352.29 481839517.16
5 37296 7459.38 488857889.75
6 44864 7494.36 491150574.57
7 52848 7676.76 503104281.98
8 60784 7756.76 508347136.11
9 68608 7835.26 513491609.52
10 76784 7902.30 517885290.67
11 84544 7935.96 520091129.45
12 92432 7916.76 518832844.57
13 100064 7855.96 514848275.43
14 107040 7692.52 504136734.09
15 114320 7499.66 491497933.56
16 121744 7436.99 487390477.85
17 129664 7438.92 487517345.01
18 136704 7326.50 480149408.39
19 144960 7587.00 497221460.09
20 153264 7796.56 510955233.33
21 160832 7814.44 512126854.90
elapsed: 21 ops: 163840 ops/sec: 7659.97 bytes/sec: 502004079.43
On Fri, Oct 25, 2019 at 11:54 AM Mike Christie <mchristi(a)redhat.com> wrote:
On 10/24/2019 11:47 PM, Ryan wrote:
I'm using CentOS 7.7.1908 with kernel
3.10.0-1062.1.2.el7.x86_64. The
workload was a VMware Storage Motion from a local SSD backed datastore
Ignore my comments. I thought you were just doing fio like tests in the vm.
to the ceph backed datastore. Performance was
measured using dstat on
the iscsi gateway for network traffic and ceph status as this cluster is
basically idle. I changed max_data_area_mb to 256 and cmdsn_depth to
128. This appears to have given a slight improvement of maybe 10MB/s.
Moving VM to the ceph backed datastore
io:
client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
Moving VM off the ceph backed datastore
io:
client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
If you run esxtop while running your test what do you see for the number
of commands in the iscsi LUN's queue?
I'm going to test bonnie++ with an rbd volume
mounted directly on the
To try and isolate if its the iscsi or rbd, you need to run fio with the
librbd io engine. We know krbd is going to be the fastest. ceph-iscsi
uses librbd so it is a better baseline. If you are not familiar with fio
you can just do something like:
fio --group_reporting --ioengine=rbd --direct=1 --name=librbdtest
--numjobs=32 --bs=512k --iodepth=128 --size=10G --rw=write
--rbd=name_of_your_image -pool=name_of_pool
iscsi gateway. Also will test bonnie++ inside a
VM on a ceph backed
datastore.
On Thu, Oct 24, 2019 at 7:15 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
On 10/24/2019 12:22 PM, Ryan wrote:
throughput?
I'm in the process of testing the iscsi
target feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving VMs to
a ceph
backed datastore is only showing 60MB/s
transfers. However moving
these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target
system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses
to
pass data between the kernel and tcmu-runner.
The default is only
8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi session
by
default. We increase that to 128, but you can
increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue depths
so if
you are using fio you want higher numjobs
and/or iodepths.
>
> What should I be looking at to track down the write performance
issue?
> In comparison with the Nimble Storage
arrays I can see 200-300MB/s
in
both
directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
9:03 p.m.
Just to clarify, it is better to separate the different performance cases:
1- regular io performance ( iops / throughput ), this should be good.
2- vmotion within datastores managed by Ceph: this will be good, as
xcopy will be used.
3. vmotion between Ceph datastore and an external datastore..this will
be bad. This seems the case you are testing. It is bad because between 2
different storage systems (iqns are served on different targets), vaai
xcopy cannot be used and vmware does its own stuff. It moves data using
64k block size, which gives low performance...to add some flavor, it
does indeed use 32 threads, but unfortunately they use co-located
addresses which does not work well in Ceph as they are hitting the same
rbd object, which gets serialized due to pg locks, so you will not get
any palatalization. Your speed will mostly be determined by a serial
64k, so with 1 ms write latency for ssd cluster, you will get around 64
MB/s..it will be slightly higher as the extra threads have some low effect.
Note your esxtop does show 32 active ios under ACTV, the QUED of zero
does is not the queue depth, but rather the "queued" io the ESX would
suspend in case your active reaches the maximum by adapater ( 128 ).
This is just to clarify, if case 3 is not your primary concern than i
would forget about it and benchmark 1 and 2 if they are relevant. Else,
if 3 is important, i am not sure you can do much as it is happening
within vmware..maybe there could be a way to map the external iqn to be
served by the same target serving the Ceph iqn then there could be a
chance the xcopy could be activated..Mike would probably know if this
has any chance of working :)
/Maged
On 25/10/2019 22:01, Ryan wrote:
esxtop is showing a queue length of 0
Storage motion to ceph
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN
ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s
DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
naa.6001405ec60d8b82342404d929fbbd03 - 128 - 32 0
25 0.25 1442.32 0.18 1440.50 0.00 89.78 21.32 0.01
21.34 0.01
Storage motion from ceph
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN
ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s
DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
naa.6001405ec60d8b82342404d929fbbd03 - 128 - 32 0
25 0.25 4065.38 4064.83 0.36 253.52 0.00 7.57 0.01
7.58 0.00
I tried using fio like you mentioned but it was hanging with
[r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS] and the ETA kept climbing. I ended
up using rbd bench on the ceph iscsi gateway. With a 64K write
workload I'm seeing 400MB/s transfers.
rbd create test --size 100G --image-feature layering
rbd map test
mkfs.ext4 /dev/rbd/rbd/test
mount /dev/rbd/rbd/test test
rbd create testec --size 100G --image-feature layering --data-pool rbd_ec
rbd map testec
mkfs.ext4 /dev/rbd/rbd/testec
mount /dev/rbd/rbd/testec testec
[root@ceph-iscsi1 mnt]# rbd bench --image test --io-size 64K --io-type
write --io-total 10G
bench type write io_size 65536 io_threads 16 bytes 10737418240
pattern sequential
SEC OPS OPS/SEC BYTES/SEC
1 6368 6377.59 417961796.64
2 12928 6462.27 423511630.71
3 19296 6420.18 420752986.78
4 26320 6585.61 431594792.67
5 33296 6662.37 436624891.04
6 40128 6754.67 442673957.25
7 46784 6765.75 443400452.26
8 53280 6809.02 446236110.93
9 60032 6739.67 441691068.73
10 66784 6698.91 439019550.77
11 73616 6690.88 438493253.66
12 80016 6654.35 436099640.00
13 85712 6485.07 425005611.11
14 91088 6202.49 406486113.46
15 96896 6021.17 394603137.62
16 102368 5741.19 376254347.24
17 107568 5501.57 360550910.38
18 113728 5603.17 367209502.58
19 120144 5820.48 381451245.32
20 126496 5917.60 387816078.53
21 132768 6089.71 399095466.00
22 139040 6306.98 413334431.09
23 145104 6276.42 411331743.63
24 151440 6256.67 410036891.68
25 157808 6261.12 410328554.98
26 163456 6140.03 402392725.36
elapsed: 26 ops: 163840 ops/sec: 6271.36 bytes/sec: 410999626.38
[root@ceph-iscsi1 mnt]# rbd bench --image testec --io-size 64K
--io-type write --io-total 10G
bench type write io_size 65536 io_threads 16 bytes 10737418240
pattern sequential
SEC OPS OPS/SEC BYTES/SEC
1 7392 7415.38 485974266.41
2 14464 7243.59 474715656.29
3 22000 7341.08 481104853.50
4 29408 7352.29 481839517.16
5 37296 7459.38 488857889.75
6 44864 7494.36 491150574.57
7 52848 7676.76 503104281.98
8 60784 7756.76 508347136.11
9 68608 7835.26 513491609.52
10 76784 7902.30 517885290.67
11 84544 7935.96 520091129.45
12 92432 7916.76 518832844.57
13 100064 7855.96 514848275.43
14 107040 7692.52 504136734.09
15 114320 7499.66 491497933.56
16 121744 7436.99 487390477.85
17 129664 7438.92 487517345.01
18 136704 7326.50 480149408.39
19 144960 7587.00 497221460.09
20 153264 7796.56 510955233.33
21 160832 7814.44 512126854.90
elapsed: 21 ops: 163840 ops/sec: 7659.97 bytes/sec: 502004079.43
On Fri, Oct 25, 2019 at 11:54 AM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
On 10/24/2019 11:47 PM, Ryan wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I'm using CentOS 7.7.1908 with kernel
3.10.0-1062.1.2.el7.x86_64. The
workload was a VMware Storage Motion from a local
SSD backed
datastore
Ignore my comments. I thought you were just doing fio like tests
in the vm.
to the ceph backed datastore. Performance was
measured using
dstat on
the iscsi gateway for network traffic and ceph
status as this
cluster is
basically idle. I changed max_data_area_mb to
256 and
cmdsn_depth to
128. This appears to have given a slight
improvement of maybe
10MB/s.
Moving VM to the ceph backed datastore
io:
client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
Moving VM off the ceph backed datastore
io:
client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
If you run esxtop while running your test what do you see for the
number
of commands in the iscsi LUN's queue?
I'm going to test bonnie++ with an rbd volume
mounted directly
on the
To try and isolate if its the iscsi or rbd, you need to run fio
with the
librbd io engine. We know krbd is going to be the fastest. ceph-iscsi
uses librbd so it is a better baseline. If you are not familiar
with fio
you can just do something like:
fio --group_reporting --ioengine=rbd --direct=1 --name=librbdtest
--numjobs=32 --bs=512k --iodepth=128 --size=10G --rw=write
--rbd=name_of_your_image -pool=name_of_pool
iscsi gateway. Also will test bonnie++ inside a
VM on a ceph backed
datastore.
On Thu, Oct 24, 2019 at 7:15 PM Mike Christie
<mchristi(a)redhat.com
<mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com
<mailto:mchristi@redhat.com>>> wrote:
On 10/24/2019 12:22 PM, Ryan wrote:
> I'm in the process of testing the iscsi target feature of
ceph.
The
> cluster is running ceph 14.2.4 and
ceph-iscsi 3.3. It
consists of 5
What kernel are you using?
> hosts with 12 SSD OSDs per host. Some basic testing moving
VMs to
a ceph
> backed datastore is only showing 60MB/s transfers. However
moving
these
throughput?
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the
If you are using fio, what arguments are you using? And,
could you
change the ioengine to rbd and re-run the
test from the
target system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s
for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer
lio/tcmu uses to
pass data between the kernel and tcmu-runner.
The default is
only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
where N is between 8 and 2048 MBs.
2. The Linux kernel target only allows 64 commands per iscsi
session by
default. We increase that to 128, but you can
increase this
to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
3. I think ceph-iscsi and lio work better with higher queue
depths so if
you are using fio you want higher numjobs
and/or iodepths.
>
> What should I be looking at to track down the write
performance
issue?
> In comparison with the Nimble Storage
arrays I can see
200-300MB/s in
> both directions.
>
> Thanks,
> Ryan
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
<mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
31 Oct
31 Oct
4:45 p.m.
On Fri, Oct 25, 2019 at 11:14 PM Maged Mokhtar <mmokhtar(a)petasan.org> wrote:
3. vmotion between Ceph datastore and an external
datastore..this will be bad. This seems the case you are testing. It is bad because
between 2 different storage systems (iqns are served on different targets), vaai xcopy
cannot be used and vmware does its own stuff. It moves data using 64k block size, which
gives low performance...to add some flavor, it does indeed use 32 threads, but
unfortunately they use co-located addresses which does not work well in Ceph as they are
hitting the same rbd object, which gets serialized due to pg locks, so you will not get
any palatalization. Your speed will mostly be determined by a serial 64k, so with 1 ms
write latency for ssd cluster, you will get around 64 MB/s..it will be slightly higher as
the extra threads have some low effect.
Yes, vmotion is the worst IO pattern ever for a sequential copy.
However, the situation you are describing can be fixed with RBD
striping v2, just make Ceph switch to another object every 64kb, see
https://docs.ceph.com/docs/master/dev/file-striping/
I'm not sure about the state of striping v2 support in the kernel
module, last time I checked it wasn't supported. But
ceph-iscsi/tcmu-runner got quite good over the past year, I don't see
any point in still using the kernel data path for iscsi nowadays.
Paul
>
> Note your esxtop does show 32 active ios under ACTV, the QUED of zero does is not the
queue depth, but rather the "queued" io the ESX would suspend in case your
active reaches the maximum by adapater ( 128 ).
>
> This is just to clarify, if case 3 is not your primary concern than i would forget
about it and benchmark 1 and 2 if they are relevant. Else, if 3 is important, i am not
sure you can do much as it is happening within vmware..maybe there could be a way to map
the external iqn to be served by the same target serving the Ceph iqn then there could be
a chance the xcopy could be activated..Mike would probably know if this has any chance of
working :)
>
> /Maged
>
>
> On 25/10/2019 22:01, Ryan wrote:
>
> esxtop is showing a queue length of 0
>
> Storage motion to ceph
> DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD
LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
> naa.6001405ec60d8b82342404d929fbbd03 - 128 - 32 0 25
0.25 1442.32 0.18 1440.50 0.00 89.78 21.32 0.01 21.34 0.01
>
> Storage motion from ceph
> DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD
LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
> naa.6001405ec60d8b82342404d929fbbd03 - 128 - 32 0 25
0.25 4065.38 4064.83 0.36 253.52 0.00 7.57 0.01 7.58 0.00
>
> I tried using fio like you mentioned but it was hanging with
[r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS] and the ETA kept climbing. I ended up using rbd bench on
the ceph iscsi gateway. With a 64K write workload I'm seeing 400MB/s transfers.
>
> rbd create test --size 100G --image-feature layering
> rbd map test
> mkfs.ext4 /dev/rbd/rbd/test
> mount /dev/rbd/rbd/test test
>
> rbd create testec --size 100G --image-feature layering --data-pool rbd_ec
> rbd map testec
> mkfs.ext4 /dev/rbd/rbd/testec
> mount /dev/rbd/rbd/testec testec
>
> [root@ceph-iscsi1 mnt]# rbd bench --image test --io-size 64K --io-type write
--io-total 10G
> bench type write io_size 65536 io_threads 16 bytes 10737418240 pattern sequential
> SEC OPS OPS/SEC BYTES/SEC
> 1 6368 6377.59 417961796.64
> 2 12928 6462.27 423511630.71
> 3 19296 6420.18 420752986.78
> 4 26320 6585.61 431594792.67
> 5 33296 6662.37 436624891.04
> 6 40128 6754.67 442673957.25
> 7 46784 6765.75 443400452.26
> 8 53280 6809.02 446236110.93
> 9 60032 6739.67 441691068.73
> 10 66784 6698.91 439019550.77
> 11 73616 6690.88 438493253.66
> 12 80016 6654.35 436099640.00
> 13 85712 6485.07 425005611.11
> 14 91088 6202.49 406486113.46
> 15 96896 6021.17 394603137.62
> 16 102368 5741.19 376254347.24
> 17 107568 5501.57 360550910.38
> 18 113728 5603.17 367209502.58
> 19 120144 5820.48 381451245.32
> 20 126496 5917.60 387816078.53
> 21 132768 6089.71 399095466.00
> 22 139040 6306.98 413334431.09
> 23 145104 6276.42 411331743.63
> 24 151440 6256.67 410036891.68
> 25 157808 6261.12 410328554.98
> 26 163456 6140.03 402392725.36
> elapsed: 26 ops: 163840 ops/sec: 6271.36 bytes/sec: 410999626.38
>
> [root@ceph-iscsi1 mnt]# rbd bench --image testec --io-size 64K --io-type write
--io-total 10G
> bench type write io_size 65536 io_threads 16 bytes 10737418240 pattern sequential
> SEC OPS OPS/SEC BYTES/SEC
> 1 7392 7415.38 485974266.41
> 2 14464 7243.59 474715656.29
> 3 22000 7341.08 481104853.50
> 4 29408 7352.29 481839517.16
> 5 37296 7459.38 488857889.75
> 6 44864 7494.36 491150574.57
> 7 52848 7676.76 503104281.98
> 8 60784 7756.76 508347136.11
> 9 68608 7835.26 513491609.52
> 10 76784 7902.30 517885290.67
> 11 84544 7935.96 520091129.45
> 12 92432 7916.76 518832844.57
> 13 100064 7855.96 514848275.43
> 14 107040 7692.52 504136734.09
> 15 114320 7499.66 491497933.56
> 16 121744 7436.99 487390477.85
> 17 129664 7438.92 487517345.01
> 18 136704 7326.50 480149408.39
> 19 144960 7587.00 497221460.09
> 20 153264 7796.56 510955233.33
> 21 160832 7814.44 512126854.90
> elapsed: 21 ops: 163840 ops/sec: 7659.97 bytes/sec: 502004079.43
>
> On Fri, Oct 25, 2019 at 11:54 AM Mike Christie <mchristi(a)redhat.com> wrote:
>>
>> On 10/24/2019 11:47 PM, Ryan wrote:
>> > I'm using CentOS 7.7.1908 with kernel 3.10.0-1062.1.2.el7.x86_64. The
>> > workload was a VMware Storage Motion from a local SSD backed datastore
>>
>> Ignore my comments. I thought you were just doing fio like tests in the vm.
>>
>> > to the ceph backed datastore. Performance was measured using dstat on
>> > the iscsi gateway for network traffic and ceph status as this cluster is
>> > basically idle. I changed max_data_area_mb to 256 and cmdsn_depth to
>> > 128. This appears to have given a slight improvement of maybe 10MB/s.
>> >
>> > Moving VM to the ceph backed datastore
>> > io:
>> > client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
>> >
>> > Moving VM off the ceph backed datastore
>> > io:
>> > client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
>> >
>>
>> If you run esxtop while running your test what do you see for the number
>> of commands in the iscsi LUN's queue?
>>
>> > I'm going to test bonnie++ with an rbd volume mounted directly on the
>>
>> To try and isolate if its the iscsi or rbd, you need to run fio with the
>> librbd io engine. We know krbd is going to be the fastest. ceph-iscsi
>> uses librbd so it is a better baseline. If you are not familiar with fio
>> you can just do something like:
>>
>> fio --group_reporting --ioengine=rbd --direct=1 --name=librbdtest
>> --numjobs=32 --bs=512k --iodepth=128 --size=10G --rw=write
>> --rbd=name_of_your_image -pool=name_of_pool
>>
>>
>> > iscsi gateway. Also will test bonnie++ inside a VM on a ceph backed
>> > datastore.
>> >
>> > On Thu, Oct 24, 2019 at 7:15 PM Mike Christie <mchristi(a)redhat.com
>> > <mailto:mchristi@redhat.com>> wrote:
>> >
>> > On 10/24/2019 12:22 PM, Ryan wrote:
>> > > I'm in the process of testing the iscsi target feature of ceph.
The
>> > > cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of
5
>> >
>> > What kernel are you using?
>> >
>> > > hosts with 12 SSD OSDs per host. Some basic testing moving VMs to
>> > a ceph
>> > > backed datastore is only showing 60MB/s transfers. However moving
>> > these
>> > > back off the datastore is fast at 200-300MB/s.
>> >
>> > What is the workload and what are you using to measure the throughput?
>> >
>> > If you are using fio, what arguments are you using? And, could you
>> > change the ioengine to rbd and re-run the test from the target system
so
>> > we can check if rbd is slow or iscsi?
>> >
>> > For small IOs, 60 is about right.
>> >
>> > For 128-512K IOs you should be able to get around 300 MB/s for writes
>> > and 600 for reads.
>> >
>> > 1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
>> > pass data between the kernel and tcmu-runner. The default is only 8MB.
>> >
>> > In gwcli cd to your disk and do:
>> >
>> > # reconfigure max_data_area_mb %N
>> >
>> > where N is between 8 and 2048 MBs.
>> >
>> > 2. The Linux kernel target only allows 64 commands per iscsi session by
>> > default. We increase that to 128, but you can increase this to 512.
>> >
>> > In gwcli cd to the target dir and do
>> >
>> > reconfigure cmdsn_depth 512
>> >
>> > 3. I think ceph-iscsi and lio work better with higher queue depths so
if
>> > you are using fio you want higher numjobs and/or iodepths.
>> >
>> > >
>> > > What should I be looking at to track down the write performance
issue?
>> > > In comparison with the Nimble Storage arrays I can see 200-300MB/s
in
>> > > both directions.
>> > >
>> > > Thanks,
>> > > Ryan
>> > >
>> > >
>> > > _______________________________________________
>> > > ceph-users mailing list -- ceph-users(a)ceph.io
>> > <mailto:ceph-users@ceph.io>
>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> > <mailto:ceph-users-leave@ceph.io>
>> > >
>> >
>>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Nov
2 Nov
12:49 p.m.
On 31/10/2019 18:45, Paul Emmerich wrote:
On Fri, Oct 25, 2019 at 11:14 PM Maged Mokhtar
<mmokhtar(a)petasan.org> wrote:
If case 3 (migration to/from Ceph to external stores) is important, then
maybe it is worth trying Veeam Quick Migration which supports live
migration. We have not tried it, but maybe this tool could be smarter
than vmware and use larger block sizes. They seem to have a free
edition, if you try it, i would recommend bumping the iSCSI MaxIoSizeKB
on the ESXi side as it helps with their backup tool.
I would not recommend using striping in this case, only very specific
cases benefit from it..but for general purpose vm workloads, striping
will probably be multiplying your queue depth and latencies with
negative effects, specially with such small sizes.
Ilya added striping to kernel rbd maybe in mainstream 4.17, we had sent
our own patches earlier but Ilya changes were more comprehensive. For
kernel rbd and librbd/tcmu, both have pros and cons, the kernel path is
used by SUSE, the main issue for us is the support for persistent
reservations which is required for setting up clustering on Windows.
/Maged
3. vmotion between Ceph datastore and an external
datastore..this will be bad. This seems the case you are testing. It is bad because
between 2 different storage systems (iqns are served on different targets), vaai xcopy
cannot be used and vmware does its own stuff. It moves data using 64k block size, which
gives low performance...to add some flavor, it does indeed use 32 threads, but
unfortunately they use co-located addresses which does not work well in Ceph as they are
hitting the same rbd object, which gets serialized due to pg locks, so you will not get
any palatalization. Your speed will mostly be determined by a serial 64k, so with 1 ms
write latency for ssd cluster, you will get around 64 MB/s..it will be slightly higher as
the extra threads have some low effect.
Yes, vmotion is the worst IO pattern ever
for a sequential copy.
However, the situation you are describing can be fixed with RBD
striping v2, just make Ceph switch to another object every 64kb, see
https://docs.ceph.com/docs/master/dev/file-striping/
I'm not sure about the state of striping v2 support in the kernel
module, last time I checked it wasn't supported. But
ceph-iscsi/tcmu-runner got quite good over the past year, I don't see
any point in still using the kernel data path for iscsi nowadays.
Paul
> Note your esxtop does show 32 active ios under
ACTV, the QUED of zero does is not the queue depth, but rather the "queued" io
the ESX would suspend in case your active reaches the maximum by adapater ( 128 ).
>
> This is just to clarify, if case 3 is not your primary concern than i would forget
about it and benchmark 1 and 2 if they are relevant. Else, if 3 is important, i am not
sure you can do much as it is happening within vmware..maybe there could be a way to map
the external iqn to be served by the same target serving the Ceph iqn then there could be
a chance the xcopy could be activated..Mike would probably know if this has any chance of
working :)
>
> /Maged
>
>
> On 25/10/2019 22:01, Ryan wrote:
>
> esxtop is showing a queue length of 0
>
> Storage motion to ceph
> DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD
LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
> naa.6001405ec60d8b82342404d929fbbd03 - 128 - 32 0 25
0.25 1442.32 0.18 1440.50 0.00 89.78 21.32 0.01 21.34 0.01
>
> Storage motion from ceph
> DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD
LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
> naa.6001405ec60d8b82342404d929fbbd03 - 128 - 32 0 25
0.25 4065.38 4064.83 0.36 253.52 0.00 7.57 0.01 7.58 0.00
>
> I tried using fio like you mentioned but it was hanging with
[r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS] and the ETA kept climbing. I ended up using rbd bench on
the ceph iscsi gateway. With a 64K write workload I'm seeing 400MB/s transfers.
>
> rbd create test --size 100G --image-feature layering
> rbd map test
> mkfs.ext4 /dev/rbd/rbd/test
> mount /dev/rbd/rbd/test test
>
> rbd create testec --size 100G --image-feature layering --data-pool rbd_ec
> rbd map testec
> mkfs.ext4 /dev/rbd/rbd/testec
> mount /dev/rbd/rbd/testec testec
>
> [root@ceph-iscsi1 mnt]# rbd bench --image test --io-size 64K --io-type write
--io-total 10G
> bench type write io_size 65536 io_threads 16 bytes 10737418240 pattern sequential
> SEC OPS OPS/SEC BYTES/SEC
> 1 6368 6377.59 417961796.64
> 2 12928 6462.27 423511630.71
> 3 19296 6420.18 420752986.78
> 4 26320 6585.61 431594792.67
> 5 33296 6662.37 436624891.04
> 6 40128 6754.67 442673957.25
> 7 46784 6765.75 443400452.26
> 8 53280 6809.02 446236110.93
> 9 60032 6739.67 441691068.73
> 10 66784 6698.91 439019550.77
> 11 73616 6690.88 438493253.66
> 12 80016 6654.35 436099640.00
> 13 85712 6485.07 425005611.11
> 14 91088 6202.49 406486113.46
> 15 96896 6021.17 394603137.62
> 16 102368 5741.19 376254347.24
> 17 107568 5501.57 360550910.38
> 18 113728 5603.17 367209502.58
> 19 120144 5820.48 381451245.32
> 20 126496 5917.60 387816078.53
> 21 132768 6089.71 399095466.00
> 22 139040 6306.98 413334431.09
> 23 145104 6276.42 411331743.63
> 24 151440 6256.67 410036891.68
> 25 157808 6261.12 410328554.98
> 26 163456 6140.03 402392725.36
> elapsed: 26 ops: 163840 ops/sec: 6271.36 bytes/sec: 410999626.38
>
> [root@ceph-iscsi1 mnt]# rbd bench --image testec --io-size 64K --io-type write
--io-total 10G
> bench type write io_size 65536 io_threads 16 bytes 10737418240 pattern sequential
> SEC OPS OPS/SEC BYTES/SEC
> 1 7392 7415.38 485974266.41
> 2 14464 7243.59 474715656.29
> 3 22000 7341.08 481104853.50
> 4 29408 7352.29 481839517.16
> 5 37296 7459.38 488857889.75
> 6 44864 7494.36 491150574.57
> 7 52848 7676.76 503104281.98
> 8 60784 7756.76 508347136.11
> 9 68608 7835.26 513491609.52
> 10 76784 7902.30 517885290.67
> 11 84544 7935.96 520091129.45
> 12 92432 7916.76 518832844.57
> 13 100064 7855.96 514848275.43
> 14 107040 7692.52 504136734.09
> 15 114320 7499.66 491497933.56
> 16 121744 7436.99 487390477.85
> 17 129664 7438.92 487517345.01
> 18 136704 7326.50 480149408.39
> 19 144960 7587.00 497221460.09
> 20 153264 7796.56 510955233.33
> 21 160832 7814.44 512126854.90
> elapsed: 21 ops: 163840 ops/sec: 7659.97 bytes/sec: 502004079.43
>
> On Fri, Oct 25, 2019 at 11:54 AM Mike Christie <mchristi(a)redhat.com> wrote:
>> On 10/24/2019 11:47 PM, Ryan wrote:
>>> I'm using CentOS 7.7.1908 with kernel 3.10.0-1062.1.2.el7.x86_64. The
>>> workload was a VMware Storage Motion from a local SSD backed datastore
>> Ignore my comments. I thought you were just doing fio like tests in the vm.
>>
>>> to the ceph backed datastore. Performance was measured using dstat on
>>> the iscsi gateway for network traffic and ceph status as this cluster is
>>> basically idle. I changed max_data_area_mb to 256 and cmdsn_depth to
>>> 128. This appears to have given a slight improvement of maybe 10MB/s.
>>>
>>> Moving VM to the ceph backed datastore
>>> io:
>>> client: 124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
>>>
>>> Moving VM off the ceph backed datastore
>>> io:
>>> client: 344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
>>>
>> If you run esxtop while running your test what do you see for the number
>> of commands in the iscsi LUN's queue?
>>
>>> I'm going to test bonnie++ with an rbd volume mounted directly on the
>> To try and isolate if its the iscsi or rbd, you need to run fio with the
>> librbd io engine. We know krbd is going to be the fastest. ceph-iscsi
>> uses librbd so it is a better baseline. If you are not familiar with fio
>> you can just do something like:
>>
>> fio --group_reporting --ioengine=rbd --direct=1 --name=librbdtest
>> --numjobs=32 --bs=512k --iodepth=128 --size=10G --rw=write
>> --rbd=name_of_your_image -pool=name_of_pool
>>
>>
>>> iscsi gateway. Also will test bonnie++ inside a VM on a ceph backed
>>> datastore.
>>>
>>> On Thu, Oct 24, 2019 at 7:15 PM Mike Christie <mchristi(a)redhat.com
>>> <mailto:mchristi@redhat.com>> wrote:
>>>
>>> On 10/24/2019 12:22 PM, Ryan wrote:
>>> > I'm in the process of testing the iscsi target feature of ceph.
The
>>> > cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of
5
>>>
>>> What kernel are you using?
>>>
>>> > hosts with 12 SSD OSDs per host. Some basic testing moving VMs to
>>> a ceph
>>> > backed datastore is only showing 60MB/s transfers. However moving
>>> these
>>> > back off the datastore is fast at 200-300MB/s.
>>>
>>> What is the workload and what are you using to measure the throughput?
>>>
>>> If you are using fio, what arguments are you using? And, could you
>>> change the ioengine to rbd and re-run the test from the target system
so
>>> we can check if rbd is slow or iscsi?
>>>
>>> For small IOs, 60 is about right.
>>>
>>> For 128-512K IOs you should be able to get around 300 MB/s for writes
>>> and 600 for reads.
>>>
>>> 1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
>>> pass data between the kernel and tcmu-runner. The default is only 8MB.
>>>
>>> In gwcli cd to your disk and do:
>>>
>>> # reconfigure max_data_area_mb %N
>>>
>>> where N is between 8 and 2048 MBs.
>>>
>>> 2. The Linux kernel target only allows 64 commands per iscsi session by
>>> default. We increase that to 128, but you can increase this to 512.
>>>
>>> In gwcli cd to the target dir and do
>>>
>>> reconfigure cmdsn_depth 512
>>>
>>> 3. I think ceph-iscsi and lio work better with higher queue depths so
if
>>> you are using fio you want higher numjobs and/or iodepths.
>>>
>>> >
>>> > What should I be looking at to track down the write performance
issue?
>>> > In comparison with the Nimble Storage arrays I can see 200-300MB/s
in
>>> > both directions.
>>> >
>>> > Thanks,
>>> > Ryan
>>> >
>>> >
>>> > _______________________________________________
>>> > ceph-users mailing list -- ceph-users(a)ceph.io
>>> <mailto:ceph-users@ceph.io>
>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> <mailto:ceph-users-leave@ceph.io>
>>> >
>>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 Oct
25 Oct
11:50 a.m.
Hi,
Em qui, 24 de out de 2019 às 20:16, Mike Christie <mchristi(a)redhat.com>
escreveu:
On 10/24/2019 12:22 PM, Ryan wrote:
where N is between 8 and 2048 MBs.
I'm in the process of testing the iscsi
target feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving VMs to a ceph
backed datastore is only showing 60MB/s transfers. However moving these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the throughput?
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
pass data between the kernel and tcmu-runner. The default is only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
2. The Linux kernel target only allows 64 commands per iscsi session by
default. We increase that to 128, but you can increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
For these commands, need the disk is detached or can be running "hot"?
3. I think ceph-iscsi and lio work better with higher
queue depths so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write performance issue?
In comparison with the Nimble Storage arrays I can see 200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:33 p.m.
I had to remove the disk from the target host in gwcli to
change max_data_area_mb. So the disk would need to be detached. For
cmdsn_depth I was able to change it live.
On Fri, Oct 25, 2019 at 7:50 AM Gesiel Galvão Bernardes <
gesiel.bernardes(a)gmail.com> wrote:
Hi,
Em qui, 24 de out de 2019 às 20:16, Mike Christie <mchristi(a)redhat.com>
escreveu:
On 10/24/2019 12:22 PM, Ryan wrote:
where N is between 8 and 2048 MBs.
I'm in the process of testing the iscsi
target feature of ceph. The
cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5
What kernel are you using?
hosts with 12 SSD OSDs per host. Some basic
testing moving VMs to a ceph
backed datastore is only showing 60MB/s transfers. However moving these
back off the datastore is fast at 200-300MB/s.
What is the workload and what are you using to measure the throughput?
If you are using fio, what arguments are you using? And, could you
change the ioengine to rbd and re-run the test from the target system so
we can check if rbd is slow or iscsi?
For small IOs, 60 is about right.
For 128-512K IOs you should be able to get around 300 MB/s for writes
and 600 for reads.
1. Increase max_data_area_mb. This is a kernel buffer lio/tcmu uses to
pass data between the kernel and tcmu-runner. The default is only 8MB.
In gwcli cd to your disk and do:
# reconfigure max_data_area_mb %N
2. The Linux kernel target only allows 64 commands per iscsi session by
default. We increase that to 128, but you can increase this to 512.
In gwcli cd to the target dir and do
reconfigure cmdsn_depth 512
For these commands, need the disk is detached or can be running "hot"?
3. I think ceph-iscsi and lio work better with
higher queue depths so if
you are using fio you want higher numjobs and/or iodepths.
What should I be looking at to track down the write performance issue?
In comparison with the Nimble Storage arrays I can see 200-300MB/s in
both directions.
Thanks,
Ryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1643
days inactive
1652
days old
20 comments
8 participants
participants (8)
-
Drew Weaver -
Gesiel Galvão Bernardes -
Maged Mokhtar -
Martin Verges -
Mike Christie -
Nathan Fish -
Paul Emmerich -
Ryan