On 10/25/2019 09:31 AM, Ryan wrote:
> I'm not seeing the emulate_3pc setting under disks/rbd/diskname when

emulate_3pc is only for kernel based backends. tcmu-runner always has
xcopy on.

> calling info. A google search shows that SUSE Enterprise Storage has it
> available. I thought I had the latest packages, but maybe not. I'm using
> tcmu-runner 1.5.2 and ceph-iscsi 3.3. Almost all of my VMs are currently
> on Nimble iSCSI storage. I've actually tested from both and performance
> is the same. Doing the math off the ceph status does show it using 64K
> blocks in both cases.
>
> Control Values
> - hw_max_sectors .. 1024
> - max_data_area_mb .. 256 (override)
> - osd_op_timeout .. 30
> - qfull_timeout .. 5
>
> On Fri, Oct 25, 2019 at 4:46 AM Maged Mokhtar <mmokhtar@petasan.org
> <mailto:mmokhtar@petasan.org>> wrote:
>
>     Actually this may not work if moving from a local datastore to Ceph.
>     For iSCSI xcopy, both the source and destination need to be
>     accessible by the target such as in moving vms across Ceph
>     datastores. So in your case, vmotion will be handled by VMWare data
>     mover which uses 64K block sizes.
>
>     On 25/10/2019 10:28, Maged Mokhtar wrote:
>>
>>     For vmotion speed, check "emulate_3pc" attribute on the LIO
>>     target. If 0 (default), VMWare will issue io in 64KB blocks which
>>     gives low speed. if set to 1  this will trigger VMWare to use vaai
>>     extended copy, which activates LIO's xcopy functionality which
>>     uses 512KB block sizes by default. We also bumped the xcopy block
>>     size to 4M (rbd object size) which gives around 400 MB/s vmotion
>>     speed, the same speed can also be achieved via Veeam backups.
>>
>>     /Maged
>>
>>     On 25/10/2019 06:47, Ryan wrote:
>>>     I'm using CentOS 7.7.1908 with kernel 3.10.0-1062.1.2.el7.x86_64.
>>>     The workload was a VMware Storage Motion from a local SSD backed
>>>     datastore to the ceph backed datastore. Performance was measured
>>>     using dstat on the iscsi gateway for network traffic and ceph
>>>     status as this cluster is basically idle.  I changed
>>>     max_data_area_mb to 256 and cmdsn_depth to 128. This appears to
>>>     have given a slight improvement of maybe 10MB/s.
>>>
>>>     Moving VM to the ceph backed datastore
>>>     io:
>>>         client:   124 KiB/s rd, 76 MiB/s wr, 95 op/s rd, 1.26k op/s wr
>>>
>>>     Moving VM off the ceph backed datastore
>>>       io:
>>>         client:   344 MiB/s rd, 625 KiB/s wr, 5.54k op/s rd, 62 op/s wr
>>>
>>>     I'm going to test bonnie++ with an rbd volume mounted directly on
>>>     the iscsi gateway. Also will test bonnie++ inside a VM on a ceph
>>>     backed datastore.
>>>
>>>     On Thu, Oct 24, 2019 at 7:15 PM Mike Christie
>>>     <mchristi@redhat.com <mailto:mchristi@redhat.com>> wrote:
>>>
>>>         On 10/24/2019 12:22 PM, Ryan wrote:
>>>         > I'm in the process of testing the iscsi target feature of
>>>         ceph. The
>>>         > cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It
>>>         consists of 5
>>>
>>>         What kernel are you using?
>>>
>>>         > hosts with 12 SSD OSDs per host. Some basic testing moving
>>>         VMs to a ceph
>>>         > backed datastore is only showing 60MB/s transfers. However
>>>         moving these
>>>         > back off the datastore is fast at 200-300MB/s.
>>>
>>>         What is the workload and what are you using to measure the
>>>         throughput?
>>>
>>>         If you are using fio, what arguments are you using? And,
>>>         could you
>>>         change the ioengine to rbd and re-run the test from the
>>>         target system so
>>>         we can check if rbd is slow or iscsi?
>>>
>>>         For small IOs, 60 is about right.
>>>
>>>         For 128-512K IOs you should be able to get around 300 MB/s
>>>         for writes
>>>         and 600 for reads.
>>>
>>>         1. Increase max_data_area_mb. This is a kernel buffer
>>>         lio/tcmu uses to
>>>         pass data between the kernel and tcmu-runner. The default is
>>>         only 8MB.
>>>
>>>         In gwcli cd to your disk and do:
>>>
>>>         # reconfigure max_data_area_mb %N
>>>
>>>         where N is between 8 and 2048 MBs.
>>>
>>>         2. The Linux kernel target only allows 64 commands per iscsi
>>>         session by
>>>         default. We increase that to 128, but you can increase this
>>>         to 512.
>>>
>>>         In gwcli cd to the target dir and do
>>>
>>>         reconfigure cmdsn_depth 512
>>>
>>>         3. I think ceph-iscsi and lio work better with higher queue
>>>         depths so if
>>>         you are using fio you want higher numjobs and/or iodepths.
>>>
>>>         >
>>>         > What should I be looking at to track down the write
>>>         performance issue?
>>>         > In comparison with the Nimble Storage arrays I can see
>>>         200-300MB/s in
>>>         > both directions.
>>>         >
>>>         > Thanks,
>>>         > Ryan
>>>         >
>>>         >
>>>         > _______________________________________________
>>>         > ceph-users mailing list -- ceph-users@ceph.io
>>>         <mailto:ceph-users@ceph.io>
>>>         > To unsubscribe send an email to ceph-users-leave@ceph.io
>>>         <mailto:ceph-users-leave@ceph.io>
>>>         >
>>>
>>>
>>>     _______________________________________________
>>>     ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io>
>>>     To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
>>
>>     _______________________________________________
>>     ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io>
>>     To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
>