‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, February 14, 2020 4:49 PM, Mike Christie <mchristi@redhat.com> wrote:
On 02/13/2020 09:56 AM, Salsa wrote:
I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs.
How are you testing client throughput? What tool and args?
Not testing. This is what my ceph grafana is showing me while I'm writing to the LUN (ISCSI LUN -> VMWARE -> DATASTORE -> VM I/O).
Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM.
How are you getting the latency and throughput values for iscsi? Is it esxtop? Were you saying you filled up the vmware iscsi queue based on the esxtop queue values, and have you increased values like the ESX queue depth value like here:
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.troubleshoo...
Latency and throughput also from ceph grafana, but I checked esxtop latency and it is almost the same. About filling the queue, I think I've filled it because the esxi froze and I had to reboot he host (HW). Haven't increased queue as I imagined it would only take longer to fill it up and freeze esxi again.
Note: Sometimes people only increase iscsivmk_LunQDepth, but then forget to also increase iscsivmk_HostQDepth.
What is your ceph-iscsi, tcmu-runner and kernel version on the target side?
TCMU-RUNNER 1.5.2 CEPH 14.2.6 Linux ceph01 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Here are some general tweaks for the target side:
1. Increase the max_data_area_mb (this affects the LUN's max queueing) and target side queue depth.
gwcli
======
cd /disks reconfigure rbd/<image_name> max_data_area_mb 128
gwcli
======
cd /iscsi-target/iqn.2003-01.com.redhat.iscsi-gw:ceph-igw reconfigure iqn.2003-01.com.redhat.iscsi-gw:ceph-igw cmdsn_depth 512
2. Are the VMs on the same iscsi LUN or different ones?
If on the same LUN then increasing the max_data_area_mb value will help, because with smaller values we will get lots of qfulls and latency will be better since IO is not sitting in the target side queue waiting for memory. On the initaitor side though it is still sometimes helpful for testing to disable the vmware SIOC and adaptive queueing features.
3. Did you check the basics like the multipathing is setup correctly?
esxcli storage nmp path list -d yourdevice
shows all the expected paths?
On the initiator and target side, did you check the logs for any errors going on when you run your test?
4. What test tool are you running and what args? If you just run a plain fio command like:
fio --filename=some_new_file --bs=128K --size=5G --name=test --iodepth=128 --direct=1 --numjobs=8 --rw=read --ioengine=libaio
what do you get?
If you run almost the same fio command from the gateway machine or vm, but use --ioengine=rbd
fio --bs=128K --size=5G --name=test --iodepth=128 --direct=1 --numjobs=8 --rw=read --ioengine=rbd --rbdname=your_image --pool=your-pool_rbd
how does it compare?
Will try these tweaks. Only 1 VM so far. Multipathing is correct and I am not using test tools. Real world usage.