13 Feb
2020
13 Feb
'20
9:26 p.m.
I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool
and some images and presented them to a Vmware host via ISCSI, but the write performance
is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and
had to reboot it's host (seems I've filled up Vmware's ISCSI queue).
Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes
peaking at 600 ms (per OSD).
Client throughput is giving me around 4 MBs.
Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM.
On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM.
I think the performance is way too slow, much more than should be and that I can fix this
by correcting some configuration.
Any advices?
--
Salsa
14 Feb
14 Feb
1:20 a.m.
Hi Salsa,
More information about your Ceph cluster and VMware infrastructure is pretty much
required.
What Ceph version?
Ceph cluster info - i.e. how many Monitors, OSD hosts, iSCIS gateways and are these
components HW or VMs?
Do the Ceph components meet recommended hardware levels for CPU, RAM, HDs (Flash or
spinners)?
Basic Ceph stats like "ceph osd tree" and "ceph df"
What VMware version?
Software or Hardware iSCSI on the VMW side?
What's the Storage network speed and are thing like jumbo frames set?
In general, 3 OSD hosts is the bare minimum for ceph so you're going to get minimum
performance.
Andrew Ferris
Network & System Management
UBC Centre for Heart & Lung Innovation
St. Paul's Hospital, Vancouver
http://www.hli.ubc.ca
>> Salsa <salsa(a)protonmail.com> 2/13/2020
7:56 AM >>>
I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I
deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI,
but the write performance is so bad the I managed to freeze a VM doing a big rsync to a
datastore inside ceph and had to reboot it's host (seems I've filled up
Vmware's ISCSI queue).
Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes
peaking at 600 ms (per OSD).
Client throughput is giving me around 4 MBs.
Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM.
On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM.
I think the performance is way too slow, much more than should be and that I can fix this
by correcting some configuration.
Any advices?
--
Salsa
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:53 a.m.
I'm currently running Nautilus:
[root@ceph01 ~]# ceph -s
cluster:
id: eb4aea44-0c63-4202-b826-e16ea60ed54d
health: HEALTH_WARN
too few PGs per OSD (25 < min 30)
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 8d)
mgr: ceph02(active, since 9d), standbys: ceph03, ceph01
osd: 30 osds: 30 up (since 9d), 30 in (since 9d)
rbd-mirror: 3 daemons active (139316, 214105, 264563)
tcmu-runner: 15 daemons active (ceph01:rbd/Disco_vm_01, ceph01:rbd/Disco_vm_02,
ceph01:rbd/Disk1, ceph01:rbd/Disk2, ceph01:rbd/Disk3, ceph02:rbd/Disco_vm_01,
ceph02:rbd/Disco_vm_02, ceph02:rbd/Disk1, ceph02:rbd/Disk2, ceph02:rbd/Disk3,
ceph03:rbd/Disco_vm_01, ceph03:rbd/Disco_vm_02, ceph03:rbd/Disk1, ceph03:rbd/Disk2,
ceph03:rbd/Disk3)
data:
pools: 1 pools, 256 pgs
objects: 1.42M objects, 4.4 TiB
usage: 13 TiB used, 96 TiB / 109 TiB avail
pgs: 255 active+clean
1 active+clean+scrubbing+deep
io:
client: 1.3 MiB/s rd, 1.8 MiB/s wr, 140 op/s rd, 82 op/s wr
[root@ceph01 ~]# ceph --version
ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
3 Hosts running: 1 monitor each, 1 MGR each, 10 OSDs each, 1 MDS each, 1 TCMU-RUNNER each,
1 rbd-mirror each. Hosts are HW.
HW wise: 64 GB RAM + 32 cores, but we are using HDDs.
VMware: 'esxcli' version: 6.0.0 and Software iSCSI on VMW side (I guess).
Storage network at 1Gb and no jumbo frames. I have a cluster network (1Gb) and a public
network (1Gb) set.
[root@ceph01 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 109.16061 root default
-3 36.38687 host ceph01
0 hdd 3.63869 osd.0 up 1.00000 1.00000
1 hdd 3.63869 osd.1 up 1.00000 1.00000
3 hdd 3.63869 osd.3 up 1.00000 1.00000
4 hdd 3.63869 osd.4 up 1.00000 1.00000
5 hdd 3.63869 osd.5 up 1.00000 1.00000
6 hdd 3.63869 osd.6 up 1.00000 1.00000
7 hdd 3.63869 osd.7 up 1.00000 1.00000
8 hdd 3.63869 osd.8 up 1.00000 1.00000
9 hdd 3.63869 osd.9 up 1.00000 1.00000
10 hdd 3.63869 osd.10 up 1.00000 1.00000
-5 36.38687 host ceph02
11 hdd 3.63869 osd.11 up 1.00000 1.00000
12 hdd 3.63869 osd.12 up 1.00000 1.00000
13 hdd 3.63869 osd.13 up 1.00000 1.00000
14 hdd 3.63869 osd.14 up 1.00000 1.00000
15 hdd 3.63869 osd.15 up 1.00000 1.00000
16 hdd 3.63869 osd.16 up 1.00000 1.00000
17 hdd 3.63869 osd.17 up 1.00000 1.00000
18 hdd 3.63869 osd.18 up 1.00000 1.00000
19 hdd 3.63869 osd.19 up 1.00000 1.00000
20 hdd 3.63869 osd.20 up 1.00000 1.00000
-7 36.38687 host ceph03
21 hdd 3.63869 osd.21 up 1.00000 1.00000
22 hdd 3.63869 osd.22 up 1.00000 1.00000
23 hdd 3.63869 osd.23 up 1.00000 1.00000
24 hdd 3.63869 osd.24 up 1.00000 1.00000
25 hdd 3.63869 osd.25 up 1.00000 1.00000
26 hdd 3.63869 osd.26 up 1.00000 1.00000
27 hdd 3.63869 osd.27 up 1.00000 1.00000
28 hdd 3.63869 osd.28 up 1.00000 1.00000
29 hdd 3.63869 osd.29 up 1.00000 1.00000
30 hdd 3.63869 osd.30 up 1.00000 1.00000
[root@ceph01 ~]# ceph df
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 109 TiB 96 TiB 13 TiB 13 TiB 11.99
TOTAL 109 TiB 96 TiB 13 TiB 13 TiB 11.99
POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL
rbd 29 4.3 TiB 1.42M 13 TiB 13.13 29 TiB
--
Salsa
Sent with [ProtonMail](https://protonmail.com) Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, February 13, 2020 4:50 PM, Andrew Ferris <Andrew.Ferris(a)hli.ubc.ca>
wrote:
> Hi Salsa,
>
> More information about your Ceph cluster and VMware infrastructure is pretty much
required.
>
> What Ceph version?
> Ceph cluster info - i.e. how many Monitors, OSD hosts, iSCIS gateways and are these
components HW or VMs?
> Do the Ceph components meet recommended hardware levels for CPU, RAM, HDs (Flash or
spinners)?
> Basic Ceph stats like "ceph osd tree" and "ceph df"
>
> What VMware version?
> Software or Hardware iSCSI on the VMW side?
>
> What's the Storage network speed and are thing like jumbo frames set?
>
> In general, 3 OSD hosts is the bare minimum for ceph so you're going to get
minimum performance.
>
> Andrew Ferris
> Network & System Management
> UBC Centre for Heart & Lung Innovation
> St. Paul's Hospital, Vancouver
> [http://www.hli.ubc.ca](http://www.hli.ubc.ca/)
>
>>>> Salsa <salsa(a)protonmail.com> 2/13/2020 7:56 AM >>>
> I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd
pool and some images and presented them to a Vmware host via ISCSI, but the write
performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside
ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue).
>
> Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes
peaking at 600 ms (per OSD).
> Client throughput is giving me around 4 MBs.
> Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM.
> On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM.
>
> I think the performance is way too slow, much more than should be and that I can fix
this by correcting some configuration.
>
> Any advices?
>
> --
> Salsa
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
15 Feb
15 Feb
1:19 a.m.
On 02/13/2020 09:56 AM, Salsa wrote:
I have a 3 hosts, 10 4TB HDDs per host ceph storage
set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host
via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big
rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up
Vmware's ISCSI queue).
Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes
peaking at 600 ms (per OSD).
Client throughput is giving me around 4 MBs.
How are you testing client throughput? What tool and args?
Using a 4MB stripe 1 image I got 1.955..359 B/s inside
the VM.
On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM.
How are you getting the latency and throughput values for iscsi? Is it
esxtop? Were you saying you filled up the vmware iscsi queue based on
the esxtop queue values, and have you increased values like the ESX
queue depth value like here:
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.troublesho…
Note: Sometimes people only increase iscsivmk_LunQDepth, but then forget
to also increase iscsivmk_HostQDepth.
What is your ceph-iscsi, tcmu-runner and kernel version on the target side?
Here are some general tweaks for the target side:
1. Increase the max_data_area_mb (this affects the LUN's max queueing)
and target side queue depth.
# gwcli
cd /disks
reconfigure rbd/<image_name> max_data_area_mb 128
# gwcli
cd /iscsi-target/iqn.2003-01.com.redhat.iscsi-gw:ceph-igw
reconfigure iqn.2003-01.com.redhat.iscsi-gw:ceph-igw cmdsn_depth 512
2. Are the VMs on the same iscsi LUN or different ones?
If on the same LUN then increasing the max_data_area_mb value will help,
because with smaller values we will get lots of qfulls and latency will
be better since IO is not sitting in the target side queue waiting for
memory. On the initaitor side though it is still sometimes helpful for
testing to disable the vmware SIOC and adaptive queueing features.
3. Did you check the basics like the multipathing is setup correctly?
esxcli storage nmp path list -d yourdevice
shows all the expected paths?
On the initiator and target side, did you check the logs for any errors
going on when you run your test?
4. What test tool are you running and what args? If you just run a
plain fio command like:
fio --filename=some_new_file --bs=128K --size=5G --name=test
--iodepth=128 --direct=1 --numjobs=8 --rw=read --ioengine=libaio
what do you get?
If you run almost the same fio command from the gateway machine or vm,
but use --ioengine=rbd
fio --bs=128K --size=5G --name=test --iodepth=128 --direct=1 --numjobs=8
--rw=read --ioengine=rbd --rbdname=your_image --pool=your-pool_rbd
how does it compare?
3:29 a.m.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, February 14, 2020 4:49 PM, Mike Christie <mchristi(a)redhat.com> wrote:
Latency and throughput also from ceph grafana, but I checked esxtop latency and it is
almost the same.
About filling the queue, I think I've filled it because the esxi froze and I had to
reboot he host (HW).
Haven't increased queue as I imagined it would only take longer to fill it up and
freeze esxi again.
On 02/13/2020 09:56 AM, Salsa wrote:
Not testing. This is what my ceph grafana is showing me while I'm writing to the LUN
(ISCSI LUN -> VMWARE -> DATASTORE -> VM I/O).
I have a 3 hosts, 10 4TB HDDs per host ceph
storage set up. I deined a 3 replica rbd pool and some images and presented them to a
Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM
doing a big rsync to a datastore inside ceph and had to reboot it's host (seems
I've filled up Vmware's ISCSI queue).
Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes
peaking at 600 ms (per OSD).
Client throughput is giving me around 4 MBs.
How are you testing client throughput? What tool and args? Using a 4MB stripe 1 image I got 1.955..359 B/s
inside the VM.
On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM.
How are you getting the latency and throughput values for iscsi? Is it
esxtop? Were you saying you filled up the vmware iscsi queue based on
the esxtop queue values, and have you increased values like the ESX
queue depth value like here:
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.troublesho…
Note: Sometimes people only increase
iscsivmk_LunQDepth, but then forget
to also increase iscsivmk_HostQDepth.
What is your ceph-iscsi, tcmu-runner and kernel version on the target side?
TCMU-RUNNER 1.5.2
CEPH 14.2.6
Linux ceph01 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
Here are some general tweaks for the target side:
1. Increase the max_data_area_mb (this affects the LUN's max queueing)
and target side queue depth.
gwcli
======
cd /disks
reconfigure rbd/<image_name> max_data_area_mb 128
gwcli
======
cd /iscsi-target/iqn.2003-01.com.redhat.iscsi-gw:ceph-igw
reconfigure iqn.2003-01.com.redhat.iscsi-gw:ceph-igw cmdsn_depth 512
2. Are the VMs on the same iscsi LUN or different ones?
If on the same LUN then increasing the max_data_area_mb value will help,
because with smaller values we will get lots of qfulls and latency will
be better since IO is not sitting in the target side queue waiting for
memory. On the initaitor side though it is still sometimes helpful for
testing to disable the vmware SIOC and adaptive queueing features.
3. Did you check the basics like the multipathing is setup correctly?
esxcli storage nmp path list -d yourdevice
shows all the expected paths?
On the initiator and target side, did you check the logs for any errors
going on when you run your test?
4. What test tool are you running and what args? If you just run a
plain fio command like:
fio --filename=some_new_file --bs=128K --size=5G --name=test
--iodepth=128 --direct=1 --numjobs=8 --rw=read --ioengine=libaio
what do you get?
If you run almost the same fio command from the gateway machine or vm,
but use --ioengine=rbd
fio --bs=128K --size=5G --name=test --iodepth=128 --direct=1 --numjobs=8
--rw=read --ioengine=rbd --rbdname=your_image --pool=your-pool_rbd
how does it compare?
Will try these tweaks.
Only 1 VM so far.
Multipathing is correct and I am not using test tools. Real world usage.
1553
days inactive
1554
days old
4 comments
3 participants
participants (3)
-
Andrew Ferris -
Mike Christie -
Salsa