10 Nov
2020
10 Nov
'20
6:53 p.m.
Hi,
We have recently deployed a Ceph cluster with
12 OSD nodes(16 Core + 200GB RAM + 30 disks each of 14TB) Running CentOS 8
3 Monitoring Nodes (8 Core + 16GB RAM) Running CentOS 8
We are using Ceph Octopus and we are using RBD block devices.
We have three Ceph client nodes(16core + 30GB RAM, Running CentOS 8) across which RBDs are
mapped and mounted, 25 RBDs each on each client node. Each RBD size is 10TB. Each RBD is
formatted as EXT4 file system.
From network side, we have 10Gbps Active/Passive Bond on all the Ceph cluster nodes,
including the clients. Jumbo frames enabled and MTU is 9000
This is a new cluster and cluster health reports Ok. But we see high IO wait during the
writes.
From one of the clients,
15:14:30 CPU %user %nice %system %iowait %steal %idle
15:14:31 all 0.06 0.00 1.00 45.03 0.00 53.91
15:14:32 all 0.06 0.00 0.94 41.28 0.00 57.72
15:14:33 all 0.06 0.00 1.25 45.78 0.00 52.91
15:14:34 all 0.00 0.00 1.06 40.07 0.00 58.86
15:14:35 all 0.19 0.00 1.38 41.04 0.00 57.39
Average: all 0.08 0.00 1.13 42.64 0.00 56.16
and the system load shows very high
top - 15:19:15 up 34 days, 41 min, 2 users, load average: 13.49, 13.62, 13.83
From 'atop'
one of the CPUs shows this
CPU | sys 7% | user 1% | irq 2% | idle 1394% | wait 195% | steal 0% | guest
0% | ipc initial | cycl initial | curf 806MHz | curscal ?%
On the OSD nodes, don't see much %utilization of the disks.
RBD caching values are default.
Are we overlooking some configuration item ?
Thanks and Regards,
At
12 Nov
12 Nov
2:13 p.m.
From different search results I read, disabling cephx can help.
Also https://static.linaro.org/connect/san19/presentations/san19-120.pdf
recommended some settings changes for the bluestore cache.
[osd]
bluestore cache autotune = 0
bluestore_cache_kv_ratio = 0.2
bluestore_cache_meta_ratio = 0.8
bluestore rocksdb options =
compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,write_buffer_size=64M,compaction_readahead_size=2M
bluestore_cache_size_hdd = 536870912 # This is size of the Cache on the HDD
osd_min_pg_log_entries = 10
osd_max_pg_log_entries = 10
osd_pg_log_dups_tracked = 10
osd_pg_log_trim_min = 10
But nothing changed much.
It looks like it is mostly with the small files, when I tested the same
with 128k or even 64k block size, the results were much better.
Any suggestions?
Thanks and Regards,
Athreya
On Tue, Nov 10, 2020 at 8:51 PM <athreyavc(a)gmail.com> wrote:
Hi,
We have recently deployed a Ceph cluster with
12 OSD nodes(16 Core + 200GB RAM + 30 disks each of 14TB) Running CentOS 8
3 Monitoring Nodes (8 Core + 16GB RAM) Running CentOS 8
We are using Ceph Octopus and we are using RBD block devices.
We have three Ceph client nodes(16core + 30GB RAM, Running CentOS 8)
across which RBDs are mapped and mounted, 25 RBDs each on each client node.
Each RBD size is 10TB. Each RBD is formatted as EXT4 file system.
From network side, we have 10Gbps Active/Passive Bond on all the Ceph
cluster nodes, including the clients. Jumbo frames enabled and MTU is 9000
This is a new cluster and cluster health reports Ok. But we see high IO
wait during the writes.
From one of the clients,
15:14:30 CPU %user %nice %system %iowait %steal
%idle
15:14:31 all 0.06 0.00 1.00 45.03 0.00
53.91
15:14:32 all 0.06 0.00 0.94 41.28 0.00
57.72
15:14:33 all 0.06 0.00 1.25 45.78 0.00
52.91
15:14:34 all 0.00 0.00 1.06 40.07 0.00
58.86
15:14:35 all 0.19 0.00 1.38 41.04 0.00
57.39
Average: all 0.08 0.00 1.13 42.64 0.00
56.16
and the system load shows very high
top - 15:19:15 up 34 days, 41 min, 2 users, load average: 13.49, 13.62,
13.83
From 'atop'
one of the CPUs shows this
CPU | sys 7% | user 1% | irq 2% | idle 1394% | wait
195% | steal 0% | guest 0% | ipc initial | cycl initial |
curf 806MHz | curscal ?%
On the OSD nodes, don't see much %utilization of the disks.
RBD caching values are default.
Are we overlooking some configuration item ?
Thanks and Regards,
At
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:49 p.m.
for certain CPU architecture, disable spectre and meltdown mitigations. (be certain
network to physical nodes is secure from internet access) (use apt proxy, http(s), curl
proxy servers)
Try to toggle on or off the physical on disk cache. (raid controller command)
^I had same issue, doing both of these fixed it. In my case the disks I had needed on disk
cache hard set to ‘on’. raid card default was not good. (be sure to have diverse power and
UPS protection if needed to run on disk cache on) (good RAID. battery if using raid cache
improves perf.)
to see the perf impact of spec. and melt. mitigation vs. off, run: dd if=/dev/zero
of=/dev/null
^i run for 5 seconds and then ctl+c
will show a max north bridge ops.
to see the difference in await and IOPs when toggle RAID card features and on disk cache I
run: iostat -xtc 2
and use fio to generate disk load for testing IOPs. (google fio example commands)
^south bridge +raid controller to disks ops and latency.
-Edward Kalk
Datacenter Virtualization
Performance Engineering
Socket Telecom
Columbia, MO, USA
ekalk(a)socket.net
On Nov 12, 2020, at 4:45 AM, athreyavc
<athreyavc(a)gmail.com> wrote:
Jumbo frames enabled and MTU is 9000
5 p.m.
Hi,
Thanks for the email, But we are not using RAID at all, we are using HBAs
LSI HBA 9400-8e. Eash HDD is configured as an OSD.
On Thu, Nov 12, 2020 at 12:19 PM Edward kalk <ekalk(a)socket.net> wrote:
for certain CPU architecture, disable spectre and
meltdown mitigations.
(be certain network to physical nodes is secure from internet access) (use
apt proxy, http(s), curl proxy servers)
Try to toggle on or off the physical on disk cache. (raid controller
command)
^I had same issue, doing both of these fixed it. In my case the disks I
had needed on disk cache hard set to ‘on’. raid card default was not good.
(be sure to have diverse power and UPS protection if needed to run on disk
cache on) (good RAID. battery if using raid cache improves perf.)
to see the perf impact of spec. and melt. mitigation vs. off, run: dd
if=/dev/zero of=/dev/null
^i run for 5 seconds and then ctl+c
will show a max north bridge ops.
to see the difference in await and IOPs when toggle RAID card features and
on disk cache I run: iostat -xtc 2
and use fio to generate disk load for testing IOPs. (google fio example
commands)
^south bridge +raid controller to disks ops and latency.
-Edward Kalk
Datacenter Virtualization
Performance Engineering
Socket Telecom
Columbia, MO, USA
ekalk(a)socket.net
On Nov 12, 2020, at 4:45 AM, athreyavc
<athreyavc(a)gmail.com> wrote:
Jumbo frames enabled and MTU is 9000
17 Nov
17 Nov
8:09 p.m.
I disabled the CephX authentication now. Though the Performance is Slightly
better, it is not yet there.
Are there any other recommendations for all HDD ceph clusters ?
From another thread
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/DFHXXN4KKI5…
*In our test based v15.2.2, i found
osd_numa_prefer_iface/osd_numa_auto_affinity make onlyehalf cpu used. for
4K RW, it make performance drop much. So you can check this whetheroccur.*
I do see "set_numa_affinity unable to identify cluster interface" alerts.
But I am not sure that is a cause for concern.
Thanks and regards,
Athreya
On Thu, Nov 12, 2020 at 1:30 PM athreyavc <athreyavc(a)gmail.com> wrote:
Hi,
Thanks for the email, But we are not using RAID at all, we are using HBAs
LSI HBA 9400-8e. Eash HDD is configured as an OSD.
On Thu, Nov 12, 2020 at 12:19 PM Edward kalk <ekalk(a)socket.net> wrote:
> for certain CPU architecture, disable spectre and meltdown mitigations.
> (be certain network to physical nodes is secure from internet access) (use
> apt proxy, http(s), curl proxy servers)
> Try to toggle on or off the physical on disk cache. (raid controller
> command)
> ^I had same issue, doing both of these fixed it. In my case the disks I
> had needed on disk cache hard set to ‘on’. raid card default was not good.
> (be sure to have diverse power and UPS protection if needed to run on disk
> cache on) (good RAID. battery if using raid cache improves perf.)
>
> to see the perf impact of spec. and melt. mitigation vs. off, run: dd
> if=/dev/zero of=/dev/null
> ^i run for 5 seconds and then ctl+c
> will show a max north bridge ops.
>
> to see the difference in await and IOPs when toggle RAID card features
> and on disk cache I run: iostat -xtc 2
> and use fio to generate disk load for testing IOPs. (google fio example
> commands)
> ^south bridge +raid controller to disks ops and latency.
>
> -Edward Kalk
> Datacenter Virtualization
> Performance Engineering
> Socket Telecom
> Columbia, MO, USA
> ekalk(a)socket.net
>
> > On Nov 12, 2020, at 4:45 AM, athreyavc <athreyavc(a)gmail.com> wrote:
> >
> > Jumbo frames enabled and MTU is 9000
>
>
9:14 p.m.
I am not sure any configuration tuning would help here.
The bottleneck is on HDD. In my case, I have a SSD for
WAL/DB and it provides pretty good write performance.
The part I don't quite understand in your case is that,
random read is quite fast. Due to the HDD seeking latency,
the random read is normally slow. Not sure how it's so fast
in your case.
Tony
-----Original Message-----
From: athreyavc <athreyavc(a)gmail.com>
Sent: Tuesday, November 17, 2020 8:40 AM
Cc: ceph-users <ceph-users(a)ceph.io>
Subject: [ceph-users] Re: Ceph RBD - High IOWait during the Writes
I disabled the CephX authentication now. Though the Performance is
Slightly better, it is not yet there.
Are there any other recommendations for all HDD ceph clusters ?
From another thread
https://lists.ceph.io/hyperkitty/list/ceph-
users(a)ceph.io/thread/DFHXXN4KKI5PS7LYPZJO4GYHU67JYVVL/
*In our test based v15.2.2, i found
osd_numa_prefer_iface/osd_numa_auto_affinity make onlyehalf cpu used.
for 4K RW, it make performance drop much. So you can check this
whetheroccur.*
I do see "set_numa_affinity unable to identify cluster interface" alerts.
But I am not sure that is a cause for concern.
Thanks and regards,
Athreya
On Thu, Nov 12, 2020 at 1:30 PM athreyavc <athreyavc(a)gmail.com> wrote:
Hi,
Thanks for the email, But we are not using RAID at all, we are using
HBAs LSI HBA 9400-8e. Eash HDD is configured as an OSD.
On Thu, Nov 12, 2020 at 12:19 PM Edward kalk <ekalk(a)socket.net> wrote:
> for certain CPU architecture, disable spectre and meltdown
mitigations.
> (be certain network to physical nodes is
secure from internet access)
> (use apt proxy, http(s), curl proxy servers) Try to toggle on or off
> the physical on disk cache. (raid controller
> command)
> ^I had same issue, doing both of these fixed it. In my case the disks
> I had needed on disk cache hard set to ‘on’. raid card default was
not good.
> (be sure to have diverse power and UPS
protection if needed to run on
> disk cache on) (good RAID. battery if using raid cache improves
> perf.)
>
> to see the perf impact of spec. and melt. mitigation vs. off, run: dd
> if=/dev/zero of=/dev/null ^i run for 5 seconds and then ctl+c will
> show a max north bridge ops.
>
> to see the difference in await and IOPs when toggle RAID card
> features and on disk cache I run: iostat -xtc 2 and use fio to
> generate disk load for testing IOPs. (google fio example
> commands)
> ^south bridge +raid controller to disks ops and latency.
>
> -Edward Kalk
> Datacenter Virtualization
> Performance Engineering
> Socket Telecom
> Columbia, MO, USA
> ekalk(a)socket.net
>
> > On Nov 12, 2020, at 4:45 AM, athreyavc <athreyavc(a)gmail.com> wrote:
> >
> > Jumbo frames enabled and MTU is 9000
>
>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
1286
days inactive
1293
days old
5 comments
4 participants
participants (4)
-
athreyavc -
athreyavc＠gmail.com
-
Edward kalk -
Tony Liu