screenshot attached showing the IOPs and LAtency from iostat -xtc 2 On 8/14/2020 9:09 AM, Ed Kalk wrote:
ubuntu@ubuntu:/mnt$ sudo fio --filename=/mnt/sda1/file1.fio:/mnt/sdb1/file2.fio:/mnt/sdc1/file3.fio:/mnt/sdd1/file4.fio:/mnt/sde1/file5.fio:/mnt/sdf1/file6.fio:/mnt/sdg1/file7.fio:/mnt/sdh1/file8.fio:/mnt/sdi1/file9.fio:/mnt/sdj1/file10.fio:/mnt/sdk1/file11.fio:/mnt/sdl1/file12.fio --size=2GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=500 --numjobs=12 --time_based --group_reporting --name=iops-test-job
^generate fio to all disks
----- # iostat -xtc 2
^monitor IOPs to all disks -----
^how to test “fio” to multiple disks and monitor via “iostat -xtc 2“. I booted via live CD, made test partitions, file systems and mounted. disks took a while to initialize and settle in after filesystem create. monitored via iostat till all disks were quiet. then ran fio to test.
we observed the fist 6 disks (seagate constellation v1 SED) have a problem, but next 6 don’t. (seagate constellation v2 non-SED)
The “w_await” shows latency is 100x on v1’s, this was a reproduction of how they performed in ceph.
***suspecting disk firmware, will try updating them.
-Edward Kalk
On Aug 9, 2020, at 10:44 AM, Edward kalk <ekalk@socket.net> wrote:
> On Aug 8, 2020, at 6:56 PM, Anthony D'Atri
<anthony.datri@gmail.com> wrote:
> > Ugh, that looks like a pretty old HBA (and system). > > On an R730xd I found an H330 mini to be something of a bottleneck with 24x SATA SSD. > > o Check your drives for available firmware updates > > o Check your HBA for a firmware update > > o Check that your HBA’s firmware is supported with your kernel’s driver version — there should be a matrix at Dell somewhere > > o If you have a BBU/supercap, check its status (perccli /c0 … ) > > o Check dmesg/logs for HBA errors > > o Check smartctl -a output on the drives, look for wear, reallocated sectors, errors > >>> On Aug 8, 2020, at 2:55 PM, Edward kalk <ekalk@socket.net> wrote: >> R0, the controller config. (faux-JBOD) >> Single disk group. NoN-Raid. >> >> -Ed >> >>>> On Aug 8, 2020, at 9:20 AM, Anthony D'Atri <anthony.datri@gmail.com> wrote: >>> By R0 you mean hdparm ? >>> >>>> On Aug 8, 2020, at 3:49 AM, Edward kalk <ekalk@socket.net> wrote: >>>> >>>> Im getting poor performance with 5 of my OSDs, Seagate Constellation ES SED (1) 10k SAS 2TB 3.5 drives. >>>> disk write latency keeps drifting high , >>>> 100ms-230ms on writes. the other 30 OSDs are performing well. avg latency 10-20ms >>>> >>>> We observe stats via “iostat -xtc 2” on CEPH server. w_await showing writes to disk latency. >>>> >>>> **anyone else have poor latency on SEDs? >>>> >>>> The disks were included on a server we acquired. Didn’t know
Thank you for the reply Anthony. Yes the R510 is of a previous gen. The H700 isn’t my top suspect though. It’s been clocked to pull 15,000+ iops (with SSD). I’m familiar with bottlenecking an hba with 24 disk chassis. I only have 6 ceph OSDs on the H700. (this system should be overkill for that. (ejected all other disks, only 8 inserted) I’ll try investigating for FW updates on disks and hba. will also take another look at disk health via smart. they are repurposed disks and have seen action to say the least. thanks. -Ed they would be SEDs.
>>>> I’ve read that seagate SED vs. non SED shouldn’t perform differently, but looks like they are awful. All the SEDs are in the newly added CEPH node. A Dell R510 with H700 raid card. all disks in entire cluster are R0, direct write to disk, no cache. >>>> The new server and disk control spec out at about 10x the ability and IOPs of other servers, so I suspect the SEDs. >>>> >>>> -Ed >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io -- Thank you for your time,
Edward H. Kalk IV Information Technology Dept. Server Specialist Datacenter Virtualization and Storage Systems Socket Telecom, LLC. 2703 Clark Lane Columbia, MO 65202 573-817-0000 or 800-socket3 X218 --
-- Thank you for your time, Edward H. Kalk IV Information Technology Dept. Server Specialist Datacenter Virtualization and Storage Systems Socket Telecom, LLC. 2703 Clark Lane Columbia, MO 65202 573-817-0000 or 800-socket3 X218