Hi both, Thanks for both quick replies, and both (of course) 100% spot-on! With 4k, IOPS is around 18530 :-) Thank you both, and apologies for the noise! Best, MJ On 19/01/2021 14:57, Marc Roos wrote:
You should test with 4k not 4M.
-----Original Message----- From: mj <lists@merit.unu.edu> Sent: 19 January 2021 14:33 To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Samsung PM883 3.84TB SSD performance
Hi,
We purchased Samsung PM883 3.84TB SSDs to be used as BlueStore SSDs in our cluster.
I ran some benchmarks (write cache disabled, and with -sync=1 and -fsync=1) but with my little understandig, the results look terribly bad. (iops around 82!)
Reading pages like yourcmc.ru, we should at least aim for > 1000s IOPS. (like 10000 up to 20000)
Here is our complete output:
root@pve:~# hdparm -W 0 /dev/sdd
/dev/sdd: setting drive write-caching to 0 (off) write-caching = 0 (off) root@pve:~# fio -ioengine=libaio -fsync=1 -invalidate=1 -name=test - bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=32 fio-2.16 Starting 1 process Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=3268778: Tue Jan 19 14:13:44 2021 write: io=19728MB, bw=336674KB/s, iops=82, runt= 60003msec slat (usec): min=1853, max=4652, avg=2428.94, stdev=361.92 clat (msec): min=9, max=652, avg=376.87, stdev=22.88 lat (msec): min=11, max=655, avg=379.30, stdev=22.94 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 367], 10.00th=[ 371], 20.00th=[ 371], | 30.00th=[ 371], 40.00th=[ 375], 50.00th=[ 375], 60.00th=[ 379], | 70.00th=[ 379], 80.00th=[ 383], 90.00th=[ 388], 95.00th=[ 392], | 99.00th=[ 404], 99.50th=[ 420], 99.90th=[ 611], 99.95th=[ 635], | 99.99th=[ 652] lat (msec) : 10=0.02%, 50=0.06%, 100=0.10%, 250=0.24%, 500=99.25% lat (msec) : 750=0.32% cpu : usr=2.95%, sys=28.30%, ctx=47835, majf=0, minf=26 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0% issued : total=r=0/w=4932/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs): WRITE: io=19728MB, aggrb=336674KB/s, minb=336674KB/s, maxb=336674KB/s, mint=60003msec, maxt=60003msec
Disk stats (read/write): sdd: ios=41/39330, merge=0/4995672, ticks=8/231816, in_queue=231440, util=79.43% root@pve:~# fio -ioengine=libaio -sync=1 -invalidate=1 -name=test - bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=32 fio-2.16 Starting 1 process Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=3270282: Tue Jan 19 14:14:58 2021 write: io=19468MB, bw=332248KB/s, iops=81, runt= 60001msec slat (usec): min=11658, max=14287, avg=12318.59, stdev=371.39 clat (usec): min=8, max=400078, avg=380931.63, stdev=19107.88 lat (msec): min=11, max=412, avg=393.25, stdev=19.16 clat percentiles (msec): | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 375], | 30.00th=[ 375], 40.00th=[ 379], 50.00th=[ 379], 60.00th=[ 383], | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 392], 95.00th=[ 396], | 99.00th=[ 396], 99.50th=[ 400], 99.90th=[ 400], 99.95th=[ 400], | 99.99th=[ 400] lat (usec) : 10=0.02% lat (msec) : 20=0.02%, 50=0.04%, 100=0.08%, 250=0.25%, 500=99.59% cpu : usr=2.22%, sys=30.01%, ctx=53455, majf=0, minf=25 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0% issued : total=r=0/w=4867/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs): WRITE: io=19468MB, aggrb=332248KB/s, minb=332248KB/s, maxb=332248KB/s, mint=60001msec, maxt=60001msec
Disk stats (read/write): sdd: ios=41/38830, merge=0/4931664, ticks=4/203944, in_queue=203700, util=79.55%
The disk seems to be properly detected properly by the OS:
[4145961.899584] scsi 0:0:16:0: Direct-Access ATA SAMSUNG MZ7LH3T8 904Q PQ: 0 ANSI: 6 [4145961.899596] scsi 0:0:16:0: SATA: handle(0x001a), sas_addr(0x500304801ef80883), phy(3), device_name(0x0000000000000000) [4145961.899598] scsi 0:0:16:0: enclosure logical id (0x500304801ef808bf), slot(3) [4145961.899599] scsi 0:0:16:0: enclosure level(0x0000), connector name( ) [4145961.899649] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) [4145961.900471] sd 0:0:16:0: Power-on or device reset occurred [4145961.900544] sd 0:0:16:0: Attached scsi generic sg3 type 0 [4145961.902367] sd 0:0:16:0: [sdd] 7501476528 512-byte logical blocks: (3.84 TB/3.49 TiB) [4145961.902369] sd 0:0:16:0: [sdd] 4096-byte physical blocks [4145961.904264] sd 0:0:16:0: [sdd] Write Protect is off [4145961.904266] sd 0:0:16:0: [sdd] Mode Sense: 9b 00 10 08 [4145961.904756] sd 0:0:16:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [4145961.915996] sd 0:0:16:0: [sdd] Attached SCSI disk
Anyone with a idea what we could be doing wrong? Or are these disks really unsuitable for OSD use?
MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io