If you are asking, maybe run this? [global] ioengine=libaio invalidate=1 ramp_time=30 iodepth=1 runtime=180 time_based direct=1 filename=/dev/sdf [write-4k-seq] stonewall bs=4k rw=write #write_bw_log=sdx-4k-write-seq.results #write_iops_log=sdx-4k-write-seq.results [randwrite-4k-seq] stonewall bs=4k rw=randwrite #write_bw_log=sdx-4k-randwrite-seq.results #write_iops_log=sdx-4k-randwrite-seq.results [read-4k-seq] stonewall bs=4k rw=read #write_bw_log=sdx-4k-read-seq.results #write_iops_log=sdx-4k-read-seq.results [randread-4k-seq] stonewall bs=4k rw=randread #write_bw_log=sdx-4k-randread-seq.results #write_iops_log=sdx-4k-randread-seq.results [rw-4k-seq] stonewall bs=4k rw=rw #write_bw_log=sdx-4k-rw-seq.results #write_iops_log=sdx-4k-rw-seq.results [randrw-4k-seq] stonewall bs=4k rw=randrw #write_bw_log=sdx-4k-randrw-seq.results #write_iops_log=sdx-4k-randrw-seq.results [write-128k-seq] stonewall bs=128k rw=write #write_bw_log=sdx-128k-write-seq.results #write_iops_log=sdx-128k-write-seq.results [randwrite-128k-seq] stonewall bs=128k rw=randwrite #write_bw_log=sdx-128k-randwrite-seq.results #write_iops_log=sdx-128k-randwrite-seq.results [read-128k-seq] stonewall bs=128k rw=read #write_bw_log=sdx-128k-read-seq.results #write_iops_log=sdx-128k-read-seq.results [randread-128k-seq] stonewall bs=128k rw=randread #write_bw_log=sdx-128k-randread-seq.results #write_iops_log=sdx-128k-randread-seq.results [rw-128k-seq] stonewall bs=128k rw=rw #write_bw_log=sdx-128k-rw-seq.results #write_iops_log=sdx-128k-rw-seq.results [randrw-128k-seq] stonewall bs=128k rw=randrw #write_bw_log=sdx-128k-randrw-seq.results #write_iops_log=sdx-128k-randrw-seq.results [write-1024k-seq] stonewall bs=1024k rw=write #write_bw_log=sdx-1024k-write-seq.results #write_iops_log=sdx-1024k-write-seq.results [randwrite-1024k-seq] stonewall bs=1024k rw=randwrite #write_bw_log=sdx-1024k-randwrite-seq.results #write_iops_log=sdx-1024k-randwrite-seq.results [read-1024k-seq] stonewall bs=1024k rw=read #write_bw_log=sdx-1024k-read-seq.results #write_iops_log=sdx-1024k-read-seq.results [randread-1024k-seq] stonewall bs=1024k rw=randread #write_bw_log=sdx-1024k-randread-seq.results #write_iops_log=sdx-1024k-randread-seq.results [rw-1024k-seq] stonewall bs=1024k rw=rw #write_bw_log=sdx-1024k-rw-seq.results #write_iops_log=sdx-1024k-rw-seq.results [randrw-1024k-seq] stonewall bs=1024k rw=randrw #write_bw_log=sdx-1024k-randrw-seq.results #write_iops_log=sdx-1024k-randrw-seq.results -----Original Message----- From: mj [mailto:lists@merit.unu.edu] Sent: 06 March 2020 10:01 To: ceph-users@ceph.io Subject: [ceph-users] Re: Ceph Performance of Micron 5210 SATA? Last monday I performed a quick test with those two disks already, probably not that relevant, but posting it anyway: I created a two-disk ceph 'cluster' on just the one local node, and ran the following:
root@ceph:~# rados bench -p scbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ceph_48906 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 107 91 363.961 364 0.0546517 0.155121 2 16 206 190 379.948 396 0.1529 0.159227 3 16 324 308 410.614 472 0.0972163 0.151421 4 16 458 442 441.942 536 0.0484349 0.141799 5 16 590 574 459.141 528 0.0445051 0.136922 6 16 727 711 473.941 548 0.181066 0.134468 7 16 856 840 479.941 516 0.187683 0.133199 8 16 970 954 476.942 456 0.070753 0.132642 9 16 1089 1073 476.831 476 0.193608 0.133754 10 16 1214 1198 479.142 500 0.0999212 0.132529 Total time run: 10.097218 Total writes made: 1215 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 481.321 Stddev Bandwidth: 60.481 Max bandwidth (MB/sec): 548 Min bandwidth (MB/sec): 364 Average IOPS: 120 Stddev IOPS: 15 Max IOPS: 137 Min IOPS: 91 Average Latency(s): 0.132889 Stddev Latency(s): 0.0645579 Max latency(s): 0.336118 Min latency(s): 0.0117049
Do let me know what else you'd want me to do. MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, So, ran the fio commands, and pasted output (as it's quite a lot) here: https://pastebin.com/KWYEu9uU I hope someone here can draw some conclusions from this output... An observation: We are using proxmox (5.4), and it displays the WEAR level as "N/A", which is unfortunate... :-( Tried upgrading to 6: the same, still no wearout in the GUI. Here is smartctl output:
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.18-26-pve] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION === Device Model: Micron_5210_MTFDDAK3T8QDE Serial Number: 200426196EEC LU WWN Device Id: 5 00a075 126196eec Firmware Version: D2MU404 User Capacity: 3,840,755,982,336 bytes [3.84 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Mar 9 15:26:00 2020 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (13395) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 24) minutes. Conveyance self-test routine recommended polling time: ( 3) minutes. SCT capabilities: (0x0035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported.
SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 100 100 050 Pre-fail Always - 0 5 Reallocated_Sector_Ct 0x0032 100 100 001 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 5 12 Power_Cycle_Count 0x0032 100 100 001 Old_age Always - 3 170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0 171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 172 Unknown_Attribute 0x0032 100 100 001 Old_age Always - 0 173 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 1 174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 1 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 3 194 Temperature_Celsius 0x0022 071 053 000 Old_age Always - 29 (Min/Max 19/47) 195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0 202 Unknown_SSD_Attribute 0x0030 100 100 001 Old_age Offline - 0 206 Unknown_SSD_Attribute 0x000e 100 100 000 Old_age Always - 0 246 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 1234755968 247 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 40518725 248 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 2271671 180 Unused_Rsvd_Blk_Cnt_Tot 0x0033 001 001 000 Pre-fail Always - 10948 210 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
SMART Error Log Version: 1 No Errors Logged
SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
I have pasted the full "smartctl /dev/sda -P showall" output here: https://pastebin.com/XUxtLA8d The attribute "202 Unknown_SSD_Attribute" is the "Estimated life remaining". Micron's own management tool (msecli, linux version available) works very good. What can be done to make proxmox display the WEAR level? Best, MJ On 06/03/2020 10:53, Marc Roos wrote:
If you are asking, maybe run this?
[global] ioengine=libaio invalidate=1 ramp_time=30 iodepth=1 runtime=180 time_based direct=1 filename=/dev/sdf
[write-4k-seq] stonewall bs=4k rw=write #write_bw_log=sdx-4k-write-seq.results #write_iops_log=sdx-4k-write-seq.results
[randwrite-4k-seq] stonewall bs=4k rw=randwrite #write_bw_log=sdx-4k-randwrite-seq.results #write_iops_log=sdx-4k-randwrite-seq.results
[read-4k-seq] stonewall bs=4k rw=read #write_bw_log=sdx-4k-read-seq.results #write_iops_log=sdx-4k-read-seq.results
[randread-4k-seq] stonewall bs=4k rw=randread #write_bw_log=sdx-4k-randread-seq.results #write_iops_log=sdx-4k-randread-seq.results
[rw-4k-seq] stonewall bs=4k rw=rw #write_bw_log=sdx-4k-rw-seq.results #write_iops_log=sdx-4k-rw-seq.results
[randrw-4k-seq] stonewall bs=4k rw=randrw #write_bw_log=sdx-4k-randrw-seq.results #write_iops_log=sdx-4k-randrw-seq.results
[write-128k-seq] stonewall bs=128k rw=write #write_bw_log=sdx-128k-write-seq.results #write_iops_log=sdx-128k-write-seq.results
[randwrite-128k-seq] stonewall bs=128k rw=randwrite #write_bw_log=sdx-128k-randwrite-seq.results #write_iops_log=sdx-128k-randwrite-seq.results
[read-128k-seq] stonewall bs=128k rw=read #write_bw_log=sdx-128k-read-seq.results #write_iops_log=sdx-128k-read-seq.results
[randread-128k-seq] stonewall bs=128k rw=randread #write_bw_log=sdx-128k-randread-seq.results #write_iops_log=sdx-128k-randread-seq.results
[rw-128k-seq] stonewall bs=128k rw=rw #write_bw_log=sdx-128k-rw-seq.results #write_iops_log=sdx-128k-rw-seq.results
[randrw-128k-seq] stonewall bs=128k rw=randrw #write_bw_log=sdx-128k-randrw-seq.results #write_iops_log=sdx-128k-randrw-seq.results
[write-1024k-seq] stonewall bs=1024k rw=write #write_bw_log=sdx-1024k-write-seq.results #write_iops_log=sdx-1024k-write-seq.results
[randwrite-1024k-seq] stonewall bs=1024k rw=randwrite #write_bw_log=sdx-1024k-randwrite-seq.results #write_iops_log=sdx-1024k-randwrite-seq.results
[read-1024k-seq] stonewall bs=1024k rw=read #write_bw_log=sdx-1024k-read-seq.results #write_iops_log=sdx-1024k-read-seq.results
[randread-1024k-seq] stonewall bs=1024k rw=randread #write_bw_log=sdx-1024k-randread-seq.results #write_iops_log=sdx-1024k-randread-seq.results
[rw-1024k-seq] stonewall bs=1024k rw=rw #write_bw_log=sdx-1024k-rw-seq.results #write_iops_log=sdx-1024k-rw-seq.results
[randrw-1024k-seq] stonewall bs=1024k rw=randrw #write_bw_log=sdx-1024k-randrw-seq.results #write_iops_log=sdx-1024k-randrw-seq.results
-----Original Message----- From: mj [mailto:lists@merit.unu.edu] Sent: 06 March 2020 10:01 To: ceph-users@ceph.io Subject: [ceph-users] Re: Ceph Performance of Micron 5210 SATA?
Last monday I performed a quick test with those two disks already, probably not that relevant, but posting it anyway:
I created a two-disk ceph 'cluster' on just the one local node, and ran the following:
root@ceph:~# rados bench -p scbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ceph_48906 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 107 91 363.961 364 0.0546517 0.155121 2 16 206 190 379.948 396 0.1529 0.159227 3 16 324 308 410.614 472 0.0972163 0.151421 4 16 458 442 441.942 536 0.0484349 0.141799 5 16 590 574 459.141 528 0.0445051 0.136922 6 16 727 711 473.941 548 0.181066 0.134468 7 16 856 840 479.941 516 0.187683 0.133199 8 16 970 954 476.942 456 0.070753 0.132642 9 16 1089 1073 476.831 476 0.193608 0.133754 10 16 1214 1198 479.142 500 0.0999212 0.132529 Total time run: 10.097218 Total writes made: 1215 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 481.321 Stddev Bandwidth: 60.481 Max bandwidth (MB/sec): 548 Min bandwidth (MB/sec): 364 Average IOPS: 120 Stddev IOPS: 15 Max IOPS: 137 Min IOPS: 91 Average Latency(s): 0.132889 Stddev Latency(s): 0.0645579 Max latency(s): 0.336118 Min latency(s): 0.0117049
Do let me know what else you'd want me to do.
MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
An observation: We are using proxmox (5.4), and it displays the WEAR level as "N/A", which is unfortunate... :-( Tried upgrading to 6: the same, still no wearout in the GUI.
Here is smartctl output:
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.18-26-pve] (local build)
Old release, probably came with your OS.
Device Model: Micron_5210_MTFDDAK3T8QDE Firmware Version: D2MU404
Make sure you run the very latest public firmware from Micron. Ask them if this is current.
Device is: Not in smartctl database [for details use: -P showall]
There you go.
202 Unknown_SSD_Attribute 0x0030 100 100 001 Old_age Offline - 0
SMART is not consistently implemented. Some manufacturers / models use different attribute numbers. Some eg. will present a counter as lifetime remaining, others as lifetime consumed. And the calculations used to present lifetime percentages varies among models and may not be the best predictor of failure, eg. right before the drive is going to fail the number may change non-linearly. But it’s better than nothing, and I digress. Some tools match the label string returned by smartctl instead of or in addition to the number. smartctl will default strings for certain attributes (which sometimes gives wrong answers) if there isn’t an exact match to the actual drive model. So I bet if you were to add "Micron_5210_(EE|MT)FDDAK(1T9|3T8|7T6)QDE|" // tested with Micron_5210_MTFDDAK1T9QDC to your drivedb.h stanza for the 5100/5200, proxmox might pick it up. Your 6.6 drivedb.h may not even have a 5100/5200 entry. The latest upstream is here https://www.smartmontools.org/browser/trunk/smartmontools/drivedb.h But beware that IIRC the bits at the beginning of the file changed with 7.0, so you won’t be able to inhale the latest file in toto, and it doesn’t actually list the 5210. Someone from Micron submitted an omnibus update that included it, but it conflicted with the existing 5100/5200 entry, and I haven’t gotten around to submitting a reconciled version. With the caveat that I’m composing this in my MUA and have not tested it, you might try adding the below to your current drivedb.h. Now that I again have a 5100 Pro to test against I need to get off my butt and submit a diff upstream so it gets into 7.2. Be careful with the syntax, it’s easy to get the |{,} characters wrong. // Reference: https://www.micron.com/resource-details/feec878a-265e-49a7-8086-15137c5f9011 // TN-FD-34: 5100 SSD SMART Implementation { "Micron 5100 Pro / 5200 SSDs", "(Micron_5100_)?(EE|MT)FDDA[KV](240|480|960|1T9|3T8|7T6)T(BY|CB|CC)|" // Matches both stock and Dell OEM "Micron_5210_(EE|MT)FDDAK(1T9|3T8|7T6)QDE|" // tested with Micron_5210_MTFDDAK1T9QDC "(Micron_5200_)?MTFDDAK(480|960|1T9|3T8|7T6)TD(C|D|N)", // tested with Micron_5200_MTFDDAK3T8TDD/D1MU505 "", "", //"-v 1,raw48,Raw_Read_Error_Rate " //"-v 5,raw48,Reallocated_Block_Count " //"-v 9,raw24(raw8),Power_On_Hours " // raw24(raw8)?? //"-v 12,raw48,Power_Cycle_Count " "-v 170,raw48,Reserved_Block_Pct " // Percentage of remaining reserved blocks available "-v 171,raw48,Program_Fail_Count " "-v 172,raw48,Erase_Fail_Count " "-v 173,raw48,Avg_Block-Erase_Count " "-v 174,raw48,Unexpect_Power_Loss_Ct " //"-v 180,raw48,Reserved_Block_Count " // absolute count of remaining reserved blocks available "-v 183,raw48,SATA_Int_Downshift_Ct " // SATA speed downshift count //"-v 184,raw48,Error_Correction_Count " //"-v 187,raw48,Reported_Uncorrect " // Number of UECC correction failures //"-v 188,raw48,Command_Timeouts " //"-v 194,tempminmax,Temperature_Celsius " // 100 - degrees C, wraps: 101 reported as 255 //"-v 195,raw48,Cumulativ_Corrected_ECC " //"-v 196,raw48,Reallocation_Event_Ct " //"-v 197,raw48,Current_Pending_Sector " // Use the raw value //"-v 198,raw48,Offline_Uncorrectable " // Use the raw value //"-v 199,raw48,UDMA_CRC_Error_Count " // Use the raw value "-v 202,raw48,Percent_Lifetime_Remain " // Remaining endurance, trips at 10% "-v 206,raw48,Write_Error_Rate " "-v 210,raw48,RAIN_Success_Recovered " // Total number of NAND pages recovered by RAIN "-v 211,raw48,Integ_Scan_Complete_Cnt " // Number of periodic data integrity scans completed "-v 212,raw48,Integ_Scan_Folding_Cnt " // Number of blocks reallocated by integrity scans "-v 213,raw48,Integ_Scan_Progress " // Current is percentage, raw is absolute number of superblocks scanned by the current in\ tegrity scan "-v 247,raw48,Host_Program_Page_Count " "-v 248,raw48,Bckgnd_Program_Page_Cnt" },
Hi Mourik Jan,
So, ran the fio commands, and pasted output (as it's quite a lot) here:
I hope someone here can draw some conclusions from this output...
Now you know, it sort of performs similar to other enterprise drives. And you know your ceph solution will never perform beyond this ;) And if your ceph overhead is more than standard, you maybe able be to identify at a later time where the problem is. With the 4k, 128k, 1024k and 4096k test results I am trying to have data, that will explain ceph performance under different use cases. This is a result from a samsung sm863 1.92TB write-4k-seq: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 randwrite-4k-seq: (g=1): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 read-4k-seq: (g=2): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 randread-4k-seq: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 rw-4k-seq: (g=4): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 randrw-4k-seq: (g=5): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 write-128k-seq: (g=6): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 randwrite-128k-seq: (g=7): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 read-128k-seq: (g=8): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 randread-128k-seq: (g=9): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 rw-128k-seq: (g=10): rw=rw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 randrw-128k-seq: (g=11): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 write-1024k-seq: (g=12): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 randwrite-1024k-seq: (g=13): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 read-1024k-seq: (g=14): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 randread-1024k-seq: (g=15): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 rw-1024k-seq: (g=16): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 randrw-1024k-seq: (g=17): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 write-4096k-seq: (g=18): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 randwrite-4096k-seq: (g=19): rw=randwrite, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 read-4096k-seq: (g=20): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 randread-4096k-seq: (g=21): rw=randread, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 rw-4096k-seq: (g=22): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 randrw-4096k-seq: (g=23): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.1 Starting 24 processes write-4k-seq: (groupid=0, jobs=1): err= 0: pid=11515: Mon Aug 26 15:45:05 2019 write: IOPS=30.8k, BW=120MiB/s (126MB/s)(21.1GiB/180001msec) slat (nsec): min=3762, max=75591, avg=7173.31, stdev=3526.20 clat (nsec): min=440, max=1057.9k, avg=23654.15, stdev=4570.40 lat (usec): min=23, max=1062, avg=31.02, stdev= 4.94 clat percentiles (nsec): | 1.00th=[15680], 5.00th=[19840], 10.00th=[21120], 20.00th=[21888], | 30.00th=[22144], 40.00th=[22400], 50.00th=[22656], 60.00th=[22912], | 70.00th=[23168], 80.00th=[24448], 90.00th=[28544], 95.00th=[29568], | 99.00th=[47360], 99.50th=[49920], 99.90th=[51968], 99.95th=[52480], | 99.99th=[84480] bw ( KiB/s): min=117983, max=146636, per=100.00%, avg=123958.14, stdev=7366.26, samples=360 iops : min=29495, max=36659, avg=30989.17, stdev=1841.60, samples=360 lat (nsec) : 500=0.01%, 750=0.01% lat (usec) : 10=0.01%, 20=5.10%, 50=94.45%, 100=0.45%, 250=0.01% lat (usec) : 500=0.01%, 750=0.01% lat (msec) : 2=0.01% cpu : usr=5.57%, sys=25.51%, ctx=8452083, majf=0, minf=7 IO depths : 1=117.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=0,5538482,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randwrite-4k-seq: (groupid=1, jobs=1): err= 0: pid=11560: Mon Aug 26 15:45:05 2019 write: IOPS=29.5k, BW=115MiB/s (121MB/s)(20.2GiB/180001msec) slat (usec): min=3, max=126, avg= 8.06, stdev= 3.59 clat (nsec): min=457, max=419756, avg=23739.16, stdev=4377.60 lat (usec): min=23, max=428, avg=31.99, stdev= 4.78 clat percentiles (usec): | 1.00th=[ 16], 5.00th=[ 21], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 24], 90.00th=[ 29], 95.00th=[ 30], | 99.00th=[ 48], 99.50th=[ 50], 99.90th=[ 52], 99.95th=[ 52], | 99.99th=[ 101] bw ( KiB/s): min=112000, max=144697, per=100.00%, avg=118764.70, stdev=6220.53, samples=360 iops : min=28000, max=36174, avg=29690.82, stdev=1555.19, samples=360 lat (nsec) : 500=0.01%, 750=0.01% lat (usec) : 10=0.01%, 20=3.33%, 50=96.16%, 100=0.50%, 250=0.01% lat (usec) : 500=0.01% cpu : usr=5.60%, sys=27.07%, ctx=8880261, majf=0, minf=5 IO depths : 1=119.3%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=0,5306473,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 read-4k-seq: (groupid=2, jobs=1): err= 0: pid=11589: Mon Aug 26 15:45:05 2019 read: IOPS=29.4k, BW=115MiB/s (121MB/s)(20.2GiB/180001msec) slat (usec): min=3, max=256, avg= 7.64, stdev= 3.53 clat (nsec): min=449, max=1125.7k, avg=24003.96, stdev=4493.67 lat (usec): min=23, max=1130, avg=31.81, stdev= 4.75 clat percentiles (nsec): | 1.00th=[15424], 5.00th=[20864], 10.00th=[21376], 20.00th=[21888], | 30.00th=[22144], 40.00th=[22144], 50.00th=[22400], 60.00th=[22656], | 70.00th=[22912], 80.00th=[28288], 90.00th=[29312], 95.00th=[30336], | 99.00th=[45312], 99.50th=[48896], 99.90th=[50944], 99.95th=[51456], | 99.99th=[55040] bw ( KiB/s): min=113771, max=149349, per=100.00%, avg=118606.15, stdev=5880.22, samples=360 iops : min=28442, max=37337, avg=29651.20, stdev=1470.06, samples=360 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=2.26%, 50=97.44%, 100=0.30% lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=4.94%, sys=27.39%, ctx=9280107, majf=0, minf=6 IO depths : 1=117.2%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=5299598,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randread-4k-seq: (groupid=3, jobs=1): err= 0: pid=12050: Mon Aug 26 15:45:05 2019 read: IOPS=24.5k, BW=95.9MiB/s (101MB/s)(16.9GiB/180001msec) slat (nsec): min=3693, max=76505, avg=5960.68, stdev=1820.53 clat (nsec): min=675, max=1052.6k, avg=33198.13, stdev=12212.51 lat (usec): min=27, max=1057, avg=39.34, stdev=12.55 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 31], | 70.00th=[ 33], 80.00th=[ 35], 90.00th=[ 40], 95.00th=[ 45], | 99.00th=[ 106], 99.50th=[ 110], 99.90th=[ 116], 99.95th=[ 118], | 99.99th=[ 123] bw ( KiB/s): min=83500, max=110084, per=100.00%, avg=98894.61, stdev=3387.20, samples=360 iops : min=20875, max=27521, avg=24723.30, stdev=846.78, samples=360 lat (nsec) : 750=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=96.78%, 100=1.84% lat (usec) : 250=1.37%, 750=0.01% lat (msec) : 2=0.01% cpu : usr=7.31%, sys=19.03%, ctx=4440118, majf=0, minf=6 IO depths : 1=116.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=4418876,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rw-4k-seq: (groupid=4, jobs=1): err= 0: pid=12073: Mon Aug 26 15:45:05 2019 read: IOPS=15.0k, BW=62.4MiB/s (65.4MB/s)(10.0GiB/180001msec) slat (nsec): min=3661, max=75395, avg=5554.24, stdev=2712.89 clat (nsec): min=487, max=143971, avg=23601.85, stdev=4187.61 lat (usec): min=22, max=148, avg=29.32, stdev= 4.52 clat percentiles (nsec): | 1.00th=[15296], 5.00th=[20352], 10.00th=[20608], 20.00th=[21120], | 30.00th=[21632], 40.00th=[21888], 50.00th=[22144], 60.00th=[22656], | 70.00th=[23680], 80.00th=[25728], 90.00th=[28800], 95.00th=[30592], | 99.00th=[40704], 99.50th=[41728], 99.90th=[45824], 99.95th=[48896], | 99.99th=[52992] bw ( KiB/s): min=57859, max=73862, per=100.00%, avg=64353.67, stdev=5203.87, samples=360 iops : min=14464, max=18465, avg=16088.08, stdev=1300.94, samples=360 write: IOPS=15.0k, BW=62.5MiB/s (65.5MB/s)(10.0GiB/180001msec) slat (nsec): min=3761, max=77015, avg=5723.77, stdev=2746.22 clat (nsec): min=459, max=579225, avg=24813.39, stdev=4312.80 lat (usec): min=23, max=584, avg=30.70, stdev= 4.74 clat percentiles (nsec): | 1.00th=[16512], 5.00th=[21120], 10.00th=[21632], 20.00th=[22144], | 30.00th=[22656], 40.00th=[22912], 50.00th=[23168], 60.00th=[23936], | 70.00th=[25472], 80.00th=[28032], 90.00th=[29824], 95.00th=[31360], | 99.00th=[41728], 99.50th=[42752], 99.90th=[47360], 99.95th=[50432], | 99.99th=[96768] bw ( KiB/s): min=57775, max=74101, per=100.00%, avg=64475.76, stdev=5298.07, samples=360 iops : min=14443, max=18525, avg=16118.61, stdev=1324.51, samples=360 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 10=0.01%, 20=2.56%, 50=97.39%, 100=0.04%, 250=0.01% lat (usec) : 750=0.01% cpu : usr=7.05%, sys=21.91%, ctx=7267832, majf=0, minf=6 IO depths : 1=116.3%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=2874379,2879839,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randrw-4k-seq: (groupid=5, jobs=1): err= 0: pid=12211: Mon Aug 26 15:45:05 2019 read: IOPS=13.8k, BW=53.0MiB/s (56.6MB/s)(9719MiB/180001msec) slat (usec): min=3, max=442, avg= 5.29, stdev= 1.73 clat (nsec): min=695, max=1834.6k, avg=33487.10, stdev=27390.90 lat (usec): min=27, max=1838, avg=38.94, stdev=27.53 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 30], | 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 38], 95.00th=[ 44], | 99.00th=[ 109], 99.50th=[ 116], 99.90th=[ 396], 99.95th=[ 457], | 99.99th=[ 881] bw ( KiB/s): min=49604, max=62173, per=100.00%, avg=55691.34, stdev=2474.35, samples=360 iops : min=12401, max=15543, avg=13922.50, stdev=618.60, samples=360 write: IOPS=13.8k, BW=54.0MiB/s (56.6MB/s)(9720MiB/180001msec) slat (nsec): min=3751, max=58621, avg=5456.94, stdev=1739.89 clat (usec): min=5, max=128, avg=25.19, stdev= 4.45 lat (usec): min=23, max=136, avg=30.81, stdev= 5.27 clat percentiles (nsec): | 1.00th=[20864], 5.00th=[21632], 10.00th=[21888], 20.00th=[22400], | 30.00th=[22912], 40.00th=[23168], 50.00th=[23680], 60.00th=[24192], | 70.00th=[25472], 80.00th=[26496], 90.00th=[30336], 95.00th=[36608], | 99.00th=[41728], 99.50th=[43776], 99.90th=[49408], 99.95th=[51968], | 99.99th=[61184] bw ( KiB/s): min=49244, max=61663, per=100.00%, avg=55697.41, stdev=2464.11, samples=360 iops : min=12311, max=15415, avg=13924.00, stdev=616.03, samples=360 lat (nsec) : 750=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.03%, 50=98.22%, 100=0.86% lat (usec) : 250=0.70%, 500=0.15%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=8.46%, sys=18.38%, ctx=4985736, majf=0, minf=6 IO depths : 1=117.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=2488163,2488417,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 write-128k-seq: (groupid=6, jobs=1): err= 0: pid=12242: Mon Aug 26 15:45:05 2019 write: IOPS=3160, BW=395MiB/s (414MB/s)(69.5GiB/180001msec) slat (nsec): min=16332, max=72183, avg=31691.27, stdev=3994.93 clat (usec): min=241, max=2658, avg=281.36, stdev=55.40 lat (usec): min=276, max=2692, avg=313.44, stdev=55.62 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[ 277], | 30.00th=[ 277], 40.00th=[ 277], 50.00th=[ 277], 60.00th=[ 281], | 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 289], 95.00th=[ 293], | 99.00th=[ 318], 99.50th=[ 326], 99.90th=[ 343], 99.95th=[ 2212], | 99.99th=[ 2606] bw ( KiB/s): min=398848, max=416129, per=100.00%, avg=407445.96, stdev=3324.34, samples=360 iops : min= 3116, max= 3251, avg=3182.76, stdev=25.94, samples=360 lat (usec) : 250=0.01%, 500=99.91%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.06% cpu : usr=4.72%, sys=11.74%, ctx=570098, majf=0, minf=7 IO depths : 1=116.4%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=0,568940,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randwrite-128k-seq: (groupid=7, jobs=1): err= 0: pid=12269: Mon Aug 26 15:45:05 2019 write: IOPS=3149, BW=394MiB/s (413MB/s)(69.2GiB/180001msec) slat (nsec): min=16078, max=73647, avg=31696.03, stdev=3965.26 clat (usec): min=236, max=2643, avg=281.93, stdev=50.94 lat (usec): min=275, max=2675, avg=314.03, stdev=51.17 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[ 277], | 30.00th=[ 277], 40.00th=[ 277], 50.00th=[ 277], 60.00th=[ 281], | 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 289], 95.00th=[ 297], | 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 379], 99.95th=[ 1729], | 99.99th=[ 2606] bw ( KiB/s): min=398848, max=415547, per=100.00%, avg=406058.61, stdev=3217.07, samples=360 iops : min= 3116, max= 3246, avg=3171.89, stdev=25.13, samples=360 lat (usec) : 250=0.01%, 500=99.92%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.03%, 4=0.04% cpu : usr=4.84%, sys=11.77%, ctx=568040, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=0,566991,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 read-128k-seq: (groupid=8, jobs=1): err= 0: pid=12297: Mon Aug 26 15:45:05 2019 read: IOPS=3258, BW=407MiB/s (427MB/s)(71.6GiB/180001msec) slat (usec): min=14, max=201, avg=29.15, stdev= 4.09 clat (usec): min=98, max=662, avg=274.56, stdev= 8.35 lat (usec): min=264, max=686, avg=304.11, stdev= 9.97 clat percentiles (usec): | 1.00th=[ 258], 5.00th=[ 262], 10.00th=[ 265], 20.00th=[ 269], | 30.00th=[ 273], 40.00th=[ 273], 50.00th=[ 277], 60.00th=[ 277], | 70.00th=[ 277], 80.00th=[ 281], 90.00th=[ 281], 95.00th=[ 285], | 99.00th=[ 302], 99.50th=[ 314], 99.90th=[ 322], 99.95th=[ 326], | 99.99th=[ 334] bw ( KiB/s): min=415225, max=440860, per=100.00%, avg=420009.72, stdev=3918.49, samples=360 iops : min= 3243, max= 3444, avg=3280.92, stdev=30.53, samples=360 lat (usec) : 100=0.01%, 250=0.05%, 500=99.95%, 750=0.01% cpu : usr=2.40%, sys=13.84%, ctx=587372, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=586463,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randread-128k-seq: (groupid=9, jobs=1): err= 0: pid=12316: Mon Aug 26 15:45:05 2019 read: IOPS=3029, BW=379MiB/s (397MB/s)(66.6GiB/180001msec) slat (usec): min=10, max=103, avg=28.55, stdev= 4.15 clat (usec): min=237, max=978, avg=297.79, stdev=30.25 lat (usec): min=272, max=1005, avg=326.75, stdev=30.75 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 269], 10.00th=[ 277], 20.00th=[ 281], | 30.00th=[ 281], 40.00th=[ 285], 50.00th=[ 285], 60.00th=[ 289], | 70.00th=[ 293], 80.00th=[ 310], 90.00th=[ 359], 95.00th=[ 371], | 99.00th=[ 379], 99.50th=[ 383], 99.90th=[ 400], 99.95th=[ 416], | 99.99th=[ 490] bw ( KiB/s): min=382976, max=396202, per=100.00%, avg=390503.28, stdev=2581.76, samples=360 iops : min= 2992, max= 3095, avg=3050.45, stdev=19.98, samples=360 lat (usec) : 250=0.01%, 500=99.99%, 750=0.01%, 1000=0.01% cpu : usr=2.36%, sys=12.93%, ctx=546079, majf=0, minf=5 IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=545272,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rw-128k-seq: (groupid=10, jobs=1): err= 0: pid=12422: Mon Aug 26 15:45:05 2019 read: IOPS=1406, BW=176MiB/s (184MB/s)(30.9GiB/180001msec) slat (nsec): min=13910, max=82839, avg=29020.55, stdev=4086.77 clat (usec): min=231, max=3707, avg=361.02, stdev=246.76 lat (usec): min=268, max=3739, avg=390.44, stdev=246.77 clat percentiles (usec): | 1.00th=[ 258], 5.00th=[ 262], 10.00th=[ 269], 20.00th=[ 273], | 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 277], 60.00th=[ 281], | 70.00th=[ 289], 80.00th=[ 302], 90.00th=[ 537], 95.00th=[ 930], | 99.00th=[ 1483], 99.50th=[ 1516], 99.90th=[ 1713], 99.95th=[ 1926], | 99.99th=[ 2343] bw ( KiB/s): min=131878, max=199741, per=100.00%, avg=181277.06, stdev=7492.15, samples=360 iops : min= 1030, max= 1560, avg=1415.71, stdev=58.46, samples=360 write: IOPS=1407, BW=176MiB/s (185MB/s)(30.9GiB/180001msec) slat (usec): min=15, max=462, avg=31.27, stdev= 4.11 clat (usec): min=243, max=2656, avg=282.69, stdev=55.31 lat (usec): min=276, max=2684, avg=314.36, stdev=55.53 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[ 277], | 30.00th=[ 277], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 281], | 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 289], 95.00th=[ 297], | 99.00th=[ 326], 99.50th=[ 330], 99.90th=[ 400], 99.95th=[ 2147], | 99.99th=[ 2606] bw ( KiB/s): min=133688, max=197841, per=100.00%, avg=181452.40, stdev=7546.36, samples=360 iops : min= 1044, max= 1545, avg=1417.08, stdev=58.93, samples=360 lat (usec) : 250=0.01%, 500=94.34%, 750=2.38%, 1000=0.97% lat (msec) : 2=2.26%, 4=0.05% cpu : usr=3.00%, sys=11.40%, ctx=507392, majf=0, minf=8 IO depths : 1=117.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=253133,253388,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randrw-128k-seq: (groupid=11, jobs=1): err= 0: pid=12481: Mon Aug 26 15:45:05 2019 read: IOPS=1420, BW=178MiB/s (186MB/s)(31.2GiB/180001msec) slat (nsec): min=15881, max=72688, avg=29081.99, stdev=4063.08 clat (usec): min=251, max=2443, avg=353.64, stdev=194.49 lat (usec): min=281, max=2466, avg=383.13, stdev=194.56 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 273], 10.00th=[ 277], 20.00th=[ 281], | 30.00th=[ 285], 40.00th=[ 285], 50.00th=[ 289], 60.00th=[ 293], | 70.00th=[ 306], 80.00th=[ 355], 90.00th=[ 502], 95.00th=[ 668], | 99.00th=[ 1369], 99.50th=[ 1467], 99.90th=[ 1844], 99.95th=[ 1876], | 99.99th=[ 1942] bw ( KiB/s): min=167936, max=199967, per=100.00%, avg=183163.65, stdev=5287.72, samples=360 iops : min= 1312, max= 1562, avg=1430.45, stdev=41.25, samples=360 write: IOPS=1425, BW=178MiB/s (187MB/s)(31.3GiB/180001msec) slat (nsec): min=16678, max=69100, avg=31409.40, stdev=4041.71 clat (usec): min=240, max=2605, avg=280.68, stdev=15.29 lat (usec): min=277, max=2636, avg=312.50, stdev=16.10 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[ 277], | 30.00th=[ 277], 40.00th=[ 277], 50.00th=[ 277], 60.00th=[ 281], | 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 289], 95.00th=[ 297], | 99.00th=[ 322], 99.50th=[ 326], 99.90th=[ 347], 99.95th=[ 359], | 99.99th=[ 412] bw ( KiB/s): min=166040, max=202752, per=100.00%, avg=183802.21, stdev=5258.34, samples=360 iops : min= 1297, max= 1584, avg=1435.43, stdev=41.04, samples=360 lat (usec) : 250=0.01%, 500=95.00%, 750=2.82%, 1000=0.87% lat (msec) : 2=1.30%, 4=0.01% cpu : usr=3.32%, sys=11.54%, ctx=513256, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=255775,256663,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 write-1024k-seq: (groupid=12, jobs=1): err= 0: pid=12511: Mon Aug 26 15:45:05 2019 write: IOPS=463, BW=463MiB/s (486MB/s)(81.4GiB/180002msec) slat (usec): min=72, max=182, avg=117.73, stdev=13.96 clat (usec): min=1924, max=13454, avg=2034.72, stdev=361.66 lat (usec): min=2031, max=13537, avg=2153.08, stdev=361.97 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 1975], 10.00th=[ 1991], 20.00th=[ 1991], | 30.00th=[ 2008], 40.00th=[ 2008], 50.00th=[ 2024], 60.00th=[ 2024], | 70.00th=[ 2040], 80.00th=[ 2040], 90.00th=[ 2057], 95.00th=[ 2057], | 99.00th=[ 2089], 99.50th=[ 2114], 99.90th=[ 9503], 99.95th=[ 9634], | 99.99th=[ 9896] bw ( KiB/s): min=458752, max=496484, per=100.00%, avg=477576.15, stdev=7844.90, samples=360 iops : min= 448, max= 484, avg=465.70, stdev= 7.62, samples=360 lat (msec) : 2=30.24%, 4=69.50%, 10=0.25%, 20=0.01% cpu : usr=3.01%, sys=3.96%, ctx=84629, majf=0, minf=6 IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=0,83363,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randwrite-1024k-seq: (groupid=13, jobs=1): err= 0: pid=12533: Mon Aug 26 15:45:05 2019 write: IOPS=464, BW=464MiB/s (487MB/s)(81.6GiB/180001msec) slat (usec): min=69, max=191, avg=116.30, stdev=13.70 clat (usec): min=1925, max=7096, avg=2029.60, stdev=243.97 lat (usec): min=2031, max=7231, avg=2146.51, stdev=244.60 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 1975], 10.00th=[ 1991], 20.00th=[ 1991], | 30.00th=[ 2008], 40.00th=[ 2008], 50.00th=[ 2024], 60.00th=[ 2024], | 70.00th=[ 2040], 80.00th=[ 2040], 90.00th=[ 2057], 95.00th=[ 2057], | 99.00th=[ 2073], 99.50th=[ 2114], 99.90th=[ 6783], 99.95th=[ 6849], | 99.99th=[ 6915] bw ( KiB/s): min=464896, max=493344, per=100.00%, avg=478765.46, stdev=6158.78, samples=360 iops : min= 454, max= 481, avg=466.85, stdev= 6.00, samples=360 lat (msec) : 2=28.52%, 4=71.22%, 10=0.26% cpu : usr=3.06%, sys=3.92%, ctx=84773, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=0,83570,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 read-1024k-seq: (groupid=14, jobs=1): err= 0: pid=12560: Mon Aug 26 15:45:05 2019 read: IOPS=467, BW=468MiB/s (491MB/s)(82.2GiB/180002msec) slat (usec): min=74, max=173, avg=100.38, stdev= 8.93 clat (usec): min=1852, max=2618, avg=2030.43, stdev=30.12 lat (usec): min=1944, max=2720, avg=2131.49, stdev=32.83 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 1975], 10.00th=[ 1991], 20.00th=[ 2008], | 30.00th=[ 2024], 40.00th=[ 2024], 50.00th=[ 2040], 60.00th=[ 2040], | 70.00th=[ 2040], 80.00th=[ 2057], 90.00th=[ 2057], 95.00th=[ 2073], | 99.00th=[ 2073], 99.50th=[ 2089], 99.90th=[ 2147], 99.95th=[ 2180], | 99.99th=[ 2311] bw ( KiB/s): min=474838, max=517192, per=100.00%, avg=482440.02, stdev=6096.02, samples=360 iops : min= 463, max= 505, avg=470.42, stdev= 5.95, samples=360 lat (msec) : 2=14.64%, 4=85.36% cpu : usr=0.57%, sys=5.91%, ctx=84982, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=84223,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randread-1024k-seq: (groupid=15, jobs=1): err= 0: pid=12579: Mon Aug 26 15:45:05 2019 read: IOPS=481, BW=481MiB/s (505MB/s)(84.6GiB/180002msec) slat (usec): min=72, max=156, avg=99.50, stdev= 9.67 clat (usec): min=1852, max=2373, avg=1971.06, stdev=65.28 lat (usec): min=1951, max=2478, avg=2071.25, stdev=66.85 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1893], 10.00th=[ 1909], 20.00th=[ 1926], | 30.00th=[ 1942], 40.00th=[ 1942], 50.00th=[ 1958], 60.00th=[ 1958], | 70.00th=[ 1975], 80.00th=[ 2008], 90.00th=[ 2073], 95.00th=[ 2147], | 99.00th=[ 2147], 99.50th=[ 2147], 99.90th=[ 2212], 99.95th=[ 2245], | 99.99th=[ 2311] bw ( KiB/s): min=485376, max=518218, per=100.00%, avg=496212.19, stdev=5915.15, samples=360 iops : min= 474, max= 506, avg=483.80, stdev= 5.80, samples=360 lat (msec) : 2=79.15%, 4=20.85% cpu : usr=0.55%, sys=5.94%, ctx=87565, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=86622,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rw-1024k-seq: (groupid=16, jobs=1): err= 0: pid=12602: Mon Aug 26 15:45:05 2019 read: IOPS=211, BW=212MiB/s (222MB/s)(37.2GiB/180001msec) slat (usec): min=73, max=176, avg=101.99, stdev= 9.64 clat (usec): min=1931, max=6860, avg=2501.48, stdev=587.24 lat (usec): min=2026, max=6961, avg=2604.16, stdev=587.28 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 1991], 10.00th=[ 2008], 20.00th=[ 2024], | 30.00th=[ 2040], 40.00th=[ 2040], 50.00th=[ 2073], 60.00th=[ 2507], | 70.00th=[ 2802], 80.00th=[ 3228], 90.00th=[ 3490], 95.00th=[ 3556], | 99.00th=[ 3654], 99.50th=[ 3851], 99.90th=[ 5211], 99.95th=[ 5735], | 99.99th=[ 6456] bw ( KiB/s): min=168272, max=271546, per=100.00%, avg=218455.45, stdev=17227.11, samples=360 iops : min= 164, max= 265, avg=212.88, stdev=16.77, samples=360 write: IOPS=206, BW=207MiB/s (217MB/s)(36.4GiB/180001msec) slat (usec): min=75, max=178, avg=118.75, stdev=12.18 clat (usec): min=1935, max=10175, avg=2034.53, stdev=278.16 lat (usec): min=2036, max=10254, avg=2153.95, stdev=278.51 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 1975], 10.00th=[ 1991], 20.00th=[ 1991], | 30.00th=[ 2008], 40.00th=[ 2024], 50.00th=[ 2024], 60.00th=[ 2024], | 70.00th=[ 2040], 80.00th=[ 2040], 90.00th=[ 2057], 95.00th=[ 2057], | 99.00th=[ 2073], 99.50th=[ 2114], 99.90th=[ 7111], 99.95th=[ 9634], | 99.99th=[10028] bw ( KiB/s): min=163096, max=270877, per=100.00%, avg=213343.87, stdev=16525.87, samples=360 iops : min= 159, max= 264, avg=207.88, stdev=16.13, samples=360 lat (msec) : 2=15.78%, 4=83.91%, 10=0.30%, 20=0.01% cpu : usr=1.57%, sys=4.59%, ctx=76136, majf=0, minf=7 IO depths : 1=117.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=38137,37246,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randrw-1024k-seq: (groupid=17, jobs=1): err= 0: pid=12621: Mon Aug 26 15:45:05 2019 read: IOPS=222, BW=222MiB/s (233MB/s)(39.1GiB/180002msec) slat (usec): min=73, max=152, avg=101.01, stdev= 9.85 clat (usec): min=1858, max=5794, avg=2242.66, stdev=409.82 lat (usec): min=1951, max=5894, avg=2344.36, stdev=409.83 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1909], 10.00th=[ 1926], 20.00th=[ 1942], | 30.00th=[ 1958], 40.00th=[ 1975], 50.00th=[ 2040], 60.00th=[ 2147], | 70.00th=[ 2343], 80.00th=[ 2573], 90.00th=[ 2868], 95.00th=[ 3097], | 99.00th=[ 3621], 99.50th=[ 3720], 99.90th=[ 3982], 99.95th=[ 4178], | 99.99th=[ 4817] bw ( KiB/s): min=189935, max=274432, per=100.00%, avg=229282.93, stdev=14988.39, samples=360 iops : min= 185, max= 268, avg=223.41, stdev=14.61, samples=360 write: IOPS=221, BW=222MiB/s (232MB/s)(38.0GiB/180002msec) slat (usec): min=75, max=176, avg=118.54, stdev=12.46 clat (usec): min=1926, max=7003, avg=2025.44, stdev=144.51 lat (usec): min=2032, max=7088, avg=2144.64, stdev=145.19 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 1958], 10.00th=[ 1991], 20.00th=[ 1991], | 30.00th=[ 2008], 40.00th=[ 2024], 50.00th=[ 2024], 60.00th=[ 2024], | 70.00th=[ 2040], 80.00th=[ 2040], 90.00th=[ 2057], 95.00th=[ 2057], | 99.00th=[ 2073], 99.50th=[ 2089], 99.90th=[ 3916], 99.95th=[ 6652], | 99.99th=[ 6915] bw ( KiB/s): min=186368, max=260654, per=100.00%, avg=228583.23, stdev=14200.29, samples=360 iops : min= 182, max= 254, avg=222.73, stdev=13.83, samples=360 lat (msec) : 2=34.68%, 4=65.22%, 10=0.10% cpu : usr=1.70%, sys=4.81%, ctx=80799, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=40027,39906,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 write-4096k-seq: (groupid=18, jobs=1): err= 0: pid=12649: Mon Aug 26 15:45:05 2019 write: IOPS=117, BW=470MiB/s (493MB/s)(82.6GiB/180008msec) slat (usec): min=359, max=868, avg=560.54, stdev=47.08 clat (usec): min=7707, max=19363, avg=7941.94, stdev=743.06 lat (usec): min=8158, max=19836, avg=8503.08, stdev=744.61 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[ 7898], | 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[ 7898], | 99.00th=[14615], 99.50th=[15401], 99.90th=[15533], 99.95th=[15664], | 99.99th=[16057] bw ( KiB/s): min=466944, max=498498, per=100.00%, avg=484736.70, stdev=7540.84, samples=360 iops : min= 114, max= 121, avg=117.88, stdev= 1.82, samples=360 lat (msec) : 10=98.96%, 20=1.04% cpu : usr=1.99%, sys=4.84%, ctx=21926, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=0,21158,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randwrite-4096k-seq: (groupid=19, jobs=1): err= 0: pid=12751: Mon Aug 26 15:45:05 2019 write: IOPS=117, BW=471MiB/s (494MB/s)(82.9GiB/180001msec) slat (usec): min=391, max=859, avg=561.31, stdev=46.23 clat (usec): min=7709, max=16870, avg=7919.13, stdev=536.92 lat (usec): min=8137, max=17416, avg=8481.01, stdev=538.50 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[ 7898], | 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[ 7898], | 99.00th=[12256], 99.50th=[12911], 99.90th=[14877], 99.95th=[15533], | 99.99th=[16450] bw ( KiB/s): min=467879, max=498498, per=100.00%, avg=485950.74, stdev=6245.26, samples=360 iops : min= 114, max= 121, avg=118.17, stdev= 1.49, samples=360 lat (msec) : 10=98.96%, 20=1.04% cpu : usr=2.05%, sys=4.80%, ctx=21985, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=0,21210,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 read-4096k-seq: (groupid=20, jobs=1): err= 0: pid=12865: Mon Aug 26 15:45:05 2019 read: IOPS=120, BW=482MiB/s (505MB/s)(84.7GiB/180005msec) slat (usec): min=375, max=585, avg=462.83, stdev=22.62 clat (usec): min=7372, max=8219, avg=7837.59, stdev=72.82 lat (usec): min=7833, max=8683, avg=8301.03, stdev=74.07 clat percentiles (usec): | 1.00th=[ 7504], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[ 7767], | 30.00th=[ 7832], 40.00th=[ 7832], 50.00th=[ 7832], 60.00th=[ 7832], | 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[ 7898], | 99.00th=[ 7963], 99.50th=[ 7963], 99.90th=[ 8094], 99.95th=[ 8160], | 99.99th=[ 8160] bw ( KiB/s): min=484296, max=523423, per=100.00%, avg=496606.33, stdev=5140.76, samples=360 iops : min= 118, max= 127, avg=120.75, stdev= 1.21, samples=360 lat (msec) : 10=100.00% cpu : usr=0.10%, sys=5.70%, ctx=22379, majf=0, minf=6 IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=21673,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randread-4096k-seq: (groupid=21, jobs=1): err= 0: pid=12886: Mon Aug 26 15:45:05 2019 read: IOPS=124, BW=497MiB/s (521MB/s)(87.3GiB/180003msec) slat (usec): min=358, max=616, avg=467.77, stdev=24.22 clat (usec): min=7345, max=8130, avg=7580.08, stdev=143.91 lat (usec): min=7782, max=8578, avg=8048.48, stdev=144.73 clat percentiles (usec): | 1.00th=[ 7373], 5.00th=[ 7439], 10.00th=[ 7439], 20.00th=[ 7504], | 30.00th=[ 7504], 40.00th=[ 7504], 50.00th=[ 7570], 60.00th=[ 7570], | 70.00th=[ 7570], 80.00th=[ 7635], 90.00th=[ 7832], 95.00th=[ 7963], | 99.00th=[ 7963], 99.50th=[ 8029], 99.90th=[ 8029], 99.95th=[ 8029], | 99.99th=[ 8094] bw ( KiB/s): min=499712, max=523423, per=100.00%, avg=512080.41, stdev=4112.59, samples=360 iops : min= 122, max= 127, avg=124.64, stdev= 0.98, samples=360 lat (msec) : 10=100.00% cpu : usr=0.11%, sys=5.97%, ctx=22989, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=22349,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rw-4096k-seq: (groupid=22, jobs=1): err= 0: pid=12909: Mon Aug 26 15:45:05 2019 read: IOPS=58, BW=234MiB/s (245MB/s)(41.1GiB/180006msec) slat (usec): min=346, max=562, avg=452.82, stdev=26.35 clat (usec): min=7728, max=16713, avg=8327.62, stdev=806.21 lat (usec): min=8137, max=17156, avg=8781.03, stdev=807.21 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7832], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[ 8291], | 70.00th=[ 8356], 80.00th=[ 8979], 90.00th=[ 9241], 95.00th=[ 9372], | 99.00th=[10159], 99.50th=[14877], 99.90th=[16450], 99.95th=[16581], | 99.99th=[16712] bw ( KiB/s): min=164168, max=339264, per=100.00%, avg=241040.85, stdev=33169.14, samples=360 iops : min= 40, max= 82, avg=58.44, stdev= 8.03, samples=360 write: IOPS=57, BW=229MiB/s (240MB/s)(40.2GiB/180006msec) slat (usec): min=389, max=875, avg=574.61, stdev=47.69 clat (usec): min=7718, max=18868, avg=7927.09, stdev=664.09 lat (usec): min=8142, max=19410, avg=8502.30, stdev=666.26 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[ 7898], | 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[ 7898], | 99.00th=[10814], 99.50th=[14746], 99.90th=[15008], 99.95th=[15270], | 99.99th=[15533] bw ( KiB/s): min=148945, max=320128, per=100.00%, avg=235860.97, stdev=31961.69, samples=360 iops : min= 36, max= 78, avg=57.19, stdev= 7.77, samples=360 lat (msec) : 10=98.93%, 20=1.07% cpu : usr=1.06%, sys=5.11%, ctx=21483, majf=0, minf=7 IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=10520,10295,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randrw-4096k-seq: (groupid=23, jobs=1): err= 0: pid=12929: Mon Aug 26 15:45:05 2019 read: IOPS=59, BW=236MiB/s (248MB/s)(41.5GiB/180008msec) slat (usec): min=346, max=567, avg=463.00, stdev=26.69 clat (usec): min=7373, max=10851, avg=7817.33, stdev=441.45 lat (usec): min=7760, max=11352, avg=8280.93, stdev=442.88 clat percentiles (usec): | 1.00th=[ 7373], 5.00th=[ 7439], 10.00th=[ 7439], 20.00th=[ 7504], | 30.00th=[ 7504], 40.00th=[ 7570], 50.00th=[ 7635], 60.00th=[ 7701], | 70.00th=[ 7898], 80.00th=[ 8094], 90.00th=[ 8455], 95.00th=[ 8717], | 99.00th=[ 9372], 99.50th=[ 9503], 99.90th=[ 9765], 99.95th=[10290], | 99.99th=[10814] bw ( KiB/s): min=132129, max=332332, per=100.00%, avg=243420.59, stdev=32701.57, samples=360 iops : min= 32, max= 81, avg=59.02, stdev= 7.95, samples=360 write: IOPS=60, BW=241MiB/s (252MB/s)(42.3GiB/180008msec) slat (usec): min=402, max=885, avg=588.78, stdev=48.47 clat (usec): min=7696, max=16431, avg=7891.27, stdev=398.70 lat (usec): min=8171, max=16917, avg=8480.66, stdev=400.14 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7832], 50.00th=[ 7898], 60.00th=[ 7898], | 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[ 7898], | 99.00th=[ 8029], 99.50th=[12256], 99.90th=[12387], 99.95th=[12387], | 99.99th=[14877] bw ( KiB/s): min=165829, max=355096, per=100.00%, avg=248270.02, stdev=31688.91, samples=360 iops : min= 40, max= 86, avg=60.21, stdev= 7.69, samples=360 lat (msec) : 10=99.59%, 20=0.41% cpu : usr=1.15%, sys=5.40%, ctx=22140, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwt: total=10625,10836,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=21.1GiB (22.7GB), run=180001-180001msec Run status group 1 (all jobs): WRITE: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s), io=20.2GiB (21.7GB), run=180001-180001msec Run status group 2 (all jobs): READ: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s), io=20.2GiB (21.7GB), run=180001-180001msec Run status group 3 (all jobs): READ: bw=95.9MiB/s (101MB/s), 95.9MiB/s-95.9MiB/s (101MB/s-101MB/s), io=16.9GiB (18.1GB), run=180001-180001msec Run status group 4 (all jobs): READ: bw=62.4MiB/s (65.4MB/s), 62.4MiB/s-62.4MiB/s (65.4MB/s-65.4MB/s), io=10.0GiB (11.8GB), run=180001-180001msec WRITE: bw=62.5MiB/s (65.5MB/s), 62.5MiB/s-62.5MiB/s (65.5MB/s-65.5MB/s), io=10.0GiB (11.8GB), run=180001-180001msec Run status group 5 (all jobs): READ: bw=53.0MiB/s (56.6MB/s), 53.0MiB/s-53.0MiB/s (56.6MB/s-56.6MB/s), io=9719MiB (10.2GB), run=180001-180001msec WRITE: bw=54.0MiB/s (56.6MB/s), 54.0MiB/s-54.0MiB/s (56.6MB/s-56.6MB/s), io=9720MiB (10.2GB), run=180001-180001msec Run status group 6 (all jobs): WRITE: bw=395MiB/s (414MB/s), 395MiB/s-395MiB/s (414MB/s-414MB/s), io=69.5GiB (74.6GB), run=180001-180001msec Run status group 7 (all jobs): WRITE: bw=394MiB/s (413MB/s), 394MiB/s-394MiB/s (413MB/s-413MB/s), io=69.2GiB (74.3GB), run=180001-180001msec Run status group 8 (all jobs): READ: bw=407MiB/s (427MB/s), 407MiB/s-407MiB/s (427MB/s-427MB/s), io=71.6GiB (76.9GB), run=180001-180001msec Run status group 9 (all jobs): READ: bw=379MiB/s (397MB/s), 379MiB/s-379MiB/s (397MB/s-397MB/s), io=66.6GiB (71.5GB), run=180001-180001msec Run status group 10 (all jobs): READ: bw=176MiB/s (184MB/s), 176MiB/s-176MiB/s (184MB/s-184MB/s), io=30.9GiB (33.2GB), run=180001-180001msec WRITE: bw=176MiB/s (185MB/s), 176MiB/s-176MiB/s (185MB/s-185MB/s), io=30.9GiB (33.2GB), run=180001-180001msec Run status group 11 (all jobs): READ: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s), io=31.2GiB (33.5GB), run=180001-180001msec WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s), io=31.3GiB (33.6GB), run=180001-180001msec Run status group 12 (all jobs): WRITE: bw=463MiB/s (486MB/s), 463MiB/s-463MiB/s (486MB/s-486MB/s), io=81.4GiB (87.4GB), run=180002-180002msec Run status group 13 (all jobs): WRITE: bw=464MiB/s (487MB/s), 464MiB/s-464MiB/s (487MB/s-487MB/s), io=81.6GiB (87.6GB), run=180001-180001msec Run status group 14 (all jobs): READ: bw=468MiB/s (491MB/s), 468MiB/s-468MiB/s (491MB/s-491MB/s), io=82.2GiB (88.3GB), run=180002-180002msec Run status group 15 (all jobs): READ: bw=481MiB/s (505MB/s), 481MiB/s-481MiB/s (505MB/s-505MB/s), io=84.6GiB (90.8GB), run=180002-180002msec Run status group 16 (all jobs): READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=37.2GiB (39.0GB), run=180001-180001msec WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=36.4GiB (39.1GB), run=180001-180001msec Run status group 17 (all jobs): READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=39.1GiB (41.0GB), run=180002-180002msec WRITE: bw=222MiB/s (232MB/s), 222MiB/s-222MiB/s (232MB/s-232MB/s), io=38.0GiB (41.8GB), run=180002-180002msec Run status group 18 (all jobs): WRITE: bw=470MiB/s (493MB/s), 470MiB/s-470MiB/s (493MB/s-493MB/s), io=82.6GiB (88.7GB), run=180008-180008msec Run status group 19 (all jobs): WRITE: bw=471MiB/s (494MB/s), 471MiB/s-471MiB/s (494MB/s-494MB/s), io=82.9GiB (88.0GB), run=180001-180001msec Run status group 20 (all jobs): READ: bw=482MiB/s (505MB/s), 482MiB/s-482MiB/s (505MB/s-505MB/s), io=84.7GiB (90.9GB), run=180005-180005msec Run status group 21 (all jobs): READ: bw=497MiB/s (521MB/s), 497MiB/s-497MiB/s (521MB/s-521MB/s), io=87.3GiB (93.7GB), run=180003-180003msec Run status group 22 (all jobs): READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=41.1GiB (44.1GB), run=180006-180006msec WRITE: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=40.2GiB (43.2GB), run=180006-180006msec Run status group 23 (all jobs): READ: bw=236MiB/s (248MB/s), 236MiB/s-236MiB/s (248MB/s-248MB/s), io=41.5GiB (44.6GB), run=180008-180008msec WRITE: bw=241MiB/s (252MB/s), 241MiB/s-241MiB/s (252MB/s-252MB/s), io=42.3GiB (45.4GB), run=180008-180008msec Disk stats (read/write): sdb: ios=20762156/22203113, merge=0/0, ticks=4810773/4551307, in_queue=9339055, util=85.66%
Hi, Can you test it slightly differently (and simpler)? Like in this googledoc: https://docs.google.com/spreadsheets/d/1E9-eXjzsKboiCCX-0u0r5fAjjufLKayaut_F... As we know that it's a QLC drive, first let it fill the SLC cache: fio -ioengine=libaio -direct=1 -name=test -bs=4M -iodepth=32 -rw=write -size=100G -filename=/dev/sdXX Then, immediately: 1) Parallel random write fio -ioengine=libaio -direct=1 -name=test -bs=4k -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdXX 2) Transactional random write (journaling) fio -ioengine=libaio -fsync=1 -direct=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -filename=/dev/sdXX 3) Linear write fio -ioengine=libaio -direct=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/sdXX 4) Parallel random read: fio -ioengine=libaio -direct=1 -name=test -bs=4k -iodepth=32 -rw=randread -runtime=60 -filename=/dev/sdXX 5) Synchronous random read: fio -ioengine=libaio -sync=1 -direct=1 -name=test -bs=4k -iodepth=1 -rw=randread -runtime=60 -filename=/dev/sdXX 6) Linear read: fio -ioengine=libaio -direct=1 -name=test -bs=4M -iodepth=32 -rw=read -runtime=60 -filename=/dev/sdXX
Hi,
So, ran the fio commands, and pasted output (as it's quite a lot) here:
I hope someone here can draw some conclusions from this output...
participants (4)
-
Anthony D'Atri
-
Marc Roos
-
mj
-
vitalif＠yourcmc.ru