I was talking about on-disk cache, but, yes, the controller cache needs to be disabled too. The first can be done with smartctl or hdparm. Check cache status with something like 'smartctl -g wcache /dev/sda' and disable with something like 'smartctl -s wcache=off /dev/sda'. Controller cache needs to be disabled in the BIOS. By the way, if you can't use pass-through, you should disable controller cache for every disk, including the HDDs. There are cases in the list demonstrating that controller cache enabled can lead to data loss on power outage. As I recommended before, please search the ceph-user list, you will find detailed instructions and also links to explanations and typical benchmarks. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Sent: 31 August 2020 14:44:07 To: Frank Schilder; 'ceph-users@ceph.io' Subject: AW: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) We have older LSi Raid controller with no HBA/JBOD option. So we expose the single disks as raid0 devices. Ceph should not be aware of cache status? But digging deeper in to it it seems that 1 out of 4 serves is performing a lot better and has super low commit/applay rates while the other have a lot mor (20+) on heavy writes. This just applys fore the ssd. For the hdds I cant see a difference... -----Ursprüngliche Nachricht----- Von: Frank Schilder <frans@dtu.dk> Gesendet: Montag, 31. August 2020 13:19 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at>; 'ceph-users@ceph.io' <ceph-users@ceph.io> Betreff: Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) Yes, they can - if volatile write cache is not disabled. There are many threads on this, also recent. Search for "disable write cache" and/or "disable volatile write cache". You will also find different methods of doing this automatically. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Sent: 31 August 2020 13:02:45 To: 'ceph-users@ceph.io' Subject: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) I have a productive 60 osd's cluster. No extra Journals. Its performing okay. Now I added an extra ssd Pool with 16 Micron 5100 MAX. And the performance is little slower or equal to the 60 hdd pool. 4K random as also sequential reads. All on dedicated 2 times 10G Network. HDDS are still on filestore. SSD on bluestore. Ceph Luminous. What should be possible 16 ssd's vs. 60 hhd's no extra journals? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Thank you. I was working in this direction. The situation is a lot better. But I think I can get still far better. I could set the controller to writethrough, direct and no read ahead for the ssds. But I cannot disable the pdcache ☹ there is an option set in the controller "Block SSD Write Disk Cache Change = Yes" which does not permit to deactivate the ssd cache. I could not find any solution in google for this controller (LSI MegaRAID SAS 9271-8i) to change this setting. I don’t know how much performance gain it will be to deactivate the ssd cache. At least the micron 5200max has capacitor so I hope it is safe for data loss in case if power failure. I wrote a request to lsi / Broadcom if they know how I can change this setting. This is really annyoing. I will check the cpu power settings. I rode also somewhere it can improve iops a lot. (if its bad set) At the moment I get 600iops 4k random write 1 thread and 1 iodepth. I get 40K - 4k random iops for some instances with 32iodepth. Its not the world but a lot better then before. Read around 100k iops. For 16 ssd's and 2 x dual 10G nic. I was reading that good tunings and hardware config can get more then 2000 iops on single thread out of the ssds. I know thet ceph does not shine with single thread. But 600 iops is not very much... philipp -----Ursprüngliche Nachricht----- Von: Reed Dier <reed.dier@focusvq.com> Gesendet: Dienstag, 01. September 2020 22:37 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) If using storcli/perccli for manipulating the LSI controller, you can disable the on-disk write cache with: storcli /cx/vx set pdcache=off You can also ensure that you turn off write caching at the controller level with storcli /cx/vx set iopolicy=direct storcli /cx/vx set wrcache=wt You can also tweak the readahead value for the vd if you want, though with an ssd, I don't think it will be much of an issue. storcli /cx/vx set rdcache=nora I'm sure the megacli alternatives are available with some quick searches. May also want to check your c-states and p-states to make sure there isn't any aggressive power saving features getting in the way. Reed

On Aug 31, 2020, at 7:44 AM, VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> wrote:

We have older LSi Raid controller with no HBA/JBOD option. So we expose the single disks as raid0 devices. Ceph should not be aware of cache status? But digging deeper in to it it seems that 1 out of 4 serves is performing a lot better and has super low commit/applay rates while the other have a lot mor (20+) on heavy writes. This just applys fore the ssd. For the hdds I cant see a difference...

-----Ursprüngliche Nachricht----- Von: Frank Schilder <frans@dtu.dk> Gesendet: Montag, 31. August 2020 13:19 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at>; 'ceph-users@ceph.io' <ceph-users@ceph.io> Betreff: Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

Yes, they can - if volatile write cache is not disabled. There are many threads on this, also recent. Search for "disable write cache" and/or "disable volatile write cache".

You will also find different methods of doing this automatically.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Sent: 31 August 2020 13:02:45 To: 'ceph-users@ceph.io' Subject: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

I have a productive 60 osd's cluster. No extra Journals. Its performing okay. Now I added an extra ssd Pool with 16 Micron 5100 MAX. And the performance is little slower or equal to the 60 hdd pool. 4K random as also sequential reads. All on dedicated 2 times 10G Network. HDDS are still on filestore. SSD on bluestore. Ceph Luminous. What should be possible 16 ssd's vs. 60 hhd's no extra journals?

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

write-4k-seq: (groupid=0, jobs=1): err= 0: pid=11017: Tue Sep 1 20:58:43 2020 write: IOPS=34.4k, BW=134MiB/s (141MB/s)(23.6GiB/180001msec) slat (nsec): min=3964, max=124499, avg=4432.71, stdev=911.13 clat (nsec): min=470, max=435529, avg=23528.70, stdev=2553.67 lat (usec): min=24, max=445, avg=28.08, stdev= 3.04 clat percentiles (nsec): | 1.00th=[22144], 5.00th=[22400], 10.00th=[22400], 20.00th=[22656], | 30.00th=[22656], 40.00th=[22656], 50.00th=[22912], 60.00th=[22912], | 70.00th=[22912], 80.00th=[23680], 90.00th=[25984], 95.00th=[27008], | 99.00th=[32384], 99.50th=[35072], 99.90th=[40192], 99.95th=[41728], | 99.99th=[54016] bw ( KiB/s): min=51928, max=140088, per=70.28%, avg=96694.64, stdev=15493.31, samples=359 iops : min=12982, max=35022, avg=24173.37, stdev=3873.39, samples=359 lat (nsec) : 500=0.01%, 750=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=99.98%, 100=0.01% lat (usec) : 250=0.01%, 500=0.01% cpu : usr=5.61%, sys=18.02%, ctx=6191054, majf=0, minf=7 IO depths : 1=116.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,

=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,6190964,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randwrite-4k-seq: (groupid=1, jobs=1): err= 0: pid=11050: Tue Sep 1 20:58:43 2020 write: IOPS=34.6k, BW=135MiB/s (142MB/s)(23.7GiB/180001msec) slat (nsec): min=3956, max=140599, avg=4368.89, stdev=537.59 clat (nsec): min=492, max=488951, avg=23289.09, stdev=2404.05 lat (usec): min=24, max=494, avg=27.77, stdev= 2.59 clat percentiles (nsec): | 1.00th=[21888], 5.00th=[22144], 10.00th=[22144], 20.00th=[22400], | 30.00th=[22400], 40.00th=[22400], 50.00th=[22656], 60.00th=[22656], | 70.00th=[22656], 80.00th=[23424], 90.00th=[25728], 95.00th=[26752], | 99.00th=[30336], 99.50th=[32128], 99.90th=[37120], 99.95th=[39168], | 99.99th=[46848] bw ( KiB/s): min=48176, max=139232, per=70.44%, avg=97396.63, stdev=14897.43, samples=359 iops : min=12044, max=34808, avg=24348.81, stdev=3724.44, samples=359 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 20=0.01%, 50=99.98%, 100=0.01%, 250=0.01% lat (usec) : 500=0.01% cpu : usr=6.06%, sys=17.91%, ctx=6222106, majf=0, minf=5 IO depths : 1=116.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,6222038,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 read-4k-seq: (groupid=2, jobs=1): err= 0: pid=11068: Tue Sep 1 20:58:43 2020 read: IOPS=22.7k, BW=88.6MiB/s (92.9MB/s)(15.6GiB/180001msec) slat (usec): min=5, max=603, avg= 6.77, stdev= 1.64 clat (nsec): min=694, max=426738, avg=35641.98, stdev=12418.44 lat (usec): min=31, max=636, avg=42.59, stdev=12.68 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 33], | 70.00th=[ 34], 80.00th=[ 36], 90.00th=[ 40], 95.00th=[ 44], | 99.00th=[ 104], 99.50th=[ 105], 99.90th=[ 109], 99.95th=[ 110], | 99.99th=[ 112] bw ( KiB/s): min=45664, max=92376, per=70.46%, avg=63903.11, stdev=10098.57, samples=359 iops : min=11416, max=23094, avg=15975.47, stdev=2524.71, samples=359 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 20=0.01%, 50=96.83%, 100=0.47%, 250=2.69% lat (usec) : 500=0.01% cpu : usr=5.61%, sys=18.15%, ctx=4081143, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=4081084,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randread-4k-seq: (groupid=3, jobs=1): err= 0: pid=11094: Tue Sep 1 20:58:43 2020 read: IOPS=17.5k, BW=68.4MiB/s (71.7MB/s)(12.0GiB/180001msec) slat (usec): min=6, max=111, avg=12.99, stdev= 4.03 clat (nsec): min=894, max=690146, avg=41739.21, stdev=16973.22 lat (usec): min=36, max=698, avg=54.95, stdev=17.78 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 38], | 30.00th=[ 38], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 40], 80.00th=[ 41], 90.00th=[ 42], 95.00th=[ 46], | 99.00th=[ 143], 99.50th=[ 161], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 180] bw ( KiB/s): min=34504, max=72251, per=70.62%, avg=49458.74, stdev=7758.59, samples=359 iops : min= 8626, max=18062, avg=12364.34, stdev=1939.72, samples=359 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=96.85%, 100=0.15% lat (usec) : 250=3.00%, 500=0.01%, 750=0.01% cpu : usr=7.15%, sys=24.25%, ctx=3151855, majf=0, minf=5 IO depths : 1=116.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=3151781,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rw-4k-seq: (groupid=4, jobs=1): err= 0: pid=11111: Tue Sep 1 20:58:43 2020 read: IOPS=5044, BW=19.7MiB/s (20.7MB/s)(3547MiB/180001msec) slat (usec): min=6, max=931, avg=11.28, stdev= 4.13 clat (nsec): min=956, max=5424.8k, avg=134143.31, stdev=127539.59 lat (usec): min=37, max=5433, avg=145.65, stdev=127.54 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 69], 10.00th=[ 70], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 103], 50.00th=[ 126], 60.00th=[ 135], | 70.00th=[ 153], 80.00th=[ 165], 90.00th=[ 180], 95.00th=[ 219], | 99.00th=[ 553], 99.50th=[ 693], 99.90th=[ 1450], 99.95th=[ 2573], | 99.99th=[ 3949] bw ( KiB/s): min= 9008, max=21530, per=70.55%, avg=14233.71, stdev=2522.39, samples=359 iops : min= 2252, max= 5382, avg=3558.09, stdev=630.67, samples=359 write: IOPS=5057, BW=19.8MiB/s (20.7MB/s)(3556MiB/180001msec) slat (nsec): min=7231, max=92378, avg=11629.01, stdev=4076.38 clat (nsec): min=958, max=252074, avg=36289.30, stdev=8070.28 lat (usec): min=30, max=260, avg=48.15, stdev=10.24 clat percentiles (nsec): | 1.00th=[27008], 5.00th=[27264], 10.00th=[27520], 20.00th=[28032], | 30.00th=[28544], 40.00th=[29824], 50.00th=[37120], 60.00th=[38656], | 70.00th=[45312], 80.00th=[45824], 90.00th=[46336], 95.00th=[46336], | 99.00th=[47872], 99.50th=[50944], 99.90th=[68096], 99.95th=[71168], | 99.99th=[85504] bw ( KiB/s): min= 8815, max=22272, per=70.53%, avg=14267.78, stdev=2542.60, samples=359 iops : min= 2203, max= 5568, avg=3566.60, stdev=635.74, samples=359 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 20=0.01%, 50=50.77%, 100=19.16%, 250=27.91% lat (usec) : 500=1.52%, 750=0.42%, 1000=0.05% lat (msec) : 2=0.14%, 4=0.03%, 10=0.01% cpu : usr=3.84%, sys=13.55%, ctx=1818301, majf=0, minf=8 IO depths : 1=116.2%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=907960,910297,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randrw-4k-seq: (groupid=5, jobs=1): err= 0: pid=11135: Tue Sep 1 20:58:43 2020 read: IOPS=11.7k, BW=45.5MiB/s (47.7MB/s)(8196MiB/180001msec) slat (nsec): min=5290, max=79471, avg=6627.78, stdev=1194.13 clat (nsec): min=762, max=4600.3k, avg=40464.37, stdev=52795.57 lat (usec): min=33, max=4606, avg=47.27, stdev=52.83 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 33], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 37], | 70.00th=[ 37], 80.00th=[ 38], 90.00th=[ 42], 95.00th=[ 45], | 99.00th=[ 157], 99.50th=[ 174], 99.90th=[ 231], 99.95th=[ 469], | 99.99th=[ 3097] bw ( KiB/s): min=23224, max=47936, per=70.40%, avg=32826.65, stdev=5019.16, samples=359 iops : min= 5806, max=11984, avg=8206.34, stdev=1254.87, samples=359 write: IOPS=11.7k, BW=45.5MiB/s (47.7MB/s)(8196MiB/180001msec) slat (nsec): min=5513, max=99434, avg=6863.99, stdev=1232.36 clat (nsec): min=746, max=159154, avg=27763.68, stdev=4111.84 lat (usec): min=28, max=167, avg=34.81, stdev= 4.78 clat percentiles (nsec): | 1.00th=[24960], 5.00th=[25472], 10.00th=[25728], 20.00th=[25728], | 30.00th=[25984], 40.00th=[26240], 50.00th=[26496], 60.00th=[26752], | 70.00th=[27008], 80.00th=[27264], 90.00th=[30848], 95.00th=[38656], | 99.00th=[43264], 99.50th=[47360], 99.90th=[49408], 99.95th=[49920], | 99.99th=[56576] bw ( KiB/s): min=22752, max=48280, per=70.42%, avg=32834.75, stdev=5123.30, samples=359 iops : min= 5688, max=12070, avg=8208.33, stdev=1280.92, samples=359 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 20=0.01%, 50=98.36%, 100=0.11%, 250=1.48% lat (usec) : 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=6.50%, sys=18.64%, ctx=4196525, majf=0, minf=7 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=2098219,2098247,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 write-128k-seq: (groupid=6, jobs=1): err= 0: pid=11154: Tue Sep 1 20:58:43 2020 write: IOPS=3080, BW=385MiB/s (404MB/s)(67.7GiB/180001msec) slat (nsec): min=15619, max=90115, avg=23848.24, stdev=2409.33 clat (usec): min=242, max=2337, avg=296.95, stdev=24.36 lat (usec): min=276, max=2360, avg=321.21, stdev=24.66 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 269], 10.00th=[ 273], 20.00th=[ 297], | 30.00th=[ 297], 40.00th=[ 297], 50.00th=[ 297], 60.00th=[ 297], | 70.00th=[ 297], 80.00th=[ 302], 90.00th=[ 302], 95.00th=[ 306], | 99.00th=[ 334], 99.50th=[ 412], 99.90th=[ 594], 99.95th=[ 734], | 99.99th=[ 947] bw ( KiB/s): min=195840, max=397568, per=70.63%, avg=278519.83, stdev=43877.21, samples=359 iops : min= 1530, max= 3106, avg=2175.51, stdev=342.91, samples=359 lat (usec) : 250=0.01%, 500=99.78%, 750=0.18%, 1000=0.04% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=4.84%, sys=6.98%, ctx=554562, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,554527,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randwrite-128k-seq: (groupid=7, jobs=1): err= 0: pid=11178: Tue Sep 1 20:58:43 2020 write: IOPS=3065, BW=383MiB/s (402MB/s)(67.4GiB/180001msec) slat (usec): min=15, max=204, avg=24.16, stdev= 2.34 clat (usec): min=257, max=2289, avg=297.39, stdev=24.32 lat (usec): min=276, max=2315, avg=321.97, stdev=24.65 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 269], 10.00th=[ 273], 20.00th=[ 297], | 30.00th=[ 297], 40.00th=[ 297], 50.00th=[ 297], 60.00th=[ 297], | 70.00th=[ 302], 80.00th=[ 302], 90.00th=[ 306], 95.00th=[ 306], | 99.00th=[ 330], 99.50th=[ 412], 99.90th=[ 594], 99.95th=[ 750], | 99.99th=[ 938] bw ( KiB/s): min=194816, max=395776, per=70.63%, avg=277186.48, stdev=43560.34, samples=359 iops : min= 1522, max= 3092, avg=2165.11, stdev=340.42, samples=359 lat (usec) : 500=99.77%, 750=0.18%, 1000=0.04% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=5.07%, sys=6.99%, ctx=551916, majf=0, minf=4 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,551875,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 read-128k-seq: (groupid=8, jobs=1): err= 0: pid=11200: Tue Sep 1 20:58:43 2020 read: IOPS=3097, BW=387MiB/s (406MB/s)(68.1GiB/180001msec) slat (nsec): min=14919, max=95610, avg=21631.52, stdev=2127.99 clat (usec): min=257, max=649, avg=297.53, stdev=11.95 lat (usec): min=279, max=671, avg=319.56, stdev=12.35 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 273], 10.00th=[ 273], 20.00th=[ 297], | 30.00th=[ 297], 40.00th=[ 302], 50.00th=[ 302], 60.00th=[ 302], | 70.00th=[ 302], 80.00th=[ 302], 90.00th=[ 306], 95.00th=[ 306], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 359], 99.95th=[ 396], | 99.99th=[ 490] bw ( KiB/s): min=198144, max=397824, per=70.58%, avg=279820.57, stdev=43306.98, samples=359 iops : min= 1548, max= 3108, avg=2185.65, stdev=338.46, samples=359 lat (usec) : 500=99.99%, 750=0.01% cpu : usr=2.73%, sys=8.43%, ctx=557545, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=557511,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randread-128k-seq: (groupid=9, jobs=1): err= 0: pid=11224: Tue Sep 1 20:58:43 2020 read: IOPS=2421, BW=303MiB/s (317MB/s)(53.2GiB/180001msec) slat (usec): min=15, max=111, avg=22.55, stdev= 3.40 clat (usec): min=244, max=1212, avg=385.92, stdev=113.52 lat (usec): min=319, max=1234, avg=408.90, stdev=113.66 clat percentiles (usec): | 1.00th=[ 306], 5.00th=[ 310], 10.00th=[ 330], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 343], | 70.00th=[ 367], 80.00th=[ 416], 90.00th=[ 482], 95.00th=[ 676], | 99.00th=[ 873], 99.50th=[ 889], 99.90th=[ 914], 99.95th=[ 922], | 99.99th=[ 930] bw ( KiB/s): min=152576, max=314368, per=70.53%, avg=218614.19, stdev=33921.86, samples=359 iops : min= 1192, max= 2456, avg=1707.48, stdev=265.13, samples=359 lat (usec) : 250=0.01%, 500=90.98%, 750=5.60%, 1000=3.41% lat (msec) : 2=0.01% cpu : usr=2.07%, sys=6.81%, ctx=435923, majf=0, minf=4 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=435883,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rw-128k-seq: (groupid=10, jobs=1): err= 0: pid=11240: Tue Sep 1 20:58:43 2020 read: IOPS=920, BW=115MiB/s (121MB/s)(20.2GiB/180001msec) slat (usec): min=16, max=117, avg=25.21, stdev= 5.36 clat (usec): min=257, max=6673, avg=722.60, stdev=470.02 lat (usec): min=282, max=6699, avg=748.28, stdev=470.83 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 474], 10.00th=[ 502], 20.00th=[ 545], | 30.00th=[ 586], 40.00th=[ 619], 50.00th=[ 644], 60.00th=[ 676], | 70.00th=[ 734], 80.00th=[ 783], 90.00th=[ 832], 95.00th=[ 914], | 99.00th=[ 3916], 99.50th=[ 4178], 99.90th=[ 4424], 99.95th=[ 4555], | 99.99th=[ 5276] bw ( KiB/s): min=53248, max=142848, per=70.79%, avg=83410.22, stdev=14505.45, samples=359 iops : min= 416, max= 1116, avg=651.18, stdev=113.44, samples=359 write: IOPS=922, BW=115MiB/s (121MB/s)(20.3GiB/180001msec) slat (nsec): min=16349, max=97430, avg=27250.23, stdev=5389.95 clat (usec): min=253, max=506, avg=301.64, stdev=17.58 lat (usec): min=278, max=532, avg=329.36, stdev=20.56 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 269], 10.00th=[ 273], 20.00th=[ 289], | 30.00th=[ 297], 40.00th=[ 297], 50.00th=[ 297], 60.00th=[ 302], | 70.00th=[ 306], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 330], 99.50th=[ 334], 99.90th=[ 338], 99.95th=[ 347], | 99.99th=[ 396] bw ( KiB/s): min=54784, max=149504, per=70.81%, avg=83573.94, stdev=14782.76, samples=359 iops : min= 428, max= 1168, avg=652.50, stdev=115.59, samples=359 lat (usec) : 500=54.69%, 750=31.93%, 1000=11.45% lat (msec) : 2=0.82%, 4=0.68%, 10=0.43% cpu : usr=2.17%, sys=5.08%, ctx=331694, majf=0, minf=7 IO depths : 1=116.4%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=165686,165974,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randrw-128k-seq: (groupid=11, jobs=1): err= 0: pid=11264: Tue Sep 1 20:58:43 2020 read: IOPS=1226, BW=153MiB/s (161MB/s)(26.0GiB/180001msec) slat (usec): min=16, max=102, avg=22.83, stdev= 2.76 clat (usec): min=293, max=5938, avg=462.78, stdev=435.24 lat (usec): min=320, max=5964, avg=486.02, stdev=435.25 clat percentiles (usec): | 1.00th=[ 306], 5.00th=[ 310], 10.00th=[ 330], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 343], 60.00th=[ 355], | 70.00th=[ 383], 80.00th=[ 445], 90.00th=[ 611], 95.00th=[ 857], | 99.00th=[ 3195], 99.50th=[ 3720], 99.90th=[ 4146], 99.95th=[ 4293], | 99.99th=[ 4621] bw ( KiB/s): min=73472, max=165632, per=70.35%, avg=110494.22, stdev=16988.70, samples=359 iops : min= 574, max= 1294, avg=862.82, stdev=132.82, samples=359 write: IOPS=1230, BW=154MiB/s (161MB/s)(27.0GiB/180001msec) slat (nsec): min=16349, max=96575, avg=24955.25, stdev=2925.99 clat (usec): min=253, max=617, avg=294.45, stdev=12.51 lat (usec): min=278, max=652, avg=319.82, stdev=13.03 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 269], 10.00th=[ 273], 20.00th=[ 289], | 30.00th=[ 297], 40.00th=[ 297], 50.00th=[ 297], 60.00th=[ 297], | 70.00th=[ 297], 80.00th=[ 302], 90.00th=[ 302], 95.00th=[ 306], | 99.00th=[ 326], 99.50th=[ 330], 99.90th=[ 338], 99.95th=[ 347], | 99.99th=[ 416] bw ( KiB/s): min=76800, max=173056, per=70.40%, avg=110875.73, stdev=17327.47, samples=359 iops : min= 600, max= 1352, avg=865.78, stdev=135.49, samples=359 lat (usec) : 500=93.22%, 750=3.15%, 1000=2.07% lat (msec) : 2=0.52%, 4=0.95%, 10=0.10% cpu : usr=2.98%, sys=6.41%, ctx=442361, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=220859,221475,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 write-1024k-seq: (groupid=12, jobs=1): err= 0: pid=11282: Tue Sep 1 20:58:43 2020 write: IOPS=463, BW=464MiB/s (486MB/s)(81.5GiB/180002msec) slat (usec): min=74, max=193, avg=110.57, stdev= 9.64 clat (usec): min=1938, max=5509, avg=2039.70, stdev=141.71 lat (usec): min=2028, max=5618, avg=2150.87, stdev=141.80 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 1991], 10.00th=[ 2008], 20.00th=[ 2024], | 30.00th=[ 2024], 40.00th=[ 2024], 50.00th=[ 2024], 60.00th=[ 2024], | 70.00th=[ 2024], 80.00th=[ 2024], 90.00th=[ 2040], 95.00th=[ 2073], | 99.00th=[ 2474], 99.50th=[ 2933], 99.90th=[ 4080], 99.95th=[ 4948], | 99.99th=[ 5211] bw ( KiB/s): min=235520, max=479871, per=70.54%, avg=335009.41, stdev=52266.29, samples=359 iops : min= 230, max= 468, avg=326.74, stdev=51.15, samples=359 lat (msec) : 2=10.22%, 4=89.66%, 10=0.12% cpu : usr=1.93%, sys=3.78%, ctx=83515, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,83488,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randwrite-1024k-seq: (groupid=13, jobs=1): err= 0: pid=11307: Tue Sep 1 20:58:43 2020 write: IOPS=463, BW=463MiB/s (486MB/s)(81.4GiB/180001msec) slat (usec): min=75, max=196, avg=111.41, stdev= 9.66 clat (usec): min=1941, max=5551, avg=2040.28, stdev=145.64 lat (usec): min=2028, max=5669, avg=2152.28, stdev=145.77 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 1991], 10.00th=[ 2008], 20.00th=[ 2024], | 30.00th=[ 2024], 40.00th=[ 2024], 50.00th=[ 2024], 60.00th=[ 2024], | 70.00th=[ 2024], 80.00th=[ 2024], 90.00th=[ 2040], 95.00th=[ 2073], | 99.00th=[ 2507], 99.50th=[ 2966], 99.90th=[ 4146], 99.95th=[ 5014], | 99.99th=[ 5276] bw ( KiB/s): min=235520, max=479232, per=70.57%, avg=334779.94, stdev=51410.32, samples=359 iops : min= 230, max= 468, avg=326.48, stdev=50.32, samples=359 lat (msec) : 2=10.27%, 4=89.60%, 10=0.13% cpu : usr=2.04%, sys=3.76%, ctx=83415, majf=0, minf=4 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,83391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 read-1024k-seq: (groupid=14, jobs=1): err= 0: pid=11325: Tue Sep 1 20:58:43 2020 read: IOPS=421, BW=422MiB/s (442MB/s)(74.1GiB/180001msec) slat (usec): min=77, max=178, avg=95.34, stdev= 5.91 clat (usec): min=1928, max=3554, avg=2271.15, stdev=125.75 lat (usec): min=2038, max=3649, avg=2367.15, stdev=125.42 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2073], 10.00th=[ 2114], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2245], 50.00th=[ 2278], 60.00th=[ 2311], | 70.00th=[ 2343], 80.00th=[ 2376], 90.00th=[ 2409], 95.00th=[ 2474], | 99.00th=[ 2638], 99.50th=[ 2671], 99.90th=[ 2737], 99.95th=[ 2737], | 99.99th=[ 2769] bw ( KiB/s): min=215040, max=447060, per=70.63%, avg=304887.11, stdev=48153.74, samples=359 iops : min= 210, max= 436, avg=297.32, stdev=47.13, samples=359 lat (msec) : 2=0.63%, 4=99.37% cpu : usr=0.40%, sys=4.16%, ctx=75909, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=75880,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randread-1024k-seq: (groupid=15, jobs=1): err= 0: pid=11346: Tue Sep 1 20:58:43 2020 read: IOPS=420, BW=420MiB/s (441MB/s)(73.9GiB/180002msec) slat (usec): min=75, max=188, avg=96.12, stdev= 6.06 clat (usec): min=2067, max=3372, avg=2277.11, stdev=243.35 lat (usec): min=2159, max=3473, avg=2373.90, stdev=243.24 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2114], 10.00th=[ 2147], 20.00th=[ 2147], | 30.00th=[ 2147], 40.00th=[ 2147], 50.00th=[ 2180], 60.00th=[ 2180], | 70.00th=[ 2212], 80.00th=[ 2311], 90.00th=[ 2737], 95.00th=[ 2900], | 99.00th=[ 3032], 99.50th=[ 3097], 99.90th=[ 3294], 99.95th=[ 3294], | 99.99th=[ 3359] bw ( KiB/s): min=210944, max=438272, per=70.30%, avg=302458.84, stdev=45593.16, samples=359 iops : min= 206, max= 428, avg=294.97, stdev=44.62, samples=359 lat (msec) : 4=100.00% cpu : usr=0.46%, sys=4.15%, ctx=75657, majf=0, minf=4 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=75631,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rw-1024k-seq: (groupid=16, jobs=1): err= 0: pid=11364: Tue Sep 1 20:58:43 2020 read: IOPS=193, BW=193MiB/s (202MB/s)(33.9GiB/180003msec) slat (usec): min=77, max=173, avg=95.35, stdev= 5.76 clat (usec): min=1949, max=9289, avg=2990.88, stdev=666.55 lat (usec): min=2047, max=9384, avg=3086.90, stdev=666.49 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2278], 10.00th=[ 2376], 20.00th=[ 2704], | 30.00th=[ 2769], 40.00th=[ 2868], 50.00th=[ 2900], 60.00th=[ 2933], | 70.00th=[ 2966], 80.00th=[ 3032], 90.00th=[ 3326], 95.00th=[ 4555], | 99.00th=[ 5800], 99.50th=[ 5997], 99.90th=[ 7111], 99.95th=[ 7504], | 99.99th=[ 8717] bw ( KiB/s): min=88064, max=221184, per=70.63%, avg=139617.82, stdev=23214.44, samples=359 iops : min= 86, max= 216, avg=135.93, stdev=22.78, samples=359 write: IOPS=188, BW=188MiB/s (198MB/s)(33.1GiB/180003msec) slat (usec): min=75, max=198, avg=111.43, stdev= 9.91 clat (usec): min=1914, max=6517, avg=2021.72, stdev=72.17 lat (usec): min=2029, max=6628, avg=2133.76, stdev=72.06 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 1991], 10.00th=[ 1991], 20.00th=[ 2008], | 30.00th=[ 2008], 40.00th=[ 2024], 50.00th=[ 2024], 60.00th=[ 2024], | 70.00th=[ 2024], 80.00th=[ 2024], 90.00th=[ 2024], 95.00th=[ 2040], | 99.00th=[ 2180], 99.50th=[ 2278], 99.90th=[ 3032], 99.95th=[ 3687], | 99.99th=[ 4883] bw ( KiB/s): min=96256, max=233472, per=70.72%, avg=136466.21, stdev=24927.65, samples=359 iops : min= 94, max= 228, avg=132.82, stdev=24.45, samples=359 lat (msec) : 2=5.58%, 4=90.85%, 10=3.57% cpu : usr=1.11%, sys=3.32%, ctx=68699, majf=0, minf=6 IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=34748,33923,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randrw-1024k-seq: (groupid=17, jobs=1): err= 0: pid=11388: Tue Sep 1 20:58:43 2020 read: IOPS=206, BW=206MiB/s (217MB/s)(36.3GiB/180003msec) slat (usec): min=76, max=175, avg=94.01, stdev= 5.68 clat (usec): min=2073, max=9991, avg=2606.48, stdev=796.29 lat (usec): min=2160, max=10085, avg=2701.15, stdev=796.24 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2147], 10.00th=[ 2147], 20.00th=[ 2147], | 30.00th=[ 2180], 40.00th=[ 2180], 50.00th=[ 2212], 60.00th=[ 2278], | 70.00th=[ 2573], 80.00th=[ 2933], 90.00th=[ 3851], 95.00th=[ 4555], | 99.00th=[ 5407], 99.50th=[ 5669], 99.90th=[ 6849], 99.95th=[ 8029], | 99.99th=[ 9241] bw ( KiB/s): min=104448, max=241664, per=70.37%, avg=148800.73, stdev=24958.00, samples=359 iops : min= 102, max= 236, avg=144.87, stdev=24.47, samples=359 write: IOPS=206, BW=206MiB/s (216MB/s)(36.2GiB/180003msec) slat (usec): min=74, max=192, avg=108.96, stdev= 9.67 clat (usec): min=1945, max=5546, avg=2022.97, stdev=65.26 lat (usec): min=2025, max=5663, avg=2132.54, stdev=65.15 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 1991], 10.00th=[ 2008], 20.00th=[ 2008], | 30.00th=[ 2024], 40.00th=[ 2024], 50.00th=[ 2024], 60.00th=[ 2024], | 70.00th=[ 2024], 80.00th=[ 2024], 90.00th=[ 2024], 95.00th=[ 2040], | 99.00th=[ 2180], 99.50th=[ 2278], 99.90th=[ 2638], 99.95th=[ 3261], | 99.99th=[ 5276] bw ( KiB/s): min=90112, max=247808, per=70.30%, avg=148395.48, stdev=24880.99, samples=359 iops : min= 88, max= 242, avg=144.48, stdev=24.38, samples=359 lat (msec) : 2=5.14%, 4=90.44%, 10=4.42% cpu : usr=1.23%, sys=3.53%, ctx=74303, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=37170,37107,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 write-4096k-seq: (groupid=18, jobs=1): err= 0: pid=11407: Tue Sep 1 20:58:43 2020 write: IOPS=118, BW=472MiB/s (495MB/s)(83.0GiB/180004msec) slat (usec): min=447, max=692, avg=567.64, stdev=32.59 clat (usec): min=7714, max=19842, avg=7893.77, stdev=406.83 lat (usec): min=8189, max=20426, avg=8462.03, stdev=407.73 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 7832], 10.00th=[ 7832], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7832], 50.00th=[ 7832], 60.00th=[ 7832], | 70.00th=[ 7832], 80.00th=[ 7832], 90.00th=[ 7832], 95.00th=[ 7898], | 99.00th=[ 9896], 99.50th=[10552], 99.90th=[11600], 99.95th=[16057], | 99.99th=[19268] bw ( KiB/s): min=237568, max=491520, per=70.76%, avg=342309.72, stdev=53824.25, samples=359 iops : min= 58, max= 120, avg=83.09, stdev=13.28, samples=359 lat (msec) : 10=99.10%, 20=0.90% cpu : usr=1.81%, sys=5.03%, ctx=21286, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,21260,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randwrite-4096k-seq: (groupid=19, jobs=1): err= 0: pid=11432: Tue Sep 1 20:58:43 2020 write: IOPS=118, BW=472MiB/s (495MB/s)(83.1GiB/180005msec) slat (usec): min=447, max=687, avg=566.66, stdev=33.29 clat (usec): min=7712, max=19793, avg=7892.86, stdev=395.96 lat (usec): min=8188, max=20361, avg=8460.08, stdev=396.18 clat percentiles (usec): | 1.00th=[ 7701], 5.00th=[ 7832], 10.00th=[ 7832], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7832], 50.00th=[ 7832], 60.00th=[ 7832], | 70.00th=[ 7832], 80.00th=[ 7832], 90.00th=[ 7832], 95.00th=[ 7898], | 99.00th=[ 9896], 99.50th=[10552], 99.90th=[11863], 99.95th=[14615], | 99.99th=[17695] bw ( KiB/s): min=237568, max=491520, per=70.71%, avg=342143.76, stdev=54080.96, samples=359 iops : min= 58, max= 120, avg=83.05, stdev=13.33, samples=359 lat (msec) : 10=99.09%, 20=0.91% cpu : usr=1.80%, sys=5.04%, ctx=21291, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,21263,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 read-4096k-seq: (groupid=20, jobs=1): err= 0: pid=11450: Tue Sep 1 20:58:43 2020 read: IOPS=117, BW=469MiB/s (492MB/s)(82.5GiB/180003msec) slat (usec): min=419, max=563, avg=461.16, stdev=14.08 clat (usec): min=7676, max=9308, avg=8055.88, stdev=181.81 lat (usec): min=8142, max=9765, avg=8517.63, stdev=183.00 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 7832], 10.00th=[ 7832], 20.00th=[ 7898], | 30.00th=[ 7963], 40.00th=[ 7963], 50.00th=[ 8029], 60.00th=[ 8029], | 70.00th=[ 8094], 80.00th=[ 8160], 90.00th=[ 8356], 95.00th=[ 8455], | 99.00th=[ 8586], 99.50th=[ 8717], 99.90th=[ 8979], 99.95th=[ 8979], | 99.99th=[ 9110] bw ( KiB/s): min=237568, max=491520, per=70.48%, avg=338749.42, stdev=51949.33, samples=359 iops : min= 58, max= 120, avg=82.16, stdev=12.84, samples=359 lat (msec) : 10=100.00% cpu : usr=0.09%, sys=5.45%, ctx=21150, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=21122,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randread-4096k-seq: (groupid=21, jobs=1): err= 0: pid=11472: Tue Sep 1 20:58:43 2020 read: IOPS=117, BW=470MiB/s (493MB/s)(82.6GiB/180002msec) slat (usec): min=420, max=565, avg=463.99, stdev=15.45 clat (usec): min=7753, max=9130, avg=8045.19, stdev=297.37 lat (usec): min=8212, max=9580, avg=8509.78, stdev=297.08 clat percentiles (usec): | 1.00th=[ 7832], 5.00th=[ 7832], 10.00th=[ 7832], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[ 7898], | 70.00th=[ 7963], 80.00th=[ 8356], 90.00th=[ 8586], 95.00th=[ 8717], | 99.00th=[ 8848], 99.50th=[ 8979], 99.90th=[ 9110], 99.95th=[ 9110], | 99.99th=[ 9110] bw ( KiB/s): min=237568, max=486707, per=70.71%, avg=340143.03, stdev=53861.71, samples=359 iops : min= 58, max= 118, avg=82.51, stdev=13.30, samples=359 lat (msec) : 10=100.00% cpu : usr=0.10%, sys=5.49%, ctx=21165, majf=0, minf=5 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=21139,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rw-4096k-seq: (groupid=22, jobs=1): err= 0: pid=11488: Tue Sep 1 20:58:43 2020 read: IOPS=57, BW=231MiB/s (242MB/s)(40.6GiB/180005msec) slat (usec): min=422, max=565, avg=464.90, stdev=13.87 clat (usec): min=7706, max=20703, avg=8584.42, stdev=810.12 lat (usec): min=8163, max=21154, avg=9049.91, stdev=810.14 clat percentiles (usec): | 1.00th=[ 7832], 5.00th=[ 7898], 10.00th=[ 7963], 20.00th=[ 8029], | 30.00th=[ 8094], 40.00th=[ 8225], 50.00th=[ 8455], 60.00th=[ 8586], | 70.00th=[ 8717], 80.00th=[ 8717], 90.00th=[ 9241], 95.00th=[10552], | 99.00th=[11600], 99.50th=[12256], 99.90th=[14222], 99.95th=[14615], | 99.99th=[16712] bw ( KiB/s): min=81920, max=303104, per=70.68%, avg=167267.41, stdev=33811.44, samples=359 iops : min= 20, max= 74, avg=40.42, stdev= 8.34, samples=359 write: IOPS=56, BW=226MiB/s (237MB/s)(39.7GiB/180005msec) slat (usec): min=456, max=656, avg=571.40, stdev=35.03 clat (usec): min=7712, max=19791, avg=7865.47, stdev=359.18 lat (usec): min=8188, max=20294, avg=8437.42, stdev=359.26 clat percentiles (usec): | 1.00th=[ 7701], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7832], 50.00th=[ 7832], 60.00th=[ 7832], | 70.00th=[ 7832], 80.00th=[ 7832], 90.00th=[ 7832], 95.00th=[ 7898], | 99.00th=[ 9241], 99.50th=[ 9896], 99.90th=[11600], 99.95th=[14353], | 99.99th=[19530] bw ( KiB/s): min=82359, max=303104, per=70.76%, avg=163713.97, stdev=35293.16, samples=359 iops : min= 20, max= 74, avg=39.55, stdev= 8.72, samples=359 lat (msec) : 10=96.16%, 20=3.84%, 50=0.01% cpu : usr=0.90%, sys=5.14%, ctx=20600, majf=0, minf=6 IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=10400,10168,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 randrw-4096k-seq: (groupid=23, jobs=1): err= 0: pid=11512: Tue Sep 1 20:58:43 2020 read: IOPS=57, BW=229MiB/s (240MB/s)(40.3GiB/180003msec) slat (usec): min=418, max=554, avg=460.33, stdev=13.36 clat (usec): min=7774, max=19905, avg=8367.66, stdev=813.58 lat (usec): min=8219, max=20357, avg=8828.58, stdev=813.42 clat percentiles (usec): | 1.00th=[ 7832], 5.00th=[ 7832], 10.00th=[ 7832], 20.00th=[ 7832], | 30.00th=[ 7898], 40.00th=[ 7898], 50.00th=[ 7963], 60.00th=[ 8225], | 70.00th=[ 8455], 80.00th=[ 8717], 90.00th=[ 9372], 95.00th=[10159], | 99.00th=[11338], 99.50th=[12256], 99.90th=[14091], 99.95th=[14615], | 99.99th=[15270] bw ( KiB/s): min=147456, max=319488, per=100.00%, avg=234851.36, stdev=32135.58, samples=360 iops : min= 36, max= 78, avg=57.32, stdev= 7.85, samples=360 write: IOPS=58, BW=234MiB/s (245MB/s)(41.1GiB/180003msec) slat (usec): min=446, max=714, avg=568.33, stdev=34.53 clat (usec): min=7714, max=19759, avg=7871.56, stdev=355.92 lat (usec): min=8188, max=20339, avg=8440.47, stdev=356.57 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7832], 20.00th=[ 7832], | 30.00th=[ 7832], 40.00th=[ 7832], 50.00th=[ 7832], 60.00th=[ 7832], | 70.00th=[ 7832], 80.00th=[ 7832], 90.00th=[ 7832], 95.00th=[ 7963], | 99.00th=[ 9110], 99.50th=[10159], 99.90th=[11731], 99.95th=[14353], | 99.99th=[19792] bw ( KiB/s): min=147456, max=327680, per=99.99%, avg=239287.79, stdev=32679.26, samples=360 iops : min= 36, max= 80, avg=58.40, stdev= 7.97, samples=360 lat (msec) : 10=96.99%, 20=3.01% cpu : usr=0.97%, sys=5.13%, ctx=20875, majf=0, minf=6 IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=10321,10517,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

On Aug 31, 2020, at 7:44 AM, VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> wrote:

We have older LSi Raid controller with no HBA/JBOD option. So we expose the single disks as raid0 devices. Ceph should not be aware of cache status? But digging deeper in to it it seems that 1 out of 4 serves is

:) this is just native disk performance with regular sata adapter nothing fancy, on the ceph hosts I have the SAS2308. -----Original Message----- Cc: 'ceph-users' Subject: AW: [ceph-users] Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) Wow 34K ios 4k iodetph 1 😊 How many nodes, ssd#s and what network? I can't find any firmware for the lsi card anymore... -----Ursprüngliche Nachricht----- Cc: ceph-users <ceph-users@ceph.io> Betreff: RE: [ceph-users] Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) Sorry I am not fully aware of what has been already discussed in this thread. But can't you flash these LSI logic cards to jbod? I have done this with my 9207 with sas2flash. I have attached my fio test of the Micron 5100 Pro/5200 SSDs MTFDDAK1T9TCC. They perform similar to my samsung sm863a 1,92TB. Only thing weird, is that the rw-4k is 3x slower on the micron. -----Original Message----- To: 'Reed Dier' Cc: 'ceph-users@ceph.io' Subject: [ceph-users] Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) Thank you. I was working in this direction. The situation is a lot better. But I think I can get still far better. I could set the controller to writethrough, direct and no read ahead for the ssds. But I cannot disable the pdcache ☹ there is an option set in the controller "Block SSD Write Disk Cache Change = Yes" which does not permit to deactivate the ssd cache. I could not find any solution in google for this controller (LSI MegaRAID SAS 9271-8i) to change this setting. I don’t know how much performance gain it will be to deactivate the ssd cache. At least the micron 5200max has capacitor so I hope it is safe for data loss in case if power failure. I wrote a request to lsi / Broadcom if they know how I can change this setting. This is really annyoing. I will check the cpu power settings. I rode also somewhere it can improve iops a lot. (if its bad set) At the moment I get 600iops 4k random write 1 thread and 1 iodepth. I get 40K - 4k random iops for some instances with 32iodepth. Its not the world but a lot better then before. Read around 100k iops. For 16 ssd's and 2 x dual 10G nic. I was reading that good tunings and hardware config can get more then 2000 iops on single thread out of the ssds. I know thet ceph does not shine with single thread. But 600 iops is not very much... philipp -----Ursprüngliche Nachricht----- Von: Reed Dier <reed.dier@focusvq.com> Gesendet: Dienstag, 01. September 2020 22:37 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) If using storcli/perccli for manipulating the LSI controller, you can disable the on-disk write cache with: storcli /cx/vx set pdcache=off You can also ensure that you turn off write caching at the controller level with storcli /cx/vx set iopolicy=direct storcli /cx/vx set wrcache=wt You can also tweak the readahead value for the vd if you want, though with an ssd, I don't think it will be much of an issue. storcli /cx/vx set rdcache=nora I'm sure the megacli alternatives are available with some quick searches. May also want to check your c-states and p-states to make sure there isn't any aggressive power saving features getting in the way. Reed performing a lot better and has super low commit/applay rates while the other have a lot mor (20+) on heavy writes. This just applys fore the ssd. For the hdds I cant see a difference...

-----Ursprüngliche Nachricht----- Von: Frank Schilder <frans@dtu.dk> Gesendet: Montag, 31. August 2020 13:19 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at>; 'ceph-users@ceph.io' <ceph-users@ceph.io> Betreff: Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

Yes, they can - if volatile write cache is not disabled. There are

many threads on this, also recent. Search for "disable write cache" and/or "disable volatile write cache".

You will also find different methods of doing this automatically.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Sent: 31 August 2020 13:02:45 To: 'ceph-users@ceph.io' Subject: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

I have a productive 60 osd's cluster. No extra Journals. Its

performing okay. Now I added an extra ssd Pool with 16 Micron 5100 MAX. And the performance is little slower or equal to the 60 hdd pool. 4K random as also sequential reads. All on dedicated 2 times 10G Network. HDDS are still on filestore. SSD on bluestore. Ceph Luminous.

What should be possible 16 ssd's vs. 60 hhd's no extra journals?

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

...

I assume you are referencing this parameter?

...

storcli /c0/v0 set ssdcaching=<on|off>

...

If so, this is for CacheCade, which is LSI's cache tiering solution, which should both be off and not in use for ceph.

No storcli /cx/vx set pdcache=off is denied because of the lsi setting "Block SSD Write Disk Cache Change = Yes" I cannot find any firmware to upload or way to change this Do you think that disabling the write cache also on the ssd helps a lot (ceph is not aware of this because 'smartctl -g wcache /dev/sdX shows cache disabled - because the cache on the lsi is disabled allready) The only way would be to buy some hba cards and add it to the server. But that’s a lot of work - not knowing that this will improve the speed a lot. I am using rbd with hyperconvergenced nodes (4 at the moment) pools are 2 and 3 times replicated. actually the performance for windows and linux vms with the hdd osd pool was ok. But with the time getting a little bit more slow. I just want to get ready for the future. and we plan to put some bigger database servers on the cluster (they are on local storage at the moment) and therefore I want to increase the random small iops of the cluster a lot -----Ursprüngliche Nachricht----- Von: Reed Dier <reed.dier@focusvq.com> Gesendet: Dienstag, 01. September 2020 23:44 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

there is an option set in the controller "Block SSD Write Disk Cache Change = Yes" which does not permit to deactivate the ssd cache. I could not find any solution in google for this controller (LSI MegaRAID SAS 9271-8i) to change this setting.

I assume you are referencing this parameter? storcli /c0/v0 set ssdcaching=<on|off> If so, this is for CacheCade, which is LSI's cache tiering solution, which should both be off and not in use for ceph. Single thread and single iodepth benchmarks will tend to be underwhelming. Ceph shines with aggregate performance from lots of clients. And in an odd twist of fate, I typically see better performance on RBD for random benchmarks rather than sequential benchmarks, as it distributes the load across more OSD's. Might also help others offer some pointers for tuning if you describe the pool/application a bit more. Ie RBD vs cephfs vs RGW, 3x replicated vs EC, etc. At least things are trending in a positive direction. Reed

On Sep 1, 2020, at 4:21 PM, VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> wrote:

Thank you. I was working in this direction. The situation is a lot better. But I think I can get still far better.

I could set the controller to writethrough, direct and no read ahead for the ssds. But I cannot disable the pdcache ☹ there is an option set in the controller "Block SSD Write Disk Cache Change = Yes" which does not permit to deactivate the ssd cache. I could not find any solution in google for this controller (LSI MegaRAID SAS 9271-8i) to change this setting.

I don’t know how much performance gain it will be to deactivate the ssd cache. At least the micron 5200max has capacitor so I hope it is safe for data loss in case if power failure. I wrote a request to lsi / Broadcom if they know how I can change this setting. This is really annyoing.

I will check the cpu power settings. I rode also somewhere it can improve iops a lot. (if its bad set)

At the moment I get 600iops 4k random write 1 thread and 1 iodepth. I get 40K - 4k random iops for some instances with 32iodepth. Its not the world but a lot better then before. Read around 100k iops. For 16 ssd's and 2 x dual 10G nic.

I was reading that good tunings and hardware config can get more then 2000 iops on single thread out of the ssds. I know thet ceph does not shine with single thread. But 600 iops is not very much...

philipp

-----Ursprüngliche Nachricht----- Von: Reed Dier <reed.dier@focusvq.com> Gesendet: Dienstag, 01. September 2020 22:37 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

If using storcli/perccli for manipulating the LSI controller, you can disable the on-disk write cache with: storcli /cx/vx set pdcache=off

You can also ensure that you turn off write caching at the controller level with storcli /cx/vx set iopolicy=direct storcli /cx/vx set wrcache=wt

You can also tweak the readahead value for the vd if you want, though with an ssd, I don't think it will be much of an issue. storcli /cx/vx set rdcache=nora

I'm sure the megacli alternatives are available with some quick searches.

May also want to check your c-states and p-states to make sure there isn't any aggressive power saving features getting in the way.

Reed

...

On Aug 31, 2020, at 7:44 AM, VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> wrote:

We have older LSi Raid controller with no HBA/JBOD option. So we expose the single disks as raid0 devices. Ceph should not be aware of cache status? But digging deeper in to it it seems that 1 out of 4 serves is performing a lot better and has super low commit/applay rates while the other have a lot mor (20+) on heavy writes. This just applys fore the ssd. For the hdds I cant see a difference...

-----Ursprüngliche Nachricht----- Von: Frank Schilder <frans@dtu.dk> Gesendet: Montag, 31. August 2020 13:19 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at>; 'ceph-users@ceph.io' <ceph-users@ceph.io> Betreff: Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

Yes, they can - if volatile write cache is not disabled. There are many threads on this, also recent. Search for "disable write cache" and/or "disable volatile write cache".

You will also find different methods of doing this automatically.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Sent: 31 August 2020 13:02:45 To: 'ceph-users@ceph.io' Subject: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)

I have a productive 60 osd's cluster. No extra Journals. Its performing okay. Now I added an extra ssd Pool with 16 Micron 5100 MAX. And the performance is little slower or equal to the 60 hdd pool. 4K random as also sequential reads. All on dedicated 2 times 10G Network. HDDS are still on filestore. SSD on bluestore. Ceph Luminous. What should be possible 16 ssd's vs. 60 hhd's no extra journals?

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

In theory it should be possible to do this (to change the Block SSD Write Disk Cache Change = Yes setting) 1. Run MegaSCU -adpsettings -write -f mfc.ini -a0 2. Edit the mfc.ini file, setting "blockSSDWriteCacheChange" to 0 instead of 1. 3. Run MegaSCU -adpsettings -read -f mfc.ini -a0 With MEGACLI but I get en error. It is not working for me to save the config file. No idea why… My Config: VD16 Properties : =============== Strip Size = 256 KB Number of Blocks = 3749642240 VD has Emulated PD = Yes Span Depth = 1 Number of Drives Per Span = 1 Write Cache(initial setting) = WriteThrough Disk Cache Policy = Enabled Encryption = None Data Protection = Disabled Active Operations = None Exposed to OS = Yes Creation Date = 25-08-2020 Creation Time = 12:05:41 PM Emulation type = None Version : ======= Firmware Package Build = 23.28.0-0010 Firmware Version = 3.400.05-3175 Bios Version = 5.46.02.0_4.16.08.00_0x06060900 NVDATA Version = 2.1403.03-0128 Boot Block Version = 2.05.00.00-0010 Bootloader Version = 07.26.26.219 Driver Name = megaraid_sas Driver Version = 07.703.05.00-rc1 Supported Adapter Operations : ============================ Rebuild Rate = Yes CC Rate = Yes BGI Rate = Yes Reconstruct Rate = Yes Patrol Read Rate = Yes Alarm Control = Yes Cluster Support = No BBU = Yes Spanning = Yes Dedicated Hot Spare = Yes Revertible Hot Spares = Yes Foreign Config Import = Yes Self Diagnostic = Yes Allow Mixed Redundancy on Array = No Global Hot Spares = Yes Deny SCSI Passthrough = No Deny SMP Passthrough = No Deny STP Passthrough = No Support more than 8 Phys = Yes FW and Event Time in GMT = No Support Enhanced Foreign Import = Yes Support Enclosure Enumeration = Yes Support Allowed Operations = Yes Abort CC on Error = Yes Support Multipath = Yes Support Odd & Even Drive count in RAID1E = No Support Security = No Support Config Page Model = Yes Support the OCE without adding drives = Yes support EKM = No Snapshot Enabled = No Support PFK = Yes Support PI = Yes Support LDPI Type1 = No Support LDPI Type2 = No Support LDPI Type3 = No Support Ld BBM Info = No Support Shield State = Yes Block SSD Write Disk Cache Change = Yes -> this is not good as it prevents to change the SSD cache! Stupid! Support Suspend Resume BG ops = Yes Support Emergency Spares = Yes Support Set Link Speed = Yes Support Boot Time PFK Change = No Support JBOD = Yes Disable Online PFK Change = No Support Perf Tuning = Yes Support SSD PatrolRead = Yes Real Time Scheduler = Yes Support Reset Now = Yes Support Emulated Drives = Yes Headless Mode = Yes Dedicated HotSpares Limited = No Point In Time Progress = Yes Supported VD Operations : ======================= Read Policy = Yes Write Policy = Yes IO Policy = Yes Access Policy = Yes Disk Cache Policy = Yes (but only HDD’s in this case) Reconstruction = Yes Deny Locate = No Deny CC = No Allow Ctrl Encryption = No Enable LDBBM = No Support FastPath = Yes Performance Metrics = Yes Power Savings = No Support Powersave Max With Cache = No Support Breakmirror = No Support SSC WriteBack = No Support SSC Association = No Von: Reed Dier <reed.dier@focusvq.com> Gesendet: Mittwoch, 02. September 2020 19:34 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at> Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) Just for the sake of curiosity, if you do a show all on /cX/vX, what is shown for the VD properties? VD0 Properties : ============== Strip Size = 256 KB Number of Blocks = 1953374208 VD has Emulated PD = No Span Depth = 1 Number of Drives Per Span = 1 Write Cache(initial setting) = WriteBack Disk Cache Policy = Disabled Encryption = None Data Protection = Disabled Active Operations = None Exposed to OS = Yes Creation Date = 17-06-2016 Creation Time = 02:49:02 PM Emulation type = default Cachebypass size = Cachebypass-64k Cachebypass Mode = Cachebypass Intelligent Is LD Ready for OS Requests = Yes SCSI NAA Id = 600304801bb4c0001ef6ca5ea0fcb283 I'm wondering if the pdcache value must be set at vd creation, as it is a creation option as well. If that's the case, maybe consider blowing away one of the SSD vd's and recreating the vd and OSD, and see if you can measure a difference on that disk specifically in testing. It might also be helpful to document some of these values from /cX show all Version : ======= Firmware Package Build = 24.7.0-0026 Firmware Version = 4.270.00-3972 Bios Version = 6.22.03.0_4.16.08.00_0x060B0200 Ctrl-R Version = 5.08-0006 Preboot CLI Version = 01.07-05:#%0000 NVDATA Version = 3.1411.00-0009 Boot Block Version = 3.06.00.00-0001 Driver Name = megaraid_sas Driver Version = 07.703.05.00-rc1 Supported Adapter Operations : ============================ Support Shield State = Yes Block SSD Write Disk Cache Change = Yes Support Suspend Resume BG ops = Yes Support Emergency Spares = Yes Support Set Link Speed = Yes Support Boot Time PFK Change = No Support JBOD = Yes Supported VD Operations : ======================= Read Policy = Yes Write Policy = Yes IO Policy = Yes Access Policy = Yes Disk Cache Policy = Yes Reconstruction = Yes Deny Locate = No Deny CC = No Allow Ctrl Encryption = No Enable LDBBM = No Support FastPath = Yes Performance Metrics = Yes Power Savings = No Support Powersave Max With Cache = No Support Breakmirror = Yes Support SSC WriteBack = No Support SSC Association = No Support VD Hide = Yes Support VD Cachebypass = Yes Support VD discardCacheDuringLDDelete = Yes Advanced Software Option : ======================== ---------------------------------------- Adv S/W Opt Time Remaining Mode ---------------------------------------- MegaRAID FastPath Unlimited - MegaRAID RAID6 Unlimited - MegaRAID RAID5 Unlimited - ---------------------------------------- Namely, on my 3108 controller, Block SSD Write Disk Cache Change = Yes, stands out to me. My controller has SAS HDD's behind it, though so I just may not be running into the same issue, that may pertain to me. Also wondering if FastPath is enabled as well. I know on some of the older controllers, it was a paid feature enable, but they then opened it up for free, though you may need a software key to enable it (for free). Just looking to widen the net and hope we catch something. Reed On Sep 2, 2020, at 7:38 AM, VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at<mailto:p.duerhammer@velartis.at>> wrote: I assume you are referencing this parameter? storcli /c0/v0 set ssdcaching=<on|off> If so, this is for CacheCade, which is LSI's cache tiering solution, which should both be off and not in use for ceph. No storcli /cx/vx set pdcache=off is denied because of the lsi setting "Block SSD Write Disk Cache Change = Yes" I cannot find any firmware to upload or way to change this Do you think that disabling the write cache also on the ssd helps a lot (ceph is not aware of this because 'smartctl -g wcache /dev/sdX shows cache disabled - because the cache on the lsi is disabled allready) The only way would be to buy some hba cards and add it to the server. But that’s a lot of work - not knowing that this will improve the speed a lot. I am using rbd with hyperconvergenced nodes (4 at the moment) pools are 2 and 3 times replicated. actually the performance for windows and linux vms with the hdd osd pool was ok. But with the time getting a little bit more slow. I just want to get ready for the future. and we plan to put some bigger database servers on the cluster (they are on local storage at the moment) and therefore I want to increase the random small iops of the cluster a lot -----Ursprüngliche Nachricht----- Von: Reed Dier <reed.dier@focusvq.com<mailto:reed.dier@focusvq.com>> Gesendet: Dienstag, 01. September 2020 23:44 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at<mailto:p.duerhammer@velartis.at>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Betreff: Re: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) there is an option set in the controller "Block SSD Write Disk Cache Change = Yes" which does not permit to deactivate the ssd cache. I could not find any solution in google for this controller (LSI MegaRAID SAS 9271-8i) to change this setting. I assume you are referencing this parameter? storcli /c0/v0 set ssdcaching=<on|off> If so, this is for CacheCade, which is LSI's cache tiering solution, which should both be off and not in use for ceph. Single thread and single iodepth benchmarks will tend to be underwhelming. Ceph shines with aggregate performance from lots of clients. And in an odd twist of fate, I typically see better performance on RBD for random benchmarks rather than sequential benchmarks, as it distributes the load across more OSD's. Might also help others offer some pointers for tuning if you describe the pool/application a bit more. Ie RBD vs cephfs vs RGW, 3x replicated vs EC, etc. At least things are trending in a positive direction. Reed On Sep 1, 2020, at 4:21 PM, VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at<mailto:p.duerhammer@velartis.at>> wrote: Thank you. I was working in this direction. The situation is a lot better. But I think I can get still far better. I could set the controller to writethrough, direct and no read ahead for the ssds. But I cannot disable the pdcache ☹ there is an option set in the controller "Block SSD Write Disk Cache Change = Yes" which does not permit to deactivate the ssd cache. I could not find any solution in google for this controller (LSI MegaRAID SAS 9271-8i) to change this setting. I don’t know how much performance gain it will be to deactivate the ssd cache. At least the micron 5200max has capacitor so I hope it is safe for data loss in case if power failure. I wrote a request to lsi / Broadcom if they know how I can change this setting. This is really annyoing. I will check the cpu power settings. I rode also somewhere it can improve iops a lot. (if its bad set) At the moment I get 600iops 4k random write 1 thread and 1 iodepth. I get 40K - 4k random iops for some instances with 32iodepth. Its not the world but a lot better then before. Read around 100k iops. For 16 ssd's and 2 x dual 10G nic. I was reading that good tunings and hardware config can get more then 2000 iops on single thread out of the ssds. I know thet ceph does not shine with single thread. But 600 iops is not very much... philipp -----Ursprüngliche Nachricht----- Von: Reed Dier <reed.dier@focusvq.com<mailto:reed.dier@focusvq.com>> Gesendet: Dienstag, 01. September 2020 22:37 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at<mailto:p.duerhammer@velartis.at>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Betreff: Re: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) If using storcli/perccli for manipulating the LSI controller, you can disable the on-disk write cache with: storcli /cx/vx set pdcache=off You can also ensure that you turn off write caching at the controller level with storcli /cx/vx set iopolicy=direct storcli /cx/vx set wrcache=wt You can also tweak the readahead value for the vd if you want, though with an ssd, I don't think it will be much of an issue. storcli /cx/vx set rdcache=nora I'm sure the megacli alternatives are available with some quick searches. May also want to check your c-states and p-states to make sure there isn't any aggressive power saving features getting in the way. Reed On Aug 31, 2020, at 7:44 AM, VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at<mailto:p.duerhammer@velartis.at>> wrote: We have older LSi Raid controller with no HBA/JBOD option. So we expose the single disks as raid0 devices. Ceph should not be aware of cache status? But digging deeper in to it it seems that 1 out of 4 serves is performing a lot better and has super low commit/applay rates while the other have a lot mor (20+) on heavy writes. This just applys fore the ssd. For the hdds I cant see a difference... -----Ursprüngliche Nachricht----- Von: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> Gesendet: Montag, 31. August 2020 13:19 An: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at<mailto:p.duerhammer@velartis.at>>; 'ceph-users@ceph.io<mailto:ceph-users@ceph.io>' <ceph-users@ceph.io<mailto:ceph-users@ceph.io>> Betreff: Re: Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) Yes, they can - if volatile write cache is not disabled. There are many threads on this, also recent. Search for "disable write cache" and/or "disable volatile write cache". You will also find different methods of doing this automatically. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: VELARTIS Philipp Dürhammer <p.duerhammer@velartis.at<mailto:p.duerhammer@velartis.at>> Sent: 31 August 2020 13:02:45 To: 'ceph-users@ceph.io<mailto:ceph-users@ceph.io>' Subject: [ceph-users] Can 16 server grade ssd's be slower then 60 hdds? (no extra journals) I have a productive 60 osd's cluster. No extra Journals. Its performing okay. Now I added an extra ssd Pool with 16 Micron 5100 MAX. And the performance is little slower or equal to the 60 hdd pool. 4K random as also sequential reads. All on dedicated 2 times 10G Network. HDDS are still on filestore. SSD on bluestore. Ceph Luminous. What should be possible 16 ssd's vs. 60 hhd's no extra journals? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>

FWIW a handful of years back there was a bug in at least some LSI firmware where the setting “Disk Default” silently turned the volatile cache *on* instead of the documented behavior, which was to leave alone.

On Sep 3, 2020, at 8:13 AM, Reed Dier <reed.dier@focusvq.com> wrote:

It looks like I ran into the same issue during configuration of that then, as that vd is set to "Default" and not "Disabled".