Small HDD cluster, switch from Bluestore to Filestore
Hi Everyone, There's been a few threads around about small HDD (spinning disk) clusters and performance on Bluestore. One recently from Christian (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036385.html) was particularly interesting to us as we have a very similar setup to what Christian has and we see similar performance. We have a 6 node cluster each with 12x 4TB SATA HDD, IT mode LSI 3008, wal/db on 33GB NVMe partitions. Each node has a single Xeon Gold 6132 CPU @ 2.60GHz and dual 10GB network. We also use bcache with 1 180GB NVMe partition shared between 6 osd's. Workload is via KVM (Proxmox) I did the same benchmark fio tests as Christian. Here's my results (M for me, C for Christian) direct=0 ======== M -- read : io=6008.0MB, bw=203264KB/s, iops=49, runt= 30267msec C -- read: IOPS=40, BW=163MiB/s (171MB/s)(7556MiB/46320msec) direct=1 ======== M -- read : io=32768MB, bw=1991.4MB/s, iops=497, runt= 16455ms C -- read: IOPS=314, BW=1257MiB/s (1318MB/s)(32.0GiB/26063msec) direct=0 ======== M -- write: io=32768MB, bw=471105KB/s, iops=115, runt= 71225msec C -- write: IOPS=119, BW=479MiB/s (503MB/s)(32.0GiB/68348msec direct=1 ======== M -- write: io=32768MB, bw=479829KB/s, iops=117, runt= 69930msec C -- write: IOPS=139, BW=560MiB/s (587MB/s)(32.0GiB/58519msec) I should probably mention that there was some active workload on the cluster at that time also, around 500iops write and 100MB/s throughput. The main problem that we're having with this cluster is how easy it is for it to hit slow requests and we have one particular vm that ends up doing scsi resets because of the latency. So we're considering switching these osd's to filestore. We have two other clusters using filestore/bcache/ssd journal and the performance seems to be much better on those - taking into account the different sizes. What are peoples thoughts on this size cluster? Is it just not a good fit with bluestore and our type of workload? Also, does anyone have any knowledge on future support for filestore? I'm concerned that we may have to migrate our other clusters off filestore sometime in the future and that'll hurt us with the current performance. Rich
I've found that these setting make Ceph clients have much more consistent latency than the default scheduler, it also reduces the impact of backfills and recoveries. It may not give you better performance (although I have seen it allow all disks to be utilized to 100% rather than only as fast as the slowest disk), but could help with the VM with the SCSI resets. Put this in your ceph.conf and restart your OSDs. osd op queue = wpq osd op queue cut off = high ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Aug 13, 2019 at 6:12 PM Richard Bade <hitrich@gmail.com> wrote:
Hi Everyone, There's been a few threads around about small HDD (spinning disk) clusters and performance on Bluestore. One recently from Christian ( http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036385.html ) was particularly interesting to us as we have a very similar setup to what Christian has and we see similar performance.
We have a 6 node cluster each with 12x 4TB SATA HDD, IT mode LSI 3008, wal/db on 33GB NVMe partitions. Each node has a single Xeon Gold 6132 CPU @ 2.60GHz and dual 10GB network. We also use bcache with 1 180GB NVMe partition shared between 6 osd's. Workload is via KVM (Proxmox)
I did the same benchmark fio tests as Christian. Here's my results (M for me, C for Christian) direct=0 ======== M -- read : io=6008.0MB, bw=203264KB/s, iops=49, runt= 30267msec C -- read: IOPS=40, BW=163MiB/s (171MB/s)(7556MiB/46320msec)
direct=1 ======== M -- read : io=32768MB, bw=1991.4MB/s, iops=497, runt= 16455ms C -- read: IOPS=314, BW=1257MiB/s (1318MB/s)(32.0GiB/26063msec)
direct=0 ======== M -- write: io=32768MB, bw=471105KB/s, iops=115, runt= 71225msec C -- write: IOPS=119, BW=479MiB/s (503MB/s)(32.0GiB/68348msec
direct=1 ======== M -- write: io=32768MB, bw=479829KB/s, iops=117, runt= 69930msec C -- write: IOPS=139, BW=560MiB/s (587MB/s)(32.0GiB/58519msec)
I should probably mention that there was some active workload on the cluster at that time also, around 500iops write and 100MB/s throughput. The main problem that we're having with this cluster is how easy it is for it to hit slow requests and we have one particular vm that ends up doing scsi resets because of the latency.
So we're considering switching these osd's to filestore. We have two other clusters using filestore/bcache/ssd journal and the performance seems to be much better on those - taking into account the different sizes. What are peoples thoughts on this size cluster? Is it just not a good fit with bluestore and our type of workload? Also, does anyone have any knowledge on future support for filestore? I'm concerned that we may have to migrate our other clusters off filestore sometime in the future and that'll hurt us with the current performance.
Rich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Robert, I'm trying those settings to see if they make a difference for our case. It's usually around the weekend we have issues so should have some idea by next week.
Unfortunately the scsi reset on this vm happened again last night so this hasn't resolved the issue. Thanks for the suggestion though. Rich
The overall latency in the cluster may be too high, but it was worth a shot. I've noticed that these settings really reduces the latency distribution so that it becomes more predictable and prevented some single VMs from hanging for long periods of time while others worked just fine usually when one drive was at a constant 100%. Did you notice any improvement in performance or system utilization in other ways? ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Aug 15, 2019 at 3:49 PM Rich Bade <richard.bade@btg.co.nz> wrote:
Unfortunately the scsi reset on this vm happened again last night so this hasn't resolved the issue. Thanks for the suggestion though.
Rich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I think one of your problem is bcache. Here is one example: https://habr.com/en/company/selectel/blog/450818 BR, Sebastian
On 16 Aug 2019, at 00:49, Rich Bade <richard.bade@btg.co.nz> wrote:
Unfortunately the scsi reset on this vm happened again last night so this hasn't resolved the issue. Thanks for the suggestion though.
Rich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I think one of your problem is bcache. Here is one example: https://habr.com/en/company/selectel/blog/450818 BR, Sebastian
On 16 Aug 2019, at 00:49, Rich Bade <richard.bade@btg.co.nz> wrote:
Unfortunately the scsi reset on this vm happened again last night so this hasn't resolved the issue. Thanks for the suggestion though.
Rich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi. It's not Ceph to blame! Linux does not support cached asynchronous I/O, except for the new io-uring! I.e. it supports aio calls, but they just block when you're trying to do them on an FD opened without O_DIRECT. So basically what happens when you benchmark it with -ioengine=libaio -direct=0 is that it's turning into SINGLE-THREADED I/O! Of course the single-threaded performance is worse.
Hi Everyone, There's been a few threads around about small HDD (spinning disk) clusters and performance on Bluestore. One recently from Christian (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036385.html) was particularly interesting to us as we have a very similar setup to what Christian has and we see similar performance.
We have a 6 node cluster each with 12x 4TB SATA HDD, IT mode LSI 3008, wal/db on 33GB NVMe partitions. Each node has a single Xeon Gold 6132 CPU @ 2.60GHz and dual 10GB network. We also use bcache with 1 180GB NVMe partition shared between 6 osd's. Workload is via KVM (Proxmox)
I did the same benchmark fio tests as Christian. Here's my results (M for me, C for Christian) direct=0 ======== M -- read : io=6008.0MB, bw=203264KB/s, iops=49, runt= 30267msec C -- read: IOPS=40, BW=163MiB/s (171MB/s)(7556MiB/46320msec)
direct=1 ======== M -- read : io=32768MB, bw=1991.4MB/s, iops=497, runt= 16455ms C -- read: IOPS=314, BW=1257MiB/s (1318MB/s)(32.0GiB/26063msec)
direct=0 ======== M -- write: io=32768MB, bw=471105KB/s, iops=115, runt= 71225msec C -- write: IOPS=119, BW=479MiB/s (503MB/s)(32.0GiB/68348msec
direct=1 ======== M -- write: io=32768MB, bw=479829KB/s, iops=117, runt= 69930msec C -- write: IOPS=139, BW=560MiB/s (587MB/s)(32.0GiB/58519msec)
I should probably mention that there was some active workload on the cluster at that time also, around 500iops write and 100MB/s throughput. The main problem that we're having with this cluster is how easy it is for it to hit slow requests and we have one particular vm that ends up doing scsi resets because of the latency.
So we're considering switching these osd's to filestore. We have two other clusters using filestore/bcache/ssd journal and the performance seems to be much better on those - taking into account the different sizes. What are peoples thoughts on this size cluster? Is it just not a good fit with bluestore and our type of workload? Also, does anyone have any knowledge on future support for filestore? I'm concerned that we may have to migrate our other clusters off filestore sometime in the future and that'll hurt us with the current performance.
Rich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Rich Bade
-
Richard Bade
-
Robert LeBlanc
-
Sebastian Trojanowski
-
vitalif＠yourcmc.ru