any experience on using Bcache on top of HDD OSD
Dear Cephers, Just curious about any one who has some experience on using Bcache on top of HDD OSD to accelerate IOPS performance? If any, how about the stability and the performance improvement, and for how long the running time? I am evaluating whether its is worth to use Bcache on top of HDD OSD. I have done some experimental on using bcache with OSD, and would like to make sure before doing it on a production cluster. thanks in advance, samuel huxiaoyu@horebdata.cn
On Sun, Apr 18, 2021 at 10:31:30PM +0200, huxiaoyu@horebdata.cn wrote:
Dear Cephers,
Just curious about any one who has some experience on using Bcache on top of HDD OSD to accelerate IOPS performance?
If any, how about the stability and the performance improvement, and for how long the running time?
Hi, I have not used bcache with Bluestore, but I use bcache in a Jewel cluster with Filestore on XFS on bcache on HDD, and I haven't seen any bcache-related trouble with this setup so far. I don't have journal on bcache, journal is separated out to SSD. From time to time I drained and detached the caching device from an OSD just to see if the added complexity still has some value, but latency (as measured by iostat on the OSD) would go up by a factor of about 2 so I kept bcache active on HDD OSDs. For Bluestore I don't know how much (if at all) bcache would improve performance where WAL/DB is placed on SSD already. A word of warning: never use non-DC-class SSDs for bcache caching devices. bcache does some write amplification, and you will regret it if using consumer grade SSDs. Might even turn out slower than plain HDD with highly irregular latency spikes. Regards Matthias
Hi, I also have used bcache extensively on filestore with journals on SSD for at least 5 years. This has worked very well in all versions up to luminous. The iops improvement was definitely beneficial for vm disk images in rbd. I am also using it under bluestore with db/wal on nvme on both Luminous and Nautilus. This is a smaller cluster also for vm disk images on rbd, and the iops appeared to be improved. However as Matthias mentioned, be sure to get the right types of SSD's as you can have strange performance issues. Make sure you do fio testing as I found on paper Intel DC S4600 looked ok but performed very badly in sequential 4k writes. Also keep an eye on your drive writes per day as in a busy cluster you may hit 1DWPD or more. Stability has been very good with bcache. I'm using it on Ubuntu and have had to use newer kernels on some of the historical LTS releases. In 16.04 the 4.15 edge kernel is recommended, newer versions stock kernel is fine. Rich On Tue, 20 Apr 2021 at 03:37, Matthias Ferdinand <mf+ml.ceph@mfedv.net> wrote:
On Sun, Apr 18, 2021 at 10:31:30PM +0200, huxiaoyu@horebdata.cn wrote:
Dear Cephers,
Just curious about any one who has some experience on using Bcache on top of HDD OSD to accelerate IOPS performance?
If any, how about the stability and the performance improvement, and for how long the running time?
Hi,
I have not used bcache with Bluestore, but I use bcache in a Jewel cluster with Filestore on XFS on bcache on HDD, and I haven't seen any bcache-related trouble with this setup so far. I don't have journal on bcache, journal is separated out to SSD. From time to time I drained and detached the caching device from an OSD just to see if the added complexity still has some value, but latency (as measured by iostat on the OSD) would go up by a factor of about 2 so I kept bcache active on HDD OSDs.
For Bluestore I don't know how much (if at all) bcache would improve performance where WAL/DB is placed on SSD already.
A word of warning: never use non-DC-class SSDs for bcache caching devices. bcache does some write amplification, and you will regret it if using consumer grade SSDs. Might even turn out slower than plain HDD with highly irregular latency spikes.
Regards Matthias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Mattias， Very glad to know that your setting with Bcache works well in production. How long have you been puting XFS on bcache on HDD in production? Which bcache version (i mean the kernel) do you use? or do you use a special version of bcache? thanks in advance, samuel huxiaoyu@horebdata.cn From: Matthias Ferdinand Date: 2021-04-19 17:35 To: huxiaoyu@horebdata.cn CC: ceph-users Subject: Re: [ceph-users] any experience on using Bcache on top of HDD OSD On Sun, Apr 18, 2021 at 10:31:30PM +0200, huxiaoyu@horebdata.cn wrote:
Dear Cephers,
Just curious about any one who has some experience on using Bcache on top of HDD OSD to accelerate IOPS performance?
If any, how about the stability and the performance improvement, and for how long the running time?
Hi, I have not used bcache with Bluestore, but I use bcache in a Jewel cluster with Filestore on XFS on bcache on HDD, and I haven't seen any bcache-related trouble with this setup so far. I don't have journal on bcache, journal is separated out to SSD. From time to time I drained and detached the caching device from an OSD just to see if the added complexity still has some value, but latency (as measured by iostat on the OSD) would go up by a factor of about 2 so I kept bcache active on HDD OSDs. For Bluestore I don't know how much (if at all) bcache would improve performance where WAL/DB is placed on SSD already. A word of warning: never use non-DC-class SSDs for bcache caching devices. bcache does some write amplification, and you will regret it if using consumer grade SSDs. Might even turn out slower than plain HDD with highly irregular latency spikes. Regards Matthias
On Tue, Apr 20, 2021 at 08:27:50AM +0200, huxiaoyu@horebdata.cn wrote:
Dear Mattias，
Very glad to know that your setting with Bcache works well in production.
How long have you been puting XFS on bcache on HDD in production? Which bcache version (i mean the kernel) do you use? or do you use a special version of bcache?
Hi, this installation has been running for at least 4 years, on Ubuntu 16.04 with Ubuntu kernel 4.15 (from package linux-image-generic-hwe-16.04). Matthias
participants (3)
-
huxiaoyu＠horebdata.cn
-
Matthias Ferdinand
-
Richard Bade