Best practices for OSD on bcache
Hi, guys I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks.
On Mon, Mar 01, 2021 at 12:37:38PM +0800, Norman.Kern wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks.
Hi, sorry to say, but since use cases and workloads differ so much, there is no easy list of best practises. Number one reason for low bcache performance is consumer-grade caching devices, since bcache does a lot of write amplification and not even "PRO" consumer devices will give you decent and consistent performance. You might even end up with worse performance than on direct HDD under load. With decent caching device, there still are quite a few tuning knobs in bcache, but it all depends on your workload. You also have to consider the added complexity of a bcache setup for maintenance operations. Moving an OSD between hosts becomes a complex operation (wait for bcache draining, detach bcache, move HDD, create new bcache caching device, attach bcache). Regards Matthias
On 2021/3/1 下午6:32, Matthias Ferdinand wrote:
On Mon, Mar 01, 2021 at 12:37:38PM +0800, Norman.Kern wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks. Hi,
sorry to say, but since use cases and workloads differ so much, there is no easy list of best practises.
Number one reason for low bcache performance is consumer-grade caching devices, since bcache does a lot of write amplification and not even "PRO" consumer devices will give you decent and consistent performance. You might even end up with worse performance than on direct HDD under load.
With decent caching device, there still are quite a few tuning knobs in bcache, but it all depends on your workload.
You also have to consider the added complexity of a bcache setup for maintenance operations. Moving an OSD between hosts becomes a complex operation (wait for bcache draining, detach bcache, move HDD, create new bcache caching device, attach bcache).
Matthias, I agreed with you for tuning. I ask this question just for that my OSDs have problems when the cache_available_percent less than 30, the SSDs almost useless and all I/Os bypass to HDDs with large latency. So I think maybe I have wrong configs for bcache.
Regards Matthias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Mar 02, 2021 at 05:47:29PM +0800, Norman.Kern wrote:
Matthias,
I agreed with you for tuning. I ask this question just for that my OSDs have problems when the
cache_available_percent less than 30, the SSDs almost useless and all I/Os bypass to HDDs with large latency.
Hi, I used to tune writeback_percent as far down as 1. I guess rapid writeback helped keep complexity (CPU, additional I/O) of handling dirty blocks low. Hoarding dirty data for a better chance to eventually turn it into sequential I/O is an important gain on MD-RAID5/6, but not so much on a single disk. Perhaps at cache_available_percent < 30 bcache needs to do some garbage collection. This would at least give some CPU spike, and probably some additional I/O spike for the on-disk data structure. This is where you really want to have decent DC-grade caching devices that can keep up with this sort of write amplification spikes. Consumer-grade devices won't be able to, and even add their own very much noticeable internal garbage collection on top. Bypassing the caching SSD device on (non-sequential) I/O is usually a symptom of bcache detecting a saturated caching device, i.e. the SSDs are probably not DC-grade. At this point you get all the latency of the backing HDD, plus some more from metadata handling. You might even tune bcache to never bypass, but at this point, it would only further add to the latency. Regards Matthias
Hallo, do you expect that to be better (faster), than having the OSD's Journal on a different disk (ssd, nvme) ? rgds, derjohn On 01.03.21 05:37, Norman.Kern wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
On 2021/3/2 上午5:09, Andreas John wrote:
Hallo,
do you expect that to be better (faster), than having the OSD's Journal on a different disk (ssd, nvme) ? No, I created the OSD storage devices using bcache devices.
rgds,
derjohn
On 01.03.21 05:37, Norman.Kern wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, we clearly understood that. But in ceph we have the concept of "OSD Journal on very fast different disk". I just asked what in theory should be the advantage of caching on bcache/NVME vs. Journal/NVME. I would not expect any performance advantage for bcache (if the Journal is reasonably sized). I might be totally wrong, though. If you just do it, because you don't want to re-create (or modify) the OSDs, it's not worth the effort IMHO. rgds, derjohn On 02.03.21 10:48, Norman.Kern wrote:
On 2021/3/2 上午5:09, Andreas John wrote:
Hallo,
do you expect that to be better (faster), than having the OSD's Journal on a different disk (ssd, nvme) ? No, I created the OSD storage devices using bcache devices.
rgds,
derjohn
On 01.03.21 05:37, Norman.Kern wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
Hi Norman On Mon, Mar 1, 2021 at 4:38 AM Norman.Kern <norman.kern@gmx.com> wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks.
I've used bcache quite a bit with Ceph with the following configuration options tweaked a) use writeback mode rather than writethrough (which is the default) This ensures that the cache device is actually used for write caching b) turn off the sequential cutoff sequential_cutoff = 0 This means that sequential writes will also always go to the cache device rather than the backing device c) disable the congestion read and write thresholds congested_read_threshold_us = congested_write_threshold_us = 0 The following repository: https://git.launchpad.net/charm-bcache-tuning/tree/src/files has a python script and systemd configuration todo b) and c) automatically on all bcache devices on boot; a) we let the provisioning system take care of. HTH
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2021/3/2 下午4:49, James Page wrote:
Hi Norman
On Mon, Mar 1, 2021 at 4:38 AM Norman.Kern <norman.kern@gmx.com> wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks.
I've used bcache quite a bit with Ceph with the following configuration options tweaked
a) use writeback mode rather than writethrough (which is the default)
This ensures that the cache device is actually used for write caching
b) turn off the sequential cutoff
sequential_cutoff = 0
This means that sequential writes will also always go to the cache device rather than the backing device
c) disable the congestion read and write thresholds
congested_read_threshold_us = congested_write_threshold_us = 0
The following repository:
https://git.launchpad.net/charm-bcache-tuning/tree/src/files
has a python script and systemd configuration todo b) and c) automatically on all bcache devices on boot; a) we let the provisioning system take care of.
HTH
I have set the variables described above. Didn't you met the latency problems when cache used increaced 30%? My cache status like this: root@WXS0089:~# cat /sys/block/sda/bcache/priority_stats Unused: 4% Clean: 28% Dirty: 70% Metadata: 0% Average: 551 Sectors per Q: 29197312 Quantiles: [27 135 167 199 230 262 294 326 358 390 422 454 486 517 549 581 613 645 677 709 741 773 804 836 844 847 851 855 860 868 881]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
James, Can you tell me what's the hardware config of your bcache? I use the 400G SATA SSD as cache device and 10T HDD as the storage device. Hardware relationed? On 2021/3/2 下午4:49, James Page wrote:
Hi Norman
On Mon, Mar 1, 2021 at 4:38 AM Norman.Kern <norman.kern@gmx.com> wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks.
I've used bcache quite a bit with Ceph with the following configuration options tweaked
a) use writeback mode rather than writethrough (which is the default)
This ensures that the cache device is actually used for write caching
b) turn off the sequential cutoff
sequential_cutoff = 0
This means that sequential writes will also always go to the cache device rather than the backing device
c) disable the congestion read and write thresholds
congested_read_threshold_us = congested_write_threshold_us = 0
The following repository:
https://git.launchpad.net/charm-bcache-tuning/tree/src/files
has a python script and systemd configuration todo b) and c) automatically on all bcache devices on boot; a) we let the provisioning system take care of.
HTH
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Norman On Wed, Mar 3, 2021 at 2:47 AM Norman.Kern <norman.kern@gmx.com> wrote:
James,
Can you tell me what's the hardware config of your bcache? I use the 400G SATA SSD as cache device and
10T HDD as the storage device. Hardware relationed?
It might be - all of the deployments I've seen/worked with use NVMe SSD devices and some more recent ones have used NVMe attached Optane devices as well (but that is usual). Backing HDD's are SAS attached 12TB ish 7K spinning disks.
On 2021/3/2 下午4:49, James Page wrote:
Hi Norman
On Mon, Mar 1, 2021 at 4:38 AM Norman.Kern <norman.kern@gmx.com> wrote:
Hi, guys
I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks.
I've used bcache quite a bit with Ceph with the following configuration options tweaked
a) use writeback mode rather than writethrough (which is the default)
This ensures that the cache device is actually used for write caching
b) turn off the sequential cutoff
sequential_cutoff = 0
This means that sequential writes will also always go to the cache device rather than the backing device
c) disable the congestion read and write thresholds
congested_read_threshold_us = congested_write_threshold_us = 0
The following repository:
https://git.launchpad.net/charm-bcache-tuning/tree/src/files
has a python script and systemd configuration todo b) and c) automatically on all bcache devices on boot; a) we let the provisioning system take care of.
HTH
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
вт, 2 мар. 2021 г. в 13:52, James Page <james.page@canonical.com>: (Disclaimer: I have never tried to run Ceph on bcache in production, and the test cluster was destroyed before reaching its first deep scrub)
b) turn off the sequential cutoff
sequential_cutoff = 0
This means that sequential writes will also always go to the cache device rather than the backing device
Could you please explain the exact mechanics of the sequential cutoff? Does it only affect big sequential writes, or big sequential reads too? I am asking because of the potential of deep scrubs being cached instead of the "real" hot data. -- Alexander E. Patrakov CV: http://u.pc.cd/wT8otalK
participants (5)
-
Alexander E. Patrakov
-
Andreas John
-
James Page
-
Matthias Ferdinand
-
Norman.Kern