Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS
Hi all, I'm trying to find performance penalty with OSD HDD when using WAL/DB in faster device (SSD/NVMe) vs WAL/DB in same device (HDD) for different workload (RBD, RGW with index bucket in SSD pool, and CephFS with metadata in SSD pool). I want to know if giving up disk slot for WAL/DB device is worth vs adding more OSD. Unfortunately I cannot find the benchmark for these kind workload. Has anyone ever done this benchmark? Thank you.
Den ons 16 sep. 2020 kl 06:27 skrev Danni Setiawan < danni.n.setiawan@gmail.com>:
Hi all,
I'm trying to find performance penalty with OSD HDD when using WAL/DB in faster device (SSD/NVMe) vs WAL/DB in same device (HDD) for different workload (RBD, RGW with index bucket in SSD pool, and CephFS with metadata in SSD pool). I want to know if giving up disk slot for WAL/DB device is worth vs adding more OSD.
Unfortunately I cannot find the benchmark for these kind workload. Has anyone ever done this benchmark?
I think this probably is a too vague and broad question. If you ask "will my cluster handle far more write iops if I have WAL/DB (or journal) on SSD/NVME instead of on the same drive as the data", then almost everyone will agree that yes, flash WAL/DB will make your writes (and recoveries) lots quicker, since NVME/SSD will do anything from 10x to 100x the amount of small writes per second than the best spin-HDDs. But how this will affect any one single end-user experience behind S3 or CephFS without diving into a ton of implementation details like "how much ram cache does the MDS have for cephfs, how many RGWs and S3 streams are you using in parallel in order to speed up S3/RGW operations" will be very hard to say in pure numbers. Also, even if flash devices are "only" used for speeding up writes, normal clusters see a lot of mixed IO so if writes theoretically take 0ms, you get lots more free time to do reads on the HDDs, and reads often can be accelerated with RAM caches in various places. So like any other storage system, if you put a flash device in front of the spinners you will see improvements, especially for many small write ops, but if your use case consists of "copy these 100 10G-images to this pool every night" or "every hour we unzip the sources to a large program and checksum the files and then clean the directory" will have a large impact on how flash helps your cluster. Also, more boxes add more performance in more ways than just "more disk", every extra cpu, every G ram, every extra network port means the overall perf of the cluster goes up by sharing the total load better. This will not show up in simple one-threaded tests but as you get 2-5-10-100 active clients doing IO it will be noticeable. -- May the most significant bit of your life be positive.
Yes, I agree that there are many knob for fine tuning Ceph performance. The problem is we don't have data which workload that benefit most from WAL/DB in SSD vs in same spinning drive and by how much. Does it really help in a cluster that mostly for object storage/RGW? Or may be just block storage/RBD workload that benefit most? IMHO, I think we need some cost-benefit analysis from this because the cost placing WAL/DB in SSD is quite noticeable (multiple OSD would be fail when SSD fail and capacity reduced). Thanks. On 16/09/20 14.45, Janne Johansson wrote:
Den ons 16 sep. 2020 kl 06:27 skrev Danni Setiawan <danni.n.setiawan@gmail.com <mailto:danni.n.setiawan@gmail.com>>:
Hi all,
I'm trying to find performance penalty with OSD HDD when using WAL/DB in faster device (SSD/NVMe) vs WAL/DB in same device (HDD) for different workload (RBD, RGW with index bucket in SSD pool, and CephFS with metadata in SSD pool). I want to know if giving up disk slot for WAL/DB device is worth vs adding more OSD.
Unfortunately I cannot find the benchmark for these kind workload. Has anyone ever done this benchmark?
I think this probably is a too vague and broad question. If you ask "will my cluster handle far more write iops if I have WAL/DB (or journal) on SSD/NVME instead of on the same drive as the data", then almost everyone will agree that yes, flash WAL/DB will make your writes (and recoveries) lots quicker, since NVME/SSD will do anything from 10x to 100x the amount of small writes per second than the best spin-HDDs. But how this will affect any one single end-user experience behind S3 or CephFS without diving into a ton of implementation details like "how much ram cache does the MDS have for cephfs, how many RGWs and S3 streams are you using in parallel in order to speed up S3/RGW operations" will be very hard to say in pure numbers.
Also, even if flash devices are "only" used for speeding up writes, normal clusters see a lot of mixed IO so if writes theoretically take 0ms, you get lots more free time to do reads on the HDDs, and reads often can be accelerated with RAM caches in various places.
So like any other storage system, if you put a flash device in front of the spinners you will see improvements, especially for many small write ops, but if your use case consists of "copy these 100 10G-images to this pool every night" or "every hour we unzip the sources to a large program and checksum the files and then clean the directory" will have a large impact on how flash helps your cluster.
Also, more boxes add more performance in more ways than just "more disk", every extra cpu, every G ram, every extra network port means the overall perf of the cluster goes up by sharing the total load better. This will not show up in simple one-threaded tests but as you get 2-5-10-100 active clients doing IO it will be noticeable.
-- May the most significant bit of your life be positive.
It does, RGW really needs SSDs for bucket indexes. CephFS also needs SSDs for metadata in any setup that's used by more than 1 user :). RBD in fact doesn't benefit much from the WAL/DB partition alone because Bluestore never does more writes per second than HDD can do on average (it flushes every 32 writes to the HDD). For RBD, the best thing is bcache. Just try to fill up your OSDs up to a decent point to see the difference because a lot of objects means a lot of metadata and when there's a lot of metadata it stops fitting in cache. The performance and the performance difference will also depend on whether your HDDs have internal SSD/media cache (a lot of them do even if you're unaware of it). +1 for hsbench, just be careful and use my repo https://github.com/vitalif/hsbench because the original has at least 2 bugs for now: 1) it only reads first 64KB when benchmarking GETs 2) it reads objects sequentially instead of reading them randomly The first one actually has a fix waiting to be merged in a someone's pull request, the second is my fix, I can submit a PR later.
Yes, I agree that there are many knob for fine tuning Ceph performance. The problem is we don't have data which workload that benefit most from WAL/DB in SSD vs in same spinning drive and by how much. Does it really help in a cluster that mostly for object storage/RGW? Or may be just block storage/RBD workload that benefit most?
IMHO, I think we need some cost-benefit analysis from this because the cost placing WAL/DB in SSD is quite noticeable (multiple OSD would be fail when SSD fail and capacity reduced).
On 9/17/20 12:21 PM, vitalif@yourcmc.ru wrote:
It does, RGW really needs SSDs for bucket indexes. CephFS also needs SSDs for metadata in any setup that's used by more than 1 user :). RBD in fact doesn't benefit much from the WAL/DB partition alone because Bluestore never does more writes per second than HDD can do on average (it flushes every 32 writes to the HDD). For RBD, the best thing is bcache.
Even just having the extra burst bandwidth available can be a big win though, especially in HDD cases with a 64k min_alloc size for the WAL and SST reads into cache for onode misses.
Just try to fill up your OSDs up to a decent point to see the difference because a lot of objects means a lot of metadata and when there's a lot of metadata it stops fitting in cache. The performance and the performance difference will also depend on whether your HDDs have internal SSD/media cache (a lot of them do even if you're unaware of it).
+1 for hsbench, just be careful and use my repo https://github.com/vitalif/hsbench because the original has at least 2 bugs for now: 1) it only reads first 64KB when benchmarking GETs 2) it reads objects sequentially instead of reading them randomly
The first one actually has a fix waiting to be merged in a someone's pull request, the second is my fix, I can submit a PR later.
Yes, please submit bug fixes! I was waiting for a reply on the read issue regarding the implementation, but For sequential vs random gets that should be fairly straightforward (Though I would make it a new mode switch preferably so we can keep the existing option as well) Mark
Yes, I agree that there are many knob for fine tuning Ceph performance. The problem is we don't have data which workload that benefit most from WAL/DB in SSD vs in same spinning drive and by how much. Does it really help in a cluster that mostly for object storage/RGW? Or may be just block storage/RBD workload that benefit most?
IMHO, I think we need some cost-benefit analysis from this because the cost placing WAL/DB in SSD is quite noticeable (multiple OSD would be fail when SSD fail and capacity reduced).
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 17/09/2020 19:21, vitalif@yourcmc.ru wrote:
RBD in fact doesn't benefit much from the WAL/DB partition alone because Bluestore never does more writes per second than HDD can do on average (it flushes every 32 writes to the HDD). For RBD, the best thing is bcache.
rbd will benefit: for each write data iop, there could be a metada read iop (unless it is cached) + a write iop, so taking these extra metadata iops away from the hdd will make a difference for small block sizes. Even for data flushes (not sure if it is 32 or 64) if the data is not totally random, the io scheduler for the hdd (cfq or deadline) will either merge blocks or order them in a way which can sustain higher client iops. we did test dm-cache, bcache and dm-writecache, we found the later to be much better. /Maged
Just try to fill up your OSDs up to a decent point to see the difference because a lot of objects means a lot of metadata and when there's a lot of metadata it stops fitting in cache. The performance and the performance difference will also depend on whether your HDDs have internal SSD/media cache (a lot of them do even if you're unaware of it).
+1 for hsbench, just be careful and use my repo https://github.com/vitalif/hsbench because the original has at least 2 bugs for now: 1) it only reads first 64KB when benchmarking GETs 2) it reads objects sequentially instead of reading them randomly
The first one actually has a fix waiting to be merged in a someone's pull request, the second is my fix, I can submit a PR later.
Yes, I agree that there are many knob for fine tuning Ceph performance. The problem is we don't have data which workload that benefit most from WAL/DB in SSD vs in same spinning drive and by how much. Does it really help in a cluster that mostly for object storage/RGW? Or may be just block storage/RBD workload that benefit most?
IMHO, I think we need some cost-benefit analysis from this because the cost placing WAL/DB in SSD is quite noticeable (multiple OSD would be fail when SSD fail and capacity reduced).
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Maged, Do you mean dm-writecache is better than B-cache in terms of small IO performance. By how much? Could you please share us a bit more details? thanks in advance, Samuel huxiaoyu@horebdata.cn From: Maged Mokhtar Date: 2020-09-18 02:12 To: ceph-users Subject: [ceph-users] Re: Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS On 17/09/2020 19:21, vitalif@yourcmc.ru wrote:
RBD in fact doesn't benefit much from the WAL/DB partition alone because Bluestore never does more writes per second than HDD can do on average (it flushes every 32 writes to the HDD). For RBD, the best thing is bcache.
rbd will benefit: for each write data iop, there could be a metada read iop (unless it is cached) + a write iop, so taking these extra metadata iops away from the hdd will make a difference for small block sizes. Even for data flushes (not sure if it is 32 or 64) if the data is not totally random, the io scheduler for the hdd (cfq or deadline) will either merge blocks or order them in a way which can sustain higher client iops. we did test dm-cache, bcache and dm-writecache, we found the later to be much better. /Maged
Just try to fill up your OSDs up to a decent point to see the difference because a lot of objects means a lot of metadata and when there's a lot of metadata it stops fitting in cache. The performance and the performance difference will also depend on whether your HDDs have internal SSD/media cache (a lot of them do even if you're unaware of it).
+1 for hsbench, just be careful and use my repo https://github.com/vitalif/hsbench because the original has at least 2 bugs for now: 1) it only reads first 64KB when benchmarking GETs 2) it reads objects sequentially instead of reading them randomly
The first one actually has a fix waiting to be merged in a someone's pull request, the second is my fix, I can submit a PR later.
Yes, I agree that there are many knob for fine tuning Ceph performance. The problem is we don't have data which workload that benefit most from WAL/DB in SSD vs in same spinning drive and by how much. Does it really help in a cluster that mostly for object storage/RGW? Or may be just block storage/RBD workload that benefit most?
IMHO, I think we need some cost-benefit analysis from this because the cost placing WAL/DB in SSD is quite noticeable (multiple OSD would be fail when SSD fail and capacity reduced).
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2020-09-17 19:21, vitalif@yourcmc.ru wrote:
It does, RGW really needs SSDs for bucket indexes. CephFS also needs SSDs for metadata in any setup that's used by more than 1 user :).
Nah. I crashed my first cephfs with my music library, a 2 TB git annex repo, just me alone (slow ops on mds). creating a cephfs on a non ssd/nvme metadata pool should require --i-really-want-this flag :) poelzi
On 16/09/2020 07:26, Danni Setiawan wrote:
Hi all,
I'm trying to find performance penalty with OSD HDD when using WAL/DB in faster device (SSD/NVMe) vs WAL/DB in same device (HDD) for different workload (RBD, RGW with index bucket in SSD pool, and CephFS with metadata in SSD pool). I want to know if giving up disk slot for WAL/DB device is worth vs adding more OSD.
Unfortunately I cannot find the benchmark for these kind workload. Has anyone ever done this benchmark?
For everything except CephFS, fio looks like a best tool for benchmarking. It can benchmark ceph on all levels: rados, rbd, http/S3. Moreover, it has excellent configuration options, detailed metrics and it can run with multi-server workload (one fio client forcing many fio servers to do benchmarking). The own fio performance is at about 15M IOPS (null engine per fio-server), and it scales horizontally.
Does fio handle S3 objects spread across many buckets well? I think bucket listing performance was maybe missing too, but It's been a while since I looked at fio's S3 support. Maybe they have those use cases covered now. I wrote a go based benchmark called hsbench based on the wasabi-tech benchmark a while back that tries to cover some of those cases, but I haven't touched it in a while: https://github.com/markhpc/hsbench FWIW fio can be used for cephfs as well and it works reasonably well if you give it a long enough run time and only expect hero run scenarios from it. For metadata intensive workloads you'll need to use mdtest or smallfile. At this point I mostly just use the io500 suite that includes both ior for hero runs and mdtest for metadata (but you need mpi to coordinate it across multiple nodes). Mark On 9/17/20 3:35 AM, George Shuklin wrote:
On 16/09/2020 07:26, Danni Setiawan wrote:
Hi all,
I'm trying to find performance penalty with OSD HDD when using WAL/DB in faster device (SSD/NVMe) vs WAL/DB in same device (HDD) for different workload (RBD, RGW with index bucket in SSD pool, and CephFS with metadata in SSD pool). I want to know if giving up disk slot for WAL/DB device is worth vs adding more OSD.
Unfortunately I cannot find the benchmark for these kind workload. Has anyone ever done this benchmark?
For everything except CephFS, fio looks like a best tool for benchmarking. It can benchmark ceph on all levels: rados, rbd, http/S3. Moreover, it has excellent configuration options, detailed metrics and it can run with multi-server workload (one fio client forcing many fio servers to do benchmarking). The own fio performance is at about 15M IOPS (null engine per fio-server), and it scales horizontally. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 17/09/2020 17:37, Mark Nelson wrote:
Does fio handle S3 objects spread across many buckets well? I think bucket listing performance was maybe missing too, but It's been a while since I looked at fio's S3 support. Maybe they have those use cases covered now. I wrote a go based benchmark called hsbench based on the wasabi-tech benchmark a while back that tries to cover some of those cases, but I haven't touched it in a while:
The way to spread across many buckets is to use 'farm' for servers under one client manage. You just give each server a different bucket to torture inside jobfile. iodepth=1 restriction for http ioengine is actually encouraging this.
FWIW fio can be used for cephfs as well and it works reasonably well if you give it a long enough run time and only expect hero run scenarios from it. For metadata intensive workloads you'll need to use mdtest or smallfile. At this point I mostly just use the io500 suite that includes both ior for hero runs and mdtest for metadata (but you need mpi to coordinate it across multiple nodes).
Yep, I've talked about metadata intensive workloads. Romping within a file or two is not a true fs-specific benchmark.
participants (8)
-
Daniel Poelzleithner
-
Danni Setiawan
-
George Shuklin
-
huxiaoyu＠horebdata.cn
-
Janne Johansson
-
Maged Mokhtar
-
Mark Nelson
-
vitalif＠yourcmc.ru