Bluestore min_alloc size space amplification cheatsheet
Hi Folks, We're discussing changing the minimum allocation size in bluestore to 4k. For flash devices this appears to be a no-brainer. We've made the write path fast enough in bluestore that we're typically seeing either the same or faster performance with a 4K min_alloc size and the space savings for small objects easily outweigh the increase in metadata for large fragmented objects. For HDDs there are tradeoffs. A smaller allocation size means more fragmentation when there are small overwrites (like in RBD) which can mean a lot more seeks. Igor was showing some fairly steep RBD performance drops for medium-large reads/writes once the OSDs started to become fragmented. For RGW this isn't nearly as big of a deal though since typically the objects shouldn't become fragmented. A small (4K) allocation size does mean however that we can write out 4K random writes sequentially and gain a big IOPS win which theoretically should benefit both RBD and RGW. Regarding space-amplification, Josh pointed out that our current 64K allocation size has huge ramifications for overall space-amp when writing out medium sized objects to EC pools. In an attempt to actually quantify this, I made a spreadsheet with some graphs showing a couple of examples of how the min_alloc size and replication/EC interact with each other at different object sizes. The gist of it is that with our current default HDD min_alloc size (64K), erasure coding can actually have worse space amplification than 3X replication, even with moderately large (128K) object sizes. How much this factors into the decision vs fragmentation is a tough call, but I wanted to at least showcase the behavior as we work through deciding what our default HDD behavior should be. https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP... Thanks, Mark
On Thu, 21 Nov 2019, Mark Nelson wrote:
Hi Folks,
We're discussing changing the minimum allocation size in bluestore to 4k. For flash devices this appears to be a no-brainer. We've made the write path fast enough in bluestore that we're typically seeing either the same or faster performance with a 4K min_alloc size and the space savings for small objects easily outweigh the increase in metadata for large fragmented objects.
For HDDs there are tradeoffs. A smaller allocation size means more fragmentation when there are small overwrites (like in RBD) which can mean a lot more seeks. Igor was showing some fairly steep RBD performance drops for medium-large reads/writes once the OSDs started to become fragmented. For RGW this isn't nearly as big of a deal though since typically the objects shouldn't become fragmented. A small (4K) allocation size does mean however that we can write out 4K random writes sequentially and gain a big IOPS win which theoretically should benefit both RBD and RGW.
Regarding space-amplification, Josh pointed out that our current 64K allocation size has huge ramifications for overall space-amp when writing out medium sized objects to EC pools. In an attempt to actually quantify this, I made a spreadsheet with some graphs showing a couple of examples of how the min_alloc size and replication/EC interact with each other at different object sizes. The gist of it is that with our current default HDD min_alloc size (64K), erasure coding can actually have worse space amplification than 3X replication, even with moderately large (128K) object sizes. How much this factors into the decision vs fragmentation is a tough call, but I wanted to at least showcase the behavior as we work through deciding what our default HDD behavior should be.
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP...
The key difference (at the bluestore level) between RGW and RBD/CephFS writes is that RGW passes down the CEPH_OSD_ALLOC_HINT_FLAG_IMMUTABLE | CEPH_OSD_ALLOC_HINT_FLAG_APPEND_ONLY hints. The immutable one in particular is what we really care about, since it's the mutable objects that get overwrites that lead to (most) fragmentation. We should use this to decide whether to create minimal (min_alloc_size) blobs or whether we should keep the blobs larger to limit fragmentation. I'm not sure what we would call the config option that isn't super confusing... maybe bluestore_mutable_min_blob_size? bluestore_baseline_min_blob_size? sage
I'd support a smaller default, if not only for immutable. We did some testing of "small" 64KB objects on a 4+2 pool, and the amp was clearly a huge issue. On Thu, Nov 21, 2019 at 1:50 PM Sage Weil <sage@newdream.net> wrote:
On Thu, 21 Nov 2019, Mark Nelson wrote:
Hi Folks,
We're discussing changing the minimum allocation size in bluestore to 4k. For flash devices this appears to be a no-brainer. We've made the write path fast enough in bluestore that we're typically seeing either the same or faster performance with a 4K min_alloc size and the space savings for small objects easily outweigh the increase in metadata for large fragmented objects.
For HDDs there are tradeoffs. A smaller allocation size means more fragmentation when there are small overwrites (like in RBD) which can mean a lot more seeks. Igor was showing some fairly steep RBD performance drops for medium-large reads/writes once the OSDs started to become fragmented. For RGW this isn't nearly as big of a deal though since typically the objects shouldn't become fragmented. A small (4K) allocation size does mean however that we can write out 4K random writes sequentially and gain a big IOPS win which theoretically should benefit both RBD and RGW.
Regarding space-amplification, Josh pointed out that our current 64K allocation size has huge ramifications for overall space-amp when writing out medium sized objects to EC pools. In an attempt to actually quantify this, I made a spreadsheet with some graphs showing a couple of examples of how the min_alloc size and replication/EC interact with each other at different object sizes. The gist of it is that with our current default HDD min_alloc size (64K), erasure coding can actually have worse space amplification than 3X replication, even with moderately large (128K) object sizes. How much this factors into the decision vs fragmentation is a tough call, but I wanted to at least showcase the behavior as we work through deciding what our default HDD behavior should be.
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP...
The key difference (at the bluestore level) between RGW and RBD/CephFS writes is that RGW passes down the CEPH_OSD_ALLOC_HINT_FLAG_IMMUTABLE | CEPH_OSD_ALLOC_HINT_FLAG_APPEND_ONLY hints. The immutable one in particular is what we really care about, since it's the mutable objects that get overwrites that lead to (most) fragmentation. We should use this to decide whether to create minimal (min_alloc_size) blobs or whether we should keep the blobs larger to limit fragmentation.
I'm not sure what we would call the config option that isn't super confusing... maybe bluestore_mutable_min_blob_size? bluestore_baseline_min_blob_size?
sage_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Mark, I can confirm that EC+HDD space amplification with the 64k min_alloc_size is quite real. On one of our first bluestore cluster we measured amplification at ~30% of the used space.This is also problematic for filestore to bluestore repaves on failures. Below are two OSDs with the same number of PGs, one on bluestore 64k, one on filestore: ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 137 hdd 7.26599 1.00000 7440G 4193G 3246G 56.36 1.14 126 218 hdd 7.26599 1.00000 7451G 6170G 1281G 82.81 1.68 126 Since then, we changed min_alloc_size to 16k and saw some good results. A 24h benchmark that wrote about 1TB of small objects a single OSD pool didn't show a significant performance difference between 64 and 16k. min_alloc_size 16k vs 64k on the same cluster (not the same as above): ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 221 hdd 7.27699 1.00000 7451G 3193G 4258G 42.85 0.78 108 579 hdd 7.27698 1.00000 7451G 4272G 3179G 57.33 1.05 108 On Thu, Nov 21, 2019 at 1:50 PM Sage Weil <sage@newdream.net> wrote:
On Thu, 21 Nov 2019, Mark Nelson wrote:
Hi Folks,
We're discussing changing the minimum allocation size in bluestore to 4k. For flash devices this appears to be a no-brainer. We've made the write path fast enough in bluestore that we're typically seeing either the same or faster performance with a 4K min_alloc size and the space savings for small objects easily outweigh the increase in metadata for large fragmented objects.
For HDDs there are tradeoffs. A smaller allocation size means more fragmentation when there are small overwrites (like in RBD) which can mean a lot more seeks. Igor was showing some fairly steep RBD performance drops for medium-large reads/writes once the OSDs started to become fragmented. For RGW this isn't nearly as big of a deal though since typically the objects shouldn't become fragmented. A small (4K) allocation size does mean however that we can write out 4K random writes sequentially and gain a big IOPS win which theoretically should benefit both RBD and RGW.
Regarding space-amplification, Josh pointed out that our current 64K allocation size has huge ramifications for overall space-amp when writing out medium sized objects to EC pools. In an attempt to actually quantify this, I made a spreadsheet with some graphs showing a couple of examples of how the min_alloc size and replication/EC interact with each other at different object sizes. The gist of it is that with our current default HDD min_alloc size (64K), erasure coding can actually have worse space amplification than 3X replication, even with moderately large (128K) object sizes. How much this factors into the decision vs fragmentation is a tough call, but I wanted to at least showcase the behavior as we work through deciding what our default HDD behavior should be.
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP...
The key difference (at the bluestore level) between RGW and RBD/CephFS writes is that RGW passes down the CEPH_OSD_ALLOC_HINT_FLAG_IMMUTABLE | CEPH_OSD_ALLOC_HINT_FLAG_APPEND_ONLY hints. The immutable one in particular is what we really care about, since it's the mutable objects that get overwrites that lead to (most) fragmentation. We should use this to decide whether to create minimal (min_alloc_size) blobs or whether we should keep the blobs larger to limit fragmentation.
I'm not sure what we would call the config option that isn't super confusing... maybe bluestore_mutable_min_blob_size? bluestore_baseline_min_blob_size?
sage_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 11/22/2019 12:50 AM, Sage Weil wrote:
Hi Folks,
We're discussing changing the minimum allocation size in bluestore to 4k. For flash devices this appears to be a no-brainer. We've made the write path fast enough in bluestore that we're typically seeing either the same or faster performance with a 4K min_alloc size and the space savings for small objects easily outweigh the increase in metadata for large fragmented objects.
For HDDs there are tradeoffs. A smaller allocation size means more fragmentation when there are small overwrites (like in RBD) which can mean a lot more seeks. Igor was showing some fairly steep RBD performance drops for medium-large reads/writes once the OSDs started to become fragmented. For RGW this isn't nearly as big of a deal though since typically the objects shouldn't become fragmented. A small (4K) allocation size does mean however that we can write out 4K random writes sequentially and gain a big IOPS win which theoretically should benefit both RBD and RGW.
Regarding space-amplification, Josh pointed out that our current 64K allocation size has huge ramifications for overall space-amp when writing out medium sized objects to EC pools. In an attempt to actually quantify this, I made a spreadsheet with some graphs showing a couple of examples of how the min_alloc size and replication/EC interact with each other at different object sizes. The gist of it is that with our current default HDD min_alloc size (64K), erasure coding can actually have worse space amplification than 3X replication, even with moderately large (128K) object sizes. How much this factors into the decision vs fragmentation is a tough call, but I wanted to at least showcase the behavior as we work through deciding what our default HDD behavior should be.
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP... The key difference (at the bluestore level) between RGW and RBD/CephFS writes is that RGW passes down the CEPH_OSD_ALLOC_HINT_FLAG_IMMUTABLE | CEPH_OSD_ALLOC_HINT_FLAG_APPEND_ONLY hints. The immutable one in
On Thu, 21 Nov 2019, Mark Nelson wrote: particular is what we really care about, since it's the mutable objects that get overwrites that lead to (most) fragmentation. We should use this to decide whether to create minimal (min_alloc_size) blobs or whether we should keep the blobs larger to limit fragmentation.
I'm not sure what we would call the config option that isn't super confusing... maybe bluestore_mutable_min_blob_size? bluestore_baseline_min_blob_size?
I'm thinking about introducing the following behavior for spinner-based stores (preliminary notes for now): 1) Unconditionally pin allocator's granularity to 4K (block size), i.e. untie it from min_alloc_size setting This will probably result in additional overhead in allocator when one needs to allocate contiguous 64K blobs (see 3) below). But it looks like new avl allocator is great in dealing with that. 2) Allocate space using 4K block size for small (<=32K or <=48K or <60K? ) objects tagged with IMMUTABLE flag 3) Use existing min_alloc_size defaults (64K) for all(?) other objects 4) Tag object with alloc size it's using. ------- The above is enough enough to deal with RGW space amplification for small objects which is the worst case for now IMO.---------------- Additional points to consider: 5) add per-object (more flexible and complex) hint or per-store setting to optimize onode keeping for size vs. for speed. May be even spit the latter into optimize for reads vs. optimize for overwrites? EC and/or other concerned entries to indicate "OPTIMIZE_FOR_SPACE". Unconditionally or using some logics? 6) apply appropriate alloc size depending on the strategy determined at 5) 7) Even more versatile approach would be vary alloc size at onode's blob level. Hence tag blob rather than object with applied alloc size (needs single bit in fact). May be useful to deal with space amplification for tail blobs or sparse objects. Thoughts? Thanks, Igor
sage
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Fri, Nov 22, 2019 at 6:02 AM Igor Fedotov <ifedotov@suse.de> wrote:
On 11/22/2019 12:50 AM, Sage Weil wrote:
On Thu, 21 Nov 2019, Mark Nelson wrote:
Hi Folks,
We're discussing changing the minimum allocation size in bluestore to 4k. For flash devices this appears to be a no-brainer. We've made the write path fast enough in bluestore that we're typically seeing either the same or faster performance with a 4K min_alloc size and the space savings for small objects easily outweigh the increase in metadata for large fragmented objects.
For HDDs there are tradeoffs. A smaller allocation size means more fragmentation when there are small overwrites (like in RBD) which can mean a lot more seeks. Igor was showing some fairly steep RBD performance drops for medium-large reads/writes once the OSDs started to become fragmented. For RGW this isn't nearly as big of a deal though since typically the objects shouldn't become fragmented. A small (4K) allocation size does mean however that we can write out 4K random writes sequentially and gain a big IOPS win which theoretically should benefit both RBD and RGW.
Regarding space-amplification, Josh pointed out that our current 64K allocation size has huge ramifications for overall space-amp when writing out medium sized objects to EC pools. In an attempt to actually quantify this, I made a spreadsheet with some graphs showing a couple of examples of how the min_alloc size and replication/EC interact with each other at different object sizes. The gist of it is that with our current default HDD min_alloc size (64K), erasure coding can actually have worse space amplification than 3X replication, even with moderately large (128K) object sizes. How much this factors into the decision vs fragmentation is a tough call, but I wanted to at least showcase the behavior as we work through deciding what our default HDD behavior should be.
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP...
The key difference (at the bluestore level) between RGW and RBD/CephFS writes is that RGW passes down the CEPH_OSD_ALLOC_HINT_FLAG_IMMUTABLE | CEPH_OSD_ALLOC_HINT_FLAG_APPEND_ONLY hints. The immutable one in particular is what we really care about, since it's the mutable objects that get overwrites that lead to (most) fragmentation. We should use this to decide whether to create minimal (min_alloc_size) blobs or whether we should keep the blobs larger to limit fragmentation.
I'm not sure what we would call the config option that isn't super confusing... maybe bluestore_mutable_min_blob_size? bluestore_baseline_min_blob_size?
I'm thinking about introducing the following behavior for spinner-based stores (preliminary notes for now):
1) Unconditionally pin allocator's granularity to 4K (block size), i.e. untie it from min_alloc_size setting
This will probably result in additional overhead in allocator when one needs to allocate contiguous 64K blobs (see 3) below).
But it looks like new avl allocator is great in dealing with that.
2) Allocate space using 4K block size for small (<=32K or <=48K or <60K? ) objects tagged with IMMUTABLE flag
3) Use existing min_alloc_size defaults (64K) for all(?) other objects
4) Tag object with alloc size it's using.
------- The above is enough enough to deal with RGW space amplification for small objects which is the worst case for now IMO.----------------
Additional points to consider:
5) add per-object (more flexible and complex) hint or per-store setting to optimize onode keeping for size vs. for speed. May be even spit the latter into optimize for reads vs. optimize for overwrites?
librados exposes allocation hints like expected object size and expected write size. Right now RBD sets these to the object size and applies the hint to each IO -- and it seems like it's currently a no-op but it is recorded in the onode. There are also lots of flags for sequential vs random reads, immutable, append-only, and random-write.
EC and/or other concerned entries to indicate "OPTIMIZE_FOR_SPACE". Unconditionally or using some logics?
6) apply appropriate alloc size depending on the strategy determined at 5)
7) Even more versatile approach would be vary alloc size at onode's blob level. Hence tag blob rather than object with applied alloc size (needs single bit in fact).
May be useful to deal with space amplification for tail blobs or sparse objects.
Thoughts?
Thanks,
Igor
sage
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason
On 11/22/2019 2:01 PM, Igor Fedotov wrote:
On 11/22/2019 12:50 AM, Sage Weil wrote:
The key difference (at the bluestore level) between RGW and RBD/CephFS writes is that RGW passes down the CEPH_OSD_ALLOC_HINT_FLAG_IMMUTABLE | CEPH_OSD_ALLOC_HINT_FLAG_APPEND_ONLY hints. The immutable one in particular is what we really care about, since it's the mutable objects that get overwrites that lead to (most) fragmentation. We should use this to decide whether to create minimal (min_alloc_size) blobs or whether we should keep the blobs larger to limit fragmentation.
I'm not sure what we would call the config option that isn't super confusing... maybe bluestore_mutable_min_blob_size? bluestore_baseline_min_blob_size?
I'm thinking about introducing the following behavior for spinner-based stores (preliminary notes for now):
1) Unconditionally pin allocator's granularity to 4K (block size), i.e. untie it from min_alloc_size setting
This will probably result in additional overhead in allocator when one needs to allocate contiguous 64K blobs (see 3) below).
But it looks like new avl allocator is great in dealing with that.
2) Allocate space using 4K block size for small (<=32K or <=48K or <60K? ) objects tagged with IMMUTABLE flag
3) Use existing min_alloc_size defaults (64K) for all(?) other objects
4) Tag object with alloc size it's using.
------- The above is enough enough to deal with RGW space amplification for small objects which is the worst case for now IMO.----------------
Additional points to consider:
5) add per-object (more flexible and complex) hint or per-store setting to optimize onode keeping for size vs. for speed. May be even spit the latter into optimize for reads vs. optimize for overwrites?
One can read 'per-pool' instead of 'per-store' above.
EC and/or other concerned entries to indicate "OPTIMIZE_FOR_SPACE". Unconditionally or using some logics?
6) apply appropriate alloc size depending on the strategy determined at 5)
7) Even more versatile approach would be vary alloc size at onode's blob level. Hence tag blob rather than object with applied alloc size (needs single bit in fact).
May be useful to deal with space amplification for tail blobs or sparse objects.
Thoughts?
Thanks,
Igor
sage
_______________________________________________ Dev mailing list --dev@ceph.io To unsubscribe send an email todev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Question: lower alloc sizes seem like a work-around. Would it be better to have an option to store small objects entirely in rocksdb? I've seen a lot of setups that would really benefit from such a feature. It's usually a case of mixed object sizes with a lot of very small (< 10kb) objects and some large objects that unfortunately can't be mapped to different directories/buckets/pools on a higher layer. (I think I did see some pull request that did something like that some time ago?) Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Nov 22, 2019 at 3:08 PM Igor Fedotov <ifedotov@suse.de> wrote:
On 11/22/2019 2:01 PM, Igor Fedotov wrote:
On 11/22/2019 12:50 AM, Sage Weil wrote:
The key difference (at the bluestore level) between RGW and RBD/CephFS
writes is that RGW passes down the CEPH_OSD_ALLOC_HINT_FLAG_IMMUTABLE | CEPH_OSD_ALLOC_HINT_FLAG_APPEND_ONLY hints. The immutable one in particular is what we really care about, since it's the mutable objects that get overwrites that lead to (most) fragmentation. We should use this to decide whether to create minimal (min_alloc_size) blobs or whether we should keep the blobs larger to limit fragmentation.
I'm not sure what we would call the config option that isn't super confusing... maybe bluestore_mutable_min_blob_size? bluestore_baseline_min_blob_size?
I'm thinking about introducing the following behavior for spinner-based stores (preliminary notes for now):
1) Unconditionally pin allocator's granularity to 4K (block size), i.e. untie it from min_alloc_size setting
This will probably result in additional overhead in allocator when one needs to allocate contiguous 64K blobs (see 3) below).
But it looks like new avl allocator is great in dealing with that.
2) Allocate space using 4K block size for small (<=32K or <=48K or <60K? ) objects tagged with IMMUTABLE flag
3) Use existing min_alloc_size defaults (64K) for all(?) other objects
4) Tag object with alloc size it's using.
------- The above is enough enough to deal with RGW space amplification for small objects which is the worst case for now IMO.----------------
Additional points to consider:
5) add per-object (more flexible and complex) hint or per-store setting to optimize onode keeping for size vs. for speed. May be even spit the latter into optimize for reads vs. optimize for overwrites?
One can read 'per-pool' instead of 'per-store' above.
EC and/or other concerned entries to indicate "OPTIMIZE_FOR_SPACE". Unconditionally or using some logics?
6) apply appropriate alloc size depending on the strategy determined at 5)
7) Even more versatile approach would be vary alloc size at onode's blob level. Hence tag blob rather than object with applied alloc size (needs single bit in fact).
May be useful to deal with space amplification for tail blobs or sparse objects.
Thoughts?
Thanks,
Igor
sage
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 11/23/19 9:51 AM, Paul Emmerich wrote:
Question: lower alloc sizes seem like a work-around. Would it be better to have an option to store small objects entirely in rocksdb?
I've seen a lot of setups that would really benefit from such a feature. It's usually a case of mixed object sizes with a lot of very small (< 10kb) objects and some large objects that unfortunately can't be mapped to different directories/buckets/pools on a higher layer.
(I think I did see some pull request that did something like that some time ago?)
Paul
There are a couple of PRs that attempt this, including one that Igor wrote a while back. I think there is both potential for reward and risk in this approach. We already are facing some issues with RocksDB not scaling well as the amount of data grows. Potentially with >4K objects this could start adding up very quickly. I think the most recent ideas here have a lot of merit though. If we can allow for a smaller allocation size when the objects have the immutable hint that would give us a win when we are most likely to want/need it. Perhaps also putting the data in the onode for <4K objects might also be worth exploring, but personally I think we'll need to do a lot of testing before we roll that one out into production. For RGW if we could get the allocation size down to 4K that would by itself be a huge win. Mark
participants (7)
-
Alexandre Marangone
-
Igor Fedotov
-
Jason Dillaman
-
Kyle Bader
-
Mark Nelson
-
Paul Emmerich
-
Sage Weil