Hi Mark, I can confirm that EC+HDD space amplification with the 64k min_alloc_size is quite real. On one of our first bluestore cluster we measured amplification at ~30% of the used space.This is also problematic for filestore to bluestore repaves on failures. Below are two OSDs with the same number of PGs, one on bluestore 64k, one on filestore: ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 137 hdd 7.26599 1.00000 7440G 4193G 3246G 56.36 1.14 126 218 hdd 7.26599 1.00000 7451G 6170G 1281G 82.81 1.68 126 Since then, we changed min_alloc_size to 16k and saw some good results. A 24h benchmark that wrote about 1TB of small objects a single OSD pool didn't show a significant performance difference between 64 and 16k. min_alloc_size 16k vs 64k on the same cluster (not the same as above): ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 221 hdd 7.27699 1.00000 7451G 3193G 4258G 42.85 0.78 108 579 hdd 7.27698 1.00000 7451G 4272G 3179G 57.33 1.05 108 On Thu, Nov 21, 2019 at 1:50 PM Sage Weil <sage@newdream.net> wrote:
On Thu, 21 Nov 2019, Mark Nelson wrote:
Hi Folks,
We're discussing changing the minimum allocation size in bluestore to 4k. For flash devices this appears to be a no-brainer. We've made the write path fast enough in bluestore that we're typically seeing either the same or faster performance with a 4K min_alloc size and the space savings for small objects easily outweigh the increase in metadata for large fragmented objects.
For HDDs there are tradeoffs. A smaller allocation size means more fragmentation when there are small overwrites (like in RBD) which can mean a lot more seeks. Igor was showing some fairly steep RBD performance drops for medium-large reads/writes once the OSDs started to become fragmented. For RGW this isn't nearly as big of a deal though since typically the objects shouldn't become fragmented. A small (4K) allocation size does mean however that we can write out 4K random writes sequentially and gain a big IOPS win which theoretically should benefit both RBD and RGW.
Regarding space-amplification, Josh pointed out that our current 64K allocation size has huge ramifications for overall space-amp when writing out medium sized objects to EC pools. In an attempt to actually quantify this, I made a spreadsheet with some graphs showing a couple of examples of how the min_alloc size and replication/EC interact with each other at different object sizes. The gist of it is that with our current default HDD min_alloc size (64K), erasure coding can actually have worse space amplification than 3X replication, even with moderately large (128K) object sizes. How much this factors into the decision vs fragmentation is a tough call, but I wanted to at least showcase the behavior as we work through deciding what our default HDD behavior should be.
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP...
The key difference (at the bluestore level) between RGW and RBD/CephFS writes is that RGW passes down the CEPH_OSD_ALLOC_HINT_FLAG_IMMUTABLE | CEPH_OSD_ALLOC_HINT_FLAG_APPEND_ONLY hints. The immutable one in particular is what we really care about, since it's the mutable objects that get overwrites that lead to (most) fragmentation. We should use this to decide whether to create minimal (min_alloc_size) blobs or whether we should keep the blobs larger to limit fragmentation.
I'm not sure what we would call the config option that isn't super confusing... maybe bluestore_mutable_min_blob_size? bluestore_baseline_min_blob_size?
sage_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io