12 Feb
2020
12 Feb
'20
1:50 p.m.
Hi Simon and Janne,
Thanks for the reply.
It seems indeed related to the bluestore_min_alloc_size.
In an old thread I've also found the following:
*S3 object saving pipeline:*
*- S3 object is divided into multipart shards by client.*
*- Rgw shards each multipart shard into rados objects of size*
*rgw_obj_stripe_size.*
*- Primary osd stripes rados object into ec stripes of width ==*
*ec.k*profile.stripe_unit, ec code them and send units into secondary*
*osds and write into object store (bluestore).*
*- Each subobject of rados object has size == (rados object size)/k.*
*- Then while writing into disk bluestore can divide rados subobject into*
*extents of minimal size == bluestore_min_alloc_size_hdd.*
*Next rules can save some space and iops:*
*- rgw_multipart_min_part_size SHOULD be multiple of rgw_obj_stripe_size*
*(client can use different value greater than)*
*- MUST rgw_obj_stripe_size == rgw_max_chunk_size*
*- ec stripe == osd_pool_erasure_code_stripe_unit or profile.stripe_unit*
*- rgw_obj_stripe_size SHOULD be multiple of profile.stripe_unit*ec.k*
*- bluestore_min_alloc_size_hdd MAY be equal to bluefs_alloc_size (to*
*avoid fragmentation)*
*- rgw_obj_stripe_size/ec.k SHOULD be multiple of*
*bluestore_min_alloc_size_hdd*
*- bluestore_min_alloc_size_hdd MAY be multiple of profile.stripe_unit*
Doing this calculation would result in the fact that smaller files of 135KB
end up in chunks of +/- 22KB. Writing this in 64KB gives me quite some
wasted space.
As far as I found, the allocation setting is kept on OSD level and set
during creation. Adapting the setting requires each OSD to be recreated. As
we have around 150 OSDs, I know what to script :-)
We will perform some testing in our test environment and I'll try to post
our feedback as long as I don't forget it...
To be sure, we just want to check the size on disk of the object. Afaik,
we'll need to export the rocks db and launch some queries on that, unless
someone else can help me on this one? Before Bluestore this was quite easy
to do...
Regards,