Bluestore compression - Which algo to choose? Zstd really still that bad?
Hey ceph-users, we've been using the default "snappy" to have Ceph compress data on certain pools - namely backups / copies of volumes of a VM environment. So it's write once, and no random access. I am now wondering if switching to another algo (there is snappy, zlib, lz4, or zstd) would improve the compression ratio (significantly)? * Does anybody have any real world data on snappy vs. $anyother? Using zstd is tempting as it's used in various other applications (btrfs, MongoDB, ...) for inline-compression with great success. For Ceph though there is a warning ([1]), about it being not recommended in the docs still. But I am wondering if this still stands with e.g. [2] merged. And there was [3] trying to improve the performance, this this reads as it only lead to a dead-end and no code changes? In any case does anybody have any numbers to help with the decision on the compression algo? Regards Christian [1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#co... [2] https://github.com/ceph/ceph/pull/33790 [3] https://github.com/facebook/zstd/issues/910
Hi Christian, I can't say anything about your primary question on zstd benefits/drawbacks but I'd like to emphasize that compression ratio at BlueStore is (to a major degree) determined by the input data flow characteristics (primarily write block size), object store allocation unit size (bluestore_min_alloc_size) and some parameters (e.g. maximum blob size) that determine how input data chunks are logically split when landing on disk. E.g. if one has min_alloc_size set to 4K and write block size is in (4K-8K] then resulting compressed block would never be less than 4K. Hence compression ratio is never more than 2. Similarly if min_alloc_size is 64K there would be no benefit in compression at all for the above input since target allocation units are always larger than input blocks. The rationale of the above behavior is that compression is applied exclusively on input blocks - there is no additional processing to merge input and existing data and compress them all together. Thanks, Igor On 26/06/2023 11:48, Christian Rohmann wrote:
Hey ceph-users,
we've been using the default "snappy" to have Ceph compress data on certain pools - namely backups / copies of volumes of a VM environment. So it's write once, and no random access. I am now wondering if switching to another algo (there is snappy, zlib, lz4, or zstd) would improve the compression ratio (significantly)?
* Does anybody have any real world data on snappy vs. $anyother?
Using zstd is tempting as it's used in various other applications (btrfs, MongoDB, ...) for inline-compression with great success. For Ceph though there is a warning ([1]), about it being not recommended in the docs still. But I am wondering if this still stands with e.g. [2] merged. And there was [3] trying to improve the performance, this this reads as it only lead to a dead-end and no code changes?
In any case does anybody have any numbers to help with the decision on the compression algo?
Regards
Christian
[1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#co... [2] https://github.com/ceph/ceph/pull/33790 [3] https://github.com/facebook/zstd/issues/910 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Igor, On 27/06/2023 12:06, Igor Fedotov wrote:
I can't say anything about your primary question on zstd benefits/drawbacks but I'd like to emphasize that compression ratio at BlueStore is (to a major degree) determined by the input data flow characteristics (primarily write block size), object store allocation unit size (bluestore_min_alloc_size) and some parameters (e.g. maximum blob size) that determine how input data chunks are logically split when landing on disk. E.g. if one has min_alloc_size set to 4K and write block size is in (4K-8K] then resulting compressed block would never be less than 4K. Hence compression ratio is never more than 2. Similarly if min_alloc_size is 64K there would be no benefit in compression at all for the above input since target allocation units are always larger than input blocks. The rationale of the above behavior is that compression is applied exclusively on input blocks - there is no additional processing to merge input and existing data and compress them all together.
Thanks for the emphasis on input data and its block-size. Yes, that is certainly the most important factor for the compression efficiency and choice of an suitable algorithm for a certain use-case. In my case the pool is RBD only, so (by default) the blocks are 4M if I am not mistaken. I also understand that even though larger blocks generally compress better, I know there is no relation between different blocks in regard to compression dictionaries (going along the lines of de-duplication). In the end in my use-case it boils down to the type of data stored on the RBD images and how compressible that might be. But since those blocks are only written once, and I am ready to invest more CPU cycles to reduce the size on disk. I am simply looking for data other might have collected on their similar use-cases. Also I am still wondering if there really is nobody that worked/played more with zstd since that has become so popular in recent months... Regards Christian
I dont have the most experience with ceph as my use case is homelab and only a few months in. I enabled compression on my vm(proxmox hosts) disk rdb pool using mode = aggressive, algorithm = lz4, all other compression settings default. After copying all of the vm disks to another storage and then back to ceph pool I saw a near %50 reduction in space needed for vm disks. I have not had a chance to benchmark the vm disks with compression yet as I am waiting for the cluster to calm down from some other disks moves. On Tue, Jun 27, 2023 at 7:01 AM Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey Igor,
On 27/06/2023 12:06, Igor Fedotov wrote:
I can't say anything about your primary question on zstd benefits/drawbacks but I'd like to emphasize that compression ratio at BlueStore is (to a major degree) determined by the input data flow characteristics (primarily write block size), object store allocation unit size (bluestore_min_alloc_size) and some parameters (e.g. maximum blob size) that determine how input data chunks are logically split when landing on disk. E.g. if one has min_alloc_size set to 4K and write block size is in (4K-8K] then resulting compressed block would never be less than 4K. Hence compression ratio is never more than 2. Similarly if min_alloc_size is 64K there would be no benefit in compression at all for the above input since target allocation units are always larger than input blocks. The rationale of the above behavior is that compression is applied exclusively on input blocks - there is no additional processing to merge input and existing data and compress them all together.
Thanks for the emphasis on input data and its block-size. Yes, that is certainly the most important factor for the compression efficiency and choice of an suitable algorithm for a certain use-case. In my case the pool is RBD only, so (by default) the blocks are 4M if I am not mistaken. I also understand that even though larger blocks generally compress better, I know there is no relation between different blocks in regard to compression dictionaries (going along the lines of de-duplication). In the end in my use-case it boils down to the type of data stored on the RBD images and how compressible that might be. But since those blocks are only written once, and I am ready to invest more CPU cycles to reduce the size on disk.
I am simply looking for data other might have collected on their similar use-cases. Also I am still wondering if there really is nobody that worked/played more with zstd since that has become so popular in recent months...
Regards
Christian
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Zach Underwood (RHCE,RHCSA,RHCT,UACA) My website <http://zachunderwood.me> advance-networking.com
participants (3)
-
Christian Rohmann
-
Igor Fedotov
-
Zach Underwood