bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type>
Hi, With Pacific and afterwards, which of the following two settings takes precedence? bluestore_compression_min_blob_size (default 0) bluestore_compression_min_blob_size_<type> (default 8192) I'd assume the more specific setting takes precedence? What's this "blob"? Is it write block? Then, there is no compression for 4K write block? I was not able to find it out from doc, hope someone could help to clarify. Thanks! Tony
Hi, First priority for compression blob is per-pool setting (taken if set) # ceph osd pool get test compression_min_blob_size Second, if set, bluestore_compression_min_blob_size config, Third is selected from *_hdd *_ssd depending on the main block device in the OSD being HDD or SSD. This can be forced by setting bluestore_debug_enforce_settings to either 'hdd' or 'ssd' if your hardware improperly reports # cat /sys/block/sda/queue/rotational 0 Blob is an organization unit in BlueStore https://github.com/ceph/ceph/pull/51933 You can get compression for blocks smaller than 4K only if your main device has native block size smaller. # cat /sys/block/nvme1n1/queue/physical_block_size 512 In such a case you can set bluestore_min_alloc_size (in effect only during mkfs) to some value smaller than 4K, but multiply of 512. This path is not often tested, and you need to own unexpected side effects. Best regards, Adam Kupczyk On Wed, 16 Aug 2023 at 19:46, Tony Liu <tonyliu0592@hotmail.com> wrote:
Hi,
With Pacific and afterwards, which of the following two settings takes precedence? bluestore_compression_min_blob_size (default 0) bluestore_compression_min_blob_size_<type> (default 8192)
I'd assume the more specific setting takes precedence? What's this "blob"? Is it write block? Then, there is no compression for 4K write block?
I was not able to find it out from doc, hope someone could help to clarify.
Thanks! Tony _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Adam, This priority order is different from common sense, where specific setting has higher priority than generic setting. With this order, compression_min_blob_size has higher priority, then what's the point of having _hdd or _ssd settings, which will never get applied? Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com> Sent: August 17, 2023 11:00 PM To: Tony Liu Cc: dev@ceph.io Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type> Hi, First priority for compression blob is per-pool setting (taken if set) # ceph osd pool get test compression_min_blob_size Second, if set, bluestore_compression_min_blob_size config, Third is selected from *_hdd *_ssd depending on the main block device in the OSD being HDD or SSD. This can be forced by setting bluestore_debug_enforce_settings to either 'hdd' or 'ssd' if your hardware improperly reports # cat /sys/block/sda/queue/rotational 0 Blob is an organization unit in BlueStore https://github.com/ceph/ceph/pull/51933 You can get compression for blocks smaller than 4K only if your main device has native block size smaller. # cat /sys/block/nvme1n1/queue/physical_block_size 512 In such a case you can set bluestore_min_alloc_size (in effect only during mkfs) to some value smaller than 4K, but multiply of 512. This path is not often tested, and you need to own unexpected side effects. Best regards, Adam Kupczyk On Wed, 16 Aug 2023 at 19:46, Tony Liu <tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com>> wrote: Hi, With Pacific and afterwards, which of the following two settings takes precedence? bluestore_compression_min_blob_size (default 0) bluestore_compression_min_blob_size_<type> (default 8192) I'd assume the more specific setting takes precedence? What's this "blob"? Is it write block? Then, there is no compression for 4K write block? I was not able to find it out from doc, hope someone could help to clarify. Thanks! Tony _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Indeed, I would have thought that the unqualified setting would only ever be derived from _hdd / _ssd. I believe we have other sets of options that behave that way.
On Aug 18, 2023, at 13:54, Tony Liu <tonyliu0592@hotmail.com> wrote:
Hi Adam,
This priority order is different from common sense, where specific setting has higher priority than generic setting. With this order, compression_min_blob_size has higher priority, then what's the point of having _hdd or _ssd settings, which will never get applied?
Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com> Sent: August 17, 2023 11:00 PM To: Tony Liu Cc: dev@ceph.io Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type>
Hi,
First priority for compression blob is per-pool setting (taken if set) # ceph osd pool get test compression_min_blob_size Second, if set, bluestore_compression_min_blob_size config, Third is selected from *_hdd *_ssd depending on the main block device in the OSD being HDD or SSD. This can be forced by setting bluestore_debug_enforce_settings to either 'hdd' or 'ssd' if your hardware improperly reports # cat /sys/block/sda/queue/rotational 0
Blob is an organization unit in BlueStore https://github.com/ceph/ceph/pull/51933
You can get compression for blocks smaller than 4K only if your main device has native block size smaller. # cat /sys/block/nvme1n1/queue/physical_block_size 512 In such a case you can set bluestore_min_alloc_size (in effect only during mkfs) to some value smaller than 4K, but multiply of 512.
This path is not often tested, and you need to own unexpected side effects.
Best regards, Adam Kupczyk
On Wed, 16 Aug 2023 at 19:46, Tony Liu <tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com>> wrote: Hi,
With Pacific and afterwards, which of the following two settings takes precedence? bluestore_compression_min_blob_size (default 0) bluestore_compression_min_blob_size_<type> (default 8192)
I'd assume the more specific setting takes precedence? What's this "blob"? Is it write block? Then, there is no compression for 4K write block?
I was not able to find it out from doc, hope someone could help to clarify.
Thanks! Tony _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Tony, Initially: 1) values for compression min and max blob size are taken from compression_(min/max)_blob_size_(ssd/hdd) but if value of compression_(min_max)_blob_size is set to non-0 2) that value is used as compression blob size in bluestore but if a pool has set compression_(min/max)_blob_size setting 3) that value is used as compression blob size I do not understand why you think of it as inverted priority. The most generic case ceph tries to guess the best blob size by type of media hw. Once you know your hw you can override blob sizes using your knowledge. And even further - you can have different expectations for blob sizes depending on type of data you have. For a pool with large unchanging objects (RGW?) you could choose a very large blob size. For a pool where you expect small accesses (RBD?) you probably want a smaller blob. Best regards, Adam Kupczyk On Fri, 18 Aug 2023 at 19:54, Tony Liu <tonyliu0592@hotmail.com> wrote:
Hi Adam,
This priority order is different from common sense, where specific setting has higher priority than generic setting. With this order, compression_min_blob_size has higher priority, then what's the point of having _hdd or _ssd settings, which will never get applied?
Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com> Sent: August 17, 2023 11:00 PM To: Tony Liu Cc: dev@ceph.io Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type>
Hi,
First priority for compression blob is per-pool setting (taken if set) # ceph osd pool get test compression_min_blob_size Second, if set, bluestore_compression_min_blob_size config, Third is selected from *_hdd *_ssd depending on the main block device in the OSD being HDD or SSD. This can be forced by setting bluestore_debug_enforce_settings to either 'hdd' or 'ssd' if your hardware improperly reports # cat /sys/block/sda/queue/rotational 0
Blob is an organization unit in BlueStore https://github.com/ceph/ceph/pull/51933
You can get compression for blocks smaller than 4K only if your main device has native block size smaller. # cat /sys/block/nvme1n1/queue/physical_block_size 512 In such a case you can set bluestore_min_alloc_size (in effect only during mkfs) to some value smaller than 4K, but multiply of 512.
This path is not often tested, and you need to own unexpected side effects.
Best regards, Adam Kupczyk
On Wed, 16 Aug 2023 at 19:46, Tony Liu <tonyliu0592@hotmail.com<mailto: tonyliu0592@hotmail.com>> wrote: Hi,
With Pacific and afterwards, which of the following two settings takes precedence? bluestore_compression_min_blob_size (default 0) bluestore_compression_min_blob_size_<type> (default 8192)
I'd assume the more specific setting takes precedence? What's this "blob"? Is it write block? Then, there is no compression for 4K write block?
I was not able to find it out from doc, hope someone could help to clarify.
Thanks! Tony _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io
Hi Adam, I've been focus on global compression, not per pool. In case of global compression, only compression_(min/max)_blob_size_(ssd/hdd) takes effect, compression_(min_max)_blob_size has no effect to global compression at all. Is that correct? If yes, it makes sense, but I'd name it compression_(min_max)_blob_size_per_pool or something more clear. Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com> Sent: August 29, 2023 01:04 AM To: Tony Liu Cc: dev@ceph.io Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type> Hi Tony, Initially: 1) values for compression min and max blob size are taken from compression_(min/max)_blob_size_(ssd/hdd) but if value of compression_(min_max)_blob_size is set to non-0 2) that value is used as compression blob size in bluestore but if a pool has set compression_(min/max)_blob_size setting 3) that value is used as compression blob size I do not understand why you think of it as inverted priority. The most generic case ceph tries to guess the best blob size by type of media hw. Once you know your hw you can override blob sizes using your knowledge. And even further - you can have different expectations for blob sizes depending on type of data you have. For a pool with large unchanging objects (RGW?) you could choose a very large blob size. For a pool where you expect small accesses (RBD?) you probably want a smaller blob. Best regards, Adam Kupczyk On Fri, 18 Aug 2023 at 19:54, Tony Liu <tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com>> wrote: Hi Adam, This priority order is different from common sense, where specific setting has higher priority than generic setting. With this order, compression_min_blob_size has higher priority, then what's the point of having _hdd or _ssd settings, which will never get applied? Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com<mailto:akupczyk@redhat.com>> Sent: August 17, 2023 11:00 PM To: Tony Liu Cc: dev@ceph.io<mailto:dev@ceph.io> Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type> Hi, First priority for compression blob is per-pool setting (taken if set) # ceph osd pool get test compression_min_blob_size Second, if set, bluestore_compression_min_blob_size config, Third is selected from *_hdd *_ssd depending on the main block device in the OSD being HDD or SSD. This can be forced by setting bluestore_debug_enforce_settings to either 'hdd' or 'ssd' if your hardware improperly reports # cat /sys/block/sda/queue/rotational 0 Blob is an organization unit in BlueStore https://github.com/ceph/ceph/pull/51933 You can get compression for blocks smaller than 4K only if your main device has native block size smaller. # cat /sys/block/nvme1n1/queue/physical_block_size 512 In such a case you can set bluestore_min_alloc_size (in effect only during mkfs) to some value smaller than 4K, but multiply of 512. This path is not often tested, and you need to own unexpected side effects. Best regards, Adam Kupczyk On Wed, 16 Aug 2023 at 19:46, Tony Liu <tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com><mailto:tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com>>> wrote: Hi, With Pacific and afterwards, which of the following two settings takes precedence? bluestore_compression_min_blob_size (default 0) bluestore_compression_min_blob_size_<type> (default 8192) I'd assume the more specific setting takes precedence? What's this "blob"? Is it write block? Then, there is no compression for 4K write block? I was not able to find it out from doc, hope someone could help to clarify. Thanks! Tony _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io><mailto:dev@ceph.io<mailto:dev@ceph.io>> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io><mailto:dev-leave@ceph.io<mailto:dev-leave@ceph.io>>
Hi Tony, Originally, there existed only compression_(min_max)_blob_size settings. Later we realized that ssd and hdd fare better with distinct values. In result compression_(min/max)_blob_size_(ssd/hdd) settings to populate compression_(min/max)_blob_size were created. Best regards, Adam On Tue, 29 Aug 2023 at 18:28, Tony Liu <tonyliu0592@hotmail.com> wrote:
Hi Adam,
I've been focus on global compression, not per pool.
In case of global compression, only compression_(min/max)_blob_size_(ssd/hdd) takes effect, compression_(min_max)_blob_size has no effect to global compression at all. Is that correct?
If yes, it makes sense, but I'd name it compression_(min_max)_blob_size_per_pool or something more clear.
Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com> Sent: August 29, 2023 01:04 AM To: Tony Liu Cc: dev@ceph.io Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type>
Hi Tony,
Initially: 1) values for compression min and max blob size are taken from compression_(min/max)_blob_size_(ssd/hdd) but if value of compression_(min_max)_blob_size is set to non-0 2) that value is used as compression blob size in bluestore but if a pool has set compression_(min/max)_blob_size setting 3) that value is used as compression blob size
I do not understand why you think of it as inverted priority. The most generic case ceph tries to guess the best blob size by type of media hw. Once you know your hw you can override blob sizes using your knowledge. And even further - you can have different expectations for blob sizes depending on type of data you have. For a pool with large unchanging objects (RGW?) you could choose a very large blob size. For a pool where you expect small accesses (RBD?) you probably want a smaller blob.
Best regards, Adam Kupczyk
On Fri, 18 Aug 2023 at 19:54, Tony Liu <tonyliu0592@hotmail.com<mailto: tonyliu0592@hotmail.com>> wrote: Hi Adam,
This priority order is different from common sense, where specific setting has higher priority than generic setting. With this order, compression_min_blob_size has higher priority, then what's the point of having _hdd or _ssd settings, which will never get applied?
Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com<mailto:akupczyk@redhat.com>> Sent: August 17, 2023 11:00 PM To: Tony Liu Cc: dev@ceph.io<mailto:dev@ceph.io> Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type>
Hi,
First priority for compression blob is per-pool setting (taken if set) # ceph osd pool get test compression_min_blob_size Second, if set, bluestore_compression_min_blob_size config, Third is selected from *_hdd *_ssd depending on the main block device in the OSD being HDD or SSD. This can be forced by setting bluestore_debug_enforce_settings to either 'hdd' or 'ssd' if your hardware improperly reports # cat /sys/block/sda/queue/rotational 0
Blob is an organization unit in BlueStore https://github.com/ceph/ceph/pull/51933
You can get compression for blocks smaller than 4K only if your main device has native block size smaller. # cat /sys/block/nvme1n1/queue/physical_block_size 512 In such a case you can set bluestore_min_alloc_size (in effect only during mkfs) to some value smaller than 4K, but multiply of 512.
This path is not often tested, and you need to own unexpected side effects.
Best regards, Adam Kupczyk
On Wed, 16 Aug 2023 at 19:46, Tony Liu <tonyliu0592@hotmail.com<mailto: tonyliu0592@hotmail.com><mailto:tonyliu0592@hotmail.com<mailto: tonyliu0592@hotmail.com>>> wrote: Hi,
With Pacific and afterwards, which of the following two settings takes precedence? bluestore_compression_min_blob_size (default 0) bluestore_compression_min_blob_size_<type> (default 8192)
I'd assume the more specific setting takes precedence? What's this "blob"? Is it write block? Then, there is no compression for 4K write block?
I was not able to find it out from doc, hope someone could help to clarify.
Thanks! Tony _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io><mailto:dev@ceph.io <mailto:dev@ceph.io>> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io
<mailto:dev-leave@ceph.io<mailto:dev-leave@ceph.io>>
Hi Adam, With the following configurations, compression_min_blob_size 4096 compression_min_blob_size_ssd 8192 which is taken by Ceph for global compression on SSD device? Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com> Sent: August 29, 2023 09:56 AM To: Tony Liu Cc: dev@ceph.io Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type> Hi Tony, Originally, there existed only compression_(min_max)_blob_size settings. Later we realized that ssd and hdd fare better with distinct values. In result compression_(min/max)_blob_size_(ssd/hdd) settings to populate compression_(min/max)_blob_size were created. Best regards, Adam On Tue, 29 Aug 2023 at 18:28, Tony Liu <tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com>> wrote: Hi Adam, I've been focus on global compression, not per pool. In case of global compression, only compression_(min/max)_blob_size_(ssd/hdd) takes effect, compression_(min_max)_blob_size has no effect to global compression at all. Is that correct? If yes, it makes sense, but I'd name it compression_(min_max)_blob_size_per_pool or something more clear. Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com<mailto:akupczyk@redhat.com>> Sent: August 29, 2023 01:04 AM To: Tony Liu Cc: dev@ceph.io<mailto:dev@ceph.io> Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type> Hi Tony, Initially: 1) values for compression min and max blob size are taken from compression_(min/max)_blob_size_(ssd/hdd) but if value of compression_(min_max)_blob_size is set to non-0 2) that value is used as compression blob size in bluestore but if a pool has set compression_(min/max)_blob_size setting 3) that value is used as compression blob size I do not understand why you think of it as inverted priority. The most generic case ceph tries to guess the best blob size by type of media hw. Once you know your hw you can override blob sizes using your knowledge. And even further - you can have different expectations for blob sizes depending on type of data you have. For a pool with large unchanging objects (RGW?) you could choose a very large blob size. For a pool where you expect small accesses (RBD?) you probably want a smaller blob. Best regards, Adam Kupczyk On Fri, 18 Aug 2023 at 19:54, Tony Liu <tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com><mailto:tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com>>> wrote: Hi Adam, This priority order is different from common sense, where specific setting has higher priority than generic setting. With this order, compression_min_blob_size has higher priority, then what's the point of having _hdd or _ssd settings, which will never get applied? Thanks! Tony ________________________________________ From: Adam Kupczyk <akupczyk@redhat.com<mailto:akupczyk@redhat.com><mailto:akupczyk@redhat.com<mailto:akupczyk@redhat.com>>> Sent: August 17, 2023 11:00 PM To: Tony Liu Cc: dev@ceph.io<mailto:dev@ceph.io><mailto:dev@ceph.io<mailto:dev@ceph.io>> Subject: Re: bluestore_compression_min_blob_size vs. bluestore_compression_min_blob_size_<type> Hi, First priority for compression blob is per-pool setting (taken if set) # ceph osd pool get test compression_min_blob_size Second, if set, bluestore_compression_min_blob_size config, Third is selected from *_hdd *_ssd depending on the main block device in the OSD being HDD or SSD. This can be forced by setting bluestore_debug_enforce_settings to either 'hdd' or 'ssd' if your hardware improperly reports # cat /sys/block/sda/queue/rotational 0 Blob is an organization unit in BlueStore https://github.com/ceph/ceph/pull/51933 You can get compression for blocks smaller than 4K only if your main device has native block size smaller. # cat /sys/block/nvme1n1/queue/physical_block_size 512 In such a case you can set bluestore_min_alloc_size (in effect only during mkfs) to some value smaller than 4K, but multiply of 512. This path is not often tested, and you need to own unexpected side effects. Best regards, Adam Kupczyk On Wed, 16 Aug 2023 at 19:46, Tony Liu <tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com><mailto:tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com>><mailto:tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com><mailto:tonyliu0592@hotmail.com<mailto:tonyliu0592@hotmail.com>>>> wrote: Hi, With Pacific and afterwards, which of the following two settings takes precedence? bluestore_compression_min_blob_size (default 0) bluestore_compression_min_blob_size_<type> (default 8192) I'd assume the more specific setting takes precedence? What's this "blob"? Is it write block? Then, there is no compression for 4K write block? I was not able to find it out from doc, hope someone could help to clarify. Thanks! Tony _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io><mailto:dev@ceph.io<mailto:dev@ceph.io>><mailto:dev@ceph.io<mailto:dev@ceph.io><mailto:dev@ceph.io<mailto:dev@ceph.io>>> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io><mailto:dev-leave@ceph.io<mailto:dev-leave@ceph.io>><mailto:dev-leave@ceph.io<mailto:dev-leave@ceph.io><mailto:dev-leave@ceph.io<mailto:dev-leave@ceph.io>>>
participants (3)
-
Adam Kupczyk
-
Anthony D'Atri
-
Tony Liu