Re: has anyone enabled bdev_enable_discard?
Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option?
Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option?
I played with this feature a while ago and recall it had visible negative impact on user operations due to the need to submit tons of discard operations - effectively each data overwrite operation triggers one or more discard operation submission to disk. And I doubt this has been widely used if any. Nevertheless recently we've got a PR to rework some aspects of thread management for this stuff, see https://github.com/ceph/ceph/pull/55469 The author claimed they needed this feature for their cluster so you might want to ask him about their user experience. W.r.t documentation - actually there are just two options - bdev_enable_discard - enables issuing discard to disk - bdev_async_discard - instructs whether discard requests are issued synchronously (along with disk extents release) or asynchronously (using a background thread). Thanks, Igor On 01/03/2024 13:06, jsterr@proxforge.de wrote:
Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have a number of drives in my fleet with old firmware that seems to have discard / TRIM bugs, as in the drives get bricked. Much worse is that since they're on legacy RAID HBAs, many of them can't be updated. ymmv.
On Mar 1, 2024, at 13:15, Igor Fedotov <igor.fedotov@croit.io> wrote:
I played with this feature a while ago and recall it had visible negative impact on user operations due to the need to submit tons of discard operations - effectively each data overwrite operation triggers one or more discard operation submission to disk.
And I doubt this has been widely used if any.
Nevertheless recently we've got a PR to rework some aspects of thread management for this stuff, see https://github.com/ceph/ceph/pull/55469
The author claimed they needed this feature for their cluster so you might want to ask him about their user experience.
W.r.t documentation - actually there are just two options
- bdev_enable_discard - enables issuing discard to disk
- bdev_async_discard - instructs whether discard requests are issued synchronously (along with disk extents release) or asynchronously (using a background thread).
Thanks,
Igor
On 01/03/2024 13:06, jsterr@proxforge.de wrote:
Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I came across an enterprise NVMe used for BlueFS DB whose performance dropped sharply after a few months of delivery (I won't mention the brand here but it was not among these 3: Intel, Samsung, Micron). It is clear that enabling bdev_enable_discard impacted performance, but this option also saved the platform after a few days of discard. IMHO the most important thing is to validate the behavior when there has been a write to the entire flash media. But this option has the merit of existing. it seems to me that the ideal would be not to have several options on bdev_*discard, and that this task should be asynchronous and with the (D)iscard instructions during a calmer period of activity (I do not see any impact if the instructions are lost during an OSD reboot) Le ven. 1 mars 2024 à 19:17, Igor Fedotov <igor.fedotov@croit.io> a écrit :
I played with this feature a while ago and recall it had visible negative impact on user operations due to the need to submit tons of discard operations - effectively each data overwrite operation triggers one or more discard operation submission to disk.
And I doubt this has been widely used if any.
Nevertheless recently we've got a PR to rework some aspects of thread management for this stuff, see https://github.com/ceph/ceph/pull/55469
The author claimed they needed this feature for their cluster so you might want to ask him about their user experience.
W.r.t documentation - actually there are just two options
- bdev_enable_discard - enables issuing discard to disk
- bdev_async_discard - instructs whether discard requests are issued synchronously (along with disk extents release) or asynchronously (using a background thread).
Thanks,
Igor
On 01/03/2024 13:06, jsterr@proxforge.de wrote:
Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We've had a specific set of drives that we've had to enable bdev_enable_discard and bdev_async_discard for in order to maintain acceptable performance on block clusters. I wrote the patch that Igor mentioned in order to try and send more parallel discards to the devices, but these ones in particular seem to process them in serial (based on observed discard counts and latency going to the device), which is unfortunate. We're also testing new firmware that suggests it should help alleviate some of the initial concerns we had about discards not keeping up which prompted the patch in the first place. Most of our drives do not need discards enabled (and definitely not without async) in order to maintain performance unless we're doing a full disk fio test or something like that where we're trying to find its cliff profile. We've used OSD classes to help target the options being applied to specific OSDs via centralized conf which helps when we would add new hosts that may have different drives so that the options weren't applied globally. Based on our experience, I wouldn't enable it unless you're seeing some sort of cliff-like behaviour as your OSDs run low on free space, or are heavily fragmented. I would also deem bdev_async_enabled = 1 to be a requirement so that it doesn't block user IO. Keep an eye on your discards being sent to devices and the discard latency, as well (via node_exporter, for example). Matt On 2024-03-02 06:18, David C. wrote:
I came across an enterprise NVMe used for BlueFS DB whose performance dropped sharply after a few months of delivery (I won't mention the brand here but it was not among these 3: Intel, Samsung, Micron). It is clear that enabling bdev_enable_discard impacted performance, but this option also saved the platform after a few days of discard.
IMHO the most important thing is to validate the behavior when there has been a write to the entire flash media. But this option has the merit of existing.
it seems to me that the ideal would be not to have several options on bdev_*discard, and that this task should be asynchronous and with the (D)iscard instructions during a calmer period of activity (I do not see any impact if the instructions are lost during an OSD reboot)
Le ven. 1 mars 2024 à 19:17, Igor Fedotov <igor.fedotov@croit.io> a écrit :
I played with this feature a while ago and recall it had visible negative impact on user operations due to the need to submit tons of discard operations - effectively each data overwrite operation triggers one or more discard operation submission to disk.
And I doubt this has been widely used if any.
Nevertheless recently we've got a PR to rework some aspects of thread management for this stuff, see https://github.com/ceph/ceph/pull/55469
The author claimed they needed this feature for their cluster so you might want to ask him about their user experience.
W.r.t documentation - actually there are just two options
- bdev_enable_discard - enables issuing discard to disk
- bdev_async_discard - instructs whether discard requests are issued synchronously (along with disk extents release) or asynchronously (using a background thread).
Thanks,
Igor
On 01/03/2024 13:06, jsterr@proxforge.de wrote:
Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Could we not consider setting up a “bluefstrim” which could be orchestrated ? This would avoid having a continuous stream of (D)iscard instructions on the disks during activity. A weekly (probably monthly) bluefstrim could probably be enough for platforms that really need it. Le sam. 2 mars 2024 à 12:58, Matt Vandermeulen <storage@reenigne.net> a écrit :
We've had a specific set of drives that we've had to enable bdev_enable_discard and bdev_async_discard for in order to maintain acceptable performance on block clusters. I wrote the patch that Igor mentioned in order to try and send more parallel discards to the devices, but these ones in particular seem to process them in serial (based on observed discard counts and latency going to the device), which is unfortunate. We're also testing new firmware that suggests it should help alleviate some of the initial concerns we had about discards not keeping up which prompted the patch in the first place.
Most of our drives do not need discards enabled (and definitely not without async) in order to maintain performance unless we're doing a full disk fio test or something like that where we're trying to find its cliff profile. We've used OSD classes to help target the options being applied to specific OSDs via centralized conf which helps when we would add new hosts that may have different drives so that the options weren't applied globally.
Based on our experience, I wouldn't enable it unless you're seeing some sort of cliff-like behaviour as your OSDs run low on free space, or are heavily fragmented. I would also deem bdev_async_enabled = 1 to be a requirement so that it doesn't block user IO. Keep an eye on your discards being sent to devices and the discard latency, as well (via node_exporter, for example).
Matt
On 2024-03-02 06:18, David C. wrote:
I came across an enterprise NVMe used for BlueFS DB whose performance dropped sharply after a few months of delivery (I won't mention the brand here but it was not among these 3: Intel, Samsung, Micron). It is clear that enabling bdev_enable_discard impacted performance, but this option also saved the platform after a few days of discard.
IMHO the most important thing is to validate the behavior when there has been a write to the entire flash media. But this option has the merit of existing.
it seems to me that the ideal would be not to have several options on bdev_*discard, and that this task should be asynchronous and with the (D)iscard instructions during a calmer period of activity (I do not see any impact if the instructions are lost during an OSD reboot)
Le ven. 1 mars 2024 à 19:17, Igor Fedotov <igor.fedotov@croit.io> a écrit :
I played with this feature a while ago and recall it had visible negative impact on user operations due to the need to submit tons of discard operations - effectively each data overwrite operation triggers one or more discard operation submission to disk.
And I doubt this has been widely used if any.
Nevertheless recently we've got a PR to rework some aspects of thread management for this stuff, see https://github.com/ceph/ceph/pull/55469
The author claimed they needed this feature for their cluster so you might want to ask him about their user experience.
W.r.t documentation - actually there are just two options
- bdev_enable_discard - enables issuing discard to disk
- bdev_async_discard - instructs whether discard requests are issued synchronously (along with disk extents release) or asynchronously (using a background thread).
Thanks,
Igor
On 01/03/2024 13:06, jsterr@proxforge.de wrote:
Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Periodic discard was actually attempted in the past: https://github.com/ceph/ceph/pull/20723 A proper implementation would probably need appropriate scheduling/throttling that can be tuned so as to balance against client I/O impact. Josh On Sat, Mar 2, 2024 at 6:20 AM David C. <david.casier@aevoo.fr> wrote:
Could we not consider setting up a “bluefstrim” which could be orchestrated ?
This would avoid having a continuous stream of (D)iscard instructions on the disks during activity.
A weekly (probably monthly) bluefstrim could probably be enough for platforms that really need it.
Le sam. 2 mars 2024 à 12:58, Matt Vandermeulen <storage@reenigne.net> a écrit :
We've had a specific set of drives that we've had to enable bdev_enable_discard and bdev_async_discard for in order to maintain acceptable performance on block clusters. I wrote the patch that Igor mentioned in order to try and send more parallel discards to the devices, but these ones in particular seem to process them in serial (based on observed discard counts and latency going to the device), which is unfortunate. We're also testing new firmware that suggests it should help alleviate some of the initial concerns we had about discards not keeping up which prompted the patch in the first place.
Most of our drives do not need discards enabled (and definitely not without async) in order to maintain performance unless we're doing a full disk fio test or something like that where we're trying to find its cliff profile. We've used OSD classes to help target the options being applied to specific OSDs via centralized conf which helps when we would add new hosts that may have different drives so that the options weren't applied globally.
Based on our experience, I wouldn't enable it unless you're seeing some sort of cliff-like behaviour as your OSDs run low on free space, or are heavily fragmented. I would also deem bdev_async_enabled = 1 to be a requirement so that it doesn't block user IO. Keep an eye on your discards being sent to devices and the discard latency, as well (via node_exporter, for example).
Matt
On 2024-03-02 06:18, David C. wrote:
I came across an enterprise NVMe used for BlueFS DB whose performance dropped sharply after a few months of delivery (I won't mention the brand here but it was not among these 3: Intel, Samsung, Micron). It is clear that enabling bdev_enable_discard impacted performance, but this option also saved the platform after a few days of discard.
IMHO the most important thing is to validate the behavior when there has been a write to the entire flash media. But this option has the merit of existing.
it seems to me that the ideal would be not to have several options on bdev_*discard, and that this task should be asynchronous and with the (D)iscard instructions during a calmer period of activity (I do not see any impact if the instructions are lost during an OSD reboot)
Le ven. 1 mars 2024 à 19:17, Igor Fedotov <igor.fedotov@croit.io> a écrit :
I played with this feature a while ago and recall it had visible negative impact on user operations due to the need to submit tons of discard operations - effectively each data overwrite operation triggers one or more discard operation submission to disk.
And I doubt this has been widely used if any.
Nevertheless recently we've got a PR to rework some aspects of thread management for this stuff, see https://github.com/ceph/ceph/pull/55469
The author claimed they needed this feature for their cluster so you might want to ask him about their user experience.
W.r.t documentation - actually there are just two options
- bdev_enable_discard - enables issuing discard to disk
- bdev_async_discard - instructs whether discard requests are issued synchronously (along with disk extents release) or asynchronously (using a background thread).
Thanks,
Igor
On 01/03/2024 13:06, jsterr@proxforge.de wrote:
Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Anthony D'Atri
-
David C.
-
Igor Fedotov
-
Joshua Baergen
-
jsterr＠deckplosion.de
-
jsterr＠proxforge.de
-
Matt Vandermeulen