has anyone enabled bdev_enable_discard?
Hi all, bdev_enable_discard has been in ceph for several major releases now but it is still off by default. Did anyone try it recently -- is it safe to use? And do you have perf numbers before and after enabling? Cheers, Dan
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Hi all,
bdev_enable_discard has been in ceph for several major releases now but it is still off by default. Did anyone try it recently -- is it safe to use? And do you have perf numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked Did I notice a real difference? Not really. It's highly debated if this still makes a difference with modern flash devices. I don't think there is a real conclusion if you still need to trim/discard blocks. Wido
Cheers, Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander <wido@42on.com> wrote:
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Hi all,
bdev_enable_discard has been in ceph for several major releases now but it is still off by default. Did anyone try it recently -- is it safe to use? And do you have perf numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
Thanks, I've enabled it on a test box and am draining data to check that it doesn't crash anything.
It's highly debated if this still makes a difference with modern flash devices. I don't think there is a real conclusion if you still need to trim/discard blocks.
Do you happen to have any more info on these debates? As you know we have seen major performance issues on hypervisors that are not running a periodic fstrim; we use similar or identical SATA ssds for HV local storage and our block.db's. If it doesn't hurt anything, why wouldn't we enable it by default? Cheers, Dan
On 4/13/21 4:07 AM, Dan van der Ster wrote:
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander <wido@42on.com> wrote:
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Hi all,
bdev_enable_discard has been in ceph for several major releases now but it is still off by default. Did anyone try it recently -- is it safe to use? And do you have perf numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
Thanks, I've enabled it on a test box and am draining data to check that it doesn't crash anything.
It's highly debated if this still makes a difference with modern flash devices. I don't think there is a real conclusion if you still need to trim/discard blocks. Do you happen to have any more info on these debates? As you know we have seen major performance issues on hypervisors that are not running a periodic fstrim; we use similar or identical SATA ssds for HV local storage and our block.db's. If it doesn't hurt anything, why wouldn't we enable it by default?
There's some good discussion in the original PR: https://github.com/ceph/ceph/pull/14727 I suspect that the primary concerns for enabling it by default are two fold: (1) the issue of having to maintain a blocklist for buggy firmware implementations (2) Even "good" firmware can potentially see slowdowns with bursts of trim commands due to needing to update the FTL metadata per this comment: https://github.com/ceph/ceph/pull/14727#issuecomment-342399578 The original issue of how to decide between online discard, periodic bulk discard, or no discard is still an issue imho. I think we probably need to get more feedback from people with real large deployments (hint hint :D) before we enable online discard by default. Mark
Cheers, Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Apr 13, 2021 at 12:35 PM Mark Nelson <mnelson@redhat.com> wrote:
On 4/13/21 4:07 AM, Dan van der Ster wrote:
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander <wido@42on.com> wrote:
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Hi all,
bdev_enable_discard has been in ceph for several major releases now but it is still off by default. Did anyone try it recently -- is it safe to use? And do you have perf numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
Thanks, I've enabled it on a test box and am draining data to check that it doesn't crash anything.
It's highly debated if this still makes a difference with modern flash devices. I don't think there is a real conclusion if you still need to trim/discard blocks. Do you happen to have any more info on these debates? As you know we have seen major performance issues on hypervisors that are not running a periodic fstrim; we use similar or identical SATA ssds for HV local storage and our block.db's. If it doesn't hurt anything, why wouldn't we enable it by default?
There's some good discussion in the original PR:
https://github.com/ceph/ceph/pull/14727
I suspect that the primary concerns for enabling it by default are two fold: (1) the issue of having to maintain a blocklist for buggy firmware implementations (2) Even "good" firmware can potentially see slowdowns with bursts of trim commands due to needing to update the FTL metadata per this comment:
https://github.com/ceph/ceph/pull/14727#issuecomment-342399578
The original issue of how to decide between online discard, periodic bulk discard, or no discard is still an issue imho. I think we probably need to get more feedback from people with real large deployments (hint hint :D) before we enable online discard by default.
Thanks for the links. And further to those I found the attempt at a periodic discard: https://github.com/ceph/ceph/pull/20723 Igor posted some performance numbers there for online and periodic, both of which seem not very promising. And I didn't find any further work on periodic discard for bitmap or beyond. Since the runtime performance impact of this looks unpredictable, maybe a conservative way to resume this work would be to allow discard via the offline bluestore tooling? Cheers, Dan
Mark
Cheers, Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 13/04/2021 11:07, Dan van der Ster wrote:
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander <wido@42on.com> wrote:
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Hi all,
bdev_enable_discard has been in ceph for several major releases now but it is still off by default. Did anyone try it recently -- is it safe to use? And do you have perf numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
Thanks, I've enabled it on a test box and am draining data to check that it doesn't crash anything.
It's highly debated if this still makes a difference with modern flash devices. I don't think there is a real conclusion if you still need to trim/discard blocks.
Do you happen to have any more info on these debates? As you know we have seen major performance issues on hypervisors that are not running a periodic fstrim; we use similar or identical SATA ssds for HV local storage and our block.db's. If it doesn't hurt anything, why wouldn't we enable it by default?
These debates are more about if it really makes sense with modern SSDs as the performance gain seems limited. With older (SATA) SSDs it might, but with the modern NVMe DC-grade ones people are doubting if it is still needed. SATA 3.0 also had the issue that the TRIM command was a blocking command where with SATA 3.1 it became async and thus non-blocking. With NVMe it is a different story again. I don't have links or papers for you, it's mainly stories I heard on conferences and such. Wido
Cheers, Dan
participants (3)
-
Dan van der Ster
-
Mark Nelson
-
Wido den Hollander