12 Apr
2021
12 Apr
'21
8:16 p.m.
Hi all,
bdev_enable_discard has been in ceph for several major releases now
but it is still off by default.
Did anyone try it recently -- is it safe to use? And do you have perf
numbers before and after enabling?
Cheers, Dan
13 Apr
13 Apr
11:30 a.m.
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Hi all,
bdev_enable_discard has been in ceph for several major releases now
but it is still off by default.
Did anyone try it recently -- is it safe to use? And do you have perf
numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
It's highly debated if this still makes a difference with modern flash
devices. I don't think there is a real conclusion if you still need to
trim/discard blocks.
Wido
> Cheers, Dan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
1:37 p.m.
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander <wido(a)42on.com> wrote:
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Thanks, I've enabled it on a test box and am draining data to check
that it doesn't crash anything.
Hi all,
bdev_enable_discard has been in ceph for several major releases now
but it is still off by default.
Did anyone try it recently -- is it safe to use? And do you have perf
numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
It's highly debated if this still makes a
difference with modern flash
devices. I don't think there is a real conclusion if you still need to
trim/discard blocks.
Do you happen to have any more info on these debates? As you know we
have seen major performance issues on hypervisors that are not running
a periodic fstrim; we use similar or identical SATA ssds for HV local
storage and our block.db's. If it doesn't hurt anything, why wouldn't
we enable it by default?
Cheers, Dan
3:03 p.m.
On 4/13/21 4:07 AM, Dan van der Ster wrote:
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander
<wido(a)42on.com> wrote:
There's some good discussion in the original PR:
https://github.com/ceph/ceph/pull/14727
I suspect that the primary concerns for enabling it by default are two
fold: (1) the issue of having to maintain a blocklist for buggy firmware
implementations (2) Even "good" firmware can potentially see slowdowns
with bursts of trim commands due to needing to update the FTL metadata
per this comment:
https://github.com/ceph/ceph/pull/14727#issuecomment-342399578
The original issue of how to decide between online discard, periodic
bulk discard, or no discard is still an issue imho. I think we probably
need to get more feedback from people with real large deployments (hint
hint :D) before we enable online discard by default.
Mark
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Thanks, I've enabled it on a test box and am draining data to check
that it doesn't crash anything.
Hi all,
bdev_enable_discard has been in ceph for several major releases now
but it is still off by default.
Did anyone try it recently -- is it safe to use? And do you have perf
numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
It's highly debated if this still makes a
difference with modern flash
devices. I don't think there is a real conclusion if you still need to
trim/discard blocks.
Do you happen to have any more info on these debates? As you
know we
have seen major performance issues on hypervisors that are not running
a periodic fstrim; we use similar or identical SATA ssds for HV local
storage and our block.db's. If it doesn't hurt anything, why wouldn't
we enable it by default?
Cheers, Dan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:26 p.m.
On Tue, Apr 13, 2021 at 12:35 PM Mark Nelson <mnelson(a)redhat.com> wrote:
On 4/13/21 4:07 AM, Dan van der Ster wrote:
Thanks for the links. And further to those I found the attempt at a
periodic discard: https://github.com/ceph/ceph/pull/20723
Igor posted some performance numbers there for online and periodic,
both of which seem not very promising.
And I didn't find any further work on periodic discard for bitmap or beyond.
Since the runtime performance impact of this looks unpredictable,
maybe a conservative way to resume this work would be to allow discard
via the offline bluestore tooling?
Cheers, Dan
>
>
> Mark
>
>
> >
> > Cheers, Dan
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
On Tue, Apr 13, 2021 at 9:00 AM Wido den
Hollander <wido(a)42on.com> wrote:
There's some good discussion in the original PR:
https://github.com/ceph/ceph/pull/14727
I suspect that the primary concerns for enabling it by default are two
fold: (1) the issue of having to maintain a blocklist for buggy firmware
implementations (2) Even "good" firmware can potentially see slowdowns
with bursts of trim commands due to needing to update the FTL metadata
per this comment:
https://github.com/ceph/ceph/pull/14727#issuecomment-342399578
The original issue of how to decide between online discard, periodic
bulk discard, or no discard is still an issue imho. I think we probably
need to get more feedback from people with real large deployments (hint
hint :D) before we enable online discard by default.
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Thanks, I've enabled it on a test box and am draining data to check
that it doesn't crash anything.
Hi all,
bdev_enable_discard has been in ceph for several major releases now
but it is still off by default.
Did anyone try it recently -- is it safe to use? And do you have perf
numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
It's highly debated if this still makes a
difference with modern flash
devices. I don't think there is a real conclusion if you still need to
trim/discard blocks.
Do you happen to have any more info on these debates? As you
know we
have seen major performance issues on hypervisors that are not running
a periodic fstrim; we use similar or identical SATA ssds for HV local
storage and our block.db's. If it doesn't hurt anything, why wouldn't
we enable it by default?
15 Apr
15 Apr
1:29 p.m.
On 13/04/2021 11:07, Dan van der Ster wrote:
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander
<wido(a)42on.com> wrote:
These debates are more about if it really makes sense with modern SSDs
as the performance gain seems limited.
With older (SATA) SSDs it might, but with the modern NVMe DC-grade ones
people are doubting if it is still needed.
SATA 3.0 also had the issue that the TRIM command was a blocking command
where with SATA 3.1 it became async and thus non-blocking.
With NVMe it is a different story again.
I don't have links or papers for you, it's mainly stories I heard on
conferences and such.
Wido
> Cheers, Dan
>
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Thanks, I've enabled it on a test box and am draining data to check
that it doesn't crash anything.
Hi all,
bdev_enable_discard has been in ceph for several major releases now
but it is still off by default.
Did anyone try it recently -- is it safe to use? And do you have perf
numbers before and after enabling?
I have done so on SATA SSDs in a few cases and: it worked
Did I notice a real difference? Not really.
It's highly debated if this still makes a
difference with modern flash
devices. I don't think there is a real conclusion if you still need to
trim/discard blocks.
Do you happen to have any more info on these debates? As you know we
have seen major performance issues on hypervisors that are not running
a periodic fstrim; we use similar or identical SATA ssds for HV local
storage and our block.db's. If it doesn't hurt anything, why wouldn't
we enable it by default?
1122
days inactive
1125
days old
5 comments
3 participants
participants (3)
-
Dan van der Ster -
Mark Nelson -
Wido den Hollander