Hi, Yes, most of the optimizations for partial reads are part of that pull request – it is referenced in the design document. There is one further enhancement we would like to make which is to reduce the size of the reads to each shard – the pull request currently reads the whole chunk for any shard that needs to be read. This is not an issue with the current default chunk size which is 4K, but if you use a larger chunk size such as 256K then this means that short client reads will read excess data and then discard it. Currently there are many reasons why using a small chunk size is a good idea, but once both partial reads and partial writes have been implemented performance can be improved further by using a larger chunk size because splitting I/Os into multiple chunks is expensive. There’s more discussion about the chunk size and how to accommodate small sized objects without large storage overheads later in the document. Cheers, Bill. bill_scales@uk.ibm.com<mailto:bill_scales@uk.ibm.com> IBM Distinguished Engineer, IBM Storage From: Sebastian Wagner <sebastian.wagner@croit.io> Date: Monday, 1 July 2024 at 09:31 To: Bill Scales <bill_scales@uk.ibm.com>, dev@ceph.io <dev@ceph.io> Subject: [EXTERNAL] Re: Erasure coding enhancements - design for review Hi Bill, Is the Partial Reads section the same as https: //github. com/ceph/ceph/pull/55196 ? Best, Sebastian Am 01. 07. 24 um 10: 08 schrieb Bill Scales: Hi, We are planning to enhance the performance of erasure coding, in particular for use with Hi Bill, Is the Partial Reads section the same as https://github.com/ceph/ceph/pull/55196 ? Best, Sebastian Am 01.07.24 um 10:08 schrieb Bill Scales: Hi, We are planning to enhance the performance of erasure coding, in particular for use with block and file. We've got a design document https://github.com/bill-scales/CephErasureCodingDesign that sets out what we are hoping to achieve. We welcome your feedback, either posting your comments in Slack on #ceph-devel , raising issues in github or getting in contact with myself Cheers, Bill. bill_scales@uk.ibm.com<mailto:bill_scales@uk.ibm.com> IBM Distinguished Engineer, IBM Storage Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io> -- Head of Software Development E-Mail: sebastian.wagner@croit.io<mailto:sebastian.wagner@croit.io> croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web<https://croit.io/> | LinkedIn<http://linkedin.com/company/croit> | Youtube<https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter<https://twitter.com/croit_io> TOP 100 Innovator Award Winner<https://croit.io/blog/croit-receives-top-100-seal> by compamedia Technology Fast50 Award<https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

It will likely never be as fast as replication, but it can be much faster than it is now.  I put benchmarks in the original PR I resurrected for partial stripe reads: https://github.com/ceph/ceph/pull/52746 I'm trying to find some time this week to run some tests on Radek's PR so we can do some comparisons. Mark On 7/2/24 02:28, Sebastian Wagner wrote:

Hi Bill,

I noticed you mentioned the PR a few moments after I clicked on send 🙈. I really hope we end up with EC being on par with replicated pools without any meaningful performance impact.

Do you think we can put those design documents into the Ceph Github organization? IIRC, this is the third proposal on this mailing list that I saw just in a few months.

Best, Sebastian

Am 01.07.24 um 12:01 schrieb Bill Scales:

...

Hi,

Yes, most of the optimizations for partial reads are part of that pull request – it is referenced in the design document.

There is one further enhancement we would like to make which is to reduce the size of the reads to each shard – the pull request currently reads the whole chunk for any shard that needs to be read. This is not an issue with the current default chunk size which is 4K, but if you use a larger chunk size such as 256K then this means that short client reads will read excess data and then discard it.

Currently there are many reasons why using a small chunk size is a good idea, but once both partial reads and partial writes have been implemented performance can be improved further by using a larger chunk size because splitting I/Os into multiple chunks is expensive. There’s more discussion about the chunk size and how to accommodate small sized objects without large storage overheads later in the document.

Cheers,

Bill. bill_scales@uk.ibm.com <mailto:bill_scales@uk.ibm.com> IBM Distinguished Engineer, IBM Storage

*From: *Sebastian Wagner <sebastian.wagner@croit.io> *Date: *Monday, 1 July 2024 at 09:31 *To: *Bill Scales <bill_scales@uk.ibm.com>, dev@ceph.io <dev@ceph.io> *Subject: *[EXTERNAL] Re: Erasure coding enhancements - design for review

Hi Bill, Is the Partial Reads section the same as https: //github. com/ceph/ceph/pull/55196 ? Best, Sebastian Am 01. 07. 24 um 10: 08 schrieb Bill Scales: Hi, We are planning to enhance the performance of erasure coding, in particular for use with

Hi Bill,

Is the Partial Reads section the same as https://github.com/ceph/ceph/pull/55196 ?

Best, Sebastian

Am 01.07.24 um 10:08 schrieb Bill Scales:

Hi,

We are planning to enhance the performance of erasure coding, in particular for use with block and file. We've got a design document https://github.com/bill-scales/CephErasureCodingDesign that sets out what we are hoping to achieve. We welcome your feedback, either posting your comments in Slack on #ceph-devel , raising issues in github or getting in contact with myself

Cheers,

Bill. bill_scales@uk.ibm.com <mailto:bill_scales@uk.ibm.com> IBM Distinguished Engineer, IBM Storage

Unless otherwise stated above:

IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

_______________________________________________

Dev mailing list --dev@ceph.io

To unsubscribe send an email todev-leave@ceph.io

-- Head of Software Development E-Mail: sebastian.wagner@croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263

Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io>

TOP 100 Innovator Award Winner <https://croit.io/blog/croit-receives-top-100-seal> by compamedia Technology Fast50 Award <https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte

Unless otherwise stated above:

IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

-- Head of Software Development E-Mail: sebastian.wagner@croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263

Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io>

TOP 100 Innovator Award Winner <https://croit.io/blog/croit-receives-top-100-seal> by compamedia Technology Fast50 Award <https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte

_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/

Hi, Firstly, lets assure everyone that we are not taking the ?fast read? option away, that will still be available and will work as it does today. Personally, I?m not convinced that fast read works as well in practice as people might think. The problem is that each delayed read will end up blocking a thread. If a delay impacts more than one or two I/Os the cluster will quickly run out of threads, and this will then stop any I/O from progressing. I can believe it improves tail-latencies when you get one off glitches, but not when you have a more pervasive problem such as a sick but not yet dead HDD. Fast writes are a trickier problem ? if you start completing writes before all the updates have been completed then you are reducing redundancy. For example, if you have a 6+2 erasure coding and you allow a write to complete with one delayed update then you will only have +1 redundancy for the whole placement group until the delayed update completes. If you allow two writes to complete each with an delayed update to different OSDs then you might have lost all your redundancy. Therefore, you probably want the cluster to coordinate which and how many OSDs it allows to fall behind in processing. It might be easier to implement both fast reads and writes by having a very aggressive I/O timeout and taking OSDs down when they fail to complete read/writes in a timely fashion. Ideally you want the OSD to stay down until the storage device completes the slow I/Os giving some indication that the device has recovered. This would then let reads and writes progress without the OSD and let recovery and backfill deal with bringing the OSD back up to date assuming it recovers. You would probably want a configuration parameter to define how much redundancy you are willing to trade off for better performance and disable the aggressive timeouts when this threshold is reached. There?s probably work to do to reduce delays to I/Os when an OSD is taken down to make this approach viable. Cheers, Bill. bill_scales@uk.ibm.com<mailto:bill_scales@uk.ibm.com> IBM Distinguished Engineer, IBM Storage From: Frank Schilder <frans@dtu.dk> Date: Tuesday, 2 July 2024 at 08:59 To: Sebastian Wagner <sebastian.wagner@croit.io>, Bill Scales <bill_scales@uk.ibm.com>, dev@ceph.io <dev@ceph.io> Subject: [EXTERNAL] Re: Erasure coding enhancements - design for review Hi Bill, Sebastian and rados developers, this seems a good opportunity to raise awareness of the discussion around this comment: https://www.mail-archive.com/ceph-users@ceph.io/msg24115.html . Many of the optimizations discussed in the design draft start targeting individual OSDs with the aim of improving performance and reducing load. Assuming all OSDs are working equally well it s likely to have the desired effect. However, in the event that not all OSDs are operating equally well, that is, the occurrence of tail latencies due to a few (temporarily) slow disks, the following write path might beat a path limited to and directly addressing a slow OSD: - assuming we use the "fast write" option proposed in the e-mail referenced above, we send new shards to all OSDs - we ack to the client when min_size OSDs reply with ack - this approach explicitly uses network- and CPU amplification in exchange for latency reduction with the simple "let the fastest win" strategy, we had great success with the already existing "fast read" option that made client IO latencies more predictable and eliminated the effect of tail latencies from reads; in our cluster design we explicitly take this into account when designing network and CPU per disk This effectively makes the write to slowest disks asynchronous from client IO. In my experience with 6+2, 8+2 and 8+3 EC profiles, this would be a significant improvement to reduce overall IO latencies with both, spinning and solid state drives. We have enterprise SSDs in our cluster that sometimes stall for up to a few seconds and then catch up again. It would be awesome if these temporarily slow drives would not need to be waited for by clients. I would be most grateful if you could consider both options for optimization: (1) reducing the IO path to the absolute minimum number of disks and shards as proposed in the draft document and (2) going the opposite way and "just send it to everyone and let the fastest win" as proposed in the comment linked above. On realistic clusters with a small fraction of mildly broken hardware I would assume option (2) to win. For the highly optimized option (1) I would expect that hardware health needs to be maintained at very high levels to be worth the effort as tail latencies will likely kill its benefits. It might also be possible to combine both ideas and get the best of both worlds? I would like to point to message https://www.mail-archive.com/ceph-users@ceph.io/msg24126.html collecting an a-priori discussion of how much effort it will be to implement the "fast write" option. It sounds not that difficult to me. Best regards, ================= Frank Schilder AIT Ris? Campus Bygning 109, rum S14 ________________________________________ From: Sebastian Wagner <sebastian.wagner@croit.io> Sent: Monday, July 1, 2024 10:31 AM To: Bill Scales; dev@ceph.io Subject: Re: Erasure coding enhancements - design for review Hi Bill, Is the Partial Reads section the same as https://github.com/ceph/ceph/pull/55196 ? Best, Sebastian Am 01.07.24 um 10:08 schrieb Bill Scales: Hi, We are planning to enhance the performance of erasure coding, in particular for use with block and file. We've got a design document https://github.com/bill-scales/CephErasureCodingDesign that sets out what we are hoping to achieve. We welcome your feedback, either posting your comments in Slack on #ceph-devel , raising issues in github or getting in contact with myself Cheers, Bill. bill_scales@uk.ibm.com<mailto:bill_scales@uk.ibm.com> IBM Distinguished Engineer, IBM Storage Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io> -- Head of Software Development E-Mail: sebastian.wagner@croit.io<mailto:sebastian.wagner@croit.io> croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web<https://croit.io/> | LinkedIn<http://linkedin.com/company/croit> | Youtube<https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter<https://twitter.com/croit_io> TOP 100 Innovator Award Winner<https://croit.io/blog/croit-receives-top-100-seal> by compamedia Technology Fast50 Award<https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Thoughts on how finding a way to implement OpenCAS might address this?

On Jul 2, 2024, at 15:44, Dan van der Ster <dan.vanderster@clyso.com> wrote:

A better approach could be to read around misses, using the new faster EC partial read to respond to the client quickly, and then asynchronously promote the whole object into the cache pool. Partial writes are more tricky, probably best handled all via a WAL.

We've had multiple conversations with Intel in the past around OpenCAS.  They had some pretty impressive performance numbers and it was fairly flexible (you could do things like pin block regions into the cache).  I had always hoped that they would get it mainlined into the kernel, but ultimately that didn't happen. There's been some more userland focused stuff coming out of the drive vendors though that might be interesting. I still would like to get back to my idea at some point of allowing bluestore overwrite extents to live on the fast device and then large (maybe not full) object writes to the slow device of the most fragmented objects.  The idea here would be that once an overwrite extent is written to the fast device, we no longer need to create a new extent on the slow device (causing fragmentation) but can instead write it in-place.  Potentially if you have compressed data you can accumulate lots of overwrite extents on the fast device and then re-compress in one go.  I know Adam has a slightly different approach he would like to take here, but the gist of it is that COW like we do in bluestore is painful when you have flash at your disposal. Mark On 7/2/24 15:00, Dan van der Ster wrote:

Totally: OpenCAS, bcache, etc could also help here.

The important thing for me if we use local device caches is to make sure our internal OSD data 'tegrity is still sound: deep-scrubbing should readthrough, for example.

Cheers, Dan

On Tue, Jul 2, 2024 at 12:56 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:

...

Thoughts on how finding a way to implement OpenCAS might address this?

On Jul 2, 2024, at 15:44, Dan van der Ster <dan.vanderster@clyso.com> wrote:

A better approach could be to read around misses, using the new faster EC partial read to respond to the client quickly, and then asynchronously promote the whole object into the cache pool. Partial writes are more tricky, probably best handled all via a WAL.

_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

---

Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io

-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/