+ dev@ceph.io I'm responding via the mailing list (with you on copy) to ensure other developers interested in Crimson and ceph-volume can follow this discussion. While I believe this very topic primarily concerns Crimson rather than ceph-volume, I'll continue the conversation in this thread to maintain context for all interested parties. On Fri, Oct 10, 2025 at 9:11 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
Including aligning with coarse IU QLC block sizes?
The problem of guaranteeing I/Os at IU size involves multiple layers: - seastar: use the minimal (or optimal) unit for performing dma writes. - crimson: use the optimal size for performing writes if seastar intends to be more conservative, and uses the minimal write size. - ceph-volume if crimson is not able to detect the i/o granularity properly, and it exposes an interface allowing tools like ceph-volume to configure it, with, for instance, a tag in the device. ceph-volume could probably detect the device's optimal setting, and persist the i/o size in the tag. Currently, seastar uses 4K for block device's dma writes. after reviewing its implementation, i created - https://github.com/scylladb/seastar/pull/3045 : Improves DMA write alignment using block device properties - https://github.com/scylladb/seastar/pull/3046 : Enhances I/O size detection mechanisms These changes move Seastar from hardcoded 4K writes to using the device's reported optimal block size. But if we are aiming at a higher performance than what is provided by minimal_io_size, and if QLC devices expose a different size with, for example, sys/block/<device>/queue/optimal_io_size, we should use it instead. probably it's not seastar's responsibility to use this number, but crimson could take it into consideration when streaming the data to the device. If crimson cannot reliably detect the I/O granularity, we could expose a configuration interface allowing tools like ceph-volume to set this via, for instance, device's superblock. Probably as you noticed, so far, we've being discussing seastar and crimson instead of ceph-volume, the reason is that the i/o granularity is more relevant to crimson than ceph-volume, unless crimson is not able to use the reported number from the system. On Oct 10, 2025, at 12:15 AM, kefu chai <tchaikov@gmail.com> wrote: I've been focusing on adding support for Seastore's secondary devices (for tiering)
-- Regards Kefu Chai
Thanks. Code was introduced several years ago to set the BlueStore min_alloc_size to queue/optimal_io_size when available, with the usual safeguards. Mark Nelson has a PR outstanding to enable this by default. The drives I’ve been been able to work with have reliably reported this to the kernel, at least with recent-ish kernels. For context: conventional SSDs, which these days are mainly TLC, have a 4KiB IU, which means that the onboard indirection table that maps between LBAs and NAND does so at 4KiB granularity, this can be roughly considered the block size. Many enterprise QLC and QLC-class drives † have larger IU sizes, for two reasons: 1) Enterprise SSDs typically have ~ 1GiB of onboard DRAM per TiB of NAND capacity for this indirection table. In order to maintain a price delta between TLC and QLC (class) SKUs, there is often less than this ratio of DRAM, because especially for larger SKUs it can be an appreciable fraction of the materials cost. 2) With larger SKUs, there may simply not be enough physical space in the form factor for large amounts of DRAM. What does this mean to us? QLC (and putative PLC to come) tends to inherently offer lower endurance and write performance compared to conventional TLC. Any write to an LBA incur a RMW cycle: existing data is read, the write is overlaid in RAM, then written to NAND. Say on a 64 KiB IU device one makes 4KiB random writes. Each of those incurs an RMW cycle, so there are op patterns that could result in that 64 KB LBA being written 16 times, that’s considerable space amp that burns PE cycles. This similarly happens with misaligned writes; a 64 KiB write that starts at the middle of one LBA thus overlaps a second, that’s 2x write amp. Drive firmware endeavors to coalesce such writes, and I’ve seen certain drives do so quite effectively with 4KiB sequential writes. There’s only so much the drive can stage for coalescing, though, and such drives really like sequential workloads. So there are considerable benefits to aligning writes with the IU size. Why bother with media like this? * There are 122 TiB SKUs on the market today, with 245 TiB SKUs incipient. This allows stratospheric density per-chassis and per-RU, with concomitant DC physical, financial, and administrative benefits. RUs cost money, so do chassis. Especially when certain brands work really hard to push you to buy a superfluous, flaky, and pricey tri-mode RAID HBA with FBWC — which almost nobody monitors well. * CapEx competitiveness with HDDs, which are increasingly bottlenecks as their capacity grows. Today’s 32 TiB HDD has the same tired SATA interface as a 3 TiB HDD did years ago, which was already a bottleneck then. Any SSD whips any HDD in terms of $/IOP, and modern QLC is competitive with TLC in terms of reads. Applications include object storage, which often is quite heavily read-mostly. One commercial RGW implementation has been seen to experience 0.01 DWPD. CephFS archival or read-mostly workloads similarly benefit. There is a Cephalocon presentation this year around these dynamics, building on my prior work. † Examples: Intel / Solidigm P5316: 64 KiB IU, P5336: 16 KiB IU, Micron P6550 (QLC-like TLC): 16 KiB IU
On Oct 11, 2025, at 2:34 AM, kefu chai <tchaikov@gmail.com> wrote:
+ dev@ceph.io <mailto:dev@ceph.io>
I'm responding via the mailing list (with you on copy) to ensure other developers interested in Crimson and ceph-volume can follow this discussion.
While I believe this very topic primarily concerns Crimson rather than ceph-volume, I'll continue the conversation in this thread to maintain context for all interested parties.
On Fri, Oct 10, 2025 at 9:11 PM Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> wrote:
Including aligning with coarse IU QLC block sizes?
The problem of guaranteeing I/Os at IU size involves multiple layers:
- seastar: use the minimal (or optimal) unit for performing dma writes. - crimson: use the optimal size for performing writes if seastar intends to be more conservative, and uses the minimal write size. - ceph-volume if crimson is not able to detect the i/o granularity properly, and it exposes an interface allowing tools like ceph-volume to configure it, with, for instance, a tag in the device. ceph-volume could probably detect the device's optimal setting, and persist the i/o size in the tag.
Currently, seastar uses 4K for block device's dma writes. after reviewing its implementation, i created
- https://github.com/scylladb/seastar/pull/3045 : Improves DMA write alignment using block device properties - https://github.com/scylladb/seastar/pull/3046 : Enhances I/O size detection mechanisms
These changes move Seastar from hardcoded 4K writes to using the device's reported optimal block size. But if we are aiming at a higher performance than what is provided by minimal_io_size, and if QLC devices expose a different size with, for example, sys/block/<device>/queue/optimal_io_size, we should use it instead. probably it's not seastar's responsibility to use this number, but crimson could take it into consideration when streaming the data to the device.
If crimson cannot reliably detect the I/O granularity, we could expose a configuration interface allowing tools like ceph-volume to set this via, for instance, device's superblock. Probably as you noticed, so far, we've being discussing seastar and crimson instead of ceph-volume, the reason is that the i/o granularity is more relevant to crimson than ceph-volume, unless crimson is not able to use the reported number from the system.
On Oct 10, 2025, at 12:15 AM, kefu chai <tchaikov@gmail.com <mailto:tchaikov@gmail.com>> wrote:
I've been focusing on adding support for Seastore's secondary devices (for tiering)
-- Regards Kefu Chai _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Anthony, Thanks for your insights in QLC support in Ceph. But as I tried to explain in previous mail, what I wanted to collect in this mail thread is the plan on crimson support in ceph-volume, not really on why we need to care about the optimal IU of QLC drives. On Sat, Oct 11, 2025 at 10:50 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
Thanks.
Code was introduced several years ago to set the BlueStore min_alloc_size to queue/optimal_io_size when available, with the usual safeguards. Mark Nelson has a PR outstanding to enable this by default. The drives I’ve been been able to work with have reliably reported this to the kernel, at least with recent-ish kernels.
For context: conventional SSDs, which these days are mainly TLC, have a 4KiB IU, which means that the onboard indirection table that maps between LBAs and NAND does so at 4KiB granularity, this can be roughly considered the block size.
Many enterprise QLC and QLC-class drives † have larger IU sizes, for two reasons:
1) Enterprise SSDs typically have ~ 1GiB of onboard DRAM per TiB of NAND capacity for this indirection table. In order to maintain a price delta between TLC and QLC (class) SKUs, there is often less than this ratio of DRAM, because especially for larger SKUs it can be an appreciable fraction of the materials cost.
2) With larger SKUs, there may simply not be enough physical space in the form factor for large amounts of DRAM.
What does this mean to us? QLC (and putative PLC to come) tends to inherently offer lower endurance and write performance compared to conventional TLC. Any write to an LBA incur a RMW cycle: existing data is read, the write is overlaid in RAM, then written to NAND. Say on a 64 KiB IU device one makes 4KiB random writes. Each of those incurs an RMW cycle, so there are op patterns that could result in that 64 KB LBA being written 16 times, that’s considerable space amp that burns PE cycles. This similarly happens with misaligned writes; a 64 KiB write that starts at the middle of one LBA thus overlaps a second, that’s 2x write amp. Drive firmware endeavors to coalesce such writes, and I’ve seen certain drives do so quite effectively with 4KiB sequential writes. There’s only so much the drive can stage for coalescing, though, and such drives really like sequential workloads.
So there are considerable benefits to aligning writes with the IU size.
Why bother with media like this?
* There are 122 TiB SKUs on the market today, with 245 TiB SKUs incipient. This allows stratospheric density per-chassis and per-RU, with concomitant DC physical, financial, and administrative benefits. RUs cost money, so do chassis. Especially when certain brands work really hard to push you to buy a superfluous, flaky, and pricey tri-mode RAID HBA with FBWC — which almost nobody monitors well.
* CapEx competitiveness with HDDs, which are increasingly bottlenecks as their capacity grows. Today’s 32 TiB HDD has the same tired SATA interface as a 3 TiB HDD did years ago, which was already a bottleneck then. Any SSD whips any HDD in terms of $/IOP, and modern QLC is competitive with TLC in terms of reads.
Applications include object storage, which often is quite heavily read-mostly. One commercial RGW implementation has been seen to experience 0.01 DWPD. CephFS archival or read-mostly workloads similarly benefit.
There is a Cephalocon presentation this year around these dynamics, building on my prior work.
† Examples: Intel / Solidigm P5316: 64 KiB IU, P5336: 16 KiB IU, Micron P6550 (QLC-like TLC): 16 KiB IU
On Oct 11, 2025, at 2:34 AM, kefu chai <tchaikov@gmail.com> wrote:
+ dev@ceph.io
I'm responding via the mailing list (with you on copy) to ensure other developers interested in Crimson and ceph-volume can follow this discussion.
While I believe this very topic primarily concerns Crimson rather than ceph-volume, I'll continue the conversation in this thread to maintain context for all interested parties.
On Fri, Oct 10, 2025 at 9:11 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
Including aligning with coarse IU QLC block sizes?
The problem of guaranteeing I/Os at IU size involves multiple layers:
- seastar: use the minimal (or optimal) unit for performing dma writes. - crimson: use the optimal size for performing writes if seastar intends to be more conservative, and uses the minimal write size. - ceph-volume if crimson is not able to detect the i/o granularity properly, and it exposes an interface allowing tools like ceph-volume to configure it, with, for instance, a tag in the device. ceph-volume could probably detect the device's optimal setting, and persist the i/o size in the tag.
Currently, seastar uses 4K for block device's dma writes. after reviewing its implementation, i created
- https://github.com/scylladb/seastar/pull/3045 : Improves DMA write alignment using block device properties - https://github.com/scylladb/seastar/pull/3046 : Enhances I/O size detection mechanisms
These changes move Seastar from hardcoded 4K writes to using the device's reported optimal block size. But if we are aiming at a higher performance than what is provided by minimal_io_size, and if QLC devices expose a different size with, for example, sys/block/<device>/queue/optimal_io_size, we should use it instead. probably it's not seastar's responsibility to use this number, but crimson could take it into consideration when streaming the data to the device.
If crimson cannot reliably detect the I/O granularity, we could expose a configuration interface allowing tools like ceph-volume to set this via, for instance, device's superblock. Probably as you noticed, so far, we've being discussing seastar and crimson instead of ceph-volume, the reason is that the i/o granularity is more relevant to crimson than ceph-volume, unless crimson is not able to use the reported number from the system.
On Oct 10, 2025, at 12:15 AM, kefu chai <tchaikov@gmail.com> wrote:
I've been focusing on adding support for Seastore's secondary devices (for tiering)
-- Regards Kefu Chai _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Regards Kefu Chai
participants (2)
-
Anthony D'Atri
-
kefu chai