Performance scaling issue with multi-SSD (CrimsonOSD/Seastore)
Hello Ceph community, I am evaluating Crimson OSD + Seastore performance for potential deployment in a distributed storage environment. With BlueStore, I have been able to achieve satisfying performance levels in my FIO tests for 4K random read/write IOPS. However, when testing Crimson OSD + Seastore, I observed that 4K random read/write IOPS do not scale as expected when increasing the number of SSDs/OSDs. The performance plateaus beyond a certain point or is much lower than expected. (See attached test results.) Test Environment: - Cluster: 8 clients, 1 OSD - Hardware: 40-core CPUs, 377 GiB DRAM - Image SHA (quay.io): e0543089a9e9cae97999761059eaccdf6bb22e9e - Configuration parameters: osd_memory_target = 34359738368 crimson_osd_scheduler_concurrency = 0 seastore_max_concurrent_transactions = 16 crimson_osd_obc_lru_size = 8192 seastore_cache_lru_size = 16G seastore_obj_data_write_amplification = 4 seastore_journal_batch_capacity = 1024 seastore_journal_batch_flush_size = 256M seastore_journal_iodepth_limit = 16 seastore_journal_batch_preferred_fullness = 0.8 seastore_segment_size = 128M seastore_device_size = 512G seastore_block_create = true seastore_default_object_metadata_reservation = 1073741824 rbd_cache = false rbd_cache_writethrough_until_flush = true rbd_op_threads = 16 Replication policy: - 4096 PGs, no replication (only 1 copy) Test Results: 1 SSD test (varying number of allocated CPUs, alien threads = 26-29, 36-39): num CPU | 4k randread | 4k randwrite | Allocated CPU sets 2 | 126772 | 14830 | 0-1 4 | 107860 | 16451 | 0-3 6 | 113741 | 17019 | 0-5 8 | 132060 | 16099 | 0-7 SSD scaling test (2 CPUs per SSD): OSD CPU mapping: OSD.0 (0-1), OSD.1 (10-11), OSD.2 (2-3), OSD.3 (12-13), ..., OSD.15 (34-35), Alien threads (26-29, 36-39) num SSD | 4k randread | 4k randwrite 4 | 861273 | 22360 8 | 1022793 | 22786 12 | 1019161 | 21211 16 | 927570 | 20502 SSD scaling test (1 CPU per SSD): OSD CPU mapping: OSD.0 (0), OSD.1 (10), OSD.2 (2), OSD.3 (12), ..., OSD.15 (24), Alien CPUs: 1, 11, 3, 13, ..., 15, 25 num SSD | 4k randread | 4k randwrite 4 | 936685 | 13730 8 | 1048204 | 18259 12 | 922727 | 23078 16 | 987838 | 30792 Questions: 1. Since Seastore is still under active development, are there any known unresolved performance issues that could explain this scaling behavior? 2. Are there recommended tuning parameters for improving small-block read scalability in multi-SSD configurations? 3. Regarding alien threads, are there best practices for CPU pinning or NUMA-aware placement that have shown measurable improvements? 4. Any additional guidance for maximizing IOPS with Crimson OSD + Seastore would be greatly appreciated. My goal is to be ready to switch from BlueStore to Crimson + Seastore after it becomes stable and shows reasonable performance compared to BlueStore, so I’d like to understand the current limitations and tuning opportunities. Thank you, Ki-taek Lee
Do you have BMC perf / thermal profiles set to performance ? Do you have TuneD installed with the latency-performance profile active to prevent deep C-states? As people start deploying Crimson I’m curious how much these affect the experience.
On Aug 11, 2025, at 11:44 PM, Ki-taek Lee <ktlee4311@gmail.com> wrote:
Hello Ceph community,
I am evaluating Crimson OSD + Seastore performance for potential deployment in a distributed storage environment. With BlueStore, I have been able to achieve satisfying performance levels in my FIO tests for 4K random read/write IOPS.
However, when testing Crimson OSD + Seastore, I observed that 4K random read/write IOPS do not scale as expected when increasing the number of SSDs/OSDs. The performance plateaus beyond a certain point or is much lower than expected. (See attached test results.)
Test Environment: - Cluster: 8 clients, 1 OSD - Hardware: 40-core CPUs, 377 GiB DRAM - Image SHA (quay.io): e0543089a9e9cae97999761059eaccdf6bb22e9e - Configuration parameters: osd_memory_target = 34359738368 crimson_osd_scheduler_concurrency = 0 seastore_max_concurrent_transactions = 16 crimson_osd_obc_lru_size = 8192 seastore_cache_lru_size = 16G seastore_obj_data_write_amplification = 4 seastore_journal_batch_capacity = 1024 seastore_journal_batch_flush_size = 256M seastore_journal_iodepth_limit = 16 seastore_journal_batch_preferred_fullness = 0.8 seastore_segment_size = 128M seastore_device_size = 512G seastore_block_create = true seastore_default_object_metadata_reservation = 1073741824 rbd_cache = false rbd_cache_writethrough_until_flush = true rbd_op_threads = 16
Replication policy: - 4096 PGs, no replication (only 1 copy)
Test Results:
1 SSD test (varying number of allocated CPUs, alien threads = 26-29, 36-39): num CPU | 4k randread | 4k randwrite | Allocated CPU sets 2 | 126772 | 14830 | 0-1 4 | 107860 | 16451 | 0-3 6 | 113741 | 17019 | 0-5 8 | 132060 | 16099 | 0-7
SSD scaling test (2 CPUs per SSD): OSD CPU mapping: OSD.0 (0-1), OSD.1 (10-11), OSD.2 (2-3), OSD.3 (12-13), ..., OSD.15 (34-35), Alien threads (26-29, 36-39) num SSD | 4k randread | 4k randwrite 4 | 861273 | 22360 8 | 1022793 | 22786 12 | 1019161 | 21211 16 | 927570 | 20502
SSD scaling test (1 CPU per SSD): OSD CPU mapping: OSD.0 (0), OSD.1 (10), OSD.2 (2), OSD.3 (12), ..., OSD.15 (24), Alien CPUs: 1, 11, 3, 13, ..., 15, 25 num SSD | 4k randread | 4k randwrite 4 | 936685 | 13730 8 | 1048204 | 18259 12 | 922727 | 23078 16 | 987838 | 30792
Questions: 1. Since Seastore is still under active development, are there any known unresolved performance issues that could explain this scaling behavior? 2. Are there recommended tuning parameters for improving small-block read scalability in multi-SSD configurations? 3. Regarding alien threads, are there best practices for CPU pinning or NUMA-aware placement that have shown measurable improvements? 4. Any additional guidance for maximizing IOPS with Crimson OSD + Seastore would be greatly appreciated.
My goal is to be ready to switch from BlueStore to Crimson + Seastore after it becomes stable and shows reasonable performance compared to BlueStore, so I’d like to understand the current limitations and tuning opportunities.
Thank you, Ki-taek Lee _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Ki-taek, The low write performance is a known issue and Sam's actively working on it afaik. I believe there are some significant changes to the write path coming, but for the moment it's expected that Seastore is slower than BlueStore for small random writes. Out of curiosity, would you mind sharing what benchmark you are using and how you are invoking it? Thanks, Mark On 8/11/25 23:42, Ki-taek Lee wrote:
Hello Ceph community,
I am evaluating Crimson OSD + Seastore performance for potential deployment in a distributed storage environment. With BlueStore, I have been able to achieve satisfying performance levels in my FIO tests for 4K random read/write IOPS.
However, when testing Crimson OSD + Seastore, I observed that 4K random read/write IOPS do not scale as expected when increasing the number of SSDs/OSDs. The performance plateaus beyond a certain point or is much lower than expected. (See attached test results.)
Test Environment: - Cluster: 8 clients, 1 OSD - Hardware: 40-core CPUs, 377 GiB DRAM - Image SHA (quay.io): e0543089a9e9cae97999761059eaccdf6bb22e9e - Configuration parameters: osd_memory_target = 34359738368 crimson_osd_scheduler_concurrency = 0 seastore_max_concurrent_transactions = 16 crimson_osd_obc_lru_size = 8192 seastore_cache_lru_size = 16G seastore_obj_data_write_amplification = 4 seastore_journal_batch_capacity = 1024 seastore_journal_batch_flush_size = 256M seastore_journal_iodepth_limit = 16 seastore_journal_batch_preferred_fullness = 0.8 seastore_segment_size = 128M seastore_device_size = 512G seastore_block_create = true seastore_default_object_metadata_reservation = 1073741824 rbd_cache = false rbd_cache_writethrough_until_flush = true rbd_op_threads = 16
Replication policy: - 4096 PGs, no replication (only 1 copy)
Test Results:
1 SSD test (varying number of allocated CPUs, alien threads = 26-29, 36-39): num CPU | 4k randread | 4k randwrite | Allocated CPU sets 2 | 126772 | 14830 | 0-1 4 | 107860 | 16451 | 0-3 6 | 113741 | 17019 | 0-5 8 | 132060 | 16099 | 0-7
SSD scaling test (2 CPUs per SSD): OSD CPU mapping: OSD.0 (0-1), OSD.1 (10-11), OSD.2 (2-3), OSD.3 (12-13), ..., OSD.15 (34-35), Alien threads (26-29, 36-39) num SSD | 4k randread | 4k randwrite 4 | 861273 | 22360 8 | 1022793 | 22786 12 | 1019161 | 21211 16 | 927570 | 20502
SSD scaling test (1 CPU per SSD): OSD CPU mapping: OSD.0 (0), OSD.1 (10), OSD.2 (2), OSD.3 (12), ..., OSD.15 (24), Alien CPUs: 1, 11, 3, 13, ..., 15, 25 num SSD | 4k randread | 4k randwrite 4 | 936685 | 13730 8 | 1048204 | 18259 12 | 922727 | 23078 16 | 987838 | 30792
Questions: 1. Since Seastore is still under active development, are there any known unresolved performance issues that could explain this scaling behavior? 2. Are there recommended tuning parameters for improving small-block read scalability in multi-SSD configurations? 3. Regarding alien threads, are there best practices for CPU pinning or NUMA-aware placement that have shown measurable improvements? 4. Any additional guidance for maximizing IOPS with Crimson OSD + Seastore would be greatly appreciated.
My goal is to be ready to switch from BlueStore to Crimson + Seastore after it becomes stable and shows reasonable performance compared to BlueStore, so I’d like to understand the current limitations and tuning opportunities.
Thank you, Ki-taek Lee _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
Hey Ki-taek, Thank you for sharing the information! Based on the results above, it appears that the allocated CPU might not have been configured correctly. Just to clarify: the "alienized" thread options are only applicable when deploying Crimson with Bluestore as the backend. Since Seastore is native to Crimson-OSD architecture, there's no need to set additional options for CPU allocation—both use the same CPU set. I'll update our documentation to better highlight this distinction, especially as our goal is for Seastore to become the default object store in Crimson. For re-testing the CPU scaling behavior, I would recommend using the crimson_seastar_num_threads option. Other options default values should already provide performant behavior out of the box. We really appreciate early deployment feedback - and it’s highly valuable to us. Would you be able to share the updated results in the #crimson Slack channel so we could discuss it more easily? Thanks, Matan On Tue, Aug 12, 2025 at 9:43 AM Ki-taek Lee <ktlee4311@gmail.com> wrote:
Hello Ceph community,
I am evaluating Crimson OSD + Seastore performance for potential deployment in a distributed storage environment. With BlueStore, I have been able to achieve satisfying performance levels in my FIO tests for 4K random read/write IOPS.
However, when testing Crimson OSD + Seastore, I observed that 4K random read/write IOPS do not scale as expected when increasing the number of SSDs/OSDs. The performance plateaus beyond a certain point or is much lower than expected. (See attached test results.)
Test Environment: - Cluster: 8 clients, 1 OSD - Hardware: 40-core CPUs, 377 GiB DRAM - Image SHA (quay.io): e0543089a9e9cae97999761059eaccdf6bb22e9e - Configuration parameters: osd_memory_target = 34359738368 crimson_osd_scheduler_concurrency = 0 seastore_max_concurrent_transactions = 16 crimson_osd_obc_lru_size = 8192 seastore_cache_lru_size = 16G seastore_obj_data_write_amplification = 4 seastore_journal_batch_capacity = 1024 seastore_journal_batch_flush_size = 256M seastore_journal_iodepth_limit = 16 seastore_journal_batch_preferred_fullness = 0.8 seastore_segment_size = 128M seastore_device_size = 512G seastore_block_create = true seastore_default_object_metadata_reservation = 1073741824 rbd_cache = false rbd_cache_writethrough_until_flush = true rbd_op_threads = 16
Replication policy: - 4096 PGs, no replication (only 1 copy)
Test Results:
1 SSD test (varying number of allocated CPUs, alien threads = 26-29, 36-39): num CPU | 4k randread | 4k randwrite | Allocated CPU sets 2 | 126772 | 14830 | 0-1 4 | 107860 | 16451 | 0-3 6 | 113741 | 17019 | 0-5 8 | 132060 | 16099 | 0-7
SSD scaling test (2 CPUs per SSD): OSD CPU mapping: OSD.0 (0-1), OSD.1 (10-11), OSD.2 (2-3), OSD.3 (12-13), ..., OSD.15 (34-35), Alien threads (26-29, 36-39) num SSD | 4k randread | 4k randwrite 4 | 861273 | 22360 8 | 1022793 | 22786 12 | 1019161 | 21211 16 | 927570 | 20502
SSD scaling test (1 CPU per SSD): OSD CPU mapping: OSD.0 (0), OSD.1 (10), OSD.2 (2), OSD.3 (12), ..., OSD.15 (24), Alien CPUs: 1, 11, 3, 13, ..., 15, 25 num SSD | 4k randread | 4k randwrite 4 | 936685 | 13730 8 | 1048204 | 18259 12 | 922727 | 23078 16 | 987838 | 30792
Questions: 1. Since Seastore is still under active development, are there any known unresolved performance issues that could explain this scaling behavior? 2. Are there recommended tuning parameters for improving small-block read scalability in multi-SSD configurations? 3. Regarding alien threads, are there best practices for CPU pinning or NUMA-aware placement that have shown measurable improvements? 4. Any additional guidance for maximizing IOPS with Crimson OSD + Seastore would be greatly appreciated.
My goal is to be ready to switch from BlueStore to Crimson + Seastore after it becomes stable and shows reasonable performance compared to BlueStore, so I’d like to understand the current limitations and tuning opportunities.
Thank you, Ki-taek Lee _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Ki-taek Lee
-
Mark Nelson
-
Matan Breizman