Hi Ki-taek, The low write performance is a known issue and Sam's actively working on it afaik. I believe there are some significant changes to the write path coming, but for the moment it's expected that Seastore is slower than BlueStore for small random writes. Out of curiosity, would you mind sharing what benchmark you are using and how you are invoking it? Thanks, Mark On 8/11/25 23:42, Ki-taek Lee wrote:
Hello Ceph community,
I am evaluating Crimson OSD + Seastore performance for potential deployment in a distributed storage environment. With BlueStore, I have been able to achieve satisfying performance levels in my FIO tests for 4K random read/write IOPS.
However, when testing Crimson OSD + Seastore, I observed that 4K random read/write IOPS do not scale as expected when increasing the number of SSDs/OSDs. The performance plateaus beyond a certain point or is much lower than expected. (See attached test results.)
Test Environment: - Cluster: 8 clients, 1 OSD - Hardware: 40-core CPUs, 377 GiB DRAM - Image SHA (quay.io): e0543089a9e9cae97999761059eaccdf6bb22e9e - Configuration parameters: osd_memory_target = 34359738368 crimson_osd_scheduler_concurrency = 0 seastore_max_concurrent_transactions = 16 crimson_osd_obc_lru_size = 8192 seastore_cache_lru_size = 16G seastore_obj_data_write_amplification = 4 seastore_journal_batch_capacity = 1024 seastore_journal_batch_flush_size = 256M seastore_journal_iodepth_limit = 16 seastore_journal_batch_preferred_fullness = 0.8 seastore_segment_size = 128M seastore_device_size = 512G seastore_block_create = true seastore_default_object_metadata_reservation = 1073741824 rbd_cache = false rbd_cache_writethrough_until_flush = true rbd_op_threads = 16
Replication policy: - 4096 PGs, no replication (only 1 copy)
Test Results:
1 SSD test (varying number of allocated CPUs, alien threads = 26-29, 36-39): num CPU | 4k randread | 4k randwrite | Allocated CPU sets 2 | 126772 | 14830 | 0-1 4 | 107860 | 16451 | 0-3 6 | 113741 | 17019 | 0-5 8 | 132060 | 16099 | 0-7
SSD scaling test (2 CPUs per SSD): OSD CPU mapping: OSD.0 (0-1), OSD.1 (10-11), OSD.2 (2-3), OSD.3 (12-13), ..., OSD.15 (34-35), Alien threads (26-29, 36-39) num SSD | 4k randread | 4k randwrite 4 | 861273 | 22360 8 | 1022793 | 22786 12 | 1019161 | 21211 16 | 927570 | 20502
SSD scaling test (1 CPU per SSD): OSD CPU mapping: OSD.0 (0), OSD.1 (10), OSD.2 (2), OSD.3 (12), ..., OSD.15 (24), Alien CPUs: 1, 11, 3, 13, ..., 15, 25 num SSD | 4k randread | 4k randwrite 4 | 936685 | 13730 8 | 1048204 | 18259 12 | 922727 | 23078 16 | 987838 | 30792
Questions: 1. Since Seastore is still under active development, are there any known unresolved performance issues that could explain this scaling behavior? 2. Are there recommended tuning parameters for improving small-block read scalability in multi-SSD configurations? 3. Regarding alien threads, are there best practices for CPU pinning or NUMA-aware placement that have shown measurable improvements? 4. Any additional guidance for maximizing IOPS with Crimson OSD + Seastore would be greatly appreciated.
My goal is to be ready to switch from BlueStore to Crimson + Seastore after it becomes stable and shows reasonable performance compared to BlueStore, so I’d like to understand the current limitations and tuning opportunities.
Thank you, Ki-taek Lee _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/