[ceph-users] Performance Related Issue

8 Oct 2026

      Hi,

I am currently working on building an *all-flash Ceph setup for
high-performance workloads*, and I am testing the performance using *8K
write I/O*.

I have tested the *same hardware* with PetaSAN as well. Since PetaSAN is
Ceph-based, I expected the performance to be relatively similar, but I am
seeing a significant difference.
Performance Observed

With *PetaSAN*, I am getting approximately *120K IOPS* from the iSCSI
workload.

With my standalone *Ceph setup*, I am getting only around *30K IOPS* from
the same type of workload.
Test Environment

   - All-flash storage
   - 100 GbE network
   - Jumbo frames enabled on both setups
   - FIO workload running over an *iSCSI share*
   - 8K write workload
   - Same/similar hardware used for both PetaSAN and Ceph
   - Comparing the performance of PetaSAN vs. my Ceph deployment

I also checked the OSD-level benchmark to make sure the OSD itself is not
showing a significant performance difference.
PetaSAN / OSD Benchmark

For 4K:

root@node3:~# ceph tell osd.0 bench 12288000 4096 4194304 100
{
    "bytes_written": 12288000,
    "blocksize": 4096,
    "elapsed_sec": 0.054923286,
    "bytes_per_sec": 223730240.75799105,
    "iops": 54621.640810056408
}

For 8K:

root@node3:~# ceph tell osd.0 bench 12288000 8192 4194304 100
{
    "bytes_written": 12288000,
    "blocksize": 8192,
    "elapsed_sec": 0.028371557,
    "bytes_per_sec": 433109821.92482424,
    "iops": 52869.851309182646
}

Standalone Ceph OSD Benchmark

For 4K:

root@c100node2:~# ceph tell osd.0 bench 12288000 4096 4194304 100
{
    "bytes_written": 12288000,
    "blocksize": 4096,
    "elapsed_sec": 0.060264142,
    "bytes_per_sec": 203902347.10385489,
    "iops": 49780.846460902074
}

For 8K:

root@c100node2:~# ceph tell osd.0 bench 12288000 8192 4194304 100
{
    "bytes_written": 12288000,
    "blocksize": 8192,
    "elapsed_sec": 0.031226269,
    "bytes_per_sec": 393514832.01531374,
    "iops": 48036.478517494354
}

Interestingly, the OSD-level benchmark results are fairly close between the
two environments (~48–53K IOPS), but the *end-to-end iSCSI FIO performance
differs significantly*, with PetaSAN reaching around *120K IOPS* while my
Ceph setup reaches only around *30K IOPS*.

Any suggestions on how to isolate where the additional ~90K IOPS difference
is coming from would be very helpful
Thanks and Regards
Vishnu Bhaskar
Acceleron Labs Pvt Ltd
Bangalore, India