Hi Vishnu I am Maged from the PetaSAN team in Cairo. Glad to hear you are happy with the performance, we try to give the best gateway performance out of the box, iscsi and nvmeof give near native ceph performance but are slightly slower, whereas nfs, smb and s3 are noticeably slower in that order. if you are specifically comparing iscsi performance, then i would expect that indeed we would be faster than the native gateway in ceph as the later is no longer maintained. if you mean you are suspecting the other native ceph cluster to be slower, that should not happen, both cluster should give comparable native rbd perfomance, and be a bit higher than the iscsi readings you got. If this is the case, it could be a system tuning issue. We have a performance tuning ui page which applies tuning settings to all nodes via bash scripts, you can copy and run the same tuning scripts on your native cluster, the most important parameters are for cpu wait states and frequency as well as some tcp parameters. Some other factors that can affect performance: any background deep scrubbing, pg auto scaler (best test on pool with a fixed/high pg count), ceph balancer...make sure such background services are off or tuned to a slow setting. If you do need iscsi, then consider using PetaSAN, it is pure ceph + extra layer for scaling gateway services using consul. We can also provide you support if you need to integrate our iscsi gateway in your native ceph setup, there is also a guthub page by Frederic Nass to do this. cheers /maged On 08/10/2026 10:39, Vishnu Bhaskar wrote:
Hi,
I am currently working on building an *all-flash Ceph setup for high-performance workloads*, and I am testing the performance using *8K write I/O*.
I have tested the *same hardware* with PetaSAN as well. Since PetaSAN is Ceph-based, I expected the performance to be relatively similar, but I am seeing a significant difference. Performance Observed
With *PetaSAN*, I am getting approximately *120K IOPS* from the iSCSI workload.
With my standalone *Ceph setup*, I am getting only around *30K IOPS* from the same type of workload. Test Environment
- All-flash storage - 100 GbE network - Jumbo frames enabled on both setups - FIO workload running over an *iSCSI share* - 8K write workload - Same/similar hardware used for both PetaSAN and Ceph - Comparing the performance of PetaSAN vs. my Ceph deployment
I also checked the OSD-level benchmark to make sure the OSD itself is not showing a significant performance difference. PetaSAN / OSD Benchmark
For 4K:
root@node3:~# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.054923286, "bytes_per_sec": 223730240.75799105, "iops": 54621.640810056408 }
For 8K:
root@node3:~# ceph tell osd.0 bench 12288000 8192 4194304 100 { "bytes_written": 12288000, "blocksize": 8192, "elapsed_sec": 0.028371557, "bytes_per_sec": 433109821.92482424, "iops": 52869.851309182646 }
Standalone Ceph OSD Benchmark
For 4K:
root@c100node2:~# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.060264142, "bytes_per_sec": 203902347.10385489, "iops": 49780.846460902074 }
For 8K:
root@c100node2:~# ceph tell osd.0 bench 12288000 8192 4194304 100 { "bytes_written": 12288000, "blocksize": 8192, "elapsed_sec": 0.031226269, "bytes_per_sec": 393514832.01531374, "iops": 48036.478517494354 }
Interestingly, the OSD-level benchmark results are fairly close between the two environments (~48–53K IOPS), but the *end-to-end iSCSI FIO performance differs significantly*, with PetaSAN reaching around *120K IOPS* while my Ceph setup reaches only around *30K IOPS*.
Any suggestions on how to isolate where the additional ~90K IOPS difference is coming from would be very helpful Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io