I am currently working on building an *all-flash Ceph setup for high-performance workloads*, and I am testing the performance using *8K write I/O*.
I have tested the *same hardware* with PetaSAN as well. Since PetaSAN is Ceph-based, I expected the performance to be relatively similar, but I am seeing a significant difference. Performance Observed
With *PetaSAN*, I am getting approximately *120K IOPS* from the iSCSI workload.
First, if you want performance, why are you using iSCSI? That said, the PetaSAN iSCSI gateway AIUI is derived from SUSE's, and is the current choice if you simply must have iSCSI for some legacy reason.
With my standalone *Ceph setup*, I am getting only around *30K IOPS* from the same type of workload. Test Environment
- All-flash storage
"All-flash" can mean a lot of things. Please be specific.
- 100 GbE network - Jumbo frames enabled on both setups - FIO workload running over an *iSCSI share*
The Ceph iSCSI gateway has not been maintained in years. It has never been performant and it has been considered deprecated for some time.
- 8K write workload - Same/similar hardware used for both PetaSAN and Ceph
Same, or similar? Similar might have a telling difference. Please give full hardware specs for both deployments. * Chassis model * RAM * CPU * BIOS / BMC settings for power and performance profiles * Specific SSDs in use * Number of nodes * Number of SSDs per node * RBD pool replica count * Number of PGs * osd_memory_target / autotune
Interestingly, the OSD-level benchmark results are fairly close between the two environments (~48–53K IOPS), but the *end-to-end iSCSI FIO performance differs significantly*, with PetaSAN reaching around *120K IOPS* while my Ceph setup reaches only around *30K IOPS*.
Simply a more-evolved gateway daemon. FredNass has a teased-out repository of the PetaSAN gateway on github for those who don't want the whole shebang. But back to my first question, why iSCSI at all?