but simply on the physical parameter of IOPS-per-TB (a "figure of merit" that is widely underestimate or ignored)
hear hear!
of HDDs, and having enough IOPS-per-TB to sustain both user and admin workload.
Even with SATA SSDs I twice had to expand a cluster to meet SLO long before it was nearly full. The SNIA TCO calculator includes a multiplier for number of drives one has to provision for semi-acceptable IOPs.
A couple of legacy Ceph instances I saw in the past had 8TB and 18TB drives and as they got full the instances basically congealed (latencies in the several seconds or even dozens of second range) even under modest user workloads, and anyhow expensive admin workloads like scrubbing (never mind deep scrubbing) got behind by a year or two, and rebalancing was nearly impossible. Again not because of Ceph.
Been there, ITSY’d. Fragmentation matters with rotational media, even with op re-ordering within the drive or the driver.
But that is completely different: SSDs have *much* higher IOPS, even SATA ones, so even large SSDs have enormously better IOPS-per-TB.
And IOPS-per-yourlocalcurrency. Coarse-IU QLC is a bit of a wrinkle depending on workload...
I would like to point out that there are scale-out storage systems that have adopted their architecture for this scenario and use large HDDs very well.
That is *physically impossible* as they just do not have enough IOPS-per-TB for many "live" workloads. The illusion that they might work well happens in one of two cases:
* Either because they have not filled up yet,
I saw this with RGW on ultradense HDD toploaders.
or because they have filled up but only a minuscule subset of the data is in active use, the IOPS-per-*active*-TB of the user workload is still good enough.
Archival workloads - sure. Sometimes even backups. Even then, prudently-sourced QLC often has superior TCO compared to spinners.
* If the *active data* is mostly read-only and gets cached on a SSD tier of sufficient size, and admin workload does not matter.
And sometimes when that data active because of full backups, that process effectively flushes the cache to boot.
I have some idea of how Qumulo does things and that is very unlikely, Ceph is not fundamentally inferior to their design. Perhaps the workload's anisotropy matches particularly well that of that particular Qumulo instance:
Like a DB that’s column-oriented vs row-oriented?