Hi, On 8/7/25 19:14, Anthony D'Atri wrote:
Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL?
As Mark wrote, there are issues with dense toploaders.
* Dense toploaders are heavy, if you have a raised floor especially you may not be able to put more than a couple in each rack safely. * Same for power, a rack that can only be populated 25% full erodes any perceived cost benefit This is true, a fully populated 60 HDD supermicro node is ~100Kg. We fill (for now) our racks with 4 nodes max, and some generic servers in
We do (supermicro 60 HDD nodes) in a 12 nodes cluster (20TB drives, 12 PiB raw) over 3 different server rooms / buildings. Our usecase is CephFS only (replicated 3, metadata pool is backed with dedicated NVMe). We didn't separate the WALL (they are on the spinners) but it's not a bottleneck for our use case. We're quite happy with this setup, even if it is not ideal. We had these supermicro servers in stock already so we use them. the remaining top half of the rack.
* If your cluster comprises a small number of these chassis, say <10, that's a very, very large blast radius. When one of those nodes halts and catches fire, IF you have sufficient spare capacity to heal, that process will be a thundering herd that will degrade performance, and could easily take MONTH to complete. During which time you are at increased risk of data unavailability or loss.
Supermicro super storage is interesting due to the possibility to swap easily all the motherboard and active components without moving the disks (tool-less removal of the back side of the server). So we've got a spare server, and in case of a chassis failure, we're able to restart (same OS/drives/OSDs...) with fresh hardware in a very short amount of time. We've considered it's acceptable in our use-case.
In most cases you would be better off with a larger number of more modestly-equipped servers, even if you stick with HDDs.
I agree (Clyso folks really know their stuff!), if we were to rebuild this from scratch I would probably buy some 24 x 3.5" HDD per 2U nodes. Still, money wise it may not be a dumb choice to use these supermicro super storage nodes, depending of your needs and setup. Cost is (was?) around €100k for 3 populated additional nodes. Not so bad I think. Anyway I'm interested in your feedback when you've made your choice. Cheers, Fabien