60/90 bays + 6 NVME Supermicro
Hi Ceph , Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL? Newest models support 144cores in single socket and several TB of ram without issues. But as far as we understood in the technical notes it use SAS Expanders for connect all disk, from 2 to 4 SAS Expanders to connect the whole chasis. We're looking for next configuration : Intel Xeon 96cores @ 2.4 1TB RAM 2 SSD for OS. 4 NVME x 30TB NVME KIOXIA 90 HDD * 22 TB HGST 4x25 Gbps or 2x100 Gbps Main use RGW. Regards [cid:image001.jpg@01DC07CB.6CB5C6C0] MANUEL RIOS FERNANDEZ CEO - EasyDataHost Phone: 677677179 Web: www.easydatahost.com<http://www.easydatahost.com/> Email: mriosfer@easydatahost.com<mailto:mriosfer@easydatahost.com> [T?tulo: LinkedIn - Descripci?n: image of LinkedIn icon] <https://es.linkedin.com/in/manuel-rios-fernandez-14880949?original_referer=https%3A%2F%2Fwww.google.com%2F> ADVERTENCIA LEGAL: Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado. Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. [Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Flor ecoTECH]No imprimas si no es necesario. Protejamos el Medio Ambiente.
Personally I prefer the 24 HDD + 4NVMe 2U chassis unless extreme density and absolute lowest cost are the highest priorities. The denser options can work, but definitely have trade-offs and are only really appropriate if you are deploying a very large cluster. Mark On 8/7/25 09:48, Manuel Rios - EDH wrote:
Hi Ceph ,
Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL?
Newest models support 144cores in single socket and several TB of ram without issues.
But as far as we understood in the technical notes it use SAS Expanders for connect all disk, from 2 to 4 SAS Expanders to connect the whole chasis.
We’re looking for next configuration :
Intel Xeon 96cores @ 2.4
1TB RAM
2 SSD for OS.
4 NVME x 30TB NVME KIOXIA
90 HDD * 22 TB HGST
4x25 Gbps or 2x100 Gbps
Main use RGW.
Regards
*MANUEL RIOS FERNANDEZ*
CEO – EasyDataHost
*Phone: * 677677179
*Web: *_www.easydatahost.com <http://www.easydatahost.com/>_
*Email: *_mriosfer@easydatahost.com <mailto:mriosfer@easydatahost.com>_
Título: LinkedIn - Descripción: image of LinkedIn icon <https://es.linkedin.com/in/manuel-rios-fernandez-14880949?original_referer=https%3A%2F%2Fwww.google.com%2F>
*_ADVERTENCIA LEGAL:_*
Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado.
Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros.
*_CONFIDENTIALITY NOTICE:_*
This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
*Descripción: Descripción: Descripción: Descripción: Descripción: Flor ecoTECHNo imprimas si no es necesario. Protejamos el Medio Ambiente.*
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w:https://clyso.com | e:mark.nelson@clyso.com We are hiring:https://www.clyso.com/jobs/
Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL?
Be aware that offloading WAL+DB is not a panacea. There are certain benefits, but it is not likely to improve throughput.
Newest models support 144cores in single socket and several TB of ram without issues.
Cores, or hyperthreads? I suspect this is the Epic 9825 with 144c / 288 threads. With a list price of USD 13,000.
But as far as we understood in the technical notes it use SAS Expanders for connect all disk, from 2 to 4 SAS Expanders to connect the whole chasis.
As I understand it, anything with more than 8 drives connecting to a single HBA is going to use expanders. If you're using rotational media, expanders are the least of your concerns, honestly.
We’re looking for next configuration :
Intel Xeon 96cores @ 2.4
That doesn't agree with what you wrote about re 144 cores. Assuming that here you mean 96c / 192t.
1TB RAM
2 SSD for OS.
4 NVME x 30TB NVME KIOXIA
Be careful that these are TLC, not one of the QLC models. QLC is not a good fit for this application. It however would be a great alternative to HDDs, assuming that this cluster is for RGW service.
90 HDD * 22 TB HGST
This gives you 2 threads / vcores per OSD, with a handful left over for mon, mgr, rgw, etc. This is on the light side, especially since you likely would want at least two and perhaps even more RGW daemons per node as well. Under expansion or component loss you may saturate CPU, leading to slow requests and all manner of unpleasantness.
4x25 Gbps or 2x100 Gbps
Main use RGW.
So it is. As Mark wrote, there are issues with dense toploaders. * If your RGW use-case includes a significant number of hot, small objects, you will want to place those on a replicated SSD pool, the HDDs will be hotspots. * 60 or 90 drives will often bottleneck a single HBA. If you do this, don't waste your money and karma on a RAID HBA. * Dense toploaders are heavy, if you have a raised floor especially you may not be able to put more than a couple in each rack safely. * Same for power, a rack that can only be populated 25% full erodes any perceived cost benefit * If your cluster comprises a small number of these chassis, say <10, that's a very, very large blast radius. When one of those nodes halts and catches fire, IF you have sufficient spare capacity to heal, that process will be a thundering herd that will degrade performance, and could easily take MONTH to complete. During which time you are at increased risk of data unavailability or loss. * Dense systems are prone to network saturation In most cases you would be better off with a larger number of more modestly-equipped servers, even if you stick with HDDs. But for most RGW workloads I would really suggest more modestly equipped 1U servers with 1-2 TLC SSDs for index/meta/log/etc and the default storage class, and the balance of drive bays populated with large QLC SSDs for bulk data.
Regards
*MANUEL RIOS FERNANDEZ*
CEO – EasyDataHost
*Phone: * 677677179
*Web: *_www.easydatahost.com <http://www.easydatahost.com/>_
*Email: *_mriosfer@easydatahost.com <mailto:mriosfer@easydatahost.com>_
Título: LinkedIn - Descripción: image of LinkedIn icon <https://es.linkedin.com/in/manuel-rios-fernandez-14880949?original_referer=https%3A%2F%2Fwww.google.com%2F>
*_ADVERTENCIA LEGAL:_*
Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado.
Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros.
*_CONFIDENTIALITY NOTICE:_*
This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
*Descripción: Descripción: Descripción: Descripción: Descripción: Flor ecoTECHNo imprimas si no es necesario. Protejamos el Medio Ambiente.*
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development
Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w:https://clyso.com | e:mark.nelson@clyso.com
We are hiring:https://www.clyso.com/jobs/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 8/7/25 19:14, Anthony D'Atri wrote:
Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL?
As Mark wrote, there are issues with dense toploaders.
* Dense toploaders are heavy, if you have a raised floor especially you may not be able to put more than a couple in each rack safely. * Same for power, a rack that can only be populated 25% full erodes any perceived cost benefit This is true, a fully populated 60 HDD supermicro node is ~100Kg. We fill (for now) our racks with 4 nodes max, and some generic servers in
We do (supermicro 60 HDD nodes) in a 12 nodes cluster (20TB drives, 12 PiB raw) over 3 different server rooms / buildings. Our usecase is CephFS only (replicated 3, metadata pool is backed with dedicated NVMe). We didn't separate the WALL (they are on the spinners) but it's not a bottleneck for our use case. We're quite happy with this setup, even if it is not ideal. We had these supermicro servers in stock already so we use them. the remaining top half of the rack.
* If your cluster comprises a small number of these chassis, say <10, that's a very, very large blast radius. When one of those nodes halts and catches fire, IF you have sufficient spare capacity to heal, that process will be a thundering herd that will degrade performance, and could easily take MONTH to complete. During which time you are at increased risk of data unavailability or loss.
Supermicro super storage is interesting due to the possibility to swap easily all the motherboard and active components without moving the disks (tool-less removal of the back side of the server). So we've got a spare server, and in case of a chassis failure, we're able to restart (same OS/drives/OSDs...) with fresh hardware in a very short amount of time. We've considered it's acceptable in our use-case.
In most cases you would be better off with a larger number of more modestly-equipped servers, even if you stick with HDDs.
I agree (Clyso folks really know their stuff!), if we were to rebuild this from scratch I would probably buy some 24 x 3.5" HDD per 2U nodes. Still, money wise it may not be a dumb choice to use these supermicro super storage nodes, depending of your needs and setup. Cost is (was?) around €100k for 3 populated additional nodes. Not so bad I think. Anyway I'm interested in your feedback when you've made your choice. Cheers, Fabien
So my question is what is the use case. S3 is not a use case it’s a protocol to access the storage. How many nodes are you looking to deploy? What model of Kioxia drive are you looking to use as a brief look said all the 30TB drives where read intensive. If you go with 90 drive chassis then you are looking at a ratio of 1:22 for NVME to HDD which could be a big issue made worse by the use of read intensive drives that may only do 80K write IOPS. Back to the first question. Last one is where are you going to put the S3 bucket metadata? Comes back too the first question of what is the use case ? Like Mark one of the configurations I use a lot is the 2U24. Whilst the 90 drive chassis looks good on paper it typically needs a deeper rack is very heavy and also consumes a lot of power. You are now at close to 2PB per node which is a massive amount of data to rebuild should a node be lost. Unless you are wanting just a very deep archive and a cluster of 30+PB then IMHO they don’t make sense.
On 7 Aug 2025, at 17:54, Mark Nelson <mark.nelson@clyso.com> wrote:
Personally I prefer the 24 HDD + 4NVMe 2U chassis unless extreme density and absolute lowest cost are the highest priorities. The denser options can work, but definitely have trade-offs and are only really appropriate if you are deploying a very large cluster.
Mark
On 8/7/25 09:48, Manuel Rios - EDH wrote:
Hi Ceph ,
Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL?
Newest models support 144cores in single socket and several TB of ram without issues.
But as far as we understood in the technical notes it use SAS Expanders for connect all disk, from 2 to 4 SAS Expanders to connect the whole chasis.
We’re looking for next configuration :
Intel Xeon 96cores @ 2.4
1TB RAM
2 SSD for OS.
4 NVME x 30TB NVME KIOXIA
90 HDD * 22 TB HGST
4x25 Gbps or 2x100 Gbps
Main use RGW.
Regards
*MANUEL RIOS FERNANDEZ*
CEO – EasyDataHost
*Phone: * 677677179
*Web: *_www.easydatahost.com <http://www.easydatahost.com/>_
*Email: *_mriosfer@easydatahost.com <mailto:mriosfer@easydatahost.com>_
Título: LinkedIn - Descripción: image of LinkedIn icon <https://es.linkedin.com/in/manuel-rios-fernandez-14880949?original_referer=https%3A%2F%2Fwww.google.com%2F>
*_ADVERTENCIA LEGAL:_*
Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado.
Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros.
*_CONFIDENTIALITY NOTICE:_*
This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
*Descripción: Descripción: Descripción: Descripción: Descripción: Flor ecoTECHNo imprimas si no es necesario. Protejamos el Medio Ambiente.*
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development
Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w:https://clyso.com | e:mark.nelson@clyso.com
We are hiring:https://www.clyso.com/jobs/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So my question is what is the use case. S3 is not a use case it’s a protocol to access the storage.
Indeed. Factors include: * read vs writes, many RGW deployments are very read-heavy * distribution of object sizes. Smaller objects (per-second checkpoints) are much more metadata-intensive than larger objects (pirated TV shows dubbed into Portuguese).
How many nodes are you looking to deploy?
I suggest that a chassis not hold more than 10% of the cluster capacity with slow media, for blast radius.
What model of Kioxia drive are you looking to use as a brief look said all the 30TB drives where read intensive.
Be very sure that those are TLC, 4KB IU drives, as I believe they have made 30TB QLC drives as well, which most likely would not be ideal for this purpose. But would be for large-object bucket data pools.
If you go with 90 drive chassis then you are looking at a ratio of 1:22 for NVME to HDD
Indeed, conventional wisdom has been at most a 10:1 ratio, though with modern PCIe 4+ servers and drives, a somewhat higher ratio is probably tolerable. But those would also want to have OSDs for the meta, log, etc pools.
which could be a big issue made worse by the use of read intensive drives that may only do 80K write IOPS. Back to the first question.
Remember that mixed-use drives are usually the same hardware with more overprovisioning (and profit margin).
Last one is where are you going to put the S3 bucket metadata?
The index pool is mainly omaps (so far), which can live on hybrid OSDs with SSD offload.
Comes back too the first question of what is the use case ?
Like Mark one of the configurations I use a lot is the 2U24. Whilst the 90 drive chassis looks good on paper it typically needs a deeper rack is very heavy and also consumes a lot of power. You are now at close to 2PB per node which is a massive amount of data to rebuild should a node be lost. Unless you are wanting just a very deep archive and a cluster of 30+PB then IMHO they don’t make sense.
Agreed.
On 7 Aug 2025, at 17:54, Mark Nelson <mark.nelson@clyso.com> wrote:
Personally I prefer the 24 HDD + 4NVMe 2U chassis unless extreme density and absolute lowest cost are the highest priorities. The denser options can work, but definitely have trade-offs and are only really appropriate if you are deploying a very large cluster.
Mark
On 8/7/25 09:48, Manuel Rios - EDH wrote:
Hi Ceph ,
Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL?
Newest models support 144cores in single socket and several TB of ram without issues.
But as far as we understood in the technical notes it use SAS Expanders for connect all disk, from 2 to 4 SAS Expanders to connect the whole chasis.
We’re looking for next configuration :
Intel Xeon 96cores @ 2.4
1TB RAM
2 SSD for OS.
4 NVME x 30TB NVME KIOXIA
90 HDD * 22 TB HGST
4x25 Gbps or 2x100 Gbps
Main use RGW.
Regards
*MANUEL RIOS FERNANDEZ*
CEO – EasyDataHost
*Phone: * 677677179
*Web: *_www.easydatahost.com <http://www.easydatahost.com/>_
*Email: *_mriosfer@easydatahost.com <mailto:mriosfer@easydatahost.com>_
Título: LinkedIn - Descripción: image of LinkedIn icon <https://es.linkedin.com/in/manuel-rios-fernandez-14880949?original_referer=https%3A%2F%2Fwww.google.com%2F>
*_ADVERTENCIA LEGAL:_*
Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado.
Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros.
*_CONFIDENTIALITY NOTICE:_*
This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
*Descripción: Descripción: Descripción: Descripción: Descripción: Flor ecoTECHNo imprimas si no es necesario. Protejamos el Medio Ambiente.*
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development
Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w:https://clyso.com | e:mark.nelson@clyso.com
We are hiring:https://www.clyso.com/jobs/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks all for response. The CPU will be Intel Xeon 6740E @ 2.4Ghz / 96 cores no HT. // DRR at 6400 Mhz Actually we manage a ceph cluster full NVME for RGW with 30TB NVME drive (Solidigm), it works like champ, but the price per TB/ disk increased during the last months. And the other cluster we managed got several Dell R740XD with 12 disk , 2 NVME , 2 SSD for OS, that works quite good, but we we're looking for something more dense. Maybe we should move to Dell R760XD2 with 24 drives, 2NVME and 2 SSD of OS. We got this "60/90node" from IBM Hardware recommendation for large clusters +2PB as they recommend use Dell DSS7000 The main target of this cluster will be 4-5PB for just S3 Storage and main customers with usage for backups storage. The power density per rack at least for us is not a problem, DCs are looking to hold racks of 7-15KW per rack, and a single rack can hold about 1000-1200KG. We know (I also know) that we best cluster for ceph is always a flash/ssd/nvme one for RGW as small writes/deletes don’t penalize the performance , but currently the price/TB didn’t drop even in the 100TB NVME/SSD drives. (10K $ per drive) Just a note, our current bandwitch usage for RGW is about 10-14 Gbps on peaks. Thanks all for amazing notes and time to response my question. Regards, Manuel -----Mensaje original----- De: Anthony D'Atri <anthony.datri@gmail.com> Enviado el: viernes, 8 de agosto de 2025 2:43 Para: darren@soothill.com CC: ceph-users@ceph.io; Mark Nelson <mark.nelson@clyso.com> Asunto: [ceph-users] Re: 60/90 bays + 6 NVME Supermicro
So my question is what is the use case. S3 is not a use case it’s a protocol to access the storage.
Indeed. Factors include: * read vs writes, many RGW deployments are very read-heavy * distribution of object sizes. Smaller objects (per-second checkpoints) are much more metadata-intensive than larger objects (pirated TV shows dubbed into Portuguese).
How many nodes are you looking to deploy?
I suggest that a chassis not hold more than 10% of the cluster capacity with slow media, for blast radius.
What model of Kioxia drive are you looking to use as a brief look said all the 30TB drives where read intensive.
Be very sure that those are TLC, 4KB IU drives, as I believe they have made 30TB QLC drives as well, which most likely would not be ideal for this purpose. But would be for large-object bucket data pools.
If you go with 90 drive chassis then you are looking at a ratio of 1:22 for NVME to HDD
Indeed, conventional wisdom has been at most a 10:1 ratio, though with modern PCIe 4+ servers and drives, a somewhat higher ratio is probably tolerable. But those would also want to have OSDs for the meta, log, etc pools.
which could be a big issue made worse by the use of read intensive drives that may only do 80K write IOPS. Back to the first question.
Remember that mixed-use drives are usually the same hardware with more overprovisioning (and profit margin).
Last one is where are you going to put the S3 bucket metadata?
The index pool is mainly omaps (so far), which can live on hybrid OSDs with SSD offload.
Comes back too the first question of what is the use case ?
Like Mark one of the configurations I use a lot is the 2U24. Whilst the 90 drive chassis looks good on paper it typically needs a deeper rack is very heavy and also consumes a lot of power. You are now at close to 2PB per node which is a massive amount of data to rebuild should a node be lost. Unless you are wanting just a very deep archive and a cluster of 30+PB then IMHO they don’t make sense.
Agreed.
On 7 Aug 2025, at 17:54, Mark Nelson <mark.nelson@clyso.com> wrote:
Personally I prefer the 24 HDD + 4NVMe 2U chassis unless extreme density and absolute lowest cost are the highest priorities. The denser options can work, but definitely have trade-offs and are only really appropriate if you are deploying a very large cluster.
Mark
On 8/7/25 09:48, Manuel Rios - EDH wrote:
Hi Ceph ,
Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL?
Newest models support 144cores in single socket and several TB of ram without issues.
But as far as we understood in the technical notes it use SAS Expanders for connect all disk, from 2 to 4 SAS Expanders to connect the whole chasis.
We’re looking for next configuration :
Intel Xeon 96cores @ 2.4
1TB RAM
2 SSD for OS.
4 NVME x 30TB NVME KIOXIA
90 HDD * 22 TB HGST
4x25 Gbps or 2x100 Gbps
Main use RGW.
Regards
*MANUEL RIOS FERNANDEZ*
CEO – EasyDataHost
*Phone: * 677677179
*Web: *_www.easydatahost.com <http://www.easydatahost.com/>_
*Email: *_mriosfer@easydatahost.com <mailto:mriosfer@easydatahost.com>_
Título: LinkedIn - Descripción: image of LinkedIn icon <https://es.linkedin.com/in/manuel-rios-fernandez-14880949?original_referer=https%3A%2F%2Fwww.google.com%2F>
*_ADVERTENCIA LEGAL:_*
Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado.
Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros.
*_CONFIDENTIALITY NOTICE:_*
This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
*Descripción: Descripción: Descripción: Descripción: Descripción: Flor ecoTECHNo imprimas si no es necesario. Protejamos el Medio Ambiente.*
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development
Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w:https://clyso.com | e:mark.nelson@clyso.com
We are hiring:https://www.clyso.com/jobs/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
darren＠soothill.com
-
Fabien Sirjean
-
Manuel Rios - EDH
-
Mark Nelson