Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ? A+ Server 1113S-WN10RT Barebone Supermicro A+ Server 1113S-WN10RT - 1U - 10x U.2 NVMe - 2x M.2 - Dual 10-Gigabit LAN - 750W Redundant Processor AMD EPYC™ 7272 Processor 12-core 2.90GHz 64MB Cache (120W) Memory 8 x 8GB PC4-25600 3200MHz DDR4 ECC RDIMM U.2/U.3 NVMe Drive 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Hard Drive 2 x 240GB Intel® SSD D3-S4610 Series 2.5" SATA 6.0Gb/s Solid State Drive Network Card 2 x Intel® 10-Gigabit Ethernet Converged Network Adapter X710-DA2 (2x SFP+) Server Management Supermicro Update Manager (SUM) (OOB Management Package), included Operating System No Windows Operating System Warranty 3 Year Depot Warranty (Return for Repair) RAID U.2 [NVMe, 10 ports] No RAID (*OS) -> 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Thanks Ignazio
On 3/10/21 5:43 PM, Ignazio Cassano wrote:
Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ?
A+ Server 1113S-WN10RT Barebone Supermicro A+ Server 1113S-WN10RT - 1U - 10x U.2 NVMe - 2x M.2 - Dual 10-Gigabit LAN - 750W Redundant Processor AMD EPYC™ 7272 Processor 12-core 2.90GHz 64MB Cache (120W) Memory 8 x 8GB PC4-25600 3200MHz DDR4 ECC RDIMM
^^ I would double that amount of RAM, especially (see below) if you plan on adding more NVMe drives.
U.2/U.3 NVMe Drive 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Hard Drive
^^ Why 5 * 8.0 TB instead of 10 * 4.0 TB? Are you planning on upgrading later? Ceph likes more OSDs better than fewer larger ones. Recovery will be faster as well, and the impact of one NVMe dying will be lower.
2 x 240GB Intel® SSD D3-S4610 Series 2.5" SATA 6.0Gb/s Solid State Drive
^^ Do you sacrifce two NVMe ports for two SATA OS disks? If so, I would advise for getting (redundant, optional) U.2 NVMe or SATADOM or similar.
Network Card 2 x Intel® 10-Gigabit Ethernet Converged Network Adapter X710-DA2 (2x SFP+) Server Management
^ Why two? One for "public" and one for "cluster"? Than most probably you won't need that, and one bond would suffice (see current Ceph best practices). If you need 40 Gb/s in one LACP trunk: perfectly fine. Gr. Stefan
On 3/10/21 8:12 PM, Stefan Kooman wrote:
On 3/10/21 5:43 PM, Ignazio Cassano wrote:
Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ?
I forgot to ask: Are you planning on only OSDs or should this be OSDs and MONs and ? In case of MONs / OSDs: give the MON a dedicated NVMe / SSD drive (not to small, in case cluster gets degraded it needs to store pglog info and storage usage will grow. Go for monstore that can hold like 800 GB or so). You want definitely want to add more memory to give MON / MGR enough. Want to add MDS?: add plenty of RAM (more is better). Gr. Stefan
Hello , non and osd. 1 small ssd is for operations system and 1 is for mon. I am agree to increase the ram. As far as nvme size it is true that more osd little disks is a better choose for performances but I should buy more servers. Ignazio Il Mer 10 Mar 2021, 20:24 Stefan Kooman <stefan@bit.nl> ha scritto:
On 3/10/21 8:12 PM, Stefan Kooman wrote:
On 3/10/21 5:43 PM, Ignazio Cassano wrote:
Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ?
I forgot to ask: Are you planning on only OSDs or should this be OSDs and MONs and ?
In case of MONs / OSDs: give the MON a dedicated NVMe / SSD drive (not to small, in case cluster gets degraded it needs to store pglog info and storage usage will grow. Go for monstore that can hold like 800 GB or so). You want definitely want to add more memory to give MON / MGR enough.
Want to add MDS?: add plenty of RAM (more is better).
Gr. Stefan
Sorry I forgot to mention I will not use cephfs Il Mer 10 Mar 2021, 20:44 Ignazio Cassano <ignaziocassano@gmail.com> ha scritto:
Hello , non and osd. 1 small ssd is for operations system and 1 is for mon. I am agree to increase the ram. As far as nvme size it is true that more osd little disks is a better choose for performances but I should buy more servers. Ignazio
Il Mer 10 Mar 2021, 20:24 Stefan Kooman <stefan@bit.nl> ha scritto:
On 3/10/21 8:12 PM, Stefan Kooman wrote:
On 3/10/21 5:43 PM, Ignazio Cassano wrote:
Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ?
I forgot to ask: Are you planning on only OSDs or should this be OSDs and MONs and ?
In case of MONs / OSDs: give the MON a dedicated NVMe / SSD drive (not to small, in case cluster gets degraded it needs to store pglog info and storage usage will grow. Go for monstore that can hold like 800 GB or so). You want definitely want to add more memory to give MON / MGR enough.
Want to add MDS?: add plenty of RAM (more is better).
Gr. Stefan
Am 10.03.21 um 20:44 schrieb Ignazio Cassano:
1 small ssd is for operations system and 1 is for mon.
Make that a RAID1 set of SSDs and be happier. ;) Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
I'm going to echo what Stefan said. I would ditch the 2x SATA drives to free up your slots. Replace with an M.2 or SATADOM. I would also recommend moving from the 2x X710-DA2 cards to 1x X710-DA4 card. It can't saturate the x8 slot, and it frees up a PCIe slot for possibly another NVMe card or something else if you need it down the line. The only other thing I would say to consider is making sure you know the endurance of the 4510 is enough for your workload long term. Reed
On Mar 10, 2021, at 1:12 PM, Stefan Kooman <stefan@bit.nl> wrote:
On 3/10/21 5:43 PM, Ignazio Cassano wrote:
Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ? A+ Server 1113S-WN10RT Barebone Supermicro A+ Server 1113S-WN10RT - 1U - 10x U.2 NVMe - 2x M.2 - Dual 10-Gigabit LAN - 750W Redundant Processor AMD EPYC™ 7272 Processor 12-core 2.90GHz 64MB Cache (120W) Memory 8 x 8GB PC4-25600 3200MHz DDR4 ECC RDIMM
^^ I would double that amount of RAM, especially (see below) if you plan on adding more NVMe drives.
U.2/U.3 NVMe Drive 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Hard Drive
^^ Why 5 * 8.0 TB instead of 10 * 4.0 TB? Are you planning on upgrading later? Ceph likes more OSDs better than fewer larger ones. Recovery will be faster as well, and the impact of one NVMe dying will be lower.
2 x 240GB Intel® SSD D3-S4610 Series 2.5" SATA 6.0Gb/s Solid State Drive
^^ Do you sacrifce two NVMe ports for two SATA OS disks? If so, I would advise for getting (redundant, optional) U.2 NVMe or SATADOM or similar.
Network Card 2 x Intel® 10-Gigabit Ethernet Converged Network Adapter X710-DA2 (2x SFP+) Server Management
^ Why two? One for "public" and one for "cluster"? Than most probably you won't need that, and one bond would suffice (see current Ceph best practices). If you need 40 Gb/s in one LACP trunk: perfectly fine.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Many thanks Ignazio Il Ven 12 Mar 2021, 00:04 Reed Dier <reed.dier@focusvq.com> ha scritto:
I'm going to echo what Stefan said.
I would ditch the 2x SATA drives to free up your slots. Replace with an M.2 or SATADOM.
I would also recommend moving from the 2x X710-DA2 cards to 1x X710-DA4 card. It can't saturate the x8 slot, and it frees up a PCIe slot for possibly another NVMe card or something else if you need it down the line.
The only other thing I would say to consider is making sure you know the endurance of the 4510 is enough for your workload long term.
Reed
On Mar 10, 2021, at 1:12 PM, Stefan Kooman <stefan@bit.nl> wrote:
On 3/10/21 5:43 PM, Ignazio Cassano wrote:
Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ? A+ Server 1113S-WN10RT Barebone Supermicro A+ Server 1113S-WN10RT - 1U - 10x U.2 NVMe - 2x M.2 - Dual 10-Gigabit LAN - 750W Redundant Processor AMD EPYC™ 7272 Processor 12-core 2.90GHz 64MB Cache (120W) Memory 8 x 8GB PC4-25600 3200MHz DDR4 ECC RDIMM
^^ I would double that amount of RAM, especially (see below) if you plan on adding more NVMe drives.
U.2/U.3 NVMe Drive 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Hard Drive
^^ Why 5 * 8.0 TB instead of 10 * 4.0 TB? Are you planning on upgrading later? Ceph likes more OSDs better than fewer larger ones. Recovery will be faster as well, and the impact of one NVMe dying will be lower.
2 x 240GB Intel® SSD D3-S4610 Series 2.5" SATA 6.0Gb/s Solid State Drive
^^ Do you sacrifce two NVMe ports for two SATA OS disks? If so, I would advise for getting (redundant, optional) U.2 NVMe or SATADOM or similar.
Network Card 2 x Intel® 10-Gigabit Ethernet Converged Network Adapter X710-DA2 (2x SFP+) Server Management
^ Why two? One for "public" and one for "cluster"? Than most probably you won't need that, and one bond would suffice (see current Ceph best practices). If you need 40 Gb/s in one LACP trunk: perfectly fine.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Am 10.03.21 um 17:43 schrieb Ignazio Cassano:
5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Hard Drive
2 x Intel® 10-Gigabit Ethernet Converged Network Adapter X710-DA2 (2x SFP+)
Have you calculated the throughput of 8 NVMe drives against 2x 10G bonded interfaces in the cluster network? I would assume that you would want to go at least 2x 25G here. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Yes, I noted that more bandwidth is required with this kind of servers. I must reconsider my network infrastructure. Many tanks Ignazio Il giorno ven 12 mar 2021 alle ore 09:26 Robert Sander < r.sander@heinlein-support.de> ha scritto:
Hi,
Am 10.03.21 um 17:43 schrieb Ignazio Cassano:
5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Hard Drive
2 x Intel® 10-Gigabit Ethernet Converged Network Adapter X710-DA2 (2x SFP+)
Have you calculated the throughput of 8 NVMe drives against 2x 10G bonded interfaces in the cluster network? I would assume that you would want to go at least 2x 25G here.
Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Ignazio Cassano
-
Reed Dier
-
Robert Sander
-
Stefan Kooman