OT: How to Build a poor man's storage with ceph
Hello List, i used to build 3 Node Cluster with spinning Rust and later with (Enterprise) SSDs. All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks and i was done. The Requirements were just 10/15TB Disk usage (30-45TB Raw). Now i was asked if i could also build a cheap 200-500TB Cluster Storage, which should also scale. Just for Data Storage such as NextCloud/OwnCloud. Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x 45k EUR = 135k EUR. Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD) How do the "big boys" do this? Just throw money at it? Would a mix of OSD SSD Metadata + Spinning Rust do the job? My experience so far is that each time i had a crash/problem it was always such a pain to wait for the spinning rust. Do you have any experience/hints on this? Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which would speed up the performance, but have a bigger impact on a dying disk. My requirements are more or less low IO Traffic but loads of disk space usage. Any hints/ideas/links are welcome. Cheers, Michael
Den tis 8 juni 2021 kl 11:39 skrev Ml Ml <mliebherr99@googlemail.com>:
Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which would speed up the performance, but have a bigger impact on a dying disk.
^^^^^^ This sounds like a very bad idea. When this 30T monster fails, you will have to wait for 30TB to rebuild elsewhere. If you think waiting for 4-6-8TB is slow, 30 will be 5x worse. -- May the most significant bit of your life be positive.
Hi Michael, El 8/6/21 a las 11:38, Ml Ml escribió:
Hello List,
i used to build 3 Node Cluster with spinning Rust and later with (Enterprise) SSDs. All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks and i was done. The Requirements were just 10/15TB Disk usage (30-45TB Raw).
Now i was asked if i could also build a cheap 200-500TB Cluster Storage, which should also scale. Just for Data Storage such as NextCloud/OwnCloud.
Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x 45k EUR = 135k EUR. Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD)
How do the "big boys" do this? Just throw money at it? Would a mix of OSD SSD Metadata + Spinning Rust do the job?
My experience so far is that each time i had a crash/problem it was always such a pain to wait for the spinning rust.
Do you have any experience/hints on this? Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which would speed up the performance, but have a bigger impact on a dying disk.
My requirements are more or less low IO Traffic but loads of disk space usage.
Any hints/ideas/links are welcome.
You don't give details about RAM and CPU, but at 24xEnterprise SSD per server I think you'll be wasting much of those SSD's performance... I suggest you consider a 4-6 server cluster, and SSD for wal + Spinning disks for storage. This will give you more redundancy for less money, and more peace of mind when a spinning disk fails. Cheers Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
Since you mention NextCloud it will probably be RWG deployment. ALso it's not clear why 3 nodes? Is rack-space a premium? Just to compare your suggestion: 3x24 (I guess 4U?) x 8TB with Replication = 576 TB raw storage + 192 TB usable Let's go 6x12 (2U) x 4TB with EC 3+2 = 288 TB raw storage + 172 TB usable. Same rack-space, a little bit less usable storage, smaller drives (means faster recovery). Put a SSD or NVME in each server for the Index pool. Plus maybe 2 NVMEs for the WAL/DB On Tue, 8 Jun 2021 at 21:39, Ml Ml <mliebherr99@googlemail.com> wrote:
Hello List,
i used to build 3 Node Cluster with spinning Rust and later with (Enterprise) SSDs. All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks and i was done. The Requirements were just 10/15TB Disk usage (30-45TB Raw).
Now i was asked if i could also build a cheap 200-500TB Cluster Storage, which should also scale. Just for Data Storage such as NextCloud/OwnCloud.
Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x 45k EUR = 135k EUR. Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD)
How do the "big boys" do this? Just throw money at it? Would a mix of OSD SSD Metadata + Spinning Rust do the job?
My experience so far is that each time i had a crash/problem it was always such a pain to wait for the spinning rust.
Do you have any experience/hints on this? Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which would speed up the performance, but have a bigger impact on a dying disk.
My requirements are more or less low IO Traffic but loads of disk space usage.
Any hints/ideas/links are welcome.
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Christian Wuerdig
-
Eneko Lacunza
-
Janne Johansson
-
Ml Ml