8 Jun
2021
8 Jun
'21
2:08 p.m.
Hello List,
i used to build 3 Node Cluster with spinning Rust and later with
(Enterprise) SSDs.
All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks
and i was done.
The Requirements were just 10/15TB Disk usage (30-45TB Raw).
Now i was asked if i could also build a cheap 200-500TB Cluster
Storage, which should also scale. Just for Data Storage such as
NextCloud/OwnCloud.
Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x
45k EUR = 135k EUR.
Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD)
How do the "big boys" do this? Just throw money at it?
Would a mix of OSD SSD Metadata + Spinning Rust do the job?
My experience so far is that each time i had a crash/problem it was
always such a pain to wait for the spinning rust.
Do you have any experience/hints on this?
Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which
would speed up the performance, but have a bigger impact on a dying
disk.
My requirements are more or less low IO Traffic but loads of disk space usage.
Any hints/ideas/links are welcome.
Cheers,
Michael
8 Jun
8 Jun
2:13 p.m.
Den tis 8 juni 2021 kl 11:39 skrev Ml Ml <mliebherr99(a)googlemail.com>om>:
Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping
Disk => which
would speed up the performance, but have a bigger impact on a dying
disk.
^^^^^^
This sounds like a very bad idea.
When this 30T monster fails, you will have to wait for 30TB to rebuild
elsewhere.
If you think waiting for 4-6-8TB is slow, 30 will be 5x worse.
--
May the most significant bit of your life be positive.
2:36 p.m.
Hi Michael,
El 8/6/21 a las 11:38, Ml Ml escribió:
Hello List,
i used to build 3 Node Cluster with spinning Rust and later with
(Enterprise) SSDs.
All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks
and i was done.
The Requirements were just 10/15TB Disk usage (30-45TB Raw).
Now i was asked if i could also build a cheap 200-500TB Cluster
Storage, which should also scale. Just for Data Storage such as
NextCloud/OwnCloud.
Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x
45k EUR = 135k EUR.
Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD)
How do the "big boys" do this? Just throw money at it?
Would a mix of OSD SSD Metadata + Spinning Rust do the job?
My experience so far is that each time i had a crash/problem it was
always such a pain to wait for the spinning rust.
Do you have any experience/hints on this?
Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which
would speed up the performance, but have a bigger impact on a dying
disk.
My requirements are more or less low IO Traffic but loads of disk space usage.
Any hints/ideas/links are welcome.
You don't give details about RAM and CPU, but at 24xEnterprise SSD per
server I think you'll be wasting much of those SSD's performance...
I suggest you consider a 4-6 server cluster, and SSD for wal + Spinning
disks for storage. This will give you more redundancy for less money,
and more peace of mind when a spinning disk fails.
Cheers
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
2:55 p.m.
Since you mention NextCloud it will probably be RWG deployment. ALso it's
not clear why 3 nodes? Is rack-space a premium?
Just to compare your suggestion:
3x24 (I guess 4U?) x 8TB with Replication = 576 TB raw storage + 192 TB
usable
Let's go 6x12 (2U) x 4TB with EC 3+2 = 288 TB raw storage + 172 TB usable.
Same rack-space, a little bit less usable storage, smaller drives (means
faster recovery). Put a SSD or NVME in each server for the Index pool. Plus
maybe 2 NVMEs for the WAL/DB
On Tue, 8 Jun 2021 at 21:39, Ml Ml <mliebherr99(a)googlemail.com> wrote:
Hello List,
i used to build 3 Node Cluster with spinning Rust and later with
(Enterprise) SSDs.
All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks
and i was done.
The Requirements were just 10/15TB Disk usage (30-45TB Raw).
Now i was asked if i could also build a cheap 200-500TB Cluster
Storage, which should also scale. Just for Data Storage such as
NextCloud/OwnCloud.
Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x
45k EUR = 135k EUR.
Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD)
How do the "big boys" do this? Just throw money at it?
Would a mix of OSD SSD Metadata + Spinning Rust do the job?
My experience so far is that each time i had a crash/problem it was
always such a pain to wait for the spinning rust.
Do you have any experience/hints on this?
Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which
would speed up the performance, but have a bigger impact on a dying
disk.
My requirements are more or less low IO Traffic but loads of disk space
usage.
Any hints/ideas/links are welcome.
Cheers,
Michael
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1072
days inactive
1072
days old
3 comments
4 participants
participants (4)
-
Christian Wuerdig -
Eneko Lacunza -
Janne Johansson -
Ml Ml