Hi Ceph users ! After years of using Ceph, we plan to build soon a new cluster bigger than what we've done in the past. As the project is still in reflection, I'd like to have your thoughts on our planned design : any feedback is welcome :) ## Requirements * ~1 PB usable space for file storage, extensible in the future * The files are mostly "hot" data, no cold storage * Purpose : storage for big files being essentially used on windows workstations (10G access) * Performance is better :) ## Global design * 8+3 Erasure Coded pool * ZFS on RBD, exposed via samba shares (cluster with failover) ## Hardware * 1 rack (multi-site would be better, of course...) * OSD nodes : 14 x supermicro servers * 24 usable bays in 2U rackspace * 16 x 10 TB nearline SAS HDD (8 bays for future needs) * 2 x Xeon Silver 4212 (12C/24T) * 128 GB RAM * 4 x 40G QSFP+ * Networking : 2 x Cisco N3K 3132Q or 3164Q * 2 x 40G per server for ceph network (LACP/VPC for HA) * 2 x 40G per server for public network (LACP/VPC for HA) * QSFP+ DAC cables ## Sizing If we've done the maths well, we expect to have : * 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB * ~1 PB of usable space if we want to keep the OSD use under 66% to allow loosing nodes without problem, extensible to 1.6 PB (same condition) ## Reflections * We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ? * We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ? * Could a 2x10G network be enough ? * ZFS on Ceph ? Any thoughts ? * What about CephFS ? We'd like to use RBD diff for backups but it looks impossible to use snapshot diff with Cephfs ? Thanks for reading, and sharing your experiences ! F.
After years of using Ceph, we plan to build soon a new cluster bigger than what we've done in the past. As the project is still in reflection, I'd like to have your thoughts on our planned design : any feedback is welcome :)
## Requirements
* ~1 PB usable space for file storage, extensible in the future * The files are mostly "hot" data, no cold storage * Purpose : storage for big files being essentially used on windows workstations (10G access) * Performance is better :)
## Global design
* 8+3 Erasure Coded pool * ZFS on RBD, exposed via samba shares (cluster with failover)
## Hardware
* 1 rack (multi-site would be better, of course...)
* OSD nodes : 14 x supermicro servers * 24 usable bays in 2U rackspace * 16 x 10 TB nearline SAS HDD (8 bays for future needs) * 2 x Xeon Silver 4212 (12C/24T) * 128 GB RAM * 4 x 40G QSFP+
* Networking : 2 x Cisco N3K 3132Q or 3164Q * 2 x 40G per server for ceph network (LACP/VPC for HA) * 2 x 40G per server for public network (LACP/VPC for HA) * QSFP+ DAC cables
## Sizing
If we've done the maths well, we expect to have :
* 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB * ~1 PB of usable space if we want to keep the OSD use under 66% to allow loosing nodes without problem, extensible to 1.6 PB (same condition)
## Reflections
* We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ?
* We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ?
* Could a 2x10G network be enough ?
I would say yes, those slow disks will not deliver more anyway. This is going to be a relative "slow" setup with limited amount of read-caching - with 16 drives / 128GB memory it'll be a few GB per OSD for read caching - menaning that all read-and-write will hit the slow drives underneath. And that in a "double slow" fashion - where one write will hit 8 + 3 OSD's and wait for sync-ack back to the master - same with reads that will hit 8+3 OSD's before returning to the client. Workload depending - this may just work for you - but it is definately not fast. Suggestions for improvements: * Hardware raid with Battery Backed write-cache - will allow OSD to ack writes before hitting spinning rust. * More memory for OSD-level read-caching. * 3x replication instead of EC .. (we have all above in a "similar" setup ~1PB - 10 OSD - hosts). SSD-tiering pool (havent been there - but would like to test it out). -- Jesper
Rather than a cache tier, I would put an NVMe device in each OSD box for Bluestore's DB and WAL. This will significantly improve small IOs. 14*16 HDDs / 11 chunks = 20 HDD's worth of write IOPs. If you expect these files to be written sequentially, this is probably ok. Mons and mgr on OSD nodes should be ok as far as I know (we are doing it). However, if you use CephFS, you will want the fastest CPUs you can get for the MDS, as it only scales to 2-3 cores in our tests. I would avoid the complexity of CephFS if you do not need POSIX semantics. On Tue, Dec 3, 2019 at 3:07 PM Fabien Sirjean <fsirjean@eddie.fdn.fr> wrote:
Hi Ceph users !
After years of using Ceph, we plan to build soon a new cluster bigger than what we've done in the past. As the project is still in reflection, I'd like to have your thoughts on our planned design : any feedback is welcome :)
## Requirements
* ~1 PB usable space for file storage, extensible in the future * The files are mostly "hot" data, no cold storage * Purpose : storage for big files being essentially used on windows workstations (10G access) * Performance is better :)
## Global design
* 8+3 Erasure Coded pool * ZFS on RBD, exposed via samba shares (cluster with failover)
## Hardware
* 1 rack (multi-site would be better, of course...)
* OSD nodes : 14 x supermicro servers * 24 usable bays in 2U rackspace * 16 x 10 TB nearline SAS HDD (8 bays for future needs) * 2 x Xeon Silver 4212 (12C/24T) * 128 GB RAM * 4 x 40G QSFP+
* Networking : 2 x Cisco N3K 3132Q or 3164Q * 2 x 40G per server for ceph network (LACP/VPC for HA) * 2 x 40G per server for public network (LACP/VPC for HA) * QSFP+ DAC cables
## Sizing
If we've done the maths well, we expect to have :
* 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB * ~1 PB of usable space if we want to keep the OSD use under 66% to allow loosing nodes without problem, extensible to 1.6 PB (same condition)
## Reflections
* We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ?
* We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ?
* Could a 2x10G network be enough ?
* ZFS on Ceph ? Any thoughts ?
* What about CephFS ? We'd like to use RBD diff for backups but it looks impossible to use snapshot diff with Cephfs ?
Thanks for reading, and sharing your experiences !
F.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, You will get slow performance: EC is slow, HDD are slow too With 400 iops per device, you get 89600 iops for the whole cluster, raw With 8+3EC, each logical write is mapped to 11 physical writes You get only 8145 write IOPS (is my math correct ?), which I find very low for a PB storage So, use flash-based storage, or at least not EC (which is for cold storage)
* We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ? Nothing troublesome here
* We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ? Cache tiering is not recommended You will not be able to manage this cache (managing which content needs to be cached), thus degrading performances
* Could a 2x10G network be enough ? If you buy hardware, get 25Gbps
Thanks for reading, and sharing your experiences !
F.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
## Requirements
* ~1 PB usable space for file storage, extensible in the future * The files are mostly "hot" data, no cold storage * Purpose : storage for big files being essentially used on windows workstations (10G access) * Performance is better :)
## Global design
* 8+3 Erasure Coded pool
EC performance for RBD is going to be mediocre at best, esp. on spinners.
* ZFS on RBD, exposed via samba shares (cluster with failover)
Why ZFS? Mind you I like ZFS, but layering it on top of RBD is more overhead and complexity.
* 128 GB RAM
Nowhere near enough. You’re going to want 256 at the very least.
* Networking : 2 x Cisco N3K 3132Q or 3164Q * 2 x 40G per server for ceph network (LACP/VPC for HA) * 2 x 40G per server for public network (LACP/VPC for HA)
Don’t bother with a replication network.
* We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ?
Contention for resources can lead to a vicious circle. Failure/maint of mon/mgr/OSD at the same time can be ugly. Put your mons on something cheap, 5 of them or 3 if you must.
* We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ?
Cache tiering is deprecated at best. Not a good idea to invest in it. If you’re going to use SSDs, there are better ways.
* Could a 2x10G network be enough ?
Yes.
* ZFS on Ceph ? Any thoughts ?
ZFS is great, but unless you have a specific need, it sounds like a lot of overhead and complexity.
* Hardware raid with Battery Backed write-cache - will allow OSD to ack writes before hitting spinning rust.
Disagree. See my litany from a few months ago. Use a plain, IT-mode HBA. Take the $$ you save and put it toward building your cluster out of SSDs instead of HDDs. That way you don’t have to mess with the management hassles of maintaining and allocating external WAL+DB partitions too.
3x replication instead of EC
This. The performance of EC RBD vols will likely disappoint you, esp on spinners. Having suffered 3R RBD on LFF spinners, I predict that you would also be unhappy unless your use-case is only archival / backups or some other cold, latency-tolerance workload.
* Hardware raid with Battery Backed write-cache - will allow OSD to ack writes before hitting spinning rust.
Disagree. See my litany from a few months ago. Use a plain, IT-mode HBA. Take the $$ you save and put it toward building your cluster out of SSDs instead of HDDs. That way you donât have to mess with the management hassles of maintaining and allocating external WAL+DB partitions too.
These things are not really comparable - are they? Cost of SSD vs. HDD is still in the 6:1 favor of HHD's. Yes SSD would be great but not nessesarily affordable - or have I missed something that makes the math work ? -- Jesper
If k=8,m=3 is too slow on HDDs, so you need replica 3 and SSD DB/WAL, vs EC 8,3 on SSD, then that's (1/3) / (8/11) = 0.45 multiplier on the SSD space required vs HDDs. That brings it from 6x to 2.7x. Then you have the benefit of not needing separate SSDs for DB/WAL both in hardware cost and complexity. SSDs will still be more expensive; but perhaps justifiable given the performance, rebuild times, etc. If you only need cold-storage, then EC 8,3 on HDDs will be cheap. But is that fast enough? On Tue, Dec 3, 2019 at 3:47 PM <jesper@krogh.cc> wrote:
* Hardware raid with Battery Backed write-cache - will allow OSD to ack writes before hitting spinning rust.
Disagree. See my litany from a few months ago. Use a plain, IT-mode HBA. Take the $$ you save and put it toward building your cluster out of SSDs instead of HDDs. That way you don’t have to mess with the management hassles of maintaining and allocating external WAL+DB partitions too.
These things are not really comparable - are they? Cost of SSD vs. HDD is still in the 6:1 favor of HHD's. Yes SSD would be great but not nessesarily affordable - or have I missed something that makes the math work ?
-- Jesper _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If k=8,m=3 is too slow on HDDs, so you need replica 3 and SSD DB/WAL, vs EC 8,3 on SSD, then that's (1/3) / (8/11) = 0.45 multiplier on the SSD space required vs HDDs. That brings it from 6x to 2.7x. Then you have the benefit of not needing separate SSDs for DB/WAL both in hardware cost and complexity. SSDs will still be more expensive; but perhaps justifiable given the performance, rebuild times, etc.
If you only need cold-storage, then EC 8,3 on HDDs will be cheap. But is that fast enough?
Ok, I understand. We have a "hot" fraction of our dataset - and 10GB cache on all 113 HDD ~1TB effective read-cache - and then writes hitting the battery-backed write-cache - this can overspill and when hitting "cold" data performance varies. But the read/write amplification of EC is still un-manageable in pratice on HDD with an active dataset. -- Jesper
It's pretty pointless to discuss erasure coding vs replicated without knowing how it'll be used. There are setups where erasure coding is faster than replicated. You do need to write less data overall, so if that's your bottleneck then erasure coding will be faster. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, Dec 3, 2019 at 9:56 PM Nathan Fish <lordcirth@gmail.com> wrote:
If k=8,m=3 is too slow on HDDs, so you need replica 3 and SSD DB/WAL, vs EC 8,3 on SSD, then that's (1/3) / (8/11) = 0.45 multiplier on the SSD space required vs HDDs. That brings it from 6x to 2.7x. Then you have the benefit of not needing separate SSDs for DB/WAL both in hardware cost and complexity. SSDs will still be more expensive; but perhaps justifiable given the performance, rebuild times, etc.
If you only need cold-storage, then EC 8,3 on HDDs will be cheap. But is that fast enough?
On Tue, Dec 3, 2019 at 3:47 PM <jesper@krogh.cc> wrote:
* Hardware raid with Battery Backed write-cache - will allow OSD to ack writes before hitting spinning rust.
Disagree. See my litany from a few months ago. Use a plain, IT-mode HBA. Take the $$ you save and put it toward building your cluster out of SSDs instead of HDDs. That way you don’t have to mess with the management hassles of maintaining and allocating external WAL+DB partitions too.
These things are not really comparable - are they? Cost of SSD vs. HDD is still in the 6:1 favor of HHD's. Yes SSD would be great but not nessesarily affordable - or have I missed something that makes the math work ?
-- Jesper _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Cost of SSD vs. HDD is still in the 6:1 favor of HHD's. It is not, you can buy fewer thing for less money, with HDDs -that is true
$/TB is better from spinning than from flash, but this is not the most important indicator, and by far: $/IOPS is another story indeed On 12/3/19 9:46 PM, jesper@krogh.cc wrote:
These things are not really comparable - are they? Yes SSD would be great but not nessesarily affordable - or have I missed something that makes the math work ?
FYI for ZFS on RBD. https://github.com/zfsonlinux/zfs/issues/3324 We go for a more modest setting with async to 64, not 20000. -- Alex Gorbachev Intelligent Systems Services Inc. On Tue, Dec 3, 2019 at 3:07 PM Fabien Sirjean <fsirjean@eddie.fdn.fr> wrote:
Hi Ceph users !
After years of using Ceph, we plan to build soon a new cluster bigger than what we've done in the past. As the project is still in reflection, I'd like to have your thoughts on our planned design : any feedback is welcome :)
## Requirements
* ~1 PB usable space for file storage, extensible in the future * The files are mostly "hot" data, no cold storage * Purpose : storage for big files being essentially used on windows workstations (10G access) * Performance is better :)
## Global design
* 8+3 Erasure Coded pool * ZFS on RBD, exposed via samba shares (cluster with failover)
## Hardware
* 1 rack (multi-site would be better, of course...)
* OSD nodes : 14 x supermicro servers * 24 usable bays in 2U rackspace * 16 x 10 TB nearline SAS HDD (8 bays for future needs) * 2 x Xeon Silver 4212 (12C/24T) * 128 GB RAM * 4 x 40G QSFP+
* Networking : 2 x Cisco N3K 3132Q or 3164Q * 2 x 40G per server for ceph network (LACP/VPC for HA) * 2 x 40G per server for public network (LACP/VPC for HA) * QSFP+ DAC cables
## Sizing
If we've done the maths well, we expect to have :
* 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB * ~1 PB of usable space if we want to keep the OSD use under 66% to allow loosing nodes without problem, extensible to 1.6 PB (same condition)
## Reflections
* We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ?
* We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ?
* Could a 2x10G network be enough ?
* ZFS on Ceph ? Any thoughts ?
* What about CephFS ? We'd like to use RBD diff for backups but it looks impossible to use snapshot diff with Cephfs ?
Thanks for reading, and sharing your experiences !
F.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, * 2 x Xeon Silver 4212 (12C/24T)
I would choose single cpu AMD EPYC systems for lower price with better performance. Supermicro does have some good systems for AMD as well. * 16 x 10 TB nearline SAS HDD (8 bays for future needs)
Don't waste money here as well. No real gain. Invest it better in more or faster (ssd) disks. * 4 x 40G QSFP+
With 24x spinning media, even a single 40G link will be enough. No gain for a lot of money again. * 2 x 40G per server for ceph network (LACP/VPC for HA)
* 2 x 40G per server for public network (LACP/VPC for HA)
Use vlans if you really want to separate the networks. Most of the time we see new customers coming in with problems on such configurations and we don't suggest tu configure Ceph that way from our experience. * ZFS on RBD, exposed via samba shares (cluster with failover)
Maybe, just maybe think about just using samba on top of cephfs to export the data. No need for all the overhead and possible bugs you would encounter. * We're used to run mons and mgrs daemons on a few of our OSD nodes,
without any issue so far : is this a bad idea for a big cluster ?
We always do so and never had a problem with it. Just make sure the MON has enough resources for your workload. * We thought using cache tiering on an SSD pool, but a large part of the PB
is used on a daily basis, so we expect the cache to be not so effective and really expensive ?
Tend to be error prone and we saw a lot of cluster meltdowns in the last 7 years due to cache tiering. Just go for an all flash cluster use db/wal devices to improve performance. * Could a 2x10G network be enough ?
Yes ;), but maybe on recovery workloads it will slow down the recovery a bit. However I don't believe that it will be a problem in your mentioned szenario. * ZFS on Ceph ? Any thoughts ?
just don't ;) * What about CephFS ? We'd like to use RBD diff for backups but it looks
impossible to use snapshot diff with Cephfs ?
Please see https://docs.ceph.com/docs/master/dev/cephfs-snapshots/ If you do have questions or want some consulting to get the best Ceph cluster for the job. Please feel free to contact us. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Di., 3. Dez. 2019 um 21:07 Uhr schrieb Fabien Sirjean < fsirjean@eddie.fdn.fr>:
Hi Ceph users !
After years of using Ceph, we plan to build soon a new cluster bigger than what we've done in the past. As the project is still in reflection, I'd like to have your thoughts on our planned design : any feedback is welcome :)
## Requirements
* ~1 PB usable space for file storage, extensible in the future * The files are mostly "hot" data, no cold storage * Purpose : storage for big files being essentially used on windows workstations (10G access) * Performance is better :)
## Global design
* 8+3 Erasure Coded pool * ZFS on RBD, exposed via samba shares (cluster with failover)
## Hardware
* 1 rack (multi-site would be better, of course...)
* OSD nodes : 14 x supermicro servers * 24 usable bays in 2U rackspace * 16 x 10 TB nearline SAS HDD (8 bays for future needs) * 2 x Xeon Silver 4212 (12C/24T) * 128 GB RAM * 4 x 40G QSFP+
* Networking : 2 x Cisco N3K 3132Q or 3164Q * 2 x 40G per server for ceph network (LACP/VPC for HA) * 2 x 40G per server for public network (LACP/VPC for HA) * QSFP+ DAC cables
## Sizing
If we've done the maths well, we expect to have :
* 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB * ~1 PB of usable space if we want to keep the OSD use under 66% to allow loosing nodes without problem, extensible to 1.6 PB (same condition)
## Reflections
* We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ?
* We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ?
* Could a 2x10G network be enough ?
* ZFS on Ceph ? Any thoughts ?
* What about CephFS ? We'd like to use RBD diff for backups but it looks impossible to use snapshot diff with Cephfs ?
Thanks for reading, and sharing your experiences !
F.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 12/4/19 3:06 AM, Fabien Sirjean wrote:
* ZFS on RBD, exposed via samba shares (cluster with failover)
Why not use samba vfs_ceph instead? It's scalable direct access.
* What about CephFS ? We'd like to use RBD diff for backups but it looks impossible to use snapshot diff with Cephfs ?
CephFS indeed support snapshots. Since Samba 4.11 support this feature too with vfs_ceph_snapshots. k
On 12/4/19 9:19 AM, Konstantin Shalygin wrote:
CephFS indeed support snapshots. Since Samba 4.11 support this feature too with vfs_ceph_snapshots. You can snapshot, but you cannot export a diff of snapshots
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You can snapshot, but you cannot export a diff of snapshots On 12/4/19 9:19 AM, Konstantin Shalygin wrote:
On 12/4/19 3:06 AM, Fabien Sirjean wrote:
* ZFS on RBD, exposed via samba shares (cluster with failover)
Why not use samba vfs_ceph instead? It's scalable direct access.
* What about CephFS ? We'd like to use RBD diff for backups but it looks impossible to use snapshot diff with Cephfs ?
CephFS indeed support snapshots. Since Samba 4.11 support this feature too with vfs_ceph_snapshots.
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Fabien, ZFS ontop of RBD really makes me shudder. ZFS expects to have individual disk devices that it can manage. It thinks it has them with this config but CEPH is masking the real data behind it. As has been said before why not just use Samba directly from CephFS and remove that layer of complexity in the middle. You havent mentioned any NVME in the servers for RocksDB and WAL. This can make a significant performance difference. The memory you have in the servers will work for the number of drives you are planning on deploying but if you increase to the full complement of 24 drives you are likely to be tight on memory. This will be a problem on servers where you are wanting to run other services if you go down the route of deploying them on the OSD's. I don't seem to be able to find 4212 processors but I would go with 4215's if you are bound to intel as you get a higher clock speed although with less cores. You could deploy a single AMD processor per server as well. Network is well over specified for what you are doing. As you are looking at deploying switches with it why not some 25G switches which have 40G or 100G uplink ports to connect to the core. For this use case I would look at 2 x 25G ports per server. That would give you far more performance and 10/25 cards are pretty much the standard these days. Cabling up 4 QSFP cables for 14 servers is a back of rack cable nightmare. 56 thick cables just for networking. 10TB drives may not be the most cost efficient point to be at but you may need the drive count for performance. I think your useable space calculations are off. 14 Servers x 16 Drives x 10TB is 2240TB as you have said. So allowing for parity overhead you end up at 1.63PB So couple of things here. You cant fill a cluster to 100% and do you really mean PB or PiB There is roughly a 7% disk space overhead for converting from disk manufactures TB to operating system TiB. Filling a system past 85% will start giving you alerts as well. So your 1.63 x 0.93 x 0.85 gets you to 1287TiB of useable space. Quite a bit less than the 1.63PB you have mentioned. How much space do you need? I would also be considering what happens if I have a node failure and should I be reserving 1 nodes worth of capacity to allow for things like disk failures or a node failure. As the cluster gets bigger the likely hood of that sort of failure increases. On 03/12/2019, 23:44, "Fabien Sirjean" <fsirjean@eddie.fdn.fr> wrote: Hi Ceph users ! After years of using Ceph, we plan to build soon a new cluster bigger than what we've done in the past. As the project is still in reflection, I'd like to have your thoughts on our planned design : any feedback is welcome :) ## Requirements * ~1 PB usable space for file storage, extensible in the future * The files are mostly "hot" data, no cold storage * Purpose : storage for big files being essentially used on windows workstations (10G access) * Performance is better :) ## Global design * 8+3 Erasure Coded pool * ZFS on RBD, exposed via samba shares (cluster with failover) ## Hardware * 1 rack (multi-site would be better, of course...) * OSD nodes : 14 x supermicro servers * 24 usable bays in 2U rackspace * 16 x 10 TB nearline SAS HDD (8 bays for future needs) * 2 x Xeon Silver 4212 (12C/24T) * 128 GB RAM * 4 x 40G QSFP+ * Networking : 2 x Cisco N3K 3132Q or 3164Q * 2 x 40G per server for ceph network (LACP/VPC for HA) * 2 x 40G per server for public network (LACP/VPC for HA) * QSFP+ DAC cables ## Sizing If we've done the maths well, we expect to have : * 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB * ~1 PB of usable space if we want to keep the OSD use under 66% to allow loosing nodes without problem, extensible to 1.6 PB (same condition) ## Reflections * We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ? * We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ? * Could a 2x10G network be enough ? * ZFS on Ceph ? Any thoughts ? * What about CephFS ? We'd like to use RBD diff for backups but it looks impossible to use snapshot diff with Cephfs ? Thanks for reading, and sharing your experiences ! F. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Darren Soothill (darren.soothill) writes:
Hi Fabien,
ZFS ontop of RBD really makes me shudder. ZFS expects to have individual disk devices that it can manage. It thinks it has them with this config but CEPH is masking the real data behind it.
As has been said before why not just use Samba directly from CephFS and remove that layer of complexity in the middle.
As a user of ZFS on ceph, I can explain some of our motivation. As it was pointed out earlier in this thread CephFS will give you snapshots but not diffs between them. I don't know what the intent was with using diffs, but in ZFS' case, snapshots provide a basis for checkpointing/ recovery, instant dataset cloning, but also for replication/offsite mirroring (although not synchronous) - so could easily back up/replicate the ZFS datasets to another location that doesn't necessarily have a CEPH installation (say, big, cheap JBOD box with SMR drives running native ZFS). And, you can diff between snapshots to see instantly which files were modified. In addition to the other benefits of running ZFS such as lz4 compression (per dataset), deduplication, etc. While it's true that ZFS on top of RBD is not optimal, it's not particularly dangerous or unreliable. You provide it with multiple RBDs, create a pool out of those (ZFS pool, not ceph pool :). It sees each RBD as an individual disk, and can issue I/O to those indepdently. If anything, you lose some of the benefits of ZFS (automatic error correction - everything is still checksummed and you detect corruption). I already run ZFS within a VM (all our customers are hosted like this, using LXD or FreeBSD jails), whether the backing store is NFS, local disk or RBD doesn't really matter. So why NOT run ZFS on top of RBD ? Complexity mostly, and some measure of lost performance... But CephFS isn't exactly simple stuff to run in a reliable manner as of yet (MDS performance and possible deadlocks are an issue). If you're planning on serving files, you're still going to need an NFS or SMB layer. If you're on CEPHFS, you can serve via Ganesha or Samba without adding the extra ZFS layering which will add latency, but either way you're still going to drag the data out of cephfs to the client mounting the FS, export that via Samba/NFS. If instead you attach, say, 10 x 1 TB RBD images from a host, assemble those into a zfs pool, and run NFS or Samba on top of that, you'll have more or less the same data path, but in addition you'll be going through ZFS which introduces latency. Now, if you're daring, you create a ceph pool with size=1, min size=1 (will ceph let you do that ? :), you map RBDs out of that, hand them over to ZFS in a raid+mirror config (or raidz2) - and let ZFS deal with failings VDEVs by giving it new RBDs to replace them. Sounds crazy ? Well, you lose the benefit of CEPH's self-healing, but you still get a super scalable ZFS running on a near limitless supply of JBOD :) And, you can quickly set up different (zfs) pools with different levels of redundancy, quotas, compression, metadata options, etc... Who says you can't do both anyway ? (CephFS and ZFS), CEPH is flexible enough...
participants (12)
-
Alex Gorbachev
-
Anthony D'Atri
-
Darren Soothill
-
Fabien Sirjean
-
Jack
-
Jack
-
jesper＠krogh.cc
-
Konstantin Shalygin
-
Martin Verges
-
Nathan Fish
-
Paul Emmerich
-
Phil Regnauld