3 Dec
2019
3 Dec
'19
8:06 p.m.
Hi Ceph users !
After years of using Ceph, we plan to build soon a new cluster bigger than what
we've done in the past. As the project is still in reflection, I'd like to
have your thoughts on our planned design : any feedback is welcome :)
## Requirements
* ~1 PB usable space for file storage, extensible in the future
* The files are mostly "hot" data, no cold storage
* Purpose : storage for big files being essentially used on windows workstations (10G
access)
* Performance is better :)
## Global design
* 8+3 Erasure Coded pool
* ZFS on RBD, exposed via samba shares (cluster with failover)
## Hardware
* 1 rack (multi-site would be better, of course...)
* OSD nodes : 14 x supermicro servers
* 24 usable bays in 2U rackspace
* 16 x 10 TB nearline SAS HDD (8 bays for future needs)
* 2 x Xeon Silver 4212 (12C/24T)
* 128 GB RAM
* 4 x 40G QSFP+
* Networking : 2 x Cisco N3K 3132Q or 3164Q
* 2 x 40G per server for ceph network (LACP/VPC for HA)
* 2 x 40G per server for public network (LACP/VPC for HA)
* QSFP+ DAC cables
## Sizing
If we've done the maths well, we expect to have :
* 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD
* 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB
* ~1 PB of usable space if we want to keep the OSD use under 66% to allow
loosing nodes without problem, extensible to 1.6 PB (same condition)
## Reflections
* We're used to run mons and mgrs daemons on a few of our OSD nodes, without
any issue so far : is this a bad idea for a big cluster ?
* We thought using cache tiering on an SSD pool, but a large part of the PB is
used on a daily basis, so we expect the cache to be not so effective and
really expensive ?
* Could a 2x10G network be enough ?
* ZFS on Ceph ? Any thoughts ?
* What about CephFS ? We'd like to use RBD diff for backups but it looks
impossible to use snapshot diff with Cephfs ?
Thanks for reading, and sharing your experiences !
F.
3 Dec
3 Dec
8:23 p.m.
After years of using Ceph, we plan to build soon a new
cluster bigger than
what
we've done in the past. As the project is still in reflection, I'd like to
have your thoughts on our planned design : any feedback is welcome :)
## Requirements
* ~1 PB usable space for file storage, extensible in the future
* The files are mostly "hot" data, no cold storage
* Purpose : storage for big files being essentially used on windows
workstations (10G access)
* Performance is better :)
## Global design
* 8+3 Erasure Coded pool
* ZFS on RBD, exposed via samba shares (cluster with failover)
## Hardware
* 1 rack (multi-site would be better, of course...)
* OSD nodes : 14 x supermicro servers
* 24 usable bays in 2U rackspace
* 16 x 10 TB nearline SAS HDD (8 bays for future needs)
* 2 x Xeon Silver 4212 (12C/24T)
* 128 GB RAM
* 4 x 40G QSFP+
* Networking : 2 x Cisco N3K 3132Q or 3164Q
* 2 x 40G per server for ceph network (LACP/VPC for HA)
* 2 x 40G per server for public network (LACP/VPC for HA)
* QSFP+ DAC cables
## Sizing
If we've done the maths well, we expect to have :
* 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD
* 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB
* ~1 PB of usable space if we want to keep the OSD use under 66% to allow
loosing nodes without problem, extensible to 1.6 PB (same condition)
## Reflections
* We're used to run mons and mgrs daemons on a few of our OSD nodes,
without
any issue so far : is this a bad idea for a big cluster ?
* We thought using cache tiering on an SSD pool, but a large part of the
PB is
used on a daily basis, so we expect the cache to be not so effective
and
really expensive ?
* Could a 2x10G network be enough ?
I would say yes, those slow disks will not deliver more anyway.
This is going to be a relative "slow" setup with limited amount of
read-caching - with 16 drives / 128GB memory it'll be a few GB per
OSD for read caching - menaning that all read-and-write will hit
the slow drives underneath.
And that in a "double slow" fashion - where one write will hit 8 + 3 OSD's
and wait for sync-ack back to the master - same with reads that
will hit 8+3 OSD's before returning to the client.
Workload depending - this may just work for you - but it is definately
not fast.
Suggestions for improvements:
* Hardware raid with Battery Backed write-cache - will allow OSD to ack
writes before hitting spinning rust.
* More memory for OSD-level read-caching.
* 3x replication instead of EC
.. (we have all above in a "similar" setup ~1PB - 10 OSD - hosts).
SSD-tiering pool (havent been there - but would like to test it out).
--
Jesper
8:27 p.m.
Rather than a cache tier, I would put an NVMe device in each OSD box
for Bluestore's DB and WAL. This will significantly improve small IOs.
14*16 HDDs / 11 chunks = 20 HDD's worth of write IOPs. If you expect
these files to be written sequentially, this is probably ok.
Mons and mgr on OSD nodes should be ok as far as I know (we are doing
it). However, if you use CephFS, you will want the fastest CPUs you
can get for the MDS, as it only scales to 2-3 cores in our tests. I
would avoid the complexity of CephFS if you do not need POSIX
semantics.
On Tue, Dec 3, 2019 at 3:07 PM Fabien Sirjean <fsirjean(a)eddie.fdn.fr> wrote:
>
> Hi Ceph users !
>
> After years of using Ceph, we plan to build soon a new cluster bigger than what
> we've done in the past. As the project is still in reflection, I'd like to
> have your thoughts on our planned design : any feedback is welcome :)
>
>
> ## Requirements
>
> * ~1 PB usable space for file storage, extensible in the future
> * The files are mostly "hot" data, no cold storage
> * Purpose : storage for big files being essentially used on windows workstations
(10G access)
> * Performance is better :)
>
>
> ## Global design
>
> * 8+3 Erasure Coded pool
> * ZFS on RBD, exposed via samba shares (cluster with failover)
>
>
> ## Hardware
>
> * 1 rack (multi-site would be better, of course...)
>
> * OSD nodes : 14 x supermicro servers
> * 24 usable bays in 2U rackspace
> * 16 x 10 TB nearline SAS HDD (8 bays for future needs)
> * 2 x Xeon Silver 4212 (12C/24T)
> * 128 GB RAM
> * 4 x 40G QSFP+
>
> * Networking : 2 x Cisco N3K 3132Q or 3164Q
> * 2 x 40G per server for ceph network (LACP/VPC for HA)
> * 2 x 40G per server for public network (LACP/VPC for HA)
> * QSFP+ DAC cables
>
>
> ## Sizing
>
> If we've done the maths well, we expect to have :
>
> * 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD
> * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB
> * ~1 PB of usable space if we want to keep the OSD use under 66% to allow
> loosing nodes without problem, extensible to 1.6 PB (same condition)
>
>
> ## Reflections
>
> * We're used to run mons and mgrs daemons on a few of our OSD nodes, without
> any issue so far : is this a bad idea for a big cluster ?
>
> * We thought using cache tiering on an SSD pool, but a large part of the PB is
> used on a daily basis, so we expect the cache to be not so effective and
> really expensive ?
>
> * Could a 2x10G network be enough ?
>
> * ZFS on Ceph ? Any thoughts ?
>
> * What about CephFS ? We'd like to use RBD diff for backups but it looks
> impossible to use snapshot diff with Cephfs ?
>
>
> Thanks for reading, and sharing your experiences !
>
> F.
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:34 p.m.
Hi,
You will get slow performance: EC is slow, HDD are slow too
With 400 iops per device, you get 89600 iops for the whole cluster, raw
With 8+3EC, each logical write is mapped to 11 physical writes
You get only 8145 write IOPS (is my math correct ?), which I find very
low for a PB storage
So, use flash-based storage, or at least not EC (which is for cold storage)
* We're used to run mons and mgrs daemons on a
few of our OSD nodes, without
any issue so far : is this a bad idea for a big cluster ?
Nothing troublesome
here
* We thought using cache tiering on an SSD pool, but
a large part of the PB is
used on a daily basis, so we expect the cache to be not so effective and
really expensive ?
Cache tiering is not recommended
You will not be able to manage this cache (managing which content needs
to be cached), thus degrading performances
* Could a 2x10G network be enough ?
If you buy
hardware, get 25Gbps
Thanks for reading, and sharing your experiences !
F.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:37 p.m.
## Requirements
* ~1 PB usable space for file storage, extensible in the future
* The files are mostly "hot" data, no cold storage
* Purpose : storage for big files being essentially used on windows workstations (10G
access)
* Performance is better :)
## Global design
* 8+3 Erasure Coded pool
EC performance for RBD is going to be mediocre at best, esp. on spinners.
* ZFS on RBD, exposed via samba shares (cluster with
failover)
Why ZFS? Mind you I like ZFS, but layering it on top of RBD is more overhead and
complexity.
* 128 GB RAM
Nowhere near enough. You’re going to want 256 at the very least.
* Networking : 2 x Cisco N3K 3132Q or 3164Q
* 2 x 40G per server for ceph network (LACP/VPC for HA)
* 2 x 40G per server for public network (LACP/VPC for HA)
Don’t bother with a replication network.
* We're used to run mons and mgrs daemons on a few
of our OSD nodes, without
any issue so far : is this a bad idea for a big cluster ?
Contention for resources can lead to a vicious circle. Failure/maint of mon/mgr/OSD at
the same time can be ugly. Put your mons on something cheap, 5 of them or 3 if you must.
* We thought using cache tiering on an SSD pool, but a
large part of the PB is
used on a daily basis, so we expect the cache to be not so effective and
really expensive ?
Cache tiering is deprecated at best. Not a good idea to invest in it. If you’re going to
use SSDs, there are better ways.
* Could a 2x10G network be enough ?
Yes.
* ZFS on Ceph ? Any thoughts ?
ZFS is great, but unless you have a specific need, it sounds like a lot of overhead and
complexity.
* Hardware raid with Battery Backed write-cache - will
allow OSD to ack writes before hitting spinning rust.
Disagree. See my litany from a few months ago. Use a plain, IT-mode HBA. Take the $$
you save and put it toward building your cluster out of SSDs instead of HDDs. That way
you don’t have to mess with the management hassles of maintaining and allocating external
WAL+DB partitions too.
3x replication instead of EC
This. The performance of EC RBD vols will likely disappoint you, esp on spinners. Having
suffered 3R RBD on LFF spinners, I predict that you would also be unhappy unless your
use-case is only archival / backups or some other cold, latency-tolerance workload.
8:46 p.m.
* Hardware
raid with Battery Backed write-cache - will allow OSD to ack
writes before hitting spinning rust.
Disagree. See my litany from a few months ago. Use a plain, IT-mode HBA.
Take the $$ you save and put it toward building your cluster out of SSDs
instead of HDDs. That way you donât have to mess with the management
hassles of maintaining and allocating external WAL+DB partitions too.
8:55 p.m.
If k=8,m=3 is too slow on HDDs, so you need replica 3 and SSD DB/WAL,
vs EC 8,3 on SSD, then that's (1/3) / (8/11) = 0.45 multiplier on the
SSD space required vs HDDs.
That brings it from 6x to 2.7x. Then you have the benefit of not
needing separate SSDs for DB/WAL both in hardware cost and complexity.
SSDs will still be more expensive; but perhaps justifiable given the
performance, rebuild times, etc.
If you only need cold-storage, then EC 8,3 on HDDs will be cheap. But
is that fast enough?
On Tue, Dec 3, 2019 at 3:47 PM <jesper(a)krogh.cc> wrote:
>
> >> * Hardware raid with Battery Backed write-cache - will allow OSD to ack
> >> writes before hitting spinning rust.
> >
> > Disagree. See my litany from a few months ago. Use a plain, IT-mode HBA.
> > Take the $$ you save and put it toward building your cluster out of SSDs
> > instead of HDDs. That way you don’t have to mess with the management
> > hassles of maintaining and allocating external WAL+DB partitions too.
>
> These things are not really comparable - are they? Cost of SSD vs. HDD is
> still in the 6:1 favor of HHD's. Yes SSD would be great but not
> nessesarily affordable - or have I missed something that makes the math
> work ?
>
> --
> Jesper
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:02 p.m.
If k=8,m=3 is too slow on HDDs, so you need replica 3
and SSD DB/WAL,
vs EC 8,3 on SSD, then that's (1/3) / (8/11) = 0.45 multiplier on the
SSD space required vs HDDs.
That brings it from 6x to 2.7x. Then you have the benefit of not
needing separate SSDs for DB/WAL both in hardware cost and complexity.
SSDs will still be more expensive; but perhaps justifiable given the
performance, rebuild times, etc.
If you only need cold-storage, then EC 8,3 on HDDs will be cheap. But
is that fast enough?
Ok, I understand.
We have a "hot" fraction of our dataset - and 10GB cache on all 113 HDD
~1TB effective read-cache - and then writes hitting the battery-backed
write-cache - this can overspill and when hitting "cold" data performance
varies. But the read/write amplification of EC is still un-manageable in
pratice on HDD with an active dataset.
--
Jesper
9:04 p.m.
It's pretty pointless to discuss erasure coding vs replicated without
knowing how it'll be used.
There are setups where erasure coding is faster than replicated. You
do need to write less data overall, so if that's your bottleneck then
erasure coding will be faster.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Tue, Dec 3, 2019 at 9:56 PM Nathan Fish <lordcirth(a)gmail.com> wrote:
>
> If k=8,m=3 is too slow on HDDs, so you need replica 3 and SSD DB/WAL,
> vs EC 8,3 on SSD, then that's (1/3) / (8/11) = 0.45 multiplier on the
> SSD space required vs HDDs.
> That brings it from 6x to 2.7x. Then you have the benefit of not
> needing separate SSDs for DB/WAL both in hardware cost and complexity.
> SSDs will still be more expensive; but perhaps justifiable given the
> performance, rebuild times, etc.
>
> If you only need cold-storage, then EC 8,3 on HDDs will be cheap. But
> is that fast enough?
>
> On Tue, Dec 3, 2019 at 3:47 PM <jesper(a)krogh.cc> wrote:
> >
> > >> * Hardware raid with Battery Backed write-cache - will allow OSD to
ack
> > >> writes before hitting spinning rust.
> > >
> > > Disagree. See my litany from a few months ago. Use a plain, IT-mode HBA.
> > > Take the $$ you save and put it toward building your cluster out of SSDs
> > > instead of HDDs. That way you don’t have to mess with the management
> > > hassles of maintaining and allocating external WAL+DB partitions too.
> >
> > These things are not really comparable - are they? Cost of SSD vs. HDD is
> > still in the 6:1 favor of HHD's. Yes SSD would be great but not
> > nessesarily affordable - or have I missed something that makes the math
> > work ?
> >
> > --
> > Jesper
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:35 p.m.
Cost of SSD vs. HDD is still in the 6:1 favor of
HHD's.
It is not, you can buy fewer thing for less money, with HDDs -that is
true
$/TB is better from spinning than from flash, but this is not the most
important indicator, and by far: $/IOPS is another story indeed
On 12/3/19 9:46 PM, jesper(a)krogh.cc wrote:
These things are not really comparable - are they?
Yes SSD would be great but not
nessesarily affordable - or have I missed something that makes the math
work ?
11:01 p.m.
FYI for ZFS on RBD.
https://github.com/zfsonlinux/zfs/issues/3324
We go for a more modest setting with async to 64, not 20000.
--
Alex Gorbachev
Intelligent Systems Services Inc.
On Tue, Dec 3, 2019 at 3:07 PM Fabien Sirjean <fsirjean(a)eddie.fdn.fr> wrote:
Hi Ceph users !
After years of using Ceph, we plan to build soon a new cluster bigger than
what
we've done in the past. As the project is still in reflection, I'd like to
have your thoughts on our planned design : any feedback is welcome :)
## Requirements
* ~1 PB usable space for file storage, extensible in the future
* The files are mostly "hot" data, no cold storage
* Purpose : storage for big files being essentially used on windows
workstations (10G access)
* Performance is better :)
## Global design
* 8+3 Erasure Coded pool
* ZFS on RBD, exposed via samba shares (cluster with failover)
## Hardware
* 1 rack (multi-site would be better, of course...)
* OSD nodes : 14 x supermicro servers
* 24 usable bays in 2U rackspace
* 16 x 10 TB nearline SAS HDD (8 bays for future needs)
* 2 x Xeon Silver 4212 (12C/24T)
* 128 GB RAM
* 4 x 40G QSFP+
* Networking : 2 x Cisco N3K 3132Q or 3164Q
* 2 x 40G per server for ceph network (LACP/VPC for HA)
* 2 x 40G per server for public network (LACP/VPC for HA)
* QSFP+ DAC cables
## Sizing
If we've done the maths well, we expect to have :
* 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD
* 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB
* ~1 PB of usable space if we want to keep the OSD use under 66% to allow
loosing nodes without problem, extensible to 1.6 PB (same condition)
## Reflections
* We're used to run mons and mgrs daemons on a few of our OSD nodes,
without
any issue so far : is this a bad idea for a big cluster ?
* We thought using cache tiering on an SSD pool, but a large part of the
PB is
used on a daily basis, so we expect the cache to be not so effective and
really expensive ?
* Could a 2x10G network be enough ?
* ZFS on Ceph ? Any thoughts ?
* What about CephFS ? We'd like to use RBD diff for backups but it looks
impossible to use snapshot diff with Cephfs ?
Thanks for reading, and sharing your experiences !
F.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4 Dec
4 Dec
7:55 a.m.
Hello,
* 2 x Xeon Silver 4212 (12C/24T)
I would choose single cpu AMD EPYC systems for lower price with better
performance. Supermicro does have some good systems for AMD as well.
* 16 x 10 TB nearline SAS HDD (8 bays for future needs)
Don't waste money here as well. No real gain. Invest it better in more or
faster (ssd) disks.
* 4 x 40G QSFP+
With 24x spinning media, even a single 40G link will be enough. No gain for
a lot of money again.
* 2 x 40G per server for ceph network (LACP/VPC for HA)
> * 2 x 40G per server for public network (LACP/VPC for HA)
Use vlans if you really want to separate the networks. Most of the time we
see new customers coming in with problems on such configurations and we
don't suggest tu configure Ceph that way from our experience.
* ZFS on RBD, exposed via samba shares (cluster with failover)
Maybe, just maybe think about just using samba on top of cephfs to export
the data. No need for all the overhead and possible bugs you would
encounter.
* We're used to run mons and mgrs daemons on a few of our OSD nodes,
> without any issue so far : is this a bad idea for a big cluster ?
We always do so and never had a problem with it. Just make sure the MON has
enough resources for your workload.
* We thought using cache tiering on an SSD pool, but a large part of the PB
> is used on a daily basis, so we expect the cache to be not so effective
> and really expensive ?
Tend to be error prone and we saw a lot of cluster meltdowns in the last 7
years due to cache tiering. Just go for an all flash cluster use db/wal
devices to improve performance.
* Could a 2x10G network be enough ?
Yes ;), but maybe on recovery workloads it will slow down the recovery a
bit. However I don't believe that it will be a problem in your mentioned
szenario.
* ZFS on Ceph ? Any thoughts ?
just don't ;)
* What about CephFS ? We'd like to use RBD diff for backups but it looks
> After years of using Ceph, we plan to build
soon a new cluster bigger than
> what
> we've done in the past. As the project is still in reflection, I'd like to
> have your thoughts on our planned design : any feedback is welcome :)
> ## Requirements
> * ~1 PB usable space for file storage,
extensible in the future
> * The files are mostly "hot" data, no cold storage
> * Purpose : storage for big files being essentially used on windows
> workstations (10G access)
> * Performance is better :)
> ## Global design
> * 8+3 Erasure Coded pool
> * ZFS on RBD, exposed via samba shares (cluster with failover)
> ## Hardware
> * 1 rack (multi-site would be better, of
course...)
> * OSD nodes : 14 x supermicro servers
> * 24 usable bays in 2U rackspace
> * 16 x 10 TB nearline SAS HDD (8 bays for future needs)
> * 2 x Xeon Silver 4212 (12C/24T)
> * 128 GB RAM
> * 4 x 40G QSFP+
> * Networking : 2 x Cisco N3K 3132Q or
3164Q
> * 2 x 40G per server for ceph network (LACP/VPC for HA)
> * 2 x 40G per server for public network (LACP/VPC for HA)
> * QSFP+ DAC cables
> ## Sizing
> If we've done the maths well, we expect
to have :
> * 2.24 PB of raw storage, extensible to
3.36 PB by adding HDD
> * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB
> * ~1 PB of usable space if we want to keep the OSD use under 66% to allow
> loosing nodes without problem, extensible to 1.6 PB (same condition)
> ## Reflections
> * We're used to run mons and mgrs
daemons on a few of our OSD nodes,
> without
> any issue so far : is this a bad idea for a big cluster ?
> * We thought using cache tiering on an SSD
pool, but a large part of the
> PB is
> used on a daily basis, so we expect the cache to be not so effective and
> really expensive ?
> * Could a 2x10G network be enough ?
> * ZFS on Ceph ? Any thoughts ?
> * What about CephFS ? We'd like to use
RBD diff for backups but it looks
> impossible to use snapshot diff with Cephfs ?
> Thanks for reading, and sharing your experiences !
> F.
>
_______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
impossible to use snapshot diff with Cephfs ?
Please see https://docs.ceph.com/docs/master/dev/cephfs-snapshots/
If you do have questions or want some consulting to get the best Ceph
cluster for the job. Please feel free to contact us.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Di., 3. Dez. 2019 um 21:07 Uhr schrieb Fabien Sirjean <
fsirjean(a)eddie.fdn.fr>gt;:
> Hi Ceph users !
8:19 a.m.
On 12/4/19 3:06 AM, Fabien Sirjean wrote:
* ZFS on RBD, exposed via samba shares (cluster with
failover)
Why not use samba vfs_ceph instead? It's scalable direct access.
* What about CephFS ? We'd like to use RBD diff
for backups but it looks
impossible to use snapshot diff with Cephfs ?
CephFS indeed support snapshots. Since Samba 4.11 support this feature
too with vfs_ceph_snapshots.
k
8:33 a.m.
On 12/4/19 9:19 AM, Konstantin Shalygin wrote:
CephFS indeed support snapshots. Since Samba 4.11
support this feature
too with vfs_ceph_snapshots.
You can snapshot, but you cannot export a diff of
snapshots
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:33 a.m.
You can snapshot, but you cannot export a diff of snapshots
On 12/4/19 9:19 AM, Konstantin Shalygin wrote:
On 12/4/19 3:06 AM, Fabien Sirjean wrote:
* ZFS on RBD, exposed via samba shares (cluster
with failover)
Why not use samba vfs_ceph instead? It's scalable direct access.
* What about CephFS ? We'd like to use RBD
diff for backups but it
looks
impossible to use snapshot diff with Cephfs ?
CephFS indeed support snapshots. Since Samba 4.11 support this feature
too with vfs_ceph_snapshots.
k
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
10:59 a.m.
Hi Fabien,
ZFS ontop of RBD really makes me shudder. ZFS expects to have individual disk devices that
it can manage. It thinks it has them with this config but CEPH is masking the real data
behind it.
As has been said before why not just use Samba directly from CephFS and remove that layer
of complexity in the middle.
You havent mentioned any NVME in the servers for RocksDB and WAL. This can make a
significant performance difference.
The memory you have in the servers will work for the number of drives you are planning on
deploying but if you increase to the full complement of 24 drives you are likely to be
tight on memory.
This will be a problem on servers where you are wanting to run other services if you go
down the route of deploying them on the OSD's.
I don't seem to be able to find 4212 processors but I would go with 4215's if you
are bound to intel as you get a higher clock speed although with less cores. You could
deploy a single AMD processor per server as well.
Network is well over specified for what you are doing. As you are looking at deploying
switches with it why not some 25G switches which have 40G or 100G uplink ports to connect
to the core.
For this use case I would look at 2 x 25G ports per server. That would give you far more
performance and 10/25 cards are pretty much the standard these days.
Cabling up 4 QSFP cables for 14 servers is a back of rack cable nightmare. 56 thick cables
just for networking.
10TB drives may not be the most cost efficient point to be at but you may need the drive
count for performance.
I think your useable space calculations are off.
14 Servers x 16 Drives x 10TB is 2240TB as you have said.
So allowing for parity overhead you end up at 1.63PB
So couple of things here.
You cant fill a cluster to 100% and do you really mean PB or PiB
There is roughly a 7% disk space overhead for converting from disk manufactures TB to
operating system TiB.
Filling a system past 85% will start giving you alerts as well.
So your 1.63 x 0.93 x 0.85 gets you to 1287TiB of useable space. Quite a bit less than the
1.63PB you have mentioned. How much space do you need?
I would also be considering what happens if I have a node failure and should I be
reserving 1 nodes worth of capacity to allow for things like disk failures or a node
failure. As the cluster gets bigger the likely hood of that sort of failure increases.
On 03/12/2019, 23:44, "Fabien Sirjean" <fsirjean(a)eddie.fdn.fr> wrote:
Hi Ceph users !
After years of using Ceph, we plan to build soon a new cluster bigger than what
we've done in the past. As the project is still in reflection, I'd like to
have your thoughts on our planned design : any feedback is welcome :)
## Requirements
* ~1 PB usable space for file storage, extensible in the future
* The files are mostly "hot" data, no cold storage
* Purpose : storage for big files being essentially used on windows workstations (10G
access)
* Performance is better :)
## Global design
* 8+3 Erasure Coded pool
* ZFS on RBD, exposed via samba shares (cluster with failover)
## Hardware
* 1 rack (multi-site would be better, of course...)
* OSD nodes : 14 x supermicro servers
* 24 usable bays in 2U rackspace
* 16 x 10 TB nearline SAS HDD (8 bays for future needs)
* 2 x Xeon Silver 4212 (12C/24T)
* 128 GB RAM
* 4 x 40G QSFP+
* Networking : 2 x Cisco N3K 3132Q or 3164Q
* 2 x 40G per server for ceph network (LACP/VPC for HA)
* 2 x 40G per server for public network (LACP/VPC for HA)
* QSFP+ DAC cables
## Sizing
If we've done the maths well, we expect to have :
* 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD
* 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB
* ~1 PB of usable space if we want to keep the OSD use under 66% to allow
loosing nodes without problem, extensible to 1.6 PB (same condition)
## Reflections
* We're used to run mons and mgrs daemons on a few of our OSD nodes, without
any issue so far : is this a bad idea for a big cluster ?
* We thought using cache tiering on an SSD pool, but a large part of the PB is
used on a daily basis, so we expect the cache to be not so effective and
really expensive ?
* Could a 2x10G network be enough ?
* ZFS on Ceph ? Any thoughts ?
* What about CephFS ? We'd like to use RBD diff for backups but it looks
impossible to use snapshot diff with Cephfs ?
Thanks for reading, and sharing your experiences !
F.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:18 p.m.
Darren Soothill (darren.soothill) writes:
Hi Fabien,
ZFS ontop of RBD really makes me shudder. ZFS expects to have individual disk devices
that it can manage. It thinks it has them with this config but CEPH is masking the real
data behind it.
As has been said before why not just use Samba directly from CephFS and remove that layer
of complexity in the middle.
As a user of ZFS on ceph, I can explain some of our motivation.
As it was pointed out earlier in this thread CephFS will give you snapshots
but not diffs between them. I don't know what the intent was with using
diffs, but in ZFS' case, snapshots provide a basis for checkpointing/
recovery, instant dataset cloning, but also for replication/offsite
mirroring (although not synchronous) - so could easily back up/replicate
the ZFS datasets to another location that doesn't necessarily have a CEPH
installation (say, big, cheap JBOD box with SMR drives running native ZFS).
And, you can diff between snapshots to see instantly which files were
modified. In addition to the other benefits of running ZFS such as lz4
compression (per dataset), deduplication, etc.
While it's true that ZFS on top of RBD is not optimal, it's not
particularly dangerous or unreliable. You provide it with multiple RBDs,
create a pool out of those (ZFS pool, not ceph pool :). It sees each
RBD as an individual disk, and can issue I/O to those indepdently.
If anything, you lose some of the benefits of ZFS (automatic error
correction - everything is still checksummed and you detect corruption).
I already run ZFS within a VM (all our customers are hosted like this,
using LXD or FreeBSD jails), whether the backing store is NFS, local disk
or RBD doesn't really matter.
So why NOT run ZFS on top of RBD ? Complexity mostly, and some measure
of lost performance... But CephFS isn't exactly simple stuff to run in a
reliable manner as of yet (MDS performance and possible deadlocks are
an issue).
If you're planning on serving files, you're still going to need an NFS
or SMB layer. If you're on CEPHFS, you can serve via Ganesha or Samba
without adding the extra ZFS layering which will add latency, but either
way you're still going to drag the data out of cephfs to the client
mounting the FS, export that via Samba/NFS. If instead you attach, say,
10 x 1 TB RBD images from a host, assemble those into a zfs pool, and
run NFS or Samba on top of that, you'll have more or less the same data
path, but in addition you'll be going through ZFS which introduces latency.
Now, if you're daring, you create a ceph pool with size=1, min size=1
(will ceph let you do that ? :), you map RBDs out of that, hand them over
to ZFS in a raid+mirror config (or raidz2) - and let ZFS deal with
failings VDEVs by giving it new RBDs to replace them. Sounds crazy ?
Well, you lose the benefit of CEPH's self-healing, but you still get
a super scalable ZFS running on a near limitless supply of JBOD :) And,
you can quickly set up different (zfs) pools with different levels of
redundancy, quotas, compression, metadata options, etc...
Who says you can't do both anyway ? (CephFS and ZFS), CEPH is flexible
enough...
1617
days inactive
1618
days old
16 comments
12 participants
participants (12)
-
Alex Gorbachev -
Anthony D'Atri -
Darren Soothill -
Fabien Sirjean -
Jack -
Jack -
jesper＠krogh.cc -
Konstantin Shalygin -
Martin Verges -
Nathan Fish -
Paul Emmerich -
Phil Regnauld