22 Oct
2020
22 Oct
'20
4:04 p.m.
Hi,
We're considering the merits of enabling CephFS for our main Ceph
cluster (which provides object storage for OpenStack), and one of the
obvious questions is what sort of hardware we would need for the MDSs
(and how many!).
These would be for our users scientific workloads, so they would need to
provide reasonably high performance. For reference, we have 3060 6TB
OSDs across 51 OSD hosts, and 6 dedicated RGW nodes.
The minimum specs are very modest (2-3GB RAM, a tiny amount of disk,
similar networking to the OSD nodes), but I'm not sure how much going
beyond that is likely to be useful in production.
I've also seen it suggested that an SSD-only pool is sensible for the
CephFS metadata pool; how big is that likely to get?
I'd be grateful for any pointers :)
Regards,
Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
22 Oct
22 Oct
6:25 p.m.
Hi Matthew,
On Thu, Oct 22, 2020 at 2:35 PM Matthew Vernon <mv3(a)sanger.ac.uk> wrote:
Hi,
We're considering the merits of enabling CephFS for our main Ceph
cluster (which provides object storage for OpenStack), and one of the
obvious questions is what sort of hardware we would need for the MDSs
(and how many!).
We've never mixed cephfs and rbd, for the simple reason that we
enforce QoS throttles on the openstack clients but cannot do that on
the cephfs clients.
This was decided years ago, and might be overly cautious these days.
These would be for our users scientific workloads, so
they would need to
provide reasonably high performance. For reference, we have 3060 6TB
OSDs across 51 OSD hosts, and 6 dedicated RGW nodes.
The minimum specs are very modest (2-3GB RAM, a tiny amount of disk,
similar networking to the OSD nodes), but I'm not sure how much going
beyond that is likely to be useful in production.
I've also seen it suggested that an SSD-only pool is sensible for the
CephFS metadata pool; how big is that likely to get?
From a smaller but active cephfs with size=3:
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 1.1 PiB 389 TiB 729 TiB 729 TiB 65.22
TOTAL 1.1 PiB 389 TiB 729 TiB 729 TiB 65.22
POOLS:
POOL ID STORED OBJECTS USED
%USED MAX AVAIL
cephfs_data 1 235 TiB 267.32M 235 TiB
42.69 105 TiB
cephfs_metadata 2 66 GiB 19.06M 66 GiB
0.02 105 TiB
Cheers, Dan
>
> I'd be grateful for any pointers :)
>
> Regards,
>
> Matthew
>
>
> --
> The Wellcome Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a
> company registered in England with number 2742969, whose registered
> office is 215 Euston Road, London, NW1 2BE.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Oct
23 Oct
9:33 a.m.
On 2020-10-22 14:34, Matthew Vernon wrote:
Hi,
We're considering the merits of enabling CephFS for our main Ceph
cluster (which provides object storage for OpenStack), and one of the
obvious questions is what sort of hardware we would need for the MDSs
(and how many!).
Is it a many parallel large writes workload without a lot fs
manipulation (file creation / deletion, attribute updates? You might
only need 2 for HA (active-standby). But when used as a regular fs with
many clients and a lot of small IO, than you might run out of the
performance of a single MDS. Add (many) more as you see fit. Keep in
mind it does make things a bit more complex (different ranks when more
than one active MDS) and that when you need to upgrade you have to
downscale that to 1. You can pin directories to a single MDS if you know
your workload well enough.
These would be for our users scientific workloads, so they would need to
provide reasonably high performance. For reference, we have 3060 6TB
OSDs across 51 OSD hosts, and 6 dedicated RGW nodes.
It really depend on the workload. If there are a lot of file / directory
operations the MDS needs to keep track of all that and needs to be able
to cache as well (inodes / dnodes). The more files/dirs, the more RAM
you need. We don't have PB of storage (but 39 TB for CephFS) but have
MDSes with 256 GB RAM for cache for all the little files and many dirs
we have. Prefer a few faster cores above many slower cores.
The minimum specs are very modest (2-3GB RAM, a tiny amount of disk,
similar networking to the OSD nodes), but I'm not sure how much going
beyond that is likely to be useful in production.
MDSes don't do a lot of traffic. Clients write directly to OSDs after
they have acquired capabilities (CAPS) from MDS.
I've also seen it suggested that an SSD-only pool is sensible for the
CephFS metadata pool; how big is that likely to get?
Yes, but CephFS, like RGW (index), stores a lot of data in OMAP and the
RocksDB databases tend to get quite large. Especially when storing many
small files and lots of dirs. So if that happens to be the workload,
make sure you have plenty of them. We once put all cephfs_metadata on 30
NVMe ... and that was not a good thing. Spread that data out over as
many SSD / NVMe as you can. Do your HDDs have their WAL / DB on flash?
Cephfs_metadaa does not take up a lot of space, but Mimic does not have
as good administration on all space occupied as newer releases. But I
guess it's in the order of 5% of CephFS size. But again, this might be
wildly different on other deployments.
I'd be grateful for any pointers :)
I would buy a CPU with high clock speed and ~ 4 -8 cores. RAM as needed,
but 32 GB will be minimum I guess.
Gr. Stefan
4:37 p.m.
Regarding MDS pinning, we have our home directories split into u{0..9}
for legacy reasons, and while adding more MDS' helped a little,
pinning certain u? to certain MDS' helped greatly. The automatic
migration between MDS' killed performance. This is an unusually
perfect workload for pinning, as we have 10 practically identical
directories, but still.
On Fri, Oct 23, 2020 at 2:04 AM Stefan Kooman <stefan(a)bit.nl> wrote:
>
> On 2020-10-22 14:34, Matthew Vernon wrote:
> > Hi,
> >
> > We're considering the merits of enabling CephFS for our main Ceph
> > cluster (which provides object storage for OpenStack), and one of the
> > obvious questions is what sort of hardware we would need for the MDSs
> > (and how many!).
>
> Is it a many parallel large writes workload without a lot fs
> manipulation (file creation / deletion, attribute updates? You might
> only need 2 for HA (active-standby). But when used as a regular fs with
> many clients and a lot of small IO, than you might run out of the
> performance of a single MDS. Add (many) more as you see fit. Keep in
> mind it does make things a bit more complex (different ranks when more
> than one active MDS) and that when you need to upgrade you have to
> downscale that to 1. You can pin directories to a single MDS if you know
> your workload well enough.
>
> >
> > These would be for our users scientific workloads, so they would need to
> > provide reasonably high performance. For reference, we have 3060 6TB
> > OSDs across 51 OSD hosts, and 6 dedicated RGW nodes.
>
> It really depend on the workload. If there are a lot of file / directory
> operations the MDS needs to keep track of all that and needs to be able
> to cache as well (inodes / dnodes). The more files/dirs, the more RAM
> you need. We don't have PB of storage (but 39 TB for CephFS) but have
> MDSes with 256 GB RAM for cache for all the little files and many dirs
> we have. Prefer a few faster cores above many slower cores.
>
>
> >
> > The minimum specs are very modest (2-3GB RAM, a tiny amount of disk,
> > similar networking to the OSD nodes), but I'm not sure how much going
> > beyond that is likely to be useful in production.
>
> MDSes don't do a lot of traffic. Clients write directly to OSDs after
> they have acquired capabilities (CAPS) from MDS.
>
> >
> > I've also seen it suggested that an SSD-only pool is sensible for the
> > CephFS metadata pool; how big is that likely to get?
>
> Yes, but CephFS, like RGW (index), stores a lot of data in OMAP and the
> RocksDB databases tend to get quite large. Especially when storing many
> small files and lots of dirs. So if that happens to be the workload,
> make sure you have plenty of them. We once put all cephfs_metadata on 30
> NVMe ... and that was not a good thing. Spread that data out over as
> many SSD / NVMe as you can. Do your HDDs have their WAL / DB on flash?
> Cephfs_metadaa does not take up a lot of space, but Mimic does not have
> as good administration on all space occupied as newer releases. But I
> guess it's in the order of 5% of CephFS size. But again, this might be
> wildly different on other deployments.
>
> >
> > I'd be grateful for any pointers :)
>
> I would buy a CPU with high clock speed and ~ 4 -8 cores. RAM as needed,
> but 32 GB will be minimum I guess.
>
> Gr. Stefan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1298
days inactive
1299
days old
3 comments
4 participants
participants (4)
-
Dan van der Ster -
Matthew Vernon -
Nathan Fish -
Stefan Kooman