7 Feb
2020
7 Feb
'20
5:01 a.m.
Hi, we are planning out a Ceph storage cluster and were choosing
between 64GB, 128GB, or even 256GB on metadata servers. We are
considering having 2 metadata servers overall.
Does going to high levels of RAM possibly yield any performance
benefits? Is there a size beyond which there are just diminishing
returns vs cost?
The expected use case would be for a cluster where there might be
10-20 concurrent users working on individual datasets of 5TB in size.
I expect there would be lots of reads of the 5TB datasets matched with
the creation of hundreds to thousands of smaller files during
processing of the images.
Thanks!
-Matt
--
Matt Larson, PhD
Madison, WI 53705 U.S.A.
7 Feb
7 Feb
5:11 a.m.
Hi,
I am running on 3 MDS servers (1 active and 2 backups and I recommend that)
each of 128 GB of RAM (the clients are running ML analysis) and I have
about 20 mil inodes loaded in ram. It's working fine except some warnings I
have "client X is failing to respond to cache pressure."
Besides that there are no complaints but I thing you would need the 256GB
of ram specially if the datasets will increase... just my 2 cents..
Will you have SSD ?
On Fri, Feb 7, 2020 at 12:02 AM Matt Larson <larsonmattr(a)gmail.com> wrote:
Hi, we are planning out a Ceph storage cluster and
were choosing
between 64GB, 128GB, or even 256GB on metadata servers. We are
considering having 2 metadata servers overall.
Does going to high levels of RAM possibly yield any performance
benefits? Is there a size beyond which there are just diminishing
returns vs cost?
The expected use case would be for a cluster where there might be
10-20 concurrent users working on individual datasets of 5TB in size.
I expect there would be lots of reads of the 5TB datasets matched with
the creation of hundreds to thousands of smaller files during
processing of the images.
Thanks!
-Matt
--
Matt Larson, PhD
Madison, WI 53705 U.S.A.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:46 a.m.
Hi Bogdan,
Are the "client failing to respond" messages indicating that you
actually exceed the 128 GB ram on your MDS hosts?
The MDS servers are not planned to have SSD drives. The storage
servers would have HD's and 1 nVME SSD drive that could hold metadata
volumes.
On Thu, Feb 6, 2020 at 4:11 PM Bogdan Adrian Velica <vbogdan(a)gmail.com> wrote:
Hi,
I am running on 3 MDS servers (1 active and 2 backups and I recommend that) each of 128
GB of RAM (the clients are running ML analysis) and I have about 20 mil inodes loaded in
ram. It's working fine except some warnings I have "client X is failing to
respond to cache pressure."
Besides that there are no complaints but I thing you would need the 256GB of ram
specially if the datasets will increase... just my 2 cents..
Will you have SSD ?
On Fri, Feb 7, 2020 at 12:02 AM Matt Larson <larsonmattr(a)gmail.com> wrote:
>
> Hi, we are planning out a Ceph storage cluster and were choosing
> between 64GB, 128GB, or even 256GB on metadata servers. We are
> considering having 2 metadata servers overall.
>
> Does going to high levels of RAM possibly yield any performance
> benefits? Is there a size beyond which there are just diminishing
> returns vs cost?
>
> The expected use case would be for a cluster where there might be
> 10-20 concurrent users working on individual datasets of 5TB in size.
> I expect there would be lots of reads of the 5TB datasets matched with
> the creation of hundreds to thousands of smaller files during
> processing of the images.
>
> Thanks!
> -Matt
>
> --
> Matt Larson, PhD
> Madison, WI 53705 U.S.A.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Matt Larson, PhD
Madison, WI 53705 U.S.A.
3:16 p.m.
Quoting Matt Larson (larsonmattr(a)gmail.com):
The MDS servers are not planned to have SSD drives. The storage
servers would have HD's and 1 nVME SSD drive that could hold metadata
volumes.
Be careful with that. One thing we have learned (the hard way) is that
the impact of a lot of OMAP data (key-value info used by the MDS ... and
RGW for indexes for that matter) condensed on a limited amount of OSDs
might hurt performance. Ideally you would want to spread it out accross
as many (flash) drives as you can. Hundreds of thousands to a couple
million files / directories will probably be not be an issue. Are you
planning on storing (processed) datasets on CephFS for the long term as
well? In that case a lot of files / directories will matter (over time).
If budget / hardware allows buy more NVMe ... and partition them in
equal parts (benchmarks show that 4 would be optimal). And put the
WAL/DB of the HDs (spinners) on them. Be careful though, if one NVMe
dies the HDs that depend on them are going down with it. It's a tradeoff
between efficiency and cost / performance.
Gr. Stefan
--
| BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
5:20 a.m.
On 2/6/20 11:01 PM, Matt Larson wrote:
Hi, we are planning out a Ceph storage cluster and
were choosing
between 64GB, 128GB, or even 256GB on metadata servers. We are
considering having 2 metadata servers overall.
Does going to high levels of RAM possibly yield any performance
benefits? Is there a size beyond which there are just diminishing
returns vs cost?
The MDS will try to cache as much inodes as you allow it to.
So the amount of users nor the total amount of bytes doesn't matter,
it's the amount of inodes, thus: files and directories.
The more you have of those, the more memory it requires.
A lot of small files? A lot of memory!
Wido
> The expected use case would be for a cluster where there might be
> 10-20 concurrent users working on individual datasets of 5TB in size.
> I expect there would be lots of reads of the 5TB datasets matched with
> the creation of hundreds to thousands of smaller files during
> processing of the images.
>
> Thanks!
> -Matt
>
3:05 p.m.
Quoting Wido den Hollander (wido(a)42on.com):
On 2/6/20 11:01 PM, Matt Larson wrote:
If clients are using unique datasets (files / directories) than the
amount of clients do matter. If that is the case you might also ask
yourself why you need a clustered filesystem, as it will definitely not
speed things up compared to a local fs (metadata operations that is).
Hi, we are planning out a Ceph storage cluster
and were choosing
between 64GB, 128GB, or even 256GB on metadata servers. We are
considering having 2 metadata servers overall.
Does going to high levels of RAM possibly yield any performance
benefits? Is there a size beyond which there are just diminishing
returns vs cost?
The MDS will try to cache as much inodes as you allow it to.
So the amount of users nor the total amount of bytes doesn't matter,
it's the amount of inodes, thus: files and directories. The more you have of those, the more memory it
requires.
To clarify: in (active) use. Just having a lot of data around does not
necessarily require a lot of memory.
A lot of small files? A lot of memory!
> The expected use case would be for a cluster
where there might be
> 10-20 concurrent users working on individual datasets of 5TB in size.
> I expect there would be lots of reads of the 5TB datasets matched with
> the creation of hundreds to thousands of smaller files during
> processing of the images.
Hundreds to thousands of files is not a lot. Are these datasets to be
stored permanently, or only temporarily? I guess it is convenient to
just configure one fs for all clients to use, but it might not be the
best fit / best performing solution in your case.
Gr. Stefan
--
| BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
11 Feb
11 Feb
4:31 a.m.
Hi together,
I am new here. I am a little bit confused about the discussion about the
amount RAM of the metadata server.
In the SUSE Deployment Guide for SUSE Enterprise Storage 6 (release
2020-01-27) in the chapter "2.2 Minimum Cluster Configuration" there the
is a sentence:
"... Metadata Servers require incremental 4 GB RAM and four cores."
Your discussion is about 128 GB and 256 GB. This is far away from the
SUSE min. requirements.
Can you explain that or give any hint to that why the value are so
different?
Marco
Am 07.02.2020 um 09:05 schrieb Stefan Kooman:
Quoting Wido den Hollander (wido(a)42on.com):
On 2/6/20 11:01 PM, Matt Larson wrote:
If clients are using
unique datasets (files / directories) than the
amount of clients do matter. If that is the case you might also ask
yourself why you need a clustered filesystem, as it will definitely not
speed things up compared to a local fs (metadata operations that is).
Hi, we are planning out a Ceph storage cluster
and were choosing
between 64GB, 128GB, or even 256GB on metadata servers. We are
considering having 2 metadata servers overall.
Does going to high levels of RAM possibly yield any performance
benefits? Is there a size beyond which there are just diminishing
returns vs cost?
The MDS will try to cache as much inodes as you allow it to.
So the amount of users nor the total amount of bytes doesn't matter,
it's the amount of inodes, thus: files and directories. The more you have of those, the more memory it
requires.
To clarify: in (active) use. Just having a lot of data around does not
necessarily require a lot of memory.
A lot of small files? A lot of memory!
> The expected use case would be for a cluster where there might be
> 10-20 concurrent users working on individual datasets of 5TB in size.
> I expect there would be lots of reads of the 5TB datasets matched with
> the creation of hundreds to thousands of smaller files during
> processing of the images.
Hundreds to thousands of files is not a lot. Are
these datasets to be
stored permanently, or only temporarily? I guess it is convenient to
just configure one fs for all clients to use, but it might not be the
best fit / best performing solution in your case.
Gr. Stefan
14 Feb
14 Feb
9:12 p.m.
Hi Marco,
the MDS cache size is depending heavily on the load and the number of
clients that access your cephFS (as always I'd say).
The mentioned 4 GB of RAM is appropriate for a few clients with no
special requirements regarding performance, so basically it's a
minimal sizing (as the deployment guide also states).
One example from "real life" is a MDS with 8 GB RAM serving mostly
home directories and some working directories for development. The
total amount of (connected) clients is around 70, but not all of them
are actually changing files constantly, I'd say the active clients
could be around 20.
Regards,
Eugen
Zitat von Marco Mühlenbeck <marco.muehlenbeck(a)ts.fujitsu.com>om>:
Hi together,
I am new here. I am a little bit confused about the discussion about
the amount RAM of the metadata server.
In the SUSE Deployment Guide for SUSE Enterprise Storage 6 (release
2020-01-27) in the chapter "2.2 Minimum Cluster Configuration" there
the is a sentence:
"... Metadata Servers require incremental 4 GB RAM and four cores."
Your discussion is about 128 GB and 256 GB. This is far away from
the SUSE min. requirements.
Can you explain that or give any hint to that why the value are so different?
Marco
Am 07.02.2020 um 09:05 schrieb Stefan Kooman:
Quoting Wido den Hollander (wido(a)42on.com):
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
On 2/6/20 11:01 PM, Matt Larson wrote:
If clients are using
unique datasets (files / directories) than the
amount of clients do matter. If that is the case you might also ask
yourself why you need a clustered filesystem, as it will definitely not
speed things up compared to a local fs (metadata operations that is).
Hi, we are planning out a Ceph storage cluster
and were choosing
between 64GB, 128GB, or even 256GB on metadata servers. We are
considering having 2 metadata servers overall.
Does going to high levels of RAM possibly yield any performance
benefits? Is there a size beyond which there are just diminishing
returns vs cost?
The MDS will try to cache as much inodes as you allow it to.
So the amount of users nor the total amount of bytes doesn't matter,
it's the amount of inodes, thus: files and directories. The more you have of those, the more memory it
requires.
To clarify: in (active) use. Just having a lot of data around does not
necessarily require a lot of memory.
A lot of small files? A lot of memory!
> The expected use case would be for a cluster where there might be
> 10-20 concurrent users working on individual datasets of 5TB in size.
> I expect there would be lots of reads of the 5TB datasets matched with
> the creation of hundreds to thousands of smaller files during
> processing of the images.
Hundreds to thousands of files is not a lot. Are
these datasets to be
stored permanently, or only temporarily? I guess it is convenient to
just configure one fs for all clients to use, but it might not be the
best fit / best performing solution in your case.
Gr. Stefan
1540
days inactive
1548
days old
7 comments
6 participants
participants (6)
-
Bogdan Adrian Velica -
Eugen Block -
Marco Mühlenbeck -
Matt Larson -
Stefan Kooman -
Wido den Hollander