10 Mar
2020
10 Mar
'20
8:24 a.m.
Quick question Ceph guru's.
For a 1.1PB raw cephfs system currently storing 191TB of data and 390 million objects
(mostly small Python, ML training files etc.) how many MDS servers should I be running?
System is Nautilus 14.2.8.
I ask because up to know I have run one MDS with one standby-replay and occasionally it
blows up with large memory consumption, 60Gb+ even though I have mds_cache_memory_limit =
32G and that was 16G until recently. It of course tries to restart on another MDS node
fails again and after several attempts usually comes back up. Today I increased to two
active MDS's but the question is what is the optimal number for a pretty active
system? The single MDS seemed to regularly run around 1400 req/s and I often get up to six
clients failing to respond to cache pressure.
The current setup is:
ceph fs status
cephfs - 71 clients
======
+------+----------------+--------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+----------------+--------+---------------+-------+-------+
| 0 | active | a | Reqs: 447 /s | 12.0M | 11.9M |
| 1 | active | b | Reqs: 154 /s | 1749k | 1686k |
| 1-s | standby-replay | c | Evts: 136 /s | 1440k | 1423k |
| 0-s | standby-replay | d | Evts: 402 /s | 16.8k | 298 |
+------+----------------+--------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 160G | 169G |
| cephfs_data | data | 574T | 140T |
+-----------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
| w |
| x |
| y |
| z |
+-------------+
MDS version: ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus
(stable)
Regards.
Robert Ruge
Systems & Network Manager
Faculty of Science, Engineering & Built Environment
[cid:image001.png@01D36789.04BE09A0]
Important Notice: The contents of this email are intended solely for the named addressee
and are confidential; any unauthorised use, reproduction or storage of the contents is
expressly prohibited. If you have received this email in error, please delete it and any
attachments immediately and advise the sender by return email or telephone.
Deakin University does not warrant that this email and any attachments are error or virus
free.
6 May
6 May
10:54 p.m.
Hello Robert,
On Mon, Mar 9, 2020 at 7:55 PM Robert Ruge <robert.ruge(a)deakin.edu.au> wrote:
For a 1.1PB raw cephfs system currently storing 191TB
of data and 390 million objects (mostly small Python, ML training files etc.) how many MDS
servers should I be running?
System is Nautilus 14.2.8.
I ask because up to know I have run one MDS with one standby-replay and occasionally it
blows up with large memory consumption, 60Gb+ even though I have mds_cache_memory_limit =
32G and that was 16G until recently. It of course tries to restart on another MDS node
fails again and after several attempts usually comes back up. Today I increased to two
active MDS’s but the question is what is the optimal number for a pretty active system?
The single MDS seemed to regularly run around 1400 req/s and I often get up to six clients
failing to respond to cache pressure.
Ideally, the only reason you should add more active MDS (increase
max_mds) is because you want to increase request throughput.
60GB RSS is not completely unexpected. A 32GB cache size would use
approximately 48GB (150%) RSS in a steady state situation. You may
ahve hit some kind of bug as others have reported which is causing the
cache size / anonymous memory to continually increase. You will need
to post more information about the client type/version, cache usage,
perf dumps, and workload to help diagnose.
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
7 May
7 May
6:51 p.m.
On Thu, May 7, 2020 at 1:27 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
Hello Robert,
On Mon, Mar 9, 2020 at 7:55 PM Robert Ruge <robert.ruge(a)deakin.edu.au> wrote:
https://github.com/ceph/ceph/pull/34571 may help if "ceph daemon mds.a
dump_mempools" shows buffer_anon uses lots of memory.
> --
> Patrick Donnelly, Ph.D.
> He / Him / His
> Senior Software Engineer
> Red Hat Sunnyvale, CA
> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
For a 1.1PB raw cephfs system currently storing
191TB of data and 390 million objects (mostly small Python, ML training files etc.) how
many MDS servers should I be running?
System is Nautilus 14.2.8.
I ask because up to know I have run one MDS with one standby-replay and occasionally it
blows up with large memory consumption, 60Gb+ even though I have mds_cache_memory_limit =
32G and that was 16G until recently. It of course tries to restart on another MDS node
fails again and after several attempts usually comes back up. Today I increased to two
active MDS’s but the question is what is the optimal number for a pretty active system?
The single MDS seemed to regularly run around 1400 req/s and I often get up to six clients
failing to respond to cache pressure.
Ideally, the only reason you should add more active MDS (increase
max_mds) is because you want to increase request throughput.
60GB RSS is not completely unexpected. A 32GB cache size would use
approximately 48GB (150%) RSS in a steady state situation. You may
ahve hit some kind of bug as others have reported which is causing the
cache size / anonymous memory to continually increase. You will need
to post more information about the client type/version, cache usage,
perf dumps, and workload to help diagnose.
10:11 p.m.
On Thu, May 7, 2020 at 6:22 AM Yan, Zheng <ukernel(a)gmail.com> wrote:
On Thu, May 7, 2020 at 1:27 AM Patrick Donnelly
<pdonnell(a)redhat.com>
wrote:
We struggle with cache management as well and I just thought it was due to
our really old kernel clients (I'm sure that doesn't help). I'll keep an
eye on the buffer_anon. Looks like that is dumped in perf dump so I can go
back and look through graphite data for it.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Hello Robert,
On Mon, Mar 9, 2020 at 7:55 PM Robert Ruge <robert.ruge(a)deakin.edu.au>
wrote:
> For a 1.1PB raw cephfs system currently
storing 191TB of data and 390
million objects (mostly small Python, ML training
files etc.) how many MDS
servers should I be running?
>
> System is Nautilus 14.2.8.
>
>
>
> I ask because up to know I have run one MDS with one standby-replay
and
occasionally it blows up with large memory consumption, 60Gb+ even
though I have mds_cache_memory_limit = 32G and that was 16G until recently.
It of course tries to restart on another MDS node fails again and after
several attempts usually comes back up. Today I increased to two active
MDS’s but the question is what is the optimal number for a pretty active
system? The single MDS seemed to regularly run around 1400 req/s and I
often get up to six clients failing to respond to cache pressure.
Ideally, the only reason you should add more active MDS (increase
max_mds) is because you want to increase request throughput.
60GB RSS is not completely unexpected. A 32GB cache size would use
approximately 48GB (150%) RSS in a steady state situation. You may
ahve hit some kind of bug as others have reported which is causing the
cache size / anonymous memory to continually increase. You will need
to post more information about the client type/version, cache usage,
perf dumps, and workload to help diagnose.
https://github.com/ceph/ceph/pull/34571 may help if "ceph daemon mds.a
dump_mempools" shows buffer_anon uses lots of memory.
10:25 p.m.
On Thu, May 7, 2020 at 9:41 AM Robert LeBlanc <robert(a)leblancnet.us> wrote:
On Thu, May 7, 2020 at 6:22 AM Yan, Zheng
<ukernel(a)gmail.com> wrote:
So, when we have trouble with our MDS, there are spikes in both
buffer_anon_bytes (sometimes GiBs) and buffer_anon_items (sometimes
millions). There are some times that there is high bytes (~2GiB) used and
low items (~1k) and we don't seem to have big issues then. Only when the
items value goes high do we seem to get into the danger zone.
Will this PR apply cleanly to Nautilus?
Thanks,
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Thu, May 7, 2020 at 1:27 AM Patrick Donnelly
<pdonnell(a)redhat.com>
wrote:
We struggle with cache management as well and I just thought it was due to
our really old kernel clients (I'm sure that doesn't help). I'll keep an
eye on the buffer_anon. Looks like that is dumped in perf dump so I can go
back and look through graphite data for it.
Hello Robert,
On Mon, Mar 9, 2020 at 7:55 PM Robert Ruge <robert.ruge(a)deakin.edu.au>
wrote:
> For a 1.1PB raw cephfs system currently
storing 191TB of data and 390
million objects (mostly small Python, ML training
files etc.) how many MDS
servers should I be running?
>
> System is Nautilus 14.2.8.
>
>
>
> I ask because up to know I have run one MDS with one standby-replay
and
occasionally it blows up with large memory consumption, 60Gb+ even
though I have mds_cache_memory_limit = 32G and that was 16G until recently.
It of course tries to restart on another MDS node fails again and after
several attempts usually comes back up. Today I increased to two active
MDS’s but the question is what is the optimal number for a pretty active
system? The single MDS seemed to regularly run around 1400 req/s and I
often get up to six clients failing to respond to cache pressure.
Ideally, the only reason you should add more active MDS (increase
max_mds) is because you want to increase request throughput.
60GB RSS is not completely unexpected. A 32GB cache size would use
approximately 48GB (150%) RSS in a steady state situation. You may
ahve hit some kind of bug as others have reported which is causing the
cache size / anonymous memory to continually increase. You will need
to post more information about the client type/version, cache usage,
perf dumps, and workload to help diagnose.
https://github.com/ceph/ceph/pull/34571 may help if "ceph daemon mds.a
dump_mempools" shows buffer_anon uses lots of memory.
1450
days inactive
1508
days old
4 comments
4 participants
participants (4)
-
Patrick Donnelly -
Robert LeBlanc -
Robert Ruge -
Yan, Zheng