10 Feb
2020
10 Feb
'20
3:30 p.m.
Upgraded to 14.2.7, doesn't appear to have affected the behavior. As requested:
~$ ceph tell mds.mds1 heap stats
2020-02-10 16:52:44.313 7fbda2cae700 0 client.59208005
ms_handle_reset on v2:x.x.x.x:6800/3372494505
2020-02-10 16:52:44.337 7fbda3cb0700 0 client.59249562
ms_handle_reset on v2:x.x.x.x:6800/3372494505
mds.mds1 tcmalloc heap stats:------------------------------------------------
MALLOC: 50000388656 (47684.1 MiB) Bytes in use by application
MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 174879528 ( 166.8 MiB) Bytes in central cache freelist
MALLOC: + 14511680 ( 13.8 MiB) Bytes in transfer cache freelist
MALLOC: + 14089320 ( 13.4 MiB) Bytes in thread cache freelists
MALLOC: + 90534048 ( 86.3 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 50294403232 (47964.5 MiB) Actual memory used (physical + swap)
MALLOC: + 50987008 ( 48.6 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 50345390240 (48013.1 MiB) Virtual address space used
MALLOC:
MALLOC: 260018 Spans in use
MALLOC: 20 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
~$ ceph tell mds.mds1 heap release
2020-02-10 16:52:47.205 7f037eff5700 0 client.59249625
ms_handle_reset on v2:x.x.x.x:6800/3372494505
2020-02-10 16:52:47.237 7f037fff7700 0 client.59249634
ms_handle_reset on v2:x.x.x.x:6800/3372494505
mds.mds1 releasing free RAM back to system.
The pools over 15 minutes or so:
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 2045,
"bytes": 3069493686
}
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 2445,
"bytes": 3111162538
}
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 7850,
"bytes": 7658678767
}
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 12274,
"bytes": 11436728978
}
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 13747,
"bytes": 11539478519
}
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 14615,
"bytes": 13859676992
}
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 23267,
"bytes": 22290063830
}
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 44944,
"bytes": 40726959425
}
And one about a minute after the heap release showing continued growth:
~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 50694,
"bytes": 47343942094
}
This is on a single active MDS with 2 standbys, scan for about a
million files with about 20 parallel threads on two clients, open and
read each if it exists.
On Wed, Jan 22, 2020 at 8:25 AM John Madden <jmadden.com(a)gmail.com> wrote:
>
> > Couldn't John confirm that this is the issue by checking the heap stats and
triggering the release via
> >
> > ceph tell mds.mds1 heap stats
> > ceph tell mds.mds1 heap release
> >
> > (this would be much less disruptive than restarting the MDS)
>
> That was my first thought as well, but `release` doesn't appear to do
> anything in this case.
>
> John
17 Feb
17 Feb
9:51 a.m.
New subject: MDS: obscene buffer_anon memory use when scanning lots of files
On Mon, Feb 10, 2020 at 8:31 PM John Madden <jmadden.com(a)gmail.com> wrote:
Upgraded to 14.2.7, doesn't appear to have affected the behavior. As requested:
In case it wasn't clear -- the fix that Patrick mentioned was
postponed to 14.2.8.
-- dan
19 Feb
19 Feb
5:38 p.m.
New subject: MDS: obscene buffer_anon memory use when scanning lots of files
Ah, no, I hadn't seen that. Patiently awaiting .8 then. Thanks!
On Mon, Feb 17, 2020 at 8:52 AM Dan van der Ster <dan(a)vanderster.com> wrote:
>
> On Mon, Feb 10, 2020 at 8:31 PM John Madden <jmadden.com(a)gmail.com> wrote:
> >
> > Upgraded to 14.2.7, doesn't appear to have affected the behavior. As
requested:
>
> In case it wasn't clear -- the fix that Patrick mentioned was
> postponed to 14.2.8.
>
> -- dan
4 Mar
4 Mar
3:38 p.m.
New subject: MDS: obscene buffer_anon memory use when scanning lots of files
Though it appears potentially(?) better, I'm still having issues with
this on 14.2.8. Kick off the ~20 threads sequentially reading ~1M
files and buffer_anon still grows apparently without bound.
mds.1 tcmalloc heap stats:------------------------------------------------
MALLOC: 53710413656 (51222.2 MiB) Bytes in use by application
MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 334028128 ( 318.6 MiB) Bytes in central cache freelist
MALLOC: + 11210608 ( 10.7 MiB) Bytes in transfer cache freelist
MALLOC: + 11105240 ( 10.6 MiB) Bytes in thread cache freelists
MALLOC: + 77525152 ( 73.9 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 54144282784 (51636.0 MiB) Actual memory used (physical + swap)
MALLOC: + 49963008 ( 47.6 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 54194245792 (51683.7 MiB) Virtual address space used
MALLOC:
MALLOC: 262021 Spans in use
MALLOC: 18 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
The byte count appears to grow even as the item count drops, though
the trend is for both to increase over the life of the workload:
ceph daemon mds.1 dump_mempools | jq .mempool.by_pool.buffer_anon:
{
"items": 28045,
"bytes": 24197601109
}
{
"items": 27132,
"bytes": 24262495865
}
{
"items": 27105,
"bytes": 24262537939
}
{
"items": 33309,
"bytes": 29754507505
}
{
"items": 36160,
"bytes": 31803033733
}
{
"items": 56772,
"bytes": 51062350351
}
Is there further data/debug I can retrieve to help track this down?
On Wed, Feb 19, 2020 at 4:38 PM John Madden <jmadden.com(a)gmail.com> wrote:
>
> Ah, no, I hadn't seen that. Patiently awaiting .8 then. Thanks!
>
> On Mon, Feb 17, 2020 at 8:52 AM Dan van der Ster <dan(a)vanderster.com> wrote:
> >
> > On Mon, Feb 10, 2020 at 8:31 PM John Madden <jmadden.com(a)gmail.com>
wrote:
> > >
> > > Upgraded to 14.2.7, doesn't appear to have affected the behavior. As
requested:
> >
> > In case it wasn't clear -- the fix that Patrick mentioned was
> > postponed to 14.2.8.
> >
> > -- dan
12 Apr
12 Apr
4:33 p.m.
New subject: MDS: obscene buffer_anon memory use when scanning lots of files
Hi John,
Did you make any progress on investigating this?
Today I also saw huge relative buffer_anon usage on our 2 active mds's
running 14.2.8:
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 2322,
"bytes": 2322
},
...
"buffer_anon": {
"items": 4947214,
"bytes": 19785847411
},
...
"osdmap": {
"items": 4036,
"bytes": 89488
},
...
"mds_co": {
"items": 9248718,
"bytes": 157725128
},
...
},
"total": {
"items": 14202290,
"bytes": 19943664349
}
}
That mds has `mds cache memory limit = 15353442304` and there was no
health warning about the mds memory usage exceeding the limit.
(I only noticed because some other crons on the mds's were going oom).
Patrick: is there any known memory leak in nautilus mds's ?
Any tips to debug this further?
Cheers, Dan
On Wed, Mar 4, 2020 at 8:38 PM John Madden <jmadden.com(a)gmail.com> wrote:
>
> Though it appears potentially(?) better, I'm still having issues with
> this on 14.2.8. Kick off the ~20 threads sequentially reading ~1M
> files and buffer_anon still grows apparently without bound.
>
> mds.1 tcmalloc heap stats:------------------------------------------------
> MALLOC: 53710413656 (51222.2 MiB) Bytes in use by application
> MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> MALLOC: + 334028128 ( 318.6 MiB) Bytes in central cache freelist
> MALLOC: + 11210608 ( 10.7 MiB) Bytes in transfer cache freelist
> MALLOC: + 11105240 ( 10.6 MiB) Bytes in thread cache freelists
> MALLOC: + 77525152 ( 73.9 MiB) Bytes in malloc metadata
> MALLOC: ------------
> MALLOC: = 54144282784 (51636.0 MiB) Actual memory used (physical + swap)
> MALLOC: + 49963008 ( 47.6 MiB) Bytes released to OS (aka unmapped)
> MALLOC: ------------
> MALLOC: = 54194245792 (51683.7 MiB) Virtual address space used
> MALLOC:
> MALLOC: 262021 Spans in use
> MALLOC: 18 Thread heaps in use
> MALLOC: 8192 Tcmalloc page size
> ------------------------------------------------
>
> The byte count appears to grow even as the item count drops, though
> the trend is for both to increase over the life of the workload:
> ceph daemon mds.1 dump_mempools | jq .mempool.by_pool.buffer_anon:
>
> {
> "items": 28045,
> "bytes": 24197601109
> }
> {
> "items": 27132,
> "bytes": 24262495865
> }
> {
> "items": 27105,
> "bytes": 24262537939
> }
> {
> "items": 33309,
> "bytes": 29754507505
> }
> {
> "items": 36160,
> "bytes": 31803033733
> }
> {
> "items": 56772,
> "bytes": 51062350351
> }
>
> Is there further data/debug I can retrieve to help track this down?
>
>
> On Wed, Feb 19, 2020 at 4:38 PM John Madden <jmadden.com(a)gmail.com> wrote:
> >
> > Ah, no, I hadn't seen that. Patiently awaiting .8 then. Thanks!
> >
> > On Mon, Feb 17, 2020 at 8:52 AM Dan van der Ster <dan(a)vanderster.com>
wrote:
> > >
> > > On Mon, Feb 10, 2020 at 8:31 PM John Madden <jmadden.com(a)gmail.com>
wrote:
> > > >
> > > > Upgraded to 14.2.7, doesn't appear to have affected the behavior.
As requested:
> > >
> > > In case it wasn't clear -- the fix that Patrick mentioned was
> > > postponed to 14.2.8.
> > >
> > > -- dan
14 Apr
14 Apr
9:50 a.m.
New subject: MDS: obscene buffer_anon memory use when scanning lots of files
On Sun, Apr 12, 2020 at 9:33 PM Dan van der Ster <dan(a)vanderster.com> wrote:
Hi John,
Did you make any progress on investigating this?
Today I also saw huge relative buffer_anon usage on our 2 active mds's
running 14.2.8:
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 2322,
"bytes": 2322
},
...
"buffer_anon": {
"items": 4947214,
"bytes": 19785847411
},
...
"osdmap": {
"items": 4036,
"bytes": 89488
},
...
"mds_co": {
"items": 9248718,
"bytes": 157725128
},
...
},
"total": {
"items": 14202290,
"bytes": 19943664349
}
}
That mds has `mds cache memory limit = 15353442304` and there was no
health warning about the mds memory usage exceeding the limit.
(I only noticed because some other crons on the mds's were going oom).
Patrick: is there any known memory leak in nautilus mds's ?
I restarted one MDS with ms_type = simple and that MDS maintained a
normal amount of buffer_anon for several hours, while the other active
MDS (with async ms type) saw its buffer_anon grow by some ~10GB
overnight.
So, it seems there are still memory leaks with ms_type = async in 14.2.8.
OTOH, the whole cluster is kinda broken now due to
https://tracker.ceph.com/issues/45080, which may be related to the
ms_type=simple .. I'm still debugging.
Cheers, Dan
> Any tips to debug this further?
>
> Cheers, Dan
>
> On Wed, Mar 4, 2020 at 8:38 PM John Madden <jmadden.com(a)gmail.com> wrote:
> >
> > Though it appears potentially(?) better, I'm still having issues with
> > this on 14.2.8. Kick off the ~20 threads sequentially reading ~1M
> > files and buffer_anon still grows apparently without bound.
> >
> > mds.1 tcmalloc heap stats:------------------------------------------------
> > MALLOC: 53710413656 (51222.2 MiB) Bytes in use by application
> > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> > MALLOC: + 334028128 ( 318.6 MiB) Bytes in central cache freelist
> > MALLOC: + 11210608 ( 10.7 MiB) Bytes in transfer cache freelist
> > MALLOC: + 11105240 ( 10.6 MiB) Bytes in thread cache freelists
> > MALLOC: + 77525152 ( 73.9 MiB) Bytes in malloc metadata
> > MALLOC: ------------
> > MALLOC: = 54144282784 (51636.0 MiB) Actual memory used (physical + swap)
> > MALLOC: + 49963008 ( 47.6 MiB) Bytes released to OS (aka unmapped)
> > MALLOC: ------------
> > MALLOC: = 54194245792 (51683.7 MiB) Virtual address space used
> > MALLOC:
> > MALLOC: 262021 Spans in use
> > MALLOC: 18 Thread heaps in use
> > MALLOC: 8192 Tcmalloc page size
> > ------------------------------------------------
> >
> > The byte count appears to grow even as the item count drops, though
> > the trend is for both to increase over the life of the workload:
> > ceph daemon mds.1 dump_mempools | jq .mempool.by_pool.buffer_anon:
> >
> > {
> > "items": 28045,
> > "bytes": 24197601109
> > }
> > {
> > "items": 27132,
> > "bytes": 24262495865
> > }
> > {
> > "items": 27105,
> > "bytes": 24262537939
> > }
> > {
> > "items": 33309,
> > "bytes": 29754507505
> > }
> > {
> > "items": 36160,
> > "bytes": 31803033733
> > }
> > {
> > "items": 56772,
> > "bytes": 51062350351
> > }
> >
> > Is there further data/debug I can retrieve to help track this down?
> >
> >
> > On Wed, Feb 19, 2020 at 4:38 PM John Madden <jmadden.com(a)gmail.com>
wrote:
> > >
> > > Ah, no, I hadn't seen that. Patiently awaiting .8 then. Thanks!
> > >
> > > On Mon, Feb 17, 2020 at 8:52 AM Dan van der Ster <dan(a)vanderster.com>
wrote:
> > > >
> > > > On Mon, Feb 10, 2020 at 8:31 PM John Madden
<jmadden.com(a)gmail.com> wrote:
> > > > >
> > > > > Upgraded to 14.2.7, doesn't appear to have affected the
behavior. As requested:
> > > >
> > > > In case it wasn't clear -- the fix that Patrick mentioned was
> > > > postponed to 14.2.8.
> > > >
> > > > -- dan
10:40 a.m.
New subject: MDS: obscene buffer_anon memory use when scanning lots of files
On Tue, Apr 14, 2020 at 2:50 PM Dan van der Ster <dan(a)vanderster.com> wrote:
On Sun, Apr 12, 2020 at 9:33 PM Dan van der Ster <dan(a)vanderster.com> wrote:
Indeed, the combination of msgr v2 and `ms type = simple` on a
ceph-mds leads to deadlocked mds ops as soon as any osd restarts.
Looks like we have to find the root cause of the memory leak rather
than working around it with ms type = simple.
Dan
>
> Cheers, Dan
>
> > Any tips to debug this further?
> >
> > Cheers, Dan
> >
> > On Wed, Mar 4, 2020 at 8:38 PM John Madden <jmadden.com(a)gmail.com> wrote:
> > >
> > > Though it appears potentially(?) better, I'm still having issues with
> > > this on 14.2.8. Kick off the ~20 threads sequentially reading ~1M
> > > files and buffer_anon still grows apparently without bound.
> > >
> > > mds.1 tcmalloc heap stats:------------------------------------------------
> > > MALLOC: 53710413656 (51222.2 MiB) Bytes in use by application
> > > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> > > MALLOC: + 334028128 ( 318.6 MiB) Bytes in central cache freelist
> > > MALLOC: + 11210608 ( 10.7 MiB) Bytes in transfer cache freelist
> > > MALLOC: + 11105240 ( 10.6 MiB) Bytes in thread cache freelists
> > > MALLOC: + 77525152 ( 73.9 MiB) Bytes in malloc metadata
> > > MALLOC: ------------
> > > MALLOC: = 54144282784 (51636.0 MiB) Actual memory used (physical + swap)
> > > MALLOC: + 49963008 ( 47.6 MiB) Bytes released to OS (aka unmapped)
> > > MALLOC: ------------
> > > MALLOC: = 54194245792 (51683.7 MiB) Virtual address space used
> > > MALLOC:
> > > MALLOC: 262021 Spans in use
> > > MALLOC: 18 Thread heaps in use
> > > MALLOC: 8192 Tcmalloc page size
> > > ------------------------------------------------
> > >
> > > The byte count appears to grow even as the item count drops, though
> > > the trend is for both to increase over the life of the workload:
> > > ceph daemon mds.1 dump_mempools | jq .mempool.by_pool.buffer_anon:
> > >
> > > {
> > > "items": 28045,
> > > "bytes": 24197601109
> > > }
> > > {
> > > "items": 27132,
> > > "bytes": 24262495865
> > > }
> > > {
> > > "items": 27105,
> > > "bytes": 24262537939
> > > }
> > > {
> > > "items": 33309,
> > > "bytes": 29754507505
> > > }
> > > {
> > > "items": 36160,
> > > "bytes": 31803033733
> > > }
> > > {
> > > "items": 56772,
> > > "bytes": 51062350351
> > > }
> > >
> > > Is there further data/debug I can retrieve to help track this down?
> > >
> > >
> > > On Wed, Feb 19, 2020 at 4:38 PM John Madden <jmadden.com(a)gmail.com>
wrote:
> > > >
> > > > Ah, no, I hadn't seen that. Patiently awaiting .8 then. Thanks!
> > > >
> > > > On Mon, Feb 17, 2020 at 8:52 AM Dan van der Ster
<dan(a)vanderster.com> wrote:
> > > > >
> > > > > On Mon, Feb 10, 2020 at 8:31 PM John Madden
<jmadden.com(a)gmail.com> wrote:
> > > > > >
> > > > > > Upgraded to 14.2.7, doesn't appear to have affected the
behavior. As requested:
> > > > >
> > > > > In case it wasn't clear -- the fix that Patrick mentioned
was
> > > > > postponed to 14.2.8.
> > > > >
> > > > > -- dan
Hi John,
Did you make any progress on investigating this?
Today I also saw huge relative buffer_anon usage on our 2 active mds's
running 14.2.8:
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 2322,
"bytes": 2322
},
...
"buffer_anon": {
"items": 4947214,
"bytes": 19785847411
},
...
"osdmap": {
"items": 4036,
"bytes": 89488
},
...
"mds_co": {
"items": 9248718,
"bytes": 157725128
},
...
},
"total": {
"items": 14202290,
"bytes": 19943664349
}
}
That mds has `mds cache memory limit = 15353442304` and there was no
health warning about the mds memory usage exceeding the limit.
(I only noticed because some other crons on the mds's were going oom).
Patrick: is there any known memory leak in nautilus mds's ?
I restarted one MDS with ms_type = simple and that MDS maintained a
normal amount of buffer_anon for several hours, while the other active
MDS (with async ms type) saw its buffer_anon grow by some ~10GB
overnight.
So, it seems there are still memory leaks with ms_type = async in 14.2.8.
OTOH, the whole cluster is kinda broken now due to
https://tracker.ceph.com/issues/45080, which may be related to the
ms_type=simple .. I'm still debugging. 
12:45 p.m.
New subject: MDS: obscene buffer_anon memory use when scanning lots of files
On Tue, Apr 14, 2020 at 9:41 PM Dan van der Ster <dan(a)vanderster.com> wrote:
On Tue, Apr 14, 2020 at 2:50 PM Dan van der Ster <dan(a)vanderster.com> wrote:
I opened https://tracker.ceph.com/issues/45090. It can explain the
buffer_anon memory use.
Regards
Yan, Zheng
>
> >
> > Cheers, Dan
> >
> > > Any tips to debug this further?
> > >
> > > Cheers, Dan
> > >
> > > On Wed, Mar 4, 2020 at 8:38 PM John Madden <jmadden.com(a)gmail.com>
wrote:
> > > >
> > > > Though it appears potentially(?) better, I'm still having issues
with
> > > > this on 14.2.8. Kick off the ~20 threads sequentially reading ~1M
> > > > files and buffer_anon still grows apparently without bound.
> > > >
> > > > mds.1 tcmalloc heap
stats:------------------------------------------------
> > > > MALLOC: 53710413656 (51222.2 MiB) Bytes in use by application
> > > > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> > > > MALLOC: + 334028128 ( 318.6 MiB) Bytes in central cache freelist
> > > > MALLOC: + 11210608 ( 10.7 MiB) Bytes in transfer cache freelist
> > > > MALLOC: + 11105240 ( 10.6 MiB) Bytes in thread cache freelists
> > > > MALLOC: + 77525152 ( 73.9 MiB) Bytes in malloc metadata
> > > > MALLOC: ------------
> > > > MALLOC: = 54144282784 (51636.0 MiB) Actual memory used (physical +
swap)
> > > > MALLOC: + 49963008 ( 47.6 MiB) Bytes released to OS (aka
unmapped)
> > > > MALLOC: ------------
> > > > MALLOC: = 54194245792 (51683.7 MiB) Virtual address space used
> > > > MALLOC:
> > > > MALLOC: 262021 Spans in use
> > > > MALLOC: 18 Thread heaps in use
> > > > MALLOC: 8192 Tcmalloc page size
> > > > ------------------------------------------------
> > > >
> > > > The byte count appears to grow even as the item count drops, though
> > > > the trend is for both to increase over the life of the workload:
> > > > ceph daemon mds.1 dump_mempools | jq .mempool.by_pool.buffer_anon:
> > > >
> > > > {
> > > > "items": 28045,
> > > > "bytes": 24197601109
> > > > }
> > > > {
> > > > "items": 27132,
> > > > "bytes": 24262495865
> > > > }
> > > > {
> > > > "items": 27105,
> > > > "bytes": 24262537939
> > > > }
> > > > {
> > > > "items": 33309,
> > > > "bytes": 29754507505
> > > > }
> > > > {
> > > > "items": 36160,
> > > > "bytes": 31803033733
> > > > }
> > > > {
> > > > "items": 56772,
> > > > "bytes": 51062350351
> > > > }
> > > >
> > > > Is there further data/debug I can retrieve to help track this down?
> > > >
> > > >
> > > > On Wed, Feb 19, 2020 at 4:38 PM John Madden
<jmadden.com(a)gmail.com> wrote:
> > > > >
> > > > > Ah, no, I hadn't seen that. Patiently awaiting .8 then.
Thanks!
> > > > >
> > > > > On Mon, Feb 17, 2020 at 8:52 AM Dan van der Ster
<dan(a)vanderster.com> wrote:
> > > > > >
> > > > > > On Mon, Feb 10, 2020 at 8:31 PM John Madden
<jmadden.com(a)gmail.com> wrote:
> > > > > > >
> > > > > > > Upgraded to 14.2.7, doesn't appear to have affected
the behavior. As requested:
> > > > > >
> > > > > > In case it wasn't clear -- the fix that Patrick
mentioned was
> > > > > > postponed to 14.2.8.
> > > > > >
> > > > > > -- dan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
On Sun, Apr 12, 2020 at 9:33 PM Dan van der Ster <dan(a)vanderster.com> wrote:
Indeed, the combination of msgr v2 and `ms type = simple` on a
ceph-mds leads to deadlocked mds ops as soon as any osd restarts.
Looks like we have to find the root cause of the memory leak rather
than working around it with ms type = simple.
Dan
Hi John,
Did you make any progress on investigating this?
Today I also saw huge relative buffer_anon usage on our 2 active mds's
running 14.2.8:
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 2322,
"bytes": 2322
},
...
"buffer_anon": {
"items": 4947214,
"bytes": 19785847411
},
...
"osdmap": {
"items": 4036,
"bytes": 89488
},
...
"mds_co": {
"items": 9248718,
"bytes": 157725128
},
...
},
"total": {
"items": 14202290,
"bytes": 19943664349
}
}
That mds has `mds cache memory limit = 15353442304` and there was no
health warning about the mds memory usage exceeding the limit.
(I only noticed because some other crons on the mds's were going oom).
Patrick: is there any known memory leak in nautilus mds's ?
I restarted one MDS with ms_type = simple and that MDS maintained a
normal amount of buffer_anon for several hours, while the other active
MDS (with async ms type) saw its buffer_anon grow by some ~10GB
overnight.
So, it seems there are still memory leaks with ms_type = async in 14.2.8.
OTOH, the whole cluster is kinda broken now due to
https://tracker.ceph.com/issues/45080, which may be related to the
ms_type=simple .. I'm still debugging.
15 Apr
15 Apr
12:01 p.m.
New subject: MDS: obscene buffer_anon memory use when scanning lots of files
On Tue, Apr 14, 2020 at 11:45 PM Yan, Zheng <ukernel(a)gmail.com> wrote:
On Tue, Apr 14, 2020 at 9:41 PM Dan van der Ster <dan(a)vanderster.com> wrote:
please try https://github.com/ceph/ceph/pull/34571 if you can compile
ceph from source.
Regards
Yan, Zheng
> Regards
> Yan, Zheng
>
> >
> > >
> > > Cheers, Dan
> > >
> > > > Any tips to debug this further?
> > > >
> > > > Cheers, Dan
> > > >
> > > > On Wed, Mar 4, 2020 at 8:38 PM John Madden
<jmadden.com(a)gmail.com> wrote:
> > > > >
> > > > > Though it appears potentially(?) better, I'm still having
issues with
> > > > > this on 14.2.8. Kick off the ~20 threads sequentially reading
~1M
> > > > > files and buffer_anon still grows apparently without bound.
> > > > >
> > > > > mds.1 tcmalloc heap
stats:------------------------------------------------
> > > > > MALLOC: 53710413656 (51222.2 MiB) Bytes in use by application
> > > > > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> > > > > MALLOC: + 334028128 ( 318.6 MiB) Bytes in central cache
freelist
> > > > > MALLOC: + 11210608 ( 10.7 MiB) Bytes in transfer cache
freelist
> > > > > MALLOC: + 11105240 ( 10.6 MiB) Bytes in thread cache
freelists
> > > > > MALLOC: + 77525152 ( 73.9 MiB) Bytes in malloc metadata
> > > > > MALLOC: ------------
> > > > > MALLOC: = 54144282784 (51636.0 MiB) Actual memory used (physical
+ swap)
> > > > > MALLOC: + 49963008 ( 47.6 MiB) Bytes released to OS (aka
unmapped)
> > > > > MALLOC: ------------
> > > > > MALLOC: = 54194245792 (51683.7 MiB) Virtual address space used
> > > > > MALLOC:
> > > > > MALLOC: 262021 Spans in use
> > > > > MALLOC: 18 Thread heaps in use
> > > > > MALLOC: 8192 Tcmalloc page size
> > > > > ------------------------------------------------
> > > > >
> > > > > The byte count appears to grow even as the item count drops,
though
> > > > > the trend is for both to increase over the life of the workload:
> > > > > ceph daemon mds.1 dump_mempools | jq
.mempool.by_pool.buffer_anon:
> > > > >
> > > > > {
> > > > > "items": 28045,
> > > > > "bytes": 24197601109
> > > > > }
> > > > > {
> > > > > "items": 27132,
> > > > > "bytes": 24262495865
> > > > > }
> > > > > {
> > > > > "items": 27105,
> > > > > "bytes": 24262537939
> > > > > }
> > > > > {
> > > > > "items": 33309,
> > > > > "bytes": 29754507505
> > > > > }
> > > > > {
> > > > > "items": 36160,
> > > > > "bytes": 31803033733
> > > > > }
> > > > > {
> > > > > "items": 56772,
> > > > > "bytes": 51062350351
> > > > > }
> > > > >
> > > > > Is there further data/debug I can retrieve to help track this
down?
> > > > >
> > > > >
> > > > > On Wed, Feb 19, 2020 at 4:38 PM John Madden
<jmadden.com(a)gmail.com> wrote:
> > > > > >
> > > > > > Ah, no, I hadn't seen that. Patiently awaiting .8 then.
Thanks!
> > > > > >
> > > > > > On Mon, Feb 17, 2020 at 8:52 AM Dan van der Ster
<dan(a)vanderster.com> wrote:
> > > > > > >
> > > > > > > On Mon, Feb 10, 2020 at 8:31 PM John Madden
<jmadden.com(a)gmail.com> wrote:
> > > > > > > >
> > > > > > > > Upgraded to 14.2.7, doesn't appear to have
affected the behavior. As requested:
> > > > > > >
> > > > > > > In case it wasn't clear -- the fix that Patrick
mentioned was
> > > > > > > postponed to 14.2.8.
> > > > > > >
> > > > > > > -- dan
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
On Tue, Apr 14, 2020 at 2:50 PM Dan van der Ster <dan(a)vanderster.com> wrote:
I opened https://tracker.ceph.com/issues/45090. It can explain the
buffer_anon memory use.
On Sun, Apr 12, 2020 at 9:33 PM Dan van der Ster <dan(a)vanderster.com> wrote:
Indeed, the combination of msgr v2 and `ms type = simple` on a
ceph-mds leads to deadlocked mds ops as soon as any osd restarts.
Looks like we have to find the root cause of the memory leak rather
than working around it with ms type = simple.
Dan
Hi John,
Did you make any progress on investigating this?
Today I also saw huge relative buffer_anon usage on our 2 active mds's
running 14.2.8:
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 2322,
"bytes": 2322
},
...
"buffer_anon": {
"items": 4947214,
"bytes": 19785847411
},
...
"osdmap": {
"items": 4036,
"bytes": 89488
},
...
"mds_co": {
"items": 9248718,
"bytes": 157725128
},
...
},
"total": {
"items": 14202290,
"bytes": 19943664349
}
}
That mds has `mds cache memory limit = 15353442304` and there was no
health warning about the mds memory usage exceeding the limit.
(I only noticed because some other crons on the mds's were going oom).
Patrick: is there any known memory leak in nautilus mds's ?
I restarted one MDS with ms_type = simple and that MDS maintained a
normal amount of buffer_anon for several hours, while the other active
MDS (with async ms type) saw its buffer_anon grow by some ~10GB
overnight.
So, it seems there are still memory leaks with ms_type = async in 14.2.8.
OTOH, the whole cluster is kinda broken now due to
https://tracker.ceph.com/issues/45080, which may be related to the
ms_type=simple .. I'm still debugging.
1477
days inactive
1542
days old
8 comments
3 participants
participants (3)
-
Dan van der Ster -
John Madden -
Yan, Zheng