On Tue, Apr 14, 2020 at 11:45 PM Yan, Zheng <ukernel@gmail.com> wrote:
On Tue, Apr 14, 2020 at 9:41 PM Dan van der Ster <dan@vanderster.com> wrote:
On Tue, Apr 14, 2020 at 2:50 PM Dan van der Ster <dan@vanderster.com> wrote:
On Sun, Apr 12, 2020 at 9:33 PM Dan van der Ster <dan@vanderster.com> wrote:
Hi John,
Did you make any progress on investigating this?
Today I also saw huge relative buffer_anon usage on our 2 active mds's running 14.2.8:
"mempool": { "by_pool": { "bloom_filter": { "items": 2322, "bytes": 2322 }, ... "buffer_anon": { "items": 4947214, "bytes": 19785847411 }, ... "osdmap": { "items": 4036, "bytes": 89488 }, ... "mds_co": { "items": 9248718, "bytes": 157725128 }, ... }, "total": { "items": 14202290, "bytes": 19943664349 } }
That mds has `mds cache memory limit = 15353442304` and there was no health warning about the mds memory usage exceeding the limit. (I only noticed because some other crons on the mds's were going oom).
Patrick: is there any known memory leak in nautilus mds's ?
I restarted one MDS with ms_type = simple and that MDS maintained a normal amount of buffer_anon for several hours, while the other active MDS (with async ms type) saw its buffer_anon grow by some ~10GB overnight. So, it seems there are still memory leaks with ms_type = async in 14.2.8.
OTOH, the whole cluster is kinda broken now due to https://tracker.ceph.com/issues/45080, which may be related to the ms_type=simple .. I'm still debugging.
Indeed, the combination of msgr v2 and `ms type = simple` on a ceph-mds leads to deadlocked mds ops as soon as any osd restarts. Looks like we have to find the root cause of the memory leak rather than working around it with ms type = simple.
Dan
I opened https://tracker.ceph.com/issues/45090. It can explain the buffer_anon memory use.
please try https://github.com/ceph/ceph/pull/34571 if you can compile ceph from source. Regards Yan, Zheng
Regards Yan, Zheng
Cheers, Dan
Any tips to debug this further?
Cheers, Dan
On Wed, Mar 4, 2020 at 8:38 PM John Madden <jmadden.com@gmail.com> wrote:
Though it appears potentially(?) better, I'm still having issues with this on 14.2.8. Kick off the ~20 threads sequentially reading ~1M files and buffer_anon still grows apparently without bound.
mds.1 tcmalloc heap stats:------------------------------------------------ MALLOC: 53710413656 (51222.2 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 334028128 ( 318.6 MiB) Bytes in central cache freelist MALLOC: + 11210608 ( 10.7 MiB) Bytes in transfer cache freelist MALLOC: + 11105240 ( 10.6 MiB) Bytes in thread cache freelists MALLOC: + 77525152 ( 73.9 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 54144282784 (51636.0 MiB) Actual memory used (physical + swap) MALLOC: + 49963008 ( 47.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 54194245792 (51683.7 MiB) Virtual address space used MALLOC: MALLOC: 262021 Spans in use MALLOC: 18 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
The byte count appears to grow even as the item count drops, though the trend is for both to increase over the life of the workload: ceph daemon mds.1 dump_mempools | jq .mempool.by_pool.buffer_anon:
{ "items": 28045, "bytes": 24197601109 } { "items": 27132, "bytes": 24262495865 } { "items": 27105, "bytes": 24262537939 } { "items": 33309, "bytes": 29754507505 } { "items": 36160, "bytes": 31803033733 } { "items": 56772, "bytes": 51062350351 }
Is there further data/debug I can retrieve to help track this down?
On Wed, Feb 19, 2020 at 4:38 PM John Madden <jmadden.com@gmail.com> wrote:
Ah, no, I hadn't seen that. Patiently awaiting .8 then. Thanks!
On Mon, Feb 17, 2020 at 8:52 AM Dan van der Ster <dan@vanderster.com> wrote: > > On Mon, Feb 10, 2020 at 8:31 PM John Madden <jmadden.com@gmail.com> wrote: > > > > Upgraded to 14.2.7, doesn't appear to have affected the behavior. As requested: > > In case it wasn't clear -- the fix that Patrick mentioned was > postponed to 14.2.8. > > -- dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io