Hi all, on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load. What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats: Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } } Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Am I looking at a memory leak here or are these heap stats expected? I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Hi Frank, So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though. Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise". Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well. Mark On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Mark, thanks for the quick answer. I would try the memory profiler if I could find any documentation on it. In fact, I just guessed the "heap stats" command and have a hard time finding anything on the OSD daemon commands. Could you possibly point me to something? Also how to interpret the mempools? Is it correct to say that out of the memory_target only the mempools total is actually used and the remaining memory is lost due to leaks? For example, for OSD 256 I get the stats below after just 2 months uptime. Am I looking at a 5.5GB memory leak here? # ceph config get osd.256 osd_memory_target 8589934592 # ceph daemon osd.256 heap stats osd.256 tcmalloc heap stats:------------------------------------------------ MALLOC: 7216067616 ( 6881.8 MiB) Bytes in use by application MALLOC: + 229376 ( 0.2 MiB) Bytes in page heap freelist MALLOC: + 1222913888 ( 1166.3 MiB) Bytes in central cache freelist MALLOC: + 278016 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 18446744073692937856 (17592186044400.2 MiB) Bytes in thread cache freelists MALLOC: + 52166656 ( 49.8 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 8475041792 ( 8082.4 MiB) Actual memory used (physical + swap) MALLOC: + 2010464256 ( 1917.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 10485506048 ( 9999.8 MiB) Virtual address space used MALLOC: MALLOC: 765182 Spans in use MALLOC: 48 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.256 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 2300682, "bytes": 18405456 }, "bluestore_cache_data": { "items": 52390, "bytes": 306843648 }, "bluestore_cache_onode": { "items": 256153, "bytes": 145494904 }, "bluestore_cache_other": { "items": 92199353, "bytes": 656620069 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 4, "bytes": 2752 }, "bluestore_writing_deferred": { "items": 122, "bytes": 1864924 }, "bluestore_writing": { "items": 3673, "bytes": 18440192 }, "bluefs": { "items": 11867, "bytes": 220504 }, "buffer_anon": { "items": 353734, "bytes": 1180837372 }, "buffer_meta": { "items": 91646, "bytes": 5865344 }, "osd": { "items": 134, "bytes": 1557616 }, "osd_mapbl": { "items": 84, "bytes": 8479562 }, "osd_pglog": { "items": 487004, "bytes": 166094788 }, "osdmap": { "items": 117697, "bytes": 2080280 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 95874543, "bytes": 2512807411 } } } Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 13 July 2020 15:39:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: OSD memory leak? Hi Frank, So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though. Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise". Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well. Mark On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, These might help: https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/ https://gperftools.github.io/gperftools/heapprofile.html https://gperftools.github.io/gperftools/heap_checker.html Regarding the mempools, they don't track all of the memory usage in Ceph, only things that were allocated using mempools. There are many other things (rocksdb block cache for example) that don't use them. It's only giving you a partial picture of memory usage. In your example below, that byte value from the thread cache freelist looks very wrong. Ignoring that for a moment though, there's a ton of memory that's been unmapped and released to the OS, but hasn't been reclaimed by the kernel. That's either because the kernel doesn't have enough memory pressure to bother reclaiming it, or because it's all fragmented chunks of a huge page that the kernel can't fully reclaim. That tells me you should definitely be looking at the transparent huge page (THP) configuration on your nodes. Looking back at batrick's PR that disables THP for Ceph, it looks like we only backported it to nautilus but not mimic. On that topic, have you considered upgrading to Nautilus? Mark On 7/14/20 2:56 AM, Frank Schilder wrote:
Dear Mark,
thanks for the quick answer. I would try the memory profiler if I could find any documentation on it. In fact, I just guessed the "heap stats" command and have a hard time finding anything on the OSD daemon commands. Could you possibly point me to something? Also how to interpret the mempools? Is it correct to say that out of the memory_target only the mempools total is actually used and the remaining memory is lost due to leaks?
For example, for OSD 256 I get the stats below after just 2 months uptime. Am I looking at a 5.5GB memory leak here?
# ceph config get osd.256 osd_memory_target 8589934592
# ceph daemon osd.256 heap stats osd.256 tcmalloc heap stats:------------------------------------------------ MALLOC: 7216067616 ( 6881.8 MiB) Bytes in use by application MALLOC: + 229376 ( 0.2 MiB) Bytes in page heap freelist MALLOC: + 1222913888 ( 1166.3 MiB) Bytes in central cache freelist MALLOC: + 278016 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 18446744073692937856 (17592186044400.2 MiB) Bytes in thread cache freelists MALLOC: + 52166656 ( 49.8 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 8475041792 ( 8082.4 MiB) Actual memory used (physical + swap) MALLOC: + 2010464256 ( 1917.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 10485506048 ( 9999.8 MiB) Virtual address space used MALLOC: MALLOC: 765182 Spans in use MALLOC: 48 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
# ceph daemon osd.256 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 2300682, "bytes": 18405456 }, "bluestore_cache_data": { "items": 52390, "bytes": 306843648 }, "bluestore_cache_onode": { "items": 256153, "bytes": 145494904 }, "bluestore_cache_other": { "items": 92199353, "bytes": 656620069 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 4, "bytes": 2752 }, "bluestore_writing_deferred": { "items": 122, "bytes": 1864924 }, "bluestore_writing": { "items": 3673, "bytes": 18440192 }, "bluefs": { "items": 11867, "bytes": 220504 }, "buffer_anon": { "items": 353734, "bytes": 1180837372 }, "buffer_meta": { "items": 91646, "bytes": 5865344 }, "osd": { "items": 134, "bytes": 1557616 }, "osd_mapbl": { "items": 84, "bytes": 8479562 }, "osd_pglog": { "items": 487004, "bytes": 166094788 }, "osdmap": { "items": 117697, "bytes": 2080280 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 95874543, "bytes": 2512807411 } } }
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 13 July 2020 15:39:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: OSD memory leak?
Hi Frank,
So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though.
Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise".
Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well.
Mark
On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Mark, thanks for the info. I forgot a few answers: THPes are disabled (set to "never"). The kernel almost certainly doesn't reclaim because there is not enough pressure yet. We have 268 OSDs. I would not consider this much. We plan to triple that soonish. In the past, the minimum recommendation was 1GB RAM per HDD blue store OSD. I'm actually not really happy about that this has been quadrupled for not really convincing reasons. Compared with other storage systems, the increase in minimum requirements really start making ceph expensive. We have set the OSDs to use the bitmap allocator. Is the fact that we get tcmalloc stats a contradiction to this? I did not consider upgrading from mimic, because a lot of people report stability issues that might be caused by a regression in the message queueing. There was a longer e-mail about clusters from nautilus and higher collapsing under trivial amounts of rebalancing, pool deletion and other admin tasks. Before I consider upgrading, I want to test this on a lab cluster we plan to set up soon. I will look at the memory profiling. If one can use this on a production system, I will give it a go. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 14 July 2020 14:48:36 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: OSD memory leak? Hi Frank, These might help: https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/ https://gperftools.github.io/gperftools/heapprofile.html https://gperftools.github.io/gperftools/heap_checker.html Regarding the mempools, they don't track all of the memory usage in Ceph, only things that were allocated using mempools. There are many other things (rocksdb block cache for example) that don't use them. It's only giving you a partial picture of memory usage. In your example below, that byte value from the thread cache freelist looks very wrong. Ignoring that for a moment though, there's a ton of memory that's been unmapped and released to the OS, but hasn't been reclaimed by the kernel. That's either because the kernel doesn't have enough memory pressure to bother reclaiming it, or because it's all fragmented chunks of a huge page that the kernel can't fully reclaim. That tells me you should definitely be looking at the transparent huge page (THP) configuration on your nodes. Looking back at batrick's PR that disables THP for Ceph, it looks like we only backported it to nautilus but not mimic. On that topic, have you considered upgrading to Nautilus? Mark On 7/14/20 2:56 AM, Frank Schilder wrote:
Dear Mark,
thanks for the quick answer. I would try the memory profiler if I could find any documentation on it. In fact, I just guessed the "heap stats" command and have a hard time finding anything on the OSD daemon commands. Could you possibly point me to something? Also how to interpret the mempools? Is it correct to say that out of the memory_target only the mempools total is actually used and the remaining memory is lost due to leaks?
For example, for OSD 256 I get the stats below after just 2 months uptime. Am I looking at a 5.5GB memory leak here?
# ceph config get osd.256 osd_memory_target 8589934592
# ceph daemon osd.256 heap stats osd.256 tcmalloc heap stats:------------------------------------------------ MALLOC: 7216067616 ( 6881.8 MiB) Bytes in use by application MALLOC: + 229376 ( 0.2 MiB) Bytes in page heap freelist MALLOC: + 1222913888 ( 1166.3 MiB) Bytes in central cache freelist MALLOC: + 278016 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 18446744073692937856 (17592186044400.2 MiB) Bytes in thread cache freelists MALLOC: + 52166656 ( 49.8 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 8475041792 ( 8082.4 MiB) Actual memory used (physical + swap) MALLOC: + 2010464256 ( 1917.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 10485506048 ( 9999.8 MiB) Virtual address space used MALLOC: MALLOC: 765182 Spans in use MALLOC: 48 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
# ceph daemon osd.256 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 2300682, "bytes": 18405456 }, "bluestore_cache_data": { "items": 52390, "bytes": 306843648 }, "bluestore_cache_onode": { "items": 256153, "bytes": 145494904 }, "bluestore_cache_other": { "items": 92199353, "bytes": 656620069 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 4, "bytes": 2752 }, "bluestore_writing_deferred": { "items": 122, "bytes": 1864924 }, "bluestore_writing": { "items": 3673, "bytes": 18440192 }, "bluefs": { "items": 11867, "bytes": 220504 }, "buffer_anon": { "items": 353734, "bytes": 1180837372 }, "buffer_meta": { "items": 91646, "bytes": 5865344 }, "osd": { "items": 134, "bytes": 1557616 }, "osd_mapbl": { "items": 84, "bytes": 8479562 }, "osd_pglog": { "items": 487004, "bytes": 166094788 }, "osdmap": { "items": 117697, "bytes": 2080280 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 95874543, "bytes": 2512807411 } } }
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 13 July 2020 15:39:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: OSD memory leak?
Hi Frank,
So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though.
Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise".
Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well.
Mark
On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 7/14/20 8:12 AM, Frank Schilder wrote:
Dear Mark,
thanks for the info. I forgot a few answers:
THPes are disabled (set to "never"). The kernel almost certainly doesn't reclaim because there is not enough pressure yet.
We have 268 OSDs. I would not consider this much. We plan to triple that soonish. In the past, the minimum recommendation was 1GB RAM per HDD blue store OSD. I'm actually not really happy about that this has been quadrupled for not really convincing reasons. Compared with other storage systems, the increase in minimum requirements really start making ceph expensive.
1GB of process memory for a single HDD backed bluestore OSD was never the recommendation. Prior to the memory autotuning we had the overall bluestore cache size set to 1GB for HDDs, but that does not mean that a bluestore OSD could ever consistently fit in a 1GB memory envelope. It would be 1GB of cache plus whatever else the OSD needed to run. In the filestore days we did (usually!) use less memory because we used the global page cache more than dedicated per-daemon caches. That works well for slower disks but isn't necessarily a great model if you've got a box full of NVMe drives and/or want to have more control over what gets cached and how. Generally the trend (and not just for Ceph) is to get the kernel out of the way and shard everything as much as possible. The page cache doesn't really fit well with that model as storage keeps getting faster and faster. Having said all of that, it wasn't necessarily uncommon for filestore OSDs to use 1-2GB of RAM either especially during recovery and if the pglog is full. The big difference is that now we are actually working toward trying to keep the OSD (and other daemons) within a certain memory boundary and historically we didn't do that at all.
We have set the OSDs to use the bitmap allocator. Is the fact that we get tcmalloc stats a contradiction to this?
No, the bitmap allocator won't prevent you from getting tcmalloc stats. tcmalloc controls memory allocations, the bitmap allocator controls disk allocations.
I did not consider upgrading from mimic, because a lot of people report stability issues that might be caused by a regression in the message queueing. There was a longer e-mail about clusters from nautilus and higher collapsing under trivial amounts of rebalancing, pool deletion and other admin tasks. Before I consider upgrading, I want to test this on a lab cluster we plan to set up soon.
I will look at the memory profiling. If one can use this on a production system, I will give it a go.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 14 July 2020 14:48:36 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: OSD memory leak?
Hi Frank,
These might help:
https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/
https://gperftools.github.io/gperftools/heapprofile.html
https://gperftools.github.io/gperftools/heap_checker.html
Regarding the mempools, they don't track all of the memory usage in Ceph, only things that were allocated using mempools. There are many other things (rocksdb block cache for example) that don't use them. It's only giving you a partial picture of memory usage. In your example below, that byte value from the thread cache freelist looks very wrong. Ignoring that for a moment though, there's a ton of memory that's been unmapped and released to the OS, but hasn't been reclaimed by the kernel. That's either because the kernel doesn't have enough memory pressure to bother reclaiming it, or because it's all fragmented chunks of a huge page that the kernel can't fully reclaim. That tells me you should definitely be looking at the transparent huge page (THP) configuration on your nodes. Looking back at batrick's PR that disables THP for Ceph, it looks like we only backported it to nautilus but not mimic. On that topic, have you considered upgrading to Nautilus?
Mark
On 7/14/20 2:56 AM, Frank Schilder wrote:
Dear Mark,
thanks for the quick answer. I would try the memory profiler if I could find any documentation on it. In fact, I just guessed the "heap stats" command and have a hard time finding anything on the OSD daemon commands. Could you possibly point me to something? Also how to interpret the mempools? Is it correct to say that out of the memory_target only the mempools total is actually used and the remaining memory is lost due to leaks?
For example, for OSD 256 I get the stats below after just 2 months uptime. Am I looking at a 5.5GB memory leak here?
# ceph config get osd.256 osd_memory_target 8589934592
# ceph daemon osd.256 heap stats osd.256 tcmalloc heap stats:------------------------------------------------ MALLOC: 7216067616 ( 6881.8 MiB) Bytes in use by application MALLOC: + 229376 ( 0.2 MiB) Bytes in page heap freelist MALLOC: + 1222913888 ( 1166.3 MiB) Bytes in central cache freelist MALLOC: + 278016 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 18446744073692937856 (17592186044400.2 MiB) Bytes in thread cache freelists MALLOC: + 52166656 ( 49.8 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 8475041792 ( 8082.4 MiB) Actual memory used (physical + swap) MALLOC: + 2010464256 ( 1917.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 10485506048 ( 9999.8 MiB) Virtual address space used MALLOC: MALLOC: 765182 Spans in use MALLOC: 48 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
# ceph daemon osd.256 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 2300682, "bytes": 18405456 }, "bluestore_cache_data": { "items": 52390, "bytes": 306843648 }, "bluestore_cache_onode": { "items": 256153, "bytes": 145494904 }, "bluestore_cache_other": { "items": 92199353, "bytes": 656620069 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 4, "bytes": 2752 }, "bluestore_writing_deferred": { "items": 122, "bytes": 1864924 }, "bluestore_writing": { "items": 3673, "bytes": 18440192 }, "bluefs": { "items": 11867, "bytes": 220504 }, "buffer_anon": { "items": 353734, "bytes": 1180837372 }, "buffer_meta": { "items": 91646, "bytes": 5865344 }, "osd": { "items": 134, "bytes": 1557616 }, "osd_mapbl": { "items": 84, "bytes": 8479562 }, "osd_pglog": { "items": 487004, "bytes": 166094788 }, "osdmap": { "items": 117697, "bytes": 2080280 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 95874543, "bytes": 2512807411 } } }
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 13 July 2020 15:39:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: OSD memory leak?
Hi Frank,
So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though.
Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise".
Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well.
Mark
On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony and Mark, thanks for your answers. I have seen recommendations derived from test clusters with bluestore OSDs that read 16GB base line + 1GB per HDD + 4GB per SSD OSD, probably from the times when bluestore had a base-line+stress dependent. I would actually consider this already quite something. I understand that for high-performance requirements one adds RAM etc. to speed things up. For a mostly cold data store with a thin layer of warm/hot data, however, this is quite a lot compared with what standard disk controllers can do with a cheap CPU, 4GB of RAM and 16 drives connected. Essentially, ceph is turning a server into a disk controller and it should be possible to run a configuration that does not require much more than an ordinary hardware controller per disk delivering reasonable performance. I'm thinking along the lines of 25MB/s throughput and maybe 10IOP/s per NL-SAS HDD OSD to the user side (simple collocated deployment, EC pool). This ought to be possible in a way similar to a RAID controller with comparably moderate hardware requirements. Good aggregated performance then comes from scale and because the layer of hot data per disk is only a few GB per drive (a full re-write of just the hot data is only a few minutes). I thought this was the idea of ceph. Instead of trying to accommodate high-performance wishes for ridiculously small ceph clusters (I do see these "I have 3 servers with 3 disks each, why is it so slow" kind of complaints, which I would simply ignore), one talks about scale-out systems with thousands of OSDs. Something like 20 hosts serving 200 disks each would count as a small cluster. If the warm/hot data is only 1% or even less, such a system will be quite satisfying. For low-cost scale-out we have ceph. For performance, we have technologies like Lustre (which by the way has much more moderate minimum hardware requirements). For anything that requires higher performance one can then start using tiering, WAL/DB devices, SSD only pools, lots of RAM, whatever. However, there should be a stable, well-tested and low-demanding base line config for a cold store use case with hardware requirements similar to a NAS box per storage unit (one server+JBODs). I start missing support for the latter. 2 or even 4GB and 1core-GHz per HDD is really a lot compared with such systems. Please don't take this as a start of a long discussion. Its just a wish from my side to have low-demanding configs available that scale easily and are easy to administrate at an overall low cost. I will look into memory profiling of some OSDs. It doesn't look like a performance killer. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: 14 July 2020 17:29 To: ceph-users@ceph.io Subject: [ceph-users] Re: OSD memory leak?
In the past, the minimum recommendation was 1GB RAM per HDD blue store OSD.
There was a rule of thumb of 1GB RAM *per TB* of HDD Filestore OSD, perhaps you were influenced by that? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Mark, On Mon, Jul 13, 2020 at 3:42 PM Mark Nelson <mnelson@redhat.com> wrote:
Hi Frank,
So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though.
Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory
Do you know if this high osdmap usage is known already? Our big block storage cluster generates a new osdmap every few seconds (due to rbd snap trimming) and we see the osdmap mempool usage growing over a few months until osds start getting OOM killed. Today we proactively restarted them because the osdmap_mempool was using close to 700MB. So it seems that whatever is supposed to be trimming is not working. (This is observed with nautilus 14.2.8 but iirc it has been the same even when we were running luminous and mimic too) Cheers, Dan
autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise".
Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well.
Mark
On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 7/15/20 9:58 AM, Dan van der Ster wrote:
Hi Mark,
On Mon, Jul 13, 2020 at 3:42 PM Mark Nelson <mnelson@redhat.com> wrote:
Hi Frank,
So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though.
Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory Do you know if this high osdmap usage is known already? Our big block storage cluster generates a new osdmap every few seconds (due to rbd snap trimming) and we see the osdmap mempool usage growing over a few months until osds start getting OOM killed.
Today we proactively restarted them because the osdmap_mempool was using close to 700MB. So it seems that whatever is supposed to be trimming is not working. (This is observed with nautilus 14.2.8 but iirc it has been the same even when we were running luminous and mimic too)
Cheers, Dan
Hrm, it hasn't been on my radar, though looking back through the mailing list there appears to be various reports over the years of high usage (some of which theoretically have been fixed). Maybe submit a tracker issue? 700MB seems quite high for osdmap, but I don't really know the retention rules so someone else who knows that code better will have to chime in.
autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise".
Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well.
Mark
On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Dan, cc Mark, this sounds exactly like the scenario I'm looking at. We have rolling snapshots on RBD images on currently ca. 200 VMs and increasing. Snapshots are daily with different retention periods. We have two pools with separate hardware backing RBD and cephfs. The mem stats I sent are from an OSD backing cephfs, which does not have any snaps currently. So the snaps on other OSDs influence the memory usage of OSDs that have nothing to do with the RBDs. I also noticed a significant drop of memory usage across the cluster after restarting the OSDs on just one host. Not sure if this is expected either. Looks like the OSDs do collect dead baggage quite fast and the memory_target reduces the caches in an attempt to accommodate for that. The fact that the kernel swaps this out in favour of disk buffers on a system with low swappiness where the only disk access is local syslog indicates that this is allocated but never used - a quite massive leak. It currently looks like that after only a couple of days the leakage exceeds the mem target already. I don't want to have the occasional OOM killer on my operations team. For now I will probably adopt a reverse strategy, give up on memory_target doing something useful, increase the minimum cache limits to ensure at least some caching, have swap take care of the leak and restart OSDs regularly (every 2-3 months). Would be good if this could be looked at. Please let me know if there is some data I can provide. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 15 July 2020 18:36:06 To: Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak? On 7/15/20 9:58 AM, Dan van der Ster wrote:
Hi Mark,
On Mon, Jul 13, 2020 at 3:42 PM Mark Nelson <mnelson@redhat.com> wrote:
Hi Frank,
So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though.
Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory Do you know if this high osdmap usage is known already? Our big block storage cluster generates a new osdmap every few seconds (due to rbd snap trimming) and we see the osdmap mempool usage growing over a few months until osds start getting OOM killed.
Today we proactively restarted them because the osdmap_mempool was using close to 700MB. So it seems that whatever is supposed to be trimming is not working. (This is observed with nautilus 14.2.8 but iirc it has been the same even when we were running luminous and mimic too)
Cheers, Dan
Hrm, it hasn't been on my radar, though looking back through the mailing list there appears to be various reports over the years of high usage (some of which theoretically have been fixed). Maybe submit a tracker issue? 700MB seems quite high for osdmap, but I don't really know the retention rules so someone else who knows that code better will have to chime in.
autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise".
Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well.
Mark
On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Mark and Dan, I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults): # Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB # Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount? Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min? Many thanks for your help with this. I can't find answers to these questions in the docs. There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps: OSD208, uptime 113-13:44:32: # ceph daemon osd.208 status { "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "osd_fsid": "16891b16-b4d8-418b-a6ba-34b85921e809", "whoami": 208, "state": "active", "oldest_map": 162084, "newest_map": 162766, "num_pgs": 96 } OSD.211, uptime 2-18:56:49: # ceph daemon osd.211 status { "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "osd_fsid": "81cb5da4-bf12-42a3-b9d4-9d4fba2a58fd", "whoami": 211, "state": "active", "oldest_map": 162084, "newest_map": 162766, "num_pgs": 98 } A long-running one and a freshly restarted one hold the same number of osd_maps. However, the long-running one has accummulated more than 1GB extra memory usage (3067.6 MiB versus 1951.4 MiB): # ceph daemon osd.208 heap stats osd.208 tcmalloc heap stats:------------------------------------------------ MALLOC: 2356637072 ( 2247.5 MiB) Bytes in use by application MALLOC: + 5742592 ( 5.5 MiB) Bytes in page heap freelist MALLOC: + 822018216 ( 783.9 MiB) Bytes in central cache freelist MALLOC: + 491520 ( 0.5 MiB) Bytes in transfer cache freelist MALLOC: + 11850184 ( 11.3 MiB) Bytes in thread cache freelists MALLOC: + 19922944 ( 19.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3216662528 ( 3067.6 MiB) Actual memory used (physical + swap) MALLOC: + 249790464 ( 238.2 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 3466452992 ( 3305.9 MiB) Virtual address space used MALLOC: MALLOC: 293267 Spans in use MALLOC: 36 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.208 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 19, "bytes": 10792 }, "bluestore_cache_other": { "items": 5720554, "bytes": 46039919 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 18, "bytes": 12384 }, "bluestore_writing_deferred": { "items": 214, "bytes": 26514641 }, "bluestore_writing": { "items": 64, "bytes": 10206779 }, "bluefs": { "items": 9735, "bytes": 188984 }, "buffer_anon": { "items": 292345, "bytes": 67468304 }, "buffer_meta": { "items": 562, "bytes": 35968 }, "osd": { "items": 96, "bytes": 1115904 }, "osd_mapbl": { "items": 80, "bytes": 8501746 }, "osd_pglog": { "items": 328703, "bytes": 117673864 }, "osdmap": { "items": 12101478, "bytes": 210941392 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 23145696, "bytes": 526245301 } } } # ceph daemon osd.211 heap stats osd.211 tcmalloc heap stats:------------------------------------------------ MALLOC: 1727399344 ( 1647.4 MiB) Bytes in use by application MALLOC: + 532480 ( 0.5 MiB) Bytes in page heap freelist MALLOC: + 262860912 ( 250.7 MiB) Bytes in central cache freelist MALLOC: + 11693568 ( 11.2 MiB) Bytes in transfer cache freelist MALLOC: + 29694944 ( 28.3 MiB) Bytes in thread cache freelists MALLOC: + 14024704 ( 13.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 2046205952 ( 1951.4 MiB) Actual memory used (physical + swap) MALLOC: + 229212160 ( 218.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 2275418112 ( 2170.0 MiB) Virtual address space used MALLOC: MALLOC: 145115 Spans in use MALLOC: 32 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.211 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 894, "bytes": 163053568 }, "bluestore_cache_onode": { "items": 165536, "bytes": 94024448 }, "bluestore_cache_other": { "items": 33936718, "bytes": 233428234 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 110, "bytes": 75680 }, "bluestore_writing_deferred": { "items": 38, "bytes": 6061245 }, "bluestore_writing": { "items": 0, "bytes": 0 }, "bluefs": { "items": 9956, "bytes": 189640 }, "buffer_anon": { "items": 293298, "bytes": 59950954 }, "buffer_meta": { "items": 1005, "bytes": 64320 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 80, "bytes": 8501690 }, "osd_pglog": { "items": 350517, "bytes": 132253139 }, "osdmap": { "items": 633498, "bytes": 10866360 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 40083576, "bytes": 747143054 } } } Same disk type, same memory_target. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 16 July 2020 09:11 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak? Dear Dan, cc Mark, this sounds exactly like the scenario I'm looking at. We have rolling snapshots on RBD images on currently ca. 200 VMs and increasing. Snapshots are daily with different retention periods. We have two pools with separate hardware backing RBD and cephfs. The mem stats I sent are from an OSD backing cephfs, which does not have any snaps currently. So the snaps on other OSDs influence the memory usage of OSDs that have nothing to do with the RBDs. I also noticed a significant drop of memory usage across the cluster after restarting the OSDs on just one host. Not sure if this is expected either. Looks like the OSDs do collect dead baggage quite fast and the memory_target reduces the caches in an attempt to accommodate for that. The fact that the kernel swaps this out in favour of disk buffers on a system with low swappiness where the only disk access is local syslog indicates that this is allocated but never used - a quite massive leak. It currently looks like that after only a couple of days the leakage exceeds the mem target already. I don't want to have the occasional OOM killer on my operations team. For now I will probably adopt a reverse strategy, give up on memory_target doing something useful, increase the minimum cache limits to ensure at least some caching, have swap take care of the leak and restart OSDs regularly (every 2-3 months). Would be good if this could be looked at. Please let me know if there is some data I can provide. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 15 July 2020 18:36:06 To: Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak? On 7/15/20 9:58 AM, Dan van der Ster wrote:
Hi Mark,
On Mon, Jul 13, 2020 at 3:42 PM Mark Nelson <mnelson@redhat.com> wrote:
Hi Frank,
So the osd_memory_target code will basically shrink the size of the bluestore and rocksdb caches to attempt to keep the overall mapped (not rss!) memory of the process below the target. It's sort of "best effort" in that it can't guarantee the process will fit within a given target, it will just (assuming we are over target) shrink the caches up to some minimum value and that's it. 2GB per OSD is a pretty ambitious target. It's the lowest osd_memory_target we recommend setting. I'm a little surprised the OSD is consuming this much memory with a 2GB target though.
Looking at your mempool dump I see very little memory allocated to the caches. In fact the majority is taken up by osdmap (looks like you have a decent number of OSDs) and pglog. That indicates that the memory Do you know if this high osdmap usage is known already? Our big block storage cluster generates a new osdmap every few seconds (due to rbd snap trimming) and we see the osdmap mempool usage growing over a few months until osds start getting OOM killed.
Today we proactively restarted them because the osdmap_mempool was using close to 700MB. So it seems that whatever is supposed to be trimming is not working. (This is observed with nautilus 14.2.8 but iirc it has been the same even when we were running luminous and mimic too)
Cheers, Dan
Hrm, it hasn't been on my radar, though looking back through the mailing list there appears to be various reports over the years of high usage (some of which theoretically have been fixed). Maybe submit a tracker issue? 700MB seems quite high for osdmap, but I don't really know the retention rules so someone else who knows that code better will have to chime in.
autotuning is probably working but simply can't do anything more to help. Something else is taking up the memory. Figure you've got a little shy of 500MB for the mempools. RocksDB will take up more (and potentially quite a bit more if you have memtables backing up waiting to be flushed to L0) and potentially some other things in the OSD itself that could take up memory. If you feel comfortable experimenting, you could try changing the rocksdb WAL/memtable settings. By default we have up to 4 256MB WAL buffers. Instead you could try something like 2 64MB buffers, but be aware this could cause slow performance or even temporary write stalls if you have fast storage. Still, this would only give you up to ~0.9GB back. Since you are on mimic, you might also want to check what your kernel's transparent huge pages configuration is. I don't remember if we backported Patrick's fix to always avoid THP for ceph processes. If your kernel is set to "always", you might consider trying it with "madvise".
Alternately, have you tried the built-in tcmalloc heap profiler? You might be able to get a better sense of where memory is being used with that as well.
Mark
On 7/13/20 7:07 AM, Frank Schilder wrote:
Hi all,
on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load.
What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats:
Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } }
Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------
Am I looking at a memory leak here or are these heap stats expected?
I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits. When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though.
Dear Mark, thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here. Thanks also for raising this as an issue. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak? On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits. When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though.
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this? If not, can I change the dump path? Its likely to overrun my log partition quickly if I cannot adjust either of the two. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak? Dear Mark, thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here. Thanks also for raising this as an issue. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak? On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits. When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Mark, here is a first collection of heap profiling data (valid 30 days): https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l This was collected with the following config settings: osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648 Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though. Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak? Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this? If not, can I change the dump path? Its likely to overrun my log partition quickly if I cannot adjust either of the two. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak? Dear Mark, thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here. Thanks also for raising this as an issue. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak? On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits. When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, I'm having trouble getting the exact version of ceph you used to create this heap profile. Could you run the google-pprof --text steps at [1] and share the output? Thanks, Dan [1] https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/ On Tue, Aug 11, 2020 at 2:37 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Mark,
here is a first collection of heap profiling data (valid 30 days):
https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l
This was collected with the following config settings:
osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648
Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though.
Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this?
If not, can I change the dump path?
Its likely to overrun my log partition quickly if I cannot adjust either of the two.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Dear Mark,
thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here.
Thanks also for raising this as an issue.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits.
When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan, I use the container docker.io/ceph/daemon:v3.2.10-stable-3.2-mimic-centos-7-x86_64. As far as I can see, it uses the packages from http://download.ceph.com/rpm-mimic/el7, its a Centos 7 build. The version is: # ceph -v ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) On Centos, the profiler packages are called different, without the "google-" prefix. The version I have installed is # pprof --version pprof (part of gperftools 2.0) Copyright 1998-2007 Google Inc. This is BSD licensed software; see the source for copying conditions and license information. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. It is possible to install pprof inside this container and analyse the *.heap-files I provided. If this doesn't work for you and you want me to generate the text output for heap-files, I can do that. Please let me know if I should do all files and with what option (eg. against a base etc.). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dan@vanderster.com> Sent: 14 August 2020 10:38:57 To: Frank Schilder Cc: Mark Nelson; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak? Hi Frank, I'm having trouble getting the exact version of ceph you used to create this heap profile. Could you run the google-pprof --text steps at [1] and share the output? Thanks, Dan [1] https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/ On Tue, Aug 11, 2020 at 2:37 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Mark,
here is a first collection of heap profiling data (valid 30 days):
https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l
This was collected with the following config settings:
osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648
Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though.
Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this?
If not, can I change the dump path?
Its likely to overrun my log partition quickly if I cannot adjust either of the two.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Dear Mark,
thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here.
Thanks also for raising this as an issue.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits.
When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan and Mark, could you please let me know if you can read the files with the version info I provided in my previous e-mail? I'm in the process of collecting data with more FS activity and would like to send it in a format that is useful for investigation. Right now I'm observing a daily growth of swap of ca. 100-200MB on servers with 16 OSDs each, 1SSD and 15HDDs. The OS+daemons operate fine, the OS manages to keep enough RAM available. Also the mempool dump still shows onode and data cached at a seemingly reasonable level. Users report a more stable performance of the FS after I increased the cach min sizes on all OSDs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 17 August 2020 09:37 To: Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak? Hi Dan, I use the container docker.io/ceph/daemon:v3.2.10-stable-3.2-mimic-centos-7-x86_64. As far as I can see, it uses the packages from http://download.ceph.com/rpm-mimic/el7, its a Centos 7 build. The version is: # ceph -v ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) On Centos, the profiler packages are called different, without the "google-" prefix. The version I have installed is # pprof --version pprof (part of gperftools 2.0) Copyright 1998-2007 Google Inc. This is BSD licensed software; see the source for copying conditions and license information. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. It is possible to install pprof inside this container and analyse the *.heap-files I provided. If this doesn't work for you and you want me to generate the text output for heap-files, I can do that. Please let me know if I should do all files and with what option (eg. against a base etc.). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dan@vanderster.com> Sent: 14 August 2020 10:38:57 To: Frank Schilder Cc: Mark Nelson; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak? Hi Frank, I'm having trouble getting the exact version of ceph you used to create this heap profile. Could you run the google-pprof --text steps at [1] and share the output? Thanks, Dan [1] https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/ On Tue, Aug 11, 2020 at 2:37 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Mark,
here is a first collection of heap profiling data (valid 30 days):
https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l
This was collected with the following config settings:
osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648
Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though.
Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this?
If not, can I change the dump path?
Its likely to overrun my log partition quickly if I cannot adjust either of the two.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Dear Mark,
thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here.
Thanks also for raising this as an issue.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits.
When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, I didn't get time yet. On our side, I was planning to see if the issue persists after upgrading to v14.2.11 -- it includes some updates to how the osdmap is referenced across OSD.cc. BTW, do you happen to have osd_map_dedup set to false? We do, and that surely increases the osdmap memory usage somewhat. -- Dan -- Dan On Thu, Aug 20, 2020 at 9:33 AM Frank Schilder <frans@dtu.dk> wrote:
Hi Dan and Mark,
could you please let me know if you can read the files with the version info I provided in my previous e-mail? I'm in the process of collecting data with more FS activity and would like to send it in a format that is useful for investigation.
Right now I'm observing a daily growth of swap of ca. 100-200MB on servers with 16 OSDs each, 1SSD and 15HDDs. The OS+daemons operate fine, the OS manages to keep enough RAM available. Also the mempool dump still shows onode and data cached at a seemingly reasonable level. Users report a more stable performance of the FS after I increased the cach min sizes on all OSDs.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 17 August 2020 09:37 To: Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Hi Dan,
I use the container docker.io/ceph/daemon:v3.2.10-stable-3.2-mimic-centos-7-x86_64. As far as I can see, it uses the packages from http://download.ceph.com/rpm-mimic/el7, its a Centos 7 build. The version is:
# ceph -v ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
On Centos, the profiler packages are called different, without the "google-" prefix. The version I have installed is
# pprof --version pprof (part of gperftools 2.0)
Copyright 1998-2007 Google Inc.
This is BSD licensed software; see the source for copying conditions and license information. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
It is possible to install pprof inside this container and analyse the *.heap-files I provided.
If this doesn't work for you and you want me to generate the text output for heap-files, I can do that. Please let me know if I should do all files and with what option (eg. against a base etc.).
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dan van der Ster <dan@vanderster.com> Sent: 14 August 2020 10:38:57 To: Frank Schilder Cc: Mark Nelson; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
Hi Frank,
I'm having trouble getting the exact version of ceph you used to create this heap profile. Could you run the google-pprof --text steps at [1] and share the output?
Thanks, Dan
[1] https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/
On Tue, Aug 11, 2020 at 2:37 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Mark,
here is a first collection of heap profiling data (valid 30 days):
https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l
This was collected with the following config settings:
osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648
Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though.
Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this?
If not, can I change the dump path?
Its likely to overrun my log partition quickly if I cannot adjust either of the two.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Dear Mark,
thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here.
Thanks also for raising this as an issue.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits.
When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan, no worries. I checked and osd_map_dedup is set to true, the default value. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dan@vanderster.com> Sent: 20 August 2020 09:41 To: Frank Schilder Cc: Mark Nelson; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak? Hi Frank, I didn't get time yet. On our side, I was planning to see if the issue persists after upgrading to v14.2.11 -- it includes some updates to how the osdmap is referenced across OSD.cc. BTW, do you happen to have osd_map_dedup set to false? We do, and that surely increases the osdmap memory usage somewhat. -- Dan -- Dan On Thu, Aug 20, 2020 at 9:33 AM Frank Schilder <frans@dtu.dk> wrote:
Hi Dan and Mark,
could you please let me know if you can read the files with the version info I provided in my previous e-mail? I'm in the process of collecting data with more FS activity and would like to send it in a format that is useful for investigation.
Right now I'm observing a daily growth of swap of ca. 100-200MB on servers with 16 OSDs each, 1SSD and 15HDDs. The OS+daemons operate fine, the OS manages to keep enough RAM available. Also the mempool dump still shows onode and data cached at a seemingly reasonable level. Users report a more stable performance of the FS after I increased the cach min sizes on all OSDs.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 17 August 2020 09:37 To: Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Hi Dan,
I use the container docker.io/ceph/daemon:v3.2.10-stable-3.2-mimic-centos-7-x86_64. As far as I can see, it uses the packages from http://download.ceph.com/rpm-mimic/el7, its a Centos 7 build. The version is:
# ceph -v ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
On Centos, the profiler packages are called different, without the "google-" prefix. The version I have installed is
# pprof --version pprof (part of gperftools 2.0)
Copyright 1998-2007 Google Inc.
This is BSD licensed software; see the source for copying conditions and license information. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
It is possible to install pprof inside this container and analyse the *.heap-files I provided.
If this doesn't work for you and you want me to generate the text output for heap-files, I can do that. Please let me know if I should do all files and with what option (eg. against a base etc.).
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dan van der Ster <dan@vanderster.com> Sent: 14 August 2020 10:38:57 To: Frank Schilder Cc: Mark Nelson; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
Hi Frank,
I'm having trouble getting the exact version of ceph you used to create this heap profile. Could you run the google-pprof --text steps at [1] and share the output?
Thanks, Dan
[1] https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/
On Tue, Aug 11, 2020 at 2:37 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Mark,
here is a first collection of heap profiling data (valid 30 days):
https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l
This was collected with the following config settings:
osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648
Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though.
Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this?
If not, can I change the dump path?
Its likely to overrun my log partition quickly if I cannot adjust either of the two.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Dear Mark,
thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here.
Thanks also for raising this as an issue.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits.
When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, I downloaded but haven't had time to get the environment setup yet either. It might be better to just generate the txt files if you can. Thanks! Mark On 8/20/20 2:33 AM, Frank Schilder wrote:
Hi Dan and Mark,
could you please let me know if you can read the files with the version info I provided in my previous e-mail? I'm in the process of collecting data with more FS activity and would like to send it in a format that is useful for investigation.
Right now I'm observing a daily growth of swap of ca. 100-200MB on servers with 16 OSDs each, 1SSD and 15HDDs. The OS+daemons operate fine, the OS manages to keep enough RAM available. Also the mempool dump still shows onode and data cached at a seemingly reasonable level. Users report a more stable performance of the FS after I increased the cach min sizes on all OSDs.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 17 August 2020 09:37 To: Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Hi Dan,
I use the container docker.io/ceph/daemon:v3.2.10-stable-3.2-mimic-centos-7-x86_64. As far as I can see, it uses the packages from http://download.ceph.com/rpm-mimic/el7, its a Centos 7 build. The version is:
# ceph -v ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
On Centos, the profiler packages are called different, without the "google-" prefix. The version I have installed is
# pprof --version pprof (part of gperftools 2.0)
Copyright 1998-2007 Google Inc.
This is BSD licensed software; see the source for copying conditions and license information. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
It is possible to install pprof inside this container and analyse the *.heap-files I provided.
If this doesn't work for you and you want me to generate the text output for heap-files, I can do that. Please let me know if I should do all files and with what option (eg. against a base etc.).
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dan van der Ster <dan@vanderster.com> Sent: 14 August 2020 10:38:57 To: Frank Schilder Cc: Mark Nelson; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
Hi Frank,
I'm having trouble getting the exact version of ceph you used to create this heap profile. Could you run the google-pprof --text steps at [1] and share the output?
Thanks, Dan
[1] https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/
On Tue, Aug 11, 2020 at 2:37 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Mark,
here is a first collection of heap profiling data (valid 30 days):
https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l
This was collected with the following config settings:
osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648
Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though.
Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this?
If not, can I change the dump path?
Its likely to overrun my log partition quickly if I cannot adjust either of the two.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Dear Mark,
thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here.
Thanks also for raising this as an issue.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits.
When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Mark and Dan, I can generate text files. Can you let me know what you would like to see? Without further instructions, I can do a simple conversion and a conversion against the first dump as a base. I will upload an archive with converted files added tomorrow afternoon. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 August 2020 21:52 To: Frank Schilder; Dan van der Ster; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak? Hi Frank, I downloaded but haven't had time to get the environment setup yet either. It might be better to just generate the txt files if you can. Thanks! Mark On 8/20/20 2:33 AM, Frank Schilder wrote:
Hi Dan and Mark,
could you please let me know if you can read the files with the version info I provided in my previous e-mail? I'm in the process of collecting data with more FS activity and would like to send it in a format that is useful for investigation.
Right now I'm observing a daily growth of swap of ca. 100-200MB on servers with 16 OSDs each, 1SSD and 15HDDs. The OS+daemons operate fine, the OS manages to keep enough RAM available. Also the mempool dump still shows onode and data cached at a seemingly reasonable level. Users report a more stable performance of the FS after I increased the cach min sizes on all OSDs.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 17 August 2020 09:37 To: Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Hi Dan,
I use the container docker.io/ceph/daemon:v3.2.10-stable-3.2-mimic-centos-7-x86_64. As far as I can see, it uses the packages from http://download.ceph.com/rpm-mimic/el7, its a Centos 7 build. The version is:
# ceph -v ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
On Centos, the profiler packages are called different, without the "google-" prefix. The version I have installed is
# pprof --version pprof (part of gperftools 2.0)
Copyright 1998-2007 Google Inc.
This is BSD licensed software; see the source for copying conditions and license information. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
It is possible to install pprof inside this container and analyse the *.heap-files I provided.
If this doesn't work for you and you want me to generate the text output for heap-files, I can do that. Please let me know if I should do all files and with what option (eg. against a base etc.).
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dan van der Ster <dan@vanderster.com> Sent: 14 August 2020 10:38:57 To: Frank Schilder Cc: Mark Nelson; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
Hi Frank,
I'm having trouble getting the exact version of ceph you used to create this heap profile. Could you run the google-pprof --text steps at [1] and share the output?
Thanks, Dan
[1] https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/
On Tue, Aug 11, 2020 at 2:37 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Mark,
here is a first collection of heap profiling data (valid 30 days):
https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l
This was collected with the following config settings:
osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648
Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though.
Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this?
If not, can I change the dump path?
Its likely to overrun my log partition quickly if I cannot adjust either of the two.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Dear Mark,
thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here.
Thanks also for raising this as an issue.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits.
When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan and Mark, sorry, took a bit longer. I uploaded a new archive containing files with the following format (https://files.dtu.dk/u/jb0uS6U9LlCfvS5L/heap_profiling-2020-08-31.tgz?l - valid 60 days): - osd.195.profile.*.heap - raw heap dump file - osd.195.profile.*.heap.txt - output of conversion with --text - osd.195.profile.*.heap-base0001.txt - output of conversion with --text against first dump as base - osd.195.*.heap_stats - output of ceph daemon osd.195 heap stats, every hour - osd.195.*.mempools - output of ceph daemon osd.195 dump_mempools, every hour - osd.195.*.perf - output of ceph daemon osd.195 perf dump, every hour, counters are reset Only for the last couple of days are converted files included, post-conversion of everything simply takes too long. Please find also attached a recording of memory usage on one of the relevant OSD nodes. I marked restarts of all OSDs/the host with vertical red lines. What is worrying is the self-amplifying nature of the leak. ts not a linear process, it looks at least quadratic if not exponential. What we are looking for is, given the comparably short uptime, probably still in the lower percentages with increasing rate. The OSDs just started to overrun their limit: top - 14:38:49 up 155 days, 19:17, 1 user, load average: 5.99, 4.59, 4.59 Tasks: 684 total, 1 running, 293 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.9 us, 0.9 sy, 0.0 ni, 89.6 id, 7.6 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 65727628 total, 6937548 free, 41921260 used, 16868820 buff/cache KiB Swap: 93532160 total, 90199040 free, 3333120 used. 6740136 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4099023 ceph 20 0 5918704 3.8g 9700 S 1.7 6.1 378:37.01 /usr/bin/ceph-osd --cluster ceph -f -i 35 --setuser cep+ 4097639 ceph 20 0 5340924 3.0g 11428 S 87.1 4.7 14636:30 /usr/bin/ceph-osd --cluster ceph -f -i 195 --setuser ce+ 4097974 ceph 20 0 3648188 2.3g 9628 S 8.3 3.6 1375:58 /usr/bin/ceph-osd --cluster ceph -f -i 201 --setuser ce+ 4098322 ceph 20 0 3478980 2.2g 9688 S 5.3 3.6 1426:05 /usr/bin/ceph-osd --cluster ceph -f -i 223 --setuser ce+ 4099374 ceph 20 0 3446784 2.2g 9252 S 4.6 3.5 1142:14 /usr/bin/ceph-osd --cluster ceph -f -i 205 --setuser ce+ 4098679 ceph 20 0 3832140 2.2g 9796 S 6.6 3.5 1248:26 /usr/bin/ceph-osd --cluster ceph -f -i 132 --setuser ce+ 4100782 ceph 20 0 3641608 2.2g 9652 S 7.9 3.5 1278:10 /usr/bin/ceph-osd --cluster ceph -f -i 207 --setuser ce+ 4095944 ceph 20 0 3375672 2.2g 8968 S 7.3 3.5 1250:02 /usr/bin/ceph-osd --cluster ceph -f -i 108 --setuser ce+ 4096956 ceph 20 0 3509376 2.2g 9456 S 7.9 3.5 1157:27 /usr/bin/ceph-osd --cluster ceph -f -i 203 --setuser ce+ 4099731 ceph 20 0 3563652 2.2g 8972 S 3.6 3.5 1421:48 /usr/bin/ceph-osd --cluster ceph -f -i 61 --setuser cep+ 4096262 ceph 20 0 3531988 2.2g 9040 S 9.9 3.5 1600:15 /usr/bin/ceph-osd --cluster ceph -f -i 121 --setuser ce+ 4100442 ceph 20 0 3359736 2.1g 9804 S 4.3 3.4 1185:53 /usr/bin/ceph-osd --cluster ceph -f -i 226 --setuser ce+ 4096617 ceph 20 0 3443060 2.1g 9432 S 5.0 3.4 1449:29 /usr/bin/ceph-osd --cluster ceph -f -i 199 --setuser ce+ 4097298 ceph 20 0 3483532 2.1g 9600 S 5.6 3.3 1265:28 /usr/bin/ceph-osd --cluster ceph -f -i 97 --setuser cep+ 4100093 ceph 20 0 3428348 2.0g 9568 S 3.3 3.2 1298:53 /usr/bin/ceph-osd --cluster ceph -f -i 197 --setuser ce+ 4095630 ceph 20 0 3440160 2.0g 8976 S 3.6 3.2 1451:35 /usr/bin/ceph-osd --cluster ceph -f -i 62 --setuser cep+ Generally speaking, increasing the cache minimum seems to help with keeping important information in RAM. Unfortunately, it also means that swap usage starts much earlier. Best regards and thanks for your help, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 August 2020 22:40 To: Mark Nelson; Dan van der Ster; ceph-users Subject: [ceph-users] Re: OSD memory leak? Hi Mark and Dan, I can generate text files. Can you let me know what you would like to see? Without further instructions, I can do a simple conversion and a conversion against the first dump as a base. I will upload an archive with converted files added tomorrow afternoon. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 August 2020 21:52 To: Frank Schilder; Dan van der Ster; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak? Hi Frank, I downloaded but haven't had time to get the environment setup yet either. It might be better to just generate the txt files if you can. Thanks! Mark On 8/20/20 2:33 AM, Frank Schilder wrote:
Hi Dan and Mark,
could you please let me know if you can read the files with the version info I provided in my previous e-mail? I'm in the process of collecting data with more FS activity and would like to send it in a format that is useful for investigation.
Right now I'm observing a daily growth of swap of ca. 100-200MB on servers with 16 OSDs each, 1SSD and 15HDDs. The OS+daemons operate fine, the OS manages to keep enough RAM available. Also the mempool dump still shows onode and data cached at a seemingly reasonable level. Users report a more stable performance of the FS after I increased the cach min sizes on all OSDs.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 17 August 2020 09:37 To: Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Hi Dan,
I use the container docker.io/ceph/daemon:v3.2.10-stable-3.2-mimic-centos-7-x86_64. As far as I can see, it uses the packages from http://download.ceph.com/rpm-mimic/el7, its a Centos 7 build. The version is:
# ceph -v ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
On Centos, the profiler packages are called different, without the "google-" prefix. The version I have installed is
# pprof --version pprof (part of gperftools 2.0)
Copyright 1998-2007 Google Inc.
This is BSD licensed software; see the source for copying conditions and license information. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
It is possible to install pprof inside this container and analyse the *.heap-files I provided.
If this doesn't work for you and you want me to generate the text output for heap-files, I can do that. Please let me know if I should do all files and with what option (eg. against a base etc.).
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Dan van der Ster <dan@vanderster.com> Sent: 14 August 2020 10:38:57 To: Frank Schilder Cc: Mark Nelson; ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
Hi Frank,
I'm having trouble getting the exact version of ceph you used to create this heap profile. Could you run the google-pprof --text steps at [1] and share the output?
Thanks, Dan
[1] https://docs.ceph.com/docs/master/rados/troubleshooting/memory-profiling/
On Tue, Aug 11, 2020 at 2:37 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Mark,
here is a first collection of heap profiling data (valid 30 days):
https://files.dtu.dk/u/53HHic_xx5P1cceJ/heap_profiling-2020-08-03.tgz?l
This was collected with the following config settings:
osd dev osd_memory_cache_min 805306368 osd basic osd_memory_target 2147483648
Setting the cache_min value seems to help keeping cache space available. Unfortunately, the above collection is for 12 days only. I needed to restart the OSD and will need to restart it soon again. I hope I can then run a longer sample. The profiling does cause slow ops though.
Maybe you can see something already? It seems to have collected some leaked memory. Unfortunately, it was a period of extremely low load. Basically, with the day of recording the utilization dropped to almost zero.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 21 July 2020 12:57:32 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Quick question: Is there a way to change the frequency of heap dumps? On this page http://goog-perftools.sourceforge.net/doc/heap_profiler.html a function HeapProfilerSetAllocationInterval() is mentioned, but no other way of configuring this. Is there a config parameter or a ceph daemon call to adjust this?
If not, can I change the dump path?
Its likely to overrun my log partition quickly if I cannot adjust either of the two.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 20 July 2020 15:19:05 To: Mark Nelson; Dan van der Ster Cc: ceph-users Subject: [ceph-users] Re: OSD memory leak?
Dear Mark,
thank you very much for the very helpful answers. I will raise osd_memory_cache_min, leave everything else alone and watch what happens. I will report back here.
Thanks also for raising this as an issue.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Mark Nelson <mnelson@redhat.com> Sent: 20 July 2020 15:08:11 To: Frank Schilder; Dan van der Ster Cc: ceph-users Subject: Re: [ceph-users] Re: OSD memory leak?
On 7/20/20 3:23 AM, Frank Schilder wrote:
Dear Mark and Dan,
I'm in the process of restarting all OSDs and could use some quick advice on bluestore cache settings. My plan is to set higher minimum values and deal with accumulated excess usage via regular restarts. Looking at the documentation (https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/), I find the following relevant options (with defaults):
# Automatic Cache Sizing osd_memory_target {4294967296} # 4GB osd_memory_base {805306368} # 768MB osd_memory_cache_min {134217728} # 128MB
# Manual Cache Sizing bluestore_cache_meta_ratio {.4} # 40% ? bluestore_cache_kv_ratio {.4} # 40% ? bluestore_cache_kv_max {512 * 1024*1024} # 512MB
Q1) If I increase osd_memory_cache_min, should I also increase osd_memory_base by the same or some other amount?
osd_memory_base is a hint at how much memory the OSD could consume outside the cache once it's reached steady state. It basically sets a hard cap on how much memory the cache will use to avoid over-committing memory and thrashing when we exceed the memory limit. It's not necessary to get it right, it just helps smooth things out by making the automatic memory tuning less aggressive. IE if you have a 2 GB memory target and a 512MB base, you'll never assign more than 1.5GB to the cache on the assumption that the rest of the OSD will eventually need 512MB to operate even if it's not using that much right now. I think you can probably just leave it alone. What you and Dan appear to be seeing is that this number isn't static in your case but increases over time any way. Eventually I'm hoping that we can automatically account for more and more of that memory by reading the data from the mempools.
Q2) The cache ratio options are shown under the section "Manual Cache Sizing". Do they also apply when cache auto tuning is enabled? If so, is it worth changing these defaults for higher values of osd_memory_cache_min?
They actually do have an effect on the automatic cache sizing and probably shouldn't only be under the manual section. When you have the automatic cache sizing enabled, those options will affect the "fair share" values of the different caches at each cache priority level. IE at priority level 0, if both caches want more memory than is available, those ratios will determine how much each cache gets. If there is more memory available than requested, each cache gets as much as they want and we move on to the next priority level and do the same thing again. So in this case the ratios end up being sort of more like fallback settings for when you don't have enough memory to fulfill all cache requests at a given priority level, but otherwise are not utilized until we hit that limit. The goal with this scheme is to make sure that "high priority" items in each cache get first dibs at the memory even if it might skew the ratios. This might be things like rocksdb bloom filters and indexes, or potentially very recent hot items in one cache vs very old items in another cache. The ratios become more like guidelines than hard limits.
When you change to manual mode, you set an overall bluestore cache size and each cache gets a flat percentage of it based on the ratios. With 0.4/0.4 you will always have 40% for onode, 40% for omap, and 20% for data even if one of those caches does not use all of it's memory.
Many thanks for your help with this. I can't find answers to these questions in the docs.
There might be two reasons for high osd_map memory usage. One is, that our OSDs seem to hold a large number of OSD maps:
I brought this up in our core team standup last week. Not sure if anyone has had time to look at it yet though. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Looks like the image attachment got removed. Please find it here: https://imgur.com/a/3tabzCN ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 31 August 2020 14:42 To: Mark Nelson; Dan van der Ster; ceph-users Subject: [ceph-users] Re: OSD memory leak? Hi Dan and Mark, sorry, took a bit longer. I uploaded a new archive containing files with the following format (https://files.dtu.dk/u/jb0uS6U9LlCfvS5L/heap_profiling-2020-08-31.tgz?l - valid 60 days): - osd.195.profile.*.heap - raw heap dump file - osd.195.profile.*.heap.txt - output of conversion with --text - osd.195.profile.*.heap-base0001.txt - output of conversion with --text against first dump as base - osd.195.*.heap_stats - output of ceph daemon osd.195 heap stats, every hour - osd.195.*.mempools - output of ceph daemon osd.195 dump_mempools, every hour - osd.195.*.perf - output of ceph daemon osd.195 perf dump, every hour, counters are reset Only for the last couple of days are converted files included, post-conversion of everything simply takes too long. Please find also attached a recording of memory usage on one of the relevant OSD nodes. I marked restarts of all OSDs/the host with vertical red lines. What is worrying is the self-amplifying nature of the leak. ts not a linear process, it looks at least quadratic if not exponential. What we are looking for is, given the comparably short uptime, probably still in the lower percentages with increasing rate. The OSDs just started to overrun their limit: top - 14:38:49 up 155 days, 19:17, 1 user, load average: 5.99, 4.59, 4.59 Tasks: 684 total, 1 running, 293 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.9 us, 0.9 sy, 0.0 ni, 89.6 id, 7.6 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 65727628 total, 6937548 free, 41921260 used, 16868820 buff/cache KiB Swap: 93532160 total, 90199040 free, 3333120 used. 6740136 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4099023 ceph 20 0 5918704 3.8g 9700 S 1.7 6.1 378:37.01 /usr/bin/ceph-osd --cluster ceph -f -i 35 --setuser cep+ 4097639 ceph 20 0 5340924 3.0g 11428 S 87.1 4.7 14636:30 /usr/bin/ceph-osd --cluster ceph -f -i 195 --setuser ce+ 4097974 ceph 20 0 3648188 2.3g 9628 S 8.3 3.6 1375:58 /usr/bin/ceph-osd --cluster ceph -f -i 201 --setuser ce+ 4098322 ceph 20 0 3478980 2.2g 9688 S 5.3 3.6 1426:05 /usr/bin/ceph-osd --cluster ceph -f -i 223 --setuser ce+ 4099374 ceph 20 0 3446784 2.2g 9252 S 4.6 3.5 1142:14 /usr/bin/ceph-osd --cluster ceph -f -i 205 --setuser ce+ 4098679 ceph 20 0 3832140 2.2g 9796 S 6.6 3.5 1248:26 /usr/bin/ceph-osd --cluster ceph -f -i 132 --setuser ce+ 4100782 ceph 20 0 3641608 2.2g 9652 S 7.9 3.5 1278:10 /usr/bin/ceph-osd --cluster ceph -f -i 207 --setuser ce+ 4095944 ceph 20 0 3375672 2.2g 8968 S 7.3 3.5 1250:02 /usr/bin/ceph-osd --cluster ceph -f -i 108 --setuser ce+ 4096956 ceph 20 0 3509376 2.2g 9456 S 7.9 3.5 1157:27 /usr/bin/ceph-osd --cluster ceph -f -i 203 --setuser ce+ 4099731 ceph 20 0 3563652 2.2g 8972 S 3.6 3.5 1421:48 /usr/bin/ceph-osd --cluster ceph -f -i 61 --setuser cep+ 4096262 ceph 20 0 3531988 2.2g 9040 S 9.9 3.5 1600:15 /usr/bin/ceph-osd --cluster ceph -f -i 121 --setuser ce+ 4100442 ceph 20 0 3359736 2.1g 9804 S 4.3 3.4 1185:53 /usr/bin/ceph-osd --cluster ceph -f -i 226 --setuser ce+ 4096617 ceph 20 0 3443060 2.1g 9432 S 5.0 3.4 1449:29 /usr/bin/ceph-osd --cluster ceph -f -i 199 --setuser ce+ 4097298 ceph 20 0 3483532 2.1g 9600 S 5.6 3.3 1265:28 /usr/bin/ceph-osd --cluster ceph -f -i 97 --setuser cep+ 4100093 ceph 20 0 3428348 2.0g 9568 S 3.3 3.2 1298:53 /usr/bin/ceph-osd --cluster ceph -f -i 197 --setuser ce+ 4095630 ceph 20 0 3440160 2.0g 8976 S 3.6 3.2 1451:35 /usr/bin/ceph-osd --cluster ceph -f -i 62 --setuser cep+ Generally speaking, increasing the cache minimum seems to help with keeping important information in RAM. Unfortunately, it also means that swap usage starts much earlier. Best regards and thanks for your help, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Dear all, I collected memory allocation data over a period of 2 months; see the graphs here: <https://imgur.com/a/R0q6nzP>. I need to revise my statement about accelerated growth. The new graphs indicate that we are looking at linear growth, that is, probably a small memory leak in a regularly called function. I think the snippets of the heap stats and memory profiling output below should give a clue about where to look. Osd 195 is using about 2.1GB more than it should, the memory limit is 2GB: osd.195 tcmalloc heap stats:------------------------------------------------ MALLOC: 4555926984 ( 4344.9 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 288132120 ( 274.8 MiB) Bytes in central cache freelist MALLOC: + 12879104 ( 12.3 MiB) Bytes in transfer cache freelist MALLOC: + 20619552 ( 19.7 MiB) Bytes in thread cache freelists MALLOC: + 33292288 ( 31.8 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 4910850048 ( 4683.4 MiB) Actual memory used (physical + swap) MALLOC: + 865198080 ( 825.1 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 5776048128 ( 5508.5 MiB) Virtual address space used MALLOC: MALLOC: 470779 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. { "error": "(0) Success", "success": true } It looks like the vast majority of this leak occurs in "ceph::decode"; see the top of the heap profiler allocation stats: Total: 4011.6 MB 1567.9 39.1% 39.1% 1815.3 45.3% ceph::decode 457.7 11.4% 50.5% 457.7 11.4% rocksdb::BlockFetcher::ReadBlockContents 269.7 6.7% 57.2% 269.7 6.7% std::vector::_M_default_append 256.0 6.4% 63.6% 256.0 6.4% rocksdb::Arena::AllocateNewBlock 243.9 6.1% 69.7% 243.9 6.1% std::_Rb_tree::_M_emplace_hint_unique 184.1 4.6% 74.3% 184.1 4.6% CrushWrapper::get_leaves 174.6 4.4% 78.6% 174.6 4.4% ceph::buffer::create_aligned_in_mempool 170.3 4.2% 82.9% 170.3 4.2% ceph::buffer::malformed_input::what 125.2 3.1% 86.0% 191.4 4.8% PGLog::IndexedLog::add 101.1 2.5% 88.5% 101.1 2.5% CrushWrapper::decode_crush_bucket Does this already help? If not, I collected 126GB of data from the heap profiler. It would be great if this leak could be closed. It would be enough to extend the uptime of an OSD to cover usual maintenance windows. By the way, increasing the cache_min value helped a lot. The OSD kept a healthy amount of ONODE items in cache despite the leak. Users noticed the improvement. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 31 August 2020 19:50:57 To: Mark Nelson; Dan van der Ster; ceph-users Subject: [ceph-users] Re: OSD memory leak? Looks like the image attachment got removed. Please find it here: https://imgur.com/a/3tabzCN ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 31 August 2020 14:42 To: Mark Nelson; Dan van der Ster; ceph-users Subject: [ceph-users] Re: OSD memory leak? Hi Dan and Mark, sorry, took a bit longer. I uploaded a new archive containing files with the following format (https://files.dtu.dk/u/jb0uS6U9LlCfvS5L/heap_profiling-2020-08-31.tgz?l - valid 60 days): - osd.195.profile.*.heap - raw heap dump file - osd.195.profile.*.heap.txt - output of conversion with --text - osd.195.profile.*.heap-base0001.txt - output of conversion with --text against first dump as base - osd.195.*.heap_stats - output of ceph daemon osd.195 heap stats, every hour - osd.195.*.mempools - output of ceph daemon osd.195 dump_mempools, every hour - osd.195.*.perf - output of ceph daemon osd.195 perf dump, every hour, counters are reset Only for the last couple of days are converted files included, post-conversion of everything simply takes too long. Please find also attached a recording of memory usage on one of the relevant OSD nodes. I marked restarts of all OSDs/the host with vertical red lines. What is worrying is the self-amplifying nature of the leak. ts not a linear process, it looks at least quadratic if not exponential. What we are looking for is, given the comparably short uptime, probably still in the lower percentages with increasing rate. The OSDs just started to overrun their limit: top - 14:38:49 up 155 days, 19:17, 1 user, load average: 5.99, 4.59, 4.59 Tasks: 684 total, 1 running, 293 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.9 us, 0.9 sy, 0.0 ni, 89.6 id, 7.6 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 65727628 total, 6937548 free, 41921260 used, 16868820 buff/cache KiB Swap: 93532160 total, 90199040 free, 3333120 used. 6740136 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4099023 ceph 20 0 5918704 3.8g 9700 S 1.7 6.1 378:37.01 /usr/bin/ceph-osd --cluster ceph -f -i 35 --setuser cep+ 4097639 ceph 20 0 5340924 3.0g 11428 S 87.1 4.7 14636:30 /usr/bin/ceph-osd --cluster ceph -f -i 195 --setuser ce+ 4097974 ceph 20 0 3648188 2.3g 9628 S 8.3 3.6 1375:58 /usr/bin/ceph-osd --cluster ceph -f -i 201 --setuser ce+ 4098322 ceph 20 0 3478980 2.2g 9688 S 5.3 3.6 1426:05 /usr/bin/ceph-osd --cluster ceph -f -i 223 --setuser ce+ 4099374 ceph 20 0 3446784 2.2g 9252 S 4.6 3.5 1142:14 /usr/bin/ceph-osd --cluster ceph -f -i 205 --setuser ce+ 4098679 ceph 20 0 3832140 2.2g 9796 S 6.6 3.5 1248:26 /usr/bin/ceph-osd --cluster ceph -f -i 132 --setuser ce+ 4100782 ceph 20 0 3641608 2.2g 9652 S 7.9 3.5 1278:10 /usr/bin/ceph-osd --cluster ceph -f -i 207 --setuser ce+ 4095944 ceph 20 0 3375672 2.2g 8968 S 7.3 3.5 1250:02 /usr/bin/ceph-osd --cluster ceph -f -i 108 --setuser ce+ 4096956 ceph 20 0 3509376 2.2g 9456 S 7.9 3.5 1157:27 /usr/bin/ceph-osd --cluster ceph -f -i 203 --setuser ce+ 4099731 ceph 20 0 3563652 2.2g 8972 S 3.6 3.5 1421:48 /usr/bin/ceph-osd --cluster ceph -f -i 61 --setuser cep+ 4096262 ceph 20 0 3531988 2.2g 9040 S 9.9 3.5 1600:15 /usr/bin/ceph-osd --cluster ceph -f -i 121 --setuser ce+ 4100442 ceph 20 0 3359736 2.1g 9804 S 4.3 3.4 1185:53 /usr/bin/ceph-osd --cluster ceph -f -i 226 --setuser ce+ 4096617 ceph 20 0 3443060 2.1g 9432 S 5.0 3.4 1449:29 /usr/bin/ceph-osd --cluster ceph -f -i 199 --setuser ce+ 4097298 ceph 20 0 3483532 2.1g 9600 S 5.6 3.3 1265:28 /usr/bin/ceph-osd --cluster ceph -f -i 97 --setuser cep+ 4100093 ceph 20 0 3428348 2.0g 9568 S 3.3 3.2 1298:53 /usr/bin/ceph-osd --cluster ceph -f -i 197 --setuser ce+ 4095630 ceph 20 0 3440160 2.0g 8976 S 3.6 3.2 1451:35 /usr/bin/ceph-osd --cluster ceph -f -i 62 --setuser cep+ Generally speaking, increasing the cache minimum seems to help with keeping important information in RAM. Unfortunately, it also means that swap usage starts much earlier. Best regards and thanks for your help, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Dan van der Ster
-
Frank Schilder
-
Mark Nelson