Nautilus OSD memory consumption?
The OOM-killer is on the rampage and striking down hapless OSDs when the cluster is under heavy client IO. The memory target does not seem to be much of a limit, is this intentional? root@cnx-11:~# ceph-conf --show-config|fgrep osd_memory_target osd_memory_target = 4294967296 osd_memory_target_cgroup_limit_ratio = 0.800000 root@cnx-31:~# pmap 4327|fgrep total total 6794892K Are there any tips for controlling the OSD memory consumption? The hosts involved have 128GB or 192GB memory, 12 x OSDs (SATA), so even with 4GB per OSD there should be a large amount of free memory.
more examples of rampant OSD memory consumption: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1326773 ceph 20 0 11.585g 0.011t 34728 S 110.3 8.6 14:26.87 ceph-osd 204622 ceph 20 0 16.414g 0.015t 34808 S 100.3 12.5 17:53.36 ceph-osd 5706 ceph 20 0 5235600 4.248g 32488 S 11.6 3.4 19:42.63 ceph-osd 5709 ceph 20 0 10.453g 9.709g 32380 S 3.6 7.7 44:51.29 ceph-osd 5826 ceph 20 0 5242920 4.245g 32408 S 2.0 3.4 17:27.84 ceph-osd 5705 ceph 20 0 5252192 4.264g 32340 S 1.7 3.4 24:06.11 ceph-osd 5716 ceph 20 0 5440428 4.436g 32556 S 1.7 3.5 24:05.89 ceph-osd 5720 ceph 20 0 5359888 4.359g 32592 S 1.7 3.5 17:18.63 ceph-osd 5836 ceph 20 0 5081448 4.099g 32468 S 1.7 3.3 16:56.52 ceph-osd OSD memory usage in the high-teens seems to be common across all hosts.
Have you tried dumping the mempools? The memory autotuner will grow or shrink the bluestore caches to try to keep the total OSD process mapped memory just under the target. If there's a memory leak or some other part of the OSD is using more memory than it should, it will shrink the caches to some base minimum at which point it can't do anything more and the memory usage will exceed the target. It sounds like you might be hitting that case. One reason this can happen for example is if you have a huge number of PGs (like many thousands per OSD). Mark On 2/25/20 9:02 PM, Nigel Williams wrote:
The OOM-killer is on the rampage and striking down hapless OSDs when the cluster is under heavy client IO.
The memory target does not seem to be much of a limit, is this intentional?
root@cnx-11:~# ceph-conf --show-config|fgrep osd_memory_target osd_memory_target = 4294967296 osd_memory_target_cgroup_limit_ratio = 0.800000
root@cnx-31:~# pmap 4327|fgrep total total 6794892K
Are there any tips for controlling the OSD memory consumption?
The hosts involved have 128GB or 192GB memory, 12 x OSDs (SATA), so even with 4GB per OSD there should be a large amount of free memory. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, 26 Feb 2020 at 23:56, Mark Nelson <mnelson@redhat.com> wrote:
Have you tried dumping the mempools? ... One reason this can happen for example is if you have a huge number of PGs (like many thousands per OSD).
We are relying on the pg autoscaler to set the PGs, and so far it seems to do the right thing. root@cnx-16:~# ceph daemon osd.158 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4794632, "bytes": 38357056 }, "bluestore_cache_data": { "items": 142, "bytes": 2265088 }, "bluestore_cache_onode": { "items": 14217, "bytes": 9326352 }, "bluestore_cache_other": { "items": 1886849, "bytes": 44817175 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 7, "bytes": 5096 }, "bluestore_writing_deferred": { "items": 59, "bytes": 296579 }, "bluestore_writing": { "items": 49, "bytes": 172642 }, "bluefs": { "items": 2133, "bytes": 65512 }, "buffer_anon": { "items": 14614, "bytes": 4312428946 }, "buffer_meta": { "items": 8751, "bytes": 770088 }, "osd": { "items": 95, "bytes": 1236520 }, "osd_mapbl": { "items": 50, "bytes": 8278656 }, "osd_pglog": { "items": 65858895, "bytes": 5338210037 }, "osdmap": { "items": 174218, "bytes": 5109376 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 72754711, "bytes": 9761339123 } } } OSD.158 pglog is 5GB (the process is using 11GB) - and we are seeing lots of these: OSD_SLOW_PING_TIME_BACK Long heartbeat ping times on back interface seen, longest is 12252.480 msec Slow heartbeat ping on back interface from osd.261 to osd.158 12252.480 msec Slow heartbeat ping on back interface from osd.312 to osd.158 10729.374 msec Slow heartbeat ping on back interface from osd.216 to osd.158 9827.114 msec Slow heartbeat ping on back interface from osd.265 to osd.158 8931.290 msec Slow heartbeat ping on back interface from osd.168 to osd.158 8889.351 msec Slow heartbeat ping on back interface from osd.159 to osd.158 8389.767 msec Slow heartbeat ping on back interface from osd.290 to osd.158 8283.217 msec Slow heartbeat ping on back interface from osd.221 to osd.158 8190.528 msec Slow heartbeat ping on back interface from osd.197 to osd.158 7973.119 msec Slow heartbeat ping on back interface from osd.96 to osd.158 7901.591 msec Truncated long network list. Use ceph daemon mgr.# dump_osd_network for more information so something is leaking memory and/or traversing some data structure is limiting the ability of the OSD to accept/transmit responses to neighbors.
participants (2)
-
Mark Nelson
-
Nigel Williams