On 5/12/20 1:57 PM, Blaine Gardner wrote: Correct Specifically the OSD will attempt to keep the mapped memory below the osd_memory_target but with potential, hopefully small, temporary overages.  The bigger a recent overage is the more aggressively the priority cache manager will react when decreasing cache memory allocation.  All of this is in relation to mapped memory however, not RSS memory. Correct Correct, we can't explicitly force the kernel to reclaim memory. That's probably fairly reasonable in most cases though off at the extremes (dedicating 21GB for a single OSD with a 4 GB memory target is almost certainly overkill). My take is that unless we are strictly allocating memory from pre-allocated pools (which we are not) we can't make any guarantee that a Ceph daemon is going to fit within a specific memory allocation at any given point in time.  The osd_memory_target code works surprisingly well for what it is, but as discussed above it's not a guarantee (which is precisely why it's called a target).  That means we can either leave the limits off, or set them to something high enough where we feel relatively confident the OSD won't go OOM, but we'll still catch a misbehaving daemon before it takes other containers (or the whole container node) down. There really are multiple ways an OSD can use more RSS memory than expected.  We can have temporary overages due to sudden allocations in the OSD that we haven't yet compensated for (We try to keep some wiggle room and periodically check memory usage to adjust the caches down, but this doesn't happen instantaneously). You can have fragmentation below the OSD where we might have freed memory that can't be reclaimed.  They may be a large memory allocation that we can't fully compensate for even after shrinking caches to their minimums.  This is especially true for smaller than default osd_memory_targets.  The more memory you give each container over the target the less likely you are to hit a situation where a spike causes an OOM kill.  How much extra you need per container is incredibly complicated and based on a number of factors including fragmentation, transparent huge pages settings, PG count, incoming write rate, rocksdb memtable sizes, rocksdb memtable flushing speed, rocksdb compaction rate, osd_memory_target, bluestore_cache_autotune_chunk_size, osd_memory_expected_fragmentation, osd_memory_cache_resize_interval, and various other things. Potentially if there was some way for the OSD to negotiate with the container it might be able to adjust that overage amount on the fly (say start at 1GB and adjust over time based on historical overages), but right now I don't think there's any capability to do that.|||||||||||||||||| Originally I think the idea was we were going to give the OSD some percentage amount higher like 20% but we kept periodically exceeding it with the default 4MB target.  That also was when THP was enabled by default and we were seeing large amounts of memory space amplification due to fragmentation (and there may have been some other bugs regarding how the limit was calculated).  I'm not sure if there are currently any container limits in place at all, but one of the rook guys can probably say what the current status is.