On 5/12/20 1:57 PM, Blaine Gardner wrote:

(Please use reply-all so that people I've explicitly tagged and I can continue to get direct email replies)

I have a conundrum around developing best practices for Rook/Ceph clusters around OSD memory targets and hardware recommendations for OSDs. I want to lay down some bulleted notes first. - 'osd_memory_target' defaults to 4GB

- OSDs attempt to keep memory allocation to 'osd_memory_target' Specifically the OSD will attempt to keep the mapped memory below the osd_memory_target but with potential, hopefully small, temporary overages.  The bigger a recent overage is the more aggressively the

Correct priority cache manager will react when decreasing cache memory allocation.  All of this is in relation to mapped memory however, not RSS memory.

- BUT... this is only best-effort

Correct

- AND... there is no guarantee that the kernel will actually reclaim memory that OSDs release/unmap Correct, we can't explicitly force the kernel to reclaim memory. - Therefore, we (SUSE) have developed a recommendation that ... Total OSD node RAM required = (num OSDs) x (1 GB + osd_memory_target) + 16 GB

That's probably fairly reasonable in most cases though off at the extremes (dedicating 21GB for a single OSD with a 4 GB memory target is almost certainly overkill).

- In Rook/Kubernetes, Ceph OSDs will read the POD_MEMORY_REQUEST and POD_MEMORY_LIMIT env vars to infer a new default value for 'osd_memory_target' - POD_MEMORY_REQUEST translates directly to 'osd_memory_target' 1:1 - POD_MEMORY_LIMIT (if REQUEST is unset) will set 'osd_memory_target' using the formula ( LIMIT x osd_memory_target_cgroup_limit_ratio ) - the default 'osd_memory_target' will be = min(REQUEST, LIMIT*ratio) - Lars has suggested that setting limits is not a best practice for Ceph; when limits are encountered, Ceph is likely in a failure state, and killing daemons could result in a "thundering herds" distributed systems problem

As you can see, there is a self-referential problem here. The OSD hardware recommendation should inform us how to set k8s resource limits for OSDs; however, doing so will affect osd_memory_target, which alters the recommendation, which further alters our k8s resource limit circularly forever.

My take is that unless we are strictly allocating memory from pre-allocated pools (which we are not) we can't make any guarantee that a Ceph daemon is going to fit within a specific memory allocation at any given point in time.  The osd_memory_target code works surprisingly well for what it is, but as discussed above it's not a guarantee (which is precisely why it's called a target).  That means we can either leave the limits off, or set them to something high enough where we feel relatively confident the OSD won't go OOM, but we'll still catch a misbehaving daemon before it takes other containers (or the whole container node) down.

We can address this issue with a semi-workaround currently: set osd_memory_target explicitly in Ceph's config, and set an appropriate k8s resource request matching (osd_memory_target + 1GB + some extra) to meet the hardware recommendation. However, means that the Ceph feature of setting osd_memory_target based on resource requests isn't really used because it doesn't behave to actual best practices. And setting a realistic k8s resource request is useful for kubernetes so that it won't schedule more daemons onto a node than the node can realistically support.

There really are multiple ways an OSD can use more RSS memory than expected.  We can have temporary overages due to sudden allocations in the OSD that we haven't yet compensated for (We try to keep some wiggle room and periodically check memory usage to adjust the caches down, but this doesn't happen instantaneously). You can have fragmentation below the OSD where we might have freed memory that can't be reclaimed.  They may be a large memory allocation that we can't fully compensate for even after shrinking caches to their minimums.  This is especially true for smaller than default osd_memory_targets.  The more memory you give each container over the target the less likely you are to hit a situation where a spike causes an OOM kill.  How much extra you need per container is incredibly complicated and based on a number of factors including fragmentation, transparent huge pages settings, PG count, incoming write rate, rocksdb memtable sizes, rocksdb memtable flushing speed, rocksdb compaction rate, osd_memory_target, bluestore_cache_autotune_chunk_size, osd_memory_expected_fragmentation, osd_memory_cache_resize_interval, and various other things. Potentially if there was some way for the OSD to negotiate with the container it might be able to adjust that overage amount on the fly (say start at 1GB and adjust over time based on historical overages), but right now I don't think there's any capability to do that.||||||||||||||||||

Long-term, I wonder if it is good to add into Ceph a computation that [[ osd_memory_target = REQUEST + osd_memory_request_overhead ]] where the osd_memory_requests overhead defaults to 1GB or somewhat higher.

Originally I think the idea was we were going to give the OSD some percentage amount higher like 20% but we kept periodically exceeding it with the default 4MB target.  That also was when THP was enabled by default and we were seeing large amounts of memory space amplification due to fragmentation (and there may have been some other bugs regarding how the limit was calculated).  I'm not sure if there are currently any container limits in place at all, but one of the rook guys can probably say what the current status is.

Please discuss, and let me know if anything here seems like I've gotten it wrong or if there are other options I haven't seen.

Cheers, and happy Tuesday! Blaine _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io