Hi Mark Thank you for you explanations! Some numbers of this example osd below. Cheers Harry From dump mempools: "buffer_anon": { "items": 29012, "bytes": 4584503367 }, From perf dump: "prioritycache": { "target_bytes": 3758096384, "mapped_bytes": 7146692608, "unmapped_bytes": 3825983488, "heap_bytes": 10972676096, "cache_bytes": 134217728 }, "prioritycache:data": { "pri0_bytes": 0, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 0, "pri4_bytes": 0, "pri5_bytes": 0, "pri6_bytes": 0, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 0, "reserved_bytes": 67108864, "committed_bytes": 67108864 }, "prioritycache:kv": { "pri0_bytes": 0, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 0, "pri4_bytes": 0, "pri5_bytes": 0, "pri6_bytes": 0, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 0, "reserved_bytes": 67108864, "committed_bytes": 67108864 }, "prioritycache:meta": { "pri0_bytes": 0, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 0, "pri4_bytes": 0, "pri5_bytes": 0, "pri6_bytes": 0, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 0, "reserved_bytes": 67108864, "committed_bytes": 67108864 }, On 20.05.20 14:05, Mark Nelson wrote:
Hi Harald,
Any idea what the priority_cache_manger perf counters show? (or you can also enable debug osd / debug priority_cache_manager) The osd memory autotuning works by shrinking the bluestore and rocksdb caches to some target value to try and keep the mapped memory of the process bellow the osd_memory_target. In some cases it's possible that something other than the caches are using the memory (usually pglog) or there's tons of pinned stuff in the cache that for some reason can't be evicted. Knowing the cache tuning stats might help tell if it's trying to shrink the caches and can't for some reason or if there's something else going on.
Thanks,
Mark
On 5/20/20 6:10 AM, Harald Staub wrote:
As a follow-up to our recent memory problems with OSDs (with high pglog values: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LJPJZPBSQRJN... ), we also see high buffer_anon values. E.g. more than 4 GB, with "osd memory target" set to 3 GB. Is there a way to restrict it?
As it is called "anon", I guess that it would first be necessary to find out what exactly is behind this?
Well maybe it is just as Wido said, with lots of small objects, there will be several problems.
Cheers Harry _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io