Hi Harald,
Thanks! So you can see from the perf dump that the target bytes are
a little below 4GB, but the mapped bytes are around 7GB. The priority
cache manager has reacted by setting the "cache_bytes" to 128MB which
is the minimum global value and each cache is getting 64MB (the local
minimum value per cache). Ultimately this means the priority cache
manager has basically told all of the caches to shrink to their
smallest possible values so it's doing the right thing. So the next
question is why buffer_anon is so huge. Looking at the mempool stats,
there are not that many items but still a lot of memory used. On
average those items in buffer_anon are ~150K. It can't be just
buffer anon though, you've got several gigabytes of mapped memory
being used beyond that and around 4GB of unmapped memory that
tcmalloc should be freeing every iteration of the priority cache
manager.
So next questions: What version of Ceph is this, and do you have
transparent huge pages enabled? We automatically disable it now, but
if you are running an older version you might want to disable (or at
least set it to madvise) manually. Also, what kind of workload is
hitting the OSDs? If you can reliably make it grow you could try
doing a heap profile at the same time the workload is going on and
see if you can see where the memory is being used.
Mark
On 5/20/20 7:36 AM, Harald Staub wrote:
Hi Mark
Thank you for you explanations! Some numbers of this example osd below.
Cheers
Harry
From dump mempools:
"buffer_anon": {
"items": 29012,
"bytes": 4584503367
},
From perf dump:
"prioritycache": {
"target_bytes": 3758096384,
"mapped_bytes": 7146692608,
"unmapped_bytes": 3825983488,
"heap_bytes": 10972676096,
"cache_bytes": 134217728
},
"prioritycache:data": {
"pri0_bytes": 0,
"pri1_bytes": 0,
"pri2_bytes": 0,
"pri3_bytes": 0,
"pri4_bytes": 0,
"pri5_bytes": 0,
"pri6_bytes": 0,
"pri7_bytes": 0,
"pri8_bytes": 0,
"pri9_bytes": 0,
"pri10_bytes": 0,
"pri11_bytes": 0,
"reserved_bytes": 67108864,
"committed_bytes": 67108864
},
"prioritycache:kv": {
"pri0_bytes": 0,
"pri1_bytes": 0,
"pri2_bytes": 0,
"pri3_bytes": 0,
"pri4_bytes": 0,
"pri5_bytes": 0,
"pri6_bytes": 0,
"pri7_bytes": 0,
"pri8_bytes": 0,
"pri9_bytes": 0,
"pri10_bytes": 0,
"pri11_bytes": 0,
"reserved_bytes": 67108864,
"committed_bytes": 67108864
},
"prioritycache:meta": {
"pri0_bytes": 0,
"pri1_bytes": 0,
"pri2_bytes": 0,
"pri3_bytes": 0,
"pri4_bytes": 0,
"pri5_bytes": 0,
"pri6_bytes": 0,
"pri7_bytes": 0,
"pri8_bytes": 0,
"pri9_bytes": 0,
"pri10_bytes": 0,
"pri11_bytes": 0,
"reserved_bytes": 67108864,
"committed_bytes": 67108864
},
On 20.05.20 14:05, Mark Nelson wrote:
Hi Harald,
Any idea what the priority_cache_manger perf counters show? (or you
can also enable debug osd / debug priority_cache_manager) The osd
memory autotuning works by shrinking the bluestore and rocksdb
caches to some target value to try and keep the mapped memory of
the process bellow the osd_memory_target. In some cases it's
possible that something other than the caches are using the memory
(usually pglog) or there's tons of pinned stuff in the cache that
for some reason can't be evicted. Knowing the cache tuning stats
might help tell if it's trying to shrink the caches and can't for
some reason or if there's something else going on.
Thanks,
Mark
On 5/20/20 6:10 AM, Harald Staub wrote:
> As a follow-up to our recent memory problems with OSDs (with high
> pglog values:
>
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LJPJZPBSQRJ…
> ), we also see high buffer_anon values. E.g. more than 4 GB, with
> "osd memory target" set to 3 GB. Is there a way to restrict it?
>
> As it is called "anon", I guess that it would first be necessary
> to find out what exactly is behind this?
>
> Well maybe it is just as Wido said, with lots of small objects,
> there will be several problems.
>
> Cheers
> Harry
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io