Replying to the other questions:
- Free Memory in ganglia is derived from "MemFree" in /proc/meminfo
- Memory Buffers in ganglia is derived from "Buffers" in /proc/meminfo
- On this host, the OSDs are 6TB. On other hosts we have 10TB OSDs
- "osd memory target" is set to ~ 4.5 GB (actually, while debugging this
issue, I have just lowered the value to 3.2 GB)
- "ceph tell osd.x heap stats" basically always reports 0 (or a very low
value) for "Bytes in page heap freelist" and a heap release doesn't change
the memory usage
- I can agree that swap is antiquated. But so far it was simply not used
and didn't cause any problems. At any rate I am now going to remove the
swap (or setting the swappiness to 0).
Thanks again !
Cheers, Massimo
On Thu, Feb 6, 2020 at 6:28 PM Anthony D'Atri <aad(a)dreamsnake.net> wrote:
Attachments are usually filtered by mailing lists.
Yours did not come
through. A URL to Skitch or some other hosting works better.
Your kernel version sounds like RHEL / CentOS? I can say that memory
accounting definitely did change between upstream 3.19 and 4.9
osd04-cephstorage1-gsc:~ # head /proc/meminfo
MemTotal: 197524684 kB
MemFree: 80388504 kB
MemAvailable: 86055708 kB
Buffers: 633768 kB
Cached: 4705408 kB
SwapCached: 0 kB
Specifically, node_memory_Active as reported by node_exporter changes
dramatically, and MemAvailable is the more meaningful metric. What is your
“FreeMem” metric actually derived from?
64GB for 10 OSDs might be on the light side, how large are those OSDs?
For sure swap is antiquated. If your systems have any swap provisioned at
all, you’re doing it wrong. I’ve had good results setting it to 1.
Do `ceph daemon osd.xx heap stats`, see if your OSD processes have much
unused memory that has not been released to the OS. If they do, “heap
release” can be useful.
On Feb 6, 2020, at 9:08 AM, Massimo Sgaravatto
<
massimo.sgaravatto(a)gmail.com> wrote:
Dear all
In the mid of January I updated my ceph cluster from Luminous to
Nautilus.
Attached you can see the memory metrics collected on one OSD node (I see
the very same behavior on all OSD hosts) graphed via Ganglia
This is Centos 7 node, with 64 GB of RAM, hosting 10 OSDs.
So before the update there were about 20 GB of FreeMem.
Now FreeMem is basically 0, but I see 20 GB of Buffers,
I guess this triggered some swapping, probably because I forgot to
set vm.swappiness to 0 (it was set to 60, the default value).
I was wondering if this the expected behavior
PS: Actually besides updating ceph, I also updated all the other packages
(yum update), so I am not sure that this different memory usage is
because
of the ceph update
For the record in this update the kernel was updated from 3.10.0-1062.1.2
to 3.10.0-1062.9.1
Thanks, Massimo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io