Thanks for your feedback The Ganglia graphs are available here: https://cernbox.cern.ch/index.php/s/0xBDVwNkRqcoGdF Replying to the other questions: - Free Memory in ganglia is derived from "MemFree" in /proc/meminfo - Memory Buffers in ganglia is derived from "Buffers" in /proc/meminfo - On this host, the OSDs are 6TB. On other hosts we have 10TB OSDs - "osd memory target" is set to ~ 4.5 GB (actually, while debugging this issue, I have just lowered the value to 3.2 GB) - "ceph tell osd.x heap stats" basically always reports 0 (or a very low value) for "Bytes in page heap freelist" and a heap release doesn't change the memory usage - I can agree that swap is antiquated. But so far it was simply not used and didn't cause any problems. At any rate I am now going to remove the swap (or setting the swappiness to 0). Thanks again ! Cheers, Massimo On Thu, Feb 6, 2020 at 6:28 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
Attachments are usually filtered by mailing lists. Yours did not come through. A URL to Skitch or some other hosting works better.
Your kernel version sounds like RHEL / CentOS? I can say that memory accounting definitely did change between upstream 3.19 and 4.9
osd04-cephstorage1-gsc:~ # head /proc/meminfo MemTotal: 197524684 kB MemFree: 80388504 kB MemAvailable: 86055708 kB Buffers: 633768 kB Cached: 4705408 kB SwapCached: 0 kB
Specifically, node_memory_Active as reported by node_exporter changes dramatically, and MemAvailable is the more meaningful metric. What is your “FreeMem” metric actually derived from?
64GB for 10 OSDs might be on the light side, how large are those OSDs?
For sure swap is antiquated. If your systems have any swap provisioned at all, you’re doing it wrong. I’ve had good results setting it to 1.
Do `ceph daemon osd.xx heap stats`, see if your OSD processes have much unused memory that has not been released to the OS. If they do, “heap release” can be useful.
On Feb 6, 2020, at 9:08 AM, Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
Dear all
In the mid of January I updated my ceph cluster from Luminous to Nautilus.
Attached you can see the memory metrics collected on one OSD node (I see the very same behavior on all OSD hosts) graphed via Ganglia This is Centos 7 node, with 64 GB of RAM, hosting 10 OSDs.
So before the update there were about 20 GB of FreeMem. Now FreeMem is basically 0, but I see 20 GB of Buffers,
I guess this triggered some swapping, probably because I forgot to set vm.swappiness to 0 (it was set to 60, the default value).
I was wondering if this the expected behavior
PS: Actually besides updating ceph, I also updated all the other packages (yum update), so I am not sure that this different memory usage is because of the ceph update For the record in this update the kernel was updated from 3.10.0-1062.1.2 to 3.10.0-1062.9.1
Thanks, Massimo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io