cephadm and kernel memory usage
Hi, So after, looking into OSDs memory usage, which seem to be fine, on a v16.2.13 running with cephadm, on el8, it seems that the kernel is using a lot of memory. # smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 65.0G 18.6G 46.4G userspace memory 50.1G 260.5M 49.9G free memory 9.9G 9.9G 0 ---------------------------------------------------------- 125.0G 28.8G 96.3G Comparing with a similar other cluster, same OS, same ceph version, but running packages instead if containers, and machines have a little bit more memory: # smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 52.8G 50.5G 2.4G userspace memory 123.9G 198.5M 123.7G free memory 10.6G 10.6G 0 ---------------------------------------------------------- 187.3G 61.3G 126.0G Does anyone have an idea why when using containers with podman the kernel needs a lot more memory? Luis Domingues Proton AG
Hi, Can you paste `free -h` output for this hosts? k Sent from my iPhone
On 24 Jul 2023, at 14:42, Luis Domingues <luis.domingues@proton.ch> wrote:
Hi,
So after, looking into OSDs memory usage, which seem to be fine, on a v16.2.13 running with cephadm, on el8, it seems that the kernel is using a lot of memory.
# smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 65.0G 18.6G 46.4G userspace memory 50.1G 260.5M 49.9G free memory 9.9G 9.9G 0 ---------------------------------------------------------- 125.0G 28.8G 96.3G
Comparing with a similar other cluster, same OS, same ceph version, but running packages instead if containers, and machines have a little bit more memory:
# smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 52.8G 50.5G 2.4G userspace memory 123.9G 198.5M 123.7G free memory 10.6G 10.6G 0 ---------------------------------------------------------- 187.3G 61.3G 126.0G
Does anyone have an idea why when using containers with podman the kernel needs a lot more memory?
Luis Domingues Proton AG _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Of course: free -h total used free shared buff/cache available Mem: 125Gi 96Gi 9.8Gi 4.0Gi 19Gi 7.6Gi Swap: 0B 0B 0B Luis Domingues Proton AG ------- Original Message ------- On Monday, July 24th, 2023 at 16:42, Konstantin Shalygin <k0ste@k0ste.ru> wrote:
Hi,
Can you paste `free -h` output for this hosts?
k Sent from my iPhone
On 24 Jul 2023, at 14:42, Luis Domingues luis.domingues@proton.ch wrote:
Hi,
So after, looking into OSDs memory usage, which seem to be fine, on a v16.2.13 running with cephadm, on el8, it seems that the kernel is using a lot of memory.
# smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 65.0G 18.6G 46.4G userspace memory 50.1G 260.5M 49.9G free memory 9.9G 9.9G 0 ---------------------------------------------------------- 125.0G 28.8G 96.3G
Comparing with a similar other cluster, same OS, same ceph version, but running packages instead if containers, and machines have a little bit more memory:
# smem -t -w -k Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 52.8G 50.5G 2.4G userspace memory 123.9G 198.5M 123.7G free memory 10.6G 10.6G 0 ---------------------------------------------------------- 187.3G 61.3G 126.0G
Does anyone have an idea why when using containers with podman the kernel needs a lot more memory?
Luis Domingues Proton AG _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Good,
On 24 Jul 2023, at 20:01, Luis Domingues <luis.domingues@proton.ch> wrote:
Of course:
free -h total used free shared buff/cache available Mem: 125Gi 96Gi 9.8Gi 4.0Gi 19Gi 7.6Gi Swap: 0B 0B 0B
As we can see, actually you have ~30GiB free (9.8GiB is not used & 19GiB is a page cache) With this command you can determine what process actually use memory & how much ``` ps -eo size,pid,user,command | \ awk '{ hr=$1/1024 ; printf("%13.6f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | \ sort -n ``` k
Hi, First, thank you for taking time to reply to me. However, my question was not on user-space memory neither on cache usage, as I can see on my machines everything sums up quite nicely. My question is: with packages, the non-cache kernel memory is around 2G to 3G, while with Podman usage, it is more around 10G, and it can go up to 40G-50G. Do anyone knows if this is expected and why this is the case? Maybe this is a podman related question and ceph-dev is not the best place to ask this kind of question, but maybe someone using cephadm saw similar behavior. Luis Domingues Proton AG ------- Original Message ------- On Tuesday, July 25th, 2023 at 11:42, Konstantin Shalygin <k0ste@k0ste.ru> wrote:
Good,
On 24 Jul 2023, at 20:01, Luis Domingues luis.domingues@proton.ch wrote:
Of course:
free -h total used free shared buff/cache available Mem: 125Gi 96Gi 9.8Gi 4.0Gi 19Gi 7.6Gi Swap: 0B 0B 0B
As we can see, actually you have ~30GiB free (9.8GiB is not used & 19GiB is a page cache) With this command you can determine what process actually use memory & how much
`ps -eo size,pid,user,command | \\ awk '{ hr=$1/1024 ; printf("%13.6f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | \\ sort -n`
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Without determining what exactly process (kernel or userspace) "eat" memory, the ceph-users can't tell what exactly use memory, because don't see your display with your eyes 🙂 You should run this commands on good & bad hosts to see the real difference. This may be related to kernel version, or Ceph options in container config or ... k Sent from my iPhone
On 26 Jul 2023, at 07:26, Luis Domingues <luis.domingues@proton.ch> wrote:
First, thank you for taking time to reply to me.
However, my question was not on user-space memory neither on cache usage, as I can see on my machines everything sums up quite nicely.
My question is: with packages, the non-cache kernel memory is around 2G to 3G, while with Podman usage, it is more around 10G, and it can go up to 40G-50G. Do anyone knows if this is expected and why this is the case?
Maybe this is a podman related question and ceph-dev is not the best place to ask this kind of question, but maybe someone using cephadm saw similar behavior.
That's the weird thing. Processes and user-space memory is the same in good memory and bad memory. ceph-osd memory usage looks good in all machines, cache is more of less the same. When I do a ps, htop or any other process review all look good, and coherent between all machines, containers or not. Only difference I can see is using smem on the noncache kernel memory on containerized machines. Maybe it's a podman issue, maybe a kernel. It does not seem related to ceph directly. I just asked here to see if anyone got the same issue. Anyway, thanks for your time. Luis Domingues Proton AG ------- Original Message ------- On Wednesday, July 26th, 2023 at 09:01, Konstantin Shalygin <k0ste@k0ste.ru> wrote:
Without determining what exactly process (kernel or userspace) "eat" memory, the ceph-users can't tell what exactly use memory, because don't see your display with your eyes 🙂
You should run this commands on good & bad hosts to see the real difference. This may be related to kernel version, or Ceph options in container config or ...
k Sent from my iPhone
On 26 Jul 2023, at 07:26, Luis Domingues luis.domingues@proton.ch wrote:
First, thank you for taking time to reply to me.
However, my question was not on user-space memory neither on cache usage, as I can see on my machines everything sums up quite nicely.
My question is: with packages, the non-cache kernel memory is around 2G to 3G, while with Podman usage, it is more around 10G, and it can go up to 40G-50G. Do anyone knows if this is expected and why this is the case?
Maybe this is a podman related question and ceph-dev is not the best place to ask this kind of question, but maybe someone using cephadm saw similar behavior.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Konstantin Shalygin
-
Luis Domingues