SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
Hi List! We are in the middle of migration from Ceph v16 to v18. As a first step of migration we decided to change OS version on all OSD hosts from Rocky Linux 8.10 to Rocky Linux 9.5 (by using clean reinstall). We also upgraded OSD Block disks from 4TB to 10TB. Now we have 6 OSD hosts running Rocky 9 with new 10TB disks (8 OSD Block disks per Host). OSD hosts have 64GB of RAM. At this moment Ceph cluster is still on version 16.2.15 "Pacific". Migration to Ceph 18.2.x will be performed later after upgrading to Rocky 9 all remaining parts (MGR+MON, MDS, RGW). Problem: After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%. Docker containers running OSD instances are restarted from time to time (and there is a lot of ugly "Call Trace" messages in dmesg). I was wondering if anyone else has similar issues? Will it be gone after migration to Ceph v18? P.S. Configuration Details: ===== [ceph: root@ceph-mon12 /]# ceph versions { "mon": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "osd": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 48 }, "mds": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "rgw": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "overall": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 58 } } [ceph: root@ceph-mon12 /]# === Old Rocky 8 OSD Host: [root@ceph-osd11 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1a4c959eb813 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-37 e37a366347db quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-1 0fdfb95fd631 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-14 276ad8a11148 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-31 9dbd30d11124 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-19 0ad59f583e6a quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-7 e551a652c3ae quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-42 07ff632ae3e1 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-25 605b7a1c00f7 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd11 fe2a6b8a35e8 quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd11 [root@ceph-osd11 ~]# uptime 10:52:59 up 216 days, 20:12, 1 user, load average: 0.20, 0.41, 0.53 [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi [root@ceph-osd11 ~]# rpm -qa | grep docker docker-buildx-plugin-0.14.0-1.el8.x86_64 docker-ce-cli-26.1.3-1.el8.x86_64 docker-ce-26.1.3-1.el8.x86_64 docker-scan-plugin-0.23.0-3.el8.x86_64 docker-ce-rootless-extras-26.1.3-1.el8.x86_64 docker-compose-plugin-2.27.0-1.el8.x86_64 [root@ceph-osd11 ~]# cat /etc/rocky-release Rocky Linux release 8.10 (Green Obsidian) [root@ceph-osd11 ~]# uname -a Linux ceph-osd11 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 8 17:47:08 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux === New Rocky 9 OSD Host: [root@ceph-osd17 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5cc58e4a77b2 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 days ago Up 4 days ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 3a60fecf648d quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 9c20407e79eb quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 9deadafef9dd quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 fcfe62a25fd9 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 38d2d96cc491 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 e29c6bbc1ae7 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 40346a7a45ea quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 43c3e3a65531 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 d9e436f9788c quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 [root@ceph-osd17 ~]# uptime 10:25:10 up 122 days, 22:37, 1 user, load average: 0.37, 0.26, 0.27 [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi [root@ceph-osd17 ~]# rpm -qa | grep docker docker-compose-plugin-2.31.0-1.el9.x86_64 docker-buildx-plugin-0.19.2-1.el9.x86_64 docker-ce-cli-27.4.0-1.el9.x86_64 docker-ce-rootless-extras-27.4.0-1.el9.x86_64 docker-ce-27.4.0-1.el9.x86_64 [root@ceph-osd17 ~]# cat /etc/rocky-release Rocky Linux release 9.5 (Blue Onyx) [root@ceph-osd17 ~]# uname -a Linux ceph-osd17 5.14.0-503.15.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 26 17:24:29 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux =====
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%.
Why do you have swap configured at all? I suggest disabling swap in fstab and rebooting serially.
Hi Anthony. Yes we have swap enabled. Old Rocky 8 and new Rocky 9 OSD hosts both configured with 8G of swap. I will try to disable swap, but I guess what we will get a lot of Out Of Memory messages on OSD hosts. = old: [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi = new: [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB): [root@ceph-osd17 ~]# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 5cc58e4a77b2 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 0.28% 3.576GiB / 62.28GiB 5.74% 0B / 0B 3.9TB / 975GB 62 3a60fecf648d ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 0.28% 2.912GiB / 62.28GiB 4.68% 0B / 0B 100TB / 45.7TB 62 9c20407e79eb ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 0.28% 2.905GiB / 62.28GiB 4.66% 0B / 0B 93TB / 35.8TB 62 9deadafef9dd ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 0.56% 3.624GiB / 62.28GiB 5.82% 0B / 0B 102TB / 39.2TB 62 fcfe62a25fd9 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 0.40% 2.968GiB / 62.28GiB 4.77% 0B / 0B 83.2TB / 34.8TB 62 38d2d96cc491 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 1.42% 2.666GiB / 62.28GiB 4.28% 0B / 0B 105TB / 38.1TB 62 e29c6bbc1ae7 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 2.01% 3.687GiB / 62.28GiB 5.92% 0B / 0B 106TB / 44.6TB 62 40346a7a45ea ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 0.69% 2.748GiB / 62.28GiB 4.41% 0B / 0B 103TB / 41.4TB 62 43c3e3a65531 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 0.00% 3.73MiB / 62.28GiB 0.01% 0B / 0B 567MB / 18MB 2 d9e436f9788c ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 15.04% 30.25MiB / 62.28GiB 0.05% 0B / 0B 410MB / 14.6MB 61 But they also are biggest swap consumers: [root@ceph-osd17 ~]# for file in /proc/*/status; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | more ceph-osd 1553520 kB ceph-osd 1447728 kB ceph-osd 1218768 kB ceph-osd 1117536 kB ceph-osd 1026548 kB ceph-osd 641632 kB ceph-osd 495080 kB ceph-osd 424392 kB firewalld 26880 kB dockerd 20352 kB containerd 11136 kB docker 6144 kB docker 6144 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5760 kB (sd-pam) 5184 kB ceph-crash 4416 kB python3 4224 kB docker 4032 kB systemd-udevd 3264 kB On 22.05.2025 18:34, Anthony D'Atri wrote:
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%.
Why do you have swap configured at all? I suggest disabling swap in fstab and rebooting serially.
Swap can possibly reduce your clusters performance, not? Osd-processes that swap data will result in supplementary and unwanted disk i/o I've got 10 OSD's per host and my memory consumption of ceph is typically 70GiB per host... Each host has about 40GiB available memory which is sufficient (for my setup) except one time I ran out of memory deleting old snapshots. But 8GiB wouldn't have helped...
-----Oorspronkelijk bericht----- Van: Dmitrijs Demidovs <dmitrijs.demidovs@carminered.eu> Verzonden: vrijdag 23 mei 2025 10:16 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] Re: SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
Hi Anthony.
Yes we have swap enabled. Old Rocky 8 and new Rocky 9 OSD hosts both configured with 8G of swap.
I will try to disable swap, but I guess what we will get a lot of Out Of Memory messages on OSD hosts.
= old: [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi
= new: [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB):
[root@ceph-osd17 ~]# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 5cc58e4a77b2 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 0.28% 3.576GiB / 62.28GiB 5.74% 0B / 0B 3.9TB / 975GB 62 3a60fecf648d ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 0.28% 2.912GiB / 62.28GiB 4.68% 0B / 0B 100TB / 45.7TB 62 9c20407e79eb ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 0.28% 2.905GiB / 62.28GiB 4.66% 0B / 0B 93TB / 35.8TB 62 9deadafef9dd ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 0.56% 3.624GiB / 62.28GiB 5.82% 0B / 0B 102TB / 39.2TB 62 fcfe62a25fd9 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 0.40% 2.968GiB / 62.28GiB 4.77% 0B / 0B 83.2TB / 34.8TB 62 38d2d96cc491 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 1.42% 2.666GiB / 62.28GiB 4.28% 0B / 0B 105TB / 38.1TB 62 e29c6bbc1ae7 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 2.01% 3.687GiB / 62.28GiB 5.92% 0B / 0B 106TB / 44.6TB 62 40346a7a45ea ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 0.69% 2.748GiB / 62.28GiB 4.41% 0B / 0B 103TB / 41.4TB 62 43c3e3a65531 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 0.00% 3.73MiB / 62.28GiB 0.01% 0B / 0B 567MB / 18MB 2 d9e436f9788c ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 15.04% 30.25MiB / 62.28GiB 0.05% 0B / 0B 410MB / 14.6MB 61
But they also are biggest swap consumers:
[root@ceph-osd17 ~]# for file in /proc/*/status; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | more ceph-osd 1553520 kB ceph-osd 1447728 kB ceph-osd 1218768 kB ceph-osd 1117536 kB ceph-osd 1026548 kB ceph-osd 641632 kB ceph-osd 495080 kB ceph-osd 424392 kB firewalld 26880 kB dockerd 20352 kB containerd 11136 kB docker 6144 kB docker 6144 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5760 kB (sd-pam) 5184 kB ceph-crash 4416 kB python3 4224 kB docker 4032 kB systemd-udevd 3264 kB
On 22.05.2025 18:34, Anthony D'Atri wrote:
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our
OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%.
Why do you have swap configured at all? I suggest disabling swap in fstab
and rebooting serially.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Swap can possibly reduce your clusters performance, not? Osd-processes that swap data will result in supplementary and unwanted disk I/o
Absolutely. Moreover I’ve seen modern(ish) Linux systems anomalously using swap space when there is available physmem, in excess of vm.min_free_kybytes.
I've got 10 OSD's per host and my memory consumption of ceph is typically 70GiB per host... Each host has about 40GiB available memory which is sufficient (for my setup) except one time I ran out of memory deleting old snapshots. But 8GiB wouldn't have helped...
Exactly. Filesystem swap can be a useful emergency tool, but in 2025 it should not be routine. In 1985 a diskless Sun 2/50 with 3MB of physmem (yes, MB) needed swap, and that was SUPER fun over 10GE ethernet against a Fuji Eagle. In years beginning with a 2, DRAM prices are such that if disabling swap causes a problem, then that’s a sign that you really, really need more physmem. Swap is 12% of your virtual memory right now. If you run hotter than 84% usage then really you need more. By default the osd_memory_target autotune should be enabled, see what values it is setting. By default it will divide 70% of physmem by however many OSDs are placed on a host: # ceph config dump | grep osd_memory_target osd host:x basic osd_memory_target 12060218196 osd host:y basic osd_memory_target 12062644163 osd host:z basic osd_memory_target 6614520783 Yes, the above cluster has lots of physmem, which is very very fortunate because it’s based on large HDDs and otherwise would have fallen over (if I told you the details, you wouldn’t sleep at night). The money would have been better spent on QLC, but I digress. If autotune isn’t on, the default osd_memory_target is 4GB. Remember that it’s a target not a limit. The docs advise a 20% headroom of available physmem, having suffered a few things I like to advise 50% at least, plus margin to run mons/mgrs/mds/etc.
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB):
Actually that’s another sign that you may be starved unless these OSDs are rather idle. Are you using cephadm, or something else to manage the containers? Is it enforcing an artificial limit on them? With 64GB of physmem cephadm’s autotuning would assign an osd_memory_target of 4.5GB. Memory allocation practice and accounting vary across kernel revisions, which may be a factor here. What model of chassis are these? Adding even 4x8GB super cheap DIMMs to each would do you a world of good, with more of course even better. Be sure to not mix SKUs within a bank, and populate slots according to your motherboard’s documentation.
-----Oorspronkelijk bericht----- Van: Dmitrijs Demidovs <dmitrijs.demidovs@carminered.eu> Verzonden: vrijdag 23 mei 2025 10:16 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] Re: SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
Hi Anthony.
Yes we have swap enabled. Old Rocky 8 and new Rocky 9 OSD hosts both configured with 8G of swap.
I will try to disable swap, but I guess what we will get a lot of Out Of Memory messages on OSD hosts.
= old: [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi
= new: [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB):
[root@ceph-osd17 ~]# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 5cc58e4a77b2 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 0.28% 3.576GiB / 62.28GiB 5.74% 0B / 0B 3.9TB / 975GB 62 3a60fecf648d ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 0.28% 2.912GiB / 62.28GiB 4.68% 0B / 0B 100TB / 45.7TB 62 9c20407e79eb ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 0.28% 2.905GiB / 62.28GiB 4.66% 0B / 0B 93TB / 35.8TB 62 9deadafef9dd ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 0.56% 3.624GiB / 62.28GiB 5.82% 0B / 0B 102TB / 39.2TB 62 fcfe62a25fd9 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 0.40% 2.968GiB / 62.28GiB 4.77% 0B / 0B 83.2TB / 34.8TB 62 38d2d96cc491 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 1.42% 2.666GiB / 62.28GiB 4.28% 0B / 0B 105TB / 38.1TB 62 e29c6bbc1ae7 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 2.01% 3.687GiB / 62.28GiB 5.92% 0B / 0B 106TB / 44.6TB 62 40346a7a45ea ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 0.69% 2.748GiB / 62.28GiB 4.41% 0B / 0B 103TB / 41.4TB 62 43c3e3a65531 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 0.00% 3.73MiB / 62.28GiB 0.01% 0B / 0B 567MB / 18MB 2 d9e436f9788c ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 15.04% 30.25MiB / 62.28GiB 0.05% 0B / 0B 410MB / 14.6MB 61
But they also are biggest swap consumers:
[root@ceph-osd17 ~]# for file in /proc/*/status; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | more ceph-osd 1553520 kB ceph-osd 1447728 kB ceph-osd 1218768 kB ceph-osd 1117536 kB ceph-osd 1026548 kB ceph-osd 641632 kB ceph-osd 495080 kB ceph-osd 424392 kB firewalld 26880 kB dockerd 20352 kB containerd 11136 kB docker 6144 kB docker 6144 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5760 kB (sd-pam) 5184 kB ceph-crash 4416 kB python3 4224 kB docker 4032 kB systemd-udevd 3264 kB
On 22.05.2025 18:34, Anthony D'Atri wrote:
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our
OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%.
Why do you have swap configured at all? I suggest disabling swap in fstab
and rebooting serially.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, A few months ago after seeing a some OSDs being OOM-killed, we were pointed to 2 references [1][2] dating from the last Nautilus version mentionning the best config for an OSD in modern Ceph versions was with bluefs_buffered_io=true (default since this last Nautilus version) and system swap disabled. We did it on all our OSD servers and we have seen a huge diminution of the allocated memory per OSD (factor ~2) Best regards, Michel [1]https://docs.ceph.com/en/latest/releases/nautilus/#notable-changes [2]https://github.com/ceph/ceph/pull/38044 Le 23/05/2025 à 14:52, Anthony D'Atri a écrit :
Swap can possibly reduce your clusters performance, not? Osd-processes that swap data will result in supplementary and unwanted disk I/o Absolutely. Moreover I’ve seen modern(ish) Linux systems anomalously using swap space when there is available physmem, in excess of vm.min_free_kybytes.
I've got 10 OSD's per host and my memory consumption of ceph is typically 70GiB per host... Each host has about 40GiB available memory which is sufficient (for my setup) except one time I ran out of memory deleting old snapshots. But 8GiB wouldn't have helped... Exactly. Filesystem swap can be a useful emergency tool, but in 2025 it should not be routine. In 1985 a diskless Sun 2/50 with 3MB of physmem (yes, MB) needed swap, and that was SUPER fun over 10GE ethernet against a Fuji Eagle.
In years beginning with a 2, DRAM prices are such that if disabling swap causes a problem, then that’s a sign that you really, really need more physmem.
Swap is 12% of your virtual memory right now. If you run hotter than 84% usage then really you need more.
By default the osd_memory_target autotune should be enabled, see what values it is setting. By default it will divide 70% of physmem by however many OSDs are placed on a host:
# ceph config dump | grep osd_memory_target osd host:x basic osd_memory_target 12060218196 osd host:y basic osd_memory_target 12062644163 osd host:z basic osd_memory_target 6614520783
Yes, the above cluster has lots of physmem, which is very very fortunate because it’s based on large HDDs and otherwise would have fallen over (if I told you the details, you wouldn’t sleep at night). The money would have been better spent on QLC, but I digress.
If autotune isn’t on, the default osd_memory_target is 4GB. Remember that it’s a target not a limit. The docs advise a 20% headroom of available physmem, having suffered a few things I like to advise 50% at least, plus margin to run mons/mgrs/mds/etc.
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB): Actually that’s another sign that you may be starved unless these OSDs are rather idle. Are you using cephadm, or something else to manage the containers? Is it enforcing an artificial limit on them?
With 64GB of physmem cephadm’s autotuning would assign an osd_memory_target of 4.5GB.
Memory allocation practice and accounting vary across kernel revisions, which may be a factor here.
What model of chassis are these? Adding even 4x8GB super cheap DIMMs to each would do you a world of good, with more of course even better. Be sure to not mix SKUs within a bank, and populate slots according to your motherboard’s documentation.
-----Oorspronkelijk bericht----- Van: Dmitrijs Demidovs <dmitrijs.demidovs@carminered.eu> Verzonden: vrijdag 23 mei 2025 10:16 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] Re: SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
Hi Anthony.
Yes we have swap enabled. Old Rocky 8 and new Rocky 9 OSD hosts both configured with 8G of swap.
I will try to disable swap, but I guess what we will get a lot of Out Of Memory messages on OSD hosts.
= old: [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi
= new: [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB):
[root@ceph-osd17 ~]# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 5cc58e4a77b2 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 0.28% 3.576GiB / 62.28GiB 5.74% 0B / 0B 3.9TB / 975GB 62 3a60fecf648d ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 0.28% 2.912GiB / 62.28GiB 4.68% 0B / 0B 100TB / 45.7TB 62 9c20407e79eb ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 0.28% 2.905GiB / 62.28GiB 4.66% 0B / 0B 93TB / 35.8TB 62 9deadafef9dd ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 0.56% 3.624GiB / 62.28GiB 5.82% 0B / 0B 102TB / 39.2TB 62 fcfe62a25fd9 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 0.40% 2.968GiB / 62.28GiB 4.77% 0B / 0B 83.2TB / 34.8TB 62 38d2d96cc491 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 1.42% 2.666GiB / 62.28GiB 4.28% 0B / 0B 105TB / 38.1TB 62 e29c6bbc1ae7 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 2.01% 3.687GiB / 62.28GiB 5.92% 0B / 0B 106TB / 44.6TB 62 40346a7a45ea ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 0.69% 2.748GiB / 62.28GiB 4.41% 0B / 0B 103TB / 41.4TB 62 43c3e3a65531 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 0.00% 3.73MiB / 62.28GiB 0.01% 0B / 0B 567MB / 18MB 2 d9e436f9788c ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 15.04% 30.25MiB / 62.28GiB 0.05% 0B / 0B 410MB / 14.6MB 61
But they also are biggest swap consumers:
[root@ceph-osd17 ~]# for file in /proc/*/status; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | more ceph-osd 1553520 kB ceph-osd 1447728 kB ceph-osd 1218768 kB ceph-osd 1117536 kB ceph-osd 1026548 kB ceph-osd 641632 kB ceph-osd 495080 kB ceph-osd 424392 kB firewalld 26880 kB dockerd 20352 kB containerd 11136 kB docker 6144 kB docker 6144 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5760 kB (sd-pam) 5184 kB ceph-crash 4416 kB python3 4224 kB docker 4032 kB systemd-udevd 3264 kB
On 22.05.2025 18:34, Anthony D'Atri wrote:
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%. Why do you have swap configured at all? I suggest disabling swap in fstab and rebooting serially.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Swap usage is not per se wrong - as long as you have enough available memory. However, it can still lead to performance issues. If you don't want to get rid of swap, check your swapiness settings (cat /proc/sys/vm/swappiness). Op vr 23 mei 2025 om 14:53 schreef Anthony D'Atri <aad@dreamsnake.net>:
Swap can possibly reduce your clusters performance, not? Osd-processes that swap data will result in supplementary and unwanted disk I/o
Absolutely. Moreover I’ve seen modern(ish) Linux systems anomalously using swap space when there is available physmem, in excess of vm.min_free_kybytes.
I've got 10 OSD's per host and my memory consumption of ceph is
typically 70GiB per host... Each host has about 40GiB available memory which is sufficient (for my setup) except one time I ran out of memory deleting old snapshots. But 8GiB wouldn't have helped...
Exactly. Filesystem swap can be a useful emergency tool, but in 2025 it should not be routine. In 1985 a diskless Sun 2/50 with 3MB of physmem (yes, MB) needed swap, and that was SUPER fun over 10GE ethernet against a Fuji Eagle.
In years beginning with a 2, DRAM prices are such that if disabling swap causes a problem, then that’s a sign that you really, really need more physmem.
Swap is 12% of your virtual memory right now. If you run hotter than 84% usage then really you need more.
By default the osd_memory_target autotune should be enabled, see what values it is setting. By default it will divide 70% of physmem by however many OSDs are placed on a host:
# ceph config dump | grep osd_memory_target osd host:x basic osd_memory_target 12060218196 osd host:y basic osd_memory_target 12062644163 osd host:z basic osd_memory_target 6614520783
Yes, the above cluster has lots of physmem, which is very very fortunate because it’s based on large HDDs and otherwise would have fallen over (if I told you the details, you wouldn’t sleep at night). The money would have been better spent on QLC, but I digress.
If autotune isn’t on, the default osd_memory_target is 4GB. Remember that it’s a target not a limit. The docs advise a 20% headroom of available physmem, having suffered a few things I like to advise 50% at least, plus margin to run mons/mgrs/mds/etc.
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB):
Actually that’s another sign that you may be starved unless these OSDs are rather idle. Are you using cephadm, or something else to manage the containers? Is it enforcing an artificial limit on them?
With 64GB of physmem cephadm’s autotuning would assign an osd_memory_target of 4.5GB.
Memory allocation practice and accounting vary across kernel revisions, which may be a factor here.
What model of chassis are these? Adding even 4x8GB super cheap DIMMs to each would do you a world of good, with more of course even better. Be sure to not mix SKUs within a bank, and populate slots according to your motherboard’s documentation.
-----Oorspronkelijk bericht----- Van: Dmitrijs Demidovs <dmitrijs.demidovs@carminered.eu> Verzonden: vrijdag 23 mei 2025 10:16 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] Re: SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
Hi Anthony.
Yes we have swap enabled. Old Rocky 8 and new Rocky 9 OSD hosts both configured with 8G of swap.
I will try to disable swap, but I guess what we will get a lot of Out
Of Memory
messages on OSD hosts.
= old: [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi
= new: [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB):
[root@ceph-osd17 ~]# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 5cc58e4a77b2 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 0.28% 3.576GiB / 62.28GiB 5.74% 0B / 0B 3.9TB / 975GB 62 3a60fecf648d ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 0.28% 2.912GiB / 62.28GiB 4.68% 0B / 0B 100TB / 45.7TB 62 9c20407e79eb ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 0.28% 2.905GiB / 62.28GiB 4.66% 0B / 0B 93TB / 35.8TB 62 9deadafef9dd ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 0.56% 3.624GiB / 62.28GiB 5.82% 0B / 0B 102TB / 39.2TB 62 fcfe62a25fd9 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 0.40% 2.968GiB / 62.28GiB 4.77% 0B / 0B 83.2TB / 34.8TB 62 38d2d96cc491 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 1.42% 2.666GiB / 62.28GiB 4.28% 0B / 0B 105TB / 38.1TB 62 e29c6bbc1ae7 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 2.01% 3.687GiB / 62.28GiB 5.92% 0B / 0B 106TB / 44.6TB 62 40346a7a45ea ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 0.69% 2.748GiB / 62.28GiB 4.41% 0B / 0B 103TB / 41.4TB 62 43c3e3a65531 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 0.00% 3.73MiB / 62.28GiB 0.01% 0B / 0B 567MB / 18MB 2 d9e436f9788c ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 15.04% 30.25MiB / 62.28GiB 0.05% 0B / 0B 410MB / 14.6MB 61
But they also are biggest swap consumers:
[root@ceph-osd17 ~]# for file in /proc/*/status; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | more ceph-osd 1553520 kB ceph-osd 1447728 kB ceph-osd 1218768 kB ceph-osd 1117536 kB ceph-osd 1026548 kB ceph-osd 641632 kB ceph-osd 495080 kB ceph-osd 424392 kB firewalld 26880 kB dockerd 20352 kB containerd 11136 kB docker 6144 kB docker 6144 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5760 kB (sd-pam) 5184 kB ceph-crash 4416 kB python3 4224 kB docker 4032 kB systemd-udevd 3264 kB
On 22.05.2025 18:34, Anthony D'Atri wrote:
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our
OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%.
Why do you have swap configured at all? I suggest disabling swap in
fstab and rebooting serially.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Anthony D'Atri
-
Anthony D'Atri
-
Dmitrijs Demidovs
-
Dominique Ramaekers
-
Michel Jouvin
-
Sinan Polat