Swap usage is not per se wrong - as long as you have enough available memory. However, it can still lead to performance issues. If you don't want to get rid of swap, check your swapiness settings (cat /proc/sys/vm/swappiness). Op vr 23 mei 2025 om 14:53 schreef Anthony D'Atri <aad@dreamsnake.net>:
Swap can possibly reduce your clusters performance, not? Osd-processes that swap data will result in supplementary and unwanted disk I/o
Absolutely. Moreover I’ve seen modern(ish) Linux systems anomalously using swap space when there is available physmem, in excess of vm.min_free_kybytes.
I've got 10 OSD's per host and my memory consumption of ceph is
typically 70GiB per host... Each host has about 40GiB available memory which is sufficient (for my setup) except one time I ran out of memory deleting old snapshots. But 8GiB wouldn't have helped...
Exactly. Filesystem swap can be a useful emergency tool, but in 2025 it should not be routine. In 1985 a diskless Sun 2/50 with 3MB of physmem (yes, MB) needed swap, and that was SUPER fun over 10GE ethernet against a Fuji Eagle.
In years beginning with a 2, DRAM prices are such that if disabling swap causes a problem, then that’s a sign that you really, really need more physmem.
Swap is 12% of your virtual memory right now. If you run hotter than 84% usage then really you need more.
By default the osd_memory_target autotune should be enabled, see what values it is setting. By default it will divide 70% of physmem by however many OSDs are placed on a host:
# ceph config dump | grep osd_memory_target osd host:x basic osd_memory_target 12060218196 osd host:y basic osd_memory_target 12062644163 osd host:z basic osd_memory_target 6614520783
Yes, the above cluster has lots of physmem, which is very very fortunate because it’s based on large HDDs and otherwise would have fallen over (if I told you the details, you wouldn’t sleep at night). The money would have been better spent on QLC, but I digress.
If autotune isn’t on, the default osd_memory_target is 4GB. Remember that it’s a target not a limit. The docs advise a 20% headroom of available physmem, having suffered a few things I like to advise 50% at least, plus margin to run mons/mgrs/mds/etc.
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB):
Actually that’s another sign that you may be starved unless these OSDs are rather idle. Are you using cephadm, or something else to manage the containers? Is it enforcing an artificial limit on them?
With 64GB of physmem cephadm’s autotuning would assign an osd_memory_target of 4.5GB.
Memory allocation practice and accounting vary across kernel revisions, which may be a factor here.
What model of chassis are these? Adding even 4x8GB super cheap DIMMs to each would do you a world of good, with more of course even better. Be sure to not mix SKUs within a bank, and populate slots according to your motherboard’s documentation.
-----Oorspronkelijk bericht----- Van: Dmitrijs Demidovs <dmitrijs.demidovs@carminered.eu> Verzonden: vrijdag 23 mei 2025 10:16 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] Re: SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
Hi Anthony.
Yes we have swap enabled. Old Rocky 8 and new Rocky 9 OSD hosts both configured with 8G of swap.
I will try to disable swap, but I guess what we will get a lot of Out
Of Memory
messages on OSD hosts.
= old: [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi
= new: [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi
OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB):
[root@ceph-osd17 ~]# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 5cc58e4a77b2 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 0.28% 3.576GiB / 62.28GiB 5.74% 0B / 0B 3.9TB / 975GB 62 3a60fecf648d ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 0.28% 2.912GiB / 62.28GiB 4.68% 0B / 0B 100TB / 45.7TB 62 9c20407e79eb ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 0.28% 2.905GiB / 62.28GiB 4.66% 0B / 0B 93TB / 35.8TB 62 9deadafef9dd ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 0.56% 3.624GiB / 62.28GiB 5.82% 0B / 0B 102TB / 39.2TB 62 fcfe62a25fd9 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 0.40% 2.968GiB / 62.28GiB 4.77% 0B / 0B 83.2TB / 34.8TB 62 38d2d96cc491 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 1.42% 2.666GiB / 62.28GiB 4.28% 0B / 0B 105TB / 38.1TB 62 e29c6bbc1ae7 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 2.01% 3.687GiB / 62.28GiB 5.92% 0B / 0B 106TB / 44.6TB 62 40346a7a45ea ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 0.69% 2.748GiB / 62.28GiB 4.41% 0B / 0B 103TB / 41.4TB 62 43c3e3a65531 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 0.00% 3.73MiB / 62.28GiB 0.01% 0B / 0B 567MB / 18MB 2 d9e436f9788c ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 15.04% 30.25MiB / 62.28GiB 0.05% 0B / 0B 410MB / 14.6MB 61
But they also are biggest swap consumers:
[root@ceph-osd17 ~]# for file in /proc/*/status; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | more ceph-osd 1553520 kB ceph-osd 1447728 kB ceph-osd 1218768 kB ceph-osd 1117536 kB ceph-osd 1026548 kB ceph-osd 641632 kB ceph-osd 495080 kB ceph-osd 424392 kB firewalld 26880 kB dockerd 20352 kB containerd 11136 kB docker 6144 kB docker 6144 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5760 kB (sd-pam) 5184 kB ceph-crash 4416 kB python3 4224 kB docker 4032 kB systemd-udevd 3264 kB
On 22.05.2025 18:34, Anthony D'Atri wrote:
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our
OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%.
Why do you have swap configured at all? I suggest disabling swap in
fstab and rebooting serially.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io