Hi Anthony. Yes we have swap enabled. Old Rocky 8 and new Rocky 9 OSD hosts both configured with 8G of swap. I will try to disable swap, but I guess what we will get a lot of Out Of Memory messages on OSD hosts. = old: [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi = new: [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi OSD containers consumes reasonable amount of RAM (~2.6GB - ~3.6 GB): [root@ceph-osd17 ~]# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 5cc58e4a77b2 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 0.28% 3.576GiB / 62.28GiB 5.74% 0B / 0B 3.9TB / 975GB 62 3a60fecf648d ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 0.28% 2.912GiB / 62.28GiB 4.68% 0B / 0B 100TB / 45.7TB 62 9c20407e79eb ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 0.28% 2.905GiB / 62.28GiB 4.66% 0B / 0B 93TB / 35.8TB 62 9deadafef9dd ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 0.56% 3.624GiB / 62.28GiB 5.82% 0B / 0B 102TB / 39.2TB 62 fcfe62a25fd9 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 0.40% 2.968GiB / 62.28GiB 4.77% 0B / 0B 83.2TB / 34.8TB 62 38d2d96cc491 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 1.42% 2.666GiB / 62.28GiB 4.28% 0B / 0B 105TB / 38.1TB 62 e29c6bbc1ae7 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 2.01% 3.687GiB / 62.28GiB 5.92% 0B / 0B 106TB / 44.6TB 62 40346a7a45ea ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 0.69% 2.748GiB / 62.28GiB 4.41% 0B / 0B 103TB / 41.4TB 62 43c3e3a65531 ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 0.00% 3.73MiB / 62.28GiB 0.01% 0B / 0B 567MB / 18MB 2 d9e436f9788c ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 15.04% 30.25MiB / 62.28GiB 0.05% 0B / 0B 410MB / 14.6MB 61 But they also are biggest swap consumers: [root@ceph-osd17 ~]# for file in /proc/*/status; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | more ceph-osd 1553520 kB ceph-osd 1447728 kB ceph-osd 1218768 kB ceph-osd 1117536 kB ceph-osd 1026548 kB ceph-osd 641632 kB ceph-osd 495080 kB ceph-osd 424392 kB firewalld 26880 kB dockerd 20352 kB containerd 11136 kB docker 6144 kB docker 6144 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5952 kB docker 5760 kB (sd-pam) 5184 kB ceph-crash 4416 kB python3 4224 kB docker 4032 kB systemd-udevd 3264 kB On 22.05.2025 18:34, Anthony D'Atri wrote:
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%.
Why do you have swap configured at all? I suggest disabling swap in fstab and rebooting serially.