High swap usage on one replication node
Hi, our Ceph 14.2.3 cluster so far runs smooth with replicated and EC pools, but since a couple of days one of the dedicated replication nodes consumes up to 99% swap and stays at that level. The other two replicated nodes use +- 50 - 60% of swap. All the 24 NVMe OSDs per node are BlueStore with default settings, 128GB RAM. The vm.swappiness is set to 10. Do you have any suggestions how to handle/reduce the swap usage? Thanks for feedback and regards . Götz
Hi there, I think we have our OSD nodes setup with vm.swappiness = 0 If I remember correctly few years ago vm.swappiness = 0 was changed and now it does not prevent swapping it just reduces the changes of memory being send to swap. Cheers, Xavier. -----Mensaje original----- De: Götz Reinicke <goetz.reinicke@filmakademie.de> Enviado el: viernes, 6 de diciembre de 2019 8:14 Para: ceph-users <ceph-users@ceph.com> Asunto: [ceph-users] High swap usage on one replication node Hi, our Ceph 14.2.3 cluster so far runs smooth with replicated and EC pools, but since a couple of days one of the dedicated replication nodes consumes up to 99% swap and stays at that level. The other two replicated nodes use +- 50 - 60% of swap. All the 24 NVMe OSDs per node are BlueStore with default settings, 128GB RAM. The vm.swappiness is set to 10. Do you have any suggestions how to handle/reduce the swap usage? Thanks for feedback and regards . Götz
Hello, I would suggest to: ~# swapoff -a ~# vi /etc/fstab ... remove swap line ... and buy additional ram if required. Without knowing your exact use case, 128GB would be our minimum ram for simple use cases and most likely not for EC and complex configurations. Swap is nothing you want to have in a Server as it is very slow and can cause long downtimes. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 7. Dez. 2019 um 12:34 Uhr schrieb Xavier Trilla < xavier.trilla@clouding.io>:
Hi there,
I think we have our OSD nodes setup with vm.swappiness = 0
If I remember correctly few years ago vm.swappiness = 0 was changed and now it does not prevent swapping it just reduces the changes of memory being send to swap.
Cheers, Xavier. -----Mensaje original----- De: Götz Reinicke <goetz.reinicke@filmakademie.de> Enviado el: viernes, 6 de diciembre de 2019 8:14 Para: ceph-users <ceph-users@ceph.com> Asunto: [ceph-users] High swap usage on one replication node
Hi,
our Ceph 14.2.3 cluster so far runs smooth with replicated and EC pools, but since a couple of days one of the dedicated replication nodes consumes up to 99% swap and stays at that level. The other two replicated nodes use +- 50 - 60% of swap.
All the 24 NVMe OSDs per node are BlueStore with default settings, 128GB RAM. The vm.swappiness is set to 10.
Do you have any suggestions how to handle/reduce the swap usage?
Thanks for feedback and regards . Götz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Sun, 8 Dec 2019 at 00:53, Martin Verges <martin.verges@croit.io> wrote:
Swap is nothing you want to have in a Server as it is very slow and can cause long downtimes.
Given the commentary on this page advocating at least some swap to enable Linux to manage memory when under pressure: https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConsequence is it worth modifying the advice to at least have some swap available (even if only say 5% of overall memory)? There was a hnews thread here but to me it seemed inconclusive about solving the overall problem (other than applications taking more responsibility for memory consumption): https://news.ycombinator.com/item?id=20641551
Without reading the links: from more then 20y Linux server and datacenter hosting environment, around 7y of Ceph, and hundreds of different systems all configured without swap. I never ever had a problem with noswap that would be solved using swap. But on my Linux desktop swap helps me a bit. btw, all multiple hundred croit based ceph deployments are 100% swap free. As we boot over the network directly into the ram, there is no swap disk that would be available. We don't have any issues with this and I doubt we will ever encounter such. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Nigel Williams <nigel.williams@tpac.org.au> schrieb am So., 8. Dez. 2019, 23:15:
On Sun, 8 Dec 2019 at 00:53, Martin Verges <martin.verges@croit.io> wrote:
Swap is nothing you want to have in a Server as it is very slow and can cause long downtimes.
Given the commentary on this page advocating at least some swap to enable Linux to manage memory when under pressure:
https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConsequence
is it worth modifying the advice to at least have some swap available (even if only say 5% of overall memory)?
There was a hnews thread here but to me it seemed inconclusive about solving the overall problem (other than applications taking more responsibility for memory consumption):
https://news.ycombinator.com/item?id=20641551 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I’ve had one or two situations where swap might have helped a memory consumption problem, but others in which it would have *worsened* cluster performance. Sometimes it’s better for the *cluster* for an OSD to die / restart / get OOMkilled than for it to limp along sluggishly. In the past RAM capacity and cost were such that one had to compromise on the amount provisioned, today with a Ceph cluster if one can’t get CapEx to properly outfit OSD nodes, one has bigger problems than just RAM. Eg. maybe one filled all DIMM slots with small-capacity modules and retrofitting would mean throwing a bunch of them away. Anyone remember how the Sun 4/110 was shipped back in the day if ordered with minimum RAM? Or the similar crap that NCD pulled? So basically I agree that there’s little or no benefit to it. Especially if only 5% of physmem as suggested below — a tiny amount of swap has a tiny benefit at best. In other words, if you really do need swap, arguably what you REALLY need is more RAM. I suggest that the OP has swap configured because NVMe is mentioned, perhaps there’s the idea that swap on NVMe is fast enough to be usable without horquing user experience. But that was only slightly implied, in the OP’s deployment it’s still possible that the swap is on a SATA/SAS/etc. boot drive, not NVMe. Swap on any kind of SSD is going to add unnecessary wear, esp. if you provision low-durability drives.
Without reading the links:
from more then 20y Linux server and datacenter hosting environment, around 7y of Ceph, and hundreds of different systems all configured without swap. I never ever had a problem with noswap that would be solved using swap.
Swap is nothing you want to have in a Server as it is very slow and can cause long downtimes.
Given the commentary on this page advocating at least some swap to enable Linux to manage memory when under pressure:
https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConsequence
is it worth modifying the advice to at least have some swap available (even if only say 5% of overall memory)?
participants (5)
-
Anthony D'Atri
-
Götz Reinicke
-
Martin Verges
-
Nigel Williams
-
Xavier Trilla