Hi, Housekeeping... I was cleaning up my snapshots and was flattening clones... Suddenly I ran out of memory on my nodes! 4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node has about 45 GiB of free (available) memory in normal operation. After flattening several images and removing dozens of snapshots, the free memory probably was already lower than the usual 45GiB. I was flattening an image of 75GiB and 2 out of 4 nodes ran out of memory. One node even automatically killed processes randomly to free up memory... After regaining control over the system, I put that node in maintenance and rebooted this node. After reboot the free mem was like 70Gib. Over night all of the nodes were back on the usual 45GiB. Today I checked the free mem of each node: 45GiB free. So I flattened again an image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast! Is there a way to avoid this behavior of the cluster? Greetings, Dominique.
Hi Dominique, Is swap enabled on OSD nodes? My experience is that when swap is enabled on OSD nodes, OSDs can exceed osd_memory_target in a completely unreasonable manner compared to when swap is disabled. Using swap on OSD nodes has been discouraged in the past [1] but I think it's not clearly documented enough. As to understand why these OSDs use so much RAM, read Mark's message from April 11th on this list. It will help you diagnose ram usage. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/releases/nautilus/#notable-changes ----- Le 25 Avr 25, à 16:36, Dominique Ramaekers dominique.ramaekers@cometal.be a écrit :
Hi,
Housekeeping... I was cleaning up my snapshots and was flattening clones... Suddenly I ran out of memory on my nodes!
4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node has about 45 GiB of free (available) memory in normal operation.
After flattening several images and removing dozens of snapshots, the free memory probably was already lower than the usual 45GiB. I was flattening an image of 75GiB and 2 out of 4 nodes ran out of memory. One node even automatically killed processes randomly to free up memory... After regaining control over the system, I put that node in maintenance and rebooted this node. After reboot the free mem was like 70Gib. Over night all of the nodes were back on the usual 45GiB.
Today I checked the free mem of each node: 45GiB free. So I flattened again an image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast!
Is there a way to avoid this behavior of the cluster?
Greetings,
Dominique. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Swap isn't enabled. I can't reproduce it anymore. This behavior only showed with such extreme on the images of one virtual machine. I don't know where the history of these images was different from the other images that can provoke this behavior... Also @Eugen and @Anthony... Thanks for the input. I'll be more attentive next time I'm cleaning up my images... Maybe next time I can catch some more relevant information.
-----Oorspronkelijk bericht----- Van: Frédéric Nass <frederic.nass@univ-lorraine.fr> Verzonden: vrijdag 25 april 2025 17:54 Aan: Dominique Ramaekers <dominique.ramaekers@cometal.be> CC: ceph-users <ceph-users@ceph.io> Onderwerp: Re: [ceph-users] Memory usage
Hi Dominique,
Is swap enabled on OSD nodes?
My experience is that when swap is enabled on OSD nodes, OSDs can exceed osd_memory_target in a completely unreasonable manner compared to when swap is disabled. Using swap on OSD nodes has been discouraged in the past [1] but I think it's not clearly documented enough.
As to understand why these OSDs use so much RAM, read Mark's message from April 11th on this list. It will help you diagnose ram usage.
Regards, Frédéric.
[1] https://docs.ceph.com/en/latest/releases/nautilus/#notable-changes
----- Le 25 Avr 25, à 16:36, Dominique Ramaekers dominique.ramaekers@cometal.be a écrit :
Hi,
Housekeeping... I was cleaning up my snapshots and was flattening clones... Suddenly I ran out of memory on my nodes!
4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node has about 45 GiB of free (available) memory in normal operation.
After flattening several images and removing dozens of snapshots, the free memory probably was already lower than the usual 45GiB. I was flattening an image of 75GiB and 2 out of 4 nodes ran out of memory. One node even automatically killed processes randomly to free up memory... After regaining control over the system, I put that node in maintenance and rebooted this node. After reboot the free mem was like 70Gib. Over night all of the nodes were back on the usual 45GiB.
Today I checked the free mem of each node: 45GiB free. So I flattened again an image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast!
Is there a way to avoid this behavior of the cluster?
Greetings,
Dominique. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Something else to experiment with is running `fstrim` every weekend on RBD clients for a more-incremental freeing of deleted data.
On Apr 28, 2025, at 2:51 AM, Dominique Ramaekers <dominique.ramaekers@cometal.be> wrote:
Swap isn't enabled.
I can't reproduce it anymore. This behavior only showed with such extreme on the images of one virtual machine. I don't know where the history of these images was different from the other images that can provoke this behavior...
Also @Eugen and @Anthony... Thanks for the input. I'll be more attentive next time I'm cleaning up my images... Maybe next time I can catch some more relevant information.
-----Oorspronkelijk bericht----- Van: Frédéric Nass <frederic.nass@univ-lorraine.fr> Verzonden: vrijdag 25 april 2025 17:54 Aan: Dominique Ramaekers <dominique.ramaekers@cometal.be> CC: ceph-users <ceph-users@ceph.io> Onderwerp: Re: [ceph-users] Memory usage
Hi Dominique,
Is swap enabled on OSD nodes?
My experience is that when swap is enabled on OSD nodes, OSDs can exceed osd_memory_target in a completely unreasonable manner compared to when swap is disabled. Using swap on OSD nodes has been discouraged in the past [1] but I think it's not clearly documented enough.
As to understand why these OSDs use so much RAM, read Mark's message from April 11th on this list. It will help you diagnose ram usage.
Regards, Frédéric.
[1] https://docs.ceph.com/en/latest/releases/nautilus/#notable-changes
----- Le 25 Avr 25, à 16:36, Dominique Ramaekers dominique.ramaekers@cometal.be a écrit :
Hi,
Housekeeping... I was cleaning up my snapshots and was flattening clones... Suddenly I ran out of memory on my nodes!
4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node has about 45 GiB of free (available) memory in normal operation.
After flattening several images and removing dozens of snapshots, the free memory probably was already lower than the usual 45GiB. I was flattening an image of 75GiB and 2 out of 4 nodes ran out of memory. One node even automatically killed processes randomly to free up memory... After regaining control over the system, I put that node in maintenance and rebooted this node. After reboot the free mem was like 70Gib. Over night all of the nodes were back on the usual 45GiB.
Today I checked the free mem of each node: 45GiB free. So I flattened again an image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast!
Is there a way to avoid this behavior of the cluster?
Greetings,
Dominique. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Interesting, I did something similar just a few weeks back, flattening some images (didn’t look at their sizes or overlaps of the snapshots). But I don’t see any spikes in memory usage at all. At that time the cluster was running latest Pacific. Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
Hi,
Housekeeping... I was cleaning up my snapshots and was flattening clones... Suddenly I ran out of memory on my nodes!
4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node has about 45 GiB of free (available) memory in normal operation.
After flattening several images and removing dozens of snapshots, the free memory probably was already lower than the usual 45GiB. I was flattening an image of 75GiB and 2 out of 4 nodes ran out of memory. One node even automatically killed processes randomly to free up memory... After regaining control over the system, I put that node in maintenance and rebooted this node. After reboot the free mem was like 70Gib. Over night all of the nodes were back on the usual 45GiB.
Today I checked the free mem of each node: 45GiB free. So I flattened again an image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast!
Is there a way to avoid this behavior of the cluster?
Greetings,
Dominique. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
In the mists of time (Luminous) I experienced a cluster reacting rather badly to a user issuing ~5000 RBD snap trims at once. At the time I raised my local values of osd_snap_trim_cost and osd_snap_trim_sleep (now osd_snap_trim_sleep_???) way up to spread the impact. The subject cluster was roughly half Filestore and half BlueStore; the BlueStore OSDs handled it much better than did Filestore.
On Apr 25, 2025, at 2:11 PM, Eugen Block <eblock@nde.ag> wrote:
Interesting, I did something similar just a few weeks back, flattening some images (didn’t look at their sizes or overlaps of the snapshots). But I don’t see any spikes in memory usage at all. At that time the cluster was running latest Pacific.
Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
Hi,
Housekeeping... I was cleaning up my snapshots and was flattening clones... Suddenly I ran out of memory on my nodes!
4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node has about 45 GiB of free (available) memory in normal operation.
After flattening several images and removing dozens of snapshots, the free memory probably was already lower than the usual 45GiB. I was flattening an image of 75GiB and 2 out of 4 nodes ran out of memory. One node even automatically killed processes randomly to free up memory... After regaining control over the system, I put that node in maintenance and rebooted this node. After reboot the free mem was like 70Gib. Over night all of the nodes were back on the usual 45GiB.
Today I checked the free mem of each node: 45GiB free. So I flattened again an image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast!
Is there a way to avoid this behavior of the cluster?
Greetings,
Dominique. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Anthony D'Atri
-
Dominique Ramaekers
-
Eugen Block
-
Frédéric Nass