OSDs ignore memory limit
Hello everyone, I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits? See e.g. the following: $ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d Best regards, Jonas -- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
I noticed the quite high reported memory stats for OSDs as well on a recently upgraded customer cluster, now running 18.2.4. But checking the top output etc. doesn't confirm those values. I don't really know where they come from, tbh. Can you confirm that those are actually OSD processes filling up the RAM? Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Hello everyone,
I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits?
See e.g. the following:
$ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d
Best regards, Jonas
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, it's the ceph-osd processes filling up the RAM. On 2025-04-09 15:13, Eugen Block wrote:
I noticed the quite high reported memory stats for OSDs as well on a recently upgraded customer cluster, now running 18.2.4. But checking the top output etc. doesn't confirm those values. I don't really know where they come from, tbh. Can you confirm that those are actually OSD processes filling up the RAM?
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Hello everyone,
I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits?
See e.g. the following:
$ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d
Best regards, Jonas
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
Then I suggest to do the usual troubleshooting [0], not necessarily in this order: - osd logs - ceph tell osd.X heap stats - ceph osd df tree (to look for unbalanced PG distribution) - check tracker.ceph.com for existing issues - How are the nodes equipped RAM wise? - Are the oom killers happening across all OSDs or only a subset, or even always the same ones? - Is the cluster healthy? 'ceph -s' output could be useful. Squid has the osd_memory_target_autotune feature enabled by default, can you check 'ceph config dump' and look for osd memory entries? [0] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd/ Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Yes, it's the ceph-osd processes filling up the RAM.
On 2025-04-09 15:13, Eugen Block wrote:
I noticed the quite high reported memory stats for OSDs as well on a recently upgraded customer cluster, now running 18.2.4. But checking the top output etc. doesn't confirm those values. I don't really know where they come from, tbh. Can you confirm that those are actually OSD processes filling up the RAM?
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Hello everyone,
I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits?
See e.g. the following:
$ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d
Best regards, Jonas
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just in case, make sure the Ceph builds you use do have tcmalloc enabled in the first place. The only time I’ve seen OSDs exceed their memory targets so far was on a Pacific cluster that used Debian 12 provided packages, and I eventually figured that those had Crimson enabled - which comes with its own allocator and completely disables tcmalloc, making osd_memory_target completely ignored.
On 9. Apr 2025, at 15:54, Eugen Block <eblock@nde.ag> wrote:
Then I suggest to do the usual troubleshooting [0], not necessarily in this order:
- osd logs - ceph tell osd.X heap stats - ceph osd df tree (to look for unbalanced PG distribution) - check tracker.ceph.com for existing issues - How are the nodes equipped RAM wise? - Are the oom killers happening across all OSDs or only a subset, or even always the same ones? - Is the cluster healthy? 'ceph -s' output could be useful.
Squid has the osd_memory_target_autotune feature enabled by default, can you check 'ceph config dump' and look for osd memory entries?
[0] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd/
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Yes, it's the ceph-osd processes filling up the RAM.
On 2025-04-09 15:13, Eugen Block wrote:
I noticed the quite high reported memory stats for OSDs as well on a recently upgraded customer cluster, now running 18.2.4. But checking the top output etc. doesn't confirm those values. I don't really know where they come from, tbh. Can you confirm that those are actually OSD processes filling up the RAM?
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Hello everyone,
I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits?
See e.g. the following:
$ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d
Best regards, Jonas
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
But checking the top output etc. doesn't confirm those values.
I suspect a startup peak that subsides for steady-state operation. I observed this with mons back in Luminous. A cluster had been expanded considerably without restarting mons, so when they tried to restart there wasn’t enough memory. The prom process collector might capture this, or frequent sampling of the OSDs’ admin socket for TCMALLOC heap stats. Logs should also have a profile of processes with info at the time of OOMkilling.
I don't really know where they come from, tbh. Can you confirm that those are actually OSD processes filling up the RAM?
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Please set your MUA to not wrap.
Hello everyone,
I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits?
Remember that osd_memory_target is a TARGET not a LIMIT. Upstream docs suggest an aggregate 20% headroom, personally I like 100% headroom, but that’s informed by some prior experiences that likely are much less of a concern these days.
See e.g. the following:
$ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
I had not noticed the MEM LIM column. Digging through the source I don’t immediately see where that comes from, but I suspect that in a non-Rook environment, at least, it reflects osd_memory_target. Your nodes each have 11 OSDs, and at least some have a mon or other daemons? How much RAM do the nodes have? `ceph osd dump | grep pool` `ceph status` `ceph config dump | grep osd_memory_target` `ceph config dump | grep osd_memory_target_autotune` Do the nodes run anything that isn’t Ceph? Do you have cron jobs or playbook runs or something that might cause an ephemeral yet hungry process to run at times? If MEM LIM is indeed the osd_memory_target, your nodes would seem to be light on RAM. The default osd_memory_target is 4GB. I’m going to SWAG that this node has 52GB of physmem? If so that IMHO is way too low. I would suggest at least 128GB for 11 OSDs. Are any of these OSDs legacy Filestore?
mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d
Best regards, Jonas
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits?
Remember that osd_memory_target is a TARGET not a LIMIT. Upstream docs suggest an aggregate 20% headroom, personally I like 100% headroom, but that’s informed by some prior experiences that likely are much less of a concern these days.
It's also only a measure of how much of the tuneable memory usage it should aim for, but OSDs also have a separate need for ram that is not negotiable (often at startup and recovery) and for that usage, preferences are not applied, it will try to use the ram it needs in order to fix itself regardless of the admins preferences. -- May the most significant bit of your life be positive.
Hi Jonas, Is swap enabled on OSD nodes? I've seen OSDs using way more memory than osd_memory_target and being OOM-killed from time to time just because swap was enabled. If that's the case, please disable swap in /etc/fstab and reboot the system. Regards, Frédéric. ________________________________ De : Jonas Schwab <jonas.schwab@uni-wuerzburg.de> Envoyé : mercredi 9 avril 2025 13:54 À : ceph-users@ceph.io Objet : [ceph-users] OSDs ignore memory limit Hello everyone, I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits? See e.g. the following: $ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d Best regards, Jonas -- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Jonas, I remembered this kernel bug [1] related to CGroupv1, which caused memory overallocation and OOM kills of OSDs. It may be worth checking you kernel version against this bug. Regards, Frédéric. [1] https://www.spinics.net/lists/ceph-users/msg80421.html ----- Le 10 Avr 25, à 7:54, Frédéric Nass frederic.nass@univ-lorraine.fr a écrit :
Hi Jonas,
Is swap enabled on OSD nodes?
I've seen OSDs using way more memory than osd_memory_target and being OOM-killed from time to time just because swap was enabled. If that's the case, please disable swap in /etc/fstab and reboot the system.
Regards, Frédéric.
________________________________ De : Jonas Schwab <jonas.schwab@uni-wuerzburg.de> Envoyé : mercredi 9 avril 2025 13:54 À : ceph-users@ceph.io Objet : [ceph-users] OSDs ignore memory limit
Hello everyone,
I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits?
See e.g. the following:
$ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d
Best regards, Jonas
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Jonas, Anthony gave some good advice for some things to check. You can also dump the mempool statistics for OSDs that you identify are over their memory target using: "ceph daemon osd.NNN dump_mempools" The osd_memory_target code basically looks at the memory usage of the process and then periodically grows or shrinks the aggregate memory for caches based on how far off the process usage is from the target. It's not perfect, but generally keeps memory close to the target size. It can't do anything if there is a memory leak or other component driving the overall memory usage higher than the target though. One example of this is that in erasure coded pools, huge xattrs on objects can drive pglog memory usage extremely high and the osd_memory_autotuning may not be able to compensate for it. Having said this, I'd suggest looking at the actual targets and the mempools and see if you can figure out where the memory is going and if its truly over the target. The targets themselves can be autotuned higher up in the stack in some cases. Mark On 4/9/25 07:52, Jonas Schwab wrote:
Hello everyone,
I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits?
See e.g. the following:
$ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d
Best regards, Jonas
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
participants (7)
-
Anthony D'Atri
-
Eric Petit
-
Eugen Block
-
Frédéric Nass
-
Janne Johansson
-
Jonas Schwab
-
Mark Nelson