Hi, On my ceph osd servers I have lot of "out of memory messages". My servers are configured with : - 32 G of memory - 11 HDD (3,5 T each) (+ 2 HDD for the system) And the error messages are : /[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child// //[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB// / Top result : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd Is there a way to limit or reduce the memory usage of each osd deamon ? Thank you, Regards, Sylvain PORTIER. --- L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast. https://www.avast.com/antivirus
You can set osd_memory_target to a lower value like 2-2.5GB. Depending on the version of Ceph you are using. On Fri, Aug 30, 2019, 10:21 AM Sylvain PORTIER <cabeur@free.fr> wrote:
Hi,
On my ceph osd servers I have lot of "out of memory messages".
My servers are configured with :
- 32 G of memory
- 11 HDD (3,5 T each) (+ 2 HDD for the system)
And the error messages are :
*[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child* *[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB*
Top result :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd
Is there a way to limit or reduce the memory usage of each osd deamon ?
Thank you,
Regards, Sylvain PORTIER.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Garanti sans virus. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> <#m_1194716995885891668_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You are looking for the config option osd_memory_target Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Aug 30, 2019 at 4:21 PM Sylvain PORTIER <cabeur@free.fr> wrote:
Hi,
On my ceph osd servers I have lot of "out of memory messages".
My servers are configured with :
- 32 G of memory
- 11 HDD (3,5 T each) (+ 2 HDD for the system)
And the error messages are :
*[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child* *[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB*
Top result :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd
Is there a way to limit or reduce the memory usage of each osd deamon ?
Thank you,
Regards, Sylvain PORTIER.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Garanti sans virus. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> <#m_-7828897276136192096_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 8/30/19 9:20 PM, Sylvain PORTIER wrote:
On my ceph osd servers I have lot of "out of memory messages".
My servers are configured with :
- 32 G of memory
- 11 HDD (3,5 T each) (+ 2 HDD for the system)
And the error messages are :
/[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child// //[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB// /
Top result :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd
Is there a way to limit or reduce the memory usage of each osd deamon ?
What is you Ceph version? k
Hi, Thank you for your response. I am using Nautilus version. Sylvain PORTIER. Le 31/08/2019 à 05:29, Konstantin Shalygin a écrit :
On 8/30/19 9:20 PM, Sylvain PORTIER wrote:
On my ceph osd servers I have lot of "out of memory messages".
My servers are configured with :
- 32 G of memory
- 11 HDD (3,5 T each) (+ 2 HDD for the system)
And the error messages are :
/[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child// //[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB// /
Top result :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd
Is there a way to limit or reduce the memory usage of each osd deamon ?
What is you Ceph version?
k
--- L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast. https://www.avast.com/antivirus
participants (4)
-
Brett Chancellor
-
Konstantin Shalygin
-
Paul Emmerich
-
Sylvain PORTIER