Hello, I am using a CEPH cluster. After monitoring it, I set: ceph config set osd osd_memory_target_autotune false ceph config set osd osd_memory_target 1G Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case? Ceph version: Quincy OSD: 3 nodes with 11 nvme each and 512GB ram per node. CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket. Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000 Thank you very much. Nguyen Huu Khoi
Hi, osd_memory_target is a "target", i.e. an OSD make an effort to consume up to the specified amount of RAM, but won't consume less than required for its operation and caches, which have some minimum values such as for example osd_memory_cache_min, bluestore_cache_size, bluestore_cache_size_hdd, bluestore_cache_size_ssd, etc. The recommended and default OSD memory target is 4 GB. Your nodes have a sufficient amount of RAM, thus I don't see why you would want to reduce OSD memory consumption below the recommended defaults, especially considering that in-memory caches are important for Ceph operations as they're many times faster than the fastest storage devices. I run my OSDs with osd_memory_target=17179869184 (16 GB) and it helps, especially with slower HDD-backed OSDs. /Z On Thu, 16 Nov 2023 at 01:02, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello, I am using a CEPH cluster. After monitoring it, I set:
ceph config set osd osd_memory_target_autotune false
ceph config set osd osd_memory_target 1G
Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case?
Ceph version: Quincy
OSD: 3 nodes with 11 nvme each and 512GB ram per node.
CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket.
Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000
Thank you very much.
Nguyen Huu Khoi _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello. Thank you very much for your explanation. Because I thought that osd_memory_target will help me limit OSD memory usage which will help prevent memory leak - I tried google and many people talked about memory leak. A nice man, @Anthony D'Atri <aad@dreamsnake.net> , on this forum helped me to understand that it wont help to limit OSD usage. I set it to 1GB because I want to see how this option works. I will read and test with caches options. Nguyen Huu Khoi On Thu, Nov 16, 2023 at 12:23 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
osd_memory_target is a "target", i.e. an OSD make an effort to consume up to the specified amount of RAM, but won't consume less than required for its operation and caches, which have some minimum values such as for example osd_memory_cache_min, bluestore_cache_size, bluestore_cache_size_hdd, bluestore_cache_size_ssd, etc. The recommended and default OSD memory target is 4 GB.
Your nodes have a sufficient amount of RAM, thus I don't see why you would want to reduce OSD memory consumption below the recommended defaults, especially considering that in-memory caches are important for Ceph operations as they're many times faster than the fastest storage devices. I run my OSDs with osd_memory_target=17179869184 (16 GB) and it helps, especially with slower HDD-backed OSDs.
/Z
On Thu, 16 Nov 2023 at 01:02, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello, I am using a CEPH cluster. After monitoring it, I set:
ceph config set osd osd_memory_target_autotune false
ceph config set osd osd_memory_target 1G
Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case?
Ceph version: Quincy
OSD: 3 nodes with 11 nvme each and 512GB ram per node.
CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket.
Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000
Thank you very much.
Nguyen Huu Khoi _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You're most welcome! I'd say that real leak issues are very rare. For example, these are my OSDs with memory target=16GB which have been running for quite a while, as you can see they don't exceed 16 GB RSS: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 92298 167 20 0 18.7g 15.8g 12264 S 1.3 4.2 1974:06 ceph-osd 94527 167 20 0 19.5g 15.8g 12248 S 2.3 4.2 2287:26 ceph-osd 93749 167 20 0 19.1g 15.7g 12804 S 2.3 4.2 1768:22 ceph-osd 89534 167 20 0 20.1g 15.7g 12412 S 4.0 4.2 2512:18 ceph-osd 3706552 167 20 0 20.5g 15.7g 15588 S 2.3 4.2 1385:26 ceph-osd 90297 167 20 0 19.5g 15.6g 12432 S 3.0 4.1 2261:00 ceph-osd 9799 167 20 0 22.9g 15.4g 12432 S 2.0 4.1 2494:00 ceph-osd 9778 167 20 0 23.1g 15.3g 12556 S 2.6 4.1 2591:25 ceph-osd 9815 167 20 0 23.4g 15.1g 12584 S 2.0 4.0 2722:28 ceph-osd 9809 167 20 0 22.3g 15.1g 12068 S 3.6 4.0 5234:52 ceph-osd 9811 167 20 0 23.4g 14.9g 12952 S 2.6 4.0 2593:19 ceph-osd 9819 167 20 0 23.9g 14.9g 12636 S 2.6 4.0 3043:19 ceph-osd 9820 167 20 0 23.3g 14.8g 12884 S 2.0 3.9 3073:43 ceph-osd 9769 167 20 0 22.4g 14.7g 12612 S 2.6 3.9 2840:22 ceph-osd 9836 167 20 0 24.0g 14.7g 12648 S 2.6 3.9 3300:34 ceph-osd 9818 167 20 0 22.0g 14.7g 12152 S 2.3 3.9 5729:06 ceph-osd Long story short, if you set reasonable targets, OSDs are unlikely to exceed them during normal operations. If you set memory targets too low, it is likely that they will be exceeded as OSDs need reasonable amounts of memory to operate. /Z On Thu, 16 Nov 2023 at 08:37, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello. Thank you very much for your explanation.
Because I thought that osd_memory_target will help me limit OSD memory usage which will help prevent memory leak - I tried google and many people talked about memory leak. A nice man, @Anthony D'Atri <aad@dreamsnake.net> , on this forum helped me to understand that it wont help to limit OSD usage.
I set it to 1GB because I want to see how this option works.
I will read and test with caches options.
Nguyen Huu Khoi
On Thu, Nov 16, 2023 at 12:23 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
osd_memory_target is a "target", i.e. an OSD make an effort to consume up to the specified amount of RAM, but won't consume less than required for its operation and caches, which have some minimum values such as for example osd_memory_cache_min, bluestore_cache_size, bluestore_cache_size_hdd, bluestore_cache_size_ssd, etc. The recommended and default OSD memory target is 4 GB.
Your nodes have a sufficient amount of RAM, thus I don't see why you would want to reduce OSD memory consumption below the recommended defaults, especially considering that in-memory caches are important for Ceph operations as they're many times faster than the fastest storage devices. I run my OSDs with osd_memory_target=17179869184 (16 GB) and it helps, especially with slower HDD-backed OSDs.
/Z
On Thu, 16 Nov 2023 at 01:02, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello, I am using a CEPH cluster. After monitoring it, I set:
ceph config set osd osd_memory_target_autotune false
ceph config set osd osd_memory_target 1G
Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case?
Ceph version: Quincy
OSD: 3 nodes with 11 nvme each and 512GB ram per node.
CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket.
Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000
Thank you very much.
Nguyen Huu Khoi _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, Yes, I see it does not exceed RSS but I see in "ceph orch ps". it is over target. Does Mem Use include cache, I am right? NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.7 sg-osd01 running (3d) 8m ago 4w 4231M 4096M 17.2.6 90a2664234e1 922185643cb8 osd.8 sg-osd03 running (3d) 7m ago 4w 3407M 4096M 17.2.6 90a2664234e1 0ec74fe54bbe osd.9 sg-osd01 running (3d) 8m ago 4w 4575M 4096M 17.2.6 90a2664234e1 c2f1c1ee2087 osd.10 sg-osd03 running (3d) 7m ago 4w 3821M 4096M 17.2.6 90a2664234e1 fecbd5e910de osd.11 sg-osd01 running (3d) 8m ago 4w 3578M 4096M 17.2.6 90a2664234e1 f201704e9026 osd.12 sg-osd03 running (3d) 7m ago 4w 3076M 4096M 17.2.6 90a2664234e1 e741b67b6582 osd.13 sg-osd01 running (3d) 8m ago 4w 3688M 4096M 17.2.6 90a2664234e1 bffa59278fc2 osd.14 sg-osd03 running (3d) 7m ago 4w 3652M 4096M 17.2.6 90a2664234e1 7d9eb3fb9c1e osd.15 sg-osd01 running (3d) 8m ago 4w 3343M 4096M 17.2.6 90a2664234e1 d96a425ae5c9 osd.16 sg-osd03 running (3d) 7m ago 4w 2492M 4096M 17.2.6 90a2664234e1 637c43176fdc osd.17 sg-osd01 running (3d) 8m ago 4w 3011M 4096M 17.2.6 90a2664234e1 a39456dd2c0c osd.18 sg-osd03 running (3d) 7m ago 4w 2341M 4096M 17.2.6 90a2664234e1 7b750672391b osd.19 sg-osd01 running (3d) 8m ago 4w 2672M 4096M 17.2.6 90a2664234e1 6358234e95f5 osd.20 sg-osd03 running (3d) 7m ago 4w 3297M 4096M 17.2.6 90a2664234e1 2ecba6b066fd osd.21 sg-osd01 running (3d) 8m ago 4w 5147M 4096M 17.2.6 90a2664234e1 1d0e4efe48bd osd.22 sg-osd03 running (3d) 7m ago 4w 3432M 4096M 17.2.6 90a2664234e1 5bb6d4f71f9d osd.23 sg-osd03 running (3d) 7m ago 4w 2893M 4096M 17.2.6 90a2664234e1 f7e1948e57d5 osd.24 sg-osd02 running (3d) 7m ago 12d 3007M 4096M 17.2.6 90a2664234e1 85d896abe467 osd.25 sg-osd02 running (3d) 7m ago 12d 2666M 4096M 17.2.6 90a2664234e1 9800cd8ff1a1 osd.26 sg-osd02 running (3d) 7m ago 12d 2918M 4096M 17.2.6 90a2664234e1 f2e0b2d50625 osd.27 sg-osd02 running (3d) 7m ago 12d 3586M 4096M 17.2.6 90a2664234e1 ee2fa3a9b40a osd.28 sg-osd02 running (3d) 7m ago 12d 2391M 4096M 17.2.6 90a2664234e1 4cf7adf9f60a osd.29 sg-osd02 running (3d) 7m ago 12d 5642M 4096M 17.2.6 90a2664234e1 8c1ba98a1738 osd.30 sg-osd02 running (3d) 7m ago 12d 4728M 4096M 17.2.6 90a2664234e1 e308497de2e5 osd.31 sg-osd02 running (3d) 7m ago 12d 3615M 4096M 17.2.6 90a2664234e1 89b80d464627 osd.32 sg-osd02 running (3d) 7m ago 12d 1703M 4096M 17.2.6 90a2664234e1 1e4608786078 osd.33 sg-osd02 running (3d) 7m ago 12d 3039M 4096M 17.2.6 90a2664234e1 16e04a1da987 osd.34 sg-osd02 running (3d) 7m ago 12d 2434M 4096M 17.2.6 90a2664234e1 014076e28182 btw as you said, I feel this value does not have much impact because if we set 1 or 4GB. It still can consume much memory when they need more memory, Nguyen Huu Khoi On Thu, Nov 16, 2023 at 2:13 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
You're most welcome!
I'd say that real leak issues are very rare. For example, these are my OSDs with memory target=16GB which have been running for quite a while, as you can see they don't exceed 16 GB RSS:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 92298 167 20 0 18.7g 15.8g 12264 S 1.3 4.2 1974:06 ceph-osd 94527 167 20 0 19.5g 15.8g 12248 S 2.3 4.2 2287:26 ceph-osd 93749 167 20 0 19.1g 15.7g 12804 S 2.3 4.2 1768:22 ceph-osd 89534 167 20 0 20.1g 15.7g 12412 S 4.0 4.2 2512:18 ceph-osd 3706552 167 20 0 20.5g 15.7g 15588 S 2.3 4.2 1385:26 ceph-osd 90297 167 20 0 19.5g 15.6g 12432 S 3.0 4.1 2261:00 ceph-osd 9799 167 20 0 22.9g 15.4g 12432 S 2.0 4.1 2494:00 ceph-osd 9778 167 20 0 23.1g 15.3g 12556 S 2.6 4.1 2591:25 ceph-osd 9815 167 20 0 23.4g 15.1g 12584 S 2.0 4.0 2722:28 ceph-osd 9809 167 20 0 22.3g 15.1g 12068 S 3.6 4.0 5234:52 ceph-osd 9811 167 20 0 23.4g 14.9g 12952 S 2.6 4.0 2593:19 ceph-osd 9819 167 20 0 23.9g 14.9g 12636 S 2.6 4.0 3043:19 ceph-osd 9820 167 20 0 23.3g 14.8g 12884 S 2.0 3.9 3073:43 ceph-osd 9769 167 20 0 22.4g 14.7g 12612 S 2.6 3.9 2840:22 ceph-osd 9836 167 20 0 24.0g 14.7g 12648 S 2.6 3.9 3300:34 ceph-osd 9818 167 20 0 22.0g 14.7g 12152 S 2.3 3.9 5729:06 ceph-osd
Long story short, if you set reasonable targets, OSDs are unlikely to exceed them during normal operations. If you set memory targets too low, it is likely that they will be exceeded as OSDs need reasonable amounts of memory to operate.
/Z
On Thu, 16 Nov 2023 at 08:37, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello. Thank you very much for your explanation.
Because I thought that osd_memory_target will help me limit OSD memory usage which will help prevent memory leak - I tried google and many people talked about memory leak. A nice man, @Anthony D'Atri <aad@dreamsnake.net> , on this forum helped me to understand that it wont help to limit OSD usage.
I set it to 1GB because I want to see how this option works.
I will read and test with caches options.
Nguyen Huu Khoi
On Thu, Nov 16, 2023 at 12:23 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
osd_memory_target is a "target", i.e. an OSD make an effort to consume up to the specified amount of RAM, but won't consume less than required for its operation and caches, which have some minimum values such as for example osd_memory_cache_min, bluestore_cache_size, bluestore_cache_size_hdd, bluestore_cache_size_ssd, etc. The recommended and default OSD memory target is 4 GB.
Your nodes have a sufficient amount of RAM, thus I don't see why you would want to reduce OSD memory consumption below the recommended defaults, especially considering that in-memory caches are important for Ceph operations as they're many times faster than the fastest storage devices. I run my OSDs with osd_memory_target=17179869184 (16 GB) and it helps, especially with slower HDD-backed OSDs.
/Z
On Thu, 16 Nov 2023 at 01:02, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello, I am using a CEPH cluster. After monitoring it, I set:
ceph config set osd osd_memory_target_autotune false
ceph config set osd osd_memory_target 1G
Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case?
Ceph version: Quincy
OSD: 3 nodes with 11 nvme each and 512GB ram per node.
CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket.
Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000
Thank you very much.
Nguyen Huu Khoi _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Orch ps seems to show virtual size set instead of resident size set. /Z On Thu, 16 Nov 2023 at 09:43, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello, Yes, I see it does not exceed RSS but I see in "ceph orch ps". it is over target. Does Mem Use include cache, I am right?
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
osd.7 sg-osd01 running (3d) 8m ago 4w 4231M 4096M 17.2.6 90a2664234e1 922185643cb8 osd.8 sg-osd03 running (3d) 7m ago 4w 3407M 4096M 17.2.6 90a2664234e1 0ec74fe54bbe osd.9 sg-osd01 running (3d) 8m ago 4w 4575M 4096M 17.2.6 90a2664234e1 c2f1c1ee2087 osd.10 sg-osd03 running (3d) 7m ago 4w 3821M 4096M 17.2.6 90a2664234e1 fecbd5e910de osd.11 sg-osd01 running (3d) 8m ago 4w 3578M 4096M 17.2.6 90a2664234e1 f201704e9026 osd.12 sg-osd03 running (3d) 7m ago 4w 3076M 4096M 17.2.6 90a2664234e1 e741b67b6582 osd.13 sg-osd01 running (3d) 8m ago 4w 3688M 4096M 17.2.6 90a2664234e1 bffa59278fc2 osd.14 sg-osd03 running (3d) 7m ago 4w 3652M 4096M 17.2.6 90a2664234e1 7d9eb3fb9c1e osd.15 sg-osd01 running (3d) 8m ago 4w 3343M 4096M 17.2.6 90a2664234e1 d96a425ae5c9 osd.16 sg-osd03 running (3d) 7m ago 4w 2492M 4096M 17.2.6 90a2664234e1 637c43176fdc osd.17 sg-osd01 running (3d) 8m ago 4w 3011M 4096M 17.2.6 90a2664234e1 a39456dd2c0c osd.18 sg-osd03 running (3d) 7m ago 4w 2341M 4096M 17.2.6 90a2664234e1 7b750672391b osd.19 sg-osd01 running (3d) 8m ago 4w 2672M 4096M 17.2.6 90a2664234e1 6358234e95f5 osd.20 sg-osd03 running (3d) 7m ago 4w 3297M 4096M 17.2.6 90a2664234e1 2ecba6b066fd osd.21 sg-osd01 running (3d) 8m ago 4w 5147M 4096M 17.2.6 90a2664234e1 1d0e4efe48bd osd.22 sg-osd03 running (3d) 7m ago 4w 3432M 4096M 17.2.6 90a2664234e1 5bb6d4f71f9d osd.23 sg-osd03 running (3d) 7m ago 4w 2893M 4096M 17.2.6 90a2664234e1 f7e1948e57d5 osd.24 sg-osd02 running (3d) 7m ago 12d 3007M 4096M 17.2.6 90a2664234e1 85d896abe467 osd.25 sg-osd02 running (3d) 7m ago 12d 2666M 4096M 17.2.6 90a2664234e1 9800cd8ff1a1 osd.26 sg-osd02 running (3d) 7m ago 12d 2918M 4096M 17.2.6 90a2664234e1 f2e0b2d50625 osd.27 sg-osd02 running (3d) 7m ago 12d 3586M 4096M 17.2.6 90a2664234e1 ee2fa3a9b40a osd.28 sg-osd02 running (3d) 7m ago 12d 2391M 4096M 17.2.6 90a2664234e1 4cf7adf9f60a osd.29 sg-osd02 running (3d) 7m ago 12d 5642M 4096M 17.2.6 90a2664234e1 8c1ba98a1738 osd.30 sg-osd02 running (3d) 7m ago 12d 4728M 4096M 17.2.6 90a2664234e1 e308497de2e5 osd.31 sg-osd02 running (3d) 7m ago 12d 3615M 4096M 17.2.6 90a2664234e1 89b80d464627 osd.32 sg-osd02 running (3d) 7m ago 12d 1703M 4096M 17.2.6 90a2664234e1 1e4608786078 osd.33 sg-osd02 running (3d) 7m ago 12d 3039M 4096M 17.2.6 90a2664234e1 16e04a1da987 osd.34 sg-osd02 running (3d) 7m ago 12d 2434M 4096M 17.2.6 90a2664234e1 014076e28182
btw as you said, I feel this value does not have much impact because if we set 1 or 4GB. It still can consume much memory when they need more memory,
Nguyen Huu Khoi
On Thu, Nov 16, 2023 at 2:13 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
You're most welcome!
I'd say that real leak issues are very rare. For example, these are my OSDs with memory target=16GB which have been running for quite a while, as you can see they don't exceed 16 GB RSS:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 92298 167 20 0 18.7g 15.8g 12264 S 1.3 4.2 1974:06 ceph-osd 94527 167 20 0 19.5g 15.8g 12248 S 2.3 4.2 2287:26 ceph-osd 93749 167 20 0 19.1g 15.7g 12804 S 2.3 4.2 1768:22 ceph-osd 89534 167 20 0 20.1g 15.7g 12412 S 4.0 4.2 2512:18 ceph-osd 3706552 167 20 0 20.5g 15.7g 15588 S 2.3 4.2 1385:26 ceph-osd 90297 167 20 0 19.5g 15.6g 12432 S 3.0 4.1 2261:00 ceph-osd 9799 167 20 0 22.9g 15.4g 12432 S 2.0 4.1 2494:00 ceph-osd 9778 167 20 0 23.1g 15.3g 12556 S 2.6 4.1 2591:25 ceph-osd 9815 167 20 0 23.4g 15.1g 12584 S 2.0 4.0 2722:28 ceph-osd 9809 167 20 0 22.3g 15.1g 12068 S 3.6 4.0 5234:52 ceph-osd 9811 167 20 0 23.4g 14.9g 12952 S 2.6 4.0 2593:19 ceph-osd 9819 167 20 0 23.9g 14.9g 12636 S 2.6 4.0 3043:19 ceph-osd 9820 167 20 0 23.3g 14.8g 12884 S 2.0 3.9 3073:43 ceph-osd 9769 167 20 0 22.4g 14.7g 12612 S 2.6 3.9 2840:22 ceph-osd 9836 167 20 0 24.0g 14.7g 12648 S 2.6 3.9 3300:34 ceph-osd 9818 167 20 0 22.0g 14.7g 12152 S 2.3 3.9 5729:06 ceph-osd
Long story short, if you set reasonable targets, OSDs are unlikely to exceed them during normal operations. If you set memory targets too low, it is likely that they will be exceeded as OSDs need reasonable amounts of memory to operate.
/Z
On Thu, 16 Nov 2023 at 08:37, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello. Thank you very much for your explanation.
Because I thought that osd_memory_target will help me limit OSD memory usage which will help prevent memory leak - I tried google and many people talked about memory leak. A nice man, @Anthony D'Atri <aad@dreamsnake.net> , on this forum helped me to understand that it wont help to limit OSD usage.
I set it to 1GB because I want to see how this option works.
I will read and test with caches options.
Nguyen Huu Khoi
On Thu, Nov 16, 2023 at 12:23 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
osd_memory_target is a "target", i.e. an OSD make an effort to consume up to the specified amount of RAM, but won't consume less than required for its operation and caches, which have some minimum values such as for example osd_memory_cache_min, bluestore_cache_size, bluestore_cache_size_hdd, bluestore_cache_size_ssd, etc. The recommended and default OSD memory target is 4 GB.
Your nodes have a sufficient amount of RAM, thus I don't see why you would want to reduce OSD memory consumption below the recommended defaults, especially considering that in-memory caches are important for Ceph operations as they're many times faster than the fastest storage devices. I run my OSDs with osd_memory_target=17179869184 (16 GB) and it helps, especially with slower HDD-backed OSDs.
/Z
On Thu, 16 Nov 2023 at 01:02, Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello, I am using a CEPH cluster. After monitoring it, I set:
ceph config set osd osd_memory_target_autotune false
ceph config set osd osd_memory_target 1G
Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case?
Ceph version: Quincy
OSD: 3 nodes with 11 nvme each and 512GB ram per node.
CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket.
Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000
Thank you very much.
Nguyen Huu Khoi _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello. I will read more about it. Thank you :) Nguyen Huu Khoi On Thu, Nov 16, 2023 at 3:21 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Orch ps seems to show virtual size set instead of resident size set.
/Z
On Thu, 16 Nov 2023 at 09:43, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello, Yes, I see it does not exceed RSS but I see in "ceph orch ps". it is over target. Does Mem Use include cache, I am right?
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
osd.7 sg-osd01 running (3d) 8m ago 4w 4231M 4096M 17.2.6 90a2664234e1 922185643cb8 osd.8 sg-osd03 running (3d) 7m ago 4w 3407M 4096M 17.2.6 90a2664234e1 0ec74fe54bbe osd.9 sg-osd01 running (3d) 8m ago 4w 4575M 4096M 17.2.6 90a2664234e1 c2f1c1ee2087 osd.10 sg-osd03 running (3d) 7m ago 4w 3821M 4096M 17.2.6 90a2664234e1 fecbd5e910de osd.11 sg-osd01 running (3d) 8m ago 4w 3578M 4096M 17.2.6 90a2664234e1 f201704e9026 osd.12 sg-osd03 running (3d) 7m ago 4w 3076M 4096M 17.2.6 90a2664234e1 e741b67b6582 osd.13 sg-osd01 running (3d) 8m ago 4w 3688M 4096M 17.2.6 90a2664234e1 bffa59278fc2 osd.14 sg-osd03 running (3d) 7m ago 4w 3652M 4096M 17.2.6 90a2664234e1 7d9eb3fb9c1e osd.15 sg-osd01 running (3d) 8m ago 4w 3343M 4096M 17.2.6 90a2664234e1 d96a425ae5c9 osd.16 sg-osd03 running (3d) 7m ago 4w 2492M 4096M 17.2.6 90a2664234e1 637c43176fdc osd.17 sg-osd01 running (3d) 8m ago 4w 3011M 4096M 17.2.6 90a2664234e1 a39456dd2c0c osd.18 sg-osd03 running (3d) 7m ago 4w 2341M 4096M 17.2.6 90a2664234e1 7b750672391b osd.19 sg-osd01 running (3d) 8m ago 4w 2672M 4096M 17.2.6 90a2664234e1 6358234e95f5 osd.20 sg-osd03 running (3d) 7m ago 4w 3297M 4096M 17.2.6 90a2664234e1 2ecba6b066fd osd.21 sg-osd01 running (3d) 8m ago 4w 5147M 4096M 17.2.6 90a2664234e1 1d0e4efe48bd osd.22 sg-osd03 running (3d) 7m ago 4w 3432M 4096M 17.2.6 90a2664234e1 5bb6d4f71f9d osd.23 sg-osd03 running (3d) 7m ago 4w 2893M 4096M 17.2.6 90a2664234e1 f7e1948e57d5 osd.24 sg-osd02 running (3d) 7m ago 12d 3007M 4096M 17.2.6 90a2664234e1 85d896abe467 osd.25 sg-osd02 running (3d) 7m ago 12d 2666M 4096M 17.2.6 90a2664234e1 9800cd8ff1a1 osd.26 sg-osd02 running (3d) 7m ago 12d 2918M 4096M 17.2.6 90a2664234e1 f2e0b2d50625 osd.27 sg-osd02 running (3d) 7m ago 12d 3586M 4096M 17.2.6 90a2664234e1 ee2fa3a9b40a osd.28 sg-osd02 running (3d) 7m ago 12d 2391M 4096M 17.2.6 90a2664234e1 4cf7adf9f60a osd.29 sg-osd02 running (3d) 7m ago 12d 5642M 4096M 17.2.6 90a2664234e1 8c1ba98a1738 osd.30 sg-osd02 running (3d) 7m ago 12d 4728M 4096M 17.2.6 90a2664234e1 e308497de2e5 osd.31 sg-osd02 running (3d) 7m ago 12d 3615M 4096M 17.2.6 90a2664234e1 89b80d464627 osd.32 sg-osd02 running (3d) 7m ago 12d 1703M 4096M 17.2.6 90a2664234e1 1e4608786078 osd.33 sg-osd02 running (3d) 7m ago 12d 3039M 4096M 17.2.6 90a2664234e1 16e04a1da987 osd.34 sg-osd02 running (3d) 7m ago 12d 2434M 4096M 17.2.6 90a2664234e1 014076e28182
btw as you said, I feel this value does not have much impact because if we set 1 or 4GB. It still can consume much memory when they need more memory,
Nguyen Huu Khoi
On Thu, Nov 16, 2023 at 2:13 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
You're most welcome!
I'd say that real leak issues are very rare. For example, these are my OSDs with memory target=16GB which have been running for quite a while, as you can see they don't exceed 16 GB RSS:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 92298 167 20 0 18.7g 15.8g 12264 S 1.3 4.2 1974:06 ceph-osd 94527 167 20 0 19.5g 15.8g 12248 S 2.3 4.2 2287:26 ceph-osd 93749 167 20 0 19.1g 15.7g 12804 S 2.3 4.2 1768:22 ceph-osd 89534 167 20 0 20.1g 15.7g 12412 S 4.0 4.2 2512:18 ceph-osd 3706552 167 20 0 20.5g 15.7g 15588 S 2.3 4.2 1385:26 ceph-osd 90297 167 20 0 19.5g 15.6g 12432 S 3.0 4.1 2261:00 ceph-osd 9799 167 20 0 22.9g 15.4g 12432 S 2.0 4.1 2494:00 ceph-osd 9778 167 20 0 23.1g 15.3g 12556 S 2.6 4.1 2591:25 ceph-osd 9815 167 20 0 23.4g 15.1g 12584 S 2.0 4.0 2722:28 ceph-osd 9809 167 20 0 22.3g 15.1g 12068 S 3.6 4.0 5234:52 ceph-osd 9811 167 20 0 23.4g 14.9g 12952 S 2.6 4.0 2593:19 ceph-osd 9819 167 20 0 23.9g 14.9g 12636 S 2.6 4.0 3043:19 ceph-osd 9820 167 20 0 23.3g 14.8g 12884 S 2.0 3.9 3073:43 ceph-osd 9769 167 20 0 22.4g 14.7g 12612 S 2.6 3.9 2840:22 ceph-osd 9836 167 20 0 24.0g 14.7g 12648 S 2.6 3.9 3300:34 ceph-osd 9818 167 20 0 22.0g 14.7g 12152 S 2.3 3.9 5729:06 ceph-osd
Long story short, if you set reasonable targets, OSDs are unlikely to exceed them during normal operations. If you set memory targets too low, it is likely that they will be exceeded as OSDs need reasonable amounts of memory to operate.
/Z
On Thu, 16 Nov 2023 at 08:37, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello. Thank you very much for your explanation.
Because I thought that osd_memory_target will help me limit OSD memory usage which will help prevent memory leak - I tried google and many people talked about memory leak. A nice man, @Anthony D'Atri <aad@dreamsnake.net> , on this forum helped me to understand that it wont help to limit OSD usage.
I set it to 1GB because I want to see how this option works.
I will read and test with caches options.
Nguyen Huu Khoi
On Thu, Nov 16, 2023 at 12:23 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
osd_memory_target is a "target", i.e. an OSD make an effort to consume up to the specified amount of RAM, but won't consume less than required for its operation and caches, which have some minimum values such as for example osd_memory_cache_min, bluestore_cache_size, bluestore_cache_size_hdd, bluestore_cache_size_ssd, etc. The recommended and default OSD memory target is 4 GB.
Your nodes have a sufficient amount of RAM, thus I don't see why you would want to reduce OSD memory consumption below the recommended defaults, especially considering that in-memory caches are important for Ceph operations as they're many times faster than the fastest storage devices. I run my OSDs with osd_memory_target=17179869184 (16 GB) and it helps, especially with slower HDD-backed OSDs.
/Z
On Thu, 16 Nov 2023 at 01:02, Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello, I am using a CEPH cluster. After monitoring it, I set:
ceph config set osd osd_memory_target_autotune false
ceph config set osd osd_memory_target 1G
Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case?
Ceph version: Quincy
OSD: 3 nodes with 11 nvme each and 512GB ram per node.
CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket.
Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000
Thank you very much.
Nguyen Huu Khoi _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den tors 16 nov. 2023 kl 08:43 skrev Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello, Yes, I see it does not exceed RSS but I see in "ceph orch ps". it is over target. Does Mem Use include cache, I am right?
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
osd.7 sg-osd01 running (3d) 8m ago 4w 4231M 4096M 17.2.6 90a2664234e1 922185643cb8
btw as you said, I feel this value does not have much impact because if we set 1 or 4GB. It still can consume much memory when they need more memory,
Don't forget that the tuneable part is only one type of memory consumption and not ALL parts of the places where and OSD needs ram, so the absolutely mandatory ram consumption is not covered by the settings, only the "optional" part, which goes to caches and other performance improving usages. During recovery and startup and other events, the required usage can and will spike, regardless of your memory usage settings. If recovery suddenly needs 20G, it would not stop recovery because you set the limit to say, max 19G ram, it would try to use 20G to recover, then go back to normal low usage and have as much as possible of the given 19G as caches again. -- May the most significant bit of your life be positive.
Hello. Great Information, I will keep in mind. Thank you :) Nguyen Huu Khoi On Thu, Nov 16, 2023 at 5:51 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den tors 16 nov. 2023 kl 08:43 skrev Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com>:
Hello, Yes, I see it does not exceed RSS but I see in "ceph orch ps". it is
over
target. Does Mem Use include cache, I am right?
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
osd.7 sg-osd01 running (3d) 8m ago 4w 4231M 4096M 17.2.6 90a2664234e1 922185643cb8
btw as you said, I feel this value does not have much impact because if we set 1 or 4GB. It still can consume much memory when they need more memory,
Don't forget that the tuneable part is only one type of memory consumption and not ALL parts of the places where and OSD needs ram, so the absolutely mandatory ram consumption is not covered by the settings, only the "optional" part, which goes to caches and other performance improving usages. During recovery and startup and other events, the required usage can and will spike, regardless of your memory usage settings.
If recovery suddenly needs 20G, it would not stop recovery because you set the limit to say, max 19G ram, it would try to use 20G to recover, then go back to normal low usage and have as much as possible of the given 19G as caches again.
-- May the most significant bit of your life be positive.
participants (3)
-
Janne Johansson
-
Nguyễn Hữu Khôi
-
Zakhar Kirpichenko