How to reduce or control memory usage during recovery?
Hi, I am using ceph mimic in a small test setup using the below configuration. OS: ubuntu 18.04 1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan 3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1 Gb lan 1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1. When running a script doing multiple folders creation ceph started throwing error late IO due to high metadata workload. once after folder creation complete PG's degraded and I am waiting for PG to complete recovery but my OSD's starting to crash due to OOM and restarting after some time. Now my question is I can wait for recovery to complete but how do I stop OOM and OSD crash? basically want to know the way to control memory usage during recovery and make it stable. I have also set very low PG metadata_pool 8 and data_pool 16. I have already set "mon osd memory target to 1Gb" and I have set max-backfill from 1 to 8. Attached msg from "kern.log" from one of the node and snippet of error msg in this mail. ---------error msg snippet ---------- -bash: fork: Cannot allocate memory Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/ mems_allowed=0 Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm: msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process 1727 (ceph-osd) score 489 or sacrifice child Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727 (ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250 regards Amudhan
I'm not aware of any memory settings that control rebuild memory usage. You are running very under on RAM, have you tried adding more swap or adjusting /proc/sys/vm/swappiness ---- On Fri, 20 Sep 2019 20:41:09 +0800 Amudhan P <mailto:amudhan83@gmail.com> wrote ---- Hi, I am using ceph mimic in a small test setup using the below configuration. OS: ubuntu 18.04 1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan 3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1 Gb lan 1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1. When running a script doing multiple folders creation ceph started throwing error late IO due to high metadata workload. once after folder creation complete PG's degraded and I am waiting for PG to complete recovery but my OSD's starting to crash due to OOM and restarting after some time. Now my question is I can wait for recovery to complete but how do I stop OOM and OSD crash? basically want to know the way to control memory usage during recovery and make it stable. I have also set very low PG metadata_pool 8 and data_pool 16. I have already set "mon osd memory target to 1Gb" and I have set max-backfill from 1 to 8. Attached msg from "kern.log" from one of the node and snippet of error msg in this mail. ---------error msg snippet ---------- -bash: fork: Cannot allocate memory Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/ mems_allowed=0 Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm: msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process 1727 (ceph-osd) score 489 or sacrifice child Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727 (ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250 regards Amudhan _______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io To unsubscribe send an email to mailto:ceph-users-leave@ceph.io
Do you think 4GB RAM for two OSD's is low, even with 1 OSD and 4GB memory i encounter OOM during recovery. This is just a test setup to try with ceph. I am planning to use ceph in 36 drive system with Xeon e5v2 proc, 32 GB RAM, 10Gb nic. So, i am trying to optimize on memory usage on test setup which can be latter used if using ceph. On Sun 22 Sep, 2019, 12:02 PM Ashley Merrick, <singapore@amerrick.co.uk> wrote:
I'm not aware of any memory settings that control rebuild memory usage.
You are running very under on RAM, have you tried adding more swap or adjusting /proc/sys/vm/*swappiness*
---- On Fri, 20 Sep 2019 20:41:09 +0800 *Amudhan P <amudhan83@gmail.com <amudhan83@gmail.com>>* wrote ----
Hi,
I am using ceph mimic in a small test setup using the below configuration.
OS: ubuntu 18.04
1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan 3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1 Gb lan 1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan
configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1.
When running a script doing multiple folders creation ceph started throwing error late IO due to high metadata workload. once after folder creation complete PG's degraded and I am waiting for PG to complete recovery but my OSD's starting to crash due to OOM and restarting after some time.
Now my question is I can wait for recovery to complete but how do I stop OOM and OSD crash? basically want to know the way to control memory usage during recovery and make it stable.
I have also set very low PG metadata_pool 8 and data_pool 16.
I have already set "mon osd memory target to 1Gb" and I have set max-backfill from 1 to 8.
Attached msg from "kern.log" from one of the node and snippet of error msg in this mail.
---------error msg snippet ---------- -bash: fork: Cannot allocate memory
Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/ mems_allowed=0 Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm: msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu
Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process 1727 (ceph-osd) score 489 or sacrifice child Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727 (ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250
regards Amudhan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi! Recomendation is: 1G RAM on 1T disk space + 1-2G for each OSD. In any case I recommend read this page: https://docs.ceph.com/docs/master/start/hardware-recommendations ----- Original Message -----
From: "Amudhan P" <amudhan83@gmail.com> To: "Ashley Merrick" <singapore@amerrick.co.uk> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Sunday, 22 September, 2019 18:50:52 Subject: [ceph-users] Re: How to reduce or control memory usage during recovery?
Do you think 4GB RAM for two OSD's is low, even with 1 OSD and 4GB memory i encounter OOM during recovery.
This is just a test setup to try with ceph.
I am planning to use ceph in 36 drive system with Xeon e5v2 proc, 32 GB RAM, 10Gb nic.
So, i am trying to optimize on memory usage on test setup which can be latter used if using ceph.
On Sun 22 Sep, 2019, 12:02 PM Ashley Merrick, < [ mailto:singapore@amerrick.co.uk | singapore@amerrick.co.uk ] > wrote:
I'm not aware of any memory settings that control rebuild memory usage.
You are running very under on RAM, have you tried adding more swap or adjusting /proc/sys/vm/ swappiness
---- On Fri, 20 Sep 2019 20:41:09 +0800 Amudhan P < [ mailto:amudhan83@gmail.com | amudhan83@gmail.com ] > wrote ----
Hi,
I am using ceph mimic in a small test setup using the below configuration.
OS: ubuntu 18.04
1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan 3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1 Gb lan 1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan
configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1.
When running a script doing multiple folders creation ceph started throwing error late IO due to high metadata workload. once after folder creation complete PG's degraded and I am waiting for PG to complete recovery but my OSD's starting to crash due to OOM and restarting after some time.
Now my question is I can wait for recovery to complete but how do I stop OOM and OSD crash? basically want to know the way to control memory usage during recovery and make it stable.
I have also set very low PG metadata_pool 8 and data_pool 16.
I have already set "mon osd memory target to 1Gb" and I have set max-backfill from 1 to 8.
Attached msg from "kern.log" from one of the node and snippet of error msg in this mail.
---------error msg snippet ---------- -bash: fork: Cannot allocate memory
Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/ mems_allowed=0 Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm: msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu
Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process 1727 (ceph-osd) score 489 or sacrifice child Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727 (ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250
regards Amudhan _______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 9/22/19 10:50 PM, Amudhan P wrote:
Do you think 4GB RAM for two OSD's is low, even with 1 OSD and 4GB memory i encounter OOM during recovery.
This is just a test setup to try with ceph.
I am planning to use ceph in 36 drive system with Xeon e5v2 proc, 32 GB RAM, 10Gb nic.
So, i am trying to optimize on memory usage on test setup which can be latter used if using ceph.
For bluestore setups by default 1 OSD need 4GB RAM, actually ~5-6GB for recovery purposes. For 36OSD's you'll need 256GB RAM. k
Why does it use such heavy RAM? I am planning to use only cephfs, no block or object, is there a way to take control over memory? On Mon, Sep 23, 2019 at 10:56 AM Konstantin Shalygin <k0ste@k0ste.ru> wrote:
On 9/22/19 10:50 PM, Amudhan P wrote:
Do you think 4GB RAM for two OSD's is low, even with 1 OSD and 4GB memory i encounter OOM during recovery.
This is just a test setup to try with ceph.
I am planning to use ceph in 36 drive system with Xeon e5v2 proc, 32 GB RAM, 10Gb nic.
So, i am trying to optimize on memory usage on test setup which can be latter used if using ceph.
For bluestore setups by default 1 OSD need 4GB RAM, actually ~5-6GB for recovery purposes.
For 36OSD's you'll need 256GB RAM.
k
On Fri, Sep 20, 2019 at 5:41 AM Amudhan P <amudhan83@gmail.com> wrote:
I have already set "mon osd memory target to 1Gb" and I have set max-backfill from 1 to 8.
Reducing the number of backfills should reduce the amount of memory, especially for EC pools. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
memory usage was high even when backfills is set to "1". On Mon, Sep 23, 2019 at 8:54 PM Robert LeBlanc <robert@leblancnet.us> wrote:
On Fri, Sep 20, 2019 at 5:41 AM Amudhan P <amudhan83@gmail.com> wrote:
I have already set "mon osd memory target to 1Gb" and I have set max-backfill from 1 to 8.
Reducing the number of backfills should reduce the amount of memory, especially for EC pools.
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Tue, Sep 24, 2019 at 12:27 AM Amudhan P <amudhan83@gmail.com> wrote:
memory usage was high even when backfills is set to "1".
Memory usage will not decrease by adding more backfills. EC is very CPU and RAM intensive during recovery as it has to rebuild the shards. I don't know if reducing stripe size or object size would help, but they you may have issues in other areas. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
participants (5)
-
Amudhan P
-
Ashley Merrick
-
Fyodor Ustinov
-
Konstantin Shalygin
-
Robert LeBlanc