20 Sep
2019
20 Sep
'19
9:43 p.m.
Hi,
I am using ceph mimic in a small test setup using the below configuration.
OS: ubuntu 18.04
1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan
3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1
Gb lan
1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan
configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure
2+1.
When running a script doing multiple folders creation ceph started throwing
error late IO due to high metadata workload.
once after folder creation complete PG's degraded and I am waiting for PG
to complete recovery but my OSD's starting to crash due to OOM and
restarting after some time.
Now my question is I can wait for recovery to complete but how do I stop
OOM and OSD crash? basically want to know the way to control memory usage
during recovery and make it stable.
I have also set very low PG metadata_pool 8 and data_pool 16.
I have already set "mon osd memory target to 1Gb" and I have set
max-backfill from 1 to 8.
Attached msg from "kern.log" from one of the node and snippet of error msg
in this mail.
---------error msg snippet ----------
-bash: fork: Cannot allocate memory
Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked
oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null),
order=0, oom_score_adj=0
Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/
mems_allowed=0
Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm:
msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu
Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill
process 1727 (ceph-osd) score 489 or sacrifice child
Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727
(ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB,
shmem-rss:0kB
Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped
process 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too
long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250
regards
Amudhan
22 Sep
22 Sep
3:32 p.m.
I'm not aware of any memory settings that control rebuild memory usage.
You are running very under on RAM, have you tried adding more swap or adjusting
/proc/sys/vm/swappiness
---- On Fri, 20 Sep 2019 20:41:09 +0800 Amudhan P <mailto:amudhan83@gmail.com> wrote
----
Hi,
I am using ceph mimic in a small test setup using the below configuration.
OS: ubuntu 18.04
1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan
3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM
and 1 Gb lan
1 node acting as cephfs client
+ 2 core cpu and 4G RAM
and 1 Gb lan
configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1.
When running a script doing multiple folders creation ceph started throwing error late IO
due to high metadata workload.
once after folder creation complete PG's degraded and I am waiting for PG to complete
recovery but my OSD's starting to crash due to OOM and restarting after some time.
Now my question is I can wait for recovery to complete but how do I stop OOM and OSD
crash? basically want to know the way to control memory usage during recovery and make it
stable.
I have also set very low PG metadata_pool 8 and data_pool 16.
I have already set "mon osd memory target to 1Gb" and I have set max-backfill
from 1 to 8.
Attached msg from "kern.log" from one of the node and snippet of error msg in
this mail.
---------error msg snippet ----------
-bash: fork: Cannot allocate memory
Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked oom-killer:
gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/ mems_allowed=0
Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm: msgr-worker-0
Not tainted 4.15.0-45-generic #48-Ubuntu
Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process 1727
(ceph-osd) score 489 or sacrifice child
Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727 (ceph-osd)
total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB
Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process 1727
(ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too long (4965
> 4938), lowering kernel.perf_event_max_sample_rate to 40250
regards
Amudhan
_______________________________________________
ceph-users mailing list -- mailto:ceph-users@ceph.io
To unsubscribe send an email to mailto:ceph-users-leave@ceph.io
23 Sep
23 Sep
12:50 a.m.
Do you think 4GB RAM for two OSD's is low, even with 1 OSD and 4GB memory i
encounter OOM during recovery.
This is just a test setup to try with ceph.
I am planning to use ceph in 36 drive system with Xeon e5v2 proc, 32 GB
RAM, 10Gb nic.
So, i am trying to optimize on memory usage on test setup which can be
latter used if using ceph.
On Sun 22 Sep, 2019, 12:02 PM Ashley Merrick, <singapore(a)amerrick.co.uk>
wrote:
I'm not aware of any memory settings that control
rebuild memory usage.
You are running very under on RAM, have you tried adding more swap or
adjusting /proc/sys/vm/*swappiness*
---- On Fri, 20 Sep 2019 20:41:09 +0800 *Amudhan P <amudhan83(a)gmail.com
<amudhan83(a)gmail.com>>* wrote ----
Hi,
I am using ceph mimic in a small test setup using the below configuration.
OS: ubuntu 18.04
1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan
3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1
Gb lan
1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan
configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure
2+1.
When running a script doing multiple folders creation ceph started
throwing error late IO due to high metadata workload.
once after folder creation complete PG's degraded and I am waiting for PG
to complete recovery but my OSD's starting to crash due to OOM and
restarting after some time.
Now my question is I can wait for recovery to complete but how do I stop
OOM and OSD crash? basically want to know the way to control memory usage
during recovery and make it stable.
I have also set very low PG metadata_pool 8 and data_pool 16.
I have already set "mon osd memory target to 1Gb" and I have set
max-backfill from 1 to 8.
Attached msg from "kern.log" from one of the node and snippet of error msg
in this mail.
---------error msg snippet ----------
-bash: fork: Cannot allocate memory
Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked
oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null),
order=0, oom_score_adj=0
Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/
mems_allowed=0
Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm:
msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu
Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill
process 1727 (ceph-osd) score 489 or sacrifice child
Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727
(ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB,
shmem-rss:0kB
Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped
process 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took
too long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250
regards
Amudhan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:26 a.m.
Hi!
Recomendation is: 1G RAM on 1T disk space + 1-2G for each OSD.
In any case I recommend read this page:
https://docs.ceph.com/docs/master/start/hardware-recommendations
----- Original Message -----
From: "Amudhan P"
<amudhan83(a)gmail.com>
To: "Ashley Merrick" <singapore(a)amerrick.co.uk>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Sunday, 22 September, 2019 18:50:52
Subject: [ceph-users] Re: How to reduce or control memory usage during recovery?
> Do you think 4GB RAM for two OSD's is low, even with 1 OSD and 4GB memory i
> encounter OOM during recovery.
>
> This is just a test setup to try with ceph.
>
> I am planning to use ceph in 36 drive system with Xeon e5v2 proc, 32 GB RAM,
> 10Gb nic.
>
> So, i am trying to optimize on memory usage on test setup which can be latter
> used if using ceph.
>
> On Sun 22 Sep, 2019, 12:02 PM Ashley Merrick, < [
> mailto:singapore@amerrick.co.uk | singapore(a)amerrick.co.uk ] > wrote:
>
>
>
> I'm not aware of any memory settings that control rebuild memory usage.
>
> You are running very under on RAM, have you tried adding more swap or adjusting
> /proc/sys/vm/ swappiness
>
>
>
> ---- On Fri, 20 Sep 2019 20:41:09 +0800 Amudhan P < [ mailto:amudhan83@gmail.com
> | amudhan83(a)gmail.com ] > wrote ----
>
>
>
>
> Hi,
>
> I am using ceph mimic in a small test setup using the below configuration.
>
> OS: ubuntu 18.04
>
> 1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan
> 3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM and 1 Gb lan
> 1 node acting as cephfs client + 2 core cpu and 4G RAM and 1 Gb lan
>
> configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1.
>
> When running a script doing multiple folders creation ceph started throwing
> error late IO due to high metadata workload.
> once after folder creation complete PG's degraded and I am waiting for PG to
> complete recovery but my OSD's starting to crash due to OOM and restarting
> after some time.
>
> Now my question is I can wait for recovery to complete but how do I stop OOM and
> OSD crash? basically want to know the way to control memory usage during
> recovery and make it stable.
>
> I have also set very low PG metadata_pool 8 and data_pool 16.
>
> I have already set "mon osd memory target to 1Gb" and I have set
max-backfill
> from 1 to 8.
>
> Attached msg from "kern.log" from one of the node and snippet of error msg
in
> this mail.
>
> ---------error msg snippet ----------
> -bash: fork: Cannot allocate memory
>
> Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked
> oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0,
> oom_score_adj=0
> Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/
> mems_allowed=0
> Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm:
> msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu
>
> Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process
> 1727 (ceph-osd) score 489 or sacrifice child
> Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727
> (ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB
> Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process
> 1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too long
> (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250
>
> regards
> Amudhan
> _______________________________________________
> ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users(a)ceph.io ]
> To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io |
> ceph-users-leave(a)ceph.io ]
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:25 p.m.
On 9/22/19 10:50 PM, Amudhan P wrote:
Do you think 4GB RAM for two OSD's is low, even
with 1 OSD and
4GB memory i encounter OOM during recovery.
This is just a test setup to try with ceph.
I am planning to use ceph in 36 drive system with Xeon e5v2 proc, 32
GB RAM, 10Gb nic.
So, i am trying to optimize on memory usage on test setup which can be
latter used if using ceph.
For bluestore setups by default 1 OSD need 4GB RAM, actually ~5-6GB for
recovery purposes.
For 36OSD's you'll need 256GB RAM.
k
4:46 p.m.
Why does it use such heavy RAM?
I am planning to use only cephfs, no block or object, is there a way to
take control over memory?
On Mon, Sep 23, 2019 at 10:56 AM Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
On 9/22/19 10:50 PM, Amudhan P wrote:
Do you think 4GB RAM for two OSD's is low,
even with 1 OSD and
4GB memory i encounter OOM during recovery.
This is just a test setup to try with ceph.
I am planning to use ceph in 36 drive system with Xeon e5v2 proc, 32
GB RAM, 10Gb nic.
So, i am trying to optimize on memory usage on test setup which can be
latter used if using ceph.
For bluestore setups by default 1 OSD need 4GB RAM, actually ~5-6GB for
recovery purposes.
For 36OSD's you'll need 256GB RAM.
k
24 Sep
24 Sep
12:23 a.m.
On Fri, Sep 20, 2019 at 5:41 AM Amudhan P <amudhan83(a)gmail.com> wrote:
I have already set "mon osd memory target to
1Gb" and I have set max-backfill from 1 to 8.
Reducing the number of backfills should reduce the amount of memory,
especially for EC pools.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
4:31 p.m.
memory usage was high even when backfills is set to "1".
On Mon, Sep 23, 2019 at 8:54 PM Robert LeBlanc <robert(a)leblancnet.us> wrote:
On Fri, Sep 20, 2019 at 5:41 AM Amudhan P
<amudhan83(a)gmail.com> wrote:
I have already set "mon osd memory target to
1Gb" and I have set
max-backfill from 1 to 8.
Reducing the number of backfills should reduce the amount of memory,
especially for EC pools.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
11:55 p.m.
On Tue, Sep 24, 2019 at 12:27 AM Amudhan P <amudhan83(a)gmail.com> wrote:
memory usage was high even when backfills is set to "1".
Memory usage will not decrease by adding more backfills. EC is very
CPU and RAM intensive during recovery as it has to rebuild the shards.
I don't know if reducing stripe size or object size would help, but
they you may have issues in other areas.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
1679
days inactive
1683
days old
8 comments
5 participants
participants (5)
-
Amudhan P -
Ashley Merrick -
Fyodor Ustinov -
Konstantin Shalygin -
Robert LeBlanc