27 Jan
2021
27 Jan
'21
7:33 p.m.
Hi folks,
I'm seeing some of our internal Red Hat builders going OOM and killing
ceph builds. This is happening across architectures.
Upstream our braggi builders have 48 vCPUs and 256GB of RAM. That's not small.
What is the minimum memory and CPU requirement for building pacific?
Internally, to use one ppc64le example, we're running with 14Gb RAM
and 16 CPUs, and the RPM spec file chooses -j5, hitting OOM. We tuned
mem_per_process from 2500 to 2700 a while back to alleviate this, but
we're still hitting OOM consistently with the pacific branch now.
- Ken
27 Jan
27 Jan
9:28 p.m.
On 27-1-2021 20:33, Ken Dreyer wrote:
Hi folks,
I'm seeing some of our internal Red Hat builders going OOM and killing
ceph builds. This is happening across architectures.
Upstream our braggi builders have 48 vCPUs and 256GB of RAM. That's not small.
What is the minimum memory and CPU requirement for building pacific?
Internally, to use one ppc64le example, we're running with 14Gb RAM
and 16 CPUs, and the RPM spec file chooses -j5, hitting OOM. We tuned
mem_per_process from 2500 to 2700 a while back to alleviate this, but
we're still hitting OOM consistently with the pacific branch now.
Is more or less equal to what I find on my FreeBSD builds.
they are running in 16Gb ram and I had to tune them down to -j4 to not
run into OOM.
The OEPS mostly happens when a rather large compile job get Clang all
excited, and then we start linking in another thread with lots of libraries.
Sure and guaranteed OOM abort.
Have not looked in detail what all steps are using but it has grown in steps
at my end. Going from -j16 to -j10 and now -j4.
--WjW
9:43 p.m.
We are constantly running into this issue. The mem_per_process does help
but it is just a heuristic and needs to be tuned quite frequently.
The last time this occurred, I did give it some thought. What if we tuned
the build system a little? We could identify the biggest memory hogs and
build them sequentially (post-parallel build?). I know ceph-dencoder was
super-memory-expensive (it required more than 4G of RAM even on 32-bit
architectures so we couldn't even build on 32-bit architectures anymore)
and it might be a first candidate on the 'sequential' list. Do we have any
other culprits?
What do you think?
Regards,
Boris
On Wed, Jan 27, 2021 at 8:34 PM Ken Dreyer <kdreyer(a)redhat.com> wrote:
Hi folks,
I'm seeing some of our internal Red Hat builders going OOM and killing
ceph builds. This is happening across architectures.
Upstream our braggi builders have 48 vCPUs and 256GB of RAM. That's not
small.
What is the minimum memory and CPU requirement for building pacific?
Internally, to use one ppc64le example, we're running with 14Gb RAM
and 16 CPUs, and the RPM spec file chooses -j5, hitting OOM. We tuned
mem_per_process from 2500 to 2700 a while back to alleviate this, but
we're still hitting OOM consistently with the pacific branch now.
- Ken
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
18 Mar
18 Mar
5:12 p.m.
On Wed, Jan 27, 2021 at 2:43 PM Boris Ranto <branto(a)redhat.com> wrote:
We are constantly running into this issue. The
mem_per_process does help but it is just a heuristic and needs to be tuned quite
frequently.
The last time this occurred, I did give it some thought. What if we tuned the build
system a little? We could identify the biggest memory hogs and build them sequentially
(post-parallel build?). I know ceph-dencoder was super-memory-expensive (it required more
than 4G of RAM even on 32-bit architectures so we couldn't even build on 32-bit
architectures anymore) and it might be a first candidate on the 'sequential' list.
Do we have any other culprits?
What do you think?
I think this is a great idea. How do we do this in CMake?
- Ken
1134
days inactive
1184
days old
3 comments
3 participants
participants (3)
-
Boris Ranto -
Ken Dreyer -
Willem Jan Withagen