We are constantly running into this issue. The mem_per_process does help but it is just a heuristic and needs to be tuned quite frequently.

The last time this occurred, I did give it some thought. What if we tuned the build system a little? We could identify the biggest memory hogs and build them sequentially (post-parallel build?). I know ceph-dencoder was super-memory-expensive (it required more than 4G of RAM even on 32-bit architectures so we couldn't even build on 32-bit architectures anymore) and it might be a first candidate on the 'sequential' list. Do we have any other culprits?

What do you think?

Regards,

Boris

On Wed, Jan 27, 2021 at 8:34 PM Ken Dreyer <kdreyer@redhat.com> wrote:

Hi folks,

I'm seeing some of our internal Red Hat builders going OOM and killing
ceph builds. This is happening across architectures.

Upstream our braggi builders have 48 vCPUs and 256GB of RAM. That's not small.

What is the minimum memory and CPU requirement for building pacific?

Internally, to use one ppc64le example, we're running with 14Gb RAM
and 16 CPUs, and the RPM spec file chooses -j5, hitting OOM. We tuned
mem_per_process from 2500 to 2700 a while back to alleviate this, but
we're still hitting OOM consistently with the pacific branch now.

- Ken
_______________________________________________
Dev mailing list -- dev@ceph.io
To unsubscribe send an email to dev-leave@ceph.io