system requirements to build pacific
Hi folks, I'm seeing some of our internal Red Hat builders going OOM and killing ceph builds. This is happening across architectures. Upstream our braggi builders have 48 vCPUs and 256GB of RAM. That's not small. What is the minimum memory and CPU requirement for building pacific? Internally, to use one ppc64le example, we're running with 14Gb RAM and 16 CPUs, and the RPM spec file chooses -j5, hitting OOM. We tuned mem_per_process from 2500 to 2700 a while back to alleviate this, but we're still hitting OOM consistently with the pacific branch now. - Ken
On 27-1-2021 20:33, Ken Dreyer wrote:
Hi folks,
I'm seeing some of our internal Red Hat builders going OOM and killing ceph builds. This is happening across architectures.
Upstream our braggi builders have 48 vCPUs and 256GB of RAM. That's not small.
What is the minimum memory and CPU requirement for building pacific?
Internally, to use one ppc64le example, we're running with 14Gb RAM and 16 CPUs, and the RPM spec file chooses -j5, hitting OOM. We tuned mem_per_process from 2500 to 2700 a while back to alleviate this, but we're still hitting OOM consistently with the pacific branch now.
Is more or less equal to what I find on my FreeBSD builds. they are running in 16Gb ram and I had to tune them down to -j4 to not run into OOM. The OEPS mostly happens when a rather large compile job get Clang all excited, and then we start linking in another thread with lots of libraries. Sure and guaranteed OOM abort. Have not looked in detail what all steps are using but it has grown in steps at my end. Going from -j16 to -j10 and now -j4. --WjW
We are constantly running into this issue. The mem_per_process does help but it is just a heuristic and needs to be tuned quite frequently. The last time this occurred, I did give it some thought. What if we tuned the build system a little? We could identify the biggest memory hogs and build them sequentially (post-parallel build?). I know ceph-dencoder was super-memory-expensive (it required more than 4G of RAM even on 32-bit architectures so we couldn't even build on 32-bit architectures anymore) and it might be a first candidate on the 'sequential' list. Do we have any other culprits? What do you think? Regards, Boris On Wed, Jan 27, 2021 at 8:34 PM Ken Dreyer <kdreyer@redhat.com> wrote:
Hi folks,
I'm seeing some of our internal Red Hat builders going OOM and killing ceph builds. This is happening across architectures.
Upstream our braggi builders have 48 vCPUs and 256GB of RAM. That's not small.
What is the minimum memory and CPU requirement for building pacific?
Internally, to use one ppc64le example, we're running with 14Gb RAM and 16 CPUs, and the RPM spec file chooses -j5, hitting OOM. We tuned mem_per_process from 2500 to 2700 a while back to alleviate this, but we're still hitting OOM consistently with the pacific branch now.
- Ken _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Wed, Jan 27, 2021 at 2:43 PM Boris Ranto <branto@redhat.com> wrote:
We are constantly running into this issue. The mem_per_process does help but it is just a heuristic and needs to be tuned quite frequently.
The last time this occurred, I did give it some thought. What if we tuned the build system a little? We could identify the biggest memory hogs and build them sequentially (post-parallel build?). I know ceph-dencoder was super-memory-expensive (it required more than 4G of RAM even on 32-bit architectures so we couldn't even build on 32-bit architectures anymore) and it might be a first candidate on the 'sequential' list. Do we have any other culprits?
What do you think?
I think this is a great idea. How do we do this in CMake? - Ken
participants (3)
-
Boris Ranto
-
Ken Dreyer
-
Willem Jan Withagen