[ceph-users] Re: Ceph deployment for HPC and research workflows

8 May 2026


      Glad to hear that helped!  OSNexus are great guys.


Thanks,

Mark


On 5/8/26 6:50 AM, Max Breitmeyer wrote:
...
Hey Mark,
Initially we had it set to use the full 192GB, but after an issue 
where starting up the MDS service was going straight to OOM, we 
brought it down to 64GB. This has prevented any further crashes. We 
have a support vendor that we're working with (OSNexus) at the moment.
On Wed, May 6, 2026 at 10:40 AM Mark Nelson via ceph-users 
<ceph-users@ceph.io> wrote:
Hi Max,
Probably the first order of business is to understand the nature
    of the
    crashes you are seeing. Just as a guess, if you have thousands of
    jobs
    being run at the same time, you may have a lot of memory pressure
    on the
    MDSses. If the crashes are just the MDSes going OOM, you might
    want to
    verify how much of the 192GB of RAM on each node the MDSes are
    allowed
    to use for cache.
Beyond that, it might take some digging to understand the
    specifics of
    your issues. There are other institutions that do use CephFS for
    HPC use
    cases though. I gave a talk at the IO500 BoF at SC24 in collaboration
    with GWDG in Germany regarding their Ceph deployment.
Shameless plug: Clyso does offer academic support packages for
    upstream
    Ceph if that is of interest to you.
Thanks,
    Mark
On 5/6/26 6:54 AM, Max Breitmeyer via ceph-users wrote:
    > Hello all,
    > At UMBC we were given a grant to run a ceph cluster as our
    primary research
    > storage for an HPC facility. The ceph cluster consists of the
    following
    > hardware:
    > Mon” nodes x3:
    > • 2x 25Gb interfaces
    > • 192GB RAM
    > • Storage:
    > -- 0 (none) OSDs/drives
    > “HDD” nodes x16:
    > • 2x 25Gb interfaces
    > • 192GB RAM
    > • Storage:
    > -- 12 OSDs w/ 20TB HDDs
    > -- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives
    > “NVMe” nodes x3:
    > • 2x 100Gb interfaces
    > • 384GB RAM
    > • Storage:
    > -- 16x OSDs w/ 8TB NVMe drives
    > “MDS” nodes x3:
    > • 2x 100Gb interfaces
    > • 192GB RAM
    > • Storage:
    > -- 8x OSDs w/ 1TB NVMe drives
    >
    >
    > In the six months since moving from a different storage solution
    (isilon),
    > we've had multiple crashes on the system that have completely
    taken down
    > our system as a result of multiple thousands of jobs being run
    at the same
    > time, causing an overload on the active MDS nodes. Has the use
    of ceph as a
    > primary research storage been used in an HPC with many dozens of
    different
    > workflows and filetypes being used at once? Or is a better to
    design to
    > come up with a tiered approach in combination with a second storage
    > solution?
    >
    -- 
    Best Regards,
    Mark Nelson
    Co-Founder and Head of R&D
Clyso Inc.
    p: +49 89 21552391 12 | a: North Vancouver, B.C.
    w: https://clyso.com | e: mark.nelson@clyso.com
We are hiring: https://www.clyso.com/jobs/
    _______________________________________________
    ceph-users mailing list -- ceph-users@ceph.io
    To unsubscribe send an email to ceph-users-leave@ceph.io
-- 
V/R,
Maxwell Breitmeyer
UMBC HPCF Specialist
Graduate Student
(443) 835-8250
-- 
Best Regards,
Mark Nelson
Co-Founder and Head of R&D

Clyso Inc.
p: +49 89 21552391 12 | a: North Vancouver, B.C.
w: https://clyso.com | e: mark.nelson@clyso.com

We are hiring: https://www.clyso.com/jobs/