I am setting up my first cluster of 9-nodes each with 8x 20T HDDs and 2x 2T NVMes. I plan to partition the NVMes into 5x 300G so that one partition can be used by cephfs_metadata (SSD only), while the other 4x partitions will be paired as db devices for 4x of the HDDs. The cluster will only be used for cephfs and data will only be stored on its EC 4+2 HDD-only pool. Just a simple and large file server, so performance isn't a primary concern. Each node has 2x 10Gb network connections (one public, one cluster). All disks encrypted (encrypted=true on OSD creation on top of LVM). I am new to Ceph, and was hoping to get advice on if these initial configurations are sensible or if I am doing something incorrect that might have a negative impact in the future: https://pastebin.com/yiQS4g3q Thanks!
Am 6/23/25 um 22:37 schrieb Ryan Sleeth:
I am setting up my first cluster of 9-nodes each with 8x 20T HDDs and 2x 2T NVMes. I plan to partition the NVMes into 5x 300G so that one partition can be used by cephfs_metadata (SSD only), while the other 4x partitions will be paired as db devices for 4x of the HDDs.
Check if the NVMe devices support multiple namespaces. https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html This way the kernel sees two block devices for each NVMe controller. This makes it easier with the cephadm orchestrator to deploy NVMe OSDs on the one namespace block device and the RocksDB volumes on the other. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi and welcome to ceph, On 23.06.25 22:37, Ryan Sleeth wrote:
I am setting up my first cluster of 9-nodes each with 8x 20T HDDs and 2x 2T NVMes. I plan to partition the NVMes into 5x 300G so that one partition can be used by cephfs_metadata (SSD only), while the other 4x partitions will be paired as db devices for 4x of the HDDs. The cluster will only be used for cephfs and data will only be stored on its EC 4+2 HDD-only pool. Just a simple and large file server, so performance isn't a primary concern. Each node has 2x 10Gb network connections (one public, one cluster). All disks encrypted (encrypted=true on OSD creation on top of LVM).
I would skip the cluster network and use a bond with both interfaces as public network. The benefits of a separate cluster network are rather small, especially in your use case. I prefer to have a reliable network connection to clients (given switches that support stacking and/or MLAG). Best regards , Burkhard
Hi, i agree to configure both interfaces as a bond. from my experience, i see the following advantages for a separate public and cluster network on the bond: the isolation of public network and cluster network traffic makes it easier to monitor client traffic and inter osd traffic. and if it is necessary later, you can also prioritise or limit client traffic via the separate interface. It's also helpful to debug and analyse issues in the ceph cluster. Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Di., 24. Juni 2025 um 10:25 Uhr schrieb Burkhard Linke < Burkhard.Linke@computational.bio.uni-giessen.de>:
Hi and welcome to ceph,
On 23.06.25 22:37, Ryan Sleeth wrote:
I am setting up my first cluster of 9-nodes each with 8x 20T HDDs and 2x 2T NVMes. I plan to partition the NVMes into 5x 300G so that one partition can be used by cephfs_metadata (SSD only), while the other 4x partitions will be paired as db devices for 4x of the HDDs. The cluster will only be used for cephfs and data will only be stored on its EC 4+2 HDD-only pool. Just a simple and large file server, so performance isn't a primary concern. Each node has 2x 10Gb network connections (one public, one cluster). All disks encrypted (encrypted=true on OSD creation on top of LVM).
I would skip the cluster network and use a bond with both interfaces as public network. The benefits of a separate cluster network are rather small, especially in your use case. I prefer to have a reliable network connection to clients (given switches that support stacking and/or MLAG).
Best regards ,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/25/25 09:19, Joachim Kraftmayer wrote:
Hi, i agree to configure both interfaces as a bond. from my experience, i see the following advantages for a separate public and cluster network on the bond:
the isolation of public network and cluster network traffic makes it easier to monitor client traffic and inter osd traffic. and if it is necessary later, you can also prioritise or limit client traffic via the separate interface. It's also helpful to debug and analyse issues in the ceph cluster.
I have a different experience. When one of the nodes had a cluster network down, but a working public network, you get hard to troubleshoot issues. Especially as a newbie in Ceph (this was a test cluster). You will see slow operations, OSDs on the storage nodes that flag their peer down, the OSD daemon itself will respond to those messages that it's still running, etc.. If you look at ceph -w you will get conflicting information. For an experienced operator this will not be too hard to troubleshoot, but for less experienced ones it will be. This was also in a time where OSD "heartbeat" check were not yet a thing and also before alertmanager et al. Ideally you don't want to have "gray" failures like this and either want your ceph node to be "UP" or "DOWN", but not something in between. A single public interface will give you that. Gr. Stefan
i agree to configure both interfaces as a bond. from my experience, i see the following advantages for a separate public and cluster network on the bond: the isolation of public network and cluster network traffic makes it easier to monitor client traffic and inter osd traffic. and if it is necessary later, you can also prioritise or limit client traffic via the separate interface. It's also helpful to debug and analyse issues in the ceph cluster.
You aren’t wrong. This however can complicate network setup, including sideband BMC interfaces and potentially MTU mismatch. With modern releases and networking, FWIW, I haven’t seen DoS issues like we used to between client and replication traffic. ymmv.
I have a different experience. When one of the nodes had a cluster network down, but a working public network, you get hard to troubleshoot issues. Especially as a newbie in Ceph (this was a test cluster). You will see slow operations, OSDs on the storage nodes that flag their peer down, the OSD daemon itself will respond to those messages that it's still running, etc..
Exactly. “I’m not dead yet!” Flap flap flap with performance impact. This is called out in the docs.
Ideally you don't want to have "gray" failures like this and either want your ceph node to be "UP" or "DOWN", but not something in between. A single public interface will give you that.
Indeed. And when there are only two network interfaces available, that redundancy (assuming active/active and appropriate xmit_hash_policy) will limit disruption when there’s a layer 1 issue, if the two links are to different switches.
participants (7)
-
Anthony D'Atri
-
Anthony D'Atri
-
Burkhard Linke
-
Joachim Kraftmayer
-
Robert Sander
-
Ryan Sleeth
-
Stefan Kooman