cephadm deploy requirements
Hi all, I'm working to deploy a new Ceph cluster using cephadm and I have some doubts about the requirements: * how should I setup container networking on OSD nodes? I see that it is advised to have two interfaces, one in the public network and one in the cluster network (for OSD-only communication), do I need to make changes from the default Docker/Podman bridge networking? * Are there any particular steps to do on SELinux-enabled systems? Thanks in advance Francesco -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email:francesco.dinucci@na.infn.it
You don’t necessarily need to separate public from cluster network, it’s optional. There have been plenty of discussions on this list in which cases it makes sense. Or do you have the requirement to separate those networks? Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Hi all,
I'm working to deploy a new Ceph cluster using cephadm and I have some doubts about the requirements:
* how should I setup container networking on OSD nodes? I see that it is advised to have two interfaces, one in the public network and one in the cluster network (for OSD-only communication), do I need to make changes from the default Docker/Podman bridge networking? * Are there any particular steps to do on SELinux-enabled systems?
Thanks in advance
Francesco
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email:francesco.dinucci@na.infn.it _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, I get that it's optional but I'd like to separate the two, both for speed and a requirement of isolation -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci@na.infn.it On 10/24/25 19:35, Eugen Block wrote:
You don’t necessarily need to separate public from cluster network, it’s optional. There have been plenty of discussions on this list in which cases it makes sense. Or do you have the requirement to separate those networks?
Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Hi all,
I'm working to deploy a new Ceph cluster using cephadm and I have some doubts about the requirements:
* how should I setup container networking on OSD nodes? I see that it is advised to have two interfaces, one in the public network and one in the cluster network (for OSD-only communication), do I need to make changes from the default Docker/Podman bridge networking? * Are there any particular steps to do on SELinux-enabled systems?
Thanks in advance
Francesco
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email:francesco.dinucci@na.infn.it _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Alright then. There's usually nothing we specifically do wrt podman. When the network setup is final (VLANs, bonds, etc.), we specify the public and cluster network during bootstrap, and that's basically it. You can also separate the cluster_network afterwards, that's quite easily done. But just keep in mind, changing the public_network is a bit more complicated and requires careful planning and execution. Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Thanks,
I get that it's optional but I'd like to separate the two, both for speed and a requirement of isolation
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email: francesco.dinucci@na.infn.it
On 10/24/25 19:35, Eugen Block wrote:
You don’t necessarily need to separate public from cluster network, it’s optional. There have been plenty of discussions on this list in which cases it makes sense. Or do you have the requirement to separate those networks?
Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Hi all,
I'm working to deploy a new Ceph cluster using cephadm and I have some doubts about the requirements:
* how should I setup container networking on OSD nodes? I see that it is advised to have two interfaces, one in the public network and one in the cluster network (for OSD-only communication), do I need to make changes from the default Docker/Podman bridge networking? * Are there any particular steps to do on SELinux-enabled systems?
Thanks in advance
Francesco
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email:francesco.dinucci@na.infn.it _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, I'll try to plan it in advance. By the way, are there any consideration on Docker vs Podman for this use case? For the bootstrap part, I see that cephadm adds a key to the root user, is it possible to use a sudoer service user instead? And do I need to pre-configure all the hosts so that they can already ssh as root into each other, or is just the first node and from then on it's cephadm that takes care of it? Thanks again -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci@na.infn.it On 10/27/25 10:14, Eugen Block wrote:
Alright then. There's usually nothing we specifically do wrt podman. When the network setup is final (VLANs, bonds, etc.), we specify the public and cluster network during bootstrap, and that's basically it. You can also separate the cluster_network afterwards, that's quite easily done. But just keep in mind, changing the public_network is a bit more complicated and requires careful planning and execution.
Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Thanks,
I get that it's optional but I'd like to separate the two, both for speed and a requirement of isolation
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email: francesco.dinucci@na.infn.it
On 10/24/25 19:35, Eugen Block wrote:
You don’t necessarily need to separate public from cluster network, it’s optional. There have been plenty of discussions on this list in which cases it makes sense. Or do you have the requirement to separate those networks?
Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Hi all,
I'm working to deploy a new Ceph cluster using cephadm and I have some doubts about the requirements:
* how should I setup container networking on OSD nodes? I see that it is advised to have two interfaces, one in the public network and one in the cluster network (for OSD-only communication), do I need to make changes from the default Docker/Podman bridge networking? * Are there any particular steps to do on SELinux-enabled systems?
Thanks in advance
Francesco
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email:francesco.dinucci@na.infn.it _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Thanks,
I'll try to plan it in advance. By the way, are there any consideration on Docker vs Podman for this use case?
I think I read somewhere that podman is preferred, but it works well with docker, too. We usually use podman, and most of our customers do, but docker is also used, so it's your choice.
For the bootstrap part, I see that cephadm adds a key to the root user, is it possible to use a sudoer service user instead? And do I need to pre-configure all the hosts so that they can already ssh as root into each other, or is just the first node and from then on it's cephadm that takes care of it?
You can specify a different user during bootstrap [0]:
The --ssh-user *<user>* option makes it possible to designate which SSH user cephadm will use to connect to hosts. The associated SSH key will be added to ~*<user>*/.ssh/authorized_keys. The user that you designate with this option must have passwordless sudo access.
You need to preconfigure all hosts as described in the docs [1]. [0] https://docs.ceph.com/en/latest/cephadm/install/#further-information-about-c... [1] https://docs.ceph.com/en/latest/cephadm/host-management/#cephadm-adding-host...
Thanks again
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email: francesco.dinucci@na.infn.it
On 10/27/25 10:14, Eugen Block wrote:
Alright then. There's usually nothing we specifically do wrt podman. When the network setup is final (VLANs, bonds, etc.), we specify the public and cluster network during bootstrap, and that's basically it. You can also separate the cluster_network afterwards, that's quite easily done. But just keep in mind, changing the public_network is a bit more complicated and requires careful planning and execution.
Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Thanks,
I get that it's optional but I'd like to separate the two, both for speed and a requirement of isolation
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email: francesco.dinucci@na.infn.it
On 10/24/25 19:35, Eugen Block wrote:
You don’t necessarily need to separate public from cluster network, it’s optional. There have been plenty of discussions on this list in which cases it makes sense. Or do you have the requirement to separate those networks?
Zitat von Francesco Di Nucci <francesco.dinucci@na.infn.it>:
Hi all,
I'm working to deploy a new Ceph cluster using cephadm and I have some doubts about the requirements:
* how should I setup container networking on OSD nodes? I see that it is advised to have two interfaces, one in the public network and one in the cluster network (for OSD-only communication), do I need to make changes from the default Docker/Podman bridge networking? * Are there any particular steps to do on SELinux-enabled systems?
Thanks in advance
Francesco
-- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples
Email:francesco.dinucci@na.infn.it _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks,
I'll try to plan it in advance. By the way, are there any consideration on Docker vs Podman for this use case?
I think I read somewhere that podman is preferred, but it works well with docker, too. We usually use podman, and most of our customers do, but docker is also used, so it's your choice.
Podman lets you manage daemons to an extent with systemd commands; Docker I think doesn't. There were some compatibility issues in the past that are listed in the docs, but with recent Ceph and OS releases Podman should pretty much Just Work.
participants (3)
-
Anthony D'Atri
-
Eugen Block
-
Francesco Di Nucci