cephadm bootstrap on 3 network clusters
Hi, I am bootstrapping a ceph cluster using cephadm, and our cluster uses 3 networks. We have - 1 network as public network (10.X.X.0/24) (pub) - 1 network as cluster network (10.X.Y.0/24) (cluster) - 1 network for management (172.Z.Z.0/24) (mgmt) The nodes are reachable using SSH only on mgmt network. However, they are reachable for our services using pub network. I want my MONs to be bind to this pub network. But when I bootstrap my cluster, I set my MON IP and CLUSTER NETWORK, and then the bootstrap process tries to add my bootstrap node using the MON IP. And then fails because it cannot reach the node. If I apply proper spec after it works fine, but the bootstrap process did not finish properly. Is there an option to tell cephadm to not use MON IP but another one for accessing the node during the bootstrap? If I tell it --skip-prepare-host, it tries to connect to it anyway, and then fails. Thanks, Luis Domingues Proton AG
Hi, On 1/3/24 14:51, Luis Domingues wrote:
But when I bootstrap my cluster, I set my MON IP and CLUSTER NETWORK, and then the bootstrap process tries to add my bootstrap node using the MON IP.
IMHO the bootstrap process has to run directly on the first node. The MON IP is local to this node. It is used to determine the public network. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert, Thanks for your reply. I am bootstrapping from the first node that will become the first monitor, let's call it mon1. I get 1 monitor and 1 manager deployed. My issue is that mon1 cannot connect via SSH to itself using pub network, and bootstrap fail at the end when cephadm tries to add mon1 to the list of hosts. When I apply a spec afterwards with my list of hosts with their IPs where cephadm can reach them, it works fine. But that means that I need to create the client-keyring rule for _admin label manually as well. Luis Domingues Proton AG On Wednesday, 3 January 2024 at 16:00, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
On 1/3/24 14:51, Luis Domingues wrote:
But when I bootstrap my cluster, I set my MON IP and CLUSTER NETWORK, and then the bootstrap process tries to add my bootstrap node using the MON IP.
IMHO the bootstrap process has to run directly on the first node. The MON IP is local to this node. It is used to determine the public network.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Luis, On 1/3/24 16:12, Luis Domingues wrote:
My issue is that mon1 cannot connect via SSH to itself using pub network, and bootstrap fail at the end when cephadm tries to add mon1 to the list of hosts.
Why? The public network should not have any restrictions between the Ceph nodes. Same with the cluster network. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Why? The public network should not have any restrictions between the Ceph nodes. Same with the cluster network.
Internal policies and network rules. Luis Domingues Proton AG On Wednesday, 3 January 2024 at 16:15, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi Luis,
On 1/3/24 16:12, Luis Domingues wrote:
My issue is that mon1 cannot connect via SSH to itself using pub network, and bootstrap fail at the end when cephadm tries to add mon1 to the list of hosts.
Why? The public network should not have any restrictions between the Ceph nodes. Same with the cluster network.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, check routing table and default gateway and eventually fix it. use IP instead of dns name. I have more complicated situation :D I have more than 3 public networks and cluster networks… BR, Sebastian
On Jan 3, 2024, at 16:40, Luis Domingues <luis.domingues@proton.ch> wrote:
Why? The public network should not have any restrictions between the Ceph nodes. Same with the cluster network.
Internal policies and network rules.
Luis Domingues Proton AG
On Wednesday, 3 January 2024 at 16:15, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi Luis,
On 1/3/24 16:12, Luis Domingues wrote:
My issue is that mon1 cannot connect via SSH to itself using pub network, and bootstrap fail at the end when cephadm tries to add mon1 to the list of hosts.
Why? The public network should not have any restrictions between the Ceph nodes. Same with the cluster network.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Sebastian as you says "more than 3 public networks", did you manage Ceph daemons listening on multiple public interface ? I'm looking for such a possibility as daemons seams binded to one interface only but do not find any how-to. Thanks Patrick Le 03/01/2024 à 21:31, Sebastian a écrit :
Hi, check routing table and default gateway and eventually fix it. use IP instead of dns name.
I have more complicated situation :D I have more than 3 public networks and cluster networks…
BR, Sebastian
On Jan 3, 2024, at 16:40, Luis Domingues <luis.domingues@proton.ch> wrote:
Why? The public network should not have any restrictions between the Ceph nodes. Same with the cluster network. Internal policies and network rules.
Luis Domingues Proton AG
On Wednesday, 3 January 2024 at 16:15, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi Luis,
On 1/3/24 16:12, Luis Domingues wrote:
My issue is that mon1 cannot connect via SSH to itself using pub network, and bootstrap fail at the end when cephadm tries to add mon1 to the list of hosts.
Why? The public network should not have any restrictions between the Ceph nodes. Same with the cluster network.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Luis Domingues
-
Patrick Begou
-
Robert Sander
-
Sebastian