Hi Ceph Users, I need to understand the Ceph network design and the way it communicates with different partners. Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x: ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24 After this I only see this as paramter in the GUI screen, under host / ceph / configuration in the section Configuration Database Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host). Is this correct? I thought, all storage traffic needs to be isolated in my storage network 10.10.x.x - no IPs in the mgmt network for ceph. Or do I have the wrong understanding of the ceph network design? Best regards, Enjoy your weekend, Beste Grüße, Schönes Wochenende, Alexander Concat AG IT Solutions | Berliner Ring 127 - 129 | 64625 Bensheim | www.concat.de alexander.leutz@concat.de | Tel.: +49 6251 7026-143 | Mobil: +49 151 18856009 | Fax: +49 6251 7026-356 Hauptsitz Bensheim a. d. Bergstraße | Vorstand Olaf von Heyer Aufsichtsratsvorsitzender Tim Patronik | Amtsgericht Darmstadt (HRB 25326)
Hi, Am 14.11.25 um 9:44 AM schrieb Alexander Leutz:
Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x:
ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24
In your case I would not setup a separate cluster network. It is optional and only recommended if you have a network that is at least twice as fast as the public network.
After this I only see this as paramter in the GUI screen, under host / ceph / configuration in the section Configuration Database Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host).
Change in the config db do not get written to the ceph.conf file. You can remove this setting from the ceph.conf file by using an editor as it is now in the config db. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
On Nov 14, 2025, at 4:37 AM, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 14.11.25 um 9:44 AM schrieb Alexander Leutz:
Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x: ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24
In your case I would not setup a separate cluster network.
Indeed. To be clear, Alexander, do you have those two 10GE ports bonded? If so, then I agree with Herr Sander that there is no benefit to defining the cluster_network, and indeed you may be confusing code by doing so. The cluster network should only be defined if it's *different* from the public network. The public network is how clients communicate with the Ceph cluster. If there's a cluster_network, it is only used for internal replication and heartbeating within Ceph.
It is optional and only recommended if you have a network that is at least twice as fast as the public network.
Well, say someone has two dual-port 10GE NICs in each system. They could either bond all four together (which I *think* works but haven't tried) for a public network, or bond one port on each NIC together for the public, and one on each for the private network. I do usually recommend that people not bother with a cluster network if they're using 25GE or better, especially with very dense nodes.
After this I only see this as paramter in the GUI screen, under host / ceph / configuration in the section Configuration Database Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host).
Change in the config db do not get written to the ceph.conf file.
I think he's writing about the Dashboard, not ceph.conf
You can remove this setting from the ceph.conf file by using an editor as it is now in the config db.
If it's in ceph.conf, then absolutely. All you need in there is the mon information, maybe the fsid.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Anthony and Robert, how can you monitor the OSD client and replication traffic separately, e.g. in grafana? It has often helped me with error analysis. Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Fr., 14. Nov. 2025 um 14:13 Uhr schrieb Anthony D'Atri < aad@dreamsnake.net>:
On Nov 14, 2025, at 4:37 AM, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 14.11.25 um 9:44 AM schrieb Alexander Leutz:
Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x: ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24
In your case I would not setup a separate cluster network.
Indeed. To be clear, Alexander, do you have those two 10GE ports bonded? If so, then I agree with Herr Sander that there is no benefit to defining the cluster_network, and indeed you may be confusing code by doing so. The cluster network should only be defined if it's *different* from the public network. The public network is how clients communicate with the Ceph cluster. If there's a cluster_network, it is only used for internal replication and heartbeating within Ceph.
It is optional and only recommended if you have a network that is at least twice as fast as the public network.
Well, say someone has two dual-port 10GE NICs in each system. They could either bond all four together (which I *think* works but haven't tried) for a public network, or bond one port on each NIC together for the public, and one on each for the private network.
I do usually recommend that people not bother with a cluster network if they're using 25GE or better, especially with very dense nodes.
After this I only see this as paramter in the GUI screen, under host /
ceph / configuration in the section Configuration Database
Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host).
Change in the config db do not get written to the ceph.conf file.
I think he's writing about the Dashboard, not ceph.conf
You can remove this setting from the ceph.conf file by using an editor
as it is now in the config db.
If it's in ceph.conf, then absolutely. All you need in there is the mon information, maybe the fsid.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
One strategy is to use VLANs. It's been a while since I've had to measure client and replication traffic at host granularity, but you make a sound point.
On Nov 14, 2025, at 9:23 AM, Joachim Kraftmayer <joachim.kraftmayer@clyso.com> wrote:
Hi,
Anthony and Robert, how can you monitor the OSD client and replication traffic separately, e.g. in grafana? It has often helped me with error analysis. Joachim
joachim.kraftmayer@clyso.com <mailto:joachim.kraftmayer@clyso.com> www.clyso.com <http://www.clyso.com/> Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677
Am Fr., 14. Nov. 2025 um 14:13 Uhr schrieb Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>>:
On Nov 14, 2025, at 4:37 AM, Robert Sander <r.sander@heinlein-support.de <mailto:r.sander@heinlein-support.de>> wrote:
Hi,
Am 14.11.25 um 9:44 AM schrieb Alexander Leutz:
Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x: ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24
In your case I would not setup a separate cluster network.
Indeed. To be clear, Alexander, do you have those two 10GE ports bonded? If so, then I agree with Herr Sander that there is no benefit to defining the cluster_network, and indeed you may be confusing code by doing so. The cluster network should only be defined if it's *different* from the public network. The public network is how clients communicate with the Ceph cluster. If there's a cluster_network, it is only used for internal replication and heartbeating within Ceph.
It is optional and only recommended if you have a network that is at least twice as fast as the public network.
Well, say someone has two dual-port 10GE NICs in each system. They could either bond all four together (which I *think* works but haven't tried) for a public network, or bond one port on each NIC together for the public, and one on each for the private network.
I do usually recommend that people not bother with a cluster network if they're using 25GE or better, especially with very dense nodes.
After this I only see this as paramter in the GUI screen, under host / ceph / configuration in the section Configuration Database Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host).
Change in the config db do not get written to the ceph.conf file.
I think he's writing about the Dashboard, not ceph.conf
You can remove this setting from the ceph.conf file by using an editor as it is now in the config db.
If it's in ceph.conf, then absolutely. All you need in there is the mon information, maybe the fsid.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de <https://www.heinlein-support.de/>
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Okay, then perhaps I misunderstood the core of the question earlier. So I would always suggest considering the advantages of a logical separation of the two networks. Joachim Kraftmayer CEO +49 89 2152527-21 <https://call.ctrlq.org/+49%2089%202152527-21> joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Fr., 14. Nov. 2025 um 15:26 Uhr schrieb Anthony D'Atri < aad@dreamsnake.net>:
One strategy is to use VLANs. It's been a while since I've had to measure client and replication traffic at host granularity, but you make a sound point.
On Nov 14, 2025, at 9:23 AM, Joachim Kraftmayer < joachim.kraftmayer@clyso.com> wrote:
Hi,
Anthony and Robert, how can you monitor the OSD client and replication traffic separately, e.g. in grafana? It has often helped me with error analysis. Joachim
joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677
Am Fr., 14. Nov. 2025 um 14:13 Uhr schrieb Anthony D'Atri < aad@dreamsnake.net>:
On Nov 14, 2025, at 4:37 AM, Robert Sander < r.sander@heinlein-support.de> wrote:
Hi,
Am 14.11.25 um 9:44 AM schrieb Alexander Leutz:
Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x: ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24
In your case I would not setup a separate cluster network.
Indeed. To be clear, Alexander, do you have those two 10GE ports bonded? If so, then I agree with Herr Sander that there is no benefit to defining the cluster_network, and indeed you may be confusing code by doing so. The cluster network should only be defined if it's *different* from the public network. The public network is how clients communicate with the Ceph cluster. If there's a cluster_network, it is only used for internal replication and heartbeating within Ceph.
It is optional and only recommended if you have a network that is at least twice as fast as the public network.
Well, say someone has two dual-port 10GE NICs in each system. They could either bond all four together (which I *think* works but haven't tried) for a public network, or bond one port on each NIC together for the public, and one on each for the private network.
I do usually recommend that people not bother with a cluster network if they're using 25GE or better, especially with very dense nodes.
After this I only see this as paramter in the GUI screen, under host /
ceph / configuration in the section Configuration Database
Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host).
Change in the config db do not get written to the ceph.conf file.
I think he's writing about the Dashboard, not ceph.conf
You can remove this setting from the ceph.conf file by using an editor
as it is now in the config db.
If it's in ceph.conf, then absolutely. All you need in there is the mon information, maybe the fsid.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Alexander Leutz
-
Anthony D'Atri
-
Joachim Kraftmayer
-
Robert Sander