Dual network board setup info
Hi, I have a CephFS instance and I am also planning on also deploying an Object Storage interface. My servers have 2 network boards each. I would like to use the current local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other. I'm quite sure that Ceph has support for this kind of setup but I can't find how to do such a thing. I already have a "public network" setting that, AFAIU, sets the interface used by Ceph to talk to clients but I don't know if it defines CephFS MDS, RGW, or all of them. And I can't see how to make the different Ceph processes to talk to each other through the other interface. I found "mon_host" setting but should I change it to get what I want? And what else? Is there some example of such a setup that I could learn from? Or maybe someone would be kind enough to scratch a basic config to achieve this setup? Or maybe there is some documentation that deals with such a scenario that I haven't found. Well, I'm in search for more info, any kind. Thanks in advance for your help and attention, Rodrigo Severo
On 11/27/19 8:04 PM, Rodrigo Severo - Fábrica wrote:
I have a CephFS instance and I am also planning on also deploying an Object Storage interface.
My servers have 2 network boards each. I would like to use the current local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other.
I'm quite sure that Ceph has support for this kind of setup but I can't find how to do such a thing.
I already have a "public network" setting that, AFAIU, sets the interface used by Ceph to talk to clients but I don't know if it defines CephFS MDS, RGW, or all of them.
And I can't see how to make the different Ceph processes to talk to each other through the other interface. I found "mon_host" setting but should I change it to get what I want? And what else?
Is there some example of such a setup that I could learn from?
Or maybe someone would be kind enough to scratch a basic config to achieve this setup?
Or maybe there is some documentation that deals with such a scenario that I haven't found.
Well, I'm in search for more info, any kind.
Ceph support `cluster network` and `public network` options. Only OSD work with cluster network. Any other is a OSD clients - public network. k
Em qui., 28 de nov. de 2019 às 00:34, Konstantin Shalygin <k0ste@k0ste.ru> escreveu:
My servers have 2 network boards each. I would like to use the current local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other.
Ceph support `cluster network` and `public network` options. Only OSD work with cluster network. Any other is a OSD clients - public network.
Great. How do I migrate from my current single network board to a dual one? Can I migrate servers one by one to the dual network setup or do I have to stop the whole ceph cluster and restart it all already on the dual setup? Regards, Rodrigo Severo
On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
Em qui., 28 de nov. de 2019 às 00:34, Konstantin Shalygin <k0ste@k0ste.ru> escreveu:
My servers have 2 network boards each. I would like to use the current local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other.
Ceph support `cluster network` and `public network` options. Only OSD work with cluster network. Any other is a OSD clients - public network.
Great. How do I migrate from my current single network board to a dual one?
Can I migrate servers one by one to the dual network setup or do I have to stop the whole ceph cluster and restart it all already on the dual setup?
Set the cluster_network in the ceph.conf and restart the OSDs one by one. However, why not bond and use the additional bandwidth for both client and cluster? I don't see a major benefit in having separate cluster+public network with Ceph. Seems like an old habit from back in the days. LACP and one IP per node works just fine with Ceph. Wido
Regards,
Rodrigo Severo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Em qui., 28 de nov. de 2019 às 13:39, Wido den Hollander <wido@42on.com> escreveu:
On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
Em qui., 28 de nov. de 2019 às 00:34, Konstantin Shalygin <k0ste@k0ste.ru> escreveu:
My servers have 2 network boards each. I would like to use the current local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other.
Ceph support `cluster network` and `public network` options. Only OSD work with cluster network. Any other is a OSD clients - public network.
Great. How do I migrate from my current single network board to a dual one?
Can I migrate servers one by one to the dual network setup or do I have to stop the whole ceph cluster and restart it all already on the dual setup?
Set the cluster_network in the ceph.conf and restart the OSDs one by one.
Just tried that. The first OSD that I'm trying to restart won't came up again. It presents the following messages which aren't that useful to me: Nov 28 18:26:33 a2-df systemd[1]: ceph-osd@1.service: Start request repeated too quickly. Nov 28 18:26:33 a2-df systemd[1]: ceph-osd@1.service: Failed with result 'exit-code'. Nov 28 18:26:33 a2-df systemd[1]: Failed to start Ceph object storage daemon osd.1. -- Subject: Unit ceph-osd@1.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit ceph-osd@1.service has failed. -- -- The result is RESULT. I also see the following error messages: Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230 7f487dee1700 -1 set_mon_vals failed to set cluster_network = 192.168.111.0/24: Configuration option 'cluster_network' may not be modified at runtime Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230 7f487dee1700 -1 set_mon_vals failed to set public_network = 192.168.109.0/24: Configuration option 'public_network' may not be modified at runtime There are 2 things that I don't understand in these messages: 1. Why is it mentioning configuration option 'public_network' in these error messages as I didn't change the public_network config, I only added a cluster_network one? 2. Why are there messages from ceph-mon when I'm trying to restart ceph-osd? And the most important issue: how can I get my osd back online? Regards, Rodrigo
Em qui., 28 de nov. de 2019 às 18:32, Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> escreveu:
Em qui., 28 de nov. de 2019 às 13:39, Wido den Hollander <wido@42on.com> escreveu:
On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
Em qui., 28 de nov. de 2019 às 00:34, Konstantin Shalygin <k0ste@k0ste.ru> escreveu:
My servers have 2 network boards each. I would like to use the current local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other.
Ceph support `cluster network` and `public network` options. Only OSD work with cluster network. Any other is a OSD clients - public network.
Great. How do I migrate from my current single network board to a dual one?
Can I migrate servers one by one to the dual network setup or do I have to stop the whole ceph cluster and restart it all already on the dual setup?
Set the cluster_network in the ceph.conf and restart the OSDs one by one.
Just tried that. The first OSD that I'm trying to restart won't came up again.
Does anybody has any suggestion on how to get my ceph fs back to healthy status? I'm even considering stop it all and restarting but I'm afraid it won't come back up with the new config. Ideias? Suggestions? Rodrigo It presents the following messages which aren't that useful
to me:
Nov 28 18:26:33 a2-df systemd[1]: ceph-osd@1.service: Start request repeated too quickly. Nov 28 18:26:33 a2-df systemd[1]: ceph-osd@1.service: Failed with result 'exit-code'. Nov 28 18:26:33 a2-df systemd[1]: Failed to start Ceph object storage daemon osd.1. -- Subject: Unit ceph-osd@1.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit ceph-osd@1.service has failed. -- -- The result is RESULT.
I also see the following error messages: Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230 7f487dee1700 -1 set_mon_vals failed to set cluster_network = 192.168.111.0/24: Configuration option 'cluster_network' may not be modified at runtime Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230 7f487dee1700 -1 set_mon_vals failed to set public_network = 192.168.109.0/24: Configuration option 'public_network' may not be modified at runtime
There are 2 things that I don't understand in these messages:
1. Why is it mentioning configuration option 'public_network' in these error messages as I didn't change the public_network config, I only added a cluster_network one?
2. Why are there messages from ceph-mon when I'm trying to restart ceph-osd?
And the most important issue: how can I get my osd back online?
Regards,
Rodrigo
Hi Rodrigo, The fact that you're getting logs from mon and the function name set_mon_vals suggests that you made use of mon provided centralized config, like `ceph config set osd cluster_network x.x.x.x` but it seems cluster_network and public_network cannot be centralized and should be set in ceph.conf. (Just an idea not really an expert). If that's the case you should be able to `ceph config rm osd cluster_network` to get your cluster healthy again. Then you could try setting in config file directly. HTH On Fri, Nov 29, 2019 at 4:55 PM Rodrigo Severo - Fábrica < rodrigo@fabricadeideias.com> wrote:
Em qui., 28 de nov. de 2019 às 18:32, Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> escreveu:
Em qui., 28 de nov. de 2019 às 13:39, Wido den Hollander <wido@42on.com> escreveu:
On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
Em qui., 28 de nov. de 2019 às 00:34, Konstantin Shalygin <k0ste@k0ste.ru> escreveu:
My servers have 2 network boards each. I would like to use the
current
local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other.
Ceph support `cluster network` and `public network` options. Only OSD work with cluster network. Any other is a OSD clients - public network.
Great. How do I migrate from my current single network board to a dual one?
Can I migrate servers one by one to the dual network setup or do I have to stop the whole ceph cluster and restart it all already on the dual setup?
Set the cluster_network in the ceph.conf and restart the OSDs one by one.
Just tried that. The first OSD that I'm trying to restart won't came up again.
Does anybody has any suggestion on how to get my ceph fs back to healthy status?
I'm even considering stop it all and restarting but I'm afraid it won't come back up with the new config.
Ideias? Suggestions?
Rodrigo
It presents the following messages which aren't that useful
to me:
Nov 28 18:26:33 a2-df systemd[1]: ceph-osd@1.service: Start request repeated too quickly. Nov 28 18:26:33 a2-df systemd[1]: ceph-osd@1.service: Failed with result 'exit-code'. Nov 28 18:26:33 a2-df systemd[1]: Failed to start Ceph object storage daemon osd.1. -- Subject: Unit ceph-osd@1.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit ceph-osd@1.service has failed. -- -- The result is RESULT.
I also see the following error messages: Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230 7f487dee1700 -1 set_mon_vals failed to set cluster_network = 192.168.111.0/24: Configuration option 'cluster_network' may not be modified at runtime Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230 7f487dee1700 -1 set_mon_vals failed to set public_network = 192.168.109.0/24: Configuration option 'public_network' may not be modified at runtime
There are 2 things that I don't understand in these messages:
1. Why is it mentioning configuration option 'public_network' in these error messages as I didn't change the public_network config, I only added a cluster_network one?
2. Why are there messages from ceph-mon when I'm trying to restart ceph-osd?
And the most important issue: how can I get my osd back online?
Regards,
Rodrigo
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- erdem agaoglu
Em dom., 1 de dez. de 2019 às 09:05, Erdem Agaoglu <erdem.agaoglu@gmail.com> escreveu:
Hi Rodrigo,
The fact that you're getting logs from mon and the function name set_mon_vals suggests that you made use of mon provided centralized config, like `ceph config set osd cluster_network x.x.x.x` but it seems cluster_network and public_network cannot be centralized and should be set in ceph.conf. (Just an idea not really an expert).
If that's the case you should be able to `ceph config rm osd cluster_network` to get your cluster healthy again. Then you could try setting in config file directly.
Unfortunately this wasn't the case. I say so because to get my cluster healthy again I just commented out the cluster_network settings in my ceph.conf files. After that I could restart my OSD with any mon errors. I did that a few days ago and forgot to post this info here. I'm sorry. My immediate issue was solved but my main issue remains: how can I migrate my current single network board setup to a dual one? As far as my experience goes, setting cluster_network in the global section of my ceph.conf files isn' t the proper way. Maybe cluster_network shouldn' t be set in the global section. Is that it? In which section should it be set instead? Regards, Rodrigo Severo
participants (4)
-
Erdem Agaoglu
-
Konstantin Shalygin
-
Rodrigo Severo - Fábrica
-
Wido den Hollander