27 Nov
2019
27 Nov
'19
2:04 p.m.
Hi,
I have a CephFS instance and I am also planning on also deploying an
Object Storage interface.
My servers have 2 network boards each. I would like to use the current
local one to talk to Cephs clients (both CephFS and Object Storage)
and use the second one to all Cephs processes to talk one to the
other.
I'm quite sure that Ceph has support for this kind of setup but I
can't find how to do such a thing.
I already have a "public network" setting that, AFAIU, sets the
interface used by Ceph to talk to clients but I don't know if it
defines CephFS MDS, RGW, or all of them.
And I can't see how to make the different Ceph processes to talk to
each other through the other interface. I found "mon_host" setting but
should I change it to get what I want? And what else?
Is there some example of such a setup that I could learn from?
Or maybe someone would be kind enough to scratch a basic config to
achieve this setup?
Or maybe there is some documentation that deals with such a scenario
that I haven't found.
Well, I'm in search for more info, any kind.
Thanks in advance for your help and attention,
Rodrigo Severo
28 Nov
28 Nov
4:34 a.m.
On 11/27/19 8:04 PM, Rodrigo Severo - Fábrica wrote:
I have a CephFS instance and I am also planning on
also deploying an
Object Storage interface.
My servers have 2 network boards each. I would like to use the current
local one to talk to Cephs clients (both CephFS and Object Storage)
and use the second one to all Cephs processes to talk one to the
other.
I'm quite sure that Ceph has support for this kind of setup but I
can't find how to do such a thing.
I already have a "public network" setting that, AFAIU, sets the
interface used by Ceph to talk to clients but I don't know if it
defines CephFS MDS, RGW, or all of them.
And I can't see how to make the different Ceph processes to talk to
each other through the other interface. I found "mon_host" setting but
should I change it to get what I want? And what else?
Is there some example of such a setup that I could learn from?
Or maybe someone would be kind enough to scratch a basic config to
achieve this setup?
Or maybe there is some documentation that deals with such a scenario
that I haven't found.
Well, I'm in search for more info, any kind.
Ceph support `cluster network` and `public network` options. Only OSD
work with cluster network. Any other is a OSD clients - public network.
k
5:23 p.m.
Em qui., 28 de nov. de 2019 às 00:34, Konstantin Shalygin
<k0ste(a)k0ste.ru> escreveu:
Great. How do I migrate from my current single network board to a dual one?
Can I migrate servers one by one to the dual network setup or do I
have to stop the whole ceph cluster and restart it all already on the
dual setup?
Regards,
Rodrigo Severo
My servers have 2 network boards each. I would
like to use the current
local one to talk to Cephs clients (both CephFS and Object Storage)
and use the second one to all Cephs processes to talk one to the
other.
Ceph support `cluster network` and `public network` options. Only OSD
work with cluster network. Any other is a OSD clients - public network. 
5:39 p.m.
On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
Em qui., 28 de nov. de 2019 às 00:34, Konstantin
Shalygin
<k0ste(a)k0ste.ru> escreveu:
Great. How do I migrate from my current single network board to a dual one?
Can I migrate servers one by one to the dual network setup or do I
have to stop the whole ceph cluster and restart it all already on the
dual setup?
Set the cluster_network in the ceph.conf and restart the OSDs one by one.
However, why not bond and use the additional bandwidth for both client
and cluster?
I don't see a major benefit in having separate cluster+public network
with Ceph. Seems like an old habit from back in the days.
LACP and one IP per node works just fine with Ceph.
Wido
>
>
> Regards,
>
> Rodrigo Severo
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
My servers have 2 network boards each. I would
like to use the current
local one to talk to Cephs clients (both CephFS and Object Storage)
and use the second one to all Cephs processes to talk one to the
other.
Ceph support `cluster network` and `public network` options. Only OSD
work with cluster network. Any other is a OSD clients - public network.
10:32 p.m.
Em qui., 28 de nov. de 2019 às 13:39, Wido den Hollander
<wido(a)42on.com> escreveu:
On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
Just tried that. The first OSD that I'm trying to restart won't came
up again. It presents the following messages which aren't that useful
to me:
Nov 28 18:26:33 a2-df systemd[1]: ceph-osd(a)1.service: Start request
repeated too quickly.
Nov 28 18:26:33 a2-df systemd[1]: ceph-osd(a)1.service: Failed with
result 'exit-code'.
Nov 28 18:26:33 a2-df systemd[1]: Failed to start Ceph object storage
daemon osd.1.
-- Subject: Unit ceph-osd(a)1.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit ceph-osd(a)1.service has failed.
--
-- The result is RESULT.
I also see the following error messages:
Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230
7f487dee1700 -1 set_mon_vals failed to set cluster_network =
192.168.111.0/24: Configuration option 'cluster_network' may not be
modified at runtime
Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230
7f487dee1700 -1 set_mon_vals failed to set public_network =
192.168.109.0/24: Configuration option 'public_network' may not be
modified at runtime
There are 2 things that I don't understand in these messages:
1. Why is it mentioning configuration option 'public_network' in these
error messages as I didn't change the public_network config, I only
added a cluster_network one?
2. Why are there messages from ceph-mon when I'm trying to restart ceph-osd?
And the most important issue: how can I get my osd back online?
Regards,
Rodrigo
Em qui., 28 de nov. de 2019 às 00:34, Konstantin
Shalygin
<k0ste(a)k0ste.ru> escreveu:
Great. How do I migrate from my current single network board to a dual one?
Can I migrate servers one by one to the dual network setup or do I
have to stop the whole ceph cluster and restart it all already on the
dual setup?
Set the cluster_network in the ceph.conf and restart the OSDs one by one. My servers have 2 network boards each. I would
like to use the current
local one to talk to Cephs clients (both CephFS and Object Storage)
and use the second one to all Cephs processes to talk one to the
other.
Ceph support `cluster network` and `public network` options. Only OSD
work with cluster network. Any other is a OSD clients - public network.
29 Nov
29 Nov
2:54 p.m.
Em qui., 28 de nov. de 2019 às 18:32, Rodrigo Severo - Fábrica
<rodrigo(a)fabricadeideias.com> escreveu:
Em qui., 28 de nov. de 2019 às 13:39, Wido den Hollander
<wido(a)42on.com> escreveu:
Does anybody has any suggestion on how to get my ceph fs back to healthy status?
I'm even considering stop it all and restarting but I'm afraid it
won't come back up with the new config.
Ideias? Suggestions?
Rodrigo
It presents the following messages which aren't that useful
> to me:
>
> Nov 28 18:26:33 a2-df systemd[1]: ceph-osd(a)1.service: Start request
> repeated too quickly.
> Nov 28 18:26:33 a2-df systemd[1]: ceph-osd(a)1.service: Failed with
> result 'exit-code'.
> Nov 28 18:26:33 a2-df systemd[1]: Failed to start Ceph object storage
> daemon osd.1.
> -- Subject: Unit ceph-osd(a)1.service has failed
> -- Defined-By: systemd
> -- Support: http://www.ubuntu.com/support
> --
> -- Unit ceph-osd(a)1.service has failed.
> --
> -- The result is RESULT.
>
> I also see the following error messages:
> Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230
> 7f487dee1700 -1 set_mon_vals failed to set cluster_network =
> 192.168.111.0/24: Configuration option 'cluster_network' may not be
> modified at runtime
> Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230
> 7f487dee1700 -1 set_mon_vals failed to set public_network =
> 192.168.109.0/24: Configuration option 'public_network' may not be
> modified at runtime
>
> There are 2 things that I don't understand in these messages:
>
> 1. Why is it mentioning configuration option 'public_network' in these
> error messages as I didn't change the public_network config, I only
> added a cluster_network one?
>
> 2. Why are there messages from ceph-mon when I'm trying to restart ceph-osd?
>
> And the most important issue: how can I get my osd back online?
>
>
> Regards,
>
> Rodrigo
On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
Just tried that. The first OSD that I'm trying to restart won't came
up again. Em qui., 28 de nov. de 2019 às 00:34, Konstantin
Shalygin
<k0ste(a)k0ste.ru> escreveu:
Set the cluster_network in the ceph.conf and restart the OSDs one by one.
> My servers have 2 network boards each. I would like to use the current
> local one to talk to Cephs clients (both CephFS and Object Storage)
> and use the second one to all Cephs processes to talk one to the
> other.
>
Ceph support `cluster network` and `public network` options. Only OSD
work with cluster network. Any other is a OSD clients - public network.
Great. How do I migrate from my current single network board to a dual one?
Can I migrate servers one by one to the dual network setup or do I
have to stop the whole ceph cluster and restart it all already on the
dual setup? 
1 Dec
1 Dec
1:05 p.m.
Hi Rodrigo,
The fact that you're getting logs from mon and the function name
set_mon_vals suggests that you made use of mon provided centralized config,
like `ceph config set osd cluster_network x.x.x.x` but it seems
cluster_network and public_network cannot be centralized and should be set
in ceph.conf. (Just an idea not really an expert).
If that's the case you should be able to `ceph config rm osd
cluster_network` to get your cluster healthy again. Then you could try
setting in config file directly.
HTH
On Fri, Nov 29, 2019 at 4:55 PM Rodrigo Severo - Fábrica <
rodrigo(a)fabricadeideias.com> wrote:
Em qui., 28 de nov. de 2019 às 18:32, Rodrigo Severo -
Fábrica
<rodrigo(a)fabricadeideias.com> escreveu:
--
erdem agaoglu
Em qui., 28 de nov. de 2019 às 13:39, Wido den Hollander
<wido(a)42on.com> escreveu:
>
> On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
> > Em qui., 28 de nov. de 2019 às 00:34, Konstantin Shalygin
> > <k0ste(a)k0ste.ru> escreveu:
> >>
> >>> My servers have 2 network boards each. I would like to use the
current
> >>> local one to talk to Cephs
clients (both CephFS and Object Storage)
> >>> and use the second one to all Cephs processes to talk one to the
> >>> other.
> >>>
> >>
> >> Ceph support `cluster network` and `public network` options. Only
OSD
> >> work with cluster network. Any
other is a OSD clients - public
network.
> >
> > Great. How do I migrate from my current single network board to a
dual
one?
> >
> > Can I migrate servers one by one to the dual network setup or do I
> > have to stop the whole ceph cluster and restart it all already on the
> > dual setup?
>
> Set the cluster_network in the ceph.conf and restart the OSDs one by
one.
Just tried that. The first OSD that I'm trying to restart won't came
up again.
Does anybody has any suggestion on how to get my ceph fs back to healthy
status?
I'm even considering stop it all and restarting but I'm afraid it
won't come back up with the new config.
Ideias? Suggestions?
Rodrigo
It presents the following messages which aren't that useful
to me:
Nov 28 18:26:33 a2-df systemd[1]: ceph-osd(a)1.service: Start request
repeated too quickly.
Nov 28 18:26:33 a2-df systemd[1]: ceph-osd(a)1.service: Failed with
result 'exit-code'.
Nov 28 18:26:33 a2-df systemd[1]: Failed to start Ceph object storage
daemon osd.1.
-- Subject: Unit ceph-osd(a)1.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit ceph-osd(a)1.service has failed.
--
-- The result is RESULT.
I also see the following error messages:
Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230
7f487dee1700 -1 set_mon_vals failed to set cluster_network =
192.168.111.0/24: Configuration option 'cluster_network' may not be
modified at runtime
Nov 28 18:26:46 a2-df ceph-mon[2526]: 2019-11-28 18:26:46.230
7f487dee1700 -1 set_mon_vals failed to set public_network =
192.168.109.0/24: Configuration option 'public_network' may not be
modified at runtime
There are 2 things that I don't understand in these messages:
1. Why is it mentioning configuration option 'public_network' in these
error messages as I didn't change the public_network config, I only
added a cluster_network one?
2. Why are there messages from ceph-mon when I'm trying to restart
ceph-osd?
And the most important issue: how can I get my osd back online?
Regards,
Rodrigo
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Dec
2 Dec
12:39 p.m.
Em dom., 1 de dez. de 2019 às 09:05, Erdem Agaoglu
<erdem.agaoglu(a)gmail.com> escreveu:
Hi Rodrigo,
The fact that you're getting logs from mon and the function name set_mon_vals
suggests that you made use of mon provided centralized config, like `ceph config set osd
cluster_network x.x.x.x` but it seems cluster_network and public_network cannot be
centralized and should be set in ceph.conf. (Just an idea not really an expert).
If that's the case you should be able to `ceph config rm osd cluster_network` to get
your cluster healthy again. Then you could try setting in config file directly.
Unfortunately this wasn't the case. I say so because to get my cluster
healthy again I just commented out the cluster_network settings in my
ceph.conf files. After that I could restart my OSD with any mon
errors. I did that a few days ago and forgot to post this info here.
I'm sorry.
My immediate issue was solved but my main issue remains: how can I
migrate my current single network board setup to a dual one?
As far as my experience goes, setting cluster_network in the global
section of my ceph.conf files isn' t the proper way.
Maybe cluster_network shouldn' t be set in the global section. Is that
it? In which section should it be set instead?
Regards,
Rodrigo Severo
1600
days inactive
1605
days old
7 comments
4 participants
participants (4)
-
Erdem Agaoglu -
Konstantin Shalygin -
Rodrigo Severo - Fábrica -
Wido den Hollander