20 May
2021
20 May
'21
9:04 p.m.
Hello
I have a weird problem on 3 node cluster. "Nautilus 14.2.9"
When I try power failure OSD's are not marking as DOWN and MDS do not
respond anymore.
If I manually set osd down then MDS becomes active again.
BTW: Only 2 node has OSD's. Third node is only for MON.
I've set mon_osd_down_out_interval = 0.3 in ceph.conf at global
section and restart all MON's but when I check it with "ceph daemon
mon.ID config show" I see mon_osd_down_out_interval: "600". I didn't
get it why its still "600" and honestly I don't know even it has any
effect on my problem.
Where should I check?
21 May
21 May
3:34 p.m.
I've figured out but I'm scared from the result.
The solution is "mon_osd_min_down_reporters = 1"
Due to "two node" cluster and "replicated 2" with "chooseleaf
host"
the reporter count should be set to 1 but on a malfunction this could
be a serious problem.
Is there any better solution?
by morphin <morphinwithyou(a)gmail.com>om>, 20 May 2021 Per, 22:04
tarihinde şunu yazdı:
>
> Hello
>
> I have a weird problem on 3 node cluster. "Nautilus 14.2.9"
> When I try power failure OSD's are not marking as DOWN and MDS do not
> respond anymore.
> If I manually set osd down then MDS becomes active again.
>
> BTW: Only 2 node has OSD's. Third node is only for MON.
>
> I've set mon_osd_down_out_interval = 0.3 in ceph.conf at global
> section and restart all MON's but when I check it with "ceph daemon
> mon.ID config show" I see mon_osd_down_out_interval: "600". I
didn't
> get it why its still "600" and honestly I don't know even it has any
> effect on my problem.
>
> Where should I check?
4:35 p.m.
Is there any better solution?
Yes, add more nodes. ;-)
Having only two OSD nodes is not the best idea, it's kind of a corner
case and I've observed some weird behaviour with corner cases in the
past, not even mentioning the 2 replicas. Is this a test environment?
Zitat von by morphin <morphinwithyou(a)gmail.com>om>:
> I've figured out but I'm scared from the result.
> The solution is "mon_osd_min_down_reporters = 1"
> Due to "two node" cluster and "replicated 2" with
"chooseleaf host"
> the reporter count should be set to 1 but on a malfunction this could
> be a serious problem.
>
Is there any better solution?
>
> by morphin <morphinwithyou(a)gmail.com>om>, 20 May 2021 Per, 22:04
> tarihinde şunu yazdı:
>>
>> Hello
>>
>> I have a weird problem on 3 node cluster. "Nautilus 14.2.9"
>> When I try power failure OSD's are not marking as DOWN and MDS do not
>> respond anymore.
>> If I manually set osd down then MDS becomes active again.
>>
>> BTW: Only 2 node has OSD's. Third node is only for MON.
>>
>> I've set mon_osd_down_out_interval = 0.3 in ceph.conf at global
>> section and restart all MON's but when I check it with "ceph daemon
>> mon.ID config show" I see mon_osd_down_out_interval: "600". I
didn't
>> get it why its still "600" and honestly I don't know even it has
any
>> effect on my problem.
>>
>> Where should I check?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:36 p.m.
New subject: orch apply mon assigns wrong IP address?
Is there a way to force '.. orch apply *' to limit ip address selection
to addresses matching the hostname in dns or /etc/hosts, or to a
specific address given at 'host add' time? I've hit a bothersome problem:
On v15, 'ceph orch apply mon ...' appears not to use the dns ip or
/etc/hosts when installing a monitor, but instead appears to select one
from the current list of interfaces up on the host. 99 of 100 times
this is correct as the host has but one address on the public cluster
subnet.
However, this or that host might have a temporary added ip interface up
on that subnet (whether for one-of diagnostic purposes or some HA
assigned movable address like a time of day server). Occasionally that
second 'unlisted' address is 'up' when the orchestrator decides then
would be a good time to add a mon to that host. Ceph somehow
occasionally prefers this 'unlisted but up' interface address, using
that permanently as the host address for the monitor docker image, even
though that 'secondary' interface is not in DNS for the host, nor in
/etc/hosts anywhere for that host.
Is this a known issue? Is there a way to direct the orchestrator to use
'dns resolvable only' or 'preferred' addresses?
Thanks
Harry Coin
4:49 p.m.
New subject: orch apply mon assigns wrong IP address?
You can define the public_network [1]:
ceph config set mon public_network *<mon-cidr-network>*
For example:
ceph config set mon public_network 10.1.2.0/24
Or is that already defined and it happens anyway?
[1]
https://docs.ceph.com/en/latest/cephadm/mon/#designating-a-particular-subne…
Zitat von "Harry G. Coin" <hgcoin(a)gmail.com>om>:
Is there a way to force '.. orch apply *' to
limit ip address selection
to addresses matching the hostname in dns or /etc/hosts, or to a
specific address given at 'host add' time? I've hit a bothersome problem:
On v15, 'ceph orch apply mon ...' appears not to use the dns ip or
/etc/hosts when installing a monitor, but instead appears to select one
from the current list of interfaces up on the host. 99 of 100 times
this is correct as the host has but one address on the public cluster
subnet.
However, this or that host might have a temporary added ip interface up
on that subnet (whether for one-of diagnostic purposes or some HA
assigned movable address like a time of day server). Occasionally that
second 'unlisted' address is 'up' when the orchestrator decides then
would be a good time to add a mon to that host. Ceph somehow
occasionally prefers this 'unlisted but up' interface address, using
that permanently as the host address for the monitor docker image, even
though that 'secondary' interface is not in DNS for the host, nor in
/etc/hosts anywhere for that host.
Is this a known issue? Is there a way to direct the orchestrator to use
'dns resolvable only' or 'preferred' addresses?
Thanks
Harry Coin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:02 p.m.
New subject: orch apply mon assigns wrong IP address?
On 5/21/21 9:49 AM, Eugen Block wrote:
You can define the public_network [1]:
ceph config set mon public_network *<mon-cidr-network>*
For example:
ceph config set mon public_network 10.1.2.0/24
Or is that already defined and it happens anyway?
The public network is defined,
and it happens anyway (the temporary,
unlisted interface address is in the public network, but not either
forward or reverse resolveable to the host in dns or /etc/hosts).
5:33 p.m.
New subject: orch apply mon assigns wrong IP address?
The public network is defined, and it happens anyway
(the temporary,
unlisted interface address is in the public network, but not either
forward or reverse resolveable to the host in dns or /etc/hosts).
Okay, then I don't have a solution for that. Maybe someone else has
dealt with this already.
Zitat von "Harry G. Coin" <hgcoin(a)gmail.com>om>:
> On 5/21/21 9:49 AM, Eugen Block wrote:
>> You can define the public_network [1]:
>>
>> ceph config set mon public_network *<mon-cidr-network>*
>>
>> For example:
>>
>> ceph config set mon public_network 10.1.2.0/24
>>
>> Or is that already defined and it happens anyway?
The public network is defined, and it happens anyway
(the temporary,
unlisted interface address is in the public network, but not either
forward or reverse resolveable to the host in dns or /etc/hosts).
>>
>> [1]
>>
https://docs.ceph.com/en/latest/cephadm/mon/#designating-a-particular-subne…
>>
1080
days inactive
1081
days old
6 comments
3 participants
participants (3)
-
by morphin -
Eugen Block -
Harry G. Coin