Ceph Dashboard suddenly gone and primary remote is not accessible [CEPHADM_HOST_CHECK_FAILED, CEPHADM_REFRESH_FAILED]
19 May
2020
19 May
'20
11:44 p.m.
Hi,
I was browsing dashboard today. Then suddently it stopped working and i got 502 errors. I
checked via root login and see thet ceph health is down to WARN.
I can access all rdb devices and CephFS. They work. All OSDs in server-1 is up.
health: HEALTH_WARN
1 hosts fail cephadm check
failed to probe daemons or devices
I even restarted server-1. No luck.
I'm on server-1. cephadm complains it cannot access to server-1. In basic term,
server-1 cannot access server-1 (192.168.0.1)
server-1: 192.168.0.1
server-2: 192.168.0.3
$ ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get
mgr/cephadm/ssh_identity_key) root@server-1
Success.
cephadm says server-1 is not reachable. Why cephadm/ceph complain about this and what
happend to my dashboard? I also redirected logs to file. I'm observing for any weird
log input but unfortunately nothing helpful. Where should I look for? What am i missing?
So, when I ran this command:
$ ceph health detail
Output was:
HEALTH_WARN 1 hosts fail cephadm check; failed to probe daemons or devices
[WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check
host server-1 failed check: Failed to connect to server-1 (server-1). Check that the
host is reachable and accepts
connections using the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get mgr/cephadm/ssh_identity_key) root@server-1
[WRN]
CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host server-1 scrape failed: Failed to connect to server-1 (server-1). Check that the
host is reachable and accepts connections using the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get mgr/cephadm/ssh_identity_key) root@server-1
host server-1
ceph-volume inventory failed: Failed to connect
to server-1 (server-1). Check that the host is reachable and accepts connections using
the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get mgr/cephadm/ssh_identity_key) root@server-1
Thanks,
Gencer.
19 May
19 May
11:56 p.m.
Hi again,
One more update:
I connected to server-2 and ran ceph -s there. I got:
Error initializing cluster client: ObjectNotFound('RADOS object not found (error
calling conf_read_file)')
Today I created a RBD pool and created 2 RDB images in this pool. Could this be reason for
all dashboard issue and this?
Thanks,
Gencer.
On 19.05.2020 23:44:25, Gencer W. Genç <gencer(a)gencgiyen.com> wrote:
Hi,
I was browsing dashboard today. Then suddently it stopped working and i got 502 errors. I
checked via root login and see thet ceph health is down to WARN.
I can access all rdb devices and CephFS. They work. All OSDs in server-1 is up.
health: HEALTH_WARN
1 hosts fail cephadm check
failed to probe daemons or devices
I even restarted server-1. No luck.
I'm on server-1. cephadm complains it cannot access to server-1. In basic term,
server-1 cannot access server-1 (192.168.0.1)
server-1: 192.168.0.1
server-2: 192.168.0.3
$ ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get
mgr/cephadm/ssh_identity_key) root@server-1
Success.
cephadm says server-1 is not reachable. Why cephadm/ceph complain about this and what
happend to my dashboard? I also redirected logs to file. I'm observing for any weird
log input but unfortunately nothing helpful. Where should I look for? What am i missing?
So, when I ran this command:
$ ceph health detail
Output was:
HEALTH_WARN 1 hosts fail cephadm check; failed to probe daemons or devices
[WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check
host server-1 failed check: Failed to connect to server-1 (server-1). Check that the
host is reachable and accepts
connections using the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get mgr/cephadm/ssh_identity_key) root@server-1
[WRN]
CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host server-1 scrape failed: Failed to connect to server-1 (server-1). Check that the
host is reachable and accepts connections using the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get mgr/cephadm/ssh_identity_key) root@server-1
host server-1
ceph-volume inventory failed: Failed to connect
to server-1 (server-1). Check that the host is reachable and accepts connections using
the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get mgr/cephadm/ssh_identity_key) root@server-1
Thanks,
Gencer.
20 May
20 May
2:37 a.m.
This happens (unfortunately) frequently to me. Look for the active mgr
(ceph -s), and go restart the mgr service there (systemctl list-units |grep
mgr then systemctl restart NAMEOFSERVICE). This normally resolves that
error for me. You can look at the journalctl output and you'll likely see
errors similar to this issue: https://tracker.ceph.com/issues/45032
I'm hopeful the PR in that issue will resolve the issue for us:
https://github.com/ceph/ceph/pull/35022
See if that works - if not - you may want to look at the logs for the mgr
(and potentially other) daemons using journalctl.
On Tue, May 19, 2020 at 3:45 PM Gencer W. Genç <gencer(a)gencgiyen.com> wrote:
Hi,
I was browsing dashboard today. Then suddently it stopped working and i
got 502 errors. I checked via root login and see thet ceph health is down
to WARN.
I can access all rdb devices and CephFS. They work. All OSDs in server-1
is up.
health: HEALTH_WARN
1 hosts fail cephadm check
failed to probe daemons or devices
I even restarted server-1. No luck.
I'm on server-1. cephadm complains it cannot access to server-1. In basic
term, server-1 cannot access server-1 (192.168.0.1)
server-1: 192.168.0.1
server-2: 192.168.0.3
$ ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get
mgr/cephadm/ssh_identity_key) root@server-1
Success.
cephadm says server-1 is not reachable. Why cephadm/ceph complain about
this and what happend to my dashboard? I also redirected logs to file. I'm
observing for any weird log input but unfortunately nothing helpful. Where
should I look for? What am i missing?
So, when I ran this command:
$ ceph health detail
Output was:
HEALTH_WARN 1 hosts fail cephadm check; failed to probe daemons or devices
[WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check
host server-1 failed check: Failed to connect to server-1 (server-1).
Check that the host is reachable and accepts
connections using the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get
mgr/cephadm/ssh_identity_key) root@server-1
[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host server-1 scrape failed: Failed to connect to server-1
(server-1). Check that the host is reachable and accepts connections using
the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get
mgr/cephadm/ssh_identity_key) root@server-1
host server-1 ceph-volume inventory failed: Failed to connect
to server-1 (server-1). Check that the host is reachable and accepts
connections using the cephadm SSH key
you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph
config-key get
mgr/cephadm/ssh_identity_key) root@server-1
Thanks,
Gencer.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1446
days inactive
1446
days old
2 comments
2 participants
participants (2)
-
David Orman -
Gencer W. Genç