Ceph Dashboard suddenly gone and primary remote is not accessible [CEPHADM_HOST_CHECK_FAILED, CEPHADM_REFRESH_FAILED]
Hi, I was browsing dashboard today. Then suddently it stopped working and i got 502 errors. I checked via root login and see thet ceph health is down to WARN. I can access all rdb devices and CephFS. They work. All OSDs in server-1 is up. health: HEALTH_WARN 1 hosts fail cephadm check failed to probe daemons or devices I even restarted server-1. No luck. I'm on server-1. cephadm complains it cannot access to server-1. In basic term, server-1 cannot access server-1 (192.168.0.1) server-1: 192.168.0.1 server-2: 192.168.0.3 $ ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1
Success.
cephadm says server-1 is not reachable. Why cephadm/ceph complain about this and what happend to my dashboard? I also redirected logs to file. I'm observing for any weird log input but unfortunately nothing helpful. Where should I look for? What am i missing? So, when I ran this command: $ ceph health detail Output was: HEALTH_WARN 1 hosts fail cephadm check; failed to probe daemons or devices [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check host server-1 failed check: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host server-1 scrape failed: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 host server-1 ceph-volume inventory failed: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1
Thanks, Gencer.
Hi again, One more update: I connected to server-2 and ran ceph -s there. I got: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)') Today I created a RBD pool and created 2 RDB images in this pool. Could this be reason for all dashboard issue and this? Thanks, Gencer. On 19.05.2020 23:44:25, Gencer W. Genç <gencer@gencgiyen.com> wrote: Hi, I was browsing dashboard today. Then suddently it stopped working and i got 502 errors. I checked via root login and see thet ceph health is down to WARN. I can access all rdb devices and CephFS. They work. All OSDs in server-1 is up. health: HEALTH_WARN 1 hosts fail cephadm check failed to probe daemons or devices I even restarted server-1. No luck. I'm on server-1. cephadm complains it cannot access to server-1. In basic term, server-1 cannot access server-1 (192.168.0.1) server-1: 192.168.0.1 server-2: 192.168.0.3 $ ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1
Success.
cephadm says server-1 is not reachable. Why cephadm/ceph complain about this and what happend to my dashboard? I also redirected logs to file. I'm observing for any weird log input but unfortunately nothing helpful. Where should I look for? What am i missing? So, when I ran this command: $ ceph health detail Output was: HEALTH_WARN 1 hosts fail cephadm check; failed to probe daemons or devices [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check host server-1 failed check: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host server-1 scrape failed: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 host server-1 ceph-volume inventory failed: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1
Thanks, Gencer.
This happens (unfortunately) frequently to me. Look for the active mgr (ceph -s), and go restart the mgr service there (systemctl list-units |grep mgr then systemctl restart NAMEOFSERVICE). This normally resolves that error for me. You can look at the journalctl output and you'll likely see errors similar to this issue: https://tracker.ceph.com/issues/45032 I'm hopeful the PR in that issue will resolve the issue for us: https://github.com/ceph/ceph/pull/35022 See if that works - if not - you may want to look at the logs for the mgr (and potentially other) daemons using journalctl. On Tue, May 19, 2020 at 3:45 PM Gencer W. Genç <gencer@gencgiyen.com> wrote:
Hi,
I was browsing dashboard today. Then suddently it stopped working and i got 502 errors. I checked via root login and see thet ceph health is down to WARN.
I can access all rdb devices and CephFS. They work. All OSDs in server-1 is up.
health: HEALTH_WARN 1 hosts fail cephadm check failed to probe daemons or devices
I even restarted server-1. No luck.
I'm on server-1. cephadm complains it cannot access to server-1. In basic term, server-1 cannot access server-1 (192.168.0.1)
server-1: 192.168.0.1 server-2: 192.168.0.3
$ ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1
Success.
cephadm says server-1 is not reachable. Why cephadm/ceph complain about this and what happend to my dashboard? I also redirected logs to file. I'm observing for any weird log input but unfortunately nothing helpful. Where should I look for? What am i missing?
So, when I ran this command:
$ ceph health detail
Output was:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host server-1 scrape failed: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using
HEALTH_WARN 1 hosts fail cephadm check; failed to probe daemons or devices [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check host server-1 failed check: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: the cephadm SSH key you may want to run:
ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 host server-1 ceph-volume inventory failed: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1
Thanks, Gencer. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
David Orman
-
Gencer W. Genç