First, I guess I would make sure that peon7 and peon12 actually could pass the host check (you can run "cephadm check-host" on the host directly if you have a copy of the cephadm binary there) Then I'd try a mgr failover (ceph mgr fail) to clear out any in memory host values cephadm might have and restart the module. If it still reproduces after that, then you might have to set mgr/cephadm/log_to_cluster_level to debug, do another mgr failover, wait until the module crashes and see if "ceph log last 100 debug cephadm" gives more info on where the crash occurred (it might have an actual traceback). On Thu, Apr 4, 2024 at 4:51 AM <arnoud@fuga.cloud> wrote:
Hi,
I’ve added some new nodes to our Ceph cluster. Only did the host add, had not added the OSD’s yet. Due to a configuration error I had to reinstall some of them. But I forgot to remove the nodes from Ceph first. I did a “ceph orch host rm peon7 --offline —force” before re-adding them to the cluster.
All the nodes are showing up in the host list (all the peons are the new ones):
# ceph orch host ls HOST ADDR LABELS STATUS ceph1 10.103.0.71 ceph2 10.103.0.72 ceph3 10.103.0.73 ceph4 10.103.0.74 compute1 10.103.0.11 compute2 10.103.0.12 compute3 10.103.0.13 compute4 10.103.0.14 controller1 10.103.0.8 controller2 10.103.0.9 controller3 10.103.0.10 peon1 10.103.0.41 peon2 10.103.0.42 peon3 10.103.0.43 peon4 10.103.0.44 peon5 10.103.0.45 peon6 10.103.0.46 peon7 10.103.0.47 peon8 10.103.0.48 peon9 10.103.0.49 peon10 10.103.0.50 peon12 10.103.0.52 peon13 10.103.0.53 peon14 10.103.0.54 peon15 10.103.0.55 peon16 10.103.0.56
But Ceph status still shows an error, which I can’t seem to get rid off.
[WRN] CEPHADM_HOST_CHECK_FAILED: 2 hosts fail cephadm check host peon7 (10.103.0.47) failed check: Can't communicate with remote host `10.103.0.47`, possibly because python3 is not installed there or you are missing NOPASSWD in sudoers. [Errno 113] Connect call failed ('10.103.0.47', 22) host peon12 (10.103.0.52) failed check: Can't communicate with remote host `10.103.0.52`, possibly because python3 is not installed there or you are missing NOPASSWD in sudoers. [Errno 113] Connect call failed ('10.103.0.52', 22) [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'peon7' Module 'cephadm' has failed: ‘peon7'
From the mgr log:
Apr 04 08:33:46 controller2 bash[4031857]: debug 2024-04-04T08:33:46.876+0000 7f2bb5710700 -1 mgr.server reply reply (5) Input/output error Module 'cephadm' has experienced an error and cannot handle commands: 'peon7'
Any idea how to clear this error?
# ceph --version ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)
Regards, Arnoud de Jonge. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io