Hi Adam, Thank you for the details. I see that the cephadm on the Ceph cluster is different from the host that is being added. I will go thru the ticket and the logs. Also the cluster is on Ubuntu Focal and the new host is on Ubuntu Jammy The utility: cephadm 16.2.13-1focal amd64 cephadm utility to bootstrap ceph daemons with systemd and containers cephadm 17.2.5-0ubuntu0.22.04.3 amd64 cephadm utility to bootstrap ceph daemons with systemd and containers Thanks again, Anantha From: Adam King <adking@redhat.com> Sent: Tuesday, June 20, 2023 4:25 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Error while adding host : Error EINVAL: Traceback (most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in _handle_command There was a cephadm bug that wasn't fixed by the time 17.2.6 came out (I'm assuming that's the version being used here, although it may have been present in some slightly earlier quincy versions) that caused this misleading error to be printed out when adding a host failed. There's a tracker for it here https://tracker.ceph.com/issues/59081 that has roughly the same traceback. The real issue is likely a connectivity or permission issue from the active mgr trying to ssh to the host. In the case I saw from the tracker, it was caused by the ssh pub key not being set up on the host. If you check the cephadm cluster logs ("ceph log last 50 debug cephadm") after trying to add the host I'm guessing you'll see some error like the second set of output in the tracker that will hopefully give some more info on why adding the host failed. On Tue, Jun 20, 2023 at 6:38 PM Adiga, Anantha <anantha.adiga@intel.com<mailto:anantha.adiga@intel.com>> wrote: Hi, I am seeing this error after an offline was deleted and while adding the host again. Thereafter, I have removed the /var/lib/cep folder and removed the ceph quincy image in the offline host. What is the cause of this issue and the solution. root@fl31ca104ja0201:/home/general# cephadm shell Inferring fsid d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e Using recent ceph image quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e<http://quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e><mailto:quay.io<mailto:quay.io>/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e> root@fl31ca104ja0201:/# root@fl31ca104ja0201:/# ceph orch host rm fl31ca104ja0302 --offline --force Removed offline host 'fl31ca104ja0302' root@fl31ca104ja0201:/# ceph -s cluster: id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e health: HEALTH_OK services: mon: 3 daemons, quorum fl31ca104ja0201,fl31ca104ja0202,fl31ca104ja0203 (age 28h) mgr: fl31ca104ja0203(active, since 6d), standbys: fl31ca104ja0202, fl31ca104ja0201 mds: 1/1 daemons up, 2 standby osd: 33 osds: 33 up (since 28h), 33 in (since 28h) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 24 pools, 737 pgs objects: 613.56k objects, 1.9 TiB usage: 2.9 TiB used, 228 TiB / 231 TiB avail pgs: 737 active+clean io: client: 161 MiB/s rd, 75 op/s rd, 0 op/s wr root@fl31ca104ja0201:/# ceph orch host add fl31ca104ja0302 10.45.219.5 Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host return self._apply_misc([s], False, Format.plain) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in _apply_misc raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in raise_if_exception e = pickle.loads(c.serialized_exception) TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' Thank you, Anantha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>