Resent due to moderation when using web interface. Hi ceph users, We have a few clusters with quincy 17.2.6 and we are preparing to migrate from ceph-deploy to cephadm for better management. We are using Ubuntu20 with latest updates (latest openssh). While testing the migration to cephadm on a test cluster with octopus (v16 latest) we had no issues replacing ceph generated cert/key with our own CA signed certs (ECDSA). After upgrading to quincy the test cluster and test again the migration we cannot add hosts due to the errors below, ssh access errors specified a while ago in a tracker. We use the following type of certs: Type: ecdsa-sha2-nistp384-cert-v01@openssh.com user certificate The certificate works everytime when using ssh client from shell to connect to all hosts in the cluster. We do a ceph mgr fail every time we replace cert/key so they are restarted. ----- cephadm logs from mgr ------ Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 111, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: yield Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 90, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: preferred_auth=['publickey'], options=ssh_options) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect Oct 06 09:23:27 ceph-m2 bash[1363]: 'Opening SSH connection to') Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect Oct 06 09:23:27 ceph-m2 bash[1363]: await conn.wait_established() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established Oct 06 09:23:27 ceph-m2 bash[1363]: await self._waiter Oct 06 09:23:27 ceph-m2 bash[1363]: asyncssh.misc.PermissionDenied: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: During handling of the above exception, another exception occurred: Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return OrchResult(f(*args, **kwargs)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2810, in apply Oct 06 09:23:27 ceph-m2 bash[1363]: results.append(self._apply(spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2558, in _apply Oct 06 09:23:27 ceph-m2 bash[1363]: return self._add_host(cast(HostSpec, spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1434, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: ip_addr = self._check_valid_addr(spec.hostname, spec.addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1415, in _check_valid_addr Oct 06 09:23:27 ceph-m2 bash[1363]: error_ok=True, no_fsid=True)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async Oct 06 09:23:27 ceph-m2 bash[1363]: return self.event_loop.get_result(coro) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result Oct 06 09:23:27 ceph-m2 bash[1363]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result Oct 06 09:23:27 ceph-m2 bash[1363]: return self.__get_result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result Oct 06 09:23:27 ceph-m2 bash[1363]: raise self._exception Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1361, in _run_cephadm Oct 06 09:23:27 ceph-m2 bash[1363]: await self.mgr.ssh._remote_connection(host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: raise Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ Oct 06 09:23:27 ceph-m2 bash[1363]: self.gen.throw(type, value, traceback) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: raise HostConnectionError(msg, host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: cephadm.ssh.HostConnectionError: Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.081+0000 7f78d86d8700 -1 log_channel(cephadm) log [ERR] : Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 111, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: yield Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 90, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: preferred_auth=['publickey'], options=ssh_options) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect Oct 06 09:23:27 ceph-m2 bash[1363]: 'Opening SSH connection to') Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect Oct 06 09:23:27 ceph-m2 bash[1363]: await conn.wait_established() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established Oct 06 09:23:27 ceph-m2 bash[1363]: await self._waiter Oct 06 09:23:27 ceph-m2 bash[1363]: asyncssh.misc.PermissionDenied: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: During handling of the above exception, another exception occurred: Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return OrchResult(f(*args, **kwargs)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2810, in apply Oct 06 09:23:27 ceph-m2 bash[1363]: results.append(self._apply(spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2558, in _apply Oct 06 09:23:27 ceph-m2 bash[1363]: return self._add_host(cast(HostSpec, spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1434, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: ip_addr = self._check_valid_addr(spec.hostname, spec.addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1415, in _check_valid_addr Oct 06 09:23:27 ceph-m2 bash[1363]: error_ok=True, no_fsid=True)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async Oct 06 09:23:27 ceph-m2 bash[1363]: return self.event_loop.get_result(coro) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result Oct 06 09:23:27 ceph-m2 bash[1363]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result Oct 06 09:23:27 ceph-m2 bash[1363]: return self.__get_result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result Oct 06 09:23:27 ceph-m2 bash[1363]: raise self._exception Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1361, in _run_cephadm Oct 06 09:23:27 ceph-m2 bash[1363]: await self.mgr.ssh._remote_connection(host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: raise Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ Oct 06 09:23:27 ceph-m2 bash[1363]: self.gen.throw(type, value, traceback) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: raise HostConnectionError(msg, host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: cephadm.ssh.HostConnectionError: Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.081+0000 7f78d86d8700 -1 mgr handle_command module 'orchestrator' command handler threw exception: __init__() missing 2 required positional arguments: > Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.093+0000 7f78d86d8700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command Oct 06 09:23:27 ceph-m2 bash[1363]: return self.handle_command(inbuf, cmd) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command Oct 06 09:23:27 ceph-m2 bash[1363]: return dispatch[cmd['prefix']].call(self, cmd, inbuf) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call Oct 06 09:23:27 ceph-m2 bash[1363]: return self.func(mgr, **kwargs) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda> Oct 06 09:23:27 ceph-m2 bash[1363]: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return func(*args, **kwargs) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: return self._apply_misc([s], False, Format.plain) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in _apply_misc Oct 06 09:23:27 ceph-m2 bash[1363]: raise_if_exception(completion) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in raise_if_exception Oct 06 09:23:27 ceph-m2 bash[1363]: e = pickle.loads(c.serialized_exception) Oct 06 09:23:27 ceph-m2 bash[1363]: TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' ----- cephadm logs from mgr ------ ----- sshd logs DEBUG3 level ------ Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: input_userauth_request: try method publickey [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: userauth_pubkey: valid user root querying public key ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzO------------ [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: userauth_pubkey: test pkalg ecdsa-sha2-nistp384 pkblob ECDSA SHA256:m6Q0ZQVjjDLWxbmCn0hcGQ2---------- [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_key_allowed entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_send entering: type 22 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive_expect entering: type 23 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: monitor_read: checking request 22 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed: key_from_blob: 0x5568f0aa7880 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: temporarily_use_uid: 0/0 (e=0/0) Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: trying public key file /etc/ssh/fake_authorized_keys Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: fd 5 clearing O_NONBLOCK Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: restore_uid: 0/0 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed: publickey authentication test: ECDSA key is not allowed Oct 6 09:33:09 ceph-m1 sshd[57168]: Failed publickey for root from 10.10.12.160 port 40854 ssh2: ECDSA SHA256:m6Q0ZQVjjDLWxbmCn0hcGQ24gbpk------------- Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_send entering: type 23 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: userauth_pubkey: authenticated 0 pkalg ecdsa-sha2-nistp384 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: user_specific_delay: user specific delay 0.000ms [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: ensure_minimum_time_since: elapsed 8.263ms, delaying 8.080ms (requested 8.171ms) [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: userauth_finish: failure partial=0 next methods="publickey" [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: send packet: type 51 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: Connection closed by authenticating user root 10.10.12.160 port 40854 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: do_cleanup [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: PAM: sshpam_thread_cleanup entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: monitor_read_log: child log fd closed Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: do_cleanup Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: PAM: cleanup Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: PAM: sshpam_thread_cleanup entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: Killing privsep child 57169 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: audit_event: unhandled event 12 Oct 6 09:33:09 ceph-m1 sshd[757]: debug1: main_sigchld_handler: Child exited --------------- I get "ECDSA key is not allowed" above. From sshd logs, it looks like the client is not sending what is required or in the expected format. Now, what was changed in quincy/mgr on ssh client? Is anyone else using ECDSA keys and it works with quincy? I could not find in PRs something specific to this that could block the access, but it might be. Any suggestion? Thank you! Paul