On Fri, Aug 2, 2019 at 5:41 PM Brad Hubbard <bhubbard(a)redhat.com> wrote:
Hmmm....trying to understand this.
$ head -1 .ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
So that is *not* the format causing the errors mentioned in
https://github.com/paramiko/paramiko/issues/1015 right?
I also do *not* see the "not a valid [key] private key file" mentioned.
Also...
$ cat test_connect.py
import os
import paramiko
user = "ubuntu"
host = "mira110.front.sepia.ceph.com"
ssh = paramiko.SSHClient()
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
connect_args = dict(
hostname=host,
username=user,
timeout=60
)
ssh_config_path = os.path.expanduser("~/.ssh/config")
if os.path.exists(ssh_config_path):
ssh_config = paramiko.SSHConfig()
ssh_config.parse(open(ssh_config_path))
opts = ssh_config.lookup(host)
ssh.connect(**connect_args)
(stdin, stdout, stderr) = ssh.exec_command("echo 'Hello World ! from '
$(hostname)")
for line in stdout.readlines():
print(line)
ssh.close()
$ source src/teuthology/virtualenv/bin/activate
$ python test_connect.py
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296:
UserWarning: implicit cast from 'char *' to a different pointer type:
will be forbidden in the future (check that the types are as you
expect; use an explicit ffi.cast() if they are correct)
group, point, conversion, buf, buflen, bn_ctx
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114:
UserWarning: implicit cast from 'char *' to a different pointer type:
will be forbidden in the future (check that the types are as you
expect; use an explicit ffi.cast() if they are correct)
operation
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:140:
UserWarning: implicit cast from 'char *' to a different pointer type:
will be forbidden in the future (check that the types are as you
expect; use an explicit ffi.cast() if they are correct)
self._backend._ffi.from_buffer(data), len(data)
Hello World ! from mira110
Also, looking at the keys repo it seems most people are using rsa keys
so I'm not sure it's the key that's the problem.
2019-08-02 03:36:03,603.603 ERROR:paramiko.transport:Exception: Error
reading SSH protocol banner
2019-08-02 03:36:03,603.603 ERROR:paramiko.transport:Traceback (most
recent call last):
2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: File
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py",
line 1966, in run
2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: self._check_banner()
2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: File
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py",
line 2143, in _check_banner
2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: "Error
reading SSH protocol banner" + str(e)
2019-08-02 03:36:03,604.604 ERROR:paramiko.transport:SSHException:
Error reading SSH protocol banner
The above seems to be the main problem but unfortunately that seems to
be a "catch-all" error for any problem with the connection.
So I mentioned earlier this problem didn't seem to affect every
connection (some appeared to "work" for some level of "work" :P) so
why is that?
After considerable efforts to uncover the differences I realised that
the rare systems where the connections succeeded were not in my
known_hosts file and therefore generated no warnings about
'known_hosts' entries when logging on (I have "StrictHostKeyChecking
no" in my ssh config file so the warnings are generated, but ignored).
So I deleted my known_hosts file and tried again and it worked. This
really is pretty piss-poor error reporting.
So I guess the latest teuthology update makes teuthology intolerant of
these 'known_hosts' warnings generated for ssh connections (or
something related in some way to that). Given I'm likely to not be the
only one stung by this and given the cryptic nature of the debugging
efforts I'm hoping this will help others similarly afflicted until a
suitable solution is found. So what's the fix for this going forward?
We could add "UserKnownHostsFile=/dev/null" in our ssh configs, but
that seems a bit extreme.
As a final note I tested this with my script above (since it doesn't
take long for known_hosts entries to build up) and got the following
results which may give more/better clues.
$ python test_connect.py
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296:
UserWarning: implicit cast from 'char *' to a different pointer type:
will be forbidden in the future (check that the types are as you
expect; use an explicit ffi.cast() if they are correct)
group, point, conversion, buf, buflen, bn_ctx
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114:
UserWarning: implicit cast from 'char *' to a different pointer type:
will be forbidden in the future (check that the types are as you
expect; use an explicit ffi.cast() if they are correct)
operation
Traceback (most recent call last):
File "test_connect.py", line 23, in <module>
ssh.connect(**connect_args)
File
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py",
line 414, in connect
raise BadHostKeyException(hostname, server_key, our_key)
paramiko.ssh_exception.BadHostKeyException:
('mira041.front.sepia.ceph.com', <paramiko.ecdsakey.ECDSAKey object at
0x7fb534521890>, <paramiko.ecdsakey.ECDSAKey object at
0x7fb534ec8a10>)
$ rm .ssh/known_hosts
$ python test_connect.py
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296:
UserWarning: implicit cast from 'char *' to a different pointer type:
will be forbidden in the future (check that the types are as you
expect; use an explicit ffi.cast() if they are correct)
group, point, conversion, buf, buflen, bn_ctx
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114:
UserWarning: implicit cast from 'char *' to a different pointer type:
will be forbidden in the future (check that the types are as you
expect; use an explicit ffi.cast() if they are correct)
operation
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:140:
UserWarning: implicit cast from 'char *' to a different pointer type:
will be forbidden in the future (check that the types are as you
expect; use an explicit ffi.cast() if they are correct)
self._backend._ffi.from_buffer(data), len(data)
Hello World ! from
mira041.front.sepia.ceph.com
So it looks like the BadHostKeyException is the one we need to try and
find a solution for I guess?
HTH.
On Thu, Aug 1, 2019 at 10:40 PM David Galloway <dgallowa(a)redhat.com> wrote:
I failed to add some context to #3 in the wiki.
The version of Paramiko was recently updated in teuthology. The newest
version doesn't support the default RSA key type in newer OpenSSH versions.
See
https://github.com/paramiko/paramiko/issues/1015
On 7/31/19 7:24 PM, Brad Hubbard wrote:
> Another thing that makes me wonder about this is I can log onto
> machines that I already had reserved.
>
> bhubbard@teuthology:~$ ssh mira110
> Last login: Wed Jul 31 23:03:08 2019 from
teuthology.front.sepia.ceph.com
>
> My key is 0600 and hasn't change/been modified for over a year.
>
>
> On Thu, Aug 1, 2019 at 8:56 AM Brad Hubbard <bhubbard(a)redhat.com> wrote:
>>
>> Hi David,
>>
>> Thanks for the response. I would wonder why any of that would have
>> changed yesterday after successfully piloting teuthology for a few
>> years but I'll certainly look into that possibility.
>>
>> On Wed, Jul 31, 2019 at 9:55 PM David Galloway <dgallowa(a)redhat.com>
wrote:
>>>
>>> Is it #3 at
>>>
https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#sshexceptionerror_re…
>>>
>>> On 7/30/19 9:57 PM, Brad Hubbard wrote:
>>>> Also this one.
>>>>
>>>>
../src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/kex_ecdh_nist.py:39:
>>>> CryptographyDeprecationWarning: encode_point has been deprecated on
>>>> EllipticCurvePublicNumbers and will be removed in a future version.
>>>> Please use EllipticCurvePublicKey.public_bytes to obtain both
>>>> compressed and uncompressed point encoding.
>>>> m.add_string(self.Q_C.public_numbers().encode_point())
>>>>
>>>> On Wed, Jul 31, 2019 at 11:48 AM Brad Hubbard
<bhubbard(a)redhat.com> wrote:
>>>>>
>>>>> I updated teuthology yesterday and since then have seen a log of
the
>>>>> following errors
>>>>>
>>>>>
...src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/ecdsakey.py:164:
>>>>> CryptographyDeprecationWarning: Support for unsafe construction of
>>>>> public numbers from encoded data will be removed in a future
version.
>>>>> Please use EllipticCurvePublicKey.from_encoded_point
>>>>> self.ecdsa_curve.curve_class(), pointinfo
>>>>>
>>>>>
>>>>> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport:Exception:
Error
>>>>> reading SSH protocol banner
>>>>> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport:Traceback
(most
>>>>> recent call last):
>>>>> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport: File
>>>>>
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py",
>>>>> line 1966, in run
>>>>> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport:
self._check_banner()
>>>>> 2019-07-31 01:45:18,977.977 ERROR:paramiko.transport: File
>>>>>
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py",
>>>>> line 2143, in _check_banner
>>>>> 2019-07-31 01:45:18,977.977 ERROR:paramiko.transport:
"Error
>>>>> reading SSH protocol banner" + str(e)
>>>>> 2019-07-31 01:45:18,977.977 ERROR:paramiko.transport:SSHException:
>>>>> Error reading SSH protocol banner
>>>>>
>>>>> Sometimes these are fatal and sometimes not. Wondering if anyone
else
>>>>> has seen them?
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Brad
>>>>
>>>>
>>>>
>>
>>
>>
>> --
>> Cheers,
>> Brad
>
>
>
> --
> Cheers,
> Brad
>
--
Cheers,
Brad