2 Aug
2019
2 Aug
'19
10:18 p.m.
Ah, it wasn't the issue I thought it was. I helped a new lab user
recently and ran into the "BEGIN OPENSSH PRIVATE KEY" thing and the
errors looked similar.
I've always had "UserKnownHostsFile /dev/null" in my SSH config so I've
never seen this problem personally. I don't think it's unreasonable to
have that config set in a lab environment especially considering the
smithi and mira host keys change every ~45min.
I've updated the Sepia wiki with a recommended SSH config.
On 8/2/19 3:41 AM, Brad Hubbard wrote:
> Hmmm....trying to understand this.
>
> $ head -1 .ssh/id_rsa
> -----BEGIN RSA PRIVATE KEY-----
>
> So that is *not* the format causing the errors mentioned in
> https://github.com/paramiko/paramiko/issues/1015 right?
>
> I also do *not* see the "not a valid [key] private key file" mentioned.
>
> Also...
>
> $ cat test_connect.py
> import os
> import paramiko
>
> user = "ubuntu"
> host = "mira110.front.sepia.ceph.com"
> ssh = paramiko.SSHClient()
>
> ssh.load_system_host_keys()
> ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>
> connect_args = dict(
> hostname=host,
> username=user,
> timeout=60
> )
>
> ssh_config_path = os.path.expanduser("~/.ssh/config")
> if os.path.exists(ssh_config_path):
> ssh_config = paramiko.SSHConfig()
> ssh_config.parse(open(ssh_config_path))
> opts = ssh_config.lookup(host)
>
> ssh.connect(**connect_args)
> (stdin, stdout, stderr) = ssh.exec_command("echo 'Hello World ! from '
> $(hostname)")
> for line in stdout.readlines():
> print(line)
> ssh.close()
>
> $ source src/teuthology/virtualenv/bin/activate
>
> $ python test_connect.py
>
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296:
> UserWarning: implicit cast from 'char *' to a different pointer type:
> will be forbidden in the future (check that the types are as you
> expect; use an explicit ffi.cast() if they are correct)
> group, point, conversion, buf, buflen, bn_ctx
>
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114:
> UserWarning: implicit cast from 'char *' to a different pointer type:
> will be forbidden in the future (check that the types are as you
> expect; use an explicit ffi.cast() if they are correct)
> operation
>
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:140:
> UserWarning: implicit cast from 'char *' to a different pointer type:
> will be forbidden in the future (check that the types are as you
> expect; use an explicit ffi.cast() if they are correct)
> self._backend._ffi.from_buffer(data), len(data)
> Hello World ! from mira110
>
> Also, looking at the keys repo it seems most people are using rsa keys
> so I'm not sure it's the key that's the problem.
>
> 2019-08-02 03:36:03,603.603 ERROR:paramiko.transport:Exception: Error
> reading SSH protocol banner
> 2019-08-02 03:36:03,603.603 ERROR:paramiko.transport:Traceback (most
> recent call last):
> 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: File
>
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py",
> line 1966, in run
> 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: self._check_banner()
> 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: File
>
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py",
> line 2143, in _check_banner
> 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: "Error
> reading SSH protocol banner" + str(e)
> 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport:SSHException:
> Error reading SSH protocol banner
>
> The above seems to be the main problem but unfortunately that seems to
> be a "catch-all" error for any problem with the connection.
>
> So I mentioned earlier this problem didn't seem to affect every
> connection (some appeared to "work" for some level of "work" :P)
so
> why is that?
>
> After considerable efforts to uncover the differences I realised that
> the rare systems where the connections succeeded were not in my
> known_hosts file and therefore generated no warnings about
> 'known_hosts' entries when logging on (I have "StrictHostKeyChecking
> no" in my ssh config file so the warnings are generated, but ignored).
> So I deleted my known_hosts file and tried again and it worked. This
> really is pretty piss-poor error reporting.
>
> So I guess the latest teuthology update makes teuthology intolerant of
> these 'known_hosts' warnings generated for ssh connections (or
> something related in some way to that). Given I'm likely to not be the
> only one stung by this and given the cryptic nature of the debugging
> efforts I'm hoping this will help others similarly afflicted until a
> suitable solution is found. So what's the fix for this going forward?
> We could add "UserKnownHostsFile=/dev/null" in our ssh configs, but
> that seems a bit extreme.
>
> As a final note I tested this with my script above (since it doesn't
> take long for known_hosts entries to build up) and got the following
> results which may give more/better clues.
>
> $ python test_connect.py
>
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296:
> UserWarning: implicit cast from 'char *' to a different pointer type:
> will be forbidden in the future (check that the types are as you
> expect; use an explicit ffi.cast() if they are correct)
> group, point, conversion, buf, buflen, bn_ctx
>
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114:
> UserWarning: implicit cast from 'char *' to a different pointer type:
> will be forbidden in the future (check that the types are as you
> expect; use an explicit ffi.cast() if they are correct)
> operation
> Traceback (most recent call last):
> File "test_connect.py", line 23, in <module>
> ssh.connect(**connect_args)
> File
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py",
> line 414, in connect
> raise BadHostKeyException(hostname, server_key, our_key)
> paramiko.ssh_exception.BadHostKeyException:
> ('mira041.front.sepia.ceph.com', <paramiko.ecdsakey.ECDSAKey object at
> 0x7fb534521890>, <paramiko.ecdsakey.ECDSAKey object at
> 0x7fb534ec8a10>)
>
> $ rm .ssh/known_hosts
>
> $ python test_connect.py
>
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296:
> UserWarning: implicit cast from 'char *' to a different pointer type:
> will be forbidden in the future (check that the types are as you
> expect; use an explicit ffi.cast() if they are correct)
> group, point, conversion, buf, buflen, bn_ctx
>
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114:
> UserWarning: implicit cast from 'char *' to a different pointer type:
> will be forbidden in the future (check that the types are as you
> expect; use an explicit ffi.cast() if they are correct)
> operation
>
/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:140:
> UserWarning: implicit cast from 'char *' to a different pointer type:
> will be forbidden in the future (check that the types are as you
> expect; use an explicit ffi.cast() if they are correct)
> self._backend._ffi.from_buffer(data), len(data)
> Hello World ! from mira041.front.sepia.ceph.com
>
> So it looks like the BadHostKeyException is the one we need to try and
> find a solution for I guess?
>
> HTH.
>
>
>
> On Thu, Aug 1, 2019 at 10:40 PM David Galloway <dgallowa(a)redhat.com> wrote:
>>
>> I failed to add some context to #3 in the wiki.
>>
>> The version of Paramiko was recently updated in teuthology. The newest
>> version doesn't support the default RSA key type in newer OpenSSH versions.
>>
>> See https://github.com/paramiko/paramiko/issues/1015
>>
>> On 7/31/19 7:24 PM, Brad Hubbard wrote:
>>> Another thing that makes me wonder about this is I can log onto
>>> machines that I already had reserved.
>>>
>>> bhubbard@teuthology:~$ ssh mira110
>>> Last login: Wed Jul 31 23:03:08 2019 from teuthology.front.sepia.ceph.com
>>>
>>> My key is 0600 and hasn't change/been modified for over a year.
>>>
>>>
>>> On Thu, Aug 1, 2019 at 8:56 AM Brad Hubbard <bhubbard(a)redhat.com>
wrote:
>>>>
>>>> Hi David,
>>>>
>>>> Thanks for the response. I would wonder why any of that would have
>>>> changed yesterday after successfully piloting teuthology for a few
>>>> years but I'll certainly look into that possibility.
>>>>
>>>> On Wed, Jul 31, 2019 at 9:55 PM David Galloway
<dgallowa(a)redhat.com> wrote:
>>>>>
>>>>> Is it #3 at
>>>>>
https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#sshexceptionerror_re…
>>>>>
>>>>> On 7/30/19 9:57 PM, Brad Hubbard wrote:
>>>>>> Also this one.
>>>>>>
>>>>>>
../src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/kex_ecdh_nist.py:39:
>>>>>> CryptographyDeprecationWarning: encode_point has been deprecated
on
>>>>>> EllipticCurvePublicNumbers and will be removed in a future
version.
>>>>>> Please use EllipticCurvePublicKey.public_bytes to obtain both
>>>>>> compressed and uncompressed point encoding.
>>>>>> m.add_string(self.Q_C.public_numbers().encode_point())
>>>>>>
>>>>>> On Wed, Jul 31, 2019 at 11:48 AM Brad Hubbard
<bhubbard(a)redhat.com> wrote:
>>>>>>>
>>>>>>> I updated teuthology yesterday and since then have seen a log
of the
>>>>>>> following errors
>>>>>>>
>>>>>>>
...src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/ecdsakey.py:164:
>>>>>>> CryptographyDeprecationWarning: Support for unsafe
construction of
>>>>>>> public numbers from encoded data will be removed in a future
version.
>>>>>>> Please use EllipticCurvePublicKey.from_encoded_point
>>>>>>> self.ecdsa_curve.curve_class(), pointinfo
>>>>>>>
>>>>>>>
>>>>>>> 2019-07-31 01:45:18,976.976
ERROR:paramiko.transport:Exception: Error
>>>>>>> reading SSH protocol banner
>>>>>>> 2019-07-31 01:45:18,976.976
ERROR:paramiko.transport:Traceback (most
>>>>>>> recent call last):
>>>>>>> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport: File
>>>>>>>
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py",
>>>>>>> line 1966, in run
>>>>>>> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport:
self._check_banner()
>>>>>>> 2019-07-31 01:45:18,977.977 ERROR:paramiko.transport: File
>>>>>>>
"/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py",
>>>>>>> line 2143, in _check_banner
>>>>>>> 2019-07-31 01:45:18,977.977 ERROR:paramiko.transport:
"Error
>>>>>>> reading SSH protocol banner" + str(e)
>>>>>>> 2019-07-31 01:45:18,977.977
ERROR:paramiko.transport:SSHException:
>>>>>>> Error reading SSH protocol banner
>>>>>>>
>>>>>>> Sometimes these are fatal and sometimes not. Wondering if
anyone else
>>>>>>> has seen them?
>>>>>>>
>>>>>>> --
>>>>>>> Cheers,
>>>>>>> Brad
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Brad
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Brad
>>>
>
>
>
> --
> Cheers,
> Brad
>