On Sat, Aug 3, 2019 at 1:18 AM David Galloway <dgallowa@redhat.com> wrote:
Ah, it wasn't the issue I thought it was. I helped a new lab user recently and ran into the "BEGIN OPENSSH PRIVATE KEY" thing and the errors looked similar.
I've always had "UserKnownHostsFile /dev/null" in my SSH config so I've never seen this problem personally. I don't think it's unreasonable to have that config set in a lab environment especially considering the smithi and mira host keys change every ~45min.
OK, I guess that's fair enough. I thought relaxing StrictHostKeyChecking was sufficient but I guess not.
I've updated the Sepia wiki with a recommended SSH config.
Thanks David. Hopefully not too many more will get caught out then. I updated https://tracker.ceph.com/issues/14500 with this information since some people may land there (like I did).
On 8/2/19 3:41 AM, Brad Hubbard wrote:
Hmmm....trying to understand this.
$ head -1 .ssh/id_rsa -----BEGIN RSA PRIVATE KEY-----
So that is *not* the format causing the errors mentioned in https://github.com/paramiko/paramiko/issues/1015 right?
I also do *not* see the "not a valid [key] private key file" mentioned.
Also...
$ cat test_connect.py import os import paramiko
user = "ubuntu" host = "mira110.front.sepia.ceph.com" ssh = paramiko.SSHClient()
ssh.load_system_host_keys() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
connect_args = dict( hostname=host, username=user, timeout=60 )
ssh_config_path = os.path.expanduser("~/.ssh/config") if os.path.exists(ssh_config_path): ssh_config = paramiko.SSHConfig() ssh_config.parse(open(ssh_config_path)) opts = ssh_config.lookup(host)
ssh.connect(**connect_args) (stdin, stdout, stderr) = ssh.exec_command("echo 'Hello World ! from ' $(hostname)") for line in stdout.readlines(): print(line) ssh.close()
$ source src/teuthology/virtualenv/bin/activate
$ python test_connect.py /home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct) group, point, conversion, buf, buflen, bn_ctx /home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct) operation /home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:140: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct) self._backend._ffi.from_buffer(data), len(data) Hello World ! from mira110
Also, looking at the keys repo it seems most people are using rsa keys so I'm not sure it's the key that's the problem.
2019-08-02 03:36:03,603.603 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2019-08-02 03:36:03,603.603 ERROR:paramiko.transport:Traceback (most recent call last): 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: File "/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 1966, in run 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: self._check_banner() 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: File "/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 2143, in _check_banner 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2019-08-02 03:36:03,604.604 ERROR:paramiko.transport:SSHException: Error reading SSH protocol banner
The above seems to be the main problem but unfortunately that seems to be a "catch-all" error for any problem with the connection.
So I mentioned earlier this problem didn't seem to affect every connection (some appeared to "work" for some level of "work" :P) so why is that?
After considerable efforts to uncover the differences I realised that the rare systems where the connections succeeded were not in my known_hosts file and therefore generated no warnings about 'known_hosts' entries when logging on (I have "StrictHostKeyChecking no" in my ssh config file so the warnings are generated, but ignored). So I deleted my known_hosts file and tried again and it worked. This really is pretty piss-poor error reporting.
So I guess the latest teuthology update makes teuthology intolerant of these 'known_hosts' warnings generated for ssh connections (or something related in some way to that). Given I'm likely to not be the only one stung by this and given the cryptic nature of the debugging efforts I'm hoping this will help others similarly afflicted until a suitable solution is found. So what's the fix for this going forward? We could add "UserKnownHostsFile=/dev/null" in our ssh configs, but that seems a bit extreme.
As a final note I tested this with my script above (since it doesn't take long for known_hosts entries to build up) and got the following results which may give more/better clues.
$ python test_connect.py /home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct) group, point, conversion, buf, buflen, bn_ctx /home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct) operation Traceback (most recent call last): File "test_connect.py", line 23, in <module> ssh.connect(**connect_args) File "/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 414, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: ('mira041.front.sepia.ceph.com', <paramiko.ecdsakey.ECDSAKey object at 0x7fb534521890>, <paramiko.ecdsakey.ECDSAKey object at 0x7fb534ec8a10>)
$ rm .ssh/known_hosts
$ python test_connect.py /home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ec.py:296: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct) group, point, conversion, buf, buflen, bn_ctx /home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:114: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct) operation /home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:140: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct) self._backend._ffi.from_buffer(data), len(data) Hello World ! from mira041.front.sepia.ceph.com
So it looks like the BadHostKeyException is the one we need to try and find a solution for I guess?
HTH.
On Thu, Aug 1, 2019 at 10:40 PM David Galloway <dgallowa@redhat.com> wrote:
I failed to add some context to #3 in the wiki.
The version of Paramiko was recently updated in teuthology. The newest version doesn't support the default RSA key type in newer OpenSSH versions.
See https://github.com/paramiko/paramiko/issues/1015
On 7/31/19 7:24 PM, Brad Hubbard wrote:
Another thing that makes me wonder about this is I can log onto machines that I already had reserved.
bhubbard@teuthology:~$ ssh mira110 Last login: Wed Jul 31 23:03:08 2019 from teuthology.front.sepia.ceph.com
My key is 0600 and hasn't change/been modified for over a year.
On Thu, Aug 1, 2019 at 8:56 AM Brad Hubbard <bhubbard@redhat.com> wrote:
Hi David,
Thanks for the response. I would wonder why any of that would have changed yesterday after successfully piloting teuthology for a few years but I'll certainly look into that possibility.
On Wed, Jul 31, 2019 at 9:55 PM David Galloway <dgallowa@redhat.com> wrote:
Is it #3 at https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#sshexceptionerror_rea...
On 7/30/19 9:57 PM, Brad Hubbard wrote: > Also this one. > > ../src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/kex_ecdh_nist.py:39: > CryptographyDeprecationWarning: encode_point has been deprecated on > EllipticCurvePublicNumbers and will be removed in a future version. > Please use EllipticCurvePublicKey.public_bytes to obtain both > compressed and uncompressed point encoding. > m.add_string(self.Q_C.public_numbers().encode_point()) > > On Wed, Jul 31, 2019 at 11:48 AM Brad Hubbard <bhubbard@redhat.com> wrote: >> >> I updated teuthology yesterday and since then have seen a log of the >> following errors >> >> ...src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/ecdsakey.py:164: >> CryptographyDeprecationWarning: Support for unsafe construction of >> public numbers from encoded data will be removed in a future version. >> Please use EllipticCurvePublicKey.from_encoded_point >> self.ecdsa_curve.curve_class(), pointinfo >> >> >> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport:Exception: Error >> reading SSH protocol banner >> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport:Traceback (most >> recent call last): >> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport: File >> "/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", >> line 1966, in run >> 2019-07-31 01:45:18,976.976 ERROR:paramiko.transport: self._check_banner() >> 2019-07-31 01:45:18,977.977 ERROR:paramiko.transport: File >> "/home/bhubbard/src/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", >> line 2143, in _check_banner >> 2019-07-31 01:45:18,977.977 ERROR:paramiko.transport: "Error >> reading SSH protocol banner" + str(e) >> 2019-07-31 01:45:18,977.977 ERROR:paramiko.transport:SSHException: >> Error reading SSH protocol banner >> >> Sometimes these are fatal and sometimes not. Wondering if anyone else >> has seen them? >> >> -- >> Cheers, >> Brad > > >
-- Cheers, Brad
-- Cheers, Brad
-- Cheers, Brad
-- Cheers, Brad