Fwd: Missing symbol under tox in openssl stuff
Hi, I'm the first to acknowledge that I do not know enough of python. But still I can get by most of the times. However during the tests of my Ceph port one of the tests complains: ============== orchestrator/_interface.py:701: ImportError ------------------------------ Captured log call ------------------------------- ERROR orchestrator._interface:_interface.py:391 _Promise failed Traceback (most recent call last): File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 334, in do_work res = self._on_complete_(*args, **kwargs) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 398, in call_self return f(self, *inner_args) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 2352, in _create_grafana return self._create_daemon('grafana', daemon_id, host) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 1874, in _create_daemon j = self._generate_grafana_config() File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 2288, in _generate_grafana_config cert, pkey = create_self_signed_cert('Ceph', 'cephadm') File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/mgr_util.py", line 134, in create_self_signed_cert from OpenSSL import crypto File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/crypto.py", line 15, in <module> from OpenSSL._util import ( File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/_util.py", line 6, in <module> from cryptography.hazmat.bindings.openssl.binding import Binding File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module> from cryptography.hazmat.bindings._openssl import ffi, lib ImportError: /home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "SSLv3_client_method" ============== This is due to the fact that on FreeBSD openSSL has its SSLv3 code disabled. Now this is fixable on an individual basis, by recompiling the openSSL port with SSLv3 enabled. But for a generic port that is not really an option. The user than has to jump thru loops to build its own openSSL, and even then he/she needs to keep up with security isssues. One should not want this. The problem stems from virtualenv/tox fetching from public sources, instead of using the ports system. This can be overruled by: --system-site-packages. I know that I'll need to load all packages before running virtualenv/tox, but that is "just" a matter of collecting the list. but I'm wondering if this is a feasable solution? --WjW
On Fri, Feb 28, 2020 at 5:39 PM Willem Jan Withagen <wjw@digiware.nl> wrote:
Hi,
I'm the first to acknowledge that I do not know enough of python. But still I can get by most of the times.
However during the tests of my Ceph port one of the tests complains: ==============
orchestrator/_interface.py:701: ImportError ------------------------------ Captured log call ------------------------------- ERROR orchestrator._interface:_interface.py:391 _Promise failed Traceback (most recent call last): File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 334, in do_work res = self._on_complete_(*args, **kwargs) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 398, in call_self return f(self, *inner_args) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 2352, in _create_grafana return self._create_daemon('grafana', daemon_id, host) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 1874, in _create_daemon j = self._generate_grafana_config() File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 2288, in _generate_grafana_config cert, pkey = create_self_signed_cert('Ceph', 'cephadm') File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/mgr_util.py", line 134, in create_self_signed_cert from OpenSSL import crypto File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/crypto.py", line 15, in <module> from OpenSSL._util import ( File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/_util.py", line 6, in <module> from cryptography.hazmat.bindings.openssl.binding import Binding File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module> from cryptography.hazmat.bindings._openssl import ffi, lib ImportError: /home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "SSLv3_client_method"
==============
This is due to the fact that on FreeBSD openSSL has its SSLv3 code disabled.
if that's the case, OPENSSL_NO_SSL3_METHOD should be defined, and hence cryptography should have this symbol defined, see https://github.com/pyca/cryptography/blob/7247665f76cf849fb5b3020a28cfc86c40... . could you check /usr/local/include/openssl/opensslfeatures.h on your build host?
Now this is fixable on an individual basis, by recompiling the openSSL port with SSLv3 enabled. But for a generic port that is not really an option. The user than has to jump thru loops to build its own openSSL, and even then he/she needs to keep up with security isssues. One should not want this.
The problem stems from virtualenv/tox fetching from public sources, instead of using the ports system. This can be overruled by: --system-site-packages. I know that I'll need to load all packages before running virtualenv/tox, but that is "just" a matter of collecting the list.
but I'm wondering if this is a feasable solution?
--WjW
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Regards Kefu Chai
On 28-2-2020 13:59, kefu chai wrote:
On Fri, Feb 28, 2020 at 5:39 PM Willem Jan Withagen <wjw@digiware.nl> wrote:
Hi,
I'm the first to acknowledge that I do not know enough of python. But still I can get by most of the times.
However during the tests of my Ceph port one of the tests complains: ==============
orchestrator/_interface.py:701: ImportError ------------------------------ Captured log call ------------------------------- ERROR orchestrator._interface:_interface.py:391 _Promise failed Traceback (most recent call last): File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 334, in do_work res = self._on_complete_(*args, **kwargs) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 398, in call_self return f(self, *inner_args) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 2352, in _create_grafana return self._create_daemon('grafana', daemon_id, host) File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 1874, in _create_daemon j = self._generate_grafana_config() File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/cephadm/module.py", line 2288, in _generate_grafana_config cert, pkey = create_self_signed_cert('Ceph', 'cephadm') File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/mgr_util.py", line 134, in create_self_signed_cert from OpenSSL import crypto File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/crypto.py", line 15, in <module> from OpenSSL._util import ( File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/OpenSSL/_util.py", line 6, in <module> from cryptography.hazmat.bindings.openssl.binding import Binding File "/home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module> from cryptography.hazmat.bindings._openssl import ffi, lib ImportError: /home/jenkins/workspace/ceph-master/src/pybind/mgr/.tox/py3/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "SSLv3_client_method"
==============
This is due to the fact that on FreeBSD openSSL has its SSLv3 code disabled.
if that's the case, OPENSSL_NO_SSL3_METHOD should be defined, and hence cryptography should have this symbol defined, see https://github.com/pyca/cryptography/blob/7247665f76cf849fb5b3020a28cfc86c40... .
could you check /usr/local/include/openssl/opensslfeatures.h on your build host?
That file does (no longer) exist in my /usr/local... I checked by looking in the ports makefile, which registers the defaults. And you can check this when running make config, it'll also tell you that SSLv3 is off. What I do find in /usr/local/include/openssl is: /usr/local/include/openssl/ssl.h:DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *SSLv3_method(void)) /* SSLv3 */ /usr/local/include/openssl/ssl.h:DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *SSLv3_server_method(void)) /usr/local/include/openssl/ssl.h:DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *SSLv3_client_method(void)) And the openssl in base is 1.1.0, and in ports is 1.1.1 so in both it should be deprecated.... But in the ports Makefile it is not default on: OPTIONS_DEFAULT=ASM ASYNC CT GOST DES EC MAN3 MD4 NEXTPROTONEG RC2 RC4 \ RMD160 SCTP SHARED SSE2 THREADS TLS1 TLS1_1 TLS1_2 Now this issue comes in a few flavours: Currently this only breaks in master, when with run-tox-mgr. This I can fix on by jenkins builder by manually installing a libcrypto that does take SSLv3. As long as the ports I make do not include cephadm, things will be fine. I still have not finished all the preparations for the ceph15 port. Ceph14-14.2.7 has just been submitted. So that will oke as long as nothing is being backported. Once that happens those versions will also suffer. Question is what happens when I release a package with cephadm in it which requires py-cryptography? As long as the FreeBSD port of that also does not need SSLv3 we're oke. Which will be since there is no tox involved. So for this it is only a problem when tox does not install the FreeBSD py-cryptography port but tries to fix it by itself. Hence my question if it would be sensible to get virtualenv/tox/pip/.... to prefer FreeBSD ports first. Or maybe even get src/tools/setup-virtualenv.sh or src/script/run_tox.sh to install if first before tox installs even more things (using py3??). Would that make sense? Now for something totally different with openssl versions: I have a similar problem of incompatible crypto I think when building my bhyve/rbd-device plugin where as things can get incompatible due to: src/common/ceph_crypto.h: namespace ceph::crypto::ssl { # if OPENSSL_VERSION_NUMBER < 0x10100000L And that if I try to run my plugin with a librados which is compiled on a platform with openssl < 0x10100000L it will crash in HMAC routines. Took me a while to figure that out. But that is more a development issue and I'm running way too many different combinations of release of Ceph and FreeBSD. Perhaps it is possible to glue a const in like namespace ceph::crypto::ssl { ........... public: const int rados_uses_ssl_version = OPENSSL_VERSION_NUMBER; Such that it is possible to ERR-out with a decent errormessage if not all versions match??? But then this needs to be mapped to C-code also, to be workable. --WjW
participants (2)
-
kefu chai
-
Willem Jan Withagen