salt-ssh: ssl.SSLError: unknown error (_ssl.c:2788)

See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=910236

As per Debian bug, importing the module directly / running the problematic code in python interpreter works fine.

On debugging, I found another error reproducible with the following:

---
>>> import ssl
>>> if ssl is not None:
... # Note that the naming of ssl.Purpose is confusing; the purpose
... # of a context is to authentiate the opposite side of the connection.
... _client_ssl_defaults = ssl.create_default_context(
... ssl.Purpose.SERVER_AUTH)
... _server_ssl_defaults = ssl.create_default_context(
... ssl.Purpose.CLIENT_AUTH)
... if hasattr(ssl, 'OP_NO_COMPRESSION'):
... # See netutil.ssl_options_to_context
... _client_ssl_defaults.options |= ssl.OP_NO_COMPRESSION
... _server_ssl_defaults.options |= ssl.OP_NO_COMPRESSION
...
>>> import salt.utils.rsax931
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/salt/utils/rsax931.py", line 85, in <module>
    libcrypto = _init_libcrypto()
  File "/usr/lib/python3/dist-packages/salt/utils/rsax931.py", line 76, in _init_libcrypto
    raise OSError("Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)")
OSError: Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)
---

I found an upstream bug (https://github.com/saltstack/salt/issues/46884) which has already been fixed with following patch.

https://patch-diff.githubusercontent.com/raw/saltstack/salt/pull/48580.patch

Back to the original error, this is a minimal test for it.

---
>>> from ctypes import cdll
>>> libcrypto = cdll.LoadLibrary('libcrypto.so.1.1')
>>> libcrypto.OPENSSL_init_crypto(140, None)
1
>>> import ssl
>>> ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/ssl.py", line 502, in create_default_context
    context = SSLContext(PROTOCOL_TLS)
  File "/usr/lib/python3.6/ssl.py", line 391, in __new__
    self = _SSLContext.__new__(cls, protocol)
ssl.SSLError: unknown error (_ssl.c:2788)
---

If you reverse the ssl and libcrypto parts, OPENSSL_init_crypto instead returns zero. So the underlying problem is the order of which imports are done. Tornado (ergo ssl) _must_ be imported first before the libcrypto library loading in salt are ran, otherwise the program dies.

I couldn't see any specific fix in salt relating to this error, however the package version (2017.7.4+dfsg1-1) is sorely out of date with the current release (2018.3.x), maybe this new version fixes this indirectly with some import refactorings.

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: salt-ssh 2017.7.4+dfsg1-1
ProcVersionSignature: Ubuntu 4.18.0-12.13-generic 4.18.17
Uname: Linux 4.18.0-12-generic x86_64
ApportVersion: 2.20.10-0ubuntu13.1
Architecture: amd64
CurrentDesktop: pop:GNOME
Date: Fri Dec 21 12:17:09 2018
PackageArchitecture: all
SourcePackage: salt
UpgradeStatus: Upgraded to cosmic on 2018-12-02 (18 days ago)
modified.conffile..etc.default.apport: [modified]
modified.conffile..etc.salt.roster: [modified]
mtime.conffile..etc.default.apport: 2014-11-26T20:33:01.966514
mtime.conffile..etc.salt.roster: 2018-07-26T00:31:37.371528