[3.4] bpo-26470: Port ssl and hashlib module to OpenSSL 1.1.0.

[vstinner]

Backport notes

• Don't add PROTOCOL_TLS
• Ignore documentation changes

(cherry picked from commit 598894f)

https://bugs.python.org/issue26470

[vstinner]

Without this change, the compilation the _ssl and _hashlib modules of Python 3.4.10rc1 fails on Fedora 29 with OpenSSL 1.1.1. The compat-openssl10 package will be retired from Fedora 30 and so it will no longer be possible to use Python 3.4 on recent Linux distributions.

This PR is a partial backport of the the commit 598894f from 3.5 to 3.4. I reverted changes related to BIO (Python 3.4 doesn't have ssl.MemoryBIO) and I adapted the doc changes.

I tried to not add ssl.PROTOCOL_TLS, but the the key thing of the change is in context_new():

-    else if (proto_version == PY_SSL_VERSION_SSL23)
-        ctx = SSL_CTX_new(SSLv23_method());
+    else if (proto_version == PY_SSL_VERSION_TLS)
+        ctx = SSL_CTX_new(TLS_method());

Maybe it would be possible to keep PY_SSL_VERSION_SSL23 but replace SSLv23_method() with TLS_method()?

[vstinner]

Maybe it would be possible to keep PY_SSL_VERSION_SSL23 but replace SSLv23_method() with TLS_method()?

Oh. I didn't notice that TLS_method is an alias to SSLv23_method :-D

 #define TLS_method SSLv23_method

So I don't think that adding PROTOCOL_TLS is a requirement. I'm reworking my PR to simplify it.

[vstinner]

Ok, I simplified the backport even more to not add PROTOCOL_TLS. On Fedora 29 with this PR, test_hashlib pass but 2 test_ssl tests fail. IMHO these 2 failures are acceptable since the same 2 tests are failing in 3.5 as well, and I don't expect a better ssl support in 3.4 than in 3.5 :-)

======================================================================
FAIL: test_options (test.test_ssl.ContextTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/prog/python/3.4/Lib/test/test_ssl.py", line 685, in test_options
    self.assertEqual(default, ctx.options)
AssertionError: 2181169236 != 2182217812

======================================================================
FAIL: test_default_ecdh_curve (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/prog/python/3.4/Lib/test/test_ssl.py", line 2651, in test_default_ecdh_curve
    self.assertIn("ECDH", s.cipher()[0])
AssertionError: 'ECDH' not found in 'TLS_AES_256_GCM_SHA384'

[hroncok]

The compat-openssl10 package will be retired from Fedora 30

Most likely Fedora 31.

[vstinner]

@stratakis @tiran: Would you mind to review this change?

[vstinner]

The compat-openssl10 package will be retired from Fedora 31.

For the Fedora background, see https://bugzilla.redhat.com/show_bug.cgi?id=1685612

[vstinner]

Note on the CI: AppVeyor uses openssl-1.0.2k. I'm not sure about the OpenSSL version used by Travis CI, but it's also 1.0.x if I recall correctly.

[vstinner]

It seems like my 3.4 checkout was outdated. I rebased this PR on top on 3.4.

(I also squashed my "Remove unused BIO_up_ref()" fix into the main commit.)

[vstinner]

@stratakis: Would you mind to review this PR?

[stratakis]

I'm taking a look

[stratakis]

The only change in the initial commit for the ssl.py, is PROTOCOL_SSLv23 changing to PROTOCOL_TLS, and since they are aliases it seems reasonable to ignore those changes for simplifying the backport, as you did.

[stratakis]

The rest of the test changes in the original commit are for test cases that do not exist in Python 3.4

[stratakis]

This was removed here: bc26646 essentially for dropping compatibility with older than 0.9.8 openssl versions.

[stratakis]

I left some comments. The rest of the changes look good to me and I'm able to compile Python 3.4 on my system with OpenSSL 1.1.1.

I get 3 test failures, but they shouldn't really matter at this point:

FAIL: test_options (test.test_ssl.ContextTests)

Traceback (most recent call last):
  File "/home/Harris/dev/cpython/Lib/test/test_ssl.py", line 707, in test_options
    self.assertEqual(default, ctx.options)
AssertionError: 2181169236 != 2182217812


FAIL: test_default_ciphers (test.test_ssl.ThreadedTests)

Traceback (most recent call last):
  File "/home/Harris/dev/cpython/Lib/test/test_ssl.py", line 2655, in test_default_ciphers
    s.connect((HOST, server.port))
AssertionError: OSError not raised


FAIL: test_default_ecdh_curve (test.test_ssl.ThreadedTests)

Traceback (most recent call last):
  File "/home/Harris/dev/cpython/Lib/test/test_ssl.py", line 2673, in test_default_ecdh_curve
    self.assertIn("ECDH", s.cipher()[0])
AssertionError: 'ECDH' not found in 'TLS_AES_256_GCM_SHA384'

[larryhastings]

Sorry, folks, but I just can't bring myself to merge this. It's too big of a last-minute change, and it introduces test failures. Also, it looks like Christian did the original work to add OpenSSL 1.1 support in 2016, and at the time he backported to 3.5--but not 3.4. If he didn't backport it then, I don't know why we need to do it now. (Sadly the bpo page sheds no light on why he didn't backport it then. My guess is it's viewed as more of a bugfix / feature than a security fix, which would still be true now.)

[vstinner]

Larry: As I told you, I'm perfectly fine with no applying this change to Python 3.4 upstream. We (Red Hat) can easily maintain a downstream patch on the Fedora python34 package.

@larryhastings:

Sorry, folks, but I just can't bring myself to merge this. It's too big of a last-minute change, and it introduces test failures.

This change doesn't make tests to fail. The difference is that without my change, Python 3.4 fails to build with OpenSSL 1.1.1. With my change, not only compilation succeed but almost all tests pass. The 3 failures as expected (see below).

Also, it looks like Christian did the original work to add OpenSSL 1.1 support in 2016, and at the time he backported to 3.5--but not 3.4. If he didn't backport it then, I don't know why we need to do it now. (Sadly the bpo page sheds no light on why he didn't backport it then. My guess is it's viewed as more of a bugfix / feature than a security fix, which would still be true now.)

Time changed since 2016. More and more Linux distributions switched to OpenSSL 1.1.x.

@stratakis:

This was removed here: bc26646 essentially for dropping compatibility with older than 0.9.8 openssl versions.

I don't think that it's a good idea to drop support for old OpenSSL versions from Python 3.4.

I get 3 test failures, but they shouldn't really matter at this point:

I have the same in Python 3.5, so they are not caused by my backport but are "expected". I don't think that it's worth it to fix these tests in Python 3.4 and 3.5.

--

@stratakis: Let's apply this change downstream ;-)

[stratakis]

Thanks for your work Victor on this PR and I understand Larry's point of view that this could be considered too big of a change for a Python version, soon to go EOL. I'll be fine with maintaining that downstream.