Regression: M2Crypto.m2.rand_bytes() hangs on Apache+mod_wsgi with pyOpenSSL 16.1.10

[porridge]

Since upgrading:

• cryptography (1.4) -> (1.5)
• pyOpenSSL (16.0.0) -> (16.1.0)

my web app reliably hangs, in a simple M2Crypto.m2.rand_bytes(16) call.
At the same time entropy_avail hovers around 800, and calling the function from a standalone python interperter works just fine. Apparently the issue is only when running inside Apache.

Downgrading cryptography alone (to 1.4) does not help. But downgrading pyOpenSSL (to 16.0.0) as well, makes it work again. I did not try downgrading pyOpenSSL alone.

I suspect this is something along the lines of pyca/cryptography#2299, but I don't pretend to understand the details.

Example stack trace of a hanging request thread:

Thread 14 (Thread 0x7f5f0cb9b700 (LWP 88)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007f5f0ea5b6d8 in PyThread_acquire_lock (
    lock=lock@entry=0x7f5f1450f9e0, waitflag=waitflag@entry=1)
    at ../Python/thread_pthread.h:324
#2  0x00007f5f0ea80386 in PyEval_RestoreThread (
    tstate=tstate@entry=0x7f5ef8bbc620) at ../Python/ceval.c:357
#3  0x00007f5f0ea807c8 in PyGILState_Ensure () at ../Python/pystate.c:611
#4  0x00007f5effa44fbd in gil_ensure () at c/misc_thread_common.h:113
#5  0x00007f5effa4a001 in cffi_call_python (
    externpy=0x7f5efff3f0a0 <_cffi_externpy__Cryptography_rand_bytes>, 
    args=0x7f5f0cb99f70 "\220S\270\370^\177") at c/call_python.c:234
#6  0x00007f5effc9843c in Cryptography_rand_bytes (a0=<optimized out>, 
    a1=<optimized out>) at build/temp.linux-x86_64-2.7/_openssl.c:11769
#7  0x00007f5efccdc223 in rand_bytes ()
   from /usr/lib/python2.7/dist-packages/M2Crypto/__m2crypto.so
#8  0x00007f5efccdc2c6 in ?? ()
   from /usr/lib/python2.7/dist-packages/M2Crypto/__m2crypto.so
#9  0x00007f5f0ea6f0d4 in call_function (oparg=<optimized out>, 
    pp_stack=0x7f5f0cb9a110) at ../Python/ceval.c:4020
#10 PyEval_EvalFrameEx (f=f@entry=0x7f5ef8089f00, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#11 0x00007f5f0ea6f059 in fast_function (nk=<optimized out>, na=3, n=3, 
    pp_stack=0x7f5f0cb9a290, func=0x7f5efcbf6b18) at ../Python/ceval.c:4106
#12 call_function (oparg=<optimized out>, pp_stack=0x7f5f0cb9a290)
    at ../Python/ceval.c:4041
#13 PyEval_EvalFrameEx (f=f@entry=0x7f5efea89050, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#14 0x00007f5f0ea6f059 in fast_function (nk=<optimized out>, na=2, n=2, 
    pp_stack=0x7f5f0cb9a410, func=0x7f5f05934758) at ../Python/ceval.c:4106
#15 call_function (oparg=<optimized out>, pp_stack=0x7f5f0cb9a410)
    at ../Python/ceval.c:4041
#16 PyEval_EvalFrameEx (f=f@entry=0x7f5efcbf8b60, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#17 0x00007f5f0ea7054d in PyEval_EvalCodeEx (co=<optimized out>, 
    globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, 
    argcount=argcount@entry=1, kws=0x7f5ef8ca9af8, kwcount=0, defs=0x0, 
    defcount=0, closure=closure@entry=0x0) at ../Python/ceval.c:3252
#18 0x00007f5f0ea6edd8 in fast_function (nk=<optimized out>, na=1, n=1, 
    pp_stack=0x7f5f0cb9a630, func=0x7f5f0592b9b0) at ../Python/ceval.c:4116
#19 call_function (oparg=<optimized out>, pp_stack=0x7f5f0cb9a630)
    at ../Python/ceval.c:4041
#20 PyEval_EvalFrameEx (f=f@entry=0x7f5ef8ca9940, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#21 0x00007f5f0ea6f059 in fast_function (nk=<optimized out>, na=3, n=3, 
    pp_stack=0x7f5f0cb9a7b0, func=0x7f5f05939758) at ../Python/ceval.c:4106
#22 call_function (oparg=<optimized out>, pp_stack=0x7f5f0cb9a7b0)
    at ../Python/ceval.c:4041
#23 PyEval_EvalFrameEx (f=f@entry=0x7f5ef8c92240, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#24 0x00007f5f0ea7054d in PyEval_EvalCodeEx (co=<optimized out>, 
    globals=<optimized out>, locals=locals@entry=0x0, 
    args=args@entry=0x7f5f13acbc98, argcount=3, kws=kws@entry=0x0, 
    kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, 
    closure=0x0) at ../Python/ceval.c:3252
#25 0x00007f5f0eaa56d0 in function_call (func=0x7f5f05939848, 
    arg=0x7f5f13acbc80, kw=0x0) at ../Objects/funcobject.c:526
#26 0x00007f5f0ea11d43 in PyObject_Call (func=func@entry=0x7f5f05939848, 
    arg=arg@entry=0x7f5f13acbc80, kw=kw@entry=0x0)
    at ../Objects/abstract.c:2529
#27 0x00007f5f0e99d7bd in instancemethod_call.8988 (func=0x7f5f05939848, 
    arg=0x7f5f13acbc80, kw=0x0) at ../Objects/classobject.c:2602
#28 0x00007f5f0ea11d43 in PyObject_Call (func=func@entry=0x7f5efeb238c0, 
    arg=arg@entry=0x7f5efcbae4d0, kw=kw@entry=0x0)
    at ../Objects/abstract.c:2529
#29 0x00007f5f0ea167e5 in slot_tp_call.26779 (self=<optimized out>, 
    args=0x7f5efcbae4d0, kwds=0x0) at ../Objects/typeobject.c:5432
#30 0x00007f5f0ea11d43 in PyObject_Call (func=func@entry=0x7f5efcc3fb10, 
    arg=arg@entry=0x7f5efcbae4d0, kw=<optimized out>)
    at ../Objects/abstract.c:2529
#31 0x00007f5f0ea8a577 in PyEval_CallObjectWithKeywords (func=0x7f5efcc3fb10, 
    arg=0x7f5efcbae4d0, kw=<optimized out>) at ../Python/ceval.c:3889
#32 0x00007f5f0ee5d644 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so
#33 0x00007f5f0ee63ba8 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so
#34 0x00007f5f131b3184 in start_thread (arg=0x7f5f0cb9b700)
    at pthread_create.c:312
#35 0x00007f5f12ee037d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[reaperhulk]

Does this problem go away if you do:

from cryptography.hazmat.backends.openssl.backend import backend
backend.activate_builtin_random()

right after importing pyopenssl?

[porridge]

I don't import pyopenssl directly, I use it through M2Crypto. Does it make sense to do this import and call after importing M2Crypto? Or should I do it before?

[reaperhulk]

I'm afraid I'm confused. m2crypto is a completely separate project that just happens to also use OpenSSL. Why would M2Crypto import pyOpenSSL?

[porridge]

The best theory I can come up with is that my code has an indirect dependency on pyOpenSSL (via some other library we happen to use), and there is some bad interaction between M2Crypto's and pyOpenSSL's use of openssl from within the python runtime embedded in Apache :-( Debugging this is going to be a bit of a nightmare I'm afraid :-/

[reaperhulk]

Yeah that's a strange setup. I suspect doing from cryptography.hazmat.backends.openssl.backend import backend ; backend.activate_builtin_random() in your code before you invoke M2Crypto.m2.rand_bytes will fix it though.

[hynek]

Has this been fixed by the 16.2.0 release?

[reaperhulk]

Yes, sorry should have closed this one :)

[porridge]

Any pointers to how this was fixed?

[reaperhulk]

There's some discussion on #542 and this is the PR that fixes it: #552

There's shared global state in the OpenSSL backend related to the way we do the random callback for cryptography. This was previously not activated for pyOpenSSL but as an import side effect it started occurring in 16.1. 16.2 moves that import to be scoped specifically to the methods that require it, so it doesn't actually fix the problem but it restores functionality to all existing code while we fix it more permanently.

[mcepl]

Hi, this is M2Crypto maintainer here. First of all, why in the world this bug was filed here, where I haven't learn about it and not in https://gitlab.com/m2crypto/m2crypto/issues , if it is a M2Crypto issue? However, if the bug was filed there, I would most likely close it as an invalid issue. Let me explain.

In my opinion, it is absolutely unnecessary for either M2Crypto or python-cryptography to provide ANY random numbers. We do it only because of providing legacy API (one of the topmost goals of M2Crypto project is to support its legacy users, so stability of API is our top goal) and perhaps for those really few who can require some weird hardware entropy generator (I don't even know, whether M2Crypto would be able to support it, OpenSSL does). For all other users, which is 99% of all projects I suppose, random module from the standard library, or open('/dev/urandom', 'rb').read(20) provides exactly the same random data (probably the former is better, because it is platform independent).

Also, why in the world do you feel the need to combine M2Crypto with pyOpenSSL, when the former‘s capabilities are superset of the latter's ones? Linking your program to OpenSSL twice (which is what you do) is effort carrying too many risks of something going wrong, that I would certainly declare it as unsupported and thus close the bug.

[tiran]

No, the random is a terrible choice if your need cryptographically secure pseudo random bits. Only random.SystemRandom is secure. open('/dev/urandom', 'rb').read(20) is also problematic for other reasons. The os.urandom() function is the correct choice here. It provides an interface on top of the best CSPRNG source on every platform.