Elide internal frames from tracebacks, to focus on user code

[Zac-HD]

Closes #848, with an approach adapted from trio.

Basically we just strip frames off the traceback until we reach one that did not come from a Hypothesis file, attach the shortened traceback to the exception, and raise it. Then our @proxies decorator overwrites the frame location so it doesn't look like a Hypothesis file... so there's a cute trick where we inject an entry into the frame globals and check for that too when unwinding.

comparison of old and new output from pytest

Here's a very simple test suite:

from hypothesis import given
from hypothesis.strategies import integers

@given(integers())
def test_simple(x):
    assert x

@given(integers())
def test_multiple_failures(x):
    if x > 100:
        raise ValueError("This number is too big!")
    elif x < -100:
        raise RuntimeError("This number is too small!")

After this change, here's the output from pytest:

================================== FAILURES ===================================
_________________________________ test_simple _________________________________
Traceback (most recent call last):
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 5, in test_simple
    def test_simple(x):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 958, in wrapped_test
    raise the_error_hypothesis_found
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 6, in test_simple
    assert x
AssertionError: assert 0
--------------------------------- Hypothesis ----------------------------------
Falsifying example: test_simple(x=0)
___________________________ test_multiple_failures ____________________________
Traceback (most recent call last):
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 9, in test_multiple_failures
    def test_multiple_failures(x):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 958, in wrapped_test
    raise the_error_hypothesis_found
hypothesis.errors.MultipleFailures: Hypothesis found 2 distinct failures.
--------------------------------- Hypothesis ----------------------------------
Falsifying example: test_multiple_failures(x=-101)
Traceback (most recent call last):
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 13, in test_multiple_failures
    raise RuntimeError("This number is too small!")
RuntimeError: This number is too small!

Falsifying example: test_multiple_failures(x=101)
Traceback (most recent call last):
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 11, in test_multiple_failures
    raise ValueError("This number is too big!")
ValueError: This number is too big!

While here's what it looked like before:

================================== FAILURES ===================================
_________________________________ test_simple _________________________________
Traceback (most recent call last):
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 5, in test_simple
    def test_simple(x):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 924, in wrapped_test
    state.run()
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 666, in run
    falsifying_example.__expected_traceback,
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 525, in execute
    result = self.test_runner(data, run)
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\executors.py", line 58, in default_new_style_executor
    return function(data)
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 523, in run
    return test(*args, **kwargs)
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 5, in test_simple
    def test_simple(x):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 470, in test
    result = self.test(*args, **kwargs)
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 6, in test_simple
    assert x
AssertionError: assert 0
--------------------------------- Hypothesis ----------------------------------
Falsifying example: test_simple(x=0)
___________________________ test_multiple_failures ____________________________
Traceback (most recent call last):
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 9, in test_multiple_failures
    def test_multiple_failures(x):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 924, in wrapped_test
    state.run()
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 723, in run
    len(self.falsifying_examples,)))
hypothesis.errors.MultipleFailures: Hypothesis found 2 distinct failures.
--------------------------------- Hypothesis ----------------------------------
Falsifying example: test_multiple_failures(x=-101)
Traceback (most recent call last):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 666, in run
    falsifying_example.__expected_traceback,
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 525, in execute
    result = self.test_runner(data, run)
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\executors.py", line 58, in default_new_style_executor
    return function(data)
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 523, in run
    return test(*args, **kwargs)
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 9, in test_multiple_failures
    def test_multiple_failures(x):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 470, in test
    result = self.test(*args, **kwargs)
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 13, in test_multiple_failures
    raise RuntimeError("This number is too small!")
RuntimeError: This number is too small!

Falsifying example: test_multiple_failures(x=101)
Traceback (most recent call last):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 666, in run
    falsifying_example.__expected_traceback,
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 525, in execute
    result = self.test_runner(data, run)
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\executors.py", line 58, in default_new_style_executor
    return function(data)
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 523, in run
    return test(*args, **kwargs)
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 9, in test_multiple_failures
    def test_multiple_failures(x):
  File "c:\users\zac\documents\github\hypothesis\hypothesis-python\src\hypothesis\core.py", line 470, in test
    result = self.test(*args, **kwargs)
  File "C:\Users\Zac\Documents\GitHub\hypothesis\test.py", line 11, in test_multiple_failures
    raise ValueError("This number is too big!")
ValueError: This number is too big!

[Zac-HD]

Um. Looks like this feature is going to be Python 3 only; it's just not possible to handle tracebacks like this in Python 2. At least the compatible form isn't too inelegant...

[njsmith]

On Python 2 you might be able to make it work by using 3-argument raise: https://docs.python.org/2/reference/simple_stmts.html#the-raise-statement

But I'm not sure it's worth bothering, since it's, y'know, python 2.

I hadn't realized that hypothesis has a MultipleFailures type... @1st1 and I spent a bunch of time last week sketching out a plan to add a built-in multi-exception type to Python 3.8, so we should talk about that and make sure it will work for hypothesis too. (There's some initial discussion at python-trio/trio#611, and the prototype will be at https://github.com/python-trio/exceptiongroup, once I get a chance to push some actual code.)

[Zac-HD]

Yuck, three-argument raise is a SyntaxError on Python 3 - I'm not keen to use it when we'd have to add an extra ugly stack frame to use it anyway. Thanks for pointing it out though!

Definitely keen to join the multi-exception conversation; and eventually to use the standard exception type like everyone else. I've even been quietly subscribed to the Trio issue for a while 😄

[Zac-HD]

...on the other hand, it turns out that the clip_traceback helper I wrote already works on Python 2, and at that point it's not that ugly to support both. So now this works on all Pythons 🐍 ✨

[DRMacIver]

In general I have a very strong preference against if PY2 code outside of compat.py. Can this not be refactored to a function?

[Zac-HD]

Not without adding another stack frame, and we've just gone to a fair bit of trouble to get rid of those! Worse, we'd have that extra frame on all Python 3 as well 😭

Otherwise it absolutely would be compat.raise_with_traceback(error, tb).

[DRMacIver]

Fair enough. Could you add an explanatory comment to the code then, please?

(In general I think this section could use some fairly verbose commenting because even knowing what it's supposed to do I'm finding it a bit eyebrow raising)

[Zac-HD]

Yep, comments incoming. I'll also try to refactor things a bit so the logic is as clear as possible without comments too!

[DRMacIver]

Sorry to expand the scope slightly but... I think this needs a flag of some sort to turn it off, e if .g. an internal use environment variable. Otherwise it has the potential to be very annoying for us when debugging things going wrong.

[DRMacIver]

I'm not sure "letting tracebacks behave like they normally do" counts as leaking information details.

[Zac-HD]

Sorry to expand the scope slightly but... I think this needs a flag of some sort to turn it off, e if .g. an internal use environment variable. Otherwise it has the potential to be very annoying for us when debugging things going wrong.

It carefully checks, and will not change any traceback raised from a Hypothesis file unless the exception is a MultipleErrors. verbosity=Verbosity.debug also disables it, since it's... well, verbose and good for debugging (FYI there's a good chance of a PR for #1238 tomorrow from a local Hactoberfest sprint). I think that covers what you had in mind?

[Zac-HD]

🎉

I'm going to wait for the ongoing GitHub incident to be fully resolved, then rebase on master before merging to avoid any conflicts or build/deploy errors. But soon!