Pytest not returning errors where I expect them

[ShmuelTreiger]

When I run the following code, pytest gives me a failure. From my understanding based on the docs and a question I asked on this plugin it should be returning an error.

I made a custom plugin just for this problem and the root of the issue seems to be that both setup and teardown pass, while call fails, hence why it's registering as a failure and not an error. I guess the better question is why doesn't setup fail?

I'm running on windows using pytest 3.1.2. Pip list shouldn't matter, though I will note that I don't have the above mentioned plugin installed and that my custom plugin only prints information.

Sorry, this might not be a bug, this might just be my misunderstanding, so I decided to post both here and on stackoverflow. Thank you for the help.

    import pytest
    import unittest
    
    class base(unittest.TestCase):
        def setUp(self):
            raise ValueError
    
    class test(base):
        def test(self):
            print("Hello")
    
    pytest.main()

[dhoomakethu]

what is the output of your test run? and what is the name of your test file. make sure the name of the file begins with test

[RonnyPfannschmidt]

unless im mistaken, that kind of usage should be pytest.main([__file__])and is discouraged

[ShmuelTreiger]

@dhoomakethu The output can be seen in the attached results. Note that I use a custom plugin to print out whether or not a section passed (i.e. setup passed, call failed, teardown passed). The name of the file is just "test.py".

@RonnyPfannschmidt Can you please elaborate/point me toward the proper resource? I'm unfamiliar with what you mean

results.pdf

[RonnyPfannschmidt]

@ShmuelTreiger i reviewed, and this is a necessary mismatch for unittest compatibility unittest setup method cannot be part of pytest setup/teardown machinery due to massive differences in the control flow

@nicoddemus is this already documented somewhere, i recall you detailing out that part of the docs lately

[nicoddemus]

@RonnyPfannschmidt I clarified a few points in the latest unittest-related documentation, but I didn't touched the point about the slight setUp/fixtures incompatibilities.

I made a custom plugin just for this problem and the root of the issue seems to be that both setup and teardown pass, while call fails

You got it right, the problem here is that all setUp/tearDown handling of unittest classes is done at the call stage, that's why your pytest_runtest_logreport sees a passing setup and a failing call. This is a fundamental incompatibility on how unittest support is implemented in Pytest in order to also support most of unittest features that happen behind the scenes (in this specific case this is done like this to support collecting errors/warnings/failures by a TestResult class).

At some point someone suggested (I think it was @hpk42) an alternative implementation by implementing setUp and tearDown support by means of injecting a special autouse fixture in unittest subclasses.

For illustration, this simple unittest class:

class Test(TestCase):

    def setUp(self):
        self.protocol = Protocol()

    def tearDown(self):
        self.protocol.close()

    def test_protocol(self):
        ...

Would look like this behind the scenes after collection:

class Test(TestCase):

    @pytest.fixture(autouse=True)
    def _internal_pytest_setup_teardown(self):
        # injected by pytest automatically during collection
        self.setUp()
        yield
        self.tearDown()

    def setUp(self):
        self.protocol = Protocol()

    def tearDown(self):
        self.protocol.close()

    def test_protocol(self):
        ...

This would make setUp/tearDown first-class fixtures and avoid this kind of surprises. At first glance this seems simple to implement, but:

• This would break running frameworks which extend unittest like twisted.trial.
• This would break backward compatibility, and I think that mixing unittest with fixtures happen specially in large test suites that started to make use of pytest features gradually, and would be a pain to fix.
• Other mechanisms would require more thought to support properly, for example addCleanup.

@ShmuelTreiger please let us know if you have more questions or some of the details are not clear. 👍

[ShmuelTreiger]

I understand. Thank you so much for the thorough and detailed response. Really helps me conceptualize and understand what's going on here.

Also, I hadn't come across the unittest page yet, and it looks super helpful, so thank you for that as well.

[nicoddemus]

Great, glad it was useful!