Please consider adding pytest-assume plugin to core plugins

[GeorgeFischhof]

Hi Folks,

please consider adding pytest-assume plugin to core plugins.

a few days ago I saw a nosetest using yields, and I did not understand it at first sight, checked the documentation and woking, and found that it is used to allow multiple asserts for one test case.

I checked this in PyTest with the following code:

def do_test_something(what):
    assert False, what

def test_something():
    yield do_test_something, 'one'
    yield do_test_something, 'two'

and found that PyTest knows this, and it is deprecated ("yield tests are deprecated, and scheduled to be removed in pytest 4.0") Remoing is not problem.

But it would be good to be able write more asserts / assumes for one test case

BR,
George

[pytestbot]

GitMate.io thinks possibly related issues are #946 (Merging pytest-capturelog into core), #1691 (Add a changelog to pytest-dev plugin requirements), #1803 (Integrate pytest-warnings plugin into core), #1652 (plugins resolved by pip instead of core pytest packages), and #1479 (pytest silently stops importing plugins on ImportError).

[nicoddemus]

Hi @GeorgeFischhof,

Thanks for the suggestion. Installing a plugin is very simple though, so I wonder if it makes sense to add this particular plugin to the core when one can just install it instead.

We try to avoid adding new things to the core specially if they are already available as external plugins; usually we merge new things to the core (which is rare) when they cover something missing from the standard library (logging the more recent case).

[nicoddemus]

Btw you can use parametrize instead of yield tests:

import pytest
@pytest.mark.parametrize('what', ['one', 'two'])
def test_something(what):
    assert False, what

Which I find much shorter and concise than yield tests. 😉

[RonnyPfannschmidt]

@GeorgeFischhof a prior version of pytest supported this, but as "yield tests" turn a test into a test genertor, they are being executed at collect time instead of test run time, thus creating completely invalid and broken results

as for doing multiple assertions per test - there is a basic plan to do this correctly which involves subtests, and allowing assertion-rewriting to be hook based (so assertions can be logged in a subtest instead of raised)

[GeorgeFischhof]

Hi folks,

thanks for the info, I have alredy know the parametrizing method, and saw the hooking tehnology.
(just forgot to mention it ...)

I had two reasons to request adding pytest-assume to core plugins.
1.) I work at a company, where the IT security is quite strict: we are not allowed (and not able) to install software from internet. We can use only on-boarded softwares. ( I have already requested pytest-assume, and it is on-boarded now. ) If someone works on similar place it will help him/her, if the assume exist at first moment. ;-)

2.) I use PyTest for functional testing as so many people does. In functional testing there is a need several times to check more things after each other (an action has several expected results, not parametrizable).
And for this purpose I found pytest-assume the best solution.
If assume would be a core plugin, people would know about it when they get PyTest, and they do not need to search for a solution.

BR,
George

[RonnyPfannschmidt]

for 1) please make them aware of a) the need to actually pin versions for that to be sensible and b) the need for exploration sessions and fast-path for requesting libs and their versions in development/prototyping

as for 2) currently pytest-assume is a really good solution for that,
however i believe the combination of "subtests" and assertion hooks will lead to a much more natural syntax for it (i hope we can experiment with it in near future)

[Sup3rGeo]

@GeorgeFischhof

I use PyTest for functional testing as so many people does. In functional testing there is a need several times to check more things after each other (an action has several expected results, not parametrizable).

Just for the sake of curiosity, can you give an example of an action with several expected results?

I do some functional testing but in my case I find that whenever I have a sequence of actions needed to test something, they are always dependent, in the sense that there is no reason to continue with the following steps (assertions) if a previous one has failed.
If they are not dependent, then they could be a different test function with common code factored as a fixture.

@RonnyPfannschmidt what do you mean by subtests (I thought pytest didn't support it) and assertion hooks?

[RonnyPfannschmidt]

@Sup3rGeo correct subtests are not supported, but i consider it a required feature for those kinds of tests

[GeorgeFischhof]

Hi @Sup3rGeo ,

example:
webshop, the customer paid with bank card, payment page redirects the customer to webshop; things to check:

• appropriate page displayed
• application log record written
• order record created in the db
• inventory decreased accordingly
• email sent to user
• book income increased
• and whatever can happen yet

Because a black-box functional test can be executed end to end. And everything must be checked / tested.

Of course it could be called subtest as well. Here is a syntax:

@pytest.mark.subtest(sequence=1)
def test_one():
    do_something()
    assert one

@pytest.mark.subtest(sequence=2)
def test_two():
    assert two

@pytest.mark.subtest(sequence=2)
def test_three():
    assert three

It would mean that the tests with same sequence number can be executed parallelly. (for example by using xdist) And test with different numbers must executed in order of sequence. ...

BR:
George

[RonnyPfannschmidt]

@GeorgeFischhof it has to be noted that your write-up is absolutely unrelated to subtests as known by unittest - dependent tests are a different beast

[Zac-HD]

Closing because upstream plugin development remains active - as https://github.com/okken/pytest-check - and IMO it would be a mistake to freeze the featureset when people are still working out the best way to do this.