Pytest-order breaks the fixture scope for parametrized fixtures

[mrbean-bremen]

This came up in a StackOverflow question. In the following code:

@pytest.fixture(params=[1, 2], scope="class")
def number(request):
    print(f"setup number {request.param}")
    yield request.param
    print(f"teardown number {request.param}")

@pytest.mark.usefixtures("number")
class TestClass:
    @pytest.mark.order(2)
    def test1(self):
        pass

    @pytest.mark.order(1)
    def test2(self):
        pass

the test fixture will be called twice instead of once for each parameter. Removing the order markers fixes this.
The same happens for session scope.

I'm not entirely sure how the correct behavior should be. Without the ordering, with the class scope the tests are reordered so that all tests with the same parameter are run consecutively - this allows the class scope to work.
With ordering, it propably shall behave the same, e.g. the ordering shall happen for each parameter separately.

[mrbean-bremen]

Copying the slightly adapted answer from Stack Overflow here:

This is actually expected behavior and not a bug, because the tests are explicitely ordered, and this ordering is done after any initial ordering due to the fixture usage.

There is a possibility to change this behavior, if using the option --indulgent-ordering. This changes the sequence of the ordering, so that the tests are first ordered by the plugin, and afterwards by the fixture.

Here is the behavior without the option:

$ python -m pytest --collect-only
<Package tests>
  <Module test_one.py>
    <Class TestClass>
        <Function test2[1]>
        <Function test2[2]>
        <Function test1[1]>
        <Function test1[2]>

With the --indulgent-ordering option, you get instead:

$ python -m pytest --collect-only --indulgent-ordering
<Package tests>
  <Module test_one.py>
    <Class TestClass>
        <Function test2[1]>
        <Function test1[1]>
        <Function test2[2]>
        <Function test1[2]>

which makes it possible for the fixture to properly work as a class-scoped fixture.

As usually, you can add the option to your pytest.ini if you want it to be effective for all tests:

[pytest]
addopts = --indulgent-ordering