Runtime Strict xfail

[jbrockmendel]

I frequently have to xfail one case out of a whole bunch of parametrized test cases, e.g.:

@pytest.mark.parametrize('box', [pd.Index, pd.Series, pd.DataFrame])
@pytest.mark.parametrize('other', [3.14, np.float64(3.14), np.array([3.14])])
@pytest.mark.parametrize('moon', ["Full", "New", "Blue"])
def test_thing(self, box, other, moon):
    if box is pd.DataFrame and isinstance(other, np.ndarray) and moon == "Blue":
        pytest.xfail(reason="...")

I would like to be able to make this xfail "strict". AFAIK the only way to do this would be to explicitly write out all the parameter combinations and xfail the appropriate one:

@pytest.mark.parametrize('box, other, moon', [
    (pd.Index, 3.14, "Full"),
    ...
    pytest.param(pd.DataFrame, np.array([3.14]), "Blue", marks=pytest.mark.xfail(reason="...", strict=True, raises=ValueError)),
    ...
]
def test_thing(self, box, other, moon):
)

But that would get really verbose and defeat the point of a really nice pytest feature.

Would it be feasible to make pytest.xfail(reason="...", strict=True, raises=ValueError) not immediately raise, but instead continue the test and behave as if it had been xfailed from outside? (Is the idea clear?)

[asottile]

a bit of a long shot, but you might find the helper I wrote here useful

[jbrockmendel]

Thanks for taking a look. Sounds like a pipe dream for the time being. Closing.