Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support -x to fail fast #134

Merged
merged 2 commits into from
Jul 7, 2024
Merged

Support -x to fail fast #134

merged 2 commits into from
Jul 7, 2024

Conversation

rhoban13
Copy link
Contributor

Attempts to progress on #23. Submitting as a draft to see if I can get some advice.

Summary: Support for pytest -x to stop after the first failing subtest. The implementation here does stop after the first failing subtest, however it also:

  1. Marks the parent test as a failure
    b/c
  2. The session.Failed exception raised from inside the subtests.test is treated as an uncaught exception from within the test. This results in the stack trace printed pointing to contextlib which feels undesirable to me.

I think the behavior in (1) is debatable. On one hand - a failing subtest does not by default fail the parent test. On the other hand, by design the parent test will not complete. In either case, (2) I think would be really confusing for users.

rhoban13
rhoban13 commented Jun 28, 2024
View reviewed changes
tests/test_subtests.py Outdated Show resolved Hide resolved
@nicoddemus
Copy link
Member

Thanks @rhoban13!

I've been pretty busy lately, but I will do my best to set aside some time this week to review this.

@nicoddemus
Copy link
Member

Thanks @rhoban13 for getting this ball rolling.

Your gut feeling of just re-raising the exception is the right approach. The ugly message we see then was because of @contextmanager -- refactoring SubTests.test to a custom context manager solves that issue, albeit not as good looking/clean as before, but a small price to pay.

@nicoddemus nicoddemus marked this pull request as ready for review July 7, 2024 12:46
@nicoddemus nicoddemus force-pushed the exitfirst_working branch from 3d9b5da to 2ca28de Compare July 7, 2024 13:00
rhoban13 and others added 2 commits July 7, 2024 10:02
@rhoban13 @nicoddemus
Support -x/--exitfirst
5ff5be6 
In order to support that and provide a good error message, we needed to stop using @contextmanager so we could have full control over the error stack.

Fix pytest-dev#23
@nicoddemus
Refactor capturing_output and capturing_logs into functions
f03ed7a
@nicoddemus nicoddemus force-pushed the exitfirst_working branch from 2ca28de to f03ed7a Compare July 7, 2024 13:03
@nicoddemus nicoddemus merged commit 96c8408 into pytest-dev:main Jul 7, 2024
14 checks passed
@nicoddemus
Copy link
Member

0.13.0 has been released. 🎉

@rhoban13 rhoban13 mentioned this pull request Jul 7, 2024
Merged
github-merge-queue bot referenced this pull request in linz/topo-imagery Sep 9, 2024
@dependabot
build(deps-dev): bump pytest-subtests from 0.11.0 to 0.13.1 (#1055)
0be07c8 
Bumps [pytest-subtests](https://github.com/pytest-dev/pytest-subtests)
from 0.11.0 to 0.13.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pytest-dev/pytest-subtests/releases">pytest-subtests's
releases</a>.</em></p>
<blockquote>
<h2>v0.13.1</h2>
<p>No release notes provided.</p>
<h2>v0.13.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Fix compatibility with pytest 8.1 by <a
href="https://github.com/nicoddemus"><code>@​nicoddemus</code></a> in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/126">pytest-dev/pytest-subtests#126</a></li>
<li>[pre-commit.ci] pre-commit autoupdate by <a
href="https://github.com/pre-commit-ci"><code>@​pre-commit-ci</code></a>
in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/127">pytest-dev/pytest-subtests#127</a></li>
<li>[pre-commit.ci] pre-commit autoupdate by <a
href="https://github.com/pre-commit-ci"><code>@​pre-commit-ci</code></a>
in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/128">pytest-dev/pytest-subtests#128</a></li>
<li>[pre-commit.ci] pre-commit autoupdate by <a
href="https://github.com/pre-commit-ci"><code>@​pre-commit-ci</code></a>
in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/129">pytest-dev/pytest-subtests#129</a></li>
<li>[pre-commit.ci] pre-commit autoupdate by <a
href="https://github.com/pre-commit-ci"><code>@​pre-commit-ci</code></a>
in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/130">pytest-dev/pytest-subtests#130</a></li>
<li>Hide <code>SubTests.test()</code> from tracebacks by <a
href="https://github.com/bswck"><code>@​bswck</code></a> in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/131">pytest-dev/pytest-subtests#131</a></li>
<li>[pre-commit.ci] pre-commit autoupdate by <a
href="https://github.com/pre-commit-ci"><code>@​pre-commit-ci</code></a>
in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/132">pytest-dev/pytest-subtests#132</a></li>
<li>[pre-commit.ci] pre-commit autoupdate by <a
href="https://github.com/pre-commit-ci"><code>@​pre-commit-ci</code></a>
in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/133">pytest-dev/pytest-subtests#133</a></li>
<li>[pre-commit.ci] pre-commit autoupdate by <a
href="https://github.com/pre-commit-ci"><code>@​pre-commit-ci</code></a>
in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/135">pytest-dev/pytest-subtests#135</a></li>
<li>Support -x to fail fast by <a
href="https://github.com/rhoban13"><code>@​rhoban13</code></a> in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/134">pytest-dev/pytest-subtests#134</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/bswck"><code>@​bswck</code></a> made
their first contribution in <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/131">pytest-dev/pytest-subtests#131</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/pytest-dev/pytest-subtests/compare/v0.12.1...v0.13.0">https://github.com/pytest-dev/pytest-subtests/compare/v0.12.1...v0.13.0</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pytest-dev/pytest-subtests/blob/main/CHANGELOG.rst">pytest-subtests's
changelog</a>.</em></p>
<blockquote>
<h2>0.13.1 (2024-07-16)</h2>
<ul>
<li>Fixed bug were an extra test would execute when
<code>-x/--exitfirst</code> was used
(<code>[#139](https://github.com/pytest-dev/pytest-subtests/issues/139)</code>_).</li>
</ul>
<p>.. _<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/139">#139</a>:
<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/139">pytest-dev/pytest-subtests#139</a></p>
<h2>0.13.0 (2024-07-07)</h2>
<ul>
<li>Dropped support for EOL Python 3.7.</li>
<li>Added support for <code>-x/--exitfirst</code>
(<code>[#134](https://github.com/pytest-dev/pytest-subtests/issues/134)</code>_).</li>
<li>Hide the traceback inside the <code>SubTests.test()</code> method
(<code>[#131](https://github.com/pytest-dev/pytest-subtests/issues/131)</code>_).</li>
</ul>
<p>.. _<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/131">#131</a>:
<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/131">pytest-dev/pytest-subtests#131</a>
.. _<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/134">#134</a>:
<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/134">pytest-dev/pytest-subtests#134</a></p>
<h2>0.12.1 (2024-03-07)</h2>
<ul>
<li>Fixed compatibility with upcoming pytest <code>8.1.x</code>.
(<code>[#125](https://github.com/pytest-dev/pytest-subtests/issues/125)</code>_).</li>
</ul>
<p>.. _<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/125">#125</a>:
<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/125">pytest-dev/pytest-subtests#125</a></p>
<h2>0.12.0 (2024-03-06)</h2>
<ul>
<li>Python 3.12 is now officially supported
(<code>[#113](https://github.com/pytest-dev/pytest-subtests/issues/113)</code>_).</li>
<li>Added typing support
(<code>[#115](https://github.com/pytest-dev/pytest-subtests/issues/115)</code>_).</li>
<li><code>SubTests</code> can be imported from
<code>pytest_subtests</code> to type-annotate the <code>subtests</code>
fixture.</li>
</ul>
<p>.. _<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/113">#113</a>:
<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/113">pytest-dev/pytest-subtests#113</a>
.. _<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/115">#115</a>:
<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/pull/115">pytest-dev/pytest-subtests#115</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/3671b40691440fcb01e96e346220ac4fe62d3580"><code>3671b40</code></a>
Release 0.13.1</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/c02c55e23c5e3b1b354716d6a93b88a9336d207b"><code>c02c55e</code></a>
Fix off by one error when using -x (<a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/139">#139</a>)</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/cbff3e1547238584a2cf9be9aa0b0da8d8cb9bc3"><code>cbff3e1</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/138">#138</a>
from nicoddemus/adjust-deploy</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/c2ccbfb53d02e7f6d3166055e73808141f12faed"><code>c2ccbfb</code></a>
Fix deploy and update instructions</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/358790b3283c8e5faf5a71aa0dfd0e17365626a5"><code>358790b</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest-subtests/issues/137">#137</a>
from pytest-dev/release-0.13.0</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/3c6a72c0f7a579c887d8a74cd587359db83cfb24"><code>3c6a72c</code></a>
Automatically merge branch during deploy</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/ed8f899e3e347d88c2bf12a29748aeb217c3218a"><code>ed8f899</code></a>
Create a GH release during deploy</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/cb5a9229de26f115e2bd4f856a68267ca95bf1e4"><code>cb5a922</code></a>
Update CHANGELOG</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/79a6bc14d9479d9cda2b0fb9e9bb32da203bc033"><code>79a6bc1</code></a>
Simplify CI configuration</li>
<li><a
href="https://github.com/pytest-dev/pytest-subtests/commit/eef3c597e1f973cd77246318bd44b1847db17b08"><code>eef3c59</code></a>
Drop support for Python 3.7</li>
<li>Additional commits viewable in <a
href="https://github.com/pytest-dev/pytest-subtests/compare/v0.11.0...v0.13.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pytest-subtests&package-manager=pip&previous-version=0.11.0&new-version=0.13.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nicoddemus nicoddemus nicoddemus approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rhoban13 @nicoddemus