Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--benchmark-disable still saves json files #205

Closed
TauPan opened this issue Jun 11, 2021 · 1 comment · Fixed by #207
Closed

--benchmark-disable still saves json files #205

TauPan opened this issue Jun 11, 2021 · 1 comment · Fixed by #207

Comments

@TauPan
Copy link
Contributor

TauPan commented Jun 11, 2021

Hi, I just upgraded from 3.2.3 to 3.4.1 (more or less by accident) and noticed that benchmark now saves json files if --benchmark-disable is given together with --benchmark-autosave

(I use these flags in pytest.ini so I can just give --benchmark-enable and don't need to give autosave as well. Maybe I should not do that, I don't know.)

I think the previous behaviour was to issue a warning.

There are a couple of problems I see with this:

1.) I get separate json files for each process I run with pytest-xdist
2.) Those json files contain no actual benchmark information
3.) This throws pytest-watch run with e.g. --spool 1000 -- --testmon --testmon-forceselect which will interrupt itself every time the json dump is written. (Of course --ignore=.benchmarks helps against that.)
4.) Those json files clutter the checkout.

My personal opinion is that saving benchmark data makes no sense if no actual benchmarks were run, so I'm happier with the warning in that case.

Any opinion or advice is appreciated here. I'm considering either downgrading, or changing the flags in pytest.ini, since I do rarely run benchmarks. The --ignore flag for pytest only helps against problem number 3.

TauPan added a commit to TauPan/pytest-benchmark that referenced this issue Jun 11, 2021
@TauPan
Copy link
Contributor Author

TauPan commented Jun 11, 2021

Looking at the code this looks like an oversight to me, so I created a PR.

TauPan added a commit to TauPan/pytest-benchmark that referenced this issue Jun 11, 2021
ionelmc pushed a commit that referenced this issue Jun 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant