Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report separate statistics for test cases in reuse, generate, and shrink phases #1555

Closed
Zac-HD opened this issue Sep 1, 2018 · 3 comments · Fixed by #2422
Closed

Report separate statistics for test cases in reuse, generate, and shrink phases #1555

Zac-HD opened this issue Sep 1, 2018 · 3 comments · Fixed by #2422
Assignees
Labels
legibility make errors helpful and Hypothesis grokable

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Sep 1, 2018

As discussed in #1554, merging all of these phases into one set of statistics makes them quite difficult to interpret if a test is or recently has been failing. The count of invalid examples tends to be higher, and it would be nice to distinguish generated, replayed, and shrunk test cases.

This will involve changes to the recording in hypothesis.internal.conjecture.engine, and to the reporting in hypothesis.statistics. Only phases with more than zero examples should be reported, to avoid e.g. pointless lines of statistics for passing tests.

@Zac-HD Zac-HD added the legibility make errors helpful and Hypothesis grokable label Sep 1, 2018
@Zalathar
Copy link
Contributor

Zalathar commented Sep 5, 2018


~~~(Haven't decided whether I'll do the front-end work as well, but I'll probably get around to it.)~~~

@Zalathar Zalathar assigned Zalathar and unassigned Zalathar Sep 9, 2018
@Zalathar
Copy link
Contributor

Zalathar commented Sep 9, 2018

I've since been distracted by other things, and might not get back to this myself.

@Zac-HD Zac-HD assigned Zac-HD and unassigned Zac-HD Sep 24, 2018
@Zac-HD
Copy link
Member Author

Zac-HD commented Sep 24, 2018

The reporting side of this should be relatively easy to build on top of #1577, and after that's merged I can have a go at the collection side too... but if anyone else wants to try go for it, and I'll post again before I start.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
legibility make errors helpful and Hypothesis grokable
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants