Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate shrinking benchmark more #4214

Merged
merged 4 commits into from
Dec 27, 2024

Conversation

tybug
Copy link
Member

@tybug tybug commented Dec 24, 2024

Brings more of the shrinking benchmark into pytest for automation. Also replaces our plotting code with a vega specification. The plotting code requires a dependency on vl-convert to generate the plot. I know about altair, but to my understanding that only aims to represent vega-lite specs, and we are moderately abusing vega (not lite) to have two axes.

Here's a benchmark result where old == new, which gives an idea of the variability in current shrinking. 10 trials (5 old, 5 new), 95% CI. blue is absolute difference, red is relative difference. (@DRMacIver you might find this interesting!)

shrinking

@tybug
Copy link
Member Author

tybug commented Dec 24, 2024

I'm happy to report that putting try_trivial_examples first in #4203 was the right decision! Though I would warn against putting too much stock in this benchmark, as our tests are likely biased more towards unusual edge cases than normal shrinks.

try_trivial_examples first try_trivial_examples after node_program("X")
shrinking_first shrinking_after

Graphs are relative to the shrinker without try_trivial_examples.

@tybug tybug merged commit 3dbfae2 into HypothesisWorks:master Dec 27, 2024
49 checks passed
@tybug tybug deleted the automate-shrinking-benchmark branch December 27, 2024 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants