-
Notifications
You must be signed in to change notification settings - Fork 591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speed up calculation of example groups used by the mutator #2390
Conversation
The first two commits are just preliminary refactoring; all of the interesting changes are in the third commit ( |
Looks like there's some sort of subtle interaction with I'll have to investigate whether this is a real problem, or some quirk of the test itself. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for continuing to look into this!
Oops, I missed this when I approved. I still think the change looks good. Let me know if you want me to rereview when you've fixed this. |
6005b6d
to
e765f88
Compare
The bad news is that something cursed is happening in the test suite on Python 3.5. The good news is that it's consistent (and locally reproducible via |
I'd be OK with just dropping support for Python 3.5 as soon as keeping it is causing noticably more work. It's ~3% of our downloads, no longer supported by Numpy et al, and pretty close to EOL (September) anyway. I think @DRMacIver feels the same way? |
For what it's worth I don't think 3.5 is actually at fault here; this is probably just a deterministically-flaky test that happens to get unlucky on 3.5 after these changes. |
b92348d
to
790198f
Compare
Yeah, I'm pretty sure the underlying test (#2322) is theoretically flaky, which normally isn't noticeable because it uses For the purposes of this PR I've worked around the problem by just increasing |
Oh, it was already fixed by 2fc4a55, so I can just rebase. |
This will let us extract the mutator into its own method.
Using calculated_example_property here is faster than the previous implementation.
790198f
to
c9ccd20
Compare
I'd be pretty OK with dropping it from our CI and putting it in a "Still technically supported but we're offloading testing to our users" state until EOL and then formally dropping it in September. I don't mind formally dropping it sooner than that but I think I'd need a reasonably convincing reason. |
I don't see any real benefit in dropping 3.5 without also doing the code cleanups that will enable, so let's just keep it - and keep all two CI jobs testing it - until EOL. |
My profiling of #2308 found that Hypothesis currently spends a lot of time generating the example groupings that the mutator uses to insert duplicate values.
I found that by switching to a dedicated
@calculated_example_property
implementation, I could speed up those calculations by a decent amount.In my informal testing, this cuts about 10 seconds out of a 50 second test run, which is a nice improvement.