Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up calculation of example groups used by the mutator #2390

Merged
merged 4 commits into from
Apr 12, 2020

Conversation

Zalathar
Copy link
Contributor

My profiling of #2308 found that Hypothesis currently spends a lot of time generating the example groupings that the mutator uses to insert duplicate values.

I found that by switching to a dedicated @calculated_example_property implementation, I could speed up those calculations by a decent amount.

In my informal testing, this cuts about 10 seconds out of a 50 second test run, which is a nice improvement.

@Zalathar Zalathar added the performance go faster! use less memory! label Apr 11, 2020
@Zalathar Zalathar requested a review from DRMacIver as a code owner April 11, 2020 10:48
@Zalathar
Copy link
Contributor Author

The first two commits are just preliminary refactoring; all of the interesting changes are in the third commit (Speed up calculation...).

@Zalathar
Copy link
Contributor Author

Looks like there's some sort of subtle interaction with test_text_containment[1] in the quality suite.

I'll have to investigate whether this is a real problem, or some quirk of the test itself.

Copy link
Member

@DRMacIver DRMacIver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for continuing to look into this!

@DRMacIver
Copy link
Member

Looks like there's some sort of subtle interaction with test_text_containment[1] in the quality suite.

I'll have to investigate whether this is a real problem, or some quirk of the test itself.

Oops, I missed this when I approved. I still think the change looks good. Let me know if you want me to rereview when you've fixed this.

@Zalathar Zalathar force-pushed the mutator-groups branch 2 times, most recently from 6005b6d to e765f88 Compare April 11, 2020 14:03
@Zalathar
Copy link
Contributor Author

The bad news is that something cursed is happening in the test suite on Python 3.5.

The good news is that it's consistent (and locally reproducible via build.sh), so at least I can investigate it.

https://dev.azure.com/HypothesisWorks/Hypothesis/_build/results?buildId=3649&view=logs&j=bf737dbf-2401-5929-1d68-053a1a5f2d7c&t=aa278db6-67f9-574a-fa2c-7dd4d540935b

@Zac-HD
Copy link
Member

Zac-HD commented Apr 12, 2020

I'd be OK with just dropping support for Python 3.5 as soon as keeping it is causing noticably more work. It's ~3% of our downloads, no longer supported by Numpy et al, and pretty close to EOL (September) anyway. I think @DRMacIver feels the same way?

@Zalathar
Copy link
Contributor Author

For what it's worth I don't think 3.5 is actually at fault here; this is probably just a deterministically-flaky test that happens to get unlucky on 3.5 after these changes.

@Zalathar Zalathar force-pushed the mutator-groups branch 3 times, most recently from b92348d to 790198f Compare April 12, 2020 06:08
@Zalathar
Copy link
Contributor Author

Yeah, I'm pretty sure the underlying test (#2322) is theoretically flaky, which normally isn't noticeable because it uses derandomize=True under the hood. So it consistently passes or fails at any point, but unrelated changes can make it suddenly break on a particular platform/version.

For the purposes of this PR I've worked around the problem by just increasing max_examples for that particular test, so it's much harder to get unlucky enough to actually fail.

@Zalathar
Copy link
Contributor Author

Oh, it was already fixed by 2fc4a55, so I can just rebase.

This will let us extract the mutator into its own method.
Using calculated_example_property here is faster than the previous
implementation.
@Zalathar Zalathar merged commit ceaa526 into HypothesisWorks:master Apr 12, 2020
@Zalathar Zalathar deleted the mutator-groups branch April 12, 2020 09:48
@DRMacIver
Copy link
Member

I'd be OK with just dropping support for Python 3.5 as soon as keeping it is causing noticably more work. It's ~3% of our downloads, no longer supported by Numpy et al, and pretty close to EOL (September) anyway. I think @DRMacIver feels the same way?

I'd be pretty OK with dropping it from our CI and putting it in a "Still technically supported but we're offloading testing to our users" state until EOL and then formally dropping it in September. I don't mind formally dropping it sooner than that but I think I'd need a reasonably convincing reason.

@Zac-HD
Copy link
Member

Zac-HD commented Apr 15, 2020

I don't see any real benefit in dropping 3.5 without also doing the code cleanups that will enable, so let's just keep it - and keep all two CI jobs testing it - until EOL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance go faster! use less memory!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants