Understand and fix hypothesis issues

[jacg]

We need to invest some time and effort into understanding and fixing the increasing number of problems that we are seeing in our hypothesis tests.

EDIT THIS COMMENT TO PROVIDE MORE DETAILS AND LINKS TO RELEVANT PRs etc. ... when I have some time ..

[jacg]

It seems that many of the problems that we have seen in recent weeks were related to the evolution of the coverage feature in hypothesis itself. For example, see this issue in the hypothesis repo.

It seems that much of this is fixed in hypothesis 3.32, as can be seen from the timings in the table in the comments of #357. #372 Proposes an immediate bump of hypothesis to 3.32.

[jjgomezcadenas]

Is this issue then solved @jacg ?

[jacg]

Oooooh, no you don't!

The whole point of this issue is to leave a reminder that some careful thought has to be invested into understanding exactly whether and how naive use of hypothesis is responsible for some of the problems we are seeing.

The whole point is that this has to be done carefully when time allows.

So closing it in a rush in a midnight pogrom of frail, defenceless and unloved orphan PR's is precicesly not what we want to do.

[jjgomezcadenas]

Well, I am glad that I asked...

[Zac-HD]

Hi @jacg, I'm a maintainer on Hypothesis, and I'd love to work through any problems you've been having so we can fix them (with either code or better documentation 😄)

If your issues were just performance-related, we're aware of them and working on those upstream - both in the issue linked above and further upstream in Coverage itself. Otherwise, I'll need some more information to help!

[jacg]

Hi @Zac-HD,

Yes, I know who you are: I have a couple of PRs open in the hypothesis repo, and you have been helping me there.

Just like those PRs, this issue is something to which I fully intend to return, but at the moment I am overstretched on things which have a higher priority for me.

Thank you very much for reaching out with the offer, which I will take up ... when I get the time to work on these things again.