Test failures with test_not_literal_with_ignorecase

[felixonmars]

Hypothesis 3.30.0, Python 2.7.13, Pytest 3.2.2, Arch Linux x86_64.

A simple retry made it passing, so I guess it's another case of flaky test.

=================================== FAILURES ===================================
__________________ test_not_literal_with_ignorecase[[^a][^b]] __________________
[gw0] linux2 -- Python 2.7.13 /usr/bin/python2
tests/cover/test_regex.py:153: in test_not_literal_with_ignorecase
    lambda s: s[0] not in (u'a', u'A') and s[1] not in (u'b', u'B')
tests/common/debug.py:127: in assert_all_examples
    assert_examples()
tests/common/debug.py:123: in assert_examples
    def assert_examples(s):
src/hypothesis/executors.py:58: in default_new_style_executor
    return function(data)
src/hypothesis/core.py:140: in run
    return test(*args, **kwargs)
tests/common/debug.py:123: in assert_examples
    def assert_examples(s):
src/hypothesis/core.py:602: in timed_test
    result = test(*args, **kwargs)
tests/common/debug.py:124: in assert_examples
    assert predicate(s), \
E   AssertionError: Found u'A00' using strategy from_regex(re.compile(ur'[^a][^b]', re.IGNORECASE)) which does not match
E   assert False
E    +  where False = <function <lambda> at 0x7f48cf556a28>('A00')
----------------------------- Captured stdout call -----------------------------
Falsifying example: assert_examples(s=u'A00')
You can add @seed(239957681856335131410575018211164889450) to this test or run pytest with --hypothesis-seed=239957681856335131410575018211164889450 to reproduce this failure.
________________ test_not_literal_with_ignorecase[(?i)[^a][^b]] ________________
[gw0] linux2 -- Python 2.7.13 /usr/bin/python2
tests/cover/test_regex.py:153: in test_not_literal_with_ignorecase
    lambda s: s[0] not in (u'a', u'A') and s[1] not in (u'b', u'B')
tests/common/debug.py:127: in assert_all_examples
    assert_examples()
tests/common/debug.py:123: in assert_examples
    def assert_examples(s):
src/hypothesis/executors.py:58: in default_new_style_executor
    return function(data)
src/hypothesis/core.py:140: in run
    return test(*args, **kwargs)
tests/common/debug.py:123: in assert_examples
    def assert_examples(s):
src/hypothesis/core.py:602: in timed_test
    result = test(*args, **kwargs)
tests/common/debug.py:124: in assert_examples
    assert predicate(s), \
E   AssertionError: Found u'A00' using strategy from_regex(u'(?i)[^a][^b]') which does not match
E   assert False
E    +  where False = <function <lambda> at 0x7f48cf53f398>('A00')
----------------------------- Captured stdout call -----------------------------
Falsifying example: assert_examples(s=u'A00')
You can add @seed(302934307671667531413257853548643485645) to this test or run pytest with --hypothesis-seed=302934307671667531413257853548643485645 to reproduce this failure.

[DRMacIver]

It is a flaky test, but it's one that I think is exposing a real problem rather than a harmless one. If you run it with the seed in the output I assume it reproduces?

[felixonmars]

No, it doesn't. I run them in a clean build environment each time, not sure if that makes a difference.

[DRMacIver]

No, it doesn't

Just to check, is this true if you pass the seed with the pytest argument? because of the way we're running the tests ourselves (which is a bit nonstandard) adding a @seed decorator won't work.

[felixonmars]

No, that's how I pass the seed. The full command:

python2 setup.py pytest --addopts="--hypothesis-seed=302934307671667531413257853548643485645"

[DRMacIver]

Wait, what? How does that work? Do you have some customisations to the setup.py? Because pytest definitely isn't a recognised command in our setup.py

[felixonmars]

That's a command added by pytest-runner, which helps to run the test suite without installing hypothesis first.

[alexwlchan]

Knowing this example was hiding in the weeds somewhere, I ran the following test:

import re
from hypothesis import given
from hypothesis.strategies import from_regex

@given(from_regex(re.compile(u'[^a][^b]', re.IGNORECASE)))
def test_x_is_not_integer(s):
    assert s != u'A00'

(I know equality tests are quite hard to hit, but I knew this example was possible.)

Eventually, it found this example, and I got the following repro which seems to work on my machine (specifically, my work laptop – macOS 10.12.4, Python 3.6.1, Hypothesis at 770e394).

import re
from hypothesis import seed, given
from hypothesis.strategies import from_regex

@seed(298794930177074211994066318099942485621)
@given(from_regex(re.compile(u'[^a][^b]', re.IGNORECASE)))
def test_x_is_not_integer(s):
    assert s[0] not in (u'a', u'A') and s[1] not in (u'b', u'B')

Are seeds portable across machines?

If so, that example should repro elsewhere. If not, the same technique is probably usable for getting a repro on another machine.

(Yes, the function name sucks, I was editing an old testing file I had around. Changing the function name seems to invalidate the seed, so I’m leaving it for now!)

[alexwlchan]

I know very little about the internals of the regex strategy (and I have to go back to my actual job), but in case it’s helpful, this is the coverage report when I run that script locally: https://gist.github.com/alexwlchan/8ca2bd68f1c598eab05a382f40066e7e

The missing lines might give a clue as to where regex.py is going.

Let me know if there’s other stuff I can do to help, but I don’t think I can debug this one myself.