Running doctests with -s fails?

[pytestbot]

Originally reported by: Luke Lee (BitBucket: durden, GitHub: durden)

---

I'm running pytest 2.5.2 with Python 2.7 and see some doctests fail unless I pass the -s flag to turn of stdout capturing. Is this expected?

This issue is either an esoteric bug in pytest or a problem with my tests.

For example, I wrote this simple test and it passes with or without the -s flag:

#!python

def test_func():
    """
    >>> print 'hello'
    hello
    """
    pass

However, this doctest will fail if I do not use the -s flag:

def saveRecentProjectPath(self, new_path):
    r"""
    Save given path to rolling unique recent XML file paths
    (projects or datasets)

    This recent file paths list is meant to be used for recent xml files
    users opened recently by name, not implicitly by opening a project
    file.  So, the implicit convention is to only update a new path here
    when the user explictly requests a file path with a dialog.

    >>> settings = UserSettings('.test_settings')
    >>> settings.clearRecentProjectPaths()
    >>> settings.saveRecentProjectPath('this/is/a/test')
    >>> settings.recentProjectPaths()
    ['this/is/a/test']
"""

The real world test is a bit more complicated, but it passes as-is with the -s flag. So, it leads me to think the issue has something to do with the stdout capturing in pytest. Also, the test passes as-is with nose and other tools regardless of log capturing.

Any ideas?

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/481

[pytestbot]

Original comment by Luke Lee (BitBucket: durden, GitHub: durden):

---

It's worth noting that I've run this test with the built-in doctest module and it passes as well:

python -m doctest settings/__init__.py

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

sounds like a interaction bug with the capture plugin

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

@durden can you please provide a traceback of the failure?
or may be you have working reproducable piece of code?

[pytestbot]

Original comment by Luke Lee (BitBucket: durden, GitHub: durden):

---

Here's a traceback of the failure when running without the -s flag. Notice how the expected text from the doctest is printed to the screen in the py.test output instead of being picked up by the test output.

The test is looking for ['this/is/a/test'] in the output, but it looks like py.test doesn't see it or pass it along to the doctest unless I use the -s flag.

#!python

=========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 2.7.6 -- py-1.4.20 -- pytest-2.5.2
collected 1 items

settings/__init__.py ['this/is/a/test']
F

================================================================================================ FAILURES =================================================================================================
__________________________________________________________________________ [doctest] settings.UserSettings.saveRecentProjectPath __________________________________________________________________________
058
059         This recent file paths list is meant to be used for recent xml files
060         users opened recently by name, not implicitly by opening a project
061         file.  So, the implicit convention is to only update a new path here
062         when the user explictly requests a file path with a dialog.
063
064         >>> settings = UserSettings('.test_settings')
065         >>> settings.clearRecentProjectPaths()
066         >>> settings.saveRecentProjectPath('this/is/a/test')
067         >>> settings.recentProjectPaths()
Expected:
    ['this/is/a/test']
Got nothing

/Users/lukelee/Documents/hmcc/settings/__init__.py:67: DocTestFailure

[pytestbot]

Original comment by Luke Lee (BitBucket: durden, GitHub: durden):

---

Also, this is not specific to my real test code posted above. I reproduced this with a very simple test. You can see the code and test output in this gist:

https://gist.github.com/durden/9955666

[pytestbot]

Original comment by Luke Lee (BitBucket: durden, GitHub: durden):

---

Just as another data point. I've found this doctest taken straight from scipy also doesn't work unless the -s flag is used:

http://scikit-image.org/docs/dev/api/skimage.transform.html#hough-line-peaks

>>> import numpy as np
>>> from skimage.transform import hough_line, hough_line_peaks
>>> from skimage.draw import line
>>> img = np.zeros((15, 15), dtype=np.bool_)
>>> rr, cc = line(0, 0, 14, 14)
>>> img[rr, cc] = 1
>>> rr, cc = line(0, 14, 14, 0)
>>> img[cc, rr] = 1
>>> hspace, angles, dists = hough_line(img)
>>> hspace, angles, dists = hough_line_peaks(hspace, angles, dists)
>>> len(angles)
2

[pytestbot]

Original comment by Brianna Laugher (BitBucket: pfctdayelise, GitHub: pfctdayelise):

---

I tried to reproduce this but so far I can't. I put the skimage example in a testfoo.txt file and it passes with or without -s. Also the code in your gist passes for me, with and without -s.

The only difference I can see between our platforms is I am on Linux.

#!python

platform linux2 -- Python 2.7.6 -- py-1.4.20 -- pytest-2.5.2 

[pytestbot]

Original comment by Luke Lee (BitBucket: durden, GitHub: durden):

---

I've run into this again in just simple tests using something like this simple gist.

Oddly enough this test will pass in Windows and Linux. However, the same test will fail if run in OS X WITHOUT the -s flag. Guess there's something odd about the stdout capturing in OS X?

[RonnyPfannschmidt]

@durden any further idea on this one, atm we don't have a OSX using volunteer that can take a look at that platform bug

[NewbiZ]

atm we don't have a OSX using volunteer that can take a look at that platform bug

I have the exact same problem on linux...

[RonnyPfannschmidt]

@NewbiZ can you please show a minimal example?

[NewbiZ]

I'm trying to, but it seems to be an environment problem.

I just reused the simple gist posted above.

• If I create a brand new virtualenv (python 2.7.10) with py.test (2.9.1), no problem.
• However with the exact same set of dependencies (py.test 2.9.1) from a conda environment (python 2.7.10), the problem triggers.

I'm still trying to find the root cause.

Example:

$ cat pytest_failure.py
def func_1():
    return ['a', 'b', 'c']

def func_2():
    """
    >>> func_1()
    ['a', 'b', 'c']
    """
    pass

From a brand new virtualenv with py.test 2.9.1:

$ py.test --version && py.test pytest_failure.py --doctest-modules -q -x
This is pytest version 2.9.1, imported from /home/avallee/venv/lib/python2.7/site-packages/pytest.pyc
.
1 passed in 0.02 seconds

From my conda environment, with the exact same python packages installed:

$ py.test --version && py.test pytest_failure.py --doctest-modules -q -x
This is pytest version 2.9.1, imported from /home/avallee/.conda/envs/testenv/lib/python2.7/site-packages/pytest.pyc
['a', 'b', 'c']
F
=========================================================================================== FAILURES ===========================================================================================
_______________________________________________________________________________ [doctest] pytest_failure.func_2 ________________________________________________________________________________
005
006     >>> func_1()
Expected:
    ['a', 'b', 'c']
Got nothing

/home/avallee/pytest_failure.py:6: DocTestFailure
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1 failed in 0.15 seconds

And now from the very same conda environment, addind "-s" to py.test:

$ py.test --version && py.test pytest_failure.py --doctest-modules -q -x -s
This is pytest version 2.9.1, imported from /home/avallee/.conda/envs/testenv/lib/python2.7/site-packages/pytest.pyc
.
1 passed in 0.02 seconds

Here is a stacktrace from pdb:

$ py.test pytest_failure.py --doctest-modules -q -x --pdb
['a', 'b', 'c']
F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
005
006     >>> func_1()
Expected:
    ['a', 'b', 'c']
Got nothing

/home/avallee/pytest_failure.py:6: DocTestFailure
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
*** SyntaxError: unexpected EOF while parsing (<stdin>, line 1)
*** SyntaxError: unexpected EOF while parsing (<stdin>, line 1)
[15] > /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/doctest.py(1808)report_failure()
-> raise DocTestFailure(test, example, got)
(Pdb++) bt
[0]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/runner.py(150)__init__()
-> self.result = func()
[1]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/runner.py(138)<lambda>()
-> return CallInfo(lambda: ihook(item=item, **kwds), when=when)
[2]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[3]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[4]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[5]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[6]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(253)_wrapped_call()
-> return call_outcome.get_result()
[7]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(279)get_result()
-> _reraise(*ex)  # noqa
[8]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[9]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[10]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/runner.py(91)pytest_runtest_call()
-> item.runtest()
[11]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/site-packages/_pytest/doctest.py(78)runtest()
-> self.runner.run(self.dtest)
[12]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/doctest.py(1799)run()
-> r = DocTestRunner.run(self, test, compileflags, out, False)
[13]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/doctest.py(1454)run()
-> return self.__run(test, compileflags, out)
[14]   /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/doctest.py(1363)__run()
-> self.report_failure(out, test, example, got)
[15] > /home/avallee/.conda/envs/rackguru-dev/lib/python2.7/doctest.py(1808)report_failure()
-> raise DocTestFailure(test, example, got)
(Pdb++) l
1803
1804        def report_unexpected_exception(self, out, test, example, exc_info):
1805            raise UnexpectedException(test, example, exc_info)
1806
1807        def report_failure(self, out, test, example, got):
1808 ->         raise DocTestFailure(test, example, got)
1809
1810    ######################################################################
1811    ## 6. Test Functions
1812    ######################################################################
1813    # These should be backwards compatible.

[decentral1se]

I can't reproduce this now on HEAD. I tried:

$ wget https://git.io/v1L6Q -O test_foo.py && pytest test_foo.py --doctest-modules

Where the link is the gist posted above (source).

This can be closed off?

[nicoddemus]

Hmm I can't reproduce in conda either:

(pytest-bug) c:\pytest\.tmp>conda list
# packages in environment at D:\Programming\Miniconda3\envs\pytest-bug:
#
colorama                  0.3.7                    py27_0
pip                       9.0.1                    py27_0
py                        1.4.31                   py27_0
pytest                    2.9.1                    py27_0
python                    2.7.12                        0
setuptools                27.2.0                   py27_1
vs2008_runtime            9.00.30729.1                  2
wheel                     0.29.0                   py27_0

(pytest-bug) c:\pytest\.tmp>py.test --version && py.test test_doctest_bug.py --doctest-modules -q -x
This is pytest version 2.9.1, imported from D:\Programming\Miniconda3\envs\pytest-bug\lib\site-packages\pytest.pyc
.
1 passed in 0.02 seconds

(pytest-bug) c:\pytest\.tmp>conda --version
conda 4.1.12

I tried the same approach in an Ubuntu VM and got the same result:

vagrant@thdengops-ubuntu-14:~$ ~/miniconda3/bin/conda create -n pytest-bug pytest==2.9.1 python=2.7
(snip env creation) 

(pytest-bug) vagrant@thdengops-ubuntu-14:~$ py.test --version && py.test pytest_failure.py --doctest-modules -q -x
This is pytest version 2.9.1, imported from /home/vagrant/miniconda3/envs/pytest-bug/lib/python2.7/site-packages/pytest.py
.
1 passed in 0.01 seconds

(pytest-bug) vagrant@thdengops-ubuntu-14:~$ conda list
# packages in environment at /home/vagrant/miniconda3/envs/pytest-bug:
#
Using Anaconda API: https://api.anaconda.org
ca-certificates           2016.9.26                     0    conda-forge
certifi                   2016.9.26                py27_0    conda-forge
ncurses                   5.9                           9    conda-forge
openssl                   1.0.2h                        2    conda-forge
pip                       9.0.1                    py27_0    conda-forge
py                        1.4.31                   py27_0    conda-forge
python                    2.7.12                        1    conda-forge
readline                  6.2                           0    conda-forge
setuptools                29.0.1                   py27_0    conda-forge
sqlite                    3.13.0                        1    conda-forge
tk                        8.5.19                        0    conda-forge
wheel                     0.29.0                   py27_0    conda-forge
zlib                      1.2.8                         3    conda-forge
pytest                    2.9.1                    py27_0

@NewbiZ do still have this issue? Can you share your full set of requirements to reproduce the issue?

[durden]

My previous test case from this comment passes with and without the -s argument in OS X. I used version 2.9.2 for the tests so looks fixed to me in OS X!

[nicoddemus]

Thanks @durden!