Subprocess coverage - strange detection of lines losing coverage

[nedbat]

Originally reported by Michał Bultrowicz (Bitbucket: Butla, GitHub: Butla)

---

I wanted to test out how well subprocess coverage works for WSGI apps and I came across a strange behavior. After running the tests a few times I deliberately commented out a test to see if the coverage dropped - it didn't.

I experimented with that a bit and it turns out, that the loss of coverage will be detected if I run "coverage erase" twice before running the tests. It also seems that it works when I run "erase" from shell and then run my "run_coverage" script with "coverage erase" lines removed. Adding "sleep" in "run_coverage" before a single "coverage erase" doesn't work.

Reproduction steps:

• clone https://github.com/butla/experiments
• checkout commit ID c4d441de7c
• go to subprocess_coverage dir
• create a Python 3.4 virtualenv and install requirements.txt (I have coverage_pth there)
• delete one of "coverage erase" lines in "run_coverage.sh"
• run "run_coverage.sh"
• comment out one of the tests in tests/test_wsgi_app.py
• run "run_coverage.sh" again - coverage still at 100%

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/492

[nedbat]

Original comment by Dan Riti (Bitbucket: danriti, GitHub: danriti)

---

I believe the link for reproduction step 2 should be:

butla/experiments@c4d441de7c

[nedbat]

Original comment by Dan Riti (Bitbucket: danriti, GitHub: danriti)

---

This seems like this may be related to this previously resolved issue by @nedbat:

https://bitbucket.org/ned/coveragepy/issues/262/when-parallel-true-erase-should-erase-all

Continuing to investigate.

[nedbat]

Original comment by Dan Riti (Bitbucket: danriti, GitHub: danriti)

---

So that above bug was patched in version 4.0b1 (on 2015-08-02), while this bug seems to be using version 4.0.3 (which came after).

https://bitbucket.org/ned/coveragepy/src/5ab728b8fd55bd78352012a15a6541fa73dd9e2c/CHANGES.rst?at=default&fileviewer=file-view-default

&

https://github.com/butla/experiments/blob/master/subprocess_coverage/requirements.txt#L3

Going to see if I can replicate this problem with a unit test.

[nedbat]

Original comment by Dan Riti (Bitbucket: danriti, GitHub: danriti)

---

Additionally, I'm able to replicate the problem with version 4.1

[nedbat]

Original comment by Dan Riti (Bitbucket: danriti, GitHub: danriti)

---

After some more investigation, it seems this is occurring because of the inclusion of the coverage-pth package:

https://github.com/butla/experiments/blob/master/subprocess_coverage/requirements.txt#L4

This seems to be causing coverage to generate coverage data files when you run any coverage command. For instance:

(.env)[driti@ubuntu subprocess_coverage]$ coverage run -m py.test tests/ 
================================== test session starts ==================================
platform linux2 -- Python 2.7.3, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /home/driti/dev/experiments/subprocess_coverage/tests, inifile: 
collected 5 items 

tests/test_bla.py .
tests/test_bottle_service.py .
tests/test_wsgi_app.py ...

=============================== 5 passed in 4.46 seconds ================================
(.env)[driti@ubuntu subprocess_coverage]$ ls -al
total 56
drwxrwxr-x 4 driti driti 4096 Jun  2 19:51 .
drwxr-xr-x 9 driti driti 4096 Jun  2 18:15 ..
-rw-rw-r-- 1 driti driti   43 Jun  2 16:55 .coveragerc
-rw-rw-r-- 1 driti driti  389 Jun  2 19:51 .coverage.ubuntu.76695.615877
-rw-rw-r-- 1 driti driti  414 Jun  2 19:51 .coverage.ubuntu.76695.989394
-rw-rw-r-- 1 driti driti  406 Jun  2 19:51 .coverage.ubuntu.76702.583650
-rw-rw-r-- 1 driti driti  409 Jun  2 19:51 .coverage.ubuntu.76703.517175
-rw-rw-r-- 1 driti driti  426 Jun  2 19:51 .coverage.ubuntu.76704.344998
-rw-rw-r-- 1 driti driti  389 Jun  2 19:51 .coverage.ubuntu.76713.186275
-rw-rw-r-- 1 driti driti  434 Jun  2 19:51 .coverage.ubuntu.76718.515879
-rw-rw-r-- 1 driti driti  264 Jun  2 18:15 requirements.txt
-rwxrwxr-x 1 driti driti  224 Jun  2 18:15 run_coverage.sh
drwxrwxr-x 3 driti driti 4096 Jun  2 18:20 some_code
drwxrwxr-x 3 driti driti 4096 Jun  2 18:38 tests

You can see in the output above, that we get the expected .coverage.ubuntu.* output files. However, notice what happens when I run coverage combine below:

(.env)[driti@ubuntu subprocess_coverage]$ coverage combine
(.env)[driti@ubuntu subprocess_coverage]$ ls -al
total 36
drwxrwxr-x 4 driti driti 4096 Jun  2 19:51 .
drwxr-xr-x 9 driti driti 4096 Jun  2 18:15 ..
-rw-rw-r-- 1 driti driti  485 Jun  2 19:51 .coverage
-rw-rw-r-- 1 driti driti   43 Jun  2 16:55 .coveragerc
-rw-rw-r-- 1 driti driti  389 Jun  2 19:51 .coverage.ubuntu.76783.398020
-rw-rw-r-- 1 driti driti  264 Jun  2 18:15 requirements.txt
-rwxrwxr-x 1 driti driti  224 Jun  2 18:15 run_coverage.sh
drwxrwxr-x 3 driti driti 4096 Jun  2 18:20 some_code
drwxrwxr-x 3 driti driti 4096 Jun  2 18:38 tests

We get out expected .coverage "combined" data file, but we've also created a new .coverage.ubuntu.76783.398020 file, which is unexpected.

With this file lingering around, it will cause follow on runs to have skewed results, which the original issue creator was able to observe. Thus, a workaround would be to manually delete the coverage data files (using rm) instead of using coverage erase.

Continuing investigation for a code fix.

[nedbat]

Original comment by Dan Riti (Bitbucket: danriti, GitHub: danriti)

---

After several discussions with @nedbat, there doesn't seem (at least to me) to be an easy way to solve this problem. So this might need some more thought to be solved.

On the bright side, here is a PR that includes a unit test that replicates this problem:

#7

[nedbat]

Original comment by Michał Bultrowicz (Bitbucket: Butla, GitHub: Butla)

---

I didn't notice that bitbucket made a link out of the commit id, but now I've set it up as a proper link.

[nedbat]

@danriti's test is here and the previous commit: https://bitbucket.org/ned/coveragepy/commits/e72ccc3a913280de298bec8757f1461bef7d8cb8

[nedbat]

Fixed in d89704fe7b17 (bb)