Ensure sqldata thread safety.

[strichter]

When running coverage with Sqlite3 on our large code base (which starts many processes and threads), we noticed failures due to race conditions. It could happen that the nest counter had been already increased or decreased but the associated action had not been completed and thus another thread started work under wrong assumptions.

This PR changes the behavior by providing every thread with a separate connection. The sqlite3 library already handles cases where database files are locked for writing, which we know works well with forked processes. Thus, this PR basically provides the same connection separation for threads as it did already for forked processes.

BTW, the nest attribute is still needed for cases where the context manager is nested inside the same thread.

[codecov-io]

Codecov Report

Merging #760 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #760      +/-   ##
==========================================
+ Coverage   89.95%   89.96%   +<.01%     
==========================================
  Files          78       78              
  Lines       10804    10809       +5     
  Branches     1118     1120       +2     
==========================================
+ Hits         9719     9724       +5     
  Misses        959      959              
  Partials      126      126

Impacted Files	Coverage Δ
coverage/sqldata.py	93.82% <100%> (+0.09%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21d3b37...dcd3674. Read the comment docs.

[strichter]

We have now run coverage across our entire code base multiple times with this fix and it works. We have not seen any more failures.

[strichter]

We have now run with this fix over 10 days in "production" and have not seen any failures. We have done 100+ runs at 9 computing hours each.

[nedbat]

Thanks, this looks good.

I wrote a test:

@~/coverage/trunk [coverage] master(+17/-0) ± tox -e py36 -- -n 0 -k test_thread_stress
py36 develop-inst-noop: /Users/ned/coverage/trunk
py36 installed: apipkg==1.5,atomicwrites==1.3.0,attrs==18.2.0,-e [email protected]:nedbat/coveragepy.git@9a6bc6e58958957f07d1d400a40a85bb5e4e0c23#egg=coverage,covtestegg1==0.0.0,decorator==4.3.2,dnspython==1.16.0,eventlet==0.24.1,execnet==1.5.0,flaky==3.4.0,future==0.17.1,gevent==1.2.2,greenlet==0.4.15,mock==2.0.0,monotonic==1.5,more-itertools==5.0.0,pbr==5.1.2,pluggy==0.8.1,py==1.7.0,PyContracts==1.8.7,pyparsing==2.3.1,pytest==4.0.2,pytest-forked==1.0.1,pytest-xdist==1.25.0,six==1.12.0,unittest-mixins==1.6
py36 run-test-pre: PYTHONHASHSEED='4003265018'
py36 runtests: commands[0] | python setup.py --quiet clean develop
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
py36 runtests: commands[1] | python igor.py zip_mods install_egg remove_extension
py36 runtests: commands[2] | python igor.py test_with_tracer py -n 0 -k test_thread_stress
=== CPython 3.6.7 with Python tracer (.tox/py36/bin/python) ===
F                                                                                                                                                        [100%]
=========================================================================== FAILURES ===========================================================================
_____________________________________________________________ ThreadedDataTest.test_thread_stress ______________________________________________________________

self = <tests.test_data.ThreadedDataTest testMethod=test_thread_stress>

    def test_thread_stress(self):
        def thread_main():
            data.add_lines(LINES_1)

        import threading
        data = CoverageData()
        threads = [threading.Thread(target=thread_main) for _ in range(10)]
        for t in threads:
            t.start()
        for t in threads:
            t.join()

>       self.assert_lines1_data(data)

tests/test_data.py:120:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_data.py:88: in assert_lines1_data
    self.assert_line_counts(covdata, SUMMARY_1)
tests/test_data.py:80: in assert_line_counts
    self.assertEqual(line_counts(covdata, fullpath), counts)
E   AssertionError: {} != {'a.py': 2, 'b.py': 1}
E   - {}
E   + {'a.py': 2, 'b.py': 1}
--------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145359286272 and this is thread id 123145348775936.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 198, in _context_id
    with self._connect() as con:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 525, in __enter__
    self.connect()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 518, in connect
    self.execute("pragma synchronous=off").close()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145359286272 and this is thread id 123145348775936.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145369796608 and this is thread id 123145343520768.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 236, in add_lines
    self._choose_lines_or_arcs(lines=True)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 279, in _choose_lines_or_arcs
    with self._connect() as con:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 160, in _connect
    self._create_db()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 117, in _create_db
    with self._db:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 525, in __enter__
    self.connect()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 518, in connect
    self.execute("pragma synchronous=off").close()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145369796608 and this is thread id 123145343520768.

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145364541440 and this is thread id 123145359286272.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 198, in _context_id
    with self._connect() as con:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 525, in __enter__
    self.connect()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 518, in connect
    self.execute("pragma synchronous=off").close()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145364541440 and this is thread id 123145359286272.
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145390817280 and this is thread id 123145364541440.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 198, in _context_id
    with self._connect() as con:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 525, in __enter__
    self.connect()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 518, in connect
    self.execute("pragma synchronous=off").close()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145390817280 and this is thread id 123145364541440.



Exception in thread Thread-3:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145390817280 and this is thread id 123145354031104.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 198, in _context_id
    with self._connect() as con:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 525, in __enter__
    self.connect()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 518, in connect
    self.execute("pragma synchronous=off").close()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145390817280 and this is thread id 123145354031104.

Exception in thread Thread-6:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145364541440 and this is thread id 123145369796608.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 198, in _context_id
    with self._connect() as con:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 525, in __enter__
    self.connect()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 518, in connect
    self.execute("pragma synchronous=off").close()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145364541440 and this is thread id 123145369796608.

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145380306944 and this is thread id 123145375051776.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 198, in _context_id
    with self._connect() as con:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 525, in __enter__
    self.connect()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 518, in connect
    self.execute("pragma synchronous=off").close()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145380306944 and this is thread id 123145375051776.
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145385562112 and this is thread id 123145380306944.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 198, in _context_id
    with self._connect() as con:
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 525, in __enter__
    self.connect()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 518, in connect
    self.execute("pragma synchronous=off").close()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145385562112 and this is thread id 123145380306944.


Exception in thread Thread-9:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145390817280 and this is thread id 123145385562112.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 199, in _context_id
    row = con.execute("select id from context where context = ?", (context,)).fetchone()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145390817280 and this is thread id 123145385562112.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 203, in _context_id
    return None
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 533, in __exit__
    self.con.__exit__(exc_type, exc_value, traceback)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145390817280 and this is thread id 123145385562112.

Exception in thread Thread-10:
Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 541, in execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: context

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/py/cpython-v3.6.7/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ned/coverage/trunk/tests/test_data.py", line 110, in thread_main
    data.add_lines(LINES_1)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 237, in add_lines
    self._set_context_id()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 215, in _set_context_id
    context_id = self._context_id(context)
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 199, in _context_id
    row = con.execute("select id from context where context = ?", (context,)).fetchone()
  File "/Users/ned/coverage/trunk/coverage/sqldata.py", line 543, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/coverage_test/tests_test_data_ThreadedDataTest_test_thread_stress_91970701/.coverage': no such table: context

=================================================================== short test summary info ====================================================================
FAIL tests/test_data.py::ThreadedDataTest::test_thread_stress
1 failed, 850 deselected in 1.03 seconds
Inefficient: ThreadedDataTest ran 1 tests, 0 made files in a temp directory
ERROR: InvocationError for command '/Users/ned/coverage/trunk/.tox/py36/bin/python igor.py test_with_tracer py -n 0 -k test_thread_stress' (exited with code 1)
___________________________________________________________________________ summary ____________________________________________________________________________
ERROR:   py36: commands failed
@~/coverage/trunk [coverage] master(+17/-0) 1 ±

This PR makes this test pass. I wish the other linked issues had the same failures, so we had some confidence that this fixes them also.

[nedbat]

But there's something I don't understand: without this PR, the error is very clear and should always happen in multi-threaded code: "SQLite objects created in a thread can only be used in that same thread."

I have tests of multi-threaded coverage measurement, and they don't raise that error? Why is that?

[nedbat]

Oh, right: in a multi-threaded program, the threads all write to the same dictionary, which is then written to the CoverageData object at the end of the program. So now I'm wondering how you produced errors when running coverage?

[strichter]

@nedbat We had race conditions appearing around the counter. If the Python interpreter decided to switch to the other thread after incrementing or decrementing the counter but before executing the DB call, we would get failures due to inconsistent state. It happened about 1% of the time, but that meant that our test suite always failed, since we have 100+ parallel workers. ;-)

[strichter]

In your test results, note that the "Could not use a SQLite object from another thread" error is a bit of a red herring. You can allow objects to be accessed across threads and the error frequency decreases but at the end it is only symptom.

The sqlite3.OperationalError: no such table: context error is more real. That is a clear indicator that something has been marked ready and yet, the DB is not setup. We saw that error a bunch.

[nedbat]

I really wish I had a way to reproduce the problems, however crude.

[strichter]

You probably can't, since the bug depends on Python's thread switching. In Python 2 you could specify the switch interval in terms of instructions and setting it to -1 would always switch . But in Python 3.2 they changed that to a time interval in seconds (to support async) and so this opportunity is gone.

You could monkey patch the DB operations and simulate thread switching in that. Or you could make the counter a int-like class that has side effects like simulating a thread switch.

[nedbat]

Can I run your test suite that originally caused you the headaches?

[strichter]

@nedbat Not easily, since it is our internal test suite.

We could find a day/time to meet up and work on together. This way you could use my laptop and some customized coveragepy code to get debugging output. (Getting a pdb() is hard in our setup.)

[nedbat]

@strichter: what about the simpler #723 as a fix? Can we tease apart the different sorts of issues to choose the right solutions?

[strichter]

How does the upsert pattern help here? There is a race condition problem due to threading. You can either implement the locking yourself or have the library manage that, since it does already do that using a file lock.

I feel that the changeset in this PR is as minimal as it gets.

[nedbat]

@strichter I'll merge this today, but I'm wondering about a race condition on the creation of the database. There's a window here between checking if the file exists, and writing the initial metadata into the database. Is there a reason we don't see failures due to that race?

[strichter]

Good call! I think that coverage starts collecting so early that it will always hit the initialization case before any threads are spawned. This is at least true in our setup where we run coverage as a shell script. In other words the first time this condition hits, we are in the main thread.