Coverage.py test suite failure since commit 0749244d134

[nedbat]

Bug report

Bug description:

Coverage.py runs its test suite against nightly builds of Python. The tests stalled last night. Running them locally, there's one test that freezes the suite. The GitHub action is eventually cancelled by timing out after six hours: https://github.com/nedbat/coveragepy/actions/runs/8000886484

Bisecting CPython, the first bad commit is 0749244

commit 0749244d13412d7cb5b53d834f586f2198f5b9a6
Author: Brett Simmers <[email protected]>
Date:   Tue Feb 20 06:57:48 2024 -0800

    gh-112175: Add `eval_breaker` to `PyThreadState` (#115194)

    This change adds an `eval_breaker` field to `PyThreadState`. The primary
    motivation is for performance in free-threaded builds: with thread-local eval
    breakers, we can stop a specific thread (e.g., for an async exception) without
    interrupting other threads.

    The source of truth for the global instrumentation version is stored in the
    `instrumentation_version` field in PyInterpreterState. Threads usually read the
    version from their local `eval_breaker`, where it continues to be colocated
    with the eval breaker bits.

I'm still investigating, but I'm hoping that someone familiar with the change (@swtaarrs?) will be able to help.

To reproduce:

% git clone https://github.com/nedbat/coveragepy
% cd coveragepy
% export COVERAGE_ANYPY=/path/to/your/python3
% pip install tox
% tox -qre anypy -- -n 0 -vvv -k test_multiprocessing_simple
=== CPython 3.13.0a4+ (rev 0749244d13) with C tracer (.tox/anypy/bin/python) ===
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.13.0a4+, pytest-8.0.1, pluggy-1.4.0 -- /Users/ned/coverage/trunk/.tox/anypy/bin/python
cachedir: .tox/anypy/.pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/Users/ned/coverage/trunk/.hypothesis/examples'))
rootdir: /Users/ned/coverage/trunk
configfile: pyproject.toml
plugins: flaky-3.7.0, xdist-3.5.0, hypothesis-6.98.9
collected 1386 items / 1384 deselected / 2 selected
run-last-failure: no previously failed tests, not deselecting items.

tests/test_concurrency.py::MultiprocessingTest::test_multiprocessing_simple[fork] PASSED                                                                 [ 50%]
tests/test_concurrency.py::MultiprocessingTest::test_multiprocessing_simple[spawn] PASSED                                                                [100%]

============================================================== 2 passed, 1384 deselected in 3.37s ==============================================================
=== CPython 3.13.0a4+ (rev 0749244d13) with sys.monitoring (.tox/anypy/bin/python) ===
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.13.0a4+, pytest-8.0.1, pluggy-1.4.0 -- /Users/ned/coverage/trunk/.tox/anypy/bin/python
cachedir: .tox/anypy/.pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/Users/ned/coverage/trunk/.hypothesis/examples'))
rootdir: /Users/ned/coverage/trunk
configfile: pyproject.toml
plugins: flaky-3.7.0, xdist-3.5.0, hypothesis-6.98.9
collected 1386 items / 1384 deselected / 2 selected
run-last-failure: no previously failed tests, not deselecting items.

tests/test_concurrency.py::MultiprocessingTest::test_multiprocessing_simple[fork] PASSED                                                                 [ 50%]
tests/test_concurrency.py::MultiprocessingTest::test_multiprocessing_simple[spawn]

and there it is stuck.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux, macOS

Linked PRs

• gh-115832: Fix instrumentation version mismatch during interpreter shutdown #115856

[nedbat]

A bit more explanation: coverage.py can run with three different "measurement cores": a sys.settrace function written in C, or in Python, or with sys.monitoring. The tests are fine with either sys.settrace, but freeze with sys.monitoring.

There are four different multiprocessing tests, which can run with fork or spawn. Any of the four freeze with spawn. Excluding "spawn", the entire test suite runs to completion.

[swtaarrs]

I'm investigating and will report back with any findings later today.

[swtaarrs]

I've figured out what's happening and I have a fix in mind, so you can assign this to me.

My commit added what's essentially a thread-local cache of the global instrumentation version. During interpreter shutdown, that can get out of sync with the global version (the root of the issue). If we run Python code later in the shutdown process (from a GC, in this case), that can cause an infinite loop because this check thinks the current code object is out of date, while this check deeper in _Py_Instrument() thinks the code object is up to date, and doesn't do anything to instrument it.

[nedbat]

Great, I would never have been able to figure that out!

[swtaarrs]

I have a fix ready, but it depends on #115837 and I'm not sure if there's a good way to do stacked PRs, so I'll wait for that to land before posting it.