python -m test test_unitttest -R3:3 fails

[sweeneyde]

For the last couple months, it seems buildbots have been failing -m test test_unittest -R3:3, then re-trying the 3 failed test cases and passing on the re-try. Look at https://buildbot.python.org/all/#/builders/123 for example. The tests consistently pass the first warmup run, then fail on the second warmup run.

Example stdout from running -m test test_unittest -R3:3 -v:

======================================================================
FAIL: test_loadTestsFromName__module_not_loaded (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromName__module_not_loaded)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 594, in test_loadTestsFromName__module_not_loaded
    self.assertEqual(list(suite), [])
AssertionError: Lists differ: [<unittest.loader._FailedTest testMethod=dummy>] != []

First list contains 1 additional elements.
First extra element 0:
<unittest.loader._FailedTest testMethod=dummy>

- [<unittest.loader._FailedTest testMethod=dummy>]
+ []

======================================================================
FAIL: test_loadTestsFromNames__module_not_loaded (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromNames__module_not_loaded)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 1019, in test_loadTestsFromNames__module_not_loaded
    self.assertEqual(list(suite), [unittest.TestSuite()])
AssertionError: Lists differ: [<uni[15 chars]tSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>] != [<uni[15 chars]tSuite tests=[]>]

First differing element 0:
<unit[13 chars]stSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>
<unit[13 chars]stSuite tests=[]>

- [<unittest.suite.TestSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>]
+ [<unittest.suite.TestSuite tests=[]>]

======================================================================
FAIL: test_loadTestsFromNames__unknown_attr_name (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromNames__unknown_attr_name)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 715, in test_loadTestsFromNames__unknown_attr_name
    error, test = self.check_deferred_error(loader, list(suite)[0])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 620, in check_deferred_error
    self.assertEqual(1, len(loader.errors))
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 1001 tests in 8.861s

FAILED (failures=3, skipped=55)
test_unittest failed (3 failures)

== Tests result: FAILURE ==

1 test failed:
    test_unittest

Total duration: 18.3 sec
Tests result: FAILURE

I bisected to here:

commit c735d54
Author: Victor Stinner [email protected]
Date: Tue Jun 21 10:27:59 2022 +0200

gh-93839: Move Lib/unttest/test/ to Lib/test/test_unittest/ (#94043)

* Move Lib/unittest/test/ to Lib/test/test_unittest/
* Remove Lib/test/test_unittest.py
* Replace unittest.test with test.test_unittest
* Remove unittest.load_tests()
* Rewrite unittest __init__.py and __main__.py
* Update build system, CODEOWNERS, and wasm_assets.py

cc @vstinner

[sweeneyde]

For convenience, here are those tests:

cpython/Lib/test/test_unittest/test_loader.py

Lines 580 to 600 in 56d9cf7

	# "The specifier can refer to modules and packages which have not been
	# imported; they will be imported as a side-effect"
	def test_loadTestsFromName__module_not_loaded(self):
	# We're going to try to load this module as a side-effect, so it
	# better not be loaded before we try.
	#
	module_name = 'test.test_unittest.dummy'
	sys.modules.pop(module_name, None)
	loader = unittest.TestLoader()
	try:
	suite = loader.loadTestsFromName(module_name)
	self.assertIsInstance(suite, loader.suiteClass)
	self.assertEqual(list(suite), [])
	# module should now be loaded, thanks to loadTestsFromName()
	self.assertIn(module_name, sys.modules)
	finally:
	if module_name in sys.modules:
	del sys.modules[module_name]

cpython/Lib/test/test_unittest/test_loader.py

Lines 1005 to 1025 in 56d9cf7

	# "The specifier can refer to modules and packages which have not been
	# imported; they will be imported as a side-effect"
	def test_loadTestsFromNames__module_not_loaded(self):
	# We're going to try to load this module as a side-effect, so it
	# better not be loaded before we try.
	#
	module_name = 'test.test_unittest.dummy'
	sys.modules.pop(module_name, None)
	loader = unittest.TestLoader()
	try:
	suite = loader.loadTestsFromNames([module_name])
	self.assertIsInstance(suite, loader.suiteClass)
	self.assertEqual(list(suite), [unittest.TestSuite()])
	# module should now be loaded, thanks to loadTestsFromName()
	self.assertIn(module_name, sys.modules)
	finally:
	if module_name in sys.modules:
	del sys.modules[module_name]

cpython/Lib/test/test_unittest/test_loader.py

Lines 704 to 720 in 56d9cf7

	# "The specifier name is a ``dotted name'' that may resolve either to
	# a module, a test case class, a TestSuite instance, a test method
	# within a test case class, or a callable object which returns a
	# TestCase or TestSuite instance."
	#
	# What happens when the module can be found, but not the attribute?
	def test_loadTestsFromNames__unknown_attr_name(self):
	loader = unittest.TestLoader()
	suite = loader.loadTestsFromNames(
	['unittest.loader.sdasfasfasdf', 'test.test_unittest.dummy'])
	error, test = self.check_deferred_error(loader, list(suite)[0])
	expected = "module 'unittest.loader' has no attribute 'sdasfasfasdf'"
	self.assertIn(
	expected, error,
	'missing error string in %r' % error)
	self.assertRaisesRegex(AttributeError, expected, test.sdasfasfasdf)

[sweeneyde]

It appears to be something a bad interaction between those three test methods and test.test_unittest.testmock.testpatch.PatchTest.test_dotted_but_module_not_loaded:

cpython/Lib/test/test_unittest/testmock/testpatch.py

Lines 1916 to 1932 in 56d9cf7

	def test_dotted_but_module_not_loaded(self):
	# This exercises the AttributeError branch of _dot_lookup.
	# make sure it's there
	import test.test_unittest.testmock.support
	# now make sure it's not:
	with patch.dict('sys.modules'):
	del sys.modules['test.test_unittest.testmock.support']
	del sys.modules['test.test_unittest.testmock']
	del sys.modules['test.test_unittest']
	del sys.modules['unittest']
	# now make sure we can patch based on a dotted path:
	@patch('test.test_unittest.testmock.support.X')
	def test(mock):
	pass
	test()

[sweeneyde]

Here's the bare essentials of what is failing:

import sys
from unittest.mock import patch

if 0: # change to 1 to see "tu.dummy" fail.

    # Inline test_dotted_but_module_not_loaded
    # This exercises the AttributeError branch of _dot_lookup.

    # make sure it's there
    import test.test_unittest.testmock.support
    # now make sure it's not:
    with patch.dict('sys.modules'):
        del sys.modules['test.test_unittest.testmock.support']
        del sys.modules['test.test_unittest.testmock']
        del sys.modules['test.test_unittest']
        del sys.modules['unittest']

        # now make sure we can patch based on a dotted path:
        @patch('test.test_unittest.testmock.support.X')
        def test(mock):
            print(mock)
        test()


# The part that fails in test_loadTestsFromName__module_not_loaded
module_name = 'test.test_unittest.dummy'
sys.modules.pop(module_name, None)
tu = __import__(module_name).test_unittest
tu.dummy

So essentially, I believe test_dotted_but_module_not_loaded needs to do a better job cleaning up after itself.

[sweeneyde]

@ezio-melotti or @gpshead or @cjw296, any suggestions on how to make test_dotted_but_module_not_loaded play more nicely? I don't know all that much about unittest.mock.

[vstinner]

So essentially, I believe test_dotted_but_module_not_loaded needs to do a better job cleaning up after itself.

Maybe save/restore sys.modules?

[sweeneyde]

I believe this is fixed now.

[vstinner]

Oh nice, thanks for the fix!