Some test modules fail when run as a script

[Eclips4]

 C:\Users\KIRILL-1\CLionProjects\cpython> ./python Lib/test/test_module.py
Running Debug|x64 interpreter...
.....................F..............
======================================================================
FAIL: test_module_repr_with_full_loader (__main__.ModuleTests.test_module_repr_with_full_loader)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\test\test_module.py", line 238, in test_module_repr_with_full_loader
    self.assertEqual(
AssertionError: "<module 'foo' (<class '__main__.FullLoader'>)>" != "<module 'foo' (<class 'test.test_module.FullLoader'>)>"      
- <module 'foo' (<class '__main__.FullLoader'>)>
?                        ^  ^^^^^
+ <module 'foo' (<class 'test.test_module.FullLoader'>)>
?                        ^^^^^^^^^  ^^^^^


----------------------------------------------------------------------
Ran 36 tests in 0.217s

FAILED (failures=1)

I'll soon send a PR to fix it.

Linked PRs

• gh-104057: Fix direct invocation of test_module #104059
• gh-104057: Fix direct invocation of test_super #104064
• gh-104057: Fix direct invocation of test_support #104069

[terryjreedy]

I grepped with IDLE for other candidates to fix.

Searching "<class 'test." in F:\dev\3x\Lib\test\*.py ...
F:\dev\3x\Lib\test\test_descrtut.py: 42:     <class 'test.test_descrtut.defaultdict'>
F:\dev\3x\Lib\test\test_descrtut.py: 49:     <class 'test.test_descrtut.defaultdict'>
F:\dev\3x\Lib\test\test_descrtut.py: 51:     <class 'test.test_descrtut.defaultdict'>
F:\dev\3x\Lib\test\test_descrtut.py: 267:     classmethod <class 'test.test_descrtut.C'> 1
F:\dev\3x\Lib\test\test_descrtut.py: 270:     classmethod <class 'test.test_descrtut.C'> 1
F:\dev\3x\Lib\test\test_descrtut.py: 276:     classmethod <class 'test.test_descrtut.D'> 1
F:\dev\3x\Lib\test\test_descrtut.py: 279:     classmethod <class 'test.test_descrtut.D'> 1
F:\dev\3x\Lib\test\test_descrtut.py: 295:     classmethod <class 'test.test_descrtut.C'> 1
F:\dev\3x\Lib\test\test_descrtut.py: 299:     classmethod <class 'test.test_descrtut.C'> 1
F:\dev\3x\Lib\test\test_metaclass.py: 121:     Prepare called: ('C', (<class 'test.test_metaclass.B'>, <class 'object'>)) {'other': 'haha'}
F:\dev\3x\Lib\test\test_module.py: 239:             repr(m), "<module 'foo' (<class 'test.test_module.FullLoader'>)>")
F:\dev\3x\Lib\test\test_pdb.py: 582:     <class 'test.test_pdb.MyClass'>
F:\dev\3x\Lib\test\test_typing.py: 120:             "<class 'test.test_typing.AnyTests.test_repr.<locals>.Sub'>",
Hits found: 13
(Hint: right-click to open locations.)

This caught the test_module line you just fixed. Some of these are in interactive output and I don't know if they will fail.

[Eclips4]

Yeah, I saw these lines, and this should be discussed. Otherwise, I found some another scenarios with the same issue (easy to fix).

[Eclips4]

However, I'll try to run file test_support on Windows and got interpreter crash. Can someone confirm that?
./python Lib/test/test_support.py
Cannot confirm this on Linux.
Please, use the current main branch if you try to reproduce it.

[terryjreedy]

main built yesterday

F:\dev\3x>python -m test.test_support
Running Debug|x64 interpreter...
.....................s.F..

Crash. Rebuild and try again. Same crash. "Debug assertion Failed."
File minkernel/ctrs/ucrt/src/appcrt/lowio/write.cpp line 50, expression (_osfile(fh) & FOPEN)

[Eclips4]

F:\dev\3x>python -m test.test_support
Running Debug|x64 interpreter...
.....................s.F..

Crash. Rebuild and try again. Same crash. "Debug assertion Failed."
File minkernel/ctrs/ucrt/src/appcrt/lowio/write.cpp line 50, expression (_osfile(fh) & FOPEN)

Thanks!
I think, it's a minimal reproducible example:

import os


def make_bad_fd():
    file = open("test_file.txt", "wb")
    try:
        return file.fileno()
    finally:
        file.close()



fd = make_bad_fd()
os.write(fd, b"foo")

[terryjreedy]

f:\dev\3x>python -m test -uall -v test_support runs fine.

Running test_support from IDLE editor: get same Assertion Failure box. After Ignore, failure is

.....................s.F.......s.........s...
======================================================================
FAIL: test_ignored_deprecations_are_silent (__main__.TestSupport.test_ignored_deprecations_are_silent)
Test support.ignore_deprecations_from() silences warnings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "f:\dev\3x\Lib\test\test_support.py", line 52, in test_ignored_deprecations_are_silent
    self.assertEqual(len(messages), 0, messages)
AssertionError: 1 != 0 : ['You should NOT be seeing this.']

[Eclips4]

f:\dev\3x>python -m test -uall -v test_support runs fine.

Running test_support from IDLE editor: get same Assertion Failure box. After Ignore, failure is

.....................s.F.......s.........s...
======================================================================
FAIL: test_ignored_deprecations_are_silent (__main__.TestSupport.test_ignored_deprecations_are_silent)
Test support.ignore_deprecations_from() silences warnings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "f:\dev\3x\Lib\test\test_support.py", line 52, in test_ignored_deprecations_are_silent
    self.assertEqual(len(messages), 0, messages)
AssertionError: 1 != 0 : ['You should NOT be seeing this.']

That's easy to fix. Just replace

        cls._test_support_token = support.ignore_deprecations_from(
            "test.test_support", like=".*You should NOT be seeing this.*"
        )

to this:

        cls._test_support_token = support.ignore_deprecations_from(
            __main__, like=".*You should NOT be seeing this.*"
        )

[chgnrdv]

Same is for test_descrtut and test_extcall. test_peg_generator and test_importlib fail too, but for other reason. Actually, there are a lot of tests that fail when directly invoked. For example, all tests that import asyncio (or some other module that does it) fail with the same exception:

$ grep -r "^import asyncio" Lib/test/test_*.py
Lib/test/test_builtin.py:import asyncio
Lib/test/test_contextlib_async.py:import asyncio
Lib/test/test_inspect.py:import asyncio
Lib/test/test_logging.py:import asyncio
Lib/test/test_os.py:import asyncio
Lib/test/test_sys_settrace.py:import asyncio
$ ./python Lib/test/test_pdb.py 
Traceback (most recent call last):
  File "/home/.../cpython/Lib/test/test_pdb.py", line 20, in <module>
    from unittest.mock import patch
  File "/home/.../cpython/Lib/unittest/mock.py", line 26, in <module>
    import asyncio
  File "/home/.../cpython/Lib/asyncio/__init__.py", line 8, in <module>
    from .base_events import *
  File "/home/.../cpython/Lib/asyncio/base_events.py", line 45, in <module>
    from . import staggered
  File "/home/.../cpython/Lib/asyncio/staggered.py", line 11, in <module>
    from . import tasks
  File "/home/.../cpython/Lib/asyncio/tasks.py", line 946, in <module>
    eager_task_factory = create_eager_task_factory(Task)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.../cpython/Lib/asyncio/tasks.py", line 936, in create_eager_task_factory
    raise TypeError(
TypeError: Provided constructor does not support eager task execution

[Eclips4]

For example, all tests that import asyncio (or some other module that does it) fail with the same exception:

Please, read the #104070, I think you got the same problem

[chgnrdv]

For example, all tests that import asyncio (or some other module that does it) fail with the same exception:

Please, read the #104070, I think you got the same problem

Yep, you're right. A lot of modules still fail for other reasons though.

[eryksun]

main built yesterday

F:\dev\3x>python -m test.test_support
Running Debug|x64 interpreter...
.....................s.F..

Crash. Rebuild and try again. Same crash. "Debug assertion Failed." File minkernel/ctrs/ucrt/src/appcrt/lowio/write.cpp line 50, expression (_osfile(fh) & FOPEN)

If you run a test directly, such as python_d -m test.test_support -vv -k test_make_bad_fd, then the reporting modes and reporting files for the debug C runtime's non-standard

• assertions -- i.e. the macros _ASSERT, _ASSERTE, and _ASSERT_EXPR, or the macros _RPT0 and _RPTN used with the _CRT_ASSERT report type;
• warnings -- i.e. the macros _RPT0 and _RPTN used with the _CRT_WARN report type; and
• errors -- i.e. the macros _RPT0 and _RPTN used with the _CRT_ERROR report type --

all remain set to their default values. The assertion and error report types default to showing a dialog box, while the warning report type defaults to outputting a message to an attached debugger, if any. For example:

>>> from msvcrt import *
>>> CrtSetReportMode(CRT_ASSERT, CRTDBG_REPORT_MODE) == CRTDBG_MODE_WNDW
True
>>> CrtSetReportMode(CRT_ERROR, CRTDBG_REPORT_MODE) == CRTDBG_MODE_WNDW
True
>>> CrtSetReportMode(CRT_WARN, CRTDBG_REPORT_MODE) == CRTDBG_MODE_DEBUG
True

On the other hand, if you run the main test suite normally, such as python_d -m test test_support -vv -m test_make_bad_fd, then _CrtSetReportMode() and _CrtSetReportFile() get called to configure the report mode for the _CRT_ASSERT, _CRT_WARN, and _CRT_ERROR report types. They either get configured to no reporting (i.e. mode 0) or, if verbose mode is enabled (i.e. -vv command-line option) to reporting to stderr (i.e. _CRTDBG_MODE_FILE mode with the file set to _CRTDBG_FILE_STDERR). Also, WinAPI SetErrorMode() is called with SEM_FAILCRITICALERRORS | SEM_NOALIGNMENTFAULTEXCEPT | SEM_NOGPFAULTERRORBOX | SEM_NOOPENFILEERRORBOX in order to suppress any message boxes due to system error reporting.

---

One step that's missing is to explicitly configure the report mode for the standard assert macro in debug builds. This gets configured by the C runtime's _set_error_mode() function. For a console application (e.g. "python.exe"), the default behavior is to write to stderr (i.e. _OUT_TO_STDERR), which is what we want. On the other hand, for a GUI application (e.g. "pythonw.exe"), the default behavior is to show a message box (i.e. _OUT_TO_MSGBOX). If we add any tests that specifically use "pythonw.exe" instead of "python.exe", then we should explicitly configure _set_error_mode(_OUT_TO_STDERR), particularly in the SuppressCrashReport context manager, and ensure that the "pythonw.exe" process is spawned with a valid stderr file. Also, it appears that the msvcrt module wraps the set_error_mode() function, but it doesn't define the required constants OUT_TO_DEFAULT, OUT_TO_STDERR, OUT_TO_MSGBOX, and REPORT_ERRMODE. That oversight should be corrected.

[eryksun]

You're running the tests incorrectly for how the debug build is supported on Windows. The debug CRT report modes are not set for each individual test, except in test cases that explicitly use the SuppressCrashReport() context manager. Generally you have to run the main test suite and supply the test you want to run as a command-line option, with an optional -m filter for test cases and -vv if you want the debug CRT assertion, warning and error messages written to stderr. For example: python_d -m test test_support -vv -m test_make_bad_fd.

[chgnrdv]

@eryksun, are you replying to me? I use Linux, so I guess that none of errors that I mentioned above relate to how test output is handled by system, and most of them happen regardless of -m flag usage (but doesn't happen if libregrtest is involved, i.e. as in python -m test test_support example).

[hugovk]

Thanks for the fixes! Anything left here, or okay to close?

[Eclips4]

Thanks for the fixes! Anything left here, or okay to close?

I think we did everything we had to do :)
Thanks for the remind, Hugo!