epoll patch

[mtelka]

I propose this wording for the epoll patch (feel free to improve/fix if you think it is needed).

Also the patch itself is much cleaner this way. I kicked build & test to make sure it really works as expected.

[mtelka]

Here are test results:

== Tests result: FAILURE ==

19 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print test_epoll
    test_kqueue test_launcher test_msilib test_perf_profiler
    test_perfmaps test_readline test_startfile test_winconsoleio
    test_winreg test_winsound test_wmi

9 tests skipped (resource denied):
    test_smtpnet test_socketserver test_tix test_tkinter test_ttk
    test_urllib2net test_urllibnet test_xmlrpc_net test_zipfile64

5 tests failed:
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_spawn.test_manager test_dtrace
    test_xml_etree test_xml_etree_c

455 tests OK.

Total duration: 23 min 39 sec
Total tests: run=41,760 failures=18 skipped=1,545
Total test files: run=479/488 failed=5 skipped=19 resource_denied=9
Result: FAILURE

[Bill-Sommerfeld]

5 tests failed:
test.test_multiprocessing_fork.test_processes
test.test_multiprocessing_spawn.test_manager test_dtrace
test_xml_etree test_xml_etree_c

Hmm. I only see the test_dtrace failure in my builds (without this change). Do you get those failures without this change?

[mtelka]

Hmm. I only see the test_dtrace failure in my builds (without this change). Do you get those failures without this change?

See #17 for test results without this PR. I'm going to run tests with again this PR because I remember Python tests were unreliable for me in past...

For test_xml_etree and test_xml_etree_c failures maybe I do not have something installed. I'll investigate...

[Bill-Sommerfeld]

This patch needs to turn off HAVE_EPOLL and this version doesn't.

Perhaps it would work to guard the epoll_create* checks in something that will skip them if HAVE_SYS_EPOLL_H or some such is not defined?

Problem didn't reproduce when I tried it again

[mtelka]

Current HEAD + epoll patch:

== Tests result: FAILURE ==
    
19 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print test_epoll
    test_kqueue test_launcher test_msilib test_perf_profiler
    test_perfmaps test_readline test_startfile test_winconsoleio
    test_winreg test_winsound test_wmi

9 tests skipped (resource denied):
    test_smtpnet test_socketserver test_tix test_tkinter test_ttk
    test_urllib2net test_urllibnet test_xmlrpc_net test_zipfile64

4 tests failed:
    test_dtrace test_multiprocessing_main_handling test_xml_etree
    test_xml_etree_c

456 tests OK.

Total duration: 15 min 23 sec
Total tests: run=41,760 failures=18 skipped=1,545
Total test files: run=479/488 failed=4 skipped=19 resource_denied=9
Result: FAILURE

Both test_xml_etree and test_xml_etree_c are well known failures here. test_multiprocessing_main_handling is this:

FAIL: test_module_in_package (test.test_multiprocessing_main_handling.SpawnCmdLineTest.test_module_in_package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/test/test_multiprocessing_main_handling.py", line 245, in test_module_in_package
    self._check_script(launch_name)
  File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/test/test_multiprocessing_main_handling.py", line 168, in _check_script
    self._check_output(script_name, rc, out, err)
  File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/test/test_multiprocessing_main_handling.py", line 159, in _check_output
    self.assertEqual(err.decode('utf-8'), '')
AssertionError: 'Traceback (most recent call last):\n  Fil[1103 chars]))\n' != ''
- Traceback (most recent call last):
-   File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/util.py", line 303, in _run_finalizers
-     finalizer()
-   File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/util.py", line 227, in __call__
-     res = self._callback(*self._args, **self._kwargs)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-   File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/synchronize.py", line 87, in _cleanup
-     sem_unlink(name)
- FileExistsError: [Errno 17] File exists
- /data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
-   warnings.warn('resource_tracker: There appear to be %d '
- /data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/resource_tracker.py:267: UserWarning: resource_tracker: '/mp-6zb34qho': [Errno 17] File exists
-   warnings.warn('resource_tracker: %r: %s' % (name, e))


----------------------------------------------------------------------

[mtelka]

When I re-run using env - gmake TESTOPTS_PYTHON_TEST=test_multiprocessing_main_handling test then the test passed. Apparently the test is flaky :-(.

[mtelka]

When I re-run using env - gmake TESTOPTS_PYTHON_TEST=test_multiprocessing_main_handling test then the test passed. Apparently the test is flaky :-(.

I re-run the same several times and it usually pass, but sometimes fail. Could you please try the same on your end few times?

[mtelka]

BTW, the failure varies. This time it is test_script_compiled, previously it was test_module_in_package:

FAIL: test_script_compiled (test.test_multiprocessing_main_handling.ForkServerCmdLineTest.test_script_compiled)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/test/test_multiprocessing_main_handling.py", line 202, in test_script_compiled
    self._check_script(pyc_file)
  File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/test/test_multiprocessing_main_handling.py", line 168, in _check_script
    self._check_output(script_name, rc, out, err)
  File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/test/test_multiprocessing_main_handling.py", line 159, in _check_output
    self.assertEqual(err.decode('utf-8'), '')
AssertionError: 'Traceback (most recent call last):\n  Fil[1103 chars]))\n' != ''
- Traceback (most recent call last):
-   File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/util.py", line 303, in _run_finalizers
-     finalizer()
-   File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/util.py", line 227, in __call__
-     res = self._callback(*self._args, **self._kwargs)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-   File "/data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/synchronize.py", line 87, in _cleanup
-     sem_unlink(name)
- FileExistsError: [Errno 17] File exists
- /data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
-   warnings.warn('resource_tracker: There appear to be %d '
- /data/builds/oi-userland/components/python/python-312/Python-3.12.2/Lib/multiprocessing/resource_tracker.py:267: UserWarning: resource_tracker: '/mp-s3tuwwqm': [Errno 17] File exists
-   warnings.warn('resource_tracker: %r: %s' % (name, e))


----------------------------------------------------------------------

[mtelka]

I reported the test_multiprocessing_main_handling failure upstream: python/cpython#115183

[Bill-Sommerfeld]

Cherry-picked to my branch; I updated the commit description to be less vague.

This is likely not the best place to track test flakes.

[mtelka]

Cherry-picked to my branch; I updated the commit description to be less vague.

Perfect. Thank you.

[Bill-Sommerfeld]

test_xml_etree test_xml_etree_c

FYI, this was due to an upstream bug when python is built with expat 2.6.0 or later: python/cpython#115164; I didn't see the bug when my build machine had expat 2.5.0 installed.

Feeding the parser by too small chunks defers parsing to prevent GHSA-gh68-jm46-84rf.

I'll pull in the upstream patch (which disables the chunking versions of the test that failed).

[mtelka]

test_xml_etree test_xml_etree_c

FYI, this was due to an upstream bug when python is built with expat 2.6.0 or later: python/cpython#115164; I didn't see the bug when my build machine had expat 2.5.0 installed.

Feeding the parser by too small chunks defers parsing to prevent GHSA-gh68-jm46-84rf.

I'll pull in the upstream patch (which disables the chunking versions of the test that failed).

Nice finding. Thank you!