Add support for coverage

[moi15moi]

I opened this PR as a draft because I still haven't implement everthing said in #772.

Also, now, to run the tests, I use coverage run -m unittest discover -v -s comtypes\test -t comtypes\test instead of python -m unittest discover -v -s ./comtypes/test -t comtypes\test, but test_eval consistently fails on Python 3.10 with a large memory leaks:

FAIL: test_eval (test_comserver.TestInproc)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\comtypes\comtypes\comtypes\test\test_comserver.py", line 99, in test_eval
    self._find_memleak(func)
  File "D:\a\comtypes\comtypes\comtypes\test\test_comserver.py", line 42, in _find_memleak
    self.assertFalse(bytes, f"Leaks {bytes} bytes")  # type: ignore
AssertionError: 5591040 is not false : Leaks 5591040 bytes

Have you an idea why it happen?
We might be hitting a element from Things that don’t work.

[junkmd]

I am investigating and re-running the failed jobs in the CI pipeline.

Even before this PR, memory leaks would occasionally occur, but re-running the jobs would result in success, so I had assumed these were one-time glitches.
However, with the changes in this PR, the leaks have become highly reproducible.

This project employs various kinds of metaprogramming and is also related to threading.

It will likely require thorough investigation.

[moi15moi]

Why do you keep running the CI? There have been 111 attempts, and they all failed.
It's pretty clear that there is a problem. ^^'

[junkmd]

In between my main work and other investigations, I was repeatedly re-running the failed jobs to see if they would pass.

As you pointed out, there are clearly problems.

One key observation is that while failures occur depending on whether third-party dependencies are included or based on the system’s bitness, at least one test run for each Python version has passed. This means that every configuration has the potential to pass — but also the potential to fail.

We should continue investigating the root cause, but to prevent memory leaks and stabilize the tests, we may need to escalate this to CPython, or open an issue in coverage-py.

[junkmd]

The coverage measurement of the test codebase might be affecting the results.
Try setting omit = ["comtypes/gen/*", "comtypes/test/*"] to measure coverage only for the production codebase.

[moi15moi]

In the documentation, they recommend to include the coverage of the test.

[junkmd]

The previous suggestion was meant to verify whether test code coverage measurement affects memory release.
The primary intent is to experiment with different conditions to investigate the root cause.

I agree that test code should be included in the coverage measurement. However, we first need to determine what is actually affecting the results.

Possibly, it’s the coverage measurement itself.
Possibly, the memory leak test for the COM server is a false positive.

[junkmd]

Thank you for the verification commit.

Since it fails no matter how many times I re-run it, it seems that test code coverage measurement is not the root cause of this issue.

Since the eval method calls Python’s eval function, it is possible that this test double implementation itself is inherently prone to memory leaks.

comtypes/comtypes/test/TestComServer.py

Lines 104 to 108 in f1e79a9

	def ITestComServer_eval(self, this, what, presult):
	self.Fire_Event(0, "EvalStarted", what)
	presult[0].value = eval(what)
	self.Fire_Event(0, "EvalCompleted", what, presult[0].value)
	return S_OK

[junkmd]

Generating a tuple might be the issue.

comtypes/comtypes/test/test_comserver.py

Lines 93 to 100 in f1e79a9

	def test_eval(self):
	obj = self.create_object()
	def func():
	return obj.eval("(1, 2, 3)")
	self.assertEqual(func(), (1, 2, 3)) # type: ignore
	self._find_memleak(func)

How about trying "1 + 2" instead?

[junkmd]

I confirmed that changing from "(1, 2, 3)" to "1 + 2" still resulted in some job failures on Python 3.12.

The failing test in a job was the same as before: test_comserver.TestInproc_win32com.test_eval.

However, what concerns me even more is that a job where all tests passed on Python 3.12 takes 818.020s.

In contrast, a job where all tests passed on Python 3.11 only took 83.091s, which means the execution time has increased nearly tenfold.

Looking at a canceled job on Python 3.12, I noticed that test_client.Test_GetModule.test_the_name_of_the_enum_member_and_the_coclass_are_duplicated was canceled.
This test generates the Python wrapper module for MSHTML (_3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0.py).

As mentioned in #524 (comment), the wrapper module for MSHTML is massive.
Could it be that despite excluding it from coverage measurement with omit = ["comtypes/gen/*"] it is still attempting to measure coverage and that’s why the execution time is so long?
It's puzzling that this issue does not occur with Python versions other than 3.12.

Currently, the latest version (7.6.10) of coverage is installed on Python 3.12 jobs.
Perhaps we should try explicitly specifying another version.

[junkmd]

I reported the Python 3.12 issue to coveragepy community: nedbat/coveragepy#1862 (comment)

[junkmd]

These issues might be related:
python/cpython#127953
python/cpython#107674