Make tests runnable on AppVeyor CI and environment-specific tests be skipped #298

junkmd · 2022-06-08T14:32:51Z

related to #267

This PR has taken over from #271 by @dmwyatt.

participants are @vasily-v-ryabov, @kdschlosser

With this change, tests can run on AppVeyor CI(and perhaps on developer machines).

The goal of this PR as is @dmwyatt's comment,

It seems like it'd be better to have a test suite that runs but doesn't cover everything than a test suite that doesn't run but hypothetically covers more.
#267 (comment)

Overview of changes

links to issue/PR comments.

Skips environment-specific tests

Fixes failed or warned tests

This module causes the whole test suite to fail. Why? I don't know!

These tests take many minutes to run on my machines. This is too long for unit tests! They either need discarded or moved to some sort of integration testing suite that can be run outside of the unit test suite.

I don't understand what this was for, but the tests pass without it. That doesn't mean the tests are correct, but I don't understand the problem being solved by this removed line.

Need to figure out if that is a good idea or if there is an alternative way to test this functionalitty that does not require admin.

I'm not sure of the utility of these tests. 1. Dozens of them fail on my machine. 2. The tests that get run will vary from machine to machine because of the way the tests are built. 3. I think that maybe more tests the other libraries on the system rather than tests comtypes itself.

15+ years ago Thomas Heller created a test running system that could enable or disable all sorts of different tests based upon various strings in this array. We have better solutions for this nowadays which the test suite should evolve to use. For now we'll use the `*` to enable all tests in this bespoke test running system.

These need investigated in more detail.

Needs further investigation.

The L suffix doesn't matter for python2, but causes python3 to not parse the file.

Needs further investigation.

…stered.

Both python2.7 and 3 support the b"x" format.

If this test is necessary we should introduce dev dependencies to comtypes.

Probably should move this to github issues once it's fleshed-out some.

dmwyatt · 2022-06-08T15:39:47Z

Thanks for picking up the baton on this, @junkmd. I got too busy to troubleshoot the AppVeyor stuff.

junkmd · 2022-06-09T12:55:22Z

@dmwyatt
You're welcome.

Your discussion of the testing policy in #267 was very interesting and helpful to me.

dmwyatt added 30 commits June 5, 2022 08:45

Skip test_server.py.

f650a78

This module causes the whole test suite to fail. Why? I don't know!

Skip the leak detection tests in test_collectitons.py.

08a90be

These tests take many minutes to run on my machines. This is too long for unit tests! They either need discarded or moved to some sort of integration testing suite that can be run outside of the unit test suite.

Skip QueryService test until we find another win32 API to use.

e2da502

Skip test_avmc.py until someone figures out how to make it work.

2e10c90

Remove check for 'ui' resource.

688288e

I don't understand what this was for, but the tests pass without it. That doesn't mean the tests are correct, but I don't understand the problem being solved by this removed line.

Disable this test because it requires the tests to be run as admin.

4800d48

Need to figure out if that is a good idea or if there is an alternative way to test this functionalitty that does not require admin.

Skip some more tests that depend on IE.

523c154

Skip test that depends on Excel.

3044606

Skip test that depends on Word.

40f7047

Add missing import.

55b7e84

Skip test failing for mysterious reasons.

051803f

Skip test failing because TestComServer isn't registered.

7b7f56b

Skip memory leak tests that fail with a memory leak.

f5f70f6

These need investigated in more detail.

Skip tests that depend on TestComServerLib, which is not registered.

fc25257

Fix typo which causes whole test file to not work.

0576d97

Skip test that fails for unknown reasons.

5a6dde9

Needs further investigation.

Fix long integer notation for python3.

d01e47f

The L suffix doesn't matter for python2, but causes python3 to not parse the file.

Skip test that causes the python interpreter to crash.

6b41eb5

Needs further investigation.

Skip a test that depends on TestComServerLib.TestComServer being regi…

eb401b6

…stered.

Fix python3 incompatibility.

baef20d

Both python2.7 and 3 support the b"x" format.

Skip test that requires the pythoncom library.

213166e

If this test is necessary we should introduce dev dependencies to comtypes.

Skip test of Internet Explorer.

9536e95

Skip test of Word.

2b96b3c

Skip more tests with dependencies on IE and Excel.

024d5e9

Add README.md explaining how to run tests.

f90d9a9

Add preliminary testing TODO list in comtypes/test/README.md

71f81ab

Probably should move this to github issues once it's fleshed-out some.

Add TODO for tox runner

929ecdd

Skip test with dependency not listed in project dependencies.

659d3a1

dmwyatt and others added 10 commits June 5, 2022 08:45

Fix each python2/python3 incompatibility in this test.

1648d95

Remove another Internet Explorer dependency.

2a16294

Remove unused imports

1a42b07

Make test suite run on python2.7

ae4307d

Cleanup unused imports

71f4d08

Add unittests to appveyor.yml

9707365

add -v option to test_script

bfaefc8

update to using context manager

28507a6

update imported module

1884d73

attributes' value assertion instead of object

9558f59

junkmd mentioned this pull request Jun 8, 2022

[WIP] Passing tests with AppVeyor CI #295

Closed

vasily-v-ryabov added a commit to vasily-v-ryabov/pywinauto that referenced this pull request Jun 9, 2022

Test enthought/comtypes#298

3fc8931

vasily-v-ryabov merged commit 7f7c283 into enthought:master Jun 9, 2022

junkmd deleted the make_minimun_tests_runnable branch June 9, 2022 12:43

This was referenced Jun 10, 2022

Make test suite runnable #271

Closed

Quit using shdocvw.dll(or ieframe.dll) in tests #302

Open

Do you know non-env specific COM library behave like Excel.Range.Value, for testing bound_named_property #303

Open

This was referenced Jun 19, 2022

test_typeinfo.py quits using shdocvw.dll and ieframe.dll #306

Merged

Argument lifetime issue when calling COM/ATL methods with array of strings (and probably others) as of at least Python 3.8.3 (which worked with Python 3.7.x) #212

Closed

junkmd mentioned this pull request Jun 26, 2022

Update test_excel #316

Merged

junkmd mentioned this pull request Jul 31, 2022

Delayed loading of npsupport #333

Closed

junkmd mentioned this pull request Sep 10, 2022

array of BSTR is not supported #347

Open

This was referenced Nov 26, 2022

FAIL in test_LoadTypeLibEx #40

Closed

Many failures in test_createwrappers under 64-bit python #9

Closed

junkmd mentioned this pull request Jan 8, 2023

fix test_getactiveobj and remove raising unittest.SkipTest #462

Merged

junkmd mentioned this pull request Jun 19, 2023

Convert the tests to use pytest #123

Closed

junkmd mentioned this pull request May 21, 2024

Implement record pointers as method parameters of a Dispatch Interface #535

Merged

junkmd mentioned this pull request Jun 15, 2024

Can we revive test_avmc? #572

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make tests runnable on AppVeyor CI and environment-specific tests be skipped #298

Make tests runnable on AppVeyor CI and environment-specific tests be skipped #298

junkmd commented Jun 8, 2022

dmwyatt commented Jun 8, 2022

junkmd commented Jun 9, 2022

Make tests runnable on AppVeyor CI and environment-specific tests be skipped #298

Make tests runnable on AppVeyor CI and environment-specific tests be skipped #298

Conversation

junkmd commented Jun 8, 2022

Overview of changes

Skips environment-specific tests

Fixes failed or warned tests

dmwyatt commented Jun 8, 2022

junkmd commented Jun 9, 2022