Run tests truly programmatically

[santiagobasulto]

Hey guys! I'm trying to execute pytest tests truly programmatically and get results from the very same python code. I see in the documentation how to "invoke pytest from Python code", but that's just executing the regular pytest command from a python interface (generating output and all that).

I'd like to invoke the underlying test runner with an arbitrary piece of code and get some sort of programmatic result. Something like:

with open('test_file.py', 'r') as fp:
    results = pytest.run_code(fp.read())
results.failed  # True/False
results.failed_tests # [TestResult, TestResult]

This is what I can do with nose for example:

suite = unittest.TestLoader().discover(abs_path)
program = TestProgram(suite=suite, exit=False, argv=sys.argv[0:1])

It's not ideal, but it's something.

[nicoddemus]

Hi @santiagobasulto,

pytest needs to startup a lot of stuff (plugins are just one example), so your best bet would be to actually follow what's suggested in the link you posted.

To obtain the programmatic result, you can implement a custom plugin with the appropriate hooks that interest you, for example (untested):

class Plugin:
    def __init__(self):
        self.passed_tests = set()
    def pytest_runtest_logreport(self, report):
        if report.passed:
            self.passed_tests.add(report.nodeid)

plugin = Plugin()
pytest.main(['test_file.py', '-p', 'no:terminal'], plugins=[plugin])
# now plugin.passed_tests contain the list of tests that passed during the session

Here I also disabled the terminal so pytest won't write to your stdout/stderr. It might not be ideal, but I'm sure you can extract any information you need from it.

[santiagobasulto]

Thanks very much @nicoddemus. I'll close the issue to remove noise, but I'll comment as I make progress.