pytest --collect-only --quiet output is not "machine readable"

[asottile]

What's the problem this feature will solve?

I would like to programmatically collect the list of test ids from pytest.

current pytest --collect-only --quiet gets close, but has some non-machine-readable output at the end which is not desirable.

$ pytest --collect-only --quiet tests | tail -5
tests/astpretty_test.py::test_typedast_support_cmdline_27
tests/astpretty_test.py::test_typedast_support_cmdline_3
tests/astpretty_test.py::test_pformat_py38_type_comments

28 tests collected in 0.02s

Describe the solution you'd like

I would like to add either a --json output to this so it can be machine-parsed. perhaps a structure like:

$ pytest --collect-only --json
{
    "testids": [
        "...",
        "...",
        "..."
    ]
}

Alternative Solutions

I could write a plugin, but...

Additional context

I'm writing a tool to detect test pollution automatically. also I suspect this would be useful for those attempting to distribute tests by id across runners -- for instance pytest-select

[pared]

I have a similar problem: I would expect that providing -q option would result in displaying only the absolutely necessary info and in the case of --collect-only that (seemingly) would be the list of tests.

Inspecting the code I noticed that decreasing the verbosity might help:

pytest/src/_pytest/terminal.py

Lines 1036 to 1038 in 00ad12b

	def summary_stats(self) -> None:
	if self.verbosity < -1:
	return

So, I tried doing that and for the following sample repo:

#!/bin/bash

mkdir tests
echo -e "def test_very_important():
    assert 1+1 == 2
" > tests/test_something.py

I tried running with -q and -qq. Here are the results:

$ pytest --collect-only -q                                                                               
tests/test_something.py::test_very_important

1 test collected in 0.00s

$ pytest --collect-only -qq                                                                                 
tests/test_something.py: 1

My question is: is this behavior expected? Is there a way to get just

tests/test_something.py::test_very_important

?

If it would be expected to display only the list of tests in one of the cases, I am willing to prepare a fix. I would appreciate some context thought, because I am not sure in which case should it be.

[elbehery95]

I would like to programmatically collect the list of test ids from pytest.

I believe there are many ways to do this E.g.

Shell approach:

pytest --collect-only -q tests/test_hello.py | head -n -2 | python -c 'import sys; tests=[i.strip() for i in sys.stdin.readlines()]; print(tests)'

Python approach:

import pytest

class MyCollectionPlugin:
  def __init__(self):
    self.tests = []
  def pytest_report_collectionfinish(self, items):
    self.tests = [item.nodeid for item in items]


if __name__ == "__main__":
  p = MyCollectionPlugin()
  pytest.main(["--collect-only"], plugins=[p])
  print(p.tests)
  # generate_json_from_items_nodeid_list(p.tests)

[RonnyPfannschmidt]

@asottile i tihnk this may be a nice chance to introduce some type of jsonlines based output for pytest that is similar to report log, but adds extras that other ux tools might be able to use

[asottile]

@elbehery95 head -2 and strip are both hacks indicating that there is not structured output which is entirely what this issue is asking for -- you'll note that the original issue shows exactly the command you're using and doesn't help

adding a plugin of course makes this possible but the ask is for pytest to do this itself

[nicoddemus]

The idea of pytest-reportlog was to have it mature outside of the core, but to be eventually be merged, so I think this in line with the plans regarding it. 👍

[The-Compiler]

I'm all for extending pytest-reportlog to allow for this (and then perhaps adding it to the core at some point) rather than adding yet another JSON output format.

[pared]

Should we create the issue in pytest-reportlog repository? I tried playing around with it in my current setup and it produces nice, machine-readable output, though I am not sure I understand what needs to be implemented there to solve the current issue. What would be an expected output while using --report-log? Are we talking here that in case of no FILE provided, the output gets written into the stdout?

[jgarte]

Hi,

What's the status on this? A --json flag would be useful for me as well as I'm trying to implement a consult-pytest like package for Emacs. Having this could clean up the code on my side 😄

[The-Compiler]

Yep, this already works fine with pytest-reportlog and --collect-only, or with a custom plugin as shown by @elbehery95 above. I don't think there's anything we should change here in the core, given that there are plenty of supported ways to do this already.