py_pytest_main and coverage

[0x2Adr1]

Hi ! I am trying to use py_pytest_main alongside pytest-cov to generate an HTML coverage report.

The py_pytest_main is working as expected, the only problem I have is that the HTML report is generated inside the bazel sandbox and I don't know how to have bazel copy it to bazel-testlogs/* or somewhere easily accessible so that I don't have to run my tests with --sandbox_debug and then having to manually find it using find -L ~/.cache/bazel -name "htmlcov" -type d

Is there a simple way of achieving that that I am missing?

My bazel test invocation is simply:

bazel test //:my_test --sandbox_debug --test_output=all

This is my BUILD file:

py_pytest_main(
    name = "__test__",
    args = [
        "--cov=my_test",
        "--cov-report=html",
    ],
)

py_library(
    name = "tests",
    srcs = glob(["*_test.py"]),
    deps = all_requirements,
)

py_test(
    name = "my_test",
    srcs = [
        ":__test__",
    ],
    main = ":__test__.py",
    deps = [
        ":tests",
    ],
)

My versions:

• rules_py 0.7.1
• rules_python 0.29.0
• bazel 7.0.2

[betaboon]

i don't know if this is the best solution, but here's what i currently do:

py_pytest_main.bzl:

load("@aspect_rules_py//py:defs.bzl", _py_pytest_main = "py_pytest_main")
load("@pip//:requirements.bzl", "requirement")

def py_pytest_main(name):
    _py_pytest_main(
        name = name,
        deps = [
            requirement("pytest"),
            requirement("pytest-asyncio"),
            requirement("pytest-cov"),
        ],
        args = [
            # report on all but passed tests
            # docs: https://docs.pytest.org/en/stable/how-to/output.html#producing-a-detailed-summary-report
            "-ra",
            # set verbositylevel
            # docs: https://docs.pytest.org/en/stable/how-to/output.html#verbosity
            "-vv",
            # enable capturing of specific warnings
            # docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html#controlling-warnings
            # docs: https://docs.pytest.org/en/stable/reference/reference.html#pytest.PytestUnhandledCoroutineWarning
            "-W error::pytest.PytestUnhandledCoroutineWarning",
            # enable automatic async test discovery
            # docs: https://pytest-asyncio.readthedocs.io/en/latest/concepts.html#auto-mode
            "--asyncio-mode=auto",
            # set package for which coverage should be collected
            # docs: https://pytest-cov.readthedocs.io/en/latest/config.html#referenc
            "--cov={}".format(native.package_name().replace("/", ".")),
            # enable branch coverage
            "--cov-branch",
            # NOTE normally we would have to set this.
            # but i have no clue how to get the output path in here.
            # thus i do it in a custom template
            # "--cov-report=lcov:$COVERAGE_OUTPUT_FILE",
        ],
        template = "//:pytest.py.tmpl",
        visibility = [":__pkg__"],
    )

pytest.py.tmpl:

# Copyright 2022 Aspect Build Systems, Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# adapted from: https://github.com/aspect-build/rules_py/blob/main/py/private/pytest.py.tmpl

import sys
import os

import pytest

if __name__ == "__main__":
    # Change to the directory where we need to run the test or execute a no-op
    $$CHDIR$$

    os.environ["ENV"] = "testing"

    args = [
        "--verbose",
        "--ignore=external/",
        # Avoid loading of the plugin "cacheprovider".
        "-p",
        "no:cacheprovider",
    ]

    junit_xml_out = os.environ.get("XML_OUTPUT_FILE")
    if junit_xml_out is not None:
        args.append(f"--junitxml={junit_xml_out}")

    # customization starts here
    coverage_lcov_out = os.environ.get("COVERAGE_OUTPUT_FILE")
    if coverage_lcov_out is not None:
        args.append(f"--cov-report=lcov:{coverage_lcov_out}")
    # customization ends here

    test_filter = os.environ.get("TESTBRIDGE_TEST_ONLY")
    if test_filter is not None:
        args.append(f"-k={test_filter}")

    user_args = [$$FLAGS$$]
    if len(user_args) > 0:
        args.extend(user_args)

    cli_args = sys.argv[1:]
    if len(cli_args) > 0:
        args.extend(cli_args)

    exit_code = pytest.main(args)

    if exit_code != 0:
        print("Pytest exit code: " + str(exit_code), file=sys.stderr)
        print("Ran pytest.main with " + str(args), file=sys.stderr)

    sys.exit(exit_code)

[qdii]

I'm trying to use your workaround, but I run into this error:

==================== Test output for //volleybot/tests:flow_roster_test:
ERROR: usage: __test__.py [options] [file_or_dir] [file_or_dir] [...]
__test__.py: error: unrecognized arguments: --asyncio-mode=auto --cov=volleybot.tests --cov-branch
  inifile: None
  rootdir: /home/qdii/.cache/bazel/_bazel_qdii/c4d89ef3654db70a58c3cc65bd37ef8e/sandbox/linux-sandbox/49/execroot/_main/bazel-out/k8-fastbuild/bin/volleybot/tests/flow_roster_test.runfiles/_main

Pytest exit code: 4
Ran pytest.main with ['--verbose', '--ignore=external/', '-p', 'no:cacheprovider', '--junitxml=/home/qdii/.cache/bazel/_bazel_qdii/c4d89ef3654db70a58c3cc65bd37ef8e/sandbox/linux-sandbox/49/execroot/_main/bazel-out/k8-fastbuild/testlogs/volleybot/tests/flow_roster_test/test.xml', '-ra', '-vv', '-W error::pytest.PytestUnhandledCoroutineWarning', '--asyncio-mode=auto', '--cov=volleybot.tests', '--cov-branch']
================================================================================

Does this ring a bell to you?

[betaboon]

@qdii not really, sorry.