Properly strip source root prefixes for V2 Pytest runner

[Eric-Arellano]

Problem

#7696 introduced support for source roots to V2 Pytest for the first time, but it did not support loose files. #8063 tried to fix this by no longer stripping the source root from source files. However, this caused two issues:

1. Regression that repr() now shows the full path, not the relativized path: Properly handle source roots and resources with V2 Pytest runner #8063 (comment)
2. Namespace packages, such as contrib code, do not work with V2. A namespace package is where multiple folders have the same package name, like pants, but may not within the same actual folder.
   • Contrib unit tests would fail because PYTHONPATH had two entries referring to pants: src/python/pants and contrib/mypy/src/python/pants. Python would use whichever entry came first and ignore the other, which does not work. Instead, we want those two to merge into one namespace package.

Solution

Follow the V1 Pytest approach of stripping the source root, like we used to, but only for source files. Loose files (i.e. files()) are not stripped so that the file system APIs work as expected. This approach comes from:

pants/src/python/pants/backend/python/subsystems/pex_build_util.py

Lines 101 to 110 in d048fd2

	def get_chroot_path(relpath: str) -> str:
	if type(tgt) == Files:
	# Loose `Files`, as opposed to `Resources` or `PythonTarget`s, have no (implied) package
	# structure and so we chroot them relative to the build root so that they can be accessed
	# via the normal Python filesystem APIs just as they would be accessed outside the
	# chrooted environment. NB: This requires we mark the pex as not zip safe so
	# these `Files` can still be accessed in the context of a built pex distribution.
	builder.info.zip_safe = False
	return relpath
	return str(Path(relpath).relative_to(tgt.target_base))

Result

V2 Pytest now supports both source roots and loose files! This allows us to run most contrib unit tests with V2 and unblocks #8113.

Because over 95% of unit tests are now remoted, we go back to only one CI shard for unit tests.

[stuhood]

Thanks a lot!

[stuhood]

Your bootstrap shards raced another PR landing. Have rebased.

[stuhood]

There appear to be some actual failures in CI.

[Eric-Arellano]

b464557 fixes the unit test failure. Test discovery was failing for test_coursier_resolve because the side-effects of importing psutil.tests caused unittest.TestCase to have __init__() defined, meaning that Pytest would no longer discover the targets.

Very unclear why this was not happening with V1 test runner or V2 before this change..not going to spend more time trying to answer that, though.