Support test discovery in sources that are directories

[shyamnamboodiripad]

At the moment, test platform only supports discovery of tests present inside test sources that are files. With the change in this PR, it can additionally support discovery of tests present inside test sources that are directories. This helps test adapters for certain languages like python which prefer to discover tests at the granularity of a directory as opposed to individual files.

Python adapter currently supplies test containers that are individual .py files to the VS test explorer. However, when these files are sent to the python test discoverer, the discoverer ignores the individual files and always jumps up to the containing project's directory to discover tests present inside all .py files present under this directory.

Python adapter has some reasons for this behavior (tickets linked below contain more context). However, there doesn't seem to be any significant reason why test platform can't support test discovery at a directory level granularity.

This PR relaxes the requirement that supplied test containers (sources) must always be files. It now also allows test containers that are directories. It also introduces a new [DirectoryBasedDiscoverer] attribute that can be applied on ITestDiscoverers that only care to process sources that are directories (similar to the existing [FileExtension] attribute that can be applied on ITestDiscoverers that only care to process sources that are files with specific extensions).

Related tickets:
https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1586970
https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1560000

[MarcoRossignoli]

@nohwnd does it work with all the "above" part related to inferring the host?

[nohwnd]

Thinking about this a bit more:

The reason this capability just (accidentally) works, is because we provide a default runtime environment for non-dll sources. The hidden assumption here is that we will start a testhost.exe to run the adapter, which will then do the work on their own or delegate to additional executable (e.g. xmlTests can be handled in process, but .py tests need to be delegated to python). And we also assume that the provided file type represents only one type of runtime, or that the adapter can detect the correct runtime and start it itself (e.g. .py can be started with python2.exe or python3.exe based on metadata in the file, or if someone would want to implement this for usage with dlls, they would have to multiplex to .NET, .NET Framework, UWP, WinUI and Native).

I guess this brings us back to discussion I think we had in the past: We use the sources to indirectly determine which TestRuntime provider should be used. In runner we check TargetFramework, and if not specified, we auto-detect TargetFramework by looking at the dll metadata. TargetFramework is then used by TestRuntime providers to determine if they can run the workload or not.

So while it is not blocking for this PR, it would be nice to determine a way to provide additional metadata on the source level, for us to skip some of those checks. Especially in cases where VS possibly has all the info already e.g. what dll type this is, what dlls the project references, and if there are any well known adapters among them.

[shyamnamboodiripad]

@nohwnd Totally agree that it would be great if source could be an open-ended concept. This could open up other interesting scenarios where the source used need not be present on disk at all (for example, it could be a URL to some online resource which serves the appropriate tests and which the adapter knows to interpret).

I don't think it is bad to have a reasonable default when a target runtime is not specified. But it would certainly be nice if there was a way to provide additional metadata about target runtimes. In cases where adapter knows it will be the same runtime for all sources (like python), perhaps the adapter could specify this declaratively via an attribute. For other cases (especially dll-based) VS could provide info about source-level target runtimes via new API / runsettings. Iirc, VS already provides the TargetFramework, does it not?

Detecting precise adapter references for a project could be done - but this may be a tricky change as it can impact backwards compatibility. Currently VS detects and passes all adapters referenced in the solution regardless of the set of adapters actually referenced by the tests being executed. But changing this would also prevent some redundant sniffs in the adapters - so this would also be nice to optimize down the line. Perhaps VS can have a special mode where it would only pass relevant adapters to Test Platform and users could explicitly opt into the new behavior... We could potentially also enable the new mode by default for newly added test projects (by changing the VS test project templates) ...

[nithinbandaru1]

Does this even work? @nohwnd

By the way it makes sense that there can be a Python Test Adaptor that works with VS Test as it used JSON based socket protocol. Probably that is part of python SDK that comes with visual studio? Good to know.

When I give directory instead of DLL path, I still get error.

Command:

D:\prototype> "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" Smpl.Calc.Lib.UT.NuTest\bin\Debug\net8.0\

Error:

Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

The test source file "D:\prototype\Smpl.Calc.Lib.UT.NuTest\bin\Debug\net8.0" provided was not found.

[nohwnd]

Yes the python adapter can be installed with VS.