No way to determine the runtime used with dotnet test

[jaredpar]

Description

Our repository has run into a situation where we have tests failing in one environment but not the other. This happens on occasion but in this specific case it's failing locally but passing in CI. There is some suspicion that it's because the environments are picking different runtimes when invoking dotnet test.

The problem is we have no way of knowing what runtime is being used when dotnet test is invoked. It's not listed in the command line, logs, etc ... It's left up to the developer to reason this out and we're failing to do so.

Expected behavior

There should be some mechanism that lets us know what runtime actually executed our tests.

Actual behavior

There is no such mechanism. The only options available are

1. Use %COREHOST_TRACE%. This produces a ~45,000 line file with multiple host activations that we have to trace through to find the runtime.
2. Attach debuggers to running processes to see what version got loaded

None of these are viable for CI environments

[MarcoRossignoli]

It's not listed in the command line, logs

.NET Core project is built as an exe without apphost to have the correct runtimeconfig.json to use during the test run(dotnet test testhost.dll --runtimeconfig ...).
If you append --diag:log.txt to the dotnet test verb you'll produce logs where in case of failure is clear why testhost failed to start. Usually we see the tipical .NET fxr error that report the missing runtime and it should be also visible in console(but sometimes we saw scenario where is not shown in console):

[MarcoRossignoli]

In the log(log.txt) you'll find also the actual command line for the test host if you search for DotnetTestHostManager:

TpTrace Information: 0 : 39608, 5, 2022/06/23, 09:48:36.837, 6689863157986, vstest.console.dll, SocketServer.Start: Listening on endpoint : 127.0.0.1:62577
TpTrace Verbose: 0 : 39608, 5, 2022/06/23, 09:48:36.841, 6689863204397, vstest.console.dll, DotnetTestHostmanager.GetTestHostProcessStartInfo: Platform environment 'X64' target architecture 'X64' framework '.NETCoreApp,Version=v7.0' OS 'Windows'
TpTrace Verbose: 0 : 39608, 5, 2022/06/23, 09:48:36.841, 6689863206029, vstest.console.dll, DotnetTestHostmanager: Adding  --runtimeconfig "C:\git\localPlayground\xxxx\bin\Debug\net7.0\jared.runtimeconfig.json" in args
TpTrace Verbose: 0 : 39608, 5, 2022/06/23, 09:48:36.842, 6689863206725, vstest.console.dll, DotnetTestHostmanager: Adding  --depsfile "C:\git\localPlayground\xxxxx\bin\Debug\net7.0\jared.deps.json" in args
TpTrace Verbose: 0 : 39608, 5, 2022/06/23, 09:48:36.842, 6689863207874, vstest.console.dll, DotnetTestHostmanager.IsWinOnArm: Current PROCESSOR_ARCHITECTURE from environment variable 'AMD64'
TpTrace Verbose: 0 : 39608, 5, 2022/06/23, 09:48:36.842, 6689863208298, vstest.console.dll, DotnetTestHostmanager.IsWinOnArm: Is Windows on ARM 'False'
TpTrace Verbose: 0 : 39608, 5, 2022/06/23, 09:48:36.842, 6689863208913, vstest.console.dll, DotnetTestHostManager: testhost.exe found at path: C:\git\localPlayground\xxxxx\bin\Debug\net7.0\testhost.exe
TpTrace Verbose: 0 : 39608, 5, 2022/06/23, 09:48:36.842, 6689863209224, vstest.console.dll, DotnetTestHostmanager: Full path of host exe is C:\git\localPlayground\xxxxx\bin\Debug\net7.0\testhost.exe
TpTrace Verbose: 0 : 39608, 5, 2022/06/23, 09:48:36.842, 6689863210040, vstest.console.dll, DotnetTestHostmanager.LaunchTestHostAsync: Found 'VSTEST_WINAPPHOST_DOTNET_ROOT' in env variables, value 'C:\Program Files\dotnet', forwarding to 'DOTNET_ROOT'
TpTrace Verbose: 0 : 39608, 7, 2022/06/23, 09:48:36.843, 6689863217250, vstest.console.dll, DotnetTestHostManager: Starting process 'C:\git\localPlayground\xxxxx\bin\Debug\net7.0\testhost.exe' with command line ' --runtimeconfig ...
TpTrace Verbose: 0 : 39608, 7, 2022/06/23, 09:48:36.849, 6689863276954, vstest.console.dll, Test Runtime launched

[jaredpar]

@MarcoRossignoli

If you append --diag:log.txt to the dotnet test verb you'll produce logs where in case of failure is clear why testhost failed to start.

The issue isn't that the test host isn't starting, the issue is that we don't know which runtime the test host is using. That makes it very hard to understand why tests fail in one environment but pass in another. One variable to consider is "which runtime is being used in each instance?". There is no reasonable way to answer this question that we're aware of.

[MarcoRossignoli]

Ok I've misunderstood,

dotnet test can run host for netcore/netfx and depends also on some switch(--arch) and env vars so it's not easy to know the runtime upfront. One idea can be find a cross way(we support netstandard and uwp) to locate/recognize the current runtime and print it at the top of host log for debugging purpose.

[MarcoRossignoli]

@jaredpar we could add on top of the host log the typeof(object).Assembly.Location(where supported, I suppose you're on .NET Core) where we can see location and version of the runtime, does it work for you?
It's useful also in case of private installation.

[MarcoRossignoli]

This is the possible log enhancement