Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XUnit projects cannot be launched under the debugger #1292

Closed
aggieben opened this issue Mar 7, 2017 · 25 comments
Closed

XUnit projects cannot be launched under the debugger #1292

aggieben opened this issue Mar 7, 2017 · 25 comments

Comments

@aggieben
Copy link

aggieben commented Mar 7, 2017

The symptoms are related to those described in #1253, but wasn't solved with the thing the closed that one. Environment info is the same.

λ dotnet --info
.NET Command Line Tools (1.0.0-rc4-004771)

Product Information:
 Version:            1.0.0-rc4-004771
 Commit SHA-1 hash:  d881d45b75

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14393
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.0.0-rc4-004771

@gregg-miskelly:

Do you always see the program has exited with code 0 or is it intermittent?
If you always get it --

  • What is program set to in launch.json?
  • Can you send me the output you get in your console window?

Yes, the issue is 100% repeatable. In this case, the problem is with an Xunit project, called Saddleback.AspNetCore.Features.Test. The project file as as follows:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>netcoreapp1.0;netcoreapp1.1</TargetFrameworks>
    <NetStandardImplicitPackageVersion>1.6.1</NetStandardImplicitPackageVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="xunit" Version="2.2.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.TestHost" Version="1.1.0"/>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="../../src/Saddleback.AspNetCore.Features/Saddleback.AspNetCore.Features.csproj"/>
    <ProjectReference Include="../Saddleback.Logging.Xunit/Saddleback.Logging.Xunit.csproj"/>
  </ItemGroup>

</Project>

The launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": ".NET Core Launch (console)",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "build",
            "program": "${workspaceRoot}\\test\\Saddleback.AspNetCore.Features.Test\\bin\\Debug\\netcoreapp1.0\\Saddleback.AspNetCore.Features.Test.dll",
            "args": [],
            "cwd": "${workspaceRoot}",
            "externalConsole": false,
            "stopAtEntry": false,
            "internalConsoleOptions": "openOnSessionStart"
        },
        {
            "name": ".NET Core Attach",
            "type": "coreclr",
            "request": "attach",
            "processId": "${command:pickProcess}"
        }
    ]
}

the console output:

Microsoft (R) Build Engine version 15.1.545.13942
Copyright (C) Microsoft Corporation. All rights reserved.
  Saddleback.AspNetCore.Features -> d:\Users\ben\proj\sb\SaddlebackFeatures\src\Saddleback.AspNetCore.Features\bin\Debug\netstandard1.4\Saddleback.AspNetCore.Features.dll
  Saddleback.Logging.Xunit -> d:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.Logging.Xunit\bin\Debug\netstandard1.4\Saddleback.Logging.Xunit.dll
  Saddleback.AspNetCore.Features.Test -> d:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.AspNetCore.Features.Test\bin\Debug\netcoreapp1.0\Saddleback.AspNetCore.Features.Test.dll
  Saddleback.AspNetCore.Features.Test -> d:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.AspNetCore.Features.Test\bin\Debug\netcoreapp1.1\Saddleback.AspNetCore.Features.Test.dll
Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:07.60

and the debug console output:

--------------------------------------------------------------------------------
You may only use the Microsoft .NET Core Debugger (clrdbg) with Visual Studio
Code, Visual Studio or Visual Studio for Mac software to help you develop and
test your applications.
--------------------------------------------------------------------------------
Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.0.3\System.Private.CoreLib.ni.dll'. Cannot find or open the symbol file.
Loaded 'd:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.AspNetCore.Features.Test\bin\Debug\netcoreapp1.0\Saddleback.AspNetCore.Features.Test.dll'. Symbols loaded.
Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.0.3\System.Runtime.dll'. Cannot find or open the symbol file.
Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.0.3\mscorlib.dll'. Cannot find or open the symbol file.
The program 'd:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.AspNetCore.Features.Test\bin\Debug\netcoreapp1.0\Saddleback.AspNetCore.Features.Test.dll' has exited with code 0 (0x00000000).
@gregg-miskelly gregg-miskelly changed the title Trouble running debugger with csproj tooling, part 2 XUnit projects cannot be launched under the debugger Mar 7, 2017
@gregg-miskelly
Copy link
Contributor

@aggieben I think the issue here is that XUnit projects aren't runnable in this way -- it either doesn't run your tests at all, or it runs them in a child process. You can confirm or Deny this theory by adding a Console.WriteLine to one of your tests to see if it runs. If it does run have it print its process id and then pause (ex: Thread.Sleep(10000);). You can then use ps -A to find out if it is the process that the debugger launched.

@gregg-miskelly
Copy link
Contributor

I should say, in theory the right way to debug a test is to use the little button next to the test method. But from #1100 I gather that that is currently broken for .csproj-based projects.

@aggieben
Copy link
Author

aggieben commented Mar 7, 2017

@gregg-miskelly indeed, the test launch links decorating the methods do not work for me. If I click run, I get the "Failed to run test because null" error, and if I click debug, nothing at all happens.

@DustinCampbell
Copy link
Member

@aggieben: The "Run Test" and "Debug Test" CodeLens links don't work yet for .csproj-based .NET Core projects. This is tracked by #1100.

@fl3pp
Copy link

fl3pp commented Jul 9, 2017

also doesnt work for me:

  <ItemGroup>
    <PackageReference Include="xunit" Version="2.3.0-beta2-build3683" />
     <DotNetCliToolReference Include="dotnet-xunit" Version="2.3.0-beta2-build3683" /> 
  </ItemGroup>

debugger doesnt hit breakpoints when xunit-cli tool is added

works again when commenting out from csproj, removing bin/ and obj/ folder and restoring

full csproj

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>netcoreapp1.1</TargetFramework>
    </PropertyGroup>


<!-- 
comment out one to debugg / run tests 
always run a restore after changing
-->

<!--
  <ItemGroup>
    <PackageReference Include="xunit" Version="2.3.0-beta2-build3683" />
     <DotNetCliToolReference Include="dotnet-xunit" Version="2.3.0-beta2-build3683" /> 
  </ItemGroup>
-->
    <ItemGroup>
        <Compile Remove="mdparser-tests/*" />
    </ItemGroup>

</Project>

I also tryed to reinstall omnisharp, c# extension and dotnet core
i got this issue on two different computers

@gregg-miskelly
Copy link
Contributor

@jflepp I might suggest opening a new issue. But either way, a few questions:

  • What do you see in the debug console?
  • You are on C# extension 1.11.0?
  • What OS are you on?

@fl3pp
Copy link

fl3pp commented Jul 10, 2017

@gregg-miskelly I created a new issue: #1630

@kamranayub
Copy link

I'm having similar problems with the new .NET 2.0 (I never tried with earlier SDKs).

I had no launch.json and so VS Code created a new one for me but I see the same thing where tests aren't run, the program just exits. Is there something special I need to initiate the running of tests?

My repo is public if you want to test: https://github.com/kamranayub/GiantBomb-CSharp/tree/net20

I'll edit/update if I end up figuring it out, I wish it wasn't so hard 😢

@DustinCampbell
Copy link
Member

The fact that you can F5 an xUnit project is really a bit strange. To be honest, C# for VS Code really shouldn't offer to create a launch.json file for them, but xUnit projects are created as "netcoreapp2.0" with no entry point.

The trick to debugging unit tests is to open a file containing unit tests and then click the "debug test" annotation above the method:

image

@kamranayub
Copy link

kamranayub commented Aug 18, 2017 via email

@DustinCampbell
Copy link
Member

Here's what I see with your branch:

Debugging method 'GiantBomb.Api.Tests.Games.game_resource_should_return_game_for_33394'...

Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  GiantBomb.Api -> c:\Projects\GiantBomb-CSharp\GiantBomb.Api\bin\Debug\netstandard1.2\GiantBomb.Api.dll
  Successfully created package 'c:\Projects\GiantBomb-CSharp\GiantBomb.Api\bin\Debug\GiantBomb.Api.2.4.0.nupkg'.
  GiantBomb.Api.Tests -> c:\Projects\GiantBomb-CSharp\GiantBomb.Api.Tests\bin\Debug\netcoreapp2.0\GiantBomb.Api.Tests.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:05.29


An exception occurred while test discoverer 'VsTestRunner' was loading tests. Exception: Object reference not set to an instance of an object.
Debugging complete.

Is that the same thing you're seeing?

@DustinCampbell
Copy link
Member

FWIW, I think this is a completely different problem than what this issue is tracking.

@kamranayub
Copy link

kamranayub commented Aug 18, 2017

Yes but I already figured the debug test link doesn't work (I think there are other issues tracking that). My issue is the regular debug launch in VS Code (using start button) doesn't seem to work and ends up with the same output as @jflepp above.

Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.0\System.Private.CoreLib.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Loaded 'e:\Development\Contrib\GiantBomb-CSharp\GiantBomb.Api.Tests\bin\Debug\netcoreapp2.0\GiantBomb.Api.Tests.dll'. Symbols loaded.
Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.0\System.Runtime.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
The program '[34748] GiantBomb.Api.Tests.dll' has exited with code 0 (0x0).

I am not sure if it's by design to not work but I'm looking for the same experience I get in Visual Studio 2017 where debugging tests by launching the debugger does work.

@DustinCampbell
Copy link
Member

As I mentioned earlier, I wouldn't expect that to work and its somewhat of a bug that we create a launch.json at all for xUnit test projects. Pressing F5 on test project is akin to launching it with 'dotnet run'. It doesn't launch it with the test framework.

@DustinCampbell
Copy link
Member

DustinCampbell commented Aug 18, 2017

FWIW, the issue I'm seeing looks like it might be a bug introduced in xUnit. If I use the version of xUnit that dotnet new xunit generates (2.2.0), run and debug tests works fine. If I moved to the latest xUnit preview that you're using (2.3.0-beta4-build3742), it fails with An exception occurred while test discoverer 'VsTestRunner' was loading tests. Exception: Object reference not set to an instance of an object.

@kamranayub
Copy link

@DustinCampbell Interesting! I just followed the latest docs of xunit, I'll try the earlier version maybe... it said the latest beta 4 adds .NET Core 2.0 support.

@DustinCampbell
Copy link
Member

The latest beta 4 definitely should be much better. IIRC, it fixes issues with running tests in VS. I'm debugging this now to see if I can figure out where the issue might be.

@DustinCampbell
Copy link
Member

Here's the exception that's happening:

TpTrace Error: 0 : 16628, 7, 2017/08/18, 13:00:27.519, 129122016173, testhost.dll, Exception: System.NullReferenceException
	Message: Object reference not set to an instance of an object.
	Stack Trace:    at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.ValidateRuntimeFramework() in C:\projects\xunit\src\xunit.runner.visualstudio\VsTestRunner.cs:line 724
   at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestDiscoverer.DiscoverTests(IEnumerable`1 sources, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink) in C:\projects\xunit\src\xunit.runner.visualstudio\VsTestRunner.cs:line 81
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Discovery.DiscovererEnumerator.LoadTestsFromAnExtension(String extensionAssembly, IEnumerable`1 sources, IRunSettings settings, IMessageLogger logger)
	BaseExceptionMessage: Object reference not set to an instance of an object.

@DustinCampbell
Copy link
Member

The issue appears to be right here:

        /// <summary>
        /// Validates the runtime target framework from test platform with the current adapter's target.
        /// </summary>
        /// <returns>True if the target frameworks match.</returns>
        static bool ValidateRuntimeFramework()
        {
#if NETCOREAPP1_0
            var targetFrameworkVersion = RunSettingsHelper.TargetFrameworkVersion;

            return targetFrameworkVersion.StartsWith(".NETCore", StringComparison.OrdinalIgnoreCase) ||
                   targetFrameworkVersion.StartsWith("FrameworkCore", StringComparison.OrdinalIgnoreCase);
#else
            return true;
#endif
        }

The problem is that our test runner passes RunSettings to the dotnet vstest that do not include a TargetFrameworkVersion and xUnit expects it to be there. The change was introduced with xunit/xunit@d22f7c1.

@DustinCampbell
Copy link
Member

Yup, that's the issue. If I change OmniSharp to pass a run settings file containing <TargetFrameworkVersion>.NETCoreApp, Version=2.0</TargetFrameworkVersion> it works fine. I don't think xUnit is at fault here, but we need to do a bit more work to make it work right. I've filed OmniSharp/omnisharp-roslyn#944 to track this work.

@kamranayub
Copy link

@DustinCampbell Great, thanks!

@Neutrino-Sunset
Copy link

Any idea when the fix will go live. Unless I've misunderstood it's currently impossible to debug either xunit or mstest test which is a major showstopper.

@DustinCampbell
Copy link
Member

Not yet, but there is a beta release that you can use in the meantime: 1.13.0-beta3. You can use the instructions here to install a beta release.

@Neutrino-Sunset
Copy link

Awesome, thanks Dustin. I've also just worked out that I can run an xunit test, but I just have to enable this horrid codelens thing. Looking forward to the time I can turn that back off.

@gregg-miskelly
Copy link
Contributor

Closing stale issue

@gregg-miskelly gregg-miskelly closed this as not planned Won't fix, can't repro, duplicate, stale Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants