Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement VSTest support to enable running and debugging .NET Core tests in .csproj projects #1100

Closed
aochsner opened this issue Dec 30, 2016 · 65 comments

Comments

@aochsner
Copy link

I realize I'm on pretty bleeding edge (.NET Core 1.1.0, .NET CLI 1.0.0 preview 4 (csproj), etc) but wanted to raise this and see if I can help w/ anything.

Environment data

.NET Core version

Microsoft .NET Core Shared Framework Host

  Version  : 1.1.0
  Build    : 928f77c4bc3f49d892459992fb6e1d5542cb5e86

dotnet --info output:

.NET Command Line Tools (1.0.0-preview4-004233)

Product Information:
 Version:            1.0.0-preview4-004233
 Commit SHA-1 hash:  8cec61c6f7

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.11-x64
 Base Path:   /usr/local/share/dotnet/sdk/1.0.0-preview4-004233

VS Code version:
Version 1.8.1 (1.8.1)

C# Extension version:
1.6.2

Steps to reproduce

Create new xunit project
dotnet new -t xunittest

Update references to 1.1.0 in csproj:

<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="15.0">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp1.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="**\*.cs" />
    <EmbeddedResource Include="**\*.resx" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NETCore.App" Version="1.1.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20161123-03" />
    <PackageReference Include="xunit" Version="2.2.0-beta4-build3444" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0-beta4-build1194" />
  </ItemGroup>

</Project>

Open in VS Code and navigate to Test.cs.
Run Test via Omnisharp

Expected behavior

Test runs

Actual behavior

"Failed to run test because null" error

@vcsjones
Copy link
Member

This is the only thing holding me back from adopting the preview4 tooling. While "run test" is a less severe of a problem (I can just do that from the command line...) "debug test" has the same problem, and I don't have a good work around for that.

@VladimirMakaev
Copy link

+1

@DustinCampbell
Copy link
Member

Upon fixing the null reference (which was due to some ugly code that looked specifically for the presence of project.json), it looks like this might be quite seriously broken. Previously, OmniSharp just called dotnet test with --port and --parentProcessId flags to start communicating with the test framework. However, now that the .NET CLI has moved over to MSBuild (and VSTest), it's not clear whether this is possible any longer.

@piotrpMSFT, do you have any thoughts here? I'm guessing OmniSharp will need to commnicate a bit differently with VSTest using the details from https://github.com/microsoft/vstest. Do we just need to use dotnet vstest?

@DustinCampbell
Copy link
Member

OK. I'm making some progress with dotnet vstest using the spec at https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0007-Editors-API-Specification.md. 😄

@jchannon
Copy link
Contributor

jchannon commented Feb 3, 2017

Keep in mind xUnit are looking to make their own command eg/dotnet xunit so not sure if that would change how tests are run/debugged in vscode @bradwilson

@DustinCampbell
Copy link
Member

I don't think it would. I think we'd still want to go through the VSTest protocol. Otherwise, we'd be writing custom code for every test framework that's supported.

@jchannon
Copy link
Contributor

jchannon commented Feb 3, 2017 via email

@bradwilson
Copy link

I think we'd still want to go through the VSTest protocol.

I agree, you should. dotnet xunit is about providing an equivalent to xunit.console.exe for .NET SDK applications. It is not intended to be automated by third parties.

@DustinCampbell DustinCampbell added Backlog and removed Bug labels Feb 6, 2017
@DustinCampbell DustinCampbell added this to the 1.8 milestone Feb 6, 2017
@DustinCampbell DustinCampbell changed the title .NET Core 1.1.0 XUnit: Failed to run test because null Running and debugging .NET Core tests is broken for .csproj projects Feb 6, 2017
@DustinCampbell DustinCampbell changed the title Running and debugging .NET Core tests is broken for .csproj projects Implement VSTest support to enable running and debugging .NET Core tests in .csproj projects Feb 6, 2017
@ErikSchierboom
Copy link

Will there be a .NET: Test command added to VS Code, similar to the .NET: Restore Packages command? That would be incredibly useful I feel.

@DustinCampbell
Copy link
Member

Eventually. This just tracks getting the CodeLens indicators for "run test" and "debug test" working for .csproj.

@ErikSchierboom
Copy link

Ah, I missed that. That would also be brilliant!

@Jonathan34
Copy link

ah well, another road blocker with xunit and vscode :(
hope the 1.8.0 is coming very soon to fully support csproj and xunit

@Jonathan34
Copy link

Jonathan34 commented Mar 8, 2017

as a workaround: you can edit your tasks.json to add the test tasks you want, bind a shortcut like this:

[{
    "key": "shift+cmd+t",
    "command": "workbench.action.tasks.runTask",
    "when": "editorTextFocus"
}]

and select the test task.
if you want to run all the tests, you can create dependencies using the tasks.json v2.0.0 (cf https://code.visualstudio.com/updates/v1_10#_more-work-on-terminal-runner)

alternatively, switch to vs 2017 :)

@DustinCampbell
Copy link
Member

Great suggestion!

@bouassaba
Copy link

@Jonathan34 Could you please clarify more the procedure that you suggested ? Could you start a debugging session for a specific test method using VSCode's tasks.json/launch.json ?

@DustinCampbell
Copy link
Member

OK. This should be working in the latest beta: 1.9.0-beta3. Please try it out!

Please use the instructions for Installing Beta Releases.

@powellcj12
Copy link

@DustinCampbell generally seems to be working for me on Mac (10.12.5 beta)!

When I initially installed the beta extension, I was still seeing various errors about not being able to attach the debugger, sometimes the test would never actually run, and one time even a message from VS Code that it was corrupt which linked me here. After reinstalling the app though (I'm running the 1.12.0-Insider Mac build), haven't run into any of those issues since.

@achimschrepfer
Copy link

@DustinCampbell seems to work on Mac VSCode 1.11.2 and OmniSharp 1.9.0-beta3 - I had the same experience as @powellcj12 when I first started the debugger. But after then it worked like a charm (without reinstall).

@DustinCampbell
Copy link
Member

@powellcj12, @achimschrepfer: Did you uninstall your previous version of C# and restart VS Code before installing the new beta? If not, that would explain why the debugger didn't work immediately (sometimes, VS Code likes to run an older version of an extension if it's still on disk).

@powellcj12
Copy link

Yes, I uninstalled the previous extension (which was the previous v1.9 beta), VS Code prompted me to reload (which I did), and then installed the new extension.

@DustinCampbell
Copy link
Member

OK. Just wanted to check.

@richlander
Copy link
Member

I wasn't able to get this to work on Windows. I installed the beta VSIX and uninstalled the prod one first. I found that the the extension was doing more than before but not the whole job and not reliably. It sometimes ran the test and sometimes gave me the error message that it currently does with the prod version. I can share my code with you if you'd like to try it (private repo).

@DustinCampbell
Copy link
Member

Hey @richlander. any chance you'll be in the office tomorrow? If so, maybe we can connect I can take a look at your machine?

@achimschrepfer
Copy link

Yep, I also uninstalled the previous version of OmniSharp and restarted VSCode as well.

@richlander
Copy link
Member

Interesting. Just tried on my work PC (both "run" and "debug"). Works! Yeahh! Last night, I was using my machine at home. I'll try again tonight. Maybe it needs a harder kick.

Thanks for getting this working!

What do you think on performance? The VS experience (unscientifically) feels significantly faster for running single tests.

@DustinCampbell
Copy link
Member

I'll try again tonight. Maybe it needs a harder kick.

Check your %UserProfile%.vscode\extensions folder. I've seen situations (though rarely) where VS Code leaves old extensions lying around and they still get loaded.

What do you think on performance? The VS experience (unscientifically) feels significantly faster for running single tests.

Baby steps @richlander. 😄 VS does a significantly amount of caching (and has an entire team devoted) to making the test experience good. Everytime we run a test, we need to shell out to the .NET CLI to build (with no fast up-to-date checks like VS), and shell out to the .NET CLI again to go through the "test discovery" phase and then do the actual running of the test once it's discovered. I have some ideas to improve this in the future, but getting it working is the high-order bit for 1.9.

@vcsjones
Copy link
Member

For what its worth, this doesn't feel any slower to me than when this was working with project.json projects.

@DustinCampbell
Copy link
Member

Thanks @vcsjones: It shouldn't, but I'd still like to make it better. 😄

@DustinCampbell
Copy link
Member

OK. A release candidate for 1.9.0 is built and ready for testing. Once we feel good about this build, we'll push it to the marketplace (hopefully tomorrow -- we'll see! 😄)

@ghost
Copy link

ghost commented Apr 24, 2017

I'm able to test now using the debugger and the launch tasks generated by VSCode.

However, when using the "debug test" option, I'm still seeing this issue.

Using 1.9.0 version of this plugin.

screenshot 2017-04-24 19 40 38

@DustinCampbell
Copy link
Member

Are you referring to the code lens items being out of order? If so, that's tracked with #936.

@ghost
Copy link

ghost commented Apr 25, 2017

No, while that is annoying, I'm referring to clicking the CodeLens "debug test" item and then receiving the message: "Failed to run test because null" as described in this issue originally.

The "run test" item works fine.

I'm running 1.9.0 of the omnisharp extension, 1.11.2 of VSCode.

@DustinCampbell
Copy link
Member

Could you go ahead and file a new issue?

@Jonathan34
Copy link

works fine with 1.9.0 and vscode 1.12.0 and .net core 1.1
thanks!

@powellcj12
Copy link

@fedoranimus I saw the same thing temporarily - can't seem to hit it any more, but my initial suspicion was that it was related to the specific .NET runtime I was targeting...

.NET Core 1.1 worked fine. I changed it to target .NET Standard 1.4 which resulted in some errors restoring packages, so I subsequently changed it to .NET Standard 1.5. At this point the 'debug' option in the extension wasn't working. I restored to .NET Core 1.1 and things started working again, but now everything works regardless of if I target .NET Core 1.0 or .NET Standard 1.4-1.6 so not really sure what might be happening. Still, might be worth checking the runtime you're targeting?

@ghost
Copy link

ghost commented Apr 25, 2017

Thanks @powellcj12 I will double check.

If the situation persists, I will open a new issue tonight @DustinCampbell.

@DustinCampbell
Copy link
Member

Thanks @fedoranimus!

@ghost
Copy link

ghost commented Apr 25, 2017

#1428 Created to address the new issue

@jmezach
Copy link

jmezach commented Apr 29, 2017

This isn't working yet for MSTest projects. Is that a separate issue, or should that work as well now?

@DustinCampbell
Copy link
Member

Separate issue please. Nothing has been done to support MSTest yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests