Tracking issue for .NET Core support

[ckansiz]

I am trying to run dotnet core 2.0 test.
But it return an exception. (System.Reflection.TargetInvocationException ---> System.BadImageFormatException)

[BowserKingKoopa]

I have the same issue. Is there a fix for this?

[jcansdale]

I'm afraid there's no working .NET Core support at the moment. I want to get it working, but I have a bunch of other commitment which have been taking up my time. Sorry about that!

There is however support for executing ad-hoc test methods in .NET Standard projects, which you might find is useful in some situations. See #117

[ploeh]

I understand that you have to prioritise other tasks, but if we paid you, how much would it take to get support for .NET Core 2?

[jcansdale]

@ploeh,

I really want to get it working with .NET Core. I'm plotting to use next hack week at GitHub to work on it.

I'm also starting to use .Net Core myself (see git-pr), so I'm feeling the pain! This acts as motivation and will help knock the kinks out when I do add support. 😄

I think this project will help as well:
https://github.com/natemcmaster/DotNetCorePlugins

I had my own hacky implementation of this when I had TestDriven.Net working with .NET Core 1.1 / xproj, but @natemcmaster knows his stuff so I imagine this will work a lot better!

Thanks for your patience.

[natemcmaster]

Happy to provide more suggestions if you want. The DotNetCorePlugins project may not be the best fit. The DotNetCorePlugins project uses AssemblyLoadContext to simulate loading side-by-side assemblies, but there are endless edge cases, especially for test projects, and some other unexpected behaviors that can be hard to workaround. I know xunit's .NET Core console runner attempted something similar using their own AssemblyLoadContext implementation, but eventually decided to deprecate support for it.

An alternative I recommend investigating first is dotnet exec. Run dotnet exec with no args to see options. This can be used to start a new .NET Core process with an alternate entry assembly, but with the test project's original dependencies and configuration. As long as your entry point assembly if fairly simple, this is usually the easiest way to make things work well. It's how dotnet vstest and dotnet ef load and execute assemblies, and so far has been the most stable approach.

[jcansdale]

@natemcmaster,

An alternative I recommend investigating first is dotnet exec. Run dotnet exec with no args to see options. This can be used to start a new .NET Core process with an alternate entry assembly, but with the test project's original dependencies and configuration.

I've just blown the dust off the work I did with .NET Core in the days of .xproj. This is actually the technique I ended up using there. Thanks for the suggestion and reassurance that this maybe isn't a crazy way to go! 😉

I've just managed to execute an ad hoc test inside a modern .NET Core project. NUnit and xUnit tests aren't working, but I suspect most people use TestDriven.Net for executing ad hoc tests these days? 🙂

@ploeh 👆

[ploeh]

I suspect most people use TestDriven.Net for executing ad hoc tests these days?

I use it for that, but also very much for running all tests in my solutions.

[jcansdale]

@ploeh,

I have a plan for adding test framework support as well. It's likely to require an assembly attribute, but I'm optimistic it will work.

[BowserKingKoopa]

I use it exclusively for ad hoc testing. And I miss it so much.

[jcansdale]

I've finally manged to get the VSIX packaging working. Here's an experimental drop from my dev machine:
TestDriven.VSPackage_NetCoreExperimental.vsix.zip

I've tested running ad-hoc test methods with .NET Core console apps. 😄

Unfortunately targeting a .NET Core class library doesn't currently work (it can't see the target assembly). I guess they don't have the required *.runtimeconfig.json file to resolve their dependencies. I don't know if it's possible to do anything with *.deps.json or probing paths? What would my best bet here be @natemcmaster? 🙏

[natemcmaster]

.NET Core projects don't generate *.runtimeconfig.json files unless OutputType=Exe. Also, their .deps.json files are slightly different and will miss some assets require to run if the project has native dependencies, like SQLite. VSTest and Xunit made a decision not to support .NET Core and .NET Standard class libraries for this and a variety of other reasons. If you want to support this scenario, you might have to go it alone. I'm not 100% sure what it would take.

[jcansdale]

@natemcmaster it's all coming back now, I remember getting frustrated with this on my first attempt!

It appears to work if I piggy back on the *.runtimeconfig.json, *.runtimeconfig.dev.json and *.deps.json files of a console app that references the class library.

I could try auto-detecting a console app that references the target class library and automatically do this (warning of no or multiple candidates can be found). This would be a bit of a fiddle, but so convenient. 😄

Can you think of anything that is likely to go wrong if I tackle it this way?

[natemcmaster]

Can you think of anything that is likely to go wrong if I tackle it this way?

In general, this approach will be fraught with peril because it coerces a project into a different output format, so it seems icky. It might work for simple cases, but I suspect you'll run into issues with projects that attempt to use .deps.json at runtime or apps which use custom shared frameworks or runtimes. E.g. tests using Microsoft.AspNetCore.App metapackage in 2.1 or 2.2, .NET Core 3.0, Microsoft.AspNetCore.Mvc.Testing, or McMaster.NETCore.Plugins. It might work, but I would be surprised if it works flawlessly.

[bradwilson]

FWIW, our (@xunit) experiences trying to "run things" in .NET Core space makes me warn you to stay away. Our v3 is being completely restructured away from class libraries at all, at least in part so that supporting .NET Core in a first class way is doable. (All test projects in @xunit v3 will be executables on their target platform, and .NET Standard will remain 100% unsupported.)

[jcansdale]

@natemcmaster, @bradwilson,

After your cautions, I'm thinking of approaching this in a different way. I'm still hoping to allow the targeting of class library projects, but rather than attempting to create runtimeconfig / deps files for the class library, instead use a "surrogate" .NET Core console app that references the target class library for the actual execution.

This surrogate project could be explicitly defined as a project property (e.g. <TestProjectSurrogate>..\GitHub.Api\GitHub.Api.csproj</TestProjectSurrogate>) or implicitly as the first .NET Core console app in the solution that references the target class library.

I think for test projects it makes sense to specify the environment you want to test as a console app. With TestDriven.Net, people are used to being able to target any old project, so I need to find some kind of solution for this. Hopefully the above solution would work for .NET Core and .NET Standard class library projects, which would be nice. 😄

Does that make sense?

[duncanawoods]

Thanks for working on TestDriven again! I can run adhoc tests in a test assembly which I have already been forced to set to Exe for other tools (NUnit adaptor? Code Coverage? I can't recall).

The reasons I still miss TestDriven.Net:

• it was faster (no waiting on text explorer and it's "finding your tests" nonsense)
• captured console output in output window (currently forced to run tests in debug and pipe through debug messages)
• less log spam from test frameworks
• Ncover integration was awesome
• adhoc tests

Adhoc tests let me use a c# project as a kind of shell. I wouldn't write things like file processing scripts in bash/powershell but in a little static method and then run/debug with testdriven.net. Since moving to .net core, I have had to create console apps. Boo to friction!

[ploeh]

Here comes an ignorant question. It's not polemic, but admit that I'm curious.

The built-in Visual Studio test runner can run unit tests on .NET Core, both xUnit.net, NUnit, and MSTest. The UX is horrendous, which is why I prefer TestDriven.Net.

When the built-in test runner can run tests, however, then why is it so hard to make TestDriven.Net do it? Is the VS runner developed around a closed-source, proprietary API?

@jcansdale, I have great trust that you know what you're doing, so apparently it's not that simple. I don't, however, understand why it's not that simple. Is this something anyone can explain in a couple of minutes?

[jcansdale]

@ploeh Part of the answer is in the minimal ITestRunner interface that TestDriven.Net uses. I recommend reading https://github.com/dotnet/cli/issues/3974 for some background (thanks @plioi).

This allows for some optimizations that aren't possible using the much more complex API the Visual Studio test runner uses. I could develop an ITestRunner implementation that simply wraps the VS test runner, but it would be slow and wouldn't bring much value (I've always aimed for speed and simplicity).

My current plan is to:

1. Get ad-hoc tests working with .NET Core console and class library projects
2. Get ad-hoc tests working with .NET Framework projects
3. Get the ITestRunner interface working with .NET Core projects

Here is the xUnit implementation of ITestRunner:
https://github.com/xunit/xunit/blob/9d10262a3694bb099ddd783d735aba2a7aac759d/src/xunit.runner.tdnet/TdNetRunner.cs

Using the [assembly: CustomTestRunner(typeof(typeof(Xunit.Runner.TdNet.TdNetRunner)))] attribute lets TestDriven.Net know which test runner to use and ensures that the required infrastructure is available in the .NET Core application.

See here for how ITestRunner can currently be registered on a .NET Desktop project:
https://github.com/xunit/xunit/blob/9d10262a3694bb099ddd783d735aba2a7aac759d/test/GlobalTestAssemblyInfo.cs#L13

On .NET Desktop, test runners can also be found be looking as the assemblies that a target test project references. Unfortunately this is unlikely to work with .NET Core projects, so we'll need to explicitly use the CustomTestRunner attribute.

[ploeh]

@jcansdale, thank you for answering my question.

I could develop an ITestRunner implementation that simply wraps the VS test runner, but it would be slow and wouldn't bring much value.

FWIW, I disagree that it wouldn't bring value. I very much appreciate the speed of TestDriven.Net, but for me, the killer feature is the UX.

If I can get the best of both worlds, then I have nothing to complain about, but if I had to choose between

1. The speed of TestDriven.Net and the UX of the Visual Studio test runner
2. The UX of TestDriven.Net and the speed of the Visual Studio test runner

I'd pick the second option.

[natemcmaster]

This surrogate project could be explicitly defined as a project property

Makes sense what you're trying to do. If you can make it work, 👏 great. Fair warning, it's not going to be simple. One of the problems you'll have to address is NuGet/Home#6091. A ProjectReference doesn't transitively flow 'build assets'. If a test app references packages like Microsoft.AspNetCore.App or SQLite, the surrogate also needs to directly reference these, too, in order to have the packages' MSBuild targets imported.

[jcansdale]

Here is the first semi-official drop of TestDriven.Net that supports .NET Core.

TestDriven.VSPackage.5.0.18096-beta+f37144875b.zip

I've tested it with .NET Core 1.0, 2.1 and 3.0. For the moment you can only execute ad-hoc tests on .NET Core console apps. It's a start. 😉

Please report back with any issues you notice. 🙏

[jcansdale]

If you'd like to try WPF on .NET Core 3.0, I've changed ad hoc tests so that they run on a STA thread (like they default to on .NET Framework).

TestDriven.VSPackage-5.0.18099-beta+f6b86d2c75.zip

[jcansdale]

I hope you don't mind these little and often updates...

This version supports targeting async methods. I've also fixed an issue when targeting methods that have an attribute. It would assume it was a test method and warn that test frameworks aren't currently supported instead of executing the method. 😊

TestDriven.VSPackage-5.0.18100-beta+52c098c41c.zip

For example you can target the lterateAsync method below:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;

class AsyncTests
{
    static async Task lterateAsync()
    {
        await foreach(var i in Mylterator())
        {
            Console.WriteLine(i);
        }
    }

    static async IAsyncEnumerable<int> Mylterator()
    {
        try
        {
            for (int i = 0; i < 10; i++)
            {
                await Task.Delay(1000);
                yield return i;
            }
        }
        finally
        {
            await Task.Delay(200);
            Console.WriteLine("finally");
        }
    }
}


namespace System.Threading.Tasks
{
    using System.Runtime.CompilerServices;
    using System.Threading.Tasks.Sources;

    internal struct ManualResetValueTaskSourceLogic<TResult>
    {
        private ManualResetValueTaskSourceCore<TResult> _core;
        public ManualResetValueTaskSourceLogic(IStrongBox<ManualResetValueTaskSourceLogic<TResult>> parent) : this() { }
        public short Version => _core.Version;
        public TResult GetResult(short token) => _core.GetResult(token);
        public ValueTaskSourceStatus GetStatus(short token) => _core.GetStatus(token);
        public void OnCompleted(Action<object> continuation, object state, short token, ValueTaskSourceOnCompletedFlags flags) => _core.OnCompleted(continuation, state, token, flags);
        public void Reset() => _core.Reset();
        public void SetResult(TResult result) => _core.SetResult(result);
        public void SetException(Exception error) => _core.SetException(error);
    }
}

namespace System.Runtime.CompilerServices
{
    internal interface IStrongBox<T> { ref T Value { get; } }
}

<Project Sdk="Microsoft.NET.Sdk.WindowsDesktop">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <LangVersion>8.0</LangVersion>
    <NullableReferenceTypes>true</NullableReferenceTypes>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="System.Runtime.Extensions" Version="4.3.0" />
    <PackageReference Include="TestDriven.Framework" Version="2.0.0-alpha2" />
  </ItemGroup>

</Project>

Explanation here about the boilerplate: https://github.com/dotnet/coreclr/issues/21379.

[jcansdale]

Here's a new drop (TestDriven.VSPackage.zip) for you to try that takes account of the "Target Framework Moniker" drop down menu (see issue #150).

When you target a method in a source file, it will execute the framework that is currently selected for that file. When targeting a project, it will always use the first TFM defined in the TargetFrameworks list.

Remember only .NET Code projects with <OutputType>Exe</OutputType> in the project file are currently supported.

[kkorus]

My unit test project is .net core app 2.1 with <OutputType>Exe</OutputType> but I'm getting:
Running .NET Core unit tests isn't currently supported. Remote the test attribute to execute an "Ah hoc" test.

How I should add this remote test attribute?

@jcansdale I don't mind often updates 😉

[bradwilson]

My guess is it's a misspelling, and he meant remove.

[jcansdale]

@kkorus @bradwilson,

Oops, yes it was a typo! I hope this is better wording:
https://github.com/jcansdale/NetCoreTestRunner/pull/1/files#diff-11b54291c74c02c13bcf1dda16023ab8R138

I don't want people to think they're executing a test using a test runner when actually they're executing an ad-hoc method (without any setup/tear down infrastructure that a test runner might use). Hence requiring them to comment out the test attribute to run.

[kkorus]

@jcansdale @bradwilson
Thanks, removing test attribute made it work. I don't have a access to this repository 😉