Need documentation/examples for setting references used by tests

[paul1956]

Update to remove duplicate issues with #598

It is not obvious how you add a reference to Roslyn into tests. Changing VerifyAnalyzerAsync is hard to discover and doesn't seem to work.

/// <inheritdoc cref="CodeFixVerifier(Of TAnalyzer, TCodeFix, TTest, TVerifier).VerifyAnalyzerAsync(String, DiagnosticResult())"/>
    Public Shared Async Function VerifyAnalyzerAsync(source As String, ParamArray expected As DiagnosticResult()) As Task
        Dim test As New Test With
        {
        .TestCode = source,
        .ReferenceAssemblies = ReferenceAssemblies.Default.
            WithAssemblies(ReferenceAssemblies.Default.Assemblies.
            Add(GetType(SyntaxTriviaList).Assembly.Location))
        }

        test.ExpectedDiagnostics.AddRange(expected)
        Await test.RunAsync(CancellationToken.None)
    End Function

With the test

        'Diagnostic And CodeFix both triggered And checked for
        <TestMethod>
        Public Async Function TestMethod2() As Task

            Dim test = "Imports Microsoft.CodeAnalysis
Imports Factory = Microsoft.CodeAnalysis.VisualBasic.SyntaxFactory

Class TYPENAME

    Sub Main()
        Dim s as New SyntaxTriviaList
        Dim Space as string = "" ""
        {|#0:s|}.Add(Factory.Whitespace(Space))
    End Sub

End Class"
            Dim expected = VerifyVB.Diagnostic("ImproperUseOfImmutable").WithLocation(0).WithArguments("TypeName")
            Await VerifyVB.VerifyAnalyzerAsync(test, expected)
        End Function
    End Class
End Namespace

You get errors

Suppress the following diagnostics to disable this analyzer: ImproperUseOfImmutable"),
    // /0/Test0.vb(7) : error BC30002: Type 'SyntaxTriviaList' is not defined.
    DiagnosticResult.CompilerError("BC30002").WithSpan(7, 22, 7, 38).WithArguments("SyntaxTriviaList"),
    // /0/Test0.vb(9) : error BC31208: Type or namespace 'Microsoft.CodeAnalysis.VisualBasic.SyntaxFactory' is not defined.
    DiagnosticResult.CompilerError("BC31208").WithSpan(9, 15, 9, 22).WithArguments("Microsoft.CodeAnalysis.VisualBasic.SyntaxFactory"),
    // /0/Test0.vb(9) : error BC30456: 'Whitespace' is not a member of 'Microsoft.CodeAnalysis.VisualBasic.SyntaxFactory'.
    DiagnosticResult.CompilerError("BC30456").WithSpan(9, 15, 9, 33).WithArguments("Whitespace", "Microsoft.CodeAnalysis.VisualBasic.SyntaxFactory"),

[sharwell]

If all of your tests need to add this reference, you can change the default ReferenceAssemblies that apply to a test in the constructor for this class. The specific instances of these type will be generated by the template when the solution is created; there are a total of 6 Test classes (analyzer, code fix, and refactoring for both C# and Visual Basic).

roslyn-sdk/src/VisualStudio.Roslyn.SDK/Roslyn.SDK/ProjectTemplates/CSharp/Diagnostic/Test/Verifiers/VBAnalyzerVerifier`1+Test.cs

Lines 10 to 15 in 4baf9ef

	public class Test : VisualBasicAnalyzerTest<TAnalyzer, MSTestVerifier>
	{
	public Test()
	{
	}
	}

roslyn-sdk/src/VisualStudio.Roslyn.SDK/Roslyn.SDK/ProjectTemplates/CSharp/Diagnostic/Test/Verifiers/VBCodeFixVerifier`2+Test.cs

Lines 12 to 14 in 4baf9ef

	public class Test : VisualBasicCodeFixTest<TAnalyzer, TCodeFix, MSTestVerifier>
	{
	}

If you only need to run a single test with a different set of reference assemblies, the recommended approach is to switch from VerifyAnalyzerAsync (the "short form") to this longer form:

await new VerifyVB.Test
{
  ReferenceAssemblies = referenceAssemblies, // <-- set this to the appropriate set of assemblies
  TestState =
  {
    Sources = { test },
    ExpectedDiagnostics = { VerifyVB.Diagnostic().WithLocation(0).WithArguments("TypeName") },
  },
}.RunAsync();

[paul1956]

Thanks, it would be great if this was documented somewhere easier to find or even in the comments.

[bakester14]

This is really a problem. The comments on this thread are not enough to really help. Running code analysis tests on sample code with external dependencies doesn't seem like it should be so hard to do.

The guidance here is "set reference assemblies to the appropriate set of assemblies". What does that actually mean? Where do the referenced assemblies have to be located? What do you do in the case of a nuget package?

[voroninp]

@sharwell What's the difference between WithAssemblies and AddAssemblies?

[voroninp]

Here's my setup:

public sealed class ResultNotDiscardedAnalyzerTests
{
    private const string Code = @"
using Utilites;

class Foo
{
    void Bar() { Result.Ok(); }
}
";

    [Fact]
    public async Task Test()
    {
        await new CSharpAnalyzerTest<ResultNotDiscardedAnalyzer, XUnitVerifier>
        {
            ReferenceAssemblies = ReferenceAssemblies.Default.WithAssemblies(
                    ImmutableArray.Create(typeof(Result).Assembly.Location)),
            TestState =
                {
                    Sources = { Code }
                }
        }.RunAsync();
    }
}

Test fails with:

Microsoft.CodeAnalysis.Testing.Verifiers.EqualWithMessageException : Mismatch between number of diagnostics returned, expected "0" actual "2"

Diagnostics:
// /0/Test0.cs(2,7): error CS0246: The type or namespace name 'Utilities' could not be found (are you missing a using directive or an assembly reference?)

[voroninp]

This one wroks:

[Fact]
public async Task Test()
{
    await new CSharpAnalyzerTest<ResultNotDiscardedAnalyzer, XUnitVerifier>
    {
        ReferenceAssemblies = new ReferenceAssemblies(
            "net6.0", new PackageIdentity("Microsoft.NETCore.App.Ref", "6.0.0"), Path.Combine("ref", "net6.0")),
        TestState =
        {
            Sources = { Code },
            AdditionalReferences = { typeof(Result).Assembly.Location }
        }
    }.RunAsync();
}

Yet I have no idea, why.
This post gave some hints, but also without any explanation.

[sharwell]

ReferenceAssemblies.WithAssemblies adds assemblies that are part of the target framework. For example, if you want to add a reference to System.Configuration you would do it with WithAssemblies. For assemblies located elsewhere (e.g. typeof(Result).Assembly), AdditionalReferences is the correct collection.

[MihailsKuzmins]

@sharwell, thanks for pointing out that AdditionalReferences is the right place for project references.
However, my project uses .net6, so then adding the assembly I get an error about different versions for System.Runtime. Does it mean that I need to scan C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.2 and manually add references? Or is there a more simple workaround?

error CS1705: Assembly 'MyProject' with identity 'MyProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' uses 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' which has a higher version than referenced assembly 'System.Runtime' with identity 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

In my case I was lucky to be able to change the project to netstandard2.0, so the error is gone. But what if I cannot change the TargetFramework.

[sharwell]

@MihailsKuzmins Use ReferenceAssemblies to indicate the target framework, and use AdditionalReferences to provide references for that framework. If you get an error about System.Runtime, it means the references are not compatible with the target framework (the same error would appear during a build).

[sergiojrdotnet]

Is there a way to add all assembly dependencies to AdditionalReferences at once?
In my case, I need to reference Azure.Data.Tables, which depends of Azure.Core, and so on...

[sharwell]

@sergiojrdotnet The only way I would recommend is using the AddPackages feature of ReferenceAssemblies. This will resolve all of the necessary dependencies from NuGet according to the target framework you are using for your tests.

Here's an example of adding the System.ValueTuple package for a test against net46:

roslyn-sdk/tests/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Analyzer.Testing.UnitTests/CompilerErrorTests.cs

Lines 349 to 357 in f9932c8

	await new CSharpTest
	{
	TestState =
	{
	Sources = { testCode },
	},
	ReferenceAssemblies = ReferenceAssemblies.NetFramework.Net46.Default
	.AddPackages(ImmutableArray.Create(new PackageIdentity("System.ValueTuple", "4.5.0"))),
	}.RunAsync();

[sharwell]

Here's an example where we needed to test against some ASP.NET packages:
https://github.com/dotnet/roslyn-analyzers/blob/c6352bf2e1bd214fce090829de1042000d021497/src/Test.Utilities/AdditionalMetadataReferences.cs#L53-L57

[sergiojrdotnet]

Great! Thank you @sharwell

[paul1956]

@sharwell where do you get new PackageIdentity("System.ValueTuple", "4.5.0")?

I was looking for WinForms where do I get its PackageIdentity for .Net 7?

[sharwell]

@paul1956 You should be able to use the same packages that appear in a real project targeting WinForms on .NET 7.

[paul1956]

@sharwell that is all a black box is there a file I can look into for the full string?

All I specify in project file is below

    <TargetFramework>net7.0-windows</TargetFramework>
    <UseWindowsForms>true</UseWindowsForms>

[sharwell]

ReferenceAssemblies.Net.Net70Windows is a predefined property. Look in a binlog from that build for package and/or assembly references added by the UseWindowsForms property to see how it modifies things.

[VissersR]

I'm running in to issues with the reference assemblies. The problem we have is that it tries to resolve nuget packages from the feed, but because we have a private feed in azure devops it failes with a 401. It does not seem to use existing tokens that visual studio or dotnet has, like a normal unit test would have.

Why can't we just reference packages in the unit test normally and have the analyzer test use these?
It seems like the only solution is to go through the source and figure out how to provide a token manually, which is a no go for obvious reasons. It is a unit test and should not have external dependencies.

The whole analyzer debugging/testing experience is extremely poor and not well documented.

[sharwell]

Why can't we just reference packages in the unit test normally and have the analyzer test use these?

These would not accurately represent builds. Some NuGet packages have different assemblies in the ref and lib folders. The analyzers need to test against the ref folder contents in these cases, but the assemblies provided by adding the package to the unit test project would come from the lib folder. This can and has changed analyzer behavior in the past, so we strive for maximum test accuracy to help users avoid these problems. The general view is it's better for a unit test to fail than to pass incorrectly.

In addition, unit tests in a single project can test against different target frameworks, e.g. netstandard2.0, net472, and net6.0. The sets of assemblies provided for each of these scenarios can be quite different, and would not be correct if they were just added to the unit test project.

I'm not aware of a solution for using authenticated feeds in unit tests, unless it's possible to do so through this feature. It might be necessary to manually construct MetadataReference instances for AdditionalReferences instead of using ReferenceAssemblies.AddPackages, but the burden of accurately reflecting the build inputs would be on the unit test project at that point.

The whole analyzer debugging/testing experience is extremely poor and not well documented.

Yes, this project is known to have a somewhat steep learning curve. It wasn't intentional, but available resources are constrained in this space.