It is hard to use test framework for source generators

[kzrnm]

from #602 (comment)

It is hard to use test framework for source generators. Some SourceGeneratorTest<TVerifier> API are not suitable for source generators.

roslyn-sdk/src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.SourceGenerators.Testing/SourceGeneratorTest`1.cs

Lines 61 to 79 in 50e2a55

	public override async Task RunAsync(CancellationToken cancellationToken = default)
	{
	Verify.NotEmpty($"{nameof(TestState)}.{nameof(SolutionState.Sources)}", TestState.Sources);
	var analyzers = GetDiagnosticAnalyzers().ToArray();
	var defaultDiagnostic = GetDefaultDiagnostic(analyzers);
	var supportedDiagnostics = analyzers.SelectMany(analyzer => analyzer.SupportedDiagnostics).ToImmutableArray();
	var fixableDiagnostics = ImmutableArray<string>.Empty;
	var rawTestState = TestState.WithInheritedValuesApplied(null, fixableDiagnostics);
	var rawFixedState = FixedState.WithInheritedValuesApplied(rawTestState, fixableDiagnostics);
	var testState = rawTestState.WithProcessedMarkup(MarkupOptions, defaultDiagnostic, supportedDiagnostics, fixableDiagnostics, DefaultFilePath);
	var fixedState = rawFixedState.WithProcessedMarkup(MarkupOptions, defaultDiagnostic, supportedDiagnostics, fixableDiagnostics, DefaultFilePath);
	await VerifyDiagnosticsAsync(new EvaluatedProjectState(testState, ReferenceAssemblies), testState.AdditionalProjects.Values.Select(additionalProject => new EvaluatedProjectState(additionalProject, ReferenceAssemblies)).ToImmutableArray(), testState.ExpectedDiagnostics.ToArray(), Verify.PushContext("Diagnostics of test state"), cancellationToken).ConfigureAwait(false);
	var diagnostics = await VerifySourceGeneratorAsync(testState, fixedState, Verify, cancellationToken).ConfigureAwait(false);
	await VerifyDiagnosticsAsync(new EvaluatedProjectState(fixedState, ReferenceAssemblies).WithAdditionalDiagnostics(diagnostics), fixedState.AdditionalProjects.Values.Select(additionalProject => new EvaluatedProjectState(additionalProject, ReferenceAssemblies)).ToImmutableArray(), fixedState.ExpectedDiagnostics.ToArray(), Verify.PushContext("Diagnostics of test state with generated sources"), cancellationToken).ConfigureAwait(false);
	}

It is hard to use FixedCode

It is hard to use FixedCode because FixedState.Sources needs original codes and generated codes.

Verify.NotEmpty should be removed

Verify.NotEmpty should be removed. Source generators can work in a project without any C# code like CsvGenerator. Even if TestState.Sources is empty, test can be valid.

The generated source code is not always deterministic

The generated source code is not always deterministic when the source generator is using System.Guid, System.Random,System.DateTime etc.

The output can also change due to the indefinite order of Dictionary<TKey, TValue> and ImmutableDictionary<TKey, TValue>.

private static StringBuilder CreateMetadataSource(StringBuilder sb, ImmutableDictionary<string, string> metadatas)
{
    sb.AppendLine("using System.Reflection;");
    foreach (var p in metadatas)
    {
        sb.Append("[assembly: AssemblyMetadataAttribute(");
        sb.Append(ToLiteral(p.Key));
        sb.Append(",");
        sb.Append(ToLiteral(p.Value));
        sb.AppendLine(")]");
    }
    return sb;
}

Name of source file

Name of genereted file is especially problematic.

In Windows, name of generated file is {AssemblyName}\{ClassName}\{hintName}, but in Mac/Linux, the name is {AssemblyName}/{ClassName}/{hintName}.

[sharwell]

It is hard to use FixedCode

In the original comment, you mentioned adding a GeneratedSources property. I really like this idea, because it means the entire Source Generator testing approach can be merged with the analyzer testing approach.

Before:

await new VerifyCS.Test
{
    TestState =
    {
        Sources =
        {
            @"// Comment",
        },
    },
    FixedState =
    {
        Sources =
        {
            @"{|#0:|}// Comment",
            ("Microsoft.CodeAnalysis.SourceGenerators.Testing.UnitTests\\Microsoft.CodeAnalysis.Testing.TestGenerators.AddEmptyFileWithDiagnostic\\EmptyGeneratedFile.cs", SourceText.From(string.Empty, Encoding.UTF8)),
        },
        ExpectedDiagnostics =
        {
            // /0/Test0.cs(1,1): warning SG0001: Message
            new DiagnosticResult(AddEmptyFileWithDiagnostic.Descriptor).WithLocation(0),
        },
    },
}.RunAsync();

After:

await new VerifyCS.Test
{
    TestState =
    {
        Sources =
        {
            @"{|#0:|}// Comment",
        },
        GeneratedSources =
        {
            ("Microsoft.CodeAnalysis.SourceGenerators.Testing.UnitTests\\Microsoft.CodeAnalysis.Testing.TestGenerators.AddEmptyFileWithDiagnostic\\EmptyGeneratedFile.cs", SourceText.From(string.Empty, Encoding.UTF8)),
        }
        ExpectedDiagnostics =
        {
            // /0/Test0.cs(1,1): warning SG0001: Message
            new DiagnosticResult(AddEmptyFileWithDiagnostic.Descriptor).WithLocation(0),
        },
    },
}.RunAsync();

[sharwell]

Verify.NotEmpty should be removed

This seems fine. If the compiler complains about the lack of source files, the test could just declare that compilation error as an expected diagnostic.

[sharwell]

The generated source code is not always deterministic

I'm not inclined to do anything with the specific goal of making this easier. Determinism is a highly desirable property of source generators. One interim solution would be overriding SourceGeneratorTest<TVerifier>.GetSourceGenerators() to construct an instance of the source generator that knows to use specific values in place of its normal non-deterministic values. If this doesn't work for you, I would suggest filing a separate issue so we can have a longer discussion about the many possible approaches.

[sharwell]

Name of source file

Yeah, this is a mess. What about this?

 GeneratedSources =
 {
-    ("Microsoft.CodeAnalysis.SourceGenerators.Testing.UnitTests\\Microsoft.CodeAnalysis.Testing.TestGenerators.AddEmptyFileWithDiagnostic\\EmptyGeneratedFile.cs", SourceText.From(string.Empty, Encoding.UTF8)),
+    (typeof(AddEmptyFileWithDiagnostic), "EmptyGeneratedFile.cs", string.Empty),
 }

The new API uses the provided Type for two purposes:

1. Automatically determining the base path
2. Automatically assuming the default encoding is UTF8, since source generators always need to specify an encoding

[aetos382]

If the Source Generator uses StringBuilder.AppendLine, the output will not be deterministic because newline characters are dependent on the Operating System.
Does this mean that using StringBuilder.Append("foo\n") instead of StringBuilder.AppendLine("foo") is the recommended policy for creating a Source Generator?

[kzrnm]

GeneratedSources

In the original comment, you mentioned adding a GeneratedSources property. I really like this idea, because it means the entire Source Generator testing approach can be merged with the analyzer testing approach.

Thank you. The reason I deleted this code in the issue was that it seemed difficult to deal with non-deterministic source code.

If deterministic source code does not be supportted, I want GeneratedSources.

But I think GeneratedSources should be a member of SourceGeneratorTest<TVerifier> instead of TestState.

In addition, I think it's better to remove the SourceGeneratorTest <TVerifier>.FixedCode.

I wrote the code. sharwell#1

deterministic

I'm not inclined to do anything with the specific goal of making this easier. Determinism is a highly desirable property of source generators. One interim solution would be overriding SourceGeneratorTest.GetSourceGenerators() to construct an instance of the source generator that knows to use specific values in place of its normal non-deterministic values. If this doesn't work for you, I would suggest filing a separate issue so we can have a longer discussion about the many possible approaches.

I understand. I’m willing to open a new issue.

[jmarolf]

@aetos382 I would just construct the expected code state using the platform specific apis like I do here for path.

[sharwell]

I'll create an API and updated tests for GeneratedSources so it's clear which direction I envision things going. Then we can work on implementing it (which will be a bit challenging but we've had worse).