Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add diagnostic verification extensibility. #584

Merged
merged 4 commits into from
Aug 13, 2020
Merged

Add diagnostic verification extensibility. #584

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
244bd36
Add diagnostic verification extensibility.
nschuessler Aug 10, 2020
67f31e0
Update API based on review feedback.
nschuessler Aug 10, 2020
0a98b50
Remove analyzers argument.
nschuessler Aug 11, 2020
1fe7f62
Update public api signatures.
nschuessler Aug 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Analyzer.Testing/AnalyzerTest`1.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,18 @@ public string TestCode
/// </summary>
public List<string> DisabledDiagnostics { get; } = new List<string>();

/// <summary>
/// Gets or sets the reference assemblies to use.
/// </summary>
public ReferenceAssemblies ReferenceAssemblies { get; set; } = ReferenceAssemblies.Default;

/// <summary>
/// Gets or sets an additional verifier for a diagnostic.
/// The action compares actual <see cref="Diagnostic"/> and the expected
/// <see cref="DiagnosticResult"/> based on custom test requirements not yet supported by the test framework.
/// </summary>
public Action<ImmutableArray<DiagnosticAnalyzer>, Diagnostic, DiagnosticResult, IVerifier>? DiagnosticVerifier { get; set; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove the analyzer portion?

Suggested change
public Action<ImmutableArray<DiagnosticAnalyzer>, Diagnostic, DiagnosticResult, IVerifier>? DiagnosticVerifier { get; set; }
public Action<Diagnostic, DiagnosticResult, IVerifier>? DiagnosticVerifier { get; set; }

💭 I'm tempted to go as far as removing the DiagnosticResult portion as well.

Copy link
Contributor Author

@nschuessler nschuessler Aug 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That means implementors can't use the following method for consistency:

     protected string FormatVerifierMessage(ImmutableArray<DiagnosticAnalyzer> analyzers, Diagnostic actual, DiagnosticResult expected, string message)

Is that ok?

Copy link
Contributor Author

@nschuessler nschuessler Aug 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: DiagnosticResult for expected. I'd like to keep it so that it can be a key to a dictionary containing expected data.

I.e. I would use DiagnosticResult.Location.Span as the key and Dictionary<string, string> for expected properties I'm validating in the Test class.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That means implementors can't use the following method for consistency

The helper wouldn't produce an actionable error message for this case, so I'm not sure how much it helps.

I'd like to keep it so that it can be a key to a dictionary containing expected data.

📝 This is already not supported. DiagnosticResult is a value type which is not equatable.

Let me think about this a bit. If I can't find an "obviously better" option then the one here seems reasonable.

Copy link
Contributor Author

@nschuessler nschuessler Aug 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do use TextSpan as keys / comparisons in some of my analyzers to determine if I'm matching up the same piece of code.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sharwell are we okay to go ahead with Niko's contribution?

Copy link
Member

@sharwell sharwell Aug 13, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are


/// <summary>
/// Gets a collection of transformation functions to apply to <see cref="Workspace.Options"/> during diagnostic
/// or code fix test setup.
Expand Down Expand Up @@ -371,6 +381,8 @@ private void VerifyDiagnosticResults(IEnumerable<Diagnostic> actualResults, Immu
StringComparer.Ordinal,
message);
}

DiagnosticVerifier?.Invoke(analyzers, actual, expected, verifier);
}
}

Expand Down
2 changes: 2 additions & 0 deletions ...soft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Analyzer.Testing/PublicAPI.Unshipped.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.AnalyzerTest() -> void
Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.CompilerDiagnostics.get -> Microsoft.CodeAnalysis.Testing.CompilerDiagnostics
Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.CompilerDiagnostics.set -> void
Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.CreateProjectAsync((string filename, Microsoft.CodeAnalysis.Text.SourceText content)[] sources, (string filename, Microsoft.CodeAnalysis.Text.SourceText content)[] additionalFiles, Microsoft.CodeAnalysis.Testing.ProjectState[] additionalProjects, Microsoft.CodeAnalysis.MetadataReference[] additionalMetadataReferences, string language, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task<Microsoft.CodeAnalysis.Project>
Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.DiagnosticVerifier.get -> System.Action<System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer>, Microsoft.CodeAnalysis.Diagnostic, Microsoft.CodeAnalysis.Testing.DiagnosticResult, Microsoft.CodeAnalysis.Testing.IVerifier>
Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.DiagnosticVerifier.set -> void
Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.DisabledDiagnostics.get -> System.Collections.Generic.List<string>
Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.ExpectedDiagnostics.get -> System.Collections.Generic.List<Microsoft.CodeAnalysis.Testing.DiagnosticResult>
Microsoft.CodeAnalysis.Testing.AnalyzerTest<TVerifier>.FormatVerifierMessage(System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer> analyzers, Microsoft.CodeAnalysis.Diagnostic actual, Microsoft.CodeAnalysis.Testing.DiagnosticResult expected, string message) -> string
Expand Down