Publish TestFramework via NuGet

[floyd-may]

In working on writing my own analyzer, it would be awesome if all the handy stuff in the test framework were available to me so that I didn't need to reinvent the wheel to write a relatively simple unit test. My strategy right now is to copy and paste DiagnosticVerifier.cs into my test project.

Can you please consider publishing the test framework via NuGet?

[tiesmaster]

@floyd-may You may consider using the Roslyn package Microsoft.CodeAnalysis.Testing designed for this. In xunit.analyzers you see an example PR of author @sharwell giving a rationale behind this libary.

If you have any questions about it, or any other Roslyn related stuff, then you can find a very active Gitter chat to get you started ;)

---

@josefpihrt Not trying to steal your thunder, however, I can imagine you're not into the efforts into publishing your internal test framework as a public package, but if you are, then don't let my comment stop you ;)

[jzabroski]

@tiesmaster Small feedback: You should specify "Roslyn package" as Microsoft.CodeAnalysis.Testing. When I read the comments in email I didn't understand how linking to Roslyn solved the problem.

My own thought is I wonder if there is a generic way to compose Expressions and compile them, serialize them as a stream and then have the analyzer walk back over that stream. You could then use a tool like AutoFixture with erecruit.Expr to generate expression graphs as SUTs.

[tiesmaster]

@jzabroski Ah, good point, thnx! Didn't realize I was a bit too sparse ;) I've updated the comment.

As your other comment, I don't really follow that. Maybe, I'm missing a bit of context, but what I know of how most of the analyzers, code fixes, and refactorings are being tested, is that they will use "concrete" source code as input, use the Roslyn APIs to convert it to a syntax tree, possible with analyzers running side-by-side, and then have the codefix, or refactoring transform it to a new tree, and "serialize" that back to source again, and assert that to the source that you'd expect.

I know AutoFixture, I've used that in the past (but moved away from that), thought, I don't really see how you would use that for testing analyzers, and such?

[jzabroski]

Just an idea. erecruit.Expr is a fluent way to write composable code. So is Dynamitey. The thought was to construct tiny fragments of code and curry those fragments and use Expression Compile and Serialize to generate examples. Why would you work so hard to write tests if a computer can find good tests for you.

[floyd-may]

@tiesmaster Thanks! That's exactly what I needed. I haven't had a great time Googling my way towards understanding how to build and test analyzers, so a little human guidance was exactly what I needed. Appreciate the assist!

[jzabroski]

@floyd-may @tiesmaster How are you pulling down Microsoft.CodeAnalysis.CSharp.Analyzer.Testing.XUnit? I get a 404 not found on nuget.org. Do you have to use the myget repo mentioned in the readme?

[sharwell]

@jzabroski Yes, the package is currently published to MyGet.
https://github.com/dotnet/roslyn-sdk/tree/master/src/Microsoft.CodeAnalysis.Testing#myget-feed-for-prerelease-packages

[jzabroski]

Thanks, 🎅!

[josefpihrt]

Roslynator Testing Framework has been finally released!

https://www.nuget.org/packages/Roslynator.Testing.CSharp.Xunit/1.0.0-rc

[sharwell]

@josefpihrt It seems unfortunate to have another parallel effort duplicating work on a feature we've spent a large amount of time polishing. Is there a reason the project isn't using the official testing library for these? If it's just a case of Roslynator not having the chance to convert the tests yet, is there a reason other projects shouldn't be starting off using the official testing library?

[jzabroski]

@sharwell I have personally found the default framework to be bare minimum and not suitable for writing cross-filter analysis tasks due to needing to detect the presence of a package in order to do per-project custom analyzers. I believe you wrote some similar "analyzer helper library" for Xunit to help that maintainer, so you should be familiar with why we want our own nuget package.

[sharwell]

@jzabroski I didn't understand these three parts of your reply:

... writing cross-filter analysis tasks ...
... needing to detect the presence of a package ...
... do per-project custom analyzers ...

Can you provide an example of how these relate to the test library, and how the standard test library restricts testing for it?

[josefpihrt]

Hi Sam,

I have mixed feelings from your post (to put it mildly).

It seems that you are not considering me as a promoter of Roslyn platform but as a some kind of a troublemaker. That's unfortunate.

is there a reason other projects shouldn't be starting off using the official testing library?

It is common that there are multiple libraries for a particular purpose and developers choose whichever suits them best. If there would be "official" testing library by the time I needed it I would have definitely used it.

Several years ago there was no testing framework so I have to create my own to be able to test analyzers, refactorings and code fixes. Over the time the testing tool evolved into regular library and I decided that it deserves to be published.

I do not plan to migrate to "official" testing framework. I see no benefit in doing so. It is better to have full control over the testing library (if you already have one).

Josef

[sharwell]

It seems that you are not considering me as a promoter of Roslyn platform but as a some kind of a troublemaker. That's unfortunate.

It's a bit different from that. In the early days of working on the testing library, I was specifically thinking of the kinds of features Roslynator might need as part of the design. It's too bad it didn't work out as a suitable replacement, but sometimes that just the way things go.

More than anything, my replies here are trying to find out if there's anything we could have done differently that would have produced a better outcome (i.e. a library that is more usable in more places).