CI test reporters

[natemcmaster]

dotnet-test-xunit and supported detecting CI build environments and reporting tests results in a format consumable by that CI environment. For example, building on AppVeyor and TeamCity would automatically light up test reporters.

How can we do the same thing with VS Test? Is it already possible?

[codito]

dotnet test already supports emitting trx files with test results. Usually the CIs parse trx and publish results.

AppVeyor supports uploading trx files. Have to check if real time reporting works (it will require AppVeyor logger to be .net core compatible).

The xunit reporter in jenkins may support parsing trx. @piotrpMSFT @livarcocc can add more details, they were trying it for dotnet-cli repo.

Teamcity supports parsing trx files via xml report parser.

VSTS supports trx with a publish results task.

[natemcmaster]

Good to know. It's close, but not quite what I had in mind as it seems this requires users to take additional configuration steps for CI's to get the trx file. dotnet-test-xunit, on the other hand, automatically detected AppVeyor or TeamCity environments and produced test reporting.

Details on how that's implemented:
TeamCity: using 'Service Messages' (special lines written to stdout) to report tests. (see https://confluence.jetbrains.com/display/TCD10/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingTests).
AppVeyor: xunit makes a small POST request with test results. https://www.appveyor.com/docs/build-worker-api/#add-tests

Here is Xunit implementation: https://github.com/xunit/xunit/tree/master/src/xunit.runner.reporters.

Is something similar possible with VSTest today? e.g. can I write a custom reporter and plug it into the VSTest host, or would this only be possible if VS Test itself adds this feature?

[codito]

There isn't auto detection on vstest side (yet). It is possible to write a custom logger which can detect the CI environment and integrate. A logger needs to implement ITestLogger interface, test platform calls a logger on test result events.

See console logger and trx logger for example.

[codito]

Triage:
We will unblock the adapter authors with command line parameters first and come back.

[clairernovotny]

This capability is currently done by the xUnit runner. We implement our own reporter extensibility mechanism that detects CI environment and interacts appropriately. An xUnit user can implement an interface and specify a command line option to pass in a custom reporter.

[tmat]

Jenkins supports xUnit format and trx. The problem with trx is size - it's >10x larger than xunit format :(. I'd like to be able to use xunit format in dotnet test.

[Faizan2304]

Few CI reporters are available now:

Scenario	Nuget Package	Source Repository
Local, CI, CD	Inbuilt	Trx Logger
Local, CI, CD	Inbuilt	Console Logger
Local, CI, CD	XunitXml.TestLogger	Xunit Logger
AppVeyor	AppVeyor.TestLogger	AppVeyor Logger
TeamCity	TeamCity.VSTest.TestAdapter	Teamcity Logger

Documentation for loggers are available here.

[omerlh]

Hey, are there plans to support Circle CI test runner? Look like they can parse test results in a specific format (see xunit/xunit#1882)