Bad performance for roslyn project

[jakubch1]

I was running performance comparison for roslyn tests projects. Results below. We were investigating why coverlet is much slower. You can clone https://github.com/dotnet/roslyn and retry this testing.

Few things that can be changed to improve performance:

1. initial instrumentation can be done in parallel for assemblies
2. review logic for probes and make minimal operations (definitely something is wrong here as Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests took 32 minutes and normal run 8 minutes. In this case instrumentation took only 1 minute)
3. report generation could first generate stuff in parallel for all assemblies (in memory) then it can push data to report.

Projects:	Microsoft.CodeAnalysis.UnitTests	Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests	Microsoft.CodeAnalysis.CSharp.Emit.UnitTests	Microsoft.CodeAnalysis.CSharp.IOperation.UnitTests	Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests	Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests	Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests	Microsoft.CodeAnalysis.CSharp.WinRT.UnitTests	VBCSCompiler.UnitTests	Microsoft.CodeAnalysis.Scripting.UnitTests	Microsoft.CodeAnalysis.CSharp.Scripting.UnitTests
NoCC:	00:21.0	00:20.9	05:03.9	00:22.7	08:17.9	01:30.2	00:21.2	00:35.6	00:21.5	00:07.5	00:35.1
Coverlet:	02:12.8	01:54.9	15:30.2	01:57.7	32:10.0	04:04.3	01:59.7	03:31.3	02:05.3	01:30.3	02:26.0
Dynamic:	00:49.5	00:43.2	07:31.9	00:50.1	15:21.2	02:30.5	00:52.7	01:15.3	00:44.5	00:30.1	00:56.7

[jakubch1]

Script that you can use:

`$src = "C:\git\roslyn\src"

$projects = New-Object System.Collections.ArrayList
$nocc = New-Object System.Collections.ArrayList
$coverlet = New-Object System.Collections.ArrayList
$dynamic = New-Object System.Collections.ArrayList

function PerformTest($projectpath)
{
Write-Output "Tests for: "
$project = [System.IO.Path]::GetFileNameWithoutExtension($projectpath)
$dir = (Split-Path $projectpath -Parent)
$projectpath

install_coverlet($dir)

$bin = "C:\\git\\roslyn\\artifacts\\bin\\$project\\Debug\\netcoreapp3.1"

if(Test-Path $bin)
{
	$projects.Add($project)
	run_no_cc $dir $bin $project
	run_coverlet_cc $dir $bin $project
	run_dynamic_cc $bin $project
	
	Write-Output "Projects: $projects"
	Write-Output "No cc: $nocc"
	Write-Output "Coverlet: $coverlet"
	Write-Output "Dynamic: $dynamic"
}

}

function install_coverlet($dir)
{
Write-Output "Installing coverlet: " + $dir
cd $dir
dotnet add package coverlet.collector
dotnet build
dotnet publish --framework netcoreapp3.1
Write-Output "Coverlet installation finished: " + $dir
}

function run_no_cc($dir, $bin, $project)
{
cd $dir

$sw = [Diagnostics.Stopwatch]::StartNew()
dotnet vstest "$bin\\$project.dll"
$sw.Stop()
$nocc.Add($sw.Elapsed) | Out-Null

}

function run_coverlet_cc($dir, $bin, $project)
{
cd $dir

$sw = [Diagnostics.Stopwatch]::StartNew()
dotnet vstest "$bin\\$project.dll" --collect:"XPlat Code Coverage"
$sw.Stop()
$coverlet.Add($sw.Elapsed) | Out-Null

}

function run_dynamic_cc($dir, $project)
{
cd $dir

if(Test-Path 'publish')
{
	Remove-Item 'publish' -Recurse
}

$sw = [Diagnostics.Stopwatch]::StartNew()
vstest.console.exe "$project.dll" /Settings:"C:\tools\dotnet\coverage.runsettings"
$sw.Stop()
$dynamic.Add($sw.Elapsed) | Out-Null

}

#run_dynamic_cc($bin)

cd $src
Get-ChildItem -Recurse *UnitTests.csproj | ForEach-Object { PerformTest($_.FullName) }
`

[MarcoRossignoli]

I know, we never worked on performance optimization, we kept focus on features/bug until now here some perf issue https://github.com/tonerdo/coverlet/labels/tenet-performance

As I've said here #777 (comment) I have some idea on what to do, we have some points to improve:

1. for the instrumentation phase, parallelize and profile where we spend time
2. for test run, fold where possible sequencepoint/branch Combine sequence point hits with adjacent branch hits #307
3. Suspect of false sharing due to array storage, try to understand if make sense use a different structure

To do this we need to improve perf project with BDN and perf test scenario.
I'm working on coverlet in my spare time so for the moment my personal priority it has always been

1. Blocking bugs
2. Coverage bugs https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Atenet-coverage
3. Features https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Afeature-request

Until now performance never was the "most important" problem for our users as you can see using label filtering.

At the moment I'm on #363 (your collegues) and I have some regression to fix, we relase every three month and first days there are some regression to fix(first two) I worked heavily on testing part: units, coverage, integration to lower down regression, but we're at the beginning, I mean we have the tool but we need the real cases.

[github-actions]

This issue is stale because it has been open for 3 months with no activity.