[Perf] Linux/x64: 20 Regressions on 4/28/2024 2:54:10 PM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorFromCollection<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SortedList - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	253.73 μs	273.59 μs	1.08	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection<String>*'

System.Collections.CtorFromCollection<String>.SortedList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_PriorityQueue<String, String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
K_Max_Elements - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	42.81 μs	48.01 μs	1.12	0.07	False
Dequeue_And_Enqueue - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	1.79 ms	1.93 ms	1.08	0.05	False
HeapSort - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	48.27 μs	51.95 μs	1.08	0.05	False
Dequeue_And_Enqueue - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	4.83 μs	5.15 μs	1.07	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue<String, String>*'

System.Collections.Tests.Perf_PriorityQueue<String, String>.K_Max_Elements(Size: 1000)

ETL Files

Histogram

JIT Disasms

System.Collections.Tests.Perf_PriorityQueue<String, String>.Dequeue_And_Enqueue(Size: 1000)

ETL Files

Histogram

JIT Disasms

System.Collections.Tests.Perf_PriorityQueue<String, String>.HeapSort(Size: 100)

ETL Files

Histogram

JIT Disasms

System.Collections.Tests.Perf_PriorityQueue<String, String>.Dequeue_And_Enqueue(Size: 10)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Sort<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
List - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	212.13 μs	249.25 μs	1.17	0.04	False
Array_ComparerClass - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	213.90 μs	253.57 μs	1.19	0.07	False
Array_Comparison - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	217.35 μs	232.99 μs	1.07	0.08	False
Array_ComparerStruct - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	223.96 μs	238.66 μs	1.07	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort<String>*'

System.Collections.Sort<String>.List(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Sort<String>.Array_ComparerClass(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Sort<String>.Array_Comparison(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Sort<String>.Array_ComparerStruct(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyTrue<String, String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SortedList - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	175.06 μs	187.33 μs	1.07	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue<String, String>*'

System.Collections.ContainsKeyTrue<String, String>.SortedList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SortedSet - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	217.85 μs	233.93 μs	1.07	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse<String>*'

System.Collections.ContainsFalse<String>.SortedSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndRemove<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
List - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	13.07 μs	14.16 μs	1.08	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove<Int32>*'

System.Collections.CreateAddAndRemove<Int32>.List(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndClear<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SortedList - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	39.49 μs	42.43 μs	1.07	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear<Int32>*'

System.Collections.CreateAddAndClear<Int32>.SortedList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueFalse<String, String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ImmutableSortedDictionary - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	223.05 μs	237.67 μs	1.07	0.02	False
SortedDictionary - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	239.78 μs	259.62 μs	1.08	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse<String, String>*'

System.Collections.TryGetValueFalse<String, String>.ImmutableSortedDictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.TryGetValueFalse<String, String>.SortedDictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SortedList - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	174.15 μs	185.70 μs	1.07	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet<String>*'

System.Collections.IndexerSet<String>.SortedList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.AddGivenSize<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SortedList - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	38.29 μs	40.93 μs	1.07	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize<Int32>*'

System.Collections.AddGivenSize<Int32>.SortedList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Char

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Char_ToUpper - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	15.23 ns	17.20 ns	1.13	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Char*'

System.Tests.Perf_Char.Char_ToUpper(c: 'İ', cultureName: en-US)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Runtime.InteropServices.Tests.Perf_Marshal

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
AllocFree - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	15.20 ns	17.00 ns	1.12	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.InteropServices.Tests.Perf_Marshal*'

System.Runtime.InteropServices.Tests.Perf_Marshal.AllocFree

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	95306c760c08d922a1d4a7a0979c16fb40c4ac3f
Compare	c3e5ce97a29317d3dab98d09c310daf11a2260c7
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SortedList - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	236.81 μs	261.17 μs	1.10	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear<String>*'

System.Collections.CreateAddAndClear<String>.SortedList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

[DrewScoggins]

Related to #101630
@richlander

[DrewScoggins]

Improvements:
Linux x64 - dotnet/perf-autofiling-issues#33506

[richlander]

We moved the build to clang/LLVM 18 in #101630. What's the usual approach for addressing performance regressions due to a toolset upgrade?

[eiriktsarpalis]

At this point it seems the only way forward is accepting the regression?

[stephentoub]

Why did a clang/LLVM update impact these particular benchmarks?

[jkotas]

The regressions from unmanaged compiler updates are typically caused by outdated native PGO data.

The native PGO data

runtime/eng/Versions.props

Line 161 in 2e1b1b4

<optimizationPGOCoreCLRVersion>1.0.0-prerelease.24223.3</optimizationPGOCoreCLRVersion>

were collected last time on April 23 (#101677) that coincides with the timing of these regressions. The PGO data update process was likely broken by the clang update.

I see there was #105236 opened yesterday that includes a fresh PGO update. We should check whether these perf regressions go away after it gets merged. If the perf regressions do not go away after the PGO update gets merged, we may have a problem in how the PGO data are getting collected or applied (note that it is important to do both with the same compiler version).

[eiriktsarpalis]

It would seem numbers have recovered since the regression was recorded, for example:

Closing as resolved.