[Perf] Linux/x64: 2 Regressions on 6/27/2024 6:42:30 AM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	58e1a7e6e499da2cd502bebb326497795101783f
Compare	51176f0bf2446f15dbd067b0001bce84b84a42f6
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Array

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ArrayAssign3D - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	431.20 μs	481.58 μs	1.12	0.11	False
ArrayAssign2D - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	385.84 μs	443.34 μs	1.15	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'

System.Tests.Perf_Array.ArrayAssign3D

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Array.ArrayAssign2D

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[DrewScoggins]

Seems related to #102739
Also related:
Linux x64 - dotnet/perf-autofiling-issues#37353

[jkotas]

@huoyaoyuan Could you please take a look?

[huoyaoyuan]

It's about SetValue, which I didn't measure performance for.

There isn't something obviously problematic at first glance. I'm also interested about the context of the improvement at May.

[jkotas]

I'm also interested about the context of the improvement at May.

It is possible that the micro-benchmark is bi-modal. It operates on a large array and it does a lot of boxing. Micro-benchmarks like this tend to be bi-modal.

Does the regression reproduce on your machine?

[huoyaoyuan]

I can't reproduce the regression on array of smaller size:

        private int[,] array2D = new int[100, 100];

        [Benchmark]
        public void ArrayAccess2D()
        {
            for (int i = 0; i < 100; i++)
                for (int j = 0; j < 100; j++)
                    array2D.SetValue(i + j, i, j);
        }

Method	Job	Toolchain	Mean	Error	StdDev	Ratio	RatioSD
ArrayAccess2D	Job-VQKCWF	\AfterMerge\corerun.exe	62.86 us	1.129 us	1.160 us	0.92	0.03
ArrayAccess2D	Job-JKQPLR	\BeforeMerge\corerun.exe	68.32 us	1.322 us	1.104 us	1.00	0.00

[jkotas]

@huoyaoyuan Thank you for checking!

This is bi-modal/unstable benchmark. It operates on large array and it does a lot of allocations (boxing) that are unrelated to the operations being tested. Benchmarks like that tend to be unstable. Closing as by design.