Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.Tests.Perf_Guid #85994

Closed
performanceautofiler bot opened this issue May 9, 2023 · 3 comments
Closed

Regressions in System.Tests.Perf_Guid #85994

performanceautofiler bot opened this issue May 9, 2023 · 3 comments
Assignees
@EgorBo
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 442141d696ee7d4ae54420974559e99738d753e6
Compare c8f43d52f699c1da289a5cb36d371aec15901ab8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MeasureReverseUsingNtoH - Duration of single invocation 331.25 ns 657.14 ns 1.98 0.00 True
MeasureReverseEndianness - Duration of single invocation 331.08 ns 656.62 ns 1.98 0.00 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.MeasureReverseUsingNtoH

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 657.1443752993548 > 348.04767438472953.
IsChangePoint: Marked as a change because one of 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -1725.4828094217587 (T) = (0 -657.7838151245886) / Math.Sqrt((0.2889242088902872 / (26)) + (0.4435805265031423 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -0.9842134137794782 = (331.5086021274592 - 657.7838151245886) / 331.5086021274592 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.MeasureReverseEndianness

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 656.6179442968024 > 347.85342306747543.
IsChangePoint: Marked as a change because one of 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -1895.5849901028187 (T) = (0 -657.5466614763055) / Math.Sqrt((0.09788024557592748 / (26)) + (0.4649324313081097 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -0.9838498204419397 = (331.44981777392036 - 657.5466614763055) / 331.44981777392036 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 442141d696ee7d4ae54420974559e99738d753e6
Compare c8f43d52f699c1da289a5cb36d371aec15901ab8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CopyTo<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Memory - Duration of single invocation 114.85 ns 185.21 ns 1.61 0.25 False
ReadOnlyMemory - Duration of single invocation 112.70 ns 184.28 ns 1.64 0.31 False
Span - Duration of single invocation 111.17 ns 181.52 ns 1.63 0.28 False
Array - Duration of single invocation 112.20 ns 182.70 ns 1.63 0.28 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CopyTo<Int32>.Memory(Size: 2048)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 185.2139209713077 > 118.96458191097815.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -11.61978521318909 (T) = (0 -184.8462357983078) / Math.Sqrt((696.3504002888097 / (26)) + (1.1656438290390554 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -0.48305469158313313 = (124.63885306953036 - 184.8462357983078) / 124.63885306953036 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CopyTo<Int32>.ReadOnlyMemory(Size: 2048)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 184.27621651852104 > 118.47282810115816.
IsChangePoint: Marked as a change because one of 4/17/2023 11:29:15 PM, 5/2/2023 4:25:00 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -6.0470748517318516 (T) = (0 -175.30536684559712) / Math.Sqrt((828.0418257010215 / (21)) + (572.8159336579739 / (23))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (23) - 2, .025) and -0.3825115960587915 = (126.80209507489892 - 175.30536684559712) / 126.80209507489892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CopyTo<Int32>.Span(Size: 2048)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 181.5209793778094 > 116.61576769905994.
IsChangePoint: Marked as a change because one of 3/16/2023 2:58:09 AM, 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -8.684139543253961 (T) = (0 -174.1069879885388) / Math.Sqrt((374.47114038979817 / (25)) + (507.0869613018526 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.487308154046983 = (117.06181231830918 - 174.1069879885388) / 117.06181231830918 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CopyTo<Int32>.Array(Size: 2048)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 182.69732039281573 > 117.76185099302873.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -13.75654105606142 (T) = (0 -183.21292876486717) / Math.Sqrt((533.4266234818236 / (26)) + (0.8952061762260242 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -0.5163226401844222 = (120.82714055010348 - 183.21292876486717) / 120.82714055010348 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 442141d696ee7d4ae54420974559e99738d753e6
Compare c8f43d52f699c1da289a5cb36d371aec15901ab8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Guid

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryWriteBytes - Duration of single invocation 0.51 ns 4.47 ns 8.71 0.04 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Guid*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Guid.TryWriteBytes

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.467159589801238 > 0.5353243770826558.
IsChangePoint: Marked as a change because one of 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -263.47363700021424 (T) = (0 -4.47755744224968) / Math.Sqrt((0.0023788205748050523 / (26)) + (0.002393072681852027 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -7.4424835622235355 = (0.5303602203366808 - 4.47755744224968) / 0.5303602203366808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels May 9, 2023
@cincuranet cincuranet changed the title [Perf] Linux/x64: 7 Regressions on 5/3/2023 1:06:54 PM Regressions in System.Tests.Perf_Guid May 9, 2023
@cincuranet cincuranet removed the untriaged New issue has not been triaged by the area owner label May 9, 2023
@cincuranet cincuranet transferred this issue from dotnet/perf-autofiling-issues May 9, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 9, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label May 9, 2023
@cincuranet cincuranet mentioned this issue May 9, 2023
Open
@cincuranet
Copy link
Contributor

Commit range 7afd85d...edd6d63.

@cincuranet cincuranet assigned EgorBo and unassigned AndyAyersMS May 9, 2023
@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label May 10, 2023
@ghost
Copy link

ghost commented May 10, 2023

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 442141d696ee7d4ae54420974559e99738d753e6
Compare c8f43d52f699c1da289a5cb36d371aec15901ab8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MeasureReverseUsingNtoH - Duration of single invocation 331.25 ns 657.14 ns 1.98 0.00 True
MeasureReverseEndianness - Duration of single invocation 331.08 ns 656.62 ns 1.98 0.00 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.MeasureReverseUsingNtoH

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 657.1443752993548 > 348.04767438472953.
IsChangePoint: Marked as a change because one of 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -1725.4828094217587 (T) = (0 -657.7838151245886) / Math.Sqrt((0.2889242088902872 / (26)) + (0.4435805265031423 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -0.9842134137794782 = (331.5086021274592 - 657.7838151245886) / 331.5086021274592 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.MeasureReverseEndianness

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 656.6179442968024 > 347.85342306747543.
IsChangePoint: Marked as a change because one of 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -1895.5849901028187 (T) = (0 -657.5466614763055) / Math.Sqrt((0.09788024557592748 / (26)) + (0.4649324313081097 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -0.9838498204419397 = (331.44981777392036 - 657.5466614763055) / 331.44981777392036 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 442141d696ee7d4ae54420974559e99738d753e6
Compare c8f43d52f699c1da289a5cb36d371aec15901ab8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CopyTo<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Memory - Duration of single invocation 114.85 ns 185.21 ns 1.61 0.25 False
ReadOnlyMemory - Duration of single invocation 112.70 ns 184.28 ns 1.64 0.31 False
Span - Duration of single invocation 111.17 ns 181.52 ns 1.63 0.28 False
Array - Duration of single invocation 112.20 ns 182.70 ns 1.63 0.28 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CopyTo<Int32>.Memory(Size: 2048)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 185.2139209713077 > 118.96458191097815.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -11.61978521318909 (T) = (0 -184.8462357983078) / Math.Sqrt((696.3504002888097 / (26)) + (1.1656438290390554 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -0.48305469158313313 = (124.63885306953036 - 184.8462357983078) / 124.63885306953036 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CopyTo<Int32>.ReadOnlyMemory(Size: 2048)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 184.27621651852104 > 118.47282810115816.
IsChangePoint: Marked as a change because one of 4/17/2023 11:29:15 PM, 5/2/2023 4:25:00 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -6.0470748517318516 (T) = (0 -175.30536684559712) / Math.Sqrt((828.0418257010215 / (21)) + (572.8159336579739 / (23))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (23) - 2, .025) and -0.3825115960587915 = (126.80209507489892 - 175.30536684559712) / 126.80209507489892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CopyTo<Int32>.Span(Size: 2048)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 181.5209793778094 > 116.61576769905994.
IsChangePoint: Marked as a change because one of 3/16/2023 2:58:09 AM, 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -8.684139543253961 (T) = (0 -174.1069879885388) / Math.Sqrt((374.47114038979817 / (25)) + (507.0869613018526 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.487308154046983 = (117.06181231830918 - 174.1069879885388) / 117.06181231830918 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CopyTo<Int32>.Array(Size: 2048)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 182.69732039281573 > 117.76185099302873.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -13.75654105606142 (T) = (0 -183.21292876486717) / Math.Sqrt((533.4266234818236 / (26)) + (0.8952061762260242 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -0.5163226401844222 = (120.82714055010348 - 183.21292876486717) / 120.82714055010348 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 442141d696ee7d4ae54420974559e99738d753e6
Compare c8f43d52f699c1da289a5cb36d371aec15901ab8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Guid

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryWriteBytes - Duration of single invocation 0.51 ns 4.47 ns 8.71 0.04 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Guid*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Guid.TryWriteBytes

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.467159589801238 > 0.5353243770826558.
IsChangePoint: Marked as a change because one of 5/3/2023 7:05:50 AM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -263.47363700021424 (T) = (0 -4.47755744224968) / Math.Sqrt((0.0023788205748050523 / (26)) + (0.002393072681852027 / (18))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (18) - 2, .025) and -7.4424835622235355 = (0.5303602203366808 - 4.47755744224968) / 0.5303602203366808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: EgorBo
Labels:

os-linux, arch-x64, area-CodeGen-coreclr, untriaged, runtime-coreclr, needs-area-label
Milestone: -

@vcsjones vcsjones removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 11, 2023
@JulieLeeMSFT JulieLeeMSFT added this to the 8.0.0 milestone May 11, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label May 11, 2023
@EgorBo
Copy link
Member

EgorBo commented Jun 9, 2023

System.Tests.Perf_Guid regression is no more
image

System.Buffers.Binary.Tests.BinaryReadAndWriteTests
Regressed due to loop alignment:

private readonly static int[] _oneThousandIntegers = new int[1000];

[Benchmark]
public int[] MeasureReverseEndianness()
{
    int[] local = _oneThousandIntegers;

    for (int j = 0; j < local.Length; j++)
    {
        local[j] = ReverseEndianness(local[j]);
    }

    return local;
}

#85559 optimized codegen for _oneThousandIntegers so it produces less instructions to load -> loop is misaligned

@EgorBo EgorBo closed this as completed Jun 9, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Jul 9, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@EgorBo EgorBo

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
6 participants
@vcsjones @EgorBo @cincuranet @jeffschwMSFT @AndyAyersMS @JulieLeeMSFT