[Perf] Alpine/x64: 11 Regressions on 3/28/2023 12:51:33 PM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	alpine 3.15
Queue	TigerAlpine
Baseline	93e91df5a7c54dd4bf133f9535acd705f91b789a
Compare	49ae58ace038dbbd3d53147e08c9ac95b24a789f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
MatchesSet - Duration of single invocation	39.16 μs	43.61 μs	1.11	0.02	True
MatchesSet - Duration of single invocation	39.20 μs	43.48 μs	1.11	0.02	True
MatchesBoundary - Duration of single invocation	41.01 μs	49.90 μs	1.22	0.01	True
Date_IsNotMatch - Duration of single invocation	86.76 ns	97.36 ns	1.12	0.01	True
OneNodeBacktracking - Duration of single invocation	58.39 ns	74.35 ns	1.27	0.12	True
Date_IsNotMatch - Duration of single invocation	87.75 ns	98.18 ns	1.12	0.01	True
Backtracking - Duration of single invocation	1.13 μs	1.27 μs	1.12	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 43.61032760416666 > 41.24964259487966.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -75.22414114054133 (T) = (0 -43439.45432859062) / Math.Sqrt((68272.1239066228 / (28)) + (26706.154441387105 / (37))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (37) - 2, .025) and -0.10784521234196856 = (39210.761435489934 - 43439.45432859062) / 39210.761435489934 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 43.484195920138895 > 41.176470560406.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -25.663018222909233 (T) = (0 -43946.66731778299) / Math.Sqrt((188438.47857281848 / (29)) + (993008.2375906764 / (37))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (37) - 2, .025) and -0.11934434414081192 = (39261.079530906674 - 43946.66731778299) / 39261.079530906674 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.89972763578275 > 43.038873278353506.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -114.82274011589874 (T) = (0 -50110.53314355519) / Math.Sqrt((11573.963314251396 / (27)) + (206249.26632446575 / (35))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (35) - 2, .025) and -0.22276816581175926 = (40981.221579552905 - 50110.53314355519) / 40981.221579552905 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 97.36306543673355 > 91.20367576957486.
IsChangePoint: Marked as a change because one of 1/27/2023 7:12:54 PM, 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -28.277271306700197 (T) = (0 -98.10556751947445) / Math.Sqrt((0.47342670667139913 / (28)) + (4.884956013060185 / (35))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (35) - 2, .025) and -0.12869024634394993 = (86.91983282150059 - 98.10556751947445) / 86.91983282150059 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 74.35229951826547 > 57.90630459778321.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -16.38060913337907 (T) = (0 -72.7072458612854) / Math.Sqrt((3.7091063450775246 / (27)) + (37.09363229266293 / (35))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (35) - 2, .025) and -0.3271574094133653 = (54.78419164567955 - 72.7072458612854) / 54.78419164567955 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 98.17901973991293 > 92.10149387468489.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -41.900493142788584 (T) = (0 -98.9944086456009) / Math.Sqrt((0.9007620371389327 / (28)) + (1.2755478617676392 / (34))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (34) - 2, .025) and -0.125787745908208 = (87.93345726617324 - 98.9944086456009) / 87.93345726617324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: None)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2660580599806448 > 1.1908465135680635.
IsChangePoint: Marked as a change because one of 2/20/2023 9:09:49 AM, 3/25/2023 10:49:06 AM, 3/28/2023 3:45:48 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -12.7371129529966 (T) = (0 -1274.1883676226917) / Math.Sqrt((1765.867205314915 / (27)) + (48.05664132193044 / (39))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (39) - 2, .025) and -0.08884960919288189 = (1170.2152040695441 - 1274.1883676226917) / 1170.2152040695441 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	alpine 3.15
Queue	TigerAlpine
Baseline	93e91df5a7c54dd4bf133f9535acd705f91b789a
Compare	49ae58ace038dbbd3d53147e08c9ac95b24a789f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	8.34 ms	9.42 ms	1.13	0.01	True
Count - Duration of single invocation	11.20 ms	12.81 ms	1.14	0.13	False
Count - Duration of single invocation	2.96 ms	3.32 ms	1.12	0.02	True
Count - Duration of single invocation	11.27 ms	12.34 ms	1.09	0.16	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\b\w+n\b", Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.417402215976331 > 8.646365512256411.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -47.47903208738034 (T) = (0 -9485974.993467225) / Math.Sqrt((3474330721.9450374 / (29)) + (19899848536.086803 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.1532770336455011 = (8225235.322238335 - 9485974.993467225) / 8225235.322238335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Ll}", Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.805346610576924 > 11.698805147115385.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -8.810349157436189 (T) = (0 -12066745.324539319) / Math.Sqrt((145773947145.79868 / (28)) + (127006316754.92084 / (37))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (37) - 2, .025) and -0.07280313984117494 = (11247865.406439584 - 12066745.324539319) / 11247865.406439584 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+\s+Holmes", Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3213264412499997 > 3.1061300618990386.
IsChangePoint: Marked as a change because one of 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -26.958442672122292 (T) = (0 -3240018.845548955) / Math.Sqrt((256254191.75127885 / (28)) + (3557775770.139129 / (37))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (37) - 2, .025) and -0.09335538438331446 = (2963372.1037340695 - 3240018.845548955) / 2963372.1037340695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{L}", Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.335949431578946 > 11.839315192692307.
IsChangePoint: Marked as a change because one of 3/21/2023 10:36:00 PM, 3/22/2023 9:02:35 AM, 3/28/2023 8:13:28 AM, 4/3/2023 9:14:01 PM falls between 3/26/2023 7:47:31 AM and 4/3/2023 9:14:01 PM.
IsRegressionStdDev: Marked as regression because -19.429234848233275 (T) = (0 -12538729.9174171) / Math.Sqrt((427071216.42220044 / (28)) + (144097536591.46198 / (35))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (35) - 2, .025) and -0.1106288327335994 = (11289757.25090391 - 12538729.9174171) / 11289757.25090391 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository