Regressions in System.Formats.Tar.Tests.Perf_TarFile

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	84fd85932fd3b371cad69cc2412404ea50158c63
Compare	18321c608e32bc5d13ad0299822cf02ab7ebfd6c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Formats.Tar.Tests.Perf_TarFile

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ExtractToDirectory_Stream - Duration of single invocation	478.32 μs	517.10 μs	1.08	0.03	False
ExtractToDirectory_Path - Duration of single invocation	502.99 μs	539.88 μs	1.07	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Formats.Tar.Tests.Perf_TarFile*'

Payloads

Baseline
Compare

System.Formats.Tar.Tests.Perf_TarFile.ExtractToDirectory_Stream

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 517.1013727678571 > 496.80518125.
IsChangePoint: Marked as a change because one of 6/22/2023 7:55:14 AM, 7/5/2023 10:47:48 AM, 7/10/2023 4:47:56 PM falls between 7/1/2023 10:14:52 PM and 7/10/2023 4:47:56 PM.
IsRegressionStdDev: Marked as regression because -15.376454381769685 (T) = (0 -515616.8345706371) / Math.Sqrt((73988476.72604911 / (25)) + (40536261.81500051 / (16))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (16) - 2, .025) and -0.07514556876581584 = (479578.6259552978 - 515616.8345706371) / 479578.6259552978 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Formats.Tar.Tests.Perf_TarFile.ExtractToDirectory_Path

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 539.8829398148146 > 528.2676646721869.
IsChangePoint: Marked as a change because one of 6/22/2023 12:06:43 AM, 7/5/2023 10:47:48 AM, 7/10/2023 4:47:56 PM falls between 7/1/2023 10:14:52 PM and 7/10/2023 4:47:56 PM.
IsRegressionStdDev: Marked as regression because -13.699217066741582 (T) = (0 -538250.606504869) / Math.Sqrt((57943146.09601676 / (25)) + (61795711.63838883 / (16))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (16) - 2, .025) and -0.06754449520001374 = (504195.0091307651 - 538250.606504869) / 504195.0091307651 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[cincuranet]

Probably caused by #88231. cc @tmds

[cincuranet]

More instances:

• [Perf] Linux/x64: 1 Regression on 7/5/2023 3:16:33 PM perf-autofiling-issues#19674

[tmds]

Probably caused by #88231. cc @tmds

We're now setting directory timestamps and that causes the extraction to take more time.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	84fd85932fd3b371cad69cc2412404ea50158c63
Compare	18321c608e32bc5d13ad0299822cf02ab7ebfd6c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Formats.Tar.Tests.Perf_TarFile

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ExtractToDirectory_Stream - Duration of single invocation	478.32 μs	517.10 μs	1.08	0.03	False
ExtractToDirectory_Path - Duration of single invocation	502.99 μs	539.88 μs	1.07	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Formats.Tar.Tests.Perf_TarFile*'

Payloads

Baseline
Compare

System.Formats.Tar.Tests.Perf_TarFile.ExtractToDirectory_Stream

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 517.1013727678571 > 496.80518125.
IsChangePoint: Marked as a change because one of 6/22/2023 7:55:14 AM, 7/5/2023 10:47:48 AM, 7/10/2023 4:47:56 PM falls between 7/1/2023 10:14:52 PM and 7/10/2023 4:47:56 PM.
IsRegressionStdDev: Marked as regression because -15.376454381769685 (T) = (0 -515616.8345706371) / Math.Sqrt((73988476.72604911 / (25)) + (40536261.81500051 / (16))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (16) - 2, .025) and -0.07514556876581584 = (479578.6259552978 - 515616.8345706371) / 479578.6259552978 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Formats.Tar.Tests.Perf_TarFile.ExtractToDirectory_Path

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 539.8829398148146 > 528.2676646721869.
IsChangePoint: Marked as a change because one of 6/22/2023 12:06:43 AM, 7/5/2023 10:47:48 AM, 7/10/2023 4:47:56 PM falls between 7/1/2023 10:14:52 PM and 7/10/2023 4:47:56 PM.
IsRegressionStdDev: Marked as regression because -13.699217066741582 (T) = (0 -538250.606504869) / Math.Sqrt((57943146.09601676 / (25)) + (61795711.63838883 / (16))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (16) - 2, .025) and -0.06754449520001374 = (504195.0091307651 - 538250.606504869) / 504195.0091307651 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	os-windows, arch-x64, untriaged, runtime-coreclr, PGO, area-System.Formats.Tar, needs-area-label
Milestone:	-

[EgorBo]

Same on win-arm64:

• [Perf] Windows/arm64: 1 Regression on 7/5/2023 5:05:14 PM perf-autofiling-issues#19785

[carlossanlop]

We believe the timestamp change is the correct thing to do, even if it impacts performance.