"Fatal error. Internal CLR error." when running System.Diagnostics.PerformanceCounter.Tests in CI

[ViktorHofer]

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1762031&view=ms.vss-test-web.build-test-results-tab&runId=47426830&resultId=201060&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Configuration: net7.0-windows-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open

Dump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69116-merge-f2fd77dfcc414e96a6/System.Diagnostics.PerformanceCounter.Tests/1/dotnet.exe.7988.dmp?helixlogtype=result

C:\h\w\BC380A28\w\B02209AD\e>"C:\h\w\BC380A28\p\dotnet.exe" exec --runtimeconfig System.Diagnostics.PerformanceCounter.Tests.runtimeconfig.json --depsfile System.Diagnostics.PerformanceCounter.Tests.deps.json xunit.console.dll System.Diagnostics.PerformanceCounter.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Diagnostics.PerformanceCounter.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Diagnostics.PerformanceCounter.Tests (found 86 of 92 test cases)
  Starting:    System.Diagnostics.PerformanceCounter.Tests (parallel test collections = on, max threads = 2)
Fatal error. Internal CLR error. (0x80131506)
Out of memory.
Out of memory.
----- end 10/05/2022 11:00:50,70 ----- exit code -1073741819 ----------------------------------------------------------
2022-05-10T11:00:51.263Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2022-05-10T11:00:51.263Z	INFO   	run.py	__init__(42)	read_results	Searching 'C:\h\w\BC380A28\w\B02209AD\e' for test results files
2022-05-10T11:00:51.263Z	INFO   	run.py	__init__(42)	read_results	Searching 'C:\h\w\BC380A28\w\B02209AD\uploads' for test results files
2022-05-10T11:00:51.263Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2022-05-10T11:00:51.263Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to 'C:\h\w\BC380A28\w\B02209AD\e\__test_report.json'
2022-05-10T11:00:51.263Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1414 bytes
read file: C:\h\w\BC380A28\p\debug-dump-template.md
writing output file: C:\h\w\BC380A28\w\B02209AD\uploads\how-to-debug-dump.md
done writing debug dump information

cc @jkotas

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jkotas]

Crash stacktrace:

Call Site
coreclr!MethodTable::GetFlag [D:\a\_work\1\s\src\coreclr\vm\methodtable.h @ 3517] 
coreclr!MethodTable::HasComponentSize [D:\a\_work\1\s\src\coreclr\vm\methodtable.h @ 1525] 
coreclr!WKS::my_get_size+0x7 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 10863] 
coreclr!WKS::gc_heap::background_mark_simple1+0x3f7 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 24243] 
coreclr!WKS::gc_heap::background_mark_simple+0x80 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 24348] 
coreclr!WKS::gc_heap::background_promote+0x64 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 24429] 
coreclr!ScanConsecutiveHandlesWithoutUserData+0x45 [D:\a\_work\1\s\src\coreclr\gc\handletablescan.cpp @ 446] 
coreclr!BlockScanBlocksWithoutUserData+0x44 [D:\a\_work\1\s\src\coreclr\gc\handletablescan.cpp @ 557] 
coreclr!ProcessScanQNode+0x44 [D:\a\_work\1\s\src\coreclr\gc\handletablescan.cpp @ 1233] 
coreclr!ProcessScanQueue+0x3f [D:\a\_work\1\s\src\coreclr\gc\handletablescan.cpp @ 1201] 
coreclr!xxxTableScanQueuedBlocksAsync+0x5d [D:\a\_work\1\s\src\coreclr\gc\handletablescan.cpp @ 1314] 
coreclr!xxxAsyncSegmentIterator+0x55 [D:\a\_work\1\s\src\coreclr\gc\handletablescan.cpp @ 1536] 
coreclr!TableScanHandles+0x80 [D:\a\_work\1\s\src\coreclr\gc\handletablescan.cpp @ 1698] 
coreclr!xxxTableScanHandlesAsync+0x97 [D:\a\_work\1\s\src\coreclr\gc\handletablescan.cpp @ 1793] 
coreclr!HndScanHandlesForGC+0x129 [D:\a\_work\1\s\src\coreclr\gc\handletable.cpp @ 782] 
coreclr!Ref_TraceNormalRoots+0x151 [D:\a\_work\1\s\src\coreclr\gc\objecthandle.cpp @ 1059] 
coreclr!GCScan::GcScanHandles+0x57 [D:\a\_work\1\s\src\coreclr\gc\gcscan.cpp @ 175] 
coreclr!WKS::gc_heap::background_mark_phase+0x1c5 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 33937] 
coreclr!WKS::gc_heap::gc1+0x4a6 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 20605] 
coreclr!WKS::gc_heap::bgc_thread_function+0xc9 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 35113] 

[jkotas]

0:014> !verifyheap
Object 00000232519d4c18 contains free object 0000023251263BB0 at 00000232519D4C40.
Last good object: 00000232519D4BF8.
0:014> !dumpobj 00000232519d4c18 
Name:        System.Collections.Hashtable+Bucket[]
MethodTable: 00007ffc8c514390
EEClass:     00007ffc8c5142f8
Tracked Type: false
Size:        140160(0x22380) bytes
Array:       Rank 1, Number of elements 5839, Type VALUETYPE (Print Array)
Fields:
None

This is bucket array from a Hashtable created by System.Diagnostics.PerformanceCounter library. Key is corrupted object reference in one of the buckets.

[jkotas]

This matches the pattern of #68443 (corrupted array in LOH). The PR that hit this crash missed the fix for #68443 by narrow margin.

Closing as duplicate of #68443.

[ViktorHofer]

Thanks for the investigation, Jan.