Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

readytorun/determinism/crossgen2determinism/crossgen2determinism.sh [FAIL] #50466

Closed
AaronRobinsonMSFT opened this issue Mar 31, 2021 · 11 comments
Closed

readytorun/determinism/crossgen2determinism/crossgen2determinism.sh [FAIL] #50466

AaronRobinsonMSFT opened this issue Mar 31, 2021 · 11 comments
Labels
arch-arm64 area-crossgen2-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' os-macos-bigsur (macOS11)
Milestone
6.0.0

Comments

@AaronRobinsonMSFT
Copy link
Member

AaronRobinsonMSFT commented Mar 31, 2021 •
edited by runfoapp bot
Loading

There is an intermittent failure with the prep for this test.

export CoreRT_DeterminismSeed=1
$CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll --map -r:$CORE_ROOT/*.dll -r:../../crossgen2/crossgen2smoke/helperdll.dll -r:../../crossgen2/crossgen2smoke/helperildll.dll -o:crossgen2smoke1.ildll ../../crossgen2/crossgen2smoke/crossgen2smoke.dll
export CoreRT_DeterminismSeed=2
$CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll --map -r:$CORE_ROOT/*.dll -r:../../crossgen2/crossgen2smoke/helperdll.dll -r:../../crossgen2/crossgen2smoke/helperildll.dll -o:crossgen2smoke2.ildll ../../crossgen2/crossgen2smoke/crossgen2smoke.dll

Observed in: #50452 #50393

The following is from the log.

    readytorun/determinism/crossgen2determinism/crossgen2determinism.sh [FAIL]
      Unhandled exception. System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
       ---> System.NullReferenceException: Object reference not set to an instance of an object.
         at ILCompiler.Sorting.Implementation.MergeSortCore`5.ParallelSort(TDataStructure arrayToSort, Int32 index, Int32 length, TComparer comparer)
         --- End of inner exception stack trace ---
         at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
         at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
         at System.Threading.Tasks.Task.Wait()
         at ILCompiler.Sorting.Implementation.MergeSortCore`5.ParallelSortApi(TDataStructure arrayToSort, TComparer comparer)
         at ILCompiler.MergeSortApi.MergeSort[T](List`1 listToSort, IComparer`1 comparer)
         at ILCompiler.DependencyAnalysis.ArrayOfEmbeddedDataNode`1.GetData(NodeFactory factory, Boolean relocsOnly)
         at ILCompiler.DependencyAnalysis.ObjectNode.GetStaticDependencies(NodeFactory factory)
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependenciesImpl(DependencyNodeCore`1 node)
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependencies(DependencyNodeCore`1 node)
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ProcessMarkStack()
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes()
         at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile)
         at ILCompiler.Program.Run(String[] args)
         at ILCompiler.Program.Main(String[] args)
      dyld: Library not loaded: @rpath/libmscordaccore.dylib
        Referenced from: /private/tmp/helix/working/A87408E4/p/createdump
        Reason: image not found
      /private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2determinism.sh: line 144: 59345 Abort trap: 6           (core dumped) $CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll --map -r:$CORE_ROOT/*.dll -r:../../crossgen2/crossgen2smoke/helperdll.dll -r:../../crossgen2/crossgen2smoke/helperildll.dll -o:crossgen2smoke2.ildll ../../crossgen2/crossgen2smoke/crossgen2smoke.dll
      Unhandled exception. System.IO.FileNotFoundException: Could not find file '/private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2smoke2.ildll'.
      File name: '/private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2smoke2.ildll'
         at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
         at Interop.CheckIo(Error error, String path, Boolean isDirectory, Func`2 errorRewriter)
         at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
         at System.IO.Strategies.FileStreamHelpers.OpenHandle(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options)
         at System.IO.Strategies.LegacyFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
         at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
         at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
         at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
         at System.IO.File.ReadAllBytes(String path) in /_/src/libraries/System.IO.FileSystem/src/System/IO/File.cs:line 332
         at Program.Main()
      dyld: Library not loaded: @rpath/libmscordaccore.dylib
        Referenced from: /private/tmp/helix/working/A87408E4/p/createdump
        Reason: image not found
      /private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2determinism.sh: line 255: 59348 Abort trap: 6           (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"
      
      Return code:      1
      Raw output file:      /private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/Reports/readytorun.determinism/crossgen2determinism/crossgen2determinism.output.txt
      Raw output:
      BEGIN EXECUTION
      Emitting R2R PE file: crossgen2smoke1.ildll
      Emitting map file: crossgen2smoke1.map
      /tmp/helix/working/A87408E4/p/corerun crossgen2determinism.dll ''
      Expected: 100
      Actual: 134
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/tmp/helix/working/A87408E4/p
      > /private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2determinism.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at readytorun_determinism._crossgen2determinism_crossgen2determinism_._crossgen2determinism_crossgen2determinism_sh()
      Output:
        Unhandled exception. System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
         ---> System.NullReferenceException: Object reference not set to an instance of an object.
           at ILCompiler.Sorting.Implementation.MergeSortCore`5.ParallelSort(TDataStructure arrayToSort, Int32 index, Int32 length, TComparer comparer)
           --- End of inner exception stack trace ---
           at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
           at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
           at System.Threading.Tasks.Task.Wait()
           at ILCompiler.Sorting.Implementation.MergeSortCore`5.ParallelSortApi(TDataStructure arrayToSort, TComparer comparer)
           at ILCompiler.MergeSortApi.MergeSort[T](List`1 listToSort, IComparer`1 comparer)
           at ILCompiler.DependencyAnalysis.ArrayOfEmbeddedDataNode`1.GetData(NodeFactory factory, Boolean relocsOnly)
           at ILCompiler.DependencyAnalysis.ObjectNode.GetStaticDependencies(NodeFactory factory)
           at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependenciesImpl(DependencyNodeCore`1 node)
           at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependencies(DependencyNodeCore`1 node)
           at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ProcessMarkStack()
           at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes()
           at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile)
           at ILCompiler.Program.Run(String[] args)
           at ILCompiler.Program.Main(String[] args)
        dyld: Library not loaded: @rpath/libmscordaccore.dylib
          Referenced from: /private/tmp/helix/working/A87408E4/p/createdump
          Reason: image not found
        /private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2determinism.sh: line 144: 59345 Abort trap: 6           (core dumped) $CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll --map -r:$CORE_ROOT/*.dll -r:../../crossgen2/crossgen2smoke/helperdll.dll -r:../../crossgen2/crossgen2smoke/helperildll.dll -o:crossgen2smoke2.ildll ../../crossgen2/crossgen2smoke/crossgen2smoke.dll
        Unhandled exception. System.IO.FileNotFoundException: Could not find file '/private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2smoke2.ildll'.
        File name: '/private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2smoke2.ildll'
           at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
           at Interop.CheckIo(Error error, String path, Boolean isDirectory, Func`2 errorRewriter)
           at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
           at System.IO.Strategies.FileStreamHelpers.OpenHandle(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options)
           at System.IO.Strategies.LegacyFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
           at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
           at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
           at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
           at System.IO.File.ReadAllBytes(String path) in /_/src/libraries/System.IO.FileSystem/src/System/IO/File.cs:line 332
           at Program.Main()
        dyld: Library not loaded: @rpath/libmscordaccore.dylib
          Referenced from: /private/tmp/helix/working/A87408E4/p/createdump
          Reason: image not found
        /private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2determinism.sh: line 255: 59348 Abort trap: 6           (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"
        
        Return code:      1
        Raw output file:      /private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/Reports/readytorun.determinism/crossgen2determinism/crossgen2determinism.output.txt
        Raw output:
        BEGIN EXECUTION
        Emitting R2R PE file: crossgen2smoke1.ildll
        Emitting map file: crossgen2smoke1.map
        /tmp/helix/working/A87408E4/p/corerun crossgen2determinism.dll ''
        Expected: 100
        Actual: 134
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/tmp/helix/working/A87408E4/p
        > /private/tmp/helix/working/A87408E4/w/B716095B/e/readytorun/determinism/crossgen2determinism/crossgen2determinism.sh

Runfo Tracking Issue: readytorun/determinism/crossgen2determinism/crossgen2determinism.sh

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1146504 runtime PR 52985 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1144495 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1144033 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1143765 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1143765 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1143562 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1143420 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1143363 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1143145 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1143083 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1142976 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1142790 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1142541 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1142369 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1142128 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1141967 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1141758 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1141555 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1141500 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1141398 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1140953 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1140906 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1140838 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1140434 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1140401 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1140321 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1140087 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1139886 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1139640 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1139011 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1138876 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1138731 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1138664 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py
1129852 runtime PR 52523 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open
1129852 runtime PR 52523 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open
1129210 runtime PR 52505 Mono Linux arm64 Release interpreter @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855
1126946 runtime PR 52424 Mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855
1126946 runtime PR 52424 Mono Linux arm64 Release interpreter @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855
1125638 runtime PR 52176 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open
1125551 runtime PR 52156 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open
1125551 runtime PR 52156 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open
1125551 runtime PR 52156 CoreCLR Linux arm Checked @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
1125551 runtime PR 52156 CoreCLR Linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
1125551 runtime PR 52156 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open
1125551 runtime PR 52156 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open
1124964 runtime PR 52288 CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open
1124964 runtime PR 52288 CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open
1124964 runtime PR 52288 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open
1124964 runtime PR 52288 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open
1124475 runtime PR 52156 CoreCLR Linux arm Checked @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
1124475 runtime PR 52156 CoreCLR Linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
1124475 runtime PR 52156 CoreCLR Linux arm64 Checked @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855
1124475 runtime PR 52156 CoreCLR Linux arm64 Checked no_tiered_compilation @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855
1124475 runtime PR 52156 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open
1124475 runtime PR 52156 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open
1124475 runtime PR 52156 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open
1124475 runtime PR 52156 CoreCLR OSX arm64 Checked no_tiered_compilation @ OSX.1100.ARM64.Open
1124475 runtime PR 52156 CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open
1118306 runtime PR 52183 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump
1118147 runtime PR 52176 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump
1118078 runtime PR 52175 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm64 Checked no_tiered_compilation @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm64 Checked @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm Checked @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1117990 runtime PR 52156 CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open core dump
1117990 runtime PR 52156 CoreCLR OSX arm64 Checked no_tiered_compilation @ OSX.1100.ARM64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR OSX arm64 Checked no_tiered_compilation @ OSX.1100.ARM64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open console.log
1117990 runtime PR 52156 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm64 Checked no_tiered_compilation @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm64 Checked @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1117990 runtime PR 52156 CoreCLR OSX arm64 Checked no_tiered_compilation @ OSX.1100.ARM64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open core dump
1117990 runtime PR 52156 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm Checked @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm64 Checked no_tiered_compilation @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 console.log core dump
1117990 runtime PR 52156 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open console.log core dump
1117990 runtime PR 52156 CoreCLR Linux arm64 Checked @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 console.log core dump
1117516 runtime PR 52156 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open console.log core dump
1117516 runtime PR 52156 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open console.log core dump
1117516 runtime PR 52156 CoreCLR Linux arm Checked @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1117516 runtime PR 52156 CoreCLR Linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump
1117516 runtime PR 52156 CoreCLR Linux arm64 Checked @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 console.log core dump
1117516 runtime PR 52156 CoreCLR Linux arm64 Checked no_tiered_compilation @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 console.log core dump
1117516 runtime PR 52156 CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump
1117516 runtime PR 52156 CoreCLR OSX arm64 Checked no_tiered_compilation @ OSX.1100.ARM64.Open console.log core dump
1110944 runtime PR 51850 CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open console.log core dump
1110944 runtime PR 51850 CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open console.log core dump
1110944 runtime PR 51850 CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open console.log core dump
1109541 runtime PR 51850 CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open core dump

Displaying 100 of 110 results

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
1 32 51
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Mar 31, 2021
@AaronRobinsonMSFT AaronRobinsonMSFT mentioned this issue Mar 31, 2021
Merged
@AaronRobinsonMSFT
Copy link
Member Author

AaronRobinsonMSFT commented Mar 31, 2021 •
edited
Loading

I was able to reproduce this on an M1 a few times under a debugger. I updated crossgen2determinism.sh as below. Note the addition of the -d flag passed to corerun - tells the process to for a debugger to attach prior to loading the runtime. The failure above was observed less than 50 % of the time but only on a Checked build of the runtime.

#export CoreRT_DeterminismSeed=1
#$CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll --map -r:$CORE_ROOT/*.dll -r:../../crossgen2/crossgen2smoke/helperdll.dll -r:../../crossgen2/crossgen2smoke/helperildll.dll -o:crossgen2smoke1.ildll ../../crossgen2/crossgen2smoke/crossgen2smoke.dll
export CoreRT_DeterminismSeed=2
$CORE_ROOT/corerun -d $CORE_ROOT/crossgen2/crossgen2.dll --map -r:$CORE_ROOT/*.dll -r:../../crossgen2/crossgen2smoke/helperdll.dll -r:../../crossgen2/crossgen2smoke/helperildll.dll -o:crossgen2smoke2.ildll ../../crossgen2/crossgen2smoke/crossgen2smoke.dll

exit 0

Stack track

(lldb) bt
libcoreclr.dylib was compiled with optimization - stepping may behave oddly; variables may not be available.
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x0000000192ccccec libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x0000000192cfdc24 libsystem_pthread.dylib`pthread_kill + 292
    frame #2: 0x0000000192c45864 libsystem_c.dylib`abort + 104
  * frame #3: 0x0000000102bc570c libcoreclr.dylib`::PROCAbort() at process.cpp:3363:5 [opt]
    frame #4: 0x0000000102bc546c libcoreclr.dylib`PROCEndProcess(hProcess=0x00000000ffffff01, uExitCode=1, bTerminateUnconditionally=YES) at process.cpp:1338:13 [opt]
    frame #5: 0x0000000102bc550c libcoreclr.dylib`::TerminateProcess(hProcess=0x00000000ffffff01, uExitCode=1) at process.cpp:1237:11 [opt]
    frame #6: 0x0000000102c73cfc libcoreclr.dylib`CrashDumpAndTerminateProcess(exitCode=<unavailable>) at excep.cpp:4189:5 [opt] [artificial]
    frame #7: 0x0000000102ef3980 libcoreclr.dylib`UnwindManagedExceptionPass1(ex=0x000000016d429a80, frameContext=<unavailable>) at exceptionhandling.cpp:4640:17 [opt]
    frame #8: 0x0000000102ef3b08 libcoreclr.dylib`DispatchManagedException(ex=0x000000016d429a80, isHardwareException=<unavailable>) at exceptionhandling.cpp:4712:17 [opt]
    frame #9: 0x0000000102e1c4d8 libcoreclr.dylib`IL_Throw(obj=<unavailable>) at jithelpers.cpp:4129:5 [opt]
    frame #10: 0x0000000281d1db14
    frame #11: 0x0000000281d13b1c
    frame #12: 0x0000000281d13a4c
    frame #13: 0x0000000281d11b74
    frame #14: 0x0000000281d119d4
    frame #15: 0x0000000281d10aec
    frame #16: 0x00000002819be5dc
    frame #17: 0x00000002819bcfc4
    frame #18: 0x00000002819bca00
    frame #19: 0x00000002819bc380
    frame #20: 0x00000002819bbb1c
    frame #21: 0x00000002819b9de0
    frame #22: 0x0000000280ce91f4
    frame #23: 0x0000000280cdb118
    frame #24: 0x0000000102f9254c libcoreclr.dylib`CallDescrWorkerInternal at calldescrworkerarm64.S:71
    frame #25: 0x0000000102d93ce8 libcoreclr.dylib`CallDescrWorkerWithHandler(pCallDescrData=0x000000016d42a780, fCriticalCall=<unavailable>) at callhelpers.cpp:71:5 [opt]
    frame #26: 0x0000000102d94630 libcoreclr.dylib`MethodDescCallSite::CallTargetWorker(this=0x000000016d42a968, pArguments=0x000000016d42a930, pReturnValue=0x000000016d42aa48, cbReturnValue=8) at callhelpers.cpp:547:9 [opt]
    frame #27: 0x0000000102bf2164 libcoreclr.dylib`RunMain(MethodDesc*, short, int*, REF<PtrArray>*) [inlined] MethodDescCallSite::Call_RetArgSlot(this=0x000000016d42a968, pArguments=0x000000016d42a930) at callhelpers.h:458:9 [opt]
    frame #28: 0x0000000102bf2130 libcoreclr.dylib`RunMain(MethodDesc*, short, int*, REF<PtrArray>*) at assembly.cpp:1481 [opt]
    frame #29: 0x0000000102bf1f50 libcoreclr.dylib`RunMain(MethodDesc*, short, int*, REF<PtrArray>*) [inlined] RunMain(this=<unavailable>, pParam=0x000000016d42a8b8)::$_0::operator()(Param*) const::'lambda'(Param*)::operator()(Param*) const at assembly.cpp:1549 [opt]
    frame #30: 0x0000000102bf1f50 libcoreclr.dylib`RunMain(MethodDesc*, short, int*, REF<PtrArray>*) at assembly.cpp:1551 [opt]
    frame #31: 0x0000000102bf1e74 libcoreclr.dylib`RunMain(pFD=<unavailable>, numSkipArgs=<unavailable>, piRetVal=<unavailable>, stringArgs=<unavailable>) at assembly.cpp:1551 [opt]
    frame #32: 0x0000000102bf24fc libcoreclr.dylib`Assembly::ExecuteMainMethod(this=<unavailable>, stringArgs=0x000000016d42ae08, waitForOtherThreads=YES) at assembly.cpp:1661:18 [opt]
    frame #33: 0x0000000102c476c4 libcoreclr.dylib`CorHost2::ExecuteAssembly(this=<unavailable>, dwAppDomainId=<unavailable>, pwzAssemblyPath=<unavailable>, argc=<unavailable>, argv=0x000000015b219970, pReturnValue=0x000000016d42afcc) at corhost.cpp:384:39 [opt]
    frame #34: 0x0000000102bd3b04 libcoreclr.dylib`::coreclr_execute_assembly(hostHandle=0x000000015b1082f0, domainId=1, argc=6, argv=<unavailable>, managedAssemblyPath="/Users/arobins/src/runtime/artifacts/tests/coreclr/OSX.arm64.Checked/Tests/Core_Root/crossgen2/crossgen2.dll", exitCode=0x000000016d42afcc) at unixinterface.cpp:446:24 [opt]
    frame #35: 0x00000001029d8230 corerun`main [inlined] run(config=0x000000016d42af80) at corerun.cpp:358:18 [opt]
    frame #36: 0x00000001029d8124 corerun`main(argc=<unavailable>, argv=<unavailable>) at corerun.cpp:509 [opt]
    frame #37: 0x0000000192d19f34 libdyld.dylib`start + 4
(lldb)

@AaronRobinsonMSFT
Copy link
Member Author

/cc @trylek @davidwrighton

@jkotas
Copy link
Member

jkotas commented Mar 31, 2021

cc @sdmaclea

This was referenced Apr 1, 2021
Merged
Merged
@trylek trylek removed the untriaged New issue has not been triaged by the area owner label Apr 11, 2021
@trylek trylek added this to the 6.0.0 milestone Apr 11, 2021
@VincentBu
Copy link
Contributor

Failed again in runtime 20210515.30

Failed test:

CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open

- readytorun/determinism/crossgen2determinism/crossgen2determinism.sh

Error message:

Unhandled exception. System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
---> System.NullReferenceException: Object reference not set to an instance of an object.
at ILCompiler.Sorting.Implementation.MergeSortCore5.ParallelSort(TDataStructure arrayToSort, Int32 index, Int32 length, TComparer comparer) in ILCompiler.DependencyAnalysisFramework.dll:token 0x600001e+0x14 --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) in System.Private.CoreLib.dll:token 0x6002cae+0x21 at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) in System.Private.CoreLib.dll:token 0x6002cca+0x3f at System.Threading.Tasks.Task.Wait() in System.Private.CoreLib.dll:token 0x6002cc6+0x0 at ILCompiler.Sorting.Implementation.MergeSortCore5.ParallelSortApi(TDataStructure arrayToSort, TComparer comparer) in ILCompiler.DependencyAnalysisFramework.dll:token 0x600001d+0x33
at ILCompiler.MergeSortApi.MergeSort[T](List1 listToSort, IComparer1 comparer) in ILCompiler.DependencyAnalysisFramework.dll:token 0x600000c+0x0
at ILCompiler.DependencyAnalysis.ArrayOfEmbeddedDataNode1.GetData(NodeFactory factory, Boolean relocsOnly) in ILCompiler.ReadyToRun.dll:token 0x600032b+0x18 at ILCompiler.DependencyAnalysis.ObjectNode.GetStaticDependencies(NodeFactory factory) in ILCompiler.ReadyToRun.dll:token 0x60002cd+0x8 at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.GetStaticDependenciesImpl(DependencyNodeCore1 node) in ILCompiler.DependencyAnalysisFramework.dll:token 0x6000038+0x0 at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.GetStaticDependencies(DependencyNodeCore1 node) in ILCompiler.DependencyAnalysisFramework.dll:token 0x6000039+0x8 at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ProcessMarkStack() in ILCompiler.DependencyAnalysisFramework.dll:token 0x600003a+0x37
at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ComputeMarkedNodes() in ILCompiler.DependencyAnalysisFramework.dll:token 0x600003b+0x19 at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile) in ILCompiler.ReadyToRun.dll:token 0x600012a+0x0 at ILCompiler.Program.RunSingleCompilation(Dictionary2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary2 unrootedInputFilePaths, HashSet1 versionBubbleModulesHash, CompilerTypeSystemContext typeSystemContext) in crossgen2.dll:token 0x600015a+0x5f6
at ILCompiler.Program.Run(String[] args) in crossgen2.dll:token 0x6000159+0x52e
at ILCompiler.Program.Main(String[] args) in crossgen2.dll:token 0x6000161+0x5
task_for_pid(43665) FAILED 5 (os/kern) failure
/private/tmp/helix/working/AF9D094C/w/A0FD08F7/e/readytorun/determinism/crossgen2determinism/crossgen2determinism.sh: line 144: 43665 Abort trap: 6           (core dumped) $CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll --map -r:$CORE_ROOT/*.dll -r:../../crossgen2/crossgen2smoke/helperdll.dll -r:../../crossgen2/crossgen2smoke/helperildll.dll -o:crossgen2smoke2.ildll ../../crossgen2/crossgen2smoke/crossgen2smoke.dll
Unhandled exception. System.IO.FileNotFoundException: Could not find file '/private/tmp/helix/working/AF9D094C/w/A0FD08F7/e/readytorun/determinism/crossgen2determinism/crossgen2smoke2.ildll'.
File name: '/private/tmp/helix/working/AF9D094C/w/A0FD08F7/e/readytorun/determinism/crossgen2determinism/crossgen2smoke2.ildll'
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func2 errorRewriter) in System.Private.CoreLib.dll:token 0x600000a+0x36 at Interop.CheckIo(Error error, String path, Boolean isDirectory, Func2 errorRewriter) in System.Private.CoreLib.dll:token 0x600000b+0x3
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode) in System.Private.CoreLib.dll:token 0x6


Stack trace
   at readytorun_determinism._crossgen2determinism_crossgen2determinism_._crossgen2determinism_crossgen2determinism_sh()

@VincentBu
Copy link
Contributor

Historical failures of this test:

Queued OS Arch Pipeline
2021-05-16T08:54:00.372Z osx.1100.arm64.open arm64 runtime-coreclr crossgen2 outerloop Checked
2021-05-16T07:28:00.434Z osx.1100.arm64.open arm64 runtime-coreclr crossgen2 Checked
2021-05-16T06:51:33.847Z osx.1100.arm64.open arm64 runtime-coreclr r2r Checked
2021-05-15T07:20:04.992Z osx.1100.arm64.open arm64 runtime-coreclr crossgen2 Checked
2021-05-15T06:47:29.129Z osx.1100.arm64.open arm64 runtime-coreclr r2r Checked
2021-05-14T07:12:00.982Z osx.1100.arm64.open arm64 runtime-coreclr crossgen2 Checked
2021-05-14T06:53:05.494Z osx.1100.arm64.open arm64 runtime-coreclr r2r Checked

@tannergooding tannergooding mentioned this issue May 17, 2021
Merged
@hoyosjs
Copy link
Member

hoyosjs commented May 20, 2021

Updating this to be a tracking issue as it came up as a hot issue in CI Council. cc: @agocke @trylek @aik-jahoda

@hoyosjs hoyosjs added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 20, 2021
@sdmaclea
Copy link
Contributor

We should probably disable this test until it is fixed.

@sdmaclea
Copy link
Contributor

hot issue in CI Council

/cc @janvorli @mangod9

@mangod9
Copy link
Member

mangod9 commented May 20, 2021

yeah should be ok to disable it just for macOS arm64

@runfoapp runfoapp bot mentioned this issue May 20, 2021
Closed
@hoyosjs hoyosjs mentioned this issue May 20, 2021
Merged
@trylek trylek mentioned this issue May 20, 2021
Closed
@mangod9
Copy link
Member

mangod9 commented Jul 26, 2021

This should be fixed once the repo moves to use preview6 sdk.

@mangod9
Copy link
Member

mangod9 commented Aug 2, 2021

I dont see this exclusion anymore in issues.targets. @trylek could you please verify if its excluded in any other way?

@mangod9 mangod9 closed this as completed Aug 2, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Sep 1, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 area-crossgen2-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' os-macos-bigsur (macOS11)
Projects
None yet
Milestone
6.0.0
Development

No branches or pull requests
7 participants
@jkotas @trylek @hoyosjs @sdmaclea @AaronRobinsonMSFT @VincentBu @mangod9