Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[mono][loader] CI test failure "appdomain.c:1591, condition !ass->image->references [i]' not met" #2305

Closed
lambdageek opened this issue Jan 28, 2020 · 6 comments · Fixed by #32622
Closed

[mono][loader] CI test failure "appdomain.c:1591, condition !ass->image->references [i]' not met" #2305

lambdageek opened this issue Jan 28, 2020 · 6 comments · Fixed by #32622
Assignees
@CoffeeFlux
Labels
area-AssemblyLoader-mono runtime-mono specific to the Mono runtime

Comments

@lambdageek
Copy link
Member

lambdageek commented Jan 28, 2020 •
edited
Loading

Full assertion:

* Assertion at /Users/runner/runners/2.164.6/work/1/s/src/mono/mono/metadata/appdomain.c:1591, condition `!ass->image->references [i] not met', function:add_assembly_to_alc, Did not expect reference 0 of /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/System.Net.Http.dll to be resolved

Happened on an OSX CI lane. Failure log here The full test log https://dev.azure.com/dnceng/public/_build/results?buildId=499602&view=logs&j=c6f8dc49-92a1-5760-c098-ba97b8142bfb&t=22b0078b-0469-5ba6-8725-2121fdbae049&l=50

Stack trace:

	0x107901602 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_dump_native_crash_info
	0x1078a3b55 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_handle_native_crash
	0x107900c1b - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : sigabrt_signal_handler
	0x7fff6249ff5a - /usr/lib/system/libsystem_platform.dylib : _sigtramp
	0x0 - Unknown
	0x7fff6223d1ae - /usr/lib/system/libsystem_c.dylib : abort
	0x107abe1c7 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : monoeg_assert_abort
	0x107a9ed67 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_log_write_logfile
	0x107abe655 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : monoeg_g_logv_nofree
	0x107abe7df - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : monoeg_assertion_message
	0x10795dd07 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_domain_fire_assembly_load
	0x1079679cc - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_assembly_invoke_load_hook_internal
	0x107968d58 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_assembly_request_load_from
	0x1079686c0 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_assembly_request_open
	0x1079021ce - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_core_preload_hook
	0x10796b591 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : invoke_assembly_preload_hook
	0x1079677a8 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_assembly_request_byname
	0x10796721e - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_assembly_load_reference
	0x10796d2eb - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_class_from_typeref_checked
	0x1079acac2 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : method_from_memberref
	0x1079aae00 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_get_method_checked
	0x10783c7e5 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mini_get_method
	0x1078188b5 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_method_to_ir
	0x1077fdae9 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mini_method_compile
	0x1077fffdc - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_jit_compile_method_inner
	0x107803de2 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_jit_compile_method_with_opt
	0x1078a667f - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : common_call_trampoline
	0x1078a7390 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_vcall_trampoline
	0x107c92293 - Unknown
	0x10c9db25a - Unknown
	0x10cb02c53 - Unknown
	0x10c9db25a - Unknown
	0x10cad0ab3 - Unknown
	0x10c9db25a - Unknown
	0x10cae7f53 - Unknown
	0x10780833e - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_jit_runtime_invoke
	0x1079dd888 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_runtime_invoke_checked
	0x1079e5292 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : mono_runtime_try_invoke_array
	0x10798f634 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : ves_icall_InternalInvoke
	0x10799b717 - /private/tmp/helix/working/AF4D0979/p/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.dylib : ves_icall_InternalInvoke_raw
	0x10b642509 - Unknown
	0x10b641689 - Unknown
	0x10c9da28b - Unknown
	0x10c9da163 - Unknown
	0x10c9d9a3b - Unknown
	0x10c9d98bb - Unknown
	0x10c9d9043 - Unknown
	0x10c9d8ed3 - Unknown
	0x10c9d7953 - Unknown
	0x10c9d7823 - Unknown
	0x10c9d1b6b - Unknown
	0x10c9d17e3 - Unknown
	0x10c9ce253 - Unknown
	0x10c9ce08b - Unknown
	0x10c9ca681 - Unknown
	0x10c9c8753 - Unknown
	0x10c9c810b - Unknown
	0x10c9c5b5b - Unknown
	0x10c9c56b3 - Unknown
	0x10c9c281b - Unknown
	0x10c9c2413 - Unknown
	0x10c9bcb6b - Unknown
	0x10c9bc663 - Unknown
	0x10c9bc533 - Unknown
	0x10c9bb84e - Unknown
	0x10c9bafeb - Unknown
	0x10c9baeb3 - Unknown
	0x10c9b9fbb - Unknown
	0x10c9b9e93 - Unknown
	0x10c9b14d3 - Unknown
	0x10c9b07eb - Unknown
	0x10c9b06a3 - Unknown
	0x10c9aca43 - Unknown
	0x10c9ac913 - Unknown
	0x10c9ab938 - Unknown
	0x10c9ab26b - Unknown
	0x10c9ab133 - Unknown
	0x10c985db3 - Unknown
	0x10c985b43 - Unknown
	0x10c984bc8 - Unknown
	0x10bb83313 - Unknown
	0x10c9845c3 - Unknown
	0x10c97d033 - Unknown
	0x10bb83b16 - Unknown
	0x10bb83313 - Unknown
	0x10bb7aeeb - Unknown
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Jan 28, 2020
@lambdageek lambdageek removed the untriaged New issue has not been triaged by the area owner label Jan 28, 2020
@safern
Copy link
Member

safern commented Jan 31, 2020

Failed again, now with: System.IO.Compression.ZipFile.dll.

Assertion at /Users/runner/runners/2.164.6/work/1/s/src/mono/mono/metadata/appdomain.c:1591, condition `!ass->image->references [i]' not met, function:add_assembly_to_alc, Did not expect reference 0 of /private/tmp/helix/working/B08109A7/p/shared/Microsoft.NETCore.App/5.0.0/System.IO.Compression.ZipFile.dll to be resolved

Here is the core dump: https://helix.dot.net/api/2019-06-17/jobs/0d54bf8d-814b-48c7-bd00-7a690a8e3c6e/workitems/System.IO.Compression.ZipFile.Tests/files/core.81383

Failed in: https://dev.azure.com/dnceng/public/_build/results?buildId=502823&view=logs&jobId=c6f8dc49-92a1-5760-c098-ba97b8142bfb
Console log: https://helix.dot.net/api/2019-06-17/jobs/0d54bf8d-814b-48c7-bd00-7a690a8e3c6e/workitems/System.IO.Compression.ZipFile.Tests/console

@safern safern mentioned this issue Jan 31, 2020
Merged
@CoffeeFlux CoffeeFlux added the runtime-mono specific to the Mono runtime label Feb 11, 2020
@ViktorHofer
Copy link
Member

ViktorHofer commented Feb 13, 2020 •
edited
Loading

Failed again: https://dnceng.visualstudio.com/public/_build/results?buildId=519073&view=ms.vss-test-web.build-test-results-tab&runId=16411134&resultId=172839&paneView=attachments

Configuration: netcoreapp5.0-OSX-Release-x64-Mono_release-OSX.1013.Amd64.Open

@CoffeeFlux what are planned next steps? Asking as we want to get our rolling builds green :)

@CoffeeFlux
Copy link
Contributor

@ViktorHofer I took a look at it last week and I'll sort it out properly this sprint, probably next week. Was dealing with some other stuff, sorry for the delay.

@ViktorHofer
Copy link
Member

Thanks a lot :)

@ViktorHofer
Copy link
Member

Happened again: https://dnceng.visualstudio.com/public/_build/results?buildId=525399&view=ms.vss-test-web.build-test-results-tab&runId=16622562&resultId=168705&paneView=attachments

@ViktorHofer
Copy link
Member

Another hit: https://dev.azure.com/dnceng/public/_build/results?buildId=536066&view=ms.vss-test-web.build-test-results-tab&runId=17016570&resultId=167358&paneView=attachments

@ViktorHofer ViktorHofer mentioned this issue Feb 25, 2020
Merged
@CoffeeFlux CoffeeFlux mentioned this issue Feb 27, 2020
Merged
monojenkins pushed a commit to monojenkins/mono that referenced this issue Mar 2, 2020
@CoffeeFlux
[loader] Clean up locking around adding assemblies to domain/ALC
7fc41aa 
This cleans up three components of assembly loading that could potentially cause races/crashes.

First, both `add_assemblies_to_domain` and `add_assembly_to_alc` now ignore resolved references on netcore. We were seeing a crash on CI related to an assertion in here that resulted from two threads using the same MonoImage simultaneously as one is being loaded in with LoadFile, and this should solve that. Fixes dotnet/runtime#2305

Second, we take both the domain and ALC loaded assemblies locks simultaneously around the assembly being added to both lists, ensuring they stay in sync.

Finally, MonoImage now takes the image lock when writing to references instead of using the global `assemblies_mutex` in assembly.c for all images. This potentially helps reduce contention. Work here is not complete, but for this PR it's good enough. See dotnet/runtime#32889
CoffeeFlux added a commit that referenced this issue Mar 4, 2020
@CoffeeFlux
[loader] Clean up locking around adding assemblies to domain/ALC (#32622
4142643 
)

This cleans up three components of assembly loading that could potentially cause races/crashes.

First, both `add_assemblies_to_domain` and `add_assembly_to_alc` now ignore resolved references on netcore. We were seeing a crash on CI related to an assertion in here that resulted from two threads using the same MonoImage simultaneously as one is being loaded in with LoadFile, and this should solve that. Fixes #2305 

Second, we take both the domain and ALC loaded assemblies locks simultaneously around the assembly being added to both lists, ensuring they stay in sync.

Finally, MonoImage now takes the image lock when writing to references instead of using the global `assemblies_mutex` in assembly.c for all images. This potentially helps reduce contention. Work here is not complete, but for this PR it's good enough. See #32889
CoffeeFlux added a commit to mono/mono that referenced this issue Mar 4, 2020
@CoffeeFlux
[loader] Clean up locking around adding assemblies to domain/ALC (#18967
a5f8905 
)

This cleans up three components of assembly loading that could potentially cause races/crashes.

First, both `add_assemblies_to_domain` and `add_assembly_to_alc` now ignore resolved references on netcore. We were seeing a crash on CI related to an assertion in here that resulted from two threads using the same MonoImage simultaneously as one is being loaded in with LoadFile, and this should solve that. Fixes dotnet/runtime#2305

Second, we take both the domain and ALC loaded assemblies locks simultaneously around the assembly being added to both lists, ensuring they stay in sync.

Finally, MonoImage now takes the image lock when writing to references instead of using the global `assemblies_mutex` in assembly.c for all images. This potentially helps reduce contention. Work here is not complete, but for this PR it's good enough. See dotnet/runtime#32889

Co-authored-by: Ryan Lucia <[email protected]>
@ghost ghost locked as resolved and limited conversation to collaborators Dec 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@CoffeeFlux CoffeeFlux

Labels
area-AssemblyLoader-mono runtime-mono specific to the Mono runtime
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[loader] Clean up locking around adding assemblies to domain/ALC CoffeeFlux/runtime
5 participants
@lambdageek @CoffeeFlux @ViktorHofer @safern @Dotnet-GitSync-Bot