System.Threading.Tasks' RunStackGuardTests AV

[tmat]

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/ubuntu_debug_prtest/1120/

All I see is
"1 test(s) failed"

but I can't find which test failed.

[stephentoub]

RunStackGuardTests

10:18:25 Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
10:18:25    at System.Runtime.CompilerServices.TaskAwaiter.OnCompletedInternal(Task task, Action continuation, Boolean continueOnCapturedContext, Boolean flowExecutionContext)
10:18:25    at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.UnsafeOnCompleted(Action continuation)
10:18:25    at System.Threading.Tasks.TaskExtensions.TransferAsynchronously[TResult,TInner](TaskCompletionSource`1 completionSource, Task`1 outer)
10:18:25    at System.Threading.Tasks.TaskExtensions.Unwrap[TResult](Task`1 task)
10:18:25    at System.Threading.Tasks.Tests.UnwrapTests.<>c__DisplayClass15_0.<RunStackGuardTests>b__0(Int32 count)
10:18:25    at System.Threading.Tasks.Tests.UnwrapTests.<>c__DisplayClass15_1.<RunStackGuardTests>b__1()
10:18:25    at System.Threading.Tasks.Task`1.InnerInvoke()
10:18:25    at System.Threading.Tasks.Task.Execute()
10:18:25    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
10:18:25    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
10:18:25    at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
10:18:25    at System.Threading.ThreadPoolWorkQueue.Dispatch()

[tmat]

Why is this not reported as a test failure with a test name? It's really hard to find in the gobs of logging text.

[stephentoub]

Why is this not reported as a test failure with a test name?

Because it went unhandled on a thread pool thread. xunit doesn't know what test it's from.

[tmat]

Interesting. Would it at least be possible to change Jenkins to highlight "Unhandled exception" in red in the log (as it does with other failures)?

[stephentoub]

Would it at least be possible to change Jenkins to highlight "Unhandled exception" in red

I don't know. @mmitche?

I usually just search for a few phrases: "unhandled", "error " (with a space at the end), "seg", "abort", "failed: 1", "failed: 2", ... etc.

[tmat]

Yeah, I tried failed :1 etc. The problem is that a contributor (like me) who's not dealing with these failures every day has no idea what to search for.

[burtonrodman]

assuming it acts the same as the full CLR, System.AccessViolationException is a Process Corrupted State Exception and can only be handled by a method decorated with the HandlesProcessCorruptedStateException. This is definitely outside the box for a unit test framework.

[tmat]

@burtonrodman I don't think it is that hard to deal with. The test runner can run tests in a separate process and keep a list of currently executing tests in that process. If the process dies unexpectedly (stack overflow, AV, etc.) it can report that one of the N tests that were running in parallel in the process failed and print out output of the process. It can then skip the tests that might have caused the failure and run rest of the tests.

[burtonrodman]

I wasn't suggesting that it's hard to deal with, but that a unit test framework usually shouldn't have to. in fact, I wrote a test that triggers and gracefully handles an access violation (on Windows). an access violation is likely a framework bug. the unit tests have done their job in triggering the issue. all that we're missing is convenient reporting of the source of a 0.01% case.

[tmat]

@burtonrodman AV is not necessarily a framework bug if the library being tested uses unsafe code or interop with native code. Also there are other bugs that occur in purely managed code such as stack overflow or out of memory. A robust unit testing framework should be able to handle these.

[burtonrodman]

agreed. I got there after I thought more about it.

[janvorli]

I've scanned all corefx jobs from the one reported till today and found no other instances of this error. I have tried to repro it locally, running the test in a loop 100 times without any repro. I have tried to lower the stack size down to 180kB, running the test in a loop again. Still no repro. Going with stack to 170kB results in stack overflow, so I was really at the edge.

[stephentoub]

This looks like it was a transient issue, with only one citing as part of a PR job. I'm going to close it.