Tests failure was not detected

[greenEkatherine]

Test results were not correct for Android arm64 device leg, it shows that it passed w/ no failing tests

Build https://dev.azure.com/dnceng/public/_build/results?buildId=1037907&view=logs&j=c693b1a1-52cc-5241-304f-e4b4c2d5f73e&t=dc5b1a4a-d7d3-5c23-61cb-3de4ef061960

Example WorkItem -> https://helix.dot.net/api/jobs/505a2465-f7f2-458e-a802-ca6cb49efd0d/workitems/System.Security.Cryptography.Algorithms.Tests?api-version=2019-06-17

Xharness detects that instrumentation return code was not correct but the build was green after all.

[steveisok]

This is pretty serious as we may be getting a false sense of security on CI.

/cc @tkapin

[MattGal]

Looking at a bunch of logs, it's clear that the Azure Devops reporter isn't getting run any more. We've made a hard-coded expectation that 0 and 1 get coerced to 0 because 1= xunit failures. I am looking at the delta to try to figure out why.

[MattGal]

Logs age out, but at least as recently as 2021-02-25 16:23:17.6280000 UTC the reporter was not set up.

[MattGal]

Pretty sure I caused this and it's an easy fix, but am doing a quick experiment to prove this first.

[MattGal]

Verified experimentally https://github.com/dotnet/arcade/pull/7103/files fixes the reporting. This may be not the only change required, but it's definitely an important one.

[tkapin]

Thanks for the fix @MattGal. @greenEkatherine will check tomorrow if there are other places where this needs to be fixed yet. If you have any tips, please let us know.

[premun]

Issue hasn't manifested for some time. closing