App completes successfully but XHarness tool fails

[premun]

Egor has built me an app from these tests:

    public class DummyTests
    {
        [Fact]
        public void PassingTest()
        {
            Assert.Equal(1, 1);
        }

        [Fact]
        public void FailingTest()
        {
            Assert.Equal(1, 2);
        }

        [Fact(Skip = "Skipped reason")]
        public void SkippedTest()
        {
            Assert.Equal(1, 1);
        }
    }

The app runs fine, it terminates but we fail to parse the results:

Could not parse xml result file: System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
   at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at Microsoft.DotNet.XHarness.iOS.Shared.XmlResultParser.ParseNUnitXml(StreamReader stream, StreamWriter writer) in /Users/prvysoky/Desktop/xharness/src/Microsoft.DotNet.XHarness.iOS.Shared/XmlResultParser.cs:line 188
   at Microsoft.DotNet.XHarness.iOS.Shared.XmlResultParser.GenerateHumanReadableResults(String source, String destination, XmlResultJargon xmlType) in /Users/prvysoky/Desktop/xharness/src/Microsoft.DotNet.XHarness.iOS.Shared/XmlResultParser.cs:line 370
   at Microsoft.DotNet.XHarness.iOS.Shared.TestReporter.ParseResultFile(AppBundleInformation appInfo, String test_log_path, Boolean timed_out) in /Users/prvysoky/Desktop/xharness/src/Microsoft.DotNet.XHarness.iOS.Shared/TestReporter.cs:line 374
File data is:
##########
01:02:43.8921440 [Local Date/Time:     04/27/2020 01:02:43]
[Remote Address:       127.0.0.1:52126]
<?xml version="1.0" encoding="utf-8"?>
<test-results name="Test results" errors="0" inconclusive="0" ignored="0" invalid="0" not-run="0" date="2020-04-27" time="01:02:45" total="3" failures="1" skipped="1">
  <environment os-version="unknown" platform="unknown" cwd="unknown" machine-name="unknown" user="unknown" user-domain="unknown" nunit-version="xUnit.net 2.4.1.0" clr-version="64-bit .NET  [collection-per-class, non-parallel]" />
  <culture-info current-culture="unknown" current-uiculture="unknown" />
  <test-suite type="Assemblies" name="xUnit.net Tests" executed="True" success="False" result="Failure" time="0.594">
    <results>
      <test-suite type="Assembly" executed="True" name="System.Numerics.Vectors.Tests.dll" result="Failure" success="False" time="0.594">
        <results>
          <test-suite type="TestCollection" executed="True" name="Test collection for DummyTestProject.DummyTests" result="Failure" success="False" time="0.191">
            <results>
              <test-case name="DummyTestProject.DummyTests.SkippedTest" executed="False" result="Skipped" time="0">
                <reason>
                  <message>Skipped reason</message>
                </reason>
              </test-case>
              <test-case name="DummyTestProject.DummyTests.FailingTest" executed="True" result="Failure" success="False" time="0.1253504">
                <failure>
                  <message>Assert.Equal() Failure\nExpected: 1\nActual:   2</message>
                  <stack-trace>   at DummyTestProject.DummyTests.FailingTest() in /Users/egorbo/prj/runtime-3/src/libraries/System.Numerics.Vectors/tests/Vector2Tests.cs:line 16
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /Users/egorbo/prj/runtime-3/src/mono/netcore/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 359</stack-trace>
                </failure>
              </test-case>
              <test-case name="DummyTestProject.DummyTests.PassingTest" executed="True" result="Success" success="True" time="0.0657749" />
            </results>
          </test-suite>
        </results>
      </test-suite>
    </results>
  </test-suite>
</test-results>

The header makes it an invalid XML:

01:02:43.8921440 [Local Date/Time:     04/27/2020 01:02:43]
[Remote Address:       127.0.0.1:52126]

Also number of tests ran doesn't correlate with the code:

System.Numerics.Vectors.Tests[35220:10197811] Tests run: 3 Passed: 1 Inconclusive: 0 Failed: 1 Ignored: 0

The skipped test is mentioned in the logs though:

System.Numerics.Vectors.Tests[35220:10196063]   [IGNORED] DummyTestProject.DummyTests.SkippedTest

Problems to be solved

• Remove invalid header from results XML - Clean-up SimpleListeners and fix invalid XML header #93
• Result XML is called nunit-test-ios-simulator-64-20200427_010223.xml - there's no nunit involved [iOS] Ensure that we do generate xml when we call the CLI. #102
• XHarness hangs after failing to parse the XML and only the timeout kills it later - it should quit immediately - [iOS] Ensure that we do generate xml when we call the CLI. #102
• Skipped test is not accounted for - [TestRunner] Fix the number of ignored tests reported #97
• Output of the app is printed on the console of the XHarness tool and shouldn't be - App stdout leaks to XHarness stderr #108

[premun]

I was able to resolve the first problem with the XML header: #93

[premun]

The reason we are generating NUnit results always is here:

xharness/src/Microsoft.DotNet.XHarness.Tests.Runners/iOSApplicationEntryPoint.cs

Lines 38 to 48 in bc6dca4

	TestRunner.Jargon jargon = Core.TestRunner.Jargon.NUnitV3;
	switch (options.XmlVersion)
	{
	case XmlVersion.NUnitV2:
	jargon = Core.TestRunner.Jargon.NUnitV2;
	break;
	case XmlVersion.NUnitV3:
	default: // nunitv3 gives os the most amount of possible details
	jargon = Core.TestRunner.Jargon.NUnitV3;
	break;
	}

@mandel-macaque currently the TestRunner can never output xUnit results ^^

So the nunit- prefix is OK actually.

[premun]

The skipped test problem might be an issue with the wrong XML result jargon

[premun]

The reason for not recognizing that tests finished successfully is here:

xharness/src/Microsoft.DotNet.XHarness.iOS.Shared/XmlResultParser.cs

Lines 54 to 58 in bc6dca4

	if (line.Contains("test-results"))
	{ // first element of the NUnitV3 test collection
	type = XmlResultJargon.NUnitV2;
	return true;
	}

But I am a bit confused. The above corresponds with what I found here:
NUnit v2: https://nunit.org/files/testresult_25.txt
NUnit v3: https://nunit.org/files/testresult_30.txt

The XML returned from Egor's app should also be NUnit v3, but somehow it is v2 and we only succeed if we try to parse it as NUnit v3... This is weird

[premun]

Oops, got auto-closed

[mandel-macaque]

@premun the test reporter can use xunit, it is supported, we need to change the default and allow to pass the jargon. Found the issue will create a pr for this and assign the fix to #77

[mandel-macaque]

The following PR should fix a number of the issues: #102

• Do generate xUnit
• Do write a clean version of the xml result in the output dir.

[premun]

So I guess after #102 we only need to find out how stuff gets to Console.Out and I just failed to figure it out. I will give it another shot tomorrow.