[2.0.0-SNAPSHOT] - Junit reports seem to not be genereated

[jaredsburrows]

All tests run but junit reports are not generated.

1.7.1

2.0.0-SNAPSHOT

[jaredsburrows]

This is really bad because we do not know which tests would have failed on the CI.

[idulkin]

I'm encountering this issue as well. I believe it's related to test orchestrator. Because each test runs as its own instrumentation, Spoon is creating an xml report after each test, and overwriting the previous report for that device. So we only get the output of the last test to run on each device.

[jaredsburrows]

@idulkin I do believe you are right, it seems to only save the "last" test because it keeps overriding the junit xml report, so there is only 1 test listed.

How can we force spoon to not run like this? Maybe using this new option, we can run all the tests together again? 53a63fe#diff-2a33694ea19ca0cbba8207fda7dcfbd6R73

[jaredsburrows]

@idulkin Can you try using --single-instrumentation-call?

[idulkin]

@jaredsburrows I was just about to try that! I'm concerned about a single instrumentation losing Test Orchestrator's ability to isolate crashes, but let's see what happens.

[jaredsburrows]

Yes, it seems to bring the junit reports back but the test are no longer isolated. This means that the CI cannot report back which tests failed unless you fail the build on the first failed test.

[idulkin]

I think the real problem isn't with Spoon, but with the XmlTestRunListener from ddmlib. When each instrumentation finishes, testRunEnded is called and a report is generated. It seems like we would need a completely different run listener to handle multiple instrumentations per test suite.

[edamyanov]

I think this PR 519 fixes this problem.

[tobi512]

@jaredsburrows The fix of Emanuil looks good, would be great to have a quick merge here, since the broken reports currently block us a little bit...

[jaredsburrows]

@tobi512 It needs to be merged here first. I am not an owner or collaborator of this repo.

[edenman]

should be fixed on master now, sorry for the delay.