Non-failsafe printStackTrace in html reporter

[seregamorph]

Problem statement

The @Test that throws Exception which toString() method throws Exception, breaks maven html reporter. Even if this exception class is defined as expected and test should be success, as a result, maven just gives failure like:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  23.568 s (Wall Clock)
[INFO] Finished at: 2022-10-04T14:23:00+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project business-logic-core-impl: There are test failures.
[ERROR] 
[ERROR] Please refer to /Users/morph/dev/Projects/miro/server/server/business-logic-core/business-logic-core-impl/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] There was an error in the forked process
[ERROR] Cannot invoke "org.apache.commons.lang3.exception.ExceptionContext.getFormattedExceptionMessage(String)" because "this.exceptionContext" is null
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] Cannot invoke "org.apache.commons.lang3.exception.ExceptionContext.getFormattedExceptionMessage(String)" because "this.exceptionContext" is null
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:656)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:196)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:186)
[ERROR] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[ERROR] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[ERROR] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[ERROR] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[ERROR] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[ERROR] 	at java.base/java.lang.Thread.run(Thread.java:833)

and it's pretty challenging to find the root cause. The issue is only reproducible in maven (not if test is run in IDE), so maven forked process should be executed in debug mode: (-Dmaven.surefire.debug flag should be used). Further investigation helped to find this trace:

// ContextedRuntimeException.getMessage() throws NPE, because exception object is created via Mockito, not via constructor; as a result has wrong state and cannot be printed
	at org.apache.commons.lang3.exception.ContextedRuntimeException.getFormattedExceptionMessage(ContextedRuntimeException.java:254)
	at org.apache.commons.lang3.exception.ContextedRuntimeException.getMessage(ContextedRuntimeException.java:235)
	at java.base/java.lang.Throwable.getLocalizedMessage(Throwable.java:397)
	at java.base/java.lang.Throwable.toString(Throwable.java:496)
	at java.base/java.lang.String.valueOf(String.java:4218)
	at java.base/java.io.PrintWriter.println(PrintWriter.java:835)
	at java.base/java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:779)
	at java.base/java.lang.Throwable.printStackTrace(Throwable.java:670)
	at java.base/java.lang.Throwable.printStackTrace(Throwable.java:736)
	at org.testng.internal.Utils.buildStackTrace(Utils.java:325)
	at org.testng.internal.Utils.longStackTrace(Utils.java:308)
	at org.testng.reporters.TestHTMLReporter.generateTable(TestHTMLReporter.java:169)
	at org.testng.reporters.TestHTMLReporter.generateLog(TestHTMLReporter.java:359)
	at org.testng.reporters.TestHTMLReporter.onFinish(TestHTMLReporter.java:38)
	at org.testng.TestRunner.fireEvent(TestRunner.java:923)
	at org.testng.TestRunner.afterRun(TestRunner.java:887)
	at org.testng.TestRunner.run(TestRunner.java:599)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:377)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:371)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:332)
	at org.testng.SuiteRunner.run(SuiteRunner.java:276)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1212)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1134)
	at org.testng.TestNG.runSuites(TestNG.java:1063)
	at org.testng.TestNG.run(TestNG.java:1031)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Expected behavior

Either failover of the report or clear diagnostics with original stack trace

Actual behavior

Confusing derivative failure

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

See first comment

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[seregamorph]

Test that led to this failure:

@Test(expectedExceptions = [UserNotFoundException::class])
fun `add should not add user when it not exists`() {
    whenever(usersService.getUserOrThrow(workspace, userId)).thenThrow(UserNotFoundException::class.java)
    // internally calls usersService.getUserOrThrow
    userGroupsUsersService.add(request)
}

which was 🟢 in IDE, but failed in maven on HTML report. The reason:
UserNotFoundException extends org.apache.commons.lang3.exception.ContextedRuntimeException, ContextedRuntimeException has field exceptionContext that should be never null. But it's null when mockito constructs exception instance. As a result testng report formatter fails (and breaks maven build while tests are success) to print such exception, because it has this line:

@Override
public String getFormattedExceptionMessage(final String baseMessage) {
    // null in test runtime (thanks, Mockito!)
    return exceptionContext.getFormattedExceptionMessage(baseMessage);
}

For sure, it's a bad practice to have definitions like thenThrow(UserNotFoundException::class.java) in UTs, but proper diagnostics could save hour of investigation of root cause.

[seregamorph]

Small detail regarding how Mockito instantiates such exception object:

@Override
protected Throwable getThrowable() {
    Instantiator instantiator = Plugins.getInstantiatorProvider().getInstantiator(null);
    return instantiator.newInstance(throwableClass);
}