Build passes when spek fail to discover tests

[jcornaz]

Hi,

When Spek fails to discover tests, it let the build be successful.

We observed the following log when running from gradle with --debug*:

*Note that without using --debug we were completely blind to it. The only indication we had is the low number of executed tests (which are only Junit5 tests running successfully).

10:20:40.561 [DEBUG] [TestEventLogger] Gradle Test Executor 2 STANDARD_ERROR
10:20:40.561 [DEBUG] [TestEventLogger]     Oct 22, 2019 10:20:40 AM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
10:20:40.561 [DEBUG] [TestEventLogger]     WARNING: TestEngine with ID 'spek2' failed to discover tests
10:20:40.561 [DEBUG] [TestEventLogger]     java.lang.AssertionError: 's1' can not be accessed in this context.
10:20:40.562 [DEBUG] [TestEventLogger]     	at org.spekframework.spek2.runtime.lifecycle.MemoizedValueAdapter.get(MemoizedValueAdapter.kt:33)
10:20:40.562 [DEBUG] [TestEventLogger]     	at org.spekframework.spek2.runtime.lifecycle.MemoizedValueAdapter.getValue(MemoizedValueAdapter.kt:22)
10:20:40.562 [DEBUG] [TestEventLogger]     	at com.mycompany.MySpec$1.invoke(MySpec.kt:91)
10:20:40.562 [DEBUG] [TestEventLogger]     	at com.mycompany.MySpec$1.invoke(MySpec.kt:37)
10:20:40.562 [DEBUG] [TestEventLogger]     	at org.spekframework.spek2.runtime.SpekRuntime.resolveSpec(SpekRuntime.kt:49)
10:20:40.562 [DEBUG] [TestEventLogger]     	at org.spekframework.spek2.runtime.SpekRuntime.discover(SpekRuntime.kt:23)
10:20:40.562 [DEBUG] [TestEventLogger]     	at org.spekframework.spek2.junit.SpekTestEngine.discover(SpekTestEngine.kt:92)
10:20:40.562 [DEBUG] [TestEventLogger]     	at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:168)
10:20:40.562 [DEBUG] [TestEventLogger]     	at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:155)
10:20:40.563 [DEBUG] [TestEventLogger]     	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
10:20:40.563 [DEBUG] [TestEventLogger]     	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:102)
10:20:40.564 [DEBUG] [TestEventLogger]     	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:82)
10:20:40.564 [DEBUG] [TestEventLogger]     	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:78)
10:20:40.564 [DEBUG] [TestEventLogger]     	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
10:20:40.564 [DEBUG] [TestEventLogger]     	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10:20:40.564 [DEBUG] [TestEventLogger]     	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
10:20:40.564 [DEBUG] [TestEventLogger]     	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
10:20:40.564 [DEBUG] [TestEventLogger]     	at java.lang.reflect.Method.invoke(Method.java:498)
10:20:40.564 [DEBUG] [TestEventLogger]     	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
10:20:40.564 [DEBUG] [TestEventLogger]     	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
10:20:40.564 [DEBUG] [TestEventLogger]     	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
10:20:40.565 [DEBUG] [TestEventLogger]     	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
10:20:40.565 [DEBUG] [TestEventLogger]     	at com.sun.proxy.$Proxy2.stop(Unknown Source)
10:20:40.565 [DEBUG] [TestEventLogger]     	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:132)
10:20:40.565 [DEBUG] [TestEventLogger]     	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10:20:40.565 [DEBUG] [TestEventLogger]     	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
10:20:40.565 [DEBUG] [TestEventLogger]     	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
10:20:40.565 [DEBUG] [TestEventLogger]     	at java.lang.reflect.Method.invoke(Method.java:498)
10:20:40.565 [DEBUG] [TestEventLogger]     	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
10:20:40.566 [DEBUG] [TestEventLogger]     	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
10:20:40.566 [DEBUG] [TestEventLogger]     	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
10:20:40.566 [DEBUG] [TestEventLogger]     	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
10:20:40.566 [DEBUG] [TestEventLogger]     	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:412)
10:20:40.566 [DEBUG] [TestEventLogger]     	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
10:20:40.566 [DEBUG] [TestEventLogger]     	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
10:20:40.566 [DEBUG] [TestEventLogger]     	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
10:20:40.566 [DEBUG] [TestEventLogger]     	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
10:20:40.566 [DEBUG] [TestEventLogger]     	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
10:20:40.566 [DEBUG] [TestEventLogger]     	at java.lang.Thread.run(Thread.java:748)

We'll continue investigate to see if solutions proposed in #801 helps us fix the problem.

But the build should definitely fail. Not pass.

[raniejade]

This should be fixed by using the latest version of JUnit Platform. I'm quite busy at work at the moment, I'll probably have some time to take a look during the weekend.

[ursjoss]

@raniejade We are already using junit5 version 5.5.2, which seems to be the latest stable version available.

I just tried upgrading to 5.6.0-M1 but with the same outcome as with 5.5.2, i.e. the build succeeding despite the spek tests not being discovered and therefore not executed.

So, unfortunately, this seems to not resolve the issue.

[raniejade]

This is the relevant issue on JUnit Platform junit-team/junit5#1909, looks like it should be fix in 1.6.0 but based on this junit-team/junit5#2028 tooling needs to update first (gradle, maven, etc...) to take advantage of it? (I'm not really sure)

[ursjoss]

@raniejade Yes, I think this is related and relevant, thanks for the links.

[ursjoss]

The relevant part from the junit 5.6.0-M1 release notes:

The Launcher now propagates errors during test discovery by default instead of only logging and thereby potentially hiding them. In order to restore the old, lenient behavior, you can set the junit.platform.discovery.listener.default configuration parameter to logging.

[ursjoss]

I can confirm that junit5 in version 6.6.0-M1 resolves the issue when triggering the tests via IntelliJ (not delegating the test run to gradle). With latest gradle version 5.6.3 we don't have the upgraded tooling yet, so the build does not fail yet.

As documented in junit-team/junit5#2028

[raniejade]

I'll move this to 2.0.10, 1.6.0 is not fully released yet but in the meantime I'll update the version used by spek to 1.5.2.