Improve error message if multiple "test" annotations are present on a method

[ctapobep]

For the following test two descriptors will be created -- one for @ParameterizedTest and another for @Test:

@ParameterizedTest
@ValueSource(strings = "blah")
@Test
void blah(String blah) {}

The parameterized test passes, while the other one fails because JUnit Juiter cannot resolve the parameter:

org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter [java.lang.String arg0] in executable [void io.qala.datagen.junit5.Junit5ParameterizedTest.blah(java.lang.String)].

I don't know if it's a bug or a feature, but because I didn't know about this I spent an hour debugging JUnit. I assume many more people could step on this, so would be great to make the logic more obvious. Couple of options:

• Ignore @Test if some other test descriptor is created
• Make error message clearer: ... This can happen e.g. if 2 different Test annotations are put on one method.
• Use mechanisms for creating Test Descriptors similar to how ParameterResolver works: boolean supportsTest(...)

Related Issues

• Introduce error handling mechanism for validation and discovery errors #242

[marcphilipp]

Agreed, there should definitely be a better error message.

[sbrannen]

Note that this is likely a more general topic involving all types of "test" annotations such as @Test, @TestTemplate, and @TestFactory.

Keep in mind that @RepeatedTest and @ParameterizedTest are specializations of @TestTemplate.

I have therefore changed this issue's title accordingly.

[akozlova]

BTW I've wrote inspections for IntelliJ to highlight such dbl tests.

[sbrannen]

BTW I've wrote inspections for IntelliJ to highlight such dbl tests.

Nice, @akozlova! 👍

[sbrannen]

@akozlova, in IntelliJ do you simply inspect for multiple annotations meta-annotated with @Testable?

Or do you check for individual "test" annotations?

[akozlova]

Now it's only check for predefined annotations, mainly due to the fact that IDEA knows nothing about custom engines and thus can't check them: who knows, probably some engine would require both parameterized and some testable annotation due to the fact that jupiter won't be available/filtered.

[sbrannen]

Now it's only check for predefined annotations, mainly due to the fact that IDEA knows nothing about custom engines and thus can't check them:

Makes sense!

who knows, probably some engine would require both parameterized and some testable annotation due to the fact that jupiter won't be available/filtered.

Yeah, I was pondering the exact same thing. 😉

[smoyer64]

I accidentally ended up with @Test and @TestTemplate on the same test method (converting from a non-parameterized test to a parameterized test) and it actually ran both. When complete, there were five tests provided by the parameterized annotation and one test by the normal test annotation. Note that the non-parameterized test failed due to lack of parameter resolution.

[sbrannen]

in progress

[sbrannen]

Update:

In my local branch, JUnit Jupiter now logs the following warning in such cases.

18:50:43.774 [main] WARN  org.junit.jupiter.engine.discovery.JavaElementsResolver - Possible configuration error: method [void org.junit.jupiter.engine.MultipleTestableAnnotationsTests.testAndRepeatedTest(org.junit.jupiter.api.RepetitionInfo)] resulted in multiple TestDescriptors [org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor, org.junit.jupiter.engine.descriptor.MethodTestDescriptor]. This is typically the result of annotating a method with multiple competing annotations such as @Test, @RepeatedTest, @ParameterizedTest, @TestFactory, etc.

How does that sound?

[sbrannen]

Note that, with the aforementioned log statement, the test would still fail in the same manner as described in this issue's description, but the warning would help to determine the cause.

[marcphilipp]

How does that sound?

SGTM! 👍

[sbrannen]

Resolved in master in commit 1ffeb82.

[yangyechi]

Good！