Demonstrate migration path from JUnit 4 to JUnit 5

[sbrannen]

Overview

Based on discussions I've had with developers in the community, it is apparent that some people fear an upgrade from JUnit 4 to JUnit 5. Aside from the fact that it's totally feasible to run JUnit 4 tests alongside JUnit 5 tests, we should still attempt to migrate an existing JUnit 4 test suite just to see where the hurdles are.

Proposal

Demonstrate how to migrate from JUnit 4's ExpectedException Rule to a JUnit 5 Extension-based alternative.

Food for Thought

Can this be achieved with simple search-and-replace?

Can a JUnit 5 extension be implemented that honors JUnit 4's ExpectedException transparently, without the need to modify a single line within test methods -- for example, by removing @Rule from the ExpectedException field and adding @ExtendWith(SomeExpectedExceptionExtension.class)?

Related Issues

• Document compatibility with JUnit 4 TestRules #343

Related Blogs

• How to Replace Rules in JUnit 5

[philwebb]

The other one I use quite often is TemporaryFolder

[sormuras]

@philwebb Me, too! See junit-team/junit5-samples#4

[kcooney]

The lack of Rule support in JUnit5 might make it hard for large projects to incrementally migrate test classes, especially if they have custom Rules.

I wonder if we could have an optional jar that defines a simple extension point that can be used for JUnit5 and JUnit4. I find that most of our Rules only need the following:

• setUp()
• tearDown() - called only if setUp() called
• verify() - called only if the test method doesn't throw

If this interests anyone, I could write up a proposal.

[akozlova]

Hi Kevin,

I thought about IDE-side migration tool to ease the transition. Of cause it
doesn't make the jar useless.

BTW If you have 'rules' for such tool in mind, it would help me a lot.

Thanks,
Anna

On Fri, Mar 25, 2016 at 4:47 PM, Kevin Cooney [email protected]
wrote:

The lack of Rule support in JUnit5 might make it hard for large projects
to incrementally migrate test classes, especially if they have custom Rules.

I wonder if we could have an optional jar that defines a simple extension
point that can be used for JUnit5 and JUnit4. I find that most of our Rules
only need the following:

• setUp()
• tearDown() - called only if setUp () called
• verify() - called only if the test method doesn't throw

If this interests anyone, I could write up a proposal.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#169 (comment)

[marcphilipp]

@kcooney: Can you please elaborate your proposal a bit more?

[sbrannen]

Related blog post:

http://www.codeaffine.com/2016/04/06/replace-rules-in-junit5/

[kcooney]

Thanks for the pointer

Unfortunately, some companies have extremely large code bases with common custom Rules. For these companies it will take many years to migrate tests to JUnit5. Having to write and maintain parallel JUnit5 versions of each of our custom Rules would be a maintenance burden

I was hoping for a common interface that we could implement that could easily be adapted to work with JUnit4-style tests or JUnit5-style tests (and maybe even JUnit 3!)

Many of our extensions need to only do work in one or more of these places:

a. Before the test is run (setup)
b. After the test is completely done (tearDown)
c. Right after the test method was called if it didn't throw (verify)

Edit: Note that some of JUnit's built in rules follow this pattern, like ExternalResource and Verifier.

I have some ideas of what this API could look like so these fixtures can use composition (ex: "start a server and return the port" composed with "start WebDrive pointing to the server port")

[kcooney]

I put together a proposal for the kind of extension point i was thinking about here: junit-team/junit4#1336

Note that this is just a prototype built in the JUnit4 code base, but the new classes don't depend on JUnit4 (except for MultipleFailureException). The classes would eventually need to move to a new project so they could be used by projects that only have JUnit Vintage tests as well as in projects that only have JUnit Jupiter tests.

[marcphilipp]

@kcooney Thanks for your proposal! We have not yet found the time to discuss it in the team but will definitely do so when we next meet (virtually).

If we end up having such an API I think opentest4j would be a reasonable place to put it.

[kcooney]

@marcphilipp if you let me know when you meet I can try to join

[marcphilipp]

@kcooney We are short on time at the moment but have briefly discussed your proposal. We are not sure to what extend it would be useful because it requires people to change their existing test code and migrate their rules to fixtures, right?

[kcooney]

@marcphilipp yes it requires migrating existing code, but unless you want to fully support @Rule for JUnit Jupiter tests, large codebases with custom Rules will have no migration path other than making parallel versions of each of their custom rules.

In addition, from what I have seen, the existing JUnit Jupiter extension points are not as easy to use and extend as rules are (Edit: they are, of course, more powerful, and I am sure they will lead to much better third-party extension points than we saw for JUnit 4).

What I have here can be supported for all kinds of JUnit tests and is easy to use. The large majority of my company's reusable extensions use a very similar mechanism (in part because it can work in JUnit3-style tests).

I could do this outside of JUnit but I would need a way to install global rules. In the past @dsaff wasn't thrilled with global rules (in part because would be nothing in the test indicating that the test has additional behavior applied to it). Maybe you feel differently?

I could do this in JUnit if we are fine with JUnit 4.13 having a dependency

[marcphilipp]

@kcooney I am terribly sorry for not responding to your comments in the last few weeks. There were just too many things on my plate.

I believe fixtures might be an alternative migration path for custom runners/rules in addition to what we plan to do in #433. If they are useful for Google, chances are someone else will find them useful, too.

Have you considered splitting junit-team/junit4#1336 so that the interfaces (e.g. TestFixture etc.) end up in a separate artifact? One place were we could put them is OpenTest4J. A completely separate small library would work, too. That way, the jupiter-engine could also provide support for fixtures.

Global extension support is being discussed in #448.

[kcooney]

@marcphilipp no need to apologize; we are all busy :-)

Does this mean that you are fine with having OpenTest4J be a required dependency for JUnit 4.13? Or would it be an optional dependency (in which case I would love to see examples of how to do that without lots of ugly reflection code)

[kcooney]

BTW, in the pull request I was referring to global rule support for JUnit 4 (to avoid independency between JUnit 4.13 and another library)

[marcphilipp]

Does this mean that you are fine with having OpenTest4J be a required dependency for JUnit 4.13? Or would it be an optional dependency (in which case I would love to see examples of how to do that without lots of ugly reflection code)

I think in the long run, I would be fine with 4.13 depending on OpenTest4J but not before OTA is GA-ready, i.e. not a milestone. For further development, I think it should suffice if you put it in a separate package for now.

BTW, in the pull request I was referring to global rule support for JUnit 4 (to avoid independency between JUnit 4.13 and another library).

I think it's usually done in a way that the dependency is available at compile-time but is not required at runtime unless the feature is used. I think @sbrannen can provide us with some examples since Spring has many optional dependencies, right?

[schauder]

I know you don't want to offer the full complexity of JUnit Rules in extensions, but I just had an idea:

How about creating a completely separate module, that offers an additional ExtensionPoint with Rule like behavior?

One could make it as discouraging to use as one whishes, by using separate artefact ids and even making it Deprecated right from the start, as a sign that this should only be used for migration purposes.

It could even come from a separate team.

[sbrannen]

I think it's usually done in a way that the dependency is available at compile-time but is not required at runtime unless the feature is used.

Yes, that's correct. It would end up being an optional dependency in the Maven POM.

I think @sbrannen can provide us with some examples since Spring has many optional dependencies, right?

Sure, if you want concrete examples, I can point you to them.

In general, however, what you do is....

1. write a utility method that checks if a given class can be loaded with the current class loader (based on a hard coded fully qualified class name that you store as a string).
2. invoke that utility method as early as possible in any code path that attempts to use the optional feature.
3. if the class cannot be loaded, throw an exception with a very helpful error message explaining that the feature cannot be used unless the optional dependency is on the classpath.

[kcooney]

@schauder junit-team/junit4#1336 is an extension point with Rule-like behavior, and I think one that the Junit5 team would be comfortable supporting without deprecated APIs or warnings. Is it missing something you think our users would need?

[schauder]

@kcooney junit-team/junit4#1336 mentiones the constraints itself:

It doesn't seem possible to change the way the test code executs:

• run it in a special thread (e.g. EDT)
• run it 10 times and consider it failed when it fails in more then two cases
• run it 100 times in 10 threads in parallel

All these can be implemented with Rules easily.

They can also be done with DynamicTests (I think), but those don't participate in the Jupiter Livecycle.

And from the reaction to my talks: There is a considerable amount of people that do stuff like this.

I try to motivate them to actually try to reimplement their Runners and Rules with JUnit5 and give feedback, but I'm not confident that will happen.

I think a problem for JUnit5 might become: You can basically do everything with DynamicTests by wrapping your Executables in whatever you want, just as Rules do.
And people will do that to solve their problems. But by that the otherwise nice Extension Model goes down the drain.

[akozlova]

+1 to support execution in a special thread without DynamicTests. It's hard to start only one DynamicTest - and that's essential in many cases.

[sbrannen]

I think a problem for JUnit5 might become: You can basically do everything with DynamicTests by wrapping your Executables in whatever you want, just as Rules do. And people will do that to solve their problems. But by that the otherwise nice Extension Model goes down the drain.

You're right: if we don't get the extension model right (i.e., powerful enough), we will in fact face that risk.

However, I think most (if not all) of these concerns will be addressed in one or more of the following issues:

• Introduce support for parameterized tests #14
• Opt-in support for parallel execution #60
• Introduce extension API for executing test in user-defined thread #157
• Support alteration of discovered tests #276 (maybe)
• Introduce extension API for dynamic test generation #371 (maybe)

[sbrannen]

@akozlova, that will be addressed in #157.

[kcooney]

Having a Rule run a test multiple times violates some of the assumptions in JUnit 4 (ex: each test method runs in a unique instance) , so I wouldn't want to support it in Fixtures.

I don't know much about Junit5 internals, but I agree with the team that Rules is a problematic extension mechanism.

[bechte]

I think a problem for JUnit5 might become: You can basically do everything with DynamicTests by wrapping your Executables in whatever you want, just as Rules do. And people will do that to solve their problems. But by that the otherwise nice Extension Model goes down the drain.

I don't think that dynamic tests will be the answer to many problems. Those are very special and (still) do not support the powerful setup, test, teardown, nor do they support hierarchies. So I believe, people would rather use the more powerful extension model and if it does not fit their needs, they will open a feature request (hopefully).

Moreover, I think that we can support most of the items with the new extension model, we just need to lead into the right direction. And this is about this ticket. What is the migration path for e.g. @Rule. For rules I think there is not only one migration path, but many. It really depends on what the rule was for and what it did internally. The rule mechanism was one of the most powerful concepts to influence the test execution in JUnit4. You basically could do anything in there. This is something that we did not want with JUnit5. The basic test execution should be controlled by the framework, but we allow to participate. This is a different approach and we have to figure out a new way to solve certain issues that were solved with rules in JUnit4.

I recommend to provide a common migration path within this ticket that does not cover rules at all (or only a few "basic rules"), and create separate issues for all the more complex rules out there.

[marcphilipp]

in progress

[marcphilipp]

ExpectedException can now be used in Jupiter by enabling an extension. See http://junit.org/junit5/docs/snapshot/user-guide/#migrating-from-junit4-rulesupport for details.