Switching the unit test runner

[DorianGrey]

I'm currently considering to swap out the karma based approach for executing unit tests in favor of a JSDOM based solution. Main reasons against the current solution are:

• It quickly becomes extremely slow on increasing project sizes and/or complexity (mostly caused by the required bundle process)
• karma-webpack isn't that well maintained; it even requires an additional plugin to make the tests fail on build errors
• Due to the way tests are picked up, the coverage reports are always a bit off (i.e. too optimistic)

After some really good experiences after migrating an angular.js project to use jest as its unit test runner (replacing karma), I think a change like this should be considered for this angular project as well - at least for pointing out the potential problems and benefits. For getting a better overview, I've picked up a mocha based solution as well - the details are listed below.

General expectable benefits:

• Tests are executing way faster than they do now, especially in watch mode (hello TDD!); mostly because the dependencies can simply be required instead of bundled
• Since they are running on node, it's a bit easier to debug your tests

General expectable downsides:

• These are JSDOM-based solutions, it might require a couple of workarounds for DOM-heavy components under test

Details about the considered alternatives:

The alternatives below have already been used on testing branches to figure out if they are even considerable (hint: they are). Branches are linked in case anyone is interested in experimenting with them. The benefits and downsides listed for each entry come in addition to the JSDOM general ones listed above.

Jest

Jest is a test runner from facebook folks, and widely used in react projects, though the vue folks seem to start to like it as well (see vue-jest).

Using jest preset:
https://github.com/thymikee/jest-preset-angular
Testing tree:
https://github.com/DorianGrey/ng-webpack-template/tree/jest-test

Expectable benefits:

• Offers a fast watch and an interactive mode, with a lot of timing information (esp. for slow tests) and code snippets on errors when possible.
• Handles most reporter types out of the box (only had to add the junit reporter manually).
• Migration from the current jasmine setup is quite easy due to a similar API.
• Very well maintained.
• Very accurate coverage reports.

Expectable downsides:

• The typings might conflict with those from jasmine, in case these are used for E2E-testing with protractor.
• babel has to be added to transpile lodash-es (uses ES module definition, but commonjs would be required here).
• Will cause a couple of duplications regarding setup, esp. when it comes to transformers and path mappings.
• Does not perform type-checking or linting on the spec files; This would only happen on build or start due to the currently used fork-ts-checker-webpack-plugin.

mocha-webpack

mocha-webpack is a utility that simplifies bundling the test code via webpack and executing the tests via mocha. The example uses chai and sinon for expectations and spies.

Library:
https://github.com/zinserjan/mocha-webpack
Testing tree:
https://github.com/DorianGrey/ng-webpack-template/tree/mocha-test

Expectable benefits :

• Faster than the current karma approach, though a bit slower than the jest alternative.
• Allows re-using the current webpack setup with all its advantages, but only a couple of its downsides.
• Type-checking and linting of spec files is still possible, though it slows down things.

Expectable downsides:

• A lot of editors do not like the chai expressions - they don't get that most of them are getters. In my test, even the linter complained about useless expressions.
• It's quite complicated to get a full source coverage report working - wasn't able to do that to the time being, it currently only lists coverage for the files involved in the tests.
• The maintenance status is a bit unclear.

Comparison

Expectable benefits of jest over mocha-webpack

• Interactive mode
• Faster
• Better coverage report (incl. setup)
• Less editor and linter complaints, at least compared to chai expectations
• Reliable maintenance status

Expectable benefits of mocha-webpack over jest

• Allows re-using the webpack configuration, thus less setup duplications
• Allows to handle lodash-es without babel
• Typings would not conflict with jasmine ones potentially used in protractor; they would even be re-usable in case mocha is used for protractor

Result

I'm currently tending a bit towards jest, but nothing has been decided yet. I'll leave this open for a while for (potential) comments, suggestions, and additional thoughts from other (and from my mind, of course).

[DorianGrey]

Decision has been made to switch to jest. Follow the corresponding PR for status updates: #15