Refactor IPC, status and reporters

[novemberborn]

Simplify how test results flow from the worker processes to the API instance to the reporters by emitting "state changes" rather than specific events.

Workers report (most) state changes as they happen, they're collated for each run, and reporters receive each state change and can write to the console in turn. Stats are computed centrally in lib/run-status.js and are emitted whenever they change, thus simplifying reporters.

Because workers report state changes as they happen we can greatly simplify their coordination with the main process. We just need to make sure the last state change has been received before exiting. If the main process decides to exit a worker it can simplify do so, it'll already have received results from completed tests.

Error conditions are either explicit or can be inferred from the computed stats.

The API now emits its plan of the tests it'll run, which can then be sent to the reporters. This leads to better separation of concerns between the reporters, the run status, the API and the watch mode
implementation.

Uncaught exceptions and unhandled rejections are now shown with a code excerpt if possible. There are other minor variations in the reporter output, but I've tried to keep changes to a minimum.

All reporters now write to process.stdout. The stdout and stderr output from workers is written to process.stderr. AVA will insert line breaks in process.stdout after writing a chunk to process.stderr that does not end in a line break.

In watch mode, files being rerun are now excluded from the previous failure count.

profile.js is now wired up to use the verbose reporter. It's still a terrible hack though.

Porting the tests surfaced some integration-type tests that no longer made sense. They've been removed. Hopefully I haven't gone overboard with that. Ideally we'd rewrite the integration tests but that's a lot of work. Code coverage still looks good though.

---

Additional changes:

• append-transform, nyc and require-precompiled are now removed from error stacks

• Reporters are now verified using integration tests. This replaces the previous reporter tests. Instead we run the reporter for varying test outputs and record the results. The environment is modified to ensure the highest fidelity colors and figures are used. Where necessary output is rewritten so the resulting logs can be compared across Node.js versions and operating systems.

  ./test/helpers/replay.js can be used to replay a log file.

• Test implementations are now called so that this ==== null. Previously we were exposing the test instance through this, and we shouldn't have been

• time-require has been removed. It was only used when DEBUG=ava but it's not super useful during development. Users can load it themselves if they want to measure how quickly their own code loads

• We now instantiate our own copy of chalk. This will make it easier to inherit the color level from the main process (Ensure workers inherit color level from main process #1701)

• When serializing errors, if the value is not an actual error object we'll format it using Concordance and display that instead

• No more repeated timeouts for the same worker

---

To do:

• Port mini and TAP reporters
• Ensure tests pass
• Ensure good coverage for new reporter code

Follow-up issues:

• File issue so color level can be defined in AVA's package.json options
• Unblock existing reporter issues

[novemberborn]

I tried to run CI on the reporter test changes before I added all the other code that now causes tests to fail. With Node.js 6 on Windows I ran into an odd issue where stdout and stderr from the worker processes was missing. See the failure here: https://ci.appveyor.com/project/ava/ava/build/job/xtg4qmqy9wy1rqfk

This may be related to https://help.appveyor.com/discussions/problems/4177-nodejs-child-processes-dont-get-complete-output. We'll have to see if this issues persists once the refactor is complete.

[novemberborn]

With Node.js 6 on Windows I ran into an odd issue where stdout and stderr from the worker processes was missing.

Based on https://nodejs.org/api/process.html#process_a_note_on_process_i_o this may be due to the process exiting before the output has been written. #1718 might help with that.

[sindresorhus]

Typo

[sindresorhus]

The changes so far look good to me. I've been running this PR (using ava -v) on lots of projects of mine and nothing has failed yet.

[novemberborn]

Thanks @sindresorhus.

I've updated the mini reporter now:

• It depends on Handle wrapping, add hideCursor option sindresorhus/ora#69
• I've removed the additional indentation
• No more truncation (Mini reporter goes mad when the terminal width is smaller than the test title #629)

The mini reporter closely follows the previous output. I'm very tempted to improve it a bit though, make it more consistent with the verbose reporter, and see where we can reuse code. The verbose reporter definitely needs another take even without output changes.

I also suspect our visual tests might be obsolete given the new reporter tests.

[novemberborn]

I've finished the main refactor. See the PR post above for updated details, as well as the commits themselves. Just trying to get tests to pass now.

I have some ideas on merging the mini and verbose reporters which would make the remaining reporter issues a lot easier to tackle, so I'm hoping to still do that before merging.

[novemberborn]

Oh my! Tests are passing!

I think I'll try and get a release out before landing this. I'd also like to see if we can communicate process-shutdown over IPC and if that helps with the job concurrency on Windows. In the meanwhile, any other feedback @avajs/core?

Unfortunately I'm out of time as far as this weekend goes, so perhaps next weekend.

[novemberborn]

The release is out, but unfortunately I've run out of time again 😄

[sindresorhus]

I've run this PR on a bunch of my projects and it seems to be working perfectly.