Latest version reports mutants as killed when they haven't run because of a runner error

[HeroBart]

I've installed the latest version and have a simple function:

function addOne(a) {
    var b = 1;
    return a + b;
}

And the unit test:

describe('Validator class', function () {
    it('adds 1', function () {
        expect(addOne(10)).toBeTruthy();
    });
});

If I understand correctly, this mutant should not be marked killed since the mutation a - b still will be truthy, so the unit test doesn't fail (and a good unit test should fail).

I found the issue after messing about in the code. If I change IsolatedTestRunnerAdapter.js and change "silent: false" to "silent: true" I can see a lot more useful output that isn't displayed even with logLevel all.
The initial unit tests are run correctly. If I make a failing unit test, it will report that the initial run failed.
FYI I faked the cpu count to 1 for less output, but it also doesn't work with 8 cpu's.

I see something like this:

22 07 2016 12:05:04.105:INFO [karma]: Karma v1.0.0 server started at http://localhost:9231/
22 07 2016 12:05:04.105:INFO [launcher]: Launching browser PhantomJS with unlimited concurrency
22 07 2016 12:05:04.113:INFO [launcher]: Starting browser PhantomJS
22 07 2016 12:05:05.863:INFO [PhantomJS 2.1.1 (Windows 7 0.0.0)]: Connected on socket /#sxiwMmWUhn7RLDb-AAAA with id 16944920
22 07 2016 12:05:06.013:INFO [KarmaTestRunner]: karma run done with  0
[2016-07-22 12:05:06.078] [INFO] Stryker - Initial test run succeeded. Ran 1 tests.
[2016-07-22 12:05:06.095] [INFO] Stryker - 1 Mutant(s) generated
[2016-07-22 12:05:06.098] [INFO] TestRunnerOrchestrator - Creating 1 test runners (based on cpu count)

But then after generating the mutants and trying to run the mutants, I see this:

[2016-07-22 12:05:07.627] [INFO] KarmaTestRunner - using config {"browsers":["PhantomJS"],"frameworks":["jasmine"],"autoWatch":false,"single
Run":false,"detached":false,"files":[{"pattern":"C:\\projects\\mobile-contract-registration\\Web\\App\\.stryker-tmp\\1855138\\test-runner-fi
les8462187\\___testSelection.js","included":true},{"pattern":"C:\\projects\\mobile-contract-registration\\Web\\App\\.stryker-tmp\\1855138\\t
est-runner-files8462187\\src\\directives\\glnlValidate\\Validator.js","included":true},{"pattern":"C:\\projects\\mobile-contract-registratio
n\\Web\\App\\.stryker-tmp\\1855138\\test-runner-files8462187\\src\\unittests\\Validator-spec.js","included":true},{"pattern":"C:\\projects\\
mobile-contract-registration\\Web\\App\\.stryker-tmp\\1855138\\test-runner-files8462187\\src\\directives\\glnlValidate\\Validator.js","inclu
ded":true}],"port":9231}
Msgggg  { type: 3, body: { runOptions: { timeout: 5178.5 } } }
Running
22 07 2016 12:05:07.930:INFO [karma]: Karma v1.0.0 server started at http://localhost:9231/
22 07 2016 12:05:07.932:INFO [launcher]: Launching browser PhantomJS with unlimited concurrency
22 07 2016 12:05:07.939:INFO [launcher]: Starting browser PhantomJS
**No captured browser, open http://localhost:9231/**
22 07 2016 12:05:07.961:INFO [KarmaTestRunner]: karma run done with  1

I messed around a little with ports and such to try to get it to work, but no such luck.
Then I tried changing KarmaTestRunner.js to use Chrome instead of PhantomJS. The initial test run will then run in Chrome and that still works correctly, but then I still get "No captured browser".

So the issue is twofold.
First of all, "No captured browser" basically results in all mutants being killed because the unit test itself isn't actually running, but some other error occurs, namely:
[TypeError: Cannot read property 'error' of null]
No clue where this error occurs. Anyway, when running the test fails like this, it should not report the mutants as killed.

But I would like to solve this issue as well of course. Does anyone have any clue why the initial test run goes fine, but running the mutants returns "No captured browser"? Could be an issue with Karma, but I just don't understand why the initial run is fine and then it fails. Any help would be greatly appreciated.

[nicojs]

@Kimpossibru Thanks for your extensive issue

Could you please provide me with version information on your stryker dependencies? This command should do the trick:

npm ls | grep stryker

The reporting of errors as killed was a recent, but deliberate, addition (#120). When creating mutants right now, we might create type errors.

@simondel maybe we should change this behavior? Can we make sure we are not creating type errors when creating mutants?

[HeroBart]

@nicojs thanks for your quick response. I've run the command and this is the output:

+-- [email protected]
+-- [email protected]
+-- [email protected]
`-- [email protected]

I've uploaded the workspace as well: bart-stryker.zip

To run it:

npm install
node node_modules/stryker/bin/stryker -c stryker.conf.js

Output will say:

2 mutants tested.
0 mutants untested.
0 mutants timed out.
2 mutants killed.
Mutation score based on covered code: 100.00%
Mutation score based on all code: 100.00%

Whereas one mutation should be killed and one should not.
The a - b mutation should not be killed since a - b and a + b are both truthy if a is 10 and b is 1.
The block statement mutation should be killed since an empty function will not return something truthy. A nice way to find more output is setting silent: false in IsolatedTestRunnerAdapter.js, this will then again say:

22 07 2016 13:44:31.698:INFO [launcher]: Starting browser PhantomJS
No captured browser, open http://localhost:9241/
22 07 2016 13:44:31.730:INFO [KarmaTestRunner]: karma run done with  1

[nicojs]

Found out what the issue was. We've been working hard on the 0.4 release and, apparently, i still had 1 commit in the testrunner-lifecycle branch (should have been in the #125 PR...). I created a new PR for this: #132. After a final review, i'll merge it and create a new release

[HeroBart]

Awesome, I tried it locally and that fixes it. Thanks!

[nicojs]

New release done, v0.4.1

Basically what happens is that we send the run command to karma before it had to time to be initialized.

This initialization step was a new addition. Now we wait for initialization to be fully completed before we start to run tests. It should result in less false positives.

Btw: you can also enable test output with --logLevel trace (command line) or logLevel: 'trace' (config file)

[nicojs]

@simondel Do you think we should add an issue to handle errors during mutation run differently? Seems a lot safer.

[simondel]

@nicojs How would you know the difference between a 'good' error (one caused by a mutation) and a bad one? (like in this case)