Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): simple mutation switching process #2269

Merged
merged 23 commits into from
Jul 3, 2020

Conversation

nicojs
Copy link
Member

@nicojs nicojs commented Jun 23, 2020

Implement a simple mutation testing process using mutation switching. It represents a new way of running mutation testing, where we place all mutants in the code at the same time. This is only the first step and lacks support for more advanced use cases.

This is a non-exhaustive list of changes:

  • Add Location to the Mutant API, for ease of reporting
  • Implement TestRunner2 factory
  • Revamped the way the main Stryker class works. All of the internals are moved to separate Executor classes. An Executor is responsible for managing a part of the process. I think this helps enormously to understand the general process of mutation testing.
    • PrepareExecutor: Responsible to read configuration and input files from disk.
    • MutantInstrumenterExecutor: Responsible for instrumenting the source code and generating mutants
    • DryRunExecutor: Responsible for performing the initial test run.
    • MutationTestExecutor: Responsible for running actual mutation testing.
  • Revamped the Sandbox class. It is now only responsible for filling a sandbox with files. It no longer manages mutants inside the sandbox or the test runner instance.
  • Revamped the SandboxPool -> TestRunnerPool, which now manages only test runner instances.
    • An improvement I've managed to sneak in, is that the test runner process from the dry run is now reused in the mutation test run. Not a huge deal, but a nice bonus.
  • Removed a bunch of dead code related to the TestFramework and Mutator plugins, see breaking changes
  • Re-enabled 3 simple e2e tests 🎉

BREAKING CHANGES:

Most of these changes are related to internals of Stryker plugins.

  • Transpilers plugins are no longer supported. Transpiler-like functionality will be possible with a build command and checker API. More on those in future PR's
  • Custom mutators are no longer supported, Stryker now brings its own mutator in the form of @stryker-mutator/instrumenter. It uses babel and an HTML parser under the hood so it has support for all the usual suspects.
  • Test Framework plugins are no longer supported. Test runners are supposed to bring their own support for test frameworks they might want to support. This has no consequences for the list of officially supported test runners.
  • Test runner v1 API is no longer supported. Please move to the new Test Runner API.

nicojs added 6 commits June 19, 2020 17:29
Implement a simple mutation testing process using mutation switching.

BREAKING CHANGE:
* Transpilers are no longer supported, please use build command instead
* Custom mutators are no longer supported, Stryker now brings it's own mutator
@nicojs
Copy link
Member Author

nicojs commented Jun 23, 2020

Todo:

  • Unit tests for PrepareExecutor
  • Unit tests for MutantInstrumenterExecutor
  • Unit tests for DryRunExecutor
  • Unit tests for MutationTestExecutor
  • Unit tests for (new) Sandbox
  • Unit tests for TestRunnerPool
  • Unit tests for the new test runner factory
  • Implement Reporter.onAllMutantsMatchedWithTests
    🤷‍♂️ Implement command test runner create a new command test runner with another PR
  • Make sure html report works (mutants are not visible a.t.m.)
  • Make sure the progress reporter is working again (onAllMutantsMatchedWithTests is needed for that).
  • Remove dead code.

@nicojs nicojs marked this pull request as ready for review June 26, 2020 19:45
@nicojs nicojs requested a review from simondel June 26, 2020 19:46
@nicojs
Copy link
Member Author

nicojs commented Jun 26, 2020

So yeah the PR turned out to be pretty massive. I've revamped the entire core of Stryker. I also had to break the reporter api in some places, we might need to break it more later. I'll update the PR's description to reflect the current state.

return this.options.coverageAnalysis === 'perTest';
}
}
// import { Mutant, StrykerOptions } from '@stryker-mutator/api/core';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commented file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, one of the ones that need to go. Probably best if I remove it indeed. I do want to keep some of it for now. For example, the new Checker code can be loosely based on the Transpiler code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I think it is better to remove it, since we could forget about it later, ok. We just need to keep it in mind :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, you're right. Classes that are already refactored/replaced should be removed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, all dead code is gone now.

nicojs added 3 commits June 26, 2020 22:34
* Remove test runner v1 stuff
* Remove mutator plugin stuff
* Remove test framework plugin stuff
* Remove `SandboxPool` (was reimagined as `TestRunnerPool`)
* Remove `MutantTestMatcher` (was reimagined as `findMutantTestCoverage`)
@nicojs nicojs merged commit 9d4671b into epic/mutation-switching Jul 3, 2020
@simondel simondel deleted the feat/simple-mutation-switching branch September 28, 2020 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants