Ability to filter test cases in test files using a custom Jest runner

[ryparker]

Updated: Draft PR available

---

🚀 Feature Proposal

Currently custom Jest runners support filtering test file paths, however it would also be nice to filter test cases in those test files.

Motivation

I'd like to setup my custom Jest runner to filter test cases by the test case's docBlock pragmas. Lets say I wanted to mark tests with a @severity pragma and wanted my custom Jest runner to only run @severity critical test cases when provided a CLI arg I.e. jest run --severity=critical

Updated: Working project (dependent on Jest PR approval)

Example

Perfect world IMO would be to create a new API TestRunner.runTestCases() similar to the TestRunner.runTests(). runTestCases() would receive an array of Jest test cases for each file with properties similar to Circus's Test.mode which could be modified as void|'todo'|'skip'|'only' before passing to super.runTestCases().

Although it looks like the current implementation of jest-runner only handles the test paths and leaves the test file processing to the test framework (circus/jasmine). So perhaps another solution would be to expose the testNamePattern config in the TestRunner.runTests(). The current implementation of testNamePattern only takes a single string/RegExp, ideally this would accept an array of test name patterns. A workaround for now could be to form a string composed of test names separated by the regex | token I.e. "(name of test 1|name of test 2)".

I've tested this theory by adding the following Get/Set to the TestRunner class in jest-runner/src/index.ts:

class TestRunner {
  private _globalConfig: Config.GlobalConfig;
  ...

  get testNamePattern() {
    return this._globalConfig.testNamePattern;
  }

  set testNamePattern(pattern: string | undefined) {
    this._globalConfig = { ...this._globalConfig, testNamePattern: pattern};
  }
...
}

This allows me to set the testNamePattern in my custom Jest runner like this:

class myCustomJestRunner extends TestRunner {
  ...
  runTests(...){
    ...
    super.testNamePattern = '(pragma is detected, when provided an @tag|pragma is detected, when provided a description located above the pragma)';
    // This will only run the two tests that's names match the above regex and skip the others
  }
...
}

However this only works as expected if all test cases use unique names. One way to prevent test name collisions between files would be to enable setting the testNamePattern for each test file. This could be achieved by adding testNamePattern to the list of ProjectConfig options, and modifying the respected testNamePattern checks in both Jasmine and Circus. The testNamePattern config would be utilized in the typical priority of ProjectConfig > GlobalConfig > Default(undefined). This would allow custom Jest runners to set a testNamePattern per test file passed through TestRunner.runTests() or it will fallback to the typical config.

If this sounds like an acceptable path I can happily open a PR.

Pitch

This would enable custom Jest runners to have more control over what is ran. Enabling more customizable test runs which is especially useful when working with large test collections. This would also allow users to write and organize test cases in ways that are run-agnostic.

[ryparker]

Here is a working draft PR for the feature. I could use some help with resolving the failing tests i'm experiencing on master's head before it is ready for review.

[ryparker]

Based on PR feedback:

That said, I'm not sure about the approach this PR makes. I think it makes more sense for a runner to be able to filter tests more directly, e.g. by getting a list of all test cases and returning the list of which to run rather than using testNamePattern. --SimenB

I'm going to try another approach to this that does not require using testNamePattern

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.