Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: introduce the new reporter API #7069

Merged
merged 89 commits into from
Jan 14, 2025
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
38b967c
feat: introduce the new reporter API
sheremet-va Dec 11, 2024
25bae18
chore: cleanup
sheremet-va Dec 13, 2024
0998e4f
feat: introduce the new reporter API
sheremet-va Dec 11, 2024
6f0da08
chore: cleanup
sheremet-va Dec 13, 2024
64abbda
feat(reporters): add first 5/7 test run life cycles
AriPerkkio Dec 19, 2024
f3686b0
test: add initial test setup
AriPerkkio Dec 31, 2024
9cd58e9
fix: report skipped tests, prevent duplicate queued
AriPerkkio Dec 31, 2024
3f6edd5
fix: review
AriPerkkio Dec 31, 2024
1b8264a
Merge branch 'feat/new-reporter-api' of github.com:sheremet-va/vitest…
sheremet-va Jan 3, 2025
506cec5
refactor: don't allow undefined in getReportedEntity
sheremet-va Jan 3, 2025
2624054
refactor: move collecton/enqueued to the test runner
sheremet-va Jan 3, 2025
ef9a086
refactor: remove test.skipped()
sheremet-va Jan 3, 2025
10f3890
fix: correct skipped state
sheremet-va Jan 3, 2025
ebf3454
docs: cleanup
sheremet-va Jan 3, 2025
5d2bd70
docs: remove skipped from suite methods
sheremet-va Jan 3, 2025
cd291db
feat: add onTestModuleCollected
sheremet-va Jan 3, 2025
0ef6608
refactor: add events to TaskResultPack
sheremet-va Jan 3, 2025
d515785
docs: update result() docs
sheremet-va Jan 3, 2025
87d313b
docs: add state() docs
sheremet-va Jan 3, 2025
934087b
docs: fix result description
sheremet-va Jan 3, 2025
6567412
docs: update metadata docs
sheremet-va Jan 3, 2025
5e3b7cb
refactor: add event
sheremet-va Jan 3, 2025
20696a5
chore: fix ts error
sheremet-va Jan 3, 2025
ed1332e
fix: set event in typechecking
sheremet-va Jan 3, 2025
89baf8e
refactor: move log to test run
sheremet-va Jan 3, 2025
4afce13
refactor: move onFinished
sheremet-va Jan 3, 2025
4557b4b
fix: report skipped tests of suites
AriPerkkio Jan 4, 2025
3e0990e
Merge remote-tracking branch 'upstream/main' into feat/new-reporter-api
AriPerkkio Jan 4, 2025
1a74b1b
feat(reporter): `'dot'` to use new reporter API
AriPerkkio Jan 4, 2025
84ddc2f
refactor: mark readonly properties
sheremet-va Jan 4, 2025
1f03389
refactor: rename spec.module to spec.testModule
sheremet-va Jan 4, 2025
19fd301
docs: update options type, remove "reporter api" warning
sheremet-va Jan 4, 2025
be5b429
feat(reporter): summary to use new reporter API, except before & afte…
AriPerkkio Jan 4, 2025
02c0279
refactor: add comments and clarifications
sheremet-va Jan 7, 2025
1c8fb37
chore: cleanup
sheremet-va Jan 7, 2025
f6c3571
refactor: remove `TaskParser`, use new reporter API completely
AriPerkkio Jan 7, 2025
eb36354
refactor: rename new APIs
AriPerkkio Jan 7, 2025
0cef2b5
fix: resolve reporter life cycle from `onTaskUpdate` event field
AriPerkkio Jan 7, 2025
b5b962f
fix: report hooks only when they actually exist
AriPerkkio Jan 7, 2025
dd6ebc5
chore: add some test run docs and make arrays readonly
sheremet-va Jan 8, 2025
44ad507
docs: fix link
sheremet-va Jan 8, 2025
23730ae
Merge branch 'main' of github.com:vitest-dev/vitest into feat/new-rep…
sheremet-va Jan 8, 2025
c62f22b
refactor: add more docs, export TestRunEndReason and SerializedError
sheremet-va Jan 8, 2025
409c401
fix: attempt to get order right
AriPerkkio Jan 9, 2025
41985f3
fix: correct order for `TestCase`
AriPerkkio Jan 9, 2025
208466b
docs: document more hooks
sheremet-va Jan 9, 2025
2852271
refactor: rename hooks option
sheremet-va Jan 9, 2025
f8b657a
refactor: return diagnostic if test was retried
sheremet-va Jan 9, 2025
44f588d
fix: single `beforeEach/afterEach` working
AriPerkkio Jan 9, 2025
d98e4eb
fix: multiple `beforeEach/afterEach` working
AriPerkkio Jan 9, 2025
d9d652a
refactor: rename test-case's Start/End -> Ready/Result
sheremet-va Jan 10, 2025
97e51bd
test: refactor and add more tests
AriPerkkio Jan 11, 2025
c812be8
fix: report `beforeAll/afterAll` of `TestModule`
AriPerkkio Jan 11, 2025
a1b2051
refactor: don't parse events, keep the order in the runner (#3)
sheremet-va Jan 11, 2025
afe9935
chore: cleanup
sheremet-va Jan 11, 2025
8d1abdd
docs: add onTestSuiteReady/onTestSuiteResult
sheremet-va Jan 11, 2025
6a48858
chore: cleanup
sheremet-va Jan 11, 2025
3b087fe
Merge branch 'main' of github.com:vitest-dev/vitest into feat/new-rep…
sheremet-va Jan 11, 2025
8592c09
test: enable tests for suites
sheremet-va Jan 11, 2025
e56ffad
fix: don't report ctx.skip tests twice
sheremet-va Jan 11, 2025
328594d
chore: events
sheremet-va Jan 11, 2025
5c44c17
test: fix test
sheremet-va Jan 11, 2025
06669e7
docs: cleanup
sheremet-va Jan 11, 2025
330deee
fix: remove coverage from merge-reports
sheremet-va Jan 11, 2025
cb0110a
chore: always report onCoverage
sheremet-va Jan 11, 2025
db86137
fix: remove error with sharding
sheremet-va Jan 11, 2025
0fd1b76
test: fix snapshot
sheremet-va Jan 11, 2025
503cae0
refactor: cleanup
sheremet-va Jan 11, 2025
9947672
test: add tests with delay to not check sync updates only
AriPerkkio Jan 12, 2025
101a542
Merge remote-tracking branch 'upstream/main' into feat/new-reporter-api
AriPerkkio Jan 12, 2025
7dc23e6
fix: dot reporter to check suite tests too
AriPerkkio Jan 12, 2025
7a4dbff
chore: review fixes
AriPerkkio Jan 12, 2025
bcb6e71
docs: broken link
AriPerkkio Jan 12, 2025
6bc4f8d
docs: correct onCoverage
AriPerkkio Jan 12, 2025
bef1eee
chore: review fixes
AriPerkkio Jan 12, 2025
306a5cd
refactor: remove unused statistics from reported tasks
AriPerkkio Jan 13, 2025
50cf790
refactor: remove unused statistics from reported tasks
AriPerkkio Jan 13, 2025
cf0e14f
Merge remote-tracking branch 'upstream/main' into feat/new-reporter-api
AriPerkkio Jan 13, 2025
e5e9638
build: fix paths
AriPerkkio Jan 13, 2025
1a155c4
doc: jsdoc for reporter hooks
AriPerkkio Jan 13, 2025
1fd9697
chore: review, typos, revert formatting-only changes
AriPerkkio Jan 13, 2025
6d3b228
fix: report skipped tests in merge-reports
AriPerkkio Jan 13, 2025
ebff697
fix: add events to typechecker
AriPerkkio Jan 13, 2025
3e93fb3
Merge branch 'main' of github.com:vitest-dev/vitest into feat/new-rep…
sheremet-va Jan 14, 2025
b46340d
chore: merge reporters types
sheremet-va Jan 14, 2025
077f230
fix: call onTestRunStart/onTestRunEnd even if there are no files to run
sheremet-va Jan 14, 2025
33c8544
fix: don't call `onCoverage` hook if there is no coverage to report
sheremet-va Jan 14, 2025
59808e7
fix: correctly report merged reports
sheremet-va Jan 14, 2025
b4cb350
fix: correctly parse typecheck events and print TS icon in summary
sheremet-va Jan 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,10 @@ export default ({ mode }: { mode: string }) => {
text: 'Running Tests',
link: '/advanced/guide/tests',
},
{
text: 'Test Lifecycle',
link: '/advanced/guide/lifecycle',
},
{
text: 'Extending Reporters',
link: '/advanced/reporters',
Expand Down
3 changes: 3 additions & 0 deletions docs/advanced/guide/lifecycle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Test Lifecycle

<!-- TODO: lifecyle diagram and reporter API -->
AriPerkkio marked this conversation as resolved.
Show resolved Hide resolved
99 changes: 84 additions & 15 deletions packages/vitest/src/node/reporters/reported-tasks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import type {
Suite as RunnerTestSuite,
TaskMeta,
} from '@vitest/runner'
import type { TestError } from '@vitest/utils'
import type { SerializedError, TestError } from '@vitest/utils'
import type { TestProject } from '../project'

class ReportedTaskImplementation {
Expand Down Expand Up @@ -122,12 +122,19 @@ export class TestCase extends ReportedTaskImplementation {
}

/**
* Test results. Will be `undefined` if test is skipped, not finished yet or was just collected.
* Test results.
* - **pending**: Test was collected, but didn't finish running yet.
* - **passed**: Test passed successfully
* - **failed**: Test failed to execute
* - **skipped**: Test was skipped during collection or dynamically with `ctx.skip()`.
*/
public result(): TestResult | undefined {
public result(): TestResult {
const result = this.task.result
if (!result || result.state === 'run') {
return undefined
return {
state: 'pending',
errors: undefined,
}
}
const state = result.state === 'fail'
? 'failed' as const
Expand Down Expand Up @@ -298,10 +305,33 @@ class TestCollection {

export type { TestCollection }

export interface TestSuiteStatistics {
total: number
completed: number
passed: number
failed: number
skipped: number
todo: number
}

function createStatistics() {
return {
total: 0,
completed: 0,
passed: 0,
failed: 0,
skipped: 0,
todo: 0,
}
}

abstract class SuiteImplementation extends ReportedTaskImplementation {
/** @internal */
declare public readonly task: RunnerTestSuite | RunnerTestFile

/** @internal */
public _statistic: TestSuiteStatistics = createStatistics()

/**
* Collection of suites and tests that are part of this suite.
*/
Expand All @@ -314,18 +344,38 @@ abstract class SuiteImplementation extends ReportedTaskImplementation {
}

/**
* Checks if the suite was skipped during collection.
* The number of tests in this suite with a specific state.
*/
public skipped(): boolean {
public statistics(): TestSuiteStatistics {
return { ...this._statistic }
}

/**
* Checks the running state of the suite.
*/
public state(): TestSuiteState {
const mode = this.task.mode
return mode === 'skip' || mode === 'todo'
const state = this.task.result?.state
if (mode === 'skip' || mode === 'todo' || state === 'skip' || state === 'todo') {
return 'skipped'
}
if (state == null || state === 'run' || state === 'only') {
return 'pending'
}
if (state === 'fail') {
return 'failed'
}
if (state === 'pass') {
return 'passed'
}
throw new Error(`Unknown suite state: ${state}`)
}

/**
* Errors that happened outside of the test run during collection, like syntax errors.
*/
public errors(): TestError[] {
return (this.task.result?.errors as TestError[] | undefined) || []
public errors(): SerializedError[] {
return (this.task.result?.errors as SerializedError[] | undefined) || []
}
}

Expand Down Expand Up @@ -402,8 +452,8 @@ export class TestModule extends SuiteImplementation {

/**
* This is usually an absolute UNIX file path.
* It can be a virtual id if the file is not on the disk.
* This value corresponds to Vite's `ModuleGraph` id.
* It can be a virtual ID if the file is not on the disk.
* This value corresponds to the ID in the Vite's module graph.
*/
public readonly moduleId: string

Expand All @@ -420,9 +470,9 @@ export class TestModule extends SuiteImplementation {
declare public ok: () => boolean

/**
* Checks if the module was skipped and didn't run.
* Checks the running state of the test file.
*/
declare public skipped: () => boolean
declare public state: () => TestSuiteState

/**
* Useful information about the module like duration, memory usage, etc.
Expand All @@ -446,6 +496,7 @@ export class TestModule extends SuiteImplementation {

export interface TaskOptions {
each: boolean | undefined
fails: boolean | undefined
concurrent: boolean | undefined
shuffle: boolean | undefined
retry: number | undefined
Expand All @@ -454,10 +505,11 @@ export interface TaskOptions {
}

function buildOptions(
task: RunnerTestCase | RunnerTestFile | RunnerTestSuite,
task: RunnerTestCase | RunnerTestSuite,
): TaskOptions {
return {
each: task.each,
fails: task.type === 'test' && task.fails,
concurrent: task.concurrent,
shuffle: task.shuffle,
retry: task.retry,
Expand All @@ -466,7 +518,24 @@ function buildOptions(
}
}

export type TestResult = TestResultPassed | TestResultFailed | TestResultSkipped
export type TestSuiteState = 'skipped' | 'pending' | 'failed' | 'passed'

export type TestResult =
| TestResultPassed
| TestResultFailed
| TestResultSkipped
| TestResultPending

export interface TestResultPending {
/**
* The test was collected, but didn't finish running yet.
*/
state: 'pending'
/**
* Pending tests have no errors.
*/
errors: undefined
}

export interface TestResultPassed {
/**
Expand Down
48 changes: 48 additions & 0 deletions packages/vitest/src/node/test-run.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import type { TaskResultPack } from '@vitest/runner'
import type { Vitest } from './core'
import type { TestModule } from './reporters/reported-tasks'
import type { TestSpecification } from './spec'

export class TestRun {
AriPerkkio marked this conversation as resolved.
Show resolved Hide resolved
private tests = emptyCounters()
private suites = emptyCounters()

constructor(private vitest: Vitest) {}

async start(specifications: TestSpecification[]) {
this.tests = emptyCounters()
this.suites = emptyCounters()
this.suites.total = specifications.length
await this.vitest.report('onTestRunStart', specifications)
}

enqueued(module: TestModule) {
// TODO
}

collected(modules: TestModule[]) {
// TODO
}

updated(update: TaskResultPack[]) {
// TODO
}

async end() {
// TODO
await this.vitest.report('onTestRunEnd', [], [], 'passed')
}
}

interface Counter {
total: number
completed: number
passed: number
failed: number
skipped: number
todo: number
}

function emptyCounters(): Counter {
return { completed: 0, passed: 0, failed: 0, skipped: 0, todo: 0, total: 0 }
}
11 changes: 11 additions & 0 deletions packages/vitest/src/node/types/reporter.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
import type { File, TaskResultPack } from '@vitest/runner'
import type { SerializedError } from '@vitest/utils'
import type { SerializedTestSpecification } from '../../runtime/types/utils'
import type { Awaitable, UserConsoleLog } from '../../types/general'
import type { Vitest } from '../core'
import type { TestModule } from '../reporters/reported-tasks'
import type { TestSpecification } from '../spec'

export interface Reporter {
onInit?: (ctx: Vitest) => void
Expand All @@ -20,4 +23,12 @@ export interface Reporter {
onServerRestart?: (reason?: string) => Awaitable<void>
onUserConsoleLog?: (log: UserConsoleLog) => Awaitable<void>
onProcessTimeout?: () => Awaitable<void>

// new API
onTestRunStart?: (specifications: TestSpecification[]) => Awaitable<void>
onTestRunEnd?: (
testModules: TestModule[],
errors: SerializedError[],
reason: 'passed' | 'interrupted' | 'failed'
) => Awaitable<void>
}
3 changes: 2 additions & 1 deletion packages/vitest/src/public/node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ export type { JUnitOptions } from '../node/reporters/junit'

export type {
ModuleDiagnostic,

TaskOptions,

TestCase,
TestCollection,
TestDiagnostic,
Expand All @@ -44,6 +44,7 @@ export type {
TestResultPassed,
TestResultSkipped,
TestSuite,
TestSuiteStatistics,
} from '../node/reporters/reported-tasks'
export { BaseSequencer } from '../node/sequencers/BaseSequencer'

Expand Down