[kbn-scout] Custom event-oriented test reporter & persistence #202906

dolaru · 2024-12-04T12:26:31Z

Summary

These changes introduce an event-oriented test reporter which should aid with analysis of test runs over time.

Scout CLI

All Scout related CLI commands are now consolidated to a single script: ./scripts/scout.js.

Scout report

This is essentially a tool for logging events that occur during a test run and save them to an ndjson file for later consumption.

Scout report events are best consumed when indexed in an ES datastream. Initializing the datastream and uploading events should be fairly simple to do using the newly added CLI commands initialize-report-datastream and upload-events.

Reporters

Easy to use reporters that produce a Scout report have been added for Playwright and FTR (Mocha).

When these are used, you can expect the reports to be saved automatically under .scout/reports relative to the Kibana repository root.

Reporters have been attached to FTR and Scout Playwright configs. For FTR, a report will only be produced if the environment variable ENABLE_SCOUT_REPORTER=true is set.

packages/kbn-scout/src/reporting/jest.ts

scripts/scout.js

packages/kbn-scout-info/README.md

packages/kbn-scout/src/cli/run_tests.ts

packages/kbn-scout/src/cli/start_server.ts

packages/kbn-scout-reporting/src/datasources/buildkite.ts

packages/kbn-scout/src/playwright/config/index.ts

pheyos

Next batch of comments - still reviewing more.

packages/kbn-scout-reporting/src/reporting/report/event.ts

packages/kbn-scout-reporting/src/reporting/report/persistence/mappings.ts

packages/kbn-scout-reporting/src/reporting/report/persistence/index_templates.ts

packages/kbn-scout-reporting/src/reporting/report/persistence/mappings.ts

packages/kbn-scout-reporting/src/reporting/report/persistence/index.ts

pheyos

Two more comments

packages/kbn-test/src/functional_test_runner/lib/mocha/reporter/scout_ftr_reporter.ts

packages/kbn-scout-reporting/src/reporting/playwright.ts

Addresses circular dependency issues detected during type checks.

Covered by ECS mappings.

dolaru · 2024-12-09T12:01:39Z

@pheyos Review comments addressed. Changes since you've last reviewed are here.

pheyos

LGTM 🚀
Also went through a number of scenarios locally (including skipped and failed tests) and uploaded results to a cloud cluster. That all worked fine for me.

elasticmachine · 2024-12-09T14:27:19Z

💚 Build Succeeded

Buildkite Build
Commit: 81289bc

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/scout-info`	-	6	+6
`@kbn/scout-reporting`	-	53	+53
`@kbn/test`	303	305	+2
total			+61

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/scout-reporting`	-	3	+3

Unknown metric groups

API count

id	before	after	diff
`@kbn/scout`	119	117	-2
`@kbn/scout-info`	-	6	+6
`@kbn/scout-reporting`	-	68	+68
`@kbn/test`	358	360	+2
total			+74

ESLint disabled line counts

id	before	after	diff
`@kbn/scout-reporting`	-	2	+2

Total ESLint disabled count

id	before	after	diff
`@kbn/scout-reporting`	-	2	+2

History

💔 Build #257977 failed be87974
💛 Build #257433 was flaky aa76745
💚 Build #257128 succeeded f5bcde0
💔 Build #257071 failed 21a67c9
💔 Build #257015 failed 73d0c76

cc @dolaru

…c#202906)

kibanamachine · 2024-12-17T16:07:11Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12376852654

kibanamachine · 2024-12-17T16:07:11Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12376852633

kibanamachine · 2024-12-17T16:12:13Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 202906

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2024-12-17T16:12:13Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 202906

Questions ?

Please refer to the Backport tool documentation

…c#202906) (cherry picked from commit ad4e8ef) # Conflicts: # .github/CODEOWNERS

dmlemeshko · 2024-12-18T09:09:35Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…202906) (#204696) # Backport This will backport the following commits from `main` to `8.x`: - [[kbn-scout] Custom event-oriented test reporter & persistence (#202906)](#202906)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  --------- Co-authored-by: David Olaru <[email protected]> Co-authored-by: kibanamachine <[email protected]>

dolaru added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting v9.0.0 test:scout labels Dec 4, 2024

dolaru self-assigned this Dec 4, 2024

wayneseymour reviewed Dec 4, 2024

View reviewed changes

packages/kbn-scout/src/reporting/jest.ts Outdated Show resolved Hide resolved

wayneseymour reviewed Dec 4, 2024

View reviewed changes

scripts/scout.js Show resolved Hide resolved

dolaru marked this pull request as ready for review December 5, 2024 13:51

dolaru requested review from a team as code owners December 5, 2024 13:51

dolaru force-pushed the scout-reporting branch from f5bcde0 to aa76745 Compare December 5, 2024 13:51

Ikuni17 approved these changes Dec 5, 2024

View reviewed changes

pheyos reviewed Dec 6, 2024

View reviewed changes

dmlemeshko reviewed Dec 6, 2024

View reviewed changes

packages/kbn-scout/src/playwright/config/index.ts Show resolved Hide resolved

pheyos reviewed Dec 6, 2024

View reviewed changes

packages/kbn-test/src/functional_test_runner/lib/mocha/reporter/scout_ftr_reporter.ts Show resolved Hide resolved

packages/kbn-scout-reporting/src/reporting/playwright.ts Show resolved Hide resolved

dolaru force-pushed the scout-reporting branch from aa76745 to 9b6eaaa Compare December 7, 2024 00:06

dolaru and others added 13 commits December 9, 2024 11:58

Custom event-oriented test reporter & persistence

62c8217

Move reporting & info to separate packages

d53025a

Addresses circular dependency issues detected during type checks.

[CI] Auto-commit changed files from 'node scripts/notice'

88caa01

Don't take over Playwright's console output

a265a6c

Address review comments

9175f65

Strip ANSI codes from error message/stack trace

db623fe

Fix skipped test detection for FTR reporter

51a4754

Add test file information to report event

538f71b

Remove unused 'event' mappings

15f41fe

Covered by ECS mappings.

Fix reporter mappings

3e65b1e

[CI] Auto-commit changed files from 'node scripts/notice'

0440157

Move data stream name to @kbn/scout-info

865551a

Populate the file.owner field

c92229f

dolaru force-pushed the scout-reporting branch from f772956 to c92229f Compare December 9, 2024 11:59

dolaru requested a review from pheyos December 9, 2024 12:00

pheyos approved these changes Dec 9, 2024

View reviewed changes

Merge branch 'main' into scout-reporting

81289bc

dolaru merged commit ad4e8ef into elastic:main Dec 9, 2024
9 checks passed

dolaru deleted the scout-reporting branch December 9, 2024 15:11

CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 9, 2024

[kbn-scout] Custom event-oriented test reporter & persistence (elasti…

d9b038e

…c#202906)

Samiul-TheSoccerFan pushed a commit to Samiul-TheSoccerFan/kibana that referenced this pull request Dec 10, 2024

[kbn-scout] Custom event-oriented test reporter & persistence (elasti…

15c44b2

…c#202906)

CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024

[kbn-scout] Custom event-oriented test reporter & persistence (elasti…

987af3f

…c#202906)

dolaru added backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) and removed backport:skip This commit does not require backporting labels Dec 17, 2024

dmlemeshko pushed a commit to dmlemeshko/kibana that referenced this pull request Dec 18, 2024

[kbn-scout] Custom event-oriented test reporter & persistence (elasti…

81b2963

…c#202906) (cherry picked from commit ad4e8ef) # Conflicts: # .github/CODEOWNERS

dmlemeshko mentioned this pull request Dec 18, 2024

[8.x] [kbn-scout] Custom event-oriented test reporter & persistence (#202906) #204696

Merged

kibanamachine added the v8.18.0 label Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kbn-scout] Custom event-oriented test reporter & persistence #202906

[kbn-scout] Custom event-oriented test reporter & persistence #202906

dolaru commented Dec 4, 2024 •

edited by kibanamachine

Loading

pheyos left a comment

pheyos left a comment

dolaru commented Dec 9, 2024 •

edited

Loading

pheyos left a comment

elasticmachine commented Dec 9, 2024

API count

ESLint disabled line counts

Total ESLint disabled count

kibanamachine commented Dec 17, 2024

kibanamachine commented Dec 17, 2024

kibanamachine commented Dec 17, 2024

kibanamachine commented Dec 17, 2024

dmlemeshko commented Dec 18, 2024

[kbn-scout] Custom event-oriented test reporter & persistence #202906

[kbn-scout] Custom event-oriented test reporter & persistence #202906

Conversation

dolaru commented Dec 4, 2024 • edited by kibanamachine Loading

Summary

Scout CLI

Scout report

Reporters

pheyos left a comment

Choose a reason for hiding this comment

pheyos left a comment

Choose a reason for hiding this comment

dolaru commented Dec 9, 2024 • edited Loading

pheyos left a comment

Choose a reason for hiding this comment

elasticmachine commented Dec 9, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

ESLint disabled line counts

Total ESLint disabled count

History

kibanamachine commented Dec 17, 2024

kibanamachine commented Dec 17, 2024

kibanamachine commented Dec 17, 2024

💔 All backports failed

Manual backport

Questions ?

kibanamachine commented Dec 17, 2024

💔 All backports failed

Manual backport

Questions ?

dmlemeshko commented Dec 18, 2024

💚 All backports created successfully

Questions ?

dolaru commented Dec 4, 2024 •

edited by kibanamachine

Loading

dolaru commented Dec 9, 2024 •

edited

Loading