♻️ use playwright for e2e

[thomas-lebeau]

Important

This is a POC / WIP started during performance friday

Motivation

Use playwright for e2e testing.

Changes

Speed 🚀

	Local	CI	CI with BrowserStack*
Before	n/a	~8m	~24m
After	~21s	~3m75s	n/a

(*): BrowserStack timing does not account for the time spend waiting for BS resources

new useful commands:

• yarn test:e2e --ui run playwright locally in UI mode
• yarn test:e2e test/e2e/scenario/rum/init.scenario.ts to run one test file
• yarn test:e2e recorder to run all test files that have recorder in the name
• BS_USERNAME=johdoe_abcD BS_ACCESS_KEY=askldjhfn yarn test:e2e:bs to run the tests in Browserstack
• yarn test:e2e --debug to playwright in debug mode

new useful method:

• await page.pause() to use with --debug
• createTest.only('foo', () => { /* ... */}) to run only this one test, also particularly useful in debug mode

And everything together:

• BS_USERNAME=johdoe_abcD BS_ACCESS_KEY=askldjhfn yarn test:e2e:bs --ui --debug recorder

status:

• developer-extension/developerExtension.scenario.ts
• recorder/recorder.scenario.ts
• recorder/shadowDow.scenario.ts
• recorder/viewports.scenario.ts
• rum/actions.scenario.ts
• rum/errors.scenario.ts
• rum/init.scenario.ts
• rum/resources.scenario.ts
• rum/s8sInject.scenario.ts
• rum/sessions.scenario.ts
• rum/tracing.scenario.ts
• rum/views.scenario.ts
• rum/vitals.scenario.ts
• eventBridge.scenario.ts
• logs.scenario.ts
• microfrontend.scenario.ts
• sessionStore.scenario.ts
• telemetry.scenario.ts
• trackingConsent.scenario.ts
• transport.scenario.ts

todo:

• CI/CD
• other scenarios
• browserstack integration
• ???

Testing

Locally, run yarn test:e2e --ui

• Local
• Staging
• Unit
• End to end

---

I have gone over the contributing documentation.

[cit-pr-commenter]

Bundles Sizes Evolution

📦 Bundle Name	Base Size	Local Size	𝚫	𝚫%	Status
Rum	145.98 KiB	145.98 KiB	0 B	0.00%	✅
Logs	51.06 KiB	51.06 KiB	0 B	0.00%	✅
Rum Slim	104.77 KiB	104.77 KiB	0 B	0.00%	✅
Worker	24.50 KiB	24.50 KiB	0 B	0.00%	✅

🚀 CPU Performance

Action Name	Base Average Cpu Time (ms)	Local Average Cpu Time (ms)	𝚫
addglobalcontext	0.004	0.002	-0.002
addaction	0.219	0.039	-0.180
addtiming	0.002	0.001	-0.001
adderror	0.124	0.058	-0.066
startstopsessionreplayrecording	0.019	0.014	-0.006
startview	0.903	0.394	-0.509
logmessage	0.062	0.024	-0.038

🧠 Memory Performance

Action Name	Base Consumption Memory (bytes)	Local Consumption Memory (bytes)	𝚫 (bytes)
addglobalcontext	27.35 KiB	28.00 KiB	664 B
addaction	55.37 KiB	54.54 KiB	-850 B
addtiming	26.34 KiB	25.70 KiB	-654 B
adderror	61.62 KiB	57.03 KiB	-4700 B
startstopsessionreplayrecording	26.09 KiB	24.05 KiB	-2097 B
startview	415.52 KiB	419.27 KiB	3.74 KiB
logmessage	58.19 KiB	58.62 KiB	437 B

🔗 RealWorld

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 25.00000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 93.67%. Comparing base (1440c99) to head (5fe7fe2).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
packages/rum/test/mutationPayloadValidator.ts	25.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3159      +/-   ##
==========================================
- Coverage   93.70%   93.67%   -0.03%     
==========================================
  Files         290      290              
  Lines        7654     7657       +3     
  Branches     1745     1748       +3     
==========================================
+ Hits         7172     7173       +1     
- Misses        482      484       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thomas-lebeau]

Unfortunately playwright recommends to NOT cache browsers

This is not ideal because it takes about as much time to install dependencies and browsers than running the tests in the CI.
I believe we can get around that by having pre-build Docker image.