Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(tests): add capturing and uploading of screen capture of ui tests #4871

Merged
merged 18 commits into from
Feb 21, 2025

Conversation

philprime
Copy link
Contributor

@philprime philprime commented Feb 19, 2025

  • Enables automatic video capture of ui tests
  • Adds upload of the captured videos on ui test failure

See #4848 for more context and details how this works.

My intention with this PR is to have visual assitance why our UI tests are failing without changes (flakiness).

Things we need to consider:

  • Maybe we should change the path of the output xcresult so that it isn't included in the raw test log upload
  • We need to decide if we want a video for all results, or only for failing results

#skip-changelog

@philprime philprime self-assigned this Feb 19, 2025
@philprime philprime changed the base branch from main to philprime/introduction-xctestplan February 19, 2025 15:38
@philprime
Copy link
Contributor Author

Additional context on the directory behaviour:

https://docs.fastlane.tools/advanced/fastlane/#directory-behavior

Copy link

github-actions bot commented Feb 19, 2025

Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1231.53 ms 1252.73 ms 21.20 ms
Size 22.31 KiB 820.51 KiB 798.20 KiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
3d8532d 1225.27 ms 1238.80 ms 13.53 ms
e072ad1 1232.43 ms 1245.51 ms 13.08 ms
5b6694b 1221.71 ms 1259.06 ms 37.35 ms
efd8ddb 1208.88 ms 1234.12 ms 25.24 ms
c4e9093 1219.04 ms 1236.00 ms 16.96 ms
6e342ac 1216.02 ms 1232.88 ms 16.86 ms
3437454 1235.54 ms 1244.82 ms 9.28 ms
228a909 1248.92 ms 1267.42 ms 18.50 ms
fc163f5 1205.20 ms 1222.32 ms 17.12 ms
addcf42 1247.33 ms 1253.58 ms 6.25 ms

App size

Revision Plain With Sentry Diff
3d8532d 22.85 KiB 414.09 KiB 391.24 KiB
e072ad1 21.58 KiB 625.83 KiB 604.24 KiB
5b6694b 20.76 KiB 426.11 KiB 405.34 KiB
efd8ddb 21.58 KiB 678.19 KiB 656.61 KiB
c4e9093 21.58 KiB 671.30 KiB 649.72 KiB
6e342ac 20.76 KiB 436.66 KiB 415.90 KiB
3437454 22.85 KiB 408.88 KiB 386.03 KiB
228a909 20.76 KiB 425.71 KiB 404.95 KiB
fc163f5 20.76 KiB 436.30 KiB 415.54 KiB
addcf42 22.85 KiB 413.42 KiB 390.57 KiB

Previous results on branch: philprime/ui-test-automatic-capturing

Startup times

Revision Plain With Sentry Diff
59b4bb8 1220.02 ms 1236.10 ms 16.08 ms

App size

Revision Plain With Sentry Diff
59b4bb8 22.32 KiB 820.51 KiB 798.19 KiB

Copy link

codecov bot commented Feb 19, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.184%. Comparing base (eff4df8) to head (cb115de).
Report is 5 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##              main     #4871       +/-   ##
=============================================
- Coverage   92.272%   92.184%   -0.088%     
=============================================
  Files          659       659               
  Lines        77331     77324        -7     
  Branches     27995     27206      -789     
=============================================
- Hits         71355     71281       -74     
- Misses        5880      5952       +72     
+ Partials        96        91        -5     

see 19 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eff4df8...cb115de. Read the comment docs.

@philprime
Copy link
Contributor Author

Example from the failed UI test:

Screenshot 2025-02-19 at 17 27 45

Btw seems like there's a race condition happening

@philprime philprime marked this pull request as ready for review February 19, 2025 16:31
Base automatically changed from philprime/introduction-xctestplan to main February 20, 2025 10:00
Copy link
Member

@philipphofmann philipphofmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, let's give this a shot 🚀

@philprime philprime merged commit a781aec into main Feb 21, 2025
48 of 52 checks passed
@philprime philprime deleted the philprime/ui-test-automatic-capturing branch February 21, 2025 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants