Streamline testing every pull request on all platforms

[AndrewGable]

Problem

Over the last year or so, we have seen many regressions introduced in our application. At best, this means that deployers of our application have to spend time tracking the source of the regression and fixing it while it’s still only deployed to our staging environment, and at worst these regressions are deployed to our production app.

Furthermore, we’ve got a single codebase where most code changes affect all platforms (iOS, Android, macOS desktop, web, and mobile web). To help prevent these regressions, we need all pull requests to be tested on all platforms. However, building the app on all these platforms takes a lot of time – even for a simple 1-line change you could easily spend 30-45 minutes building the app on all different platforms, running the testing steps, and collecting screen recordings as proof that you’ve done your due diligence. This takes up a lot of our team’s time, and is also just usually not very fun.

Solution

Build a CI integration that performs a release build of the app on every platform for every pull request. The integration would leave a comment on the pull request when it’s created, including links to the built code:

• On web, this would likely be a URL to the site built from the pull request head, deployed to an S3 bucket (similar to our actual production deploy)
• The desktop app for the PR could be downloaded and installed from the web URL in the previous step
• On Android, maybe this could use the Expo Go client or internal app sharing of some clicking the link or scanning the QR code would install the build on your device or emulator
• On iOS, we could have a solution similar to the one used for Android

This comment would update anytime new code is pushed to the pull request. A good example of a similar CI integration exists in the react-navigation repo: example

[AndrewGable]

@staszekscp and I are working on this. Can you link to this on all future PRs @staszekscp? Thanks!

[AndrewGable]

@staszekscp I tested #13433 and here are the results:

1. Looks like workflow dispatch did not work ❌ https://github.com/Expensify/App/actions/runs/3689373686
2. PR Test did pass ✅ https://github.com/Expensify/App/actions/runs/3689448500/jobs/6245401950
3. However clicking the desktop link does not work, attached error here:

[AndrewGable]

High level: this is working! Going to close for now.