☂️ Integration tests to run on-device

[gnprice]

For most of the app, we have no need for integration tests that run on an actual Android or iOS device. Flutter provides excellent support for widget testing, and that already allows us to test our UIs far more thoroughly than our practice was in the legacy zulip-mobile RN app.

But there are some places where it'd be good for us to have some integration tests.

• The key criterion for where this applies is that it involves interactions with the host platform. For anything that's about pure Dart code, including widgets, the tests we write using package:flutter_test can cover it just fine.
• A secondary criterion is that it's something especially important to users' experience. In principle we'd have integration tests for everything that can't be covered by widget tests… but in practice, integration tests are expensive to write and to run, so we're going to have to prioritize.

Flutter upstream's support for integration tests is evolving; see chat discussion. So one thing I'm hoping is that by the time we turn to investing in integration tests, some of those improvements will have come to fruition and it'll be smoother than it would be today.

Examples of where integration tests would be desired:

• Notifications on Android: Write on-device integration tests for Android notifications #758

  This involves a significant amount of our code using a significant amount of platform API surface area. It's therefore vulnerable to quirks in the platform's behavior which we don't yet know about.

  A prime example of this was notif: Enable autoCancel flag for summary notification #755.

  As that example also illustrates, a key part of the value of integration tests will be running them on old OS versions. More generally, we'll want to run them on a variety of platforms: so also different Android vendors, and different models of iPhone (like with and without a notch, or with and without biometric sensors for testing auth flows).

• Notifications on iOS. We currently have less complexity there to test so it's less of a priority than Android notifications, but still vulnerable to platform quirks.

• Login flows using web auth (e.g. authenticating with Google or GitHub).

  In general, login and other onboarding flows are a high priority for automated testing (both integration tests and widget tests) because on the one hand they're essential for a user to successfully use Zulip, and on the other they tend not to get naturally tested by our own daily use and that of our beta users, because we already have the app installed and only occasionally re-install it or log into a new account.

• Install and first launch of the app, including permission flows in particular.

  In practice this is likely to get covered as part of the work toward covering any of the other areas, because we'll need to install the app on a device in order to test it.

This is an umbrella issue; we'll file issues for specific areas of tests on the way toward implementing them.

[gnprice]

As @rajveermalviya points out at #755 (comment), we might not want to include these in every CI run (i.e. on every revision of every PR, and every push to main), because they'll be slow and will require firing up a battery of different devices.

We might instead have some scheme where they run periodically on main, and where we can explicitly choose to run them on a given PR or a specific commit from main. And we'll definitely want a straightforward way to run them locally on a device (physical or emulated) that you have lying around; that'll be essential for writing these tests, as well as debugging.