Optimize CI runs with Bazel [Blocked: #59]

[BenHenning]

There are a few things we can do to improve CI runs once Bazel becomes the primary build system for the project, in particular:

1. We could have a multi-stage workflow, where the first set of jobs runs linters, creates the emulator image & related dependencies for Fix #1832 and part of #1831: Add example Espresso emulator test to GitHub CI #1844, and uses Bazel to build the APK and all tests. The outputs from these jobs would be uploaded as artifacts. The second stage would download these artifacts and run the tests in several parallel jobs (likely Robolectric for 1 job and N jobs for the emulator tests).

2. We could create a feedback loop by re-downloading the Bazel artifacts in stage 1 for subsequent pushes for a new copy of the codebase. If done correctly, this could result in an incremental Bazel build with only the latest changes. The dependencies and emulator artifacts don't need to be recreated. This would result in stage 1 becoming very quick for subsequent commits, leading to a significantly shorter overall test time.

3. For push-based workflow initiations, we should use Bazel to determine exactly which test targets are actually affected by the changes in the PR and only run those tests (we can still build everything--this is just an optimization for stage 2 so that it runs faster).

The result of this work, if possible, would be a bit slower initial commit workflow run but significantly faster subsequent runs up to the PR submission. develop & release branch pushes should still run the entire test suite from scratch.

Some caveats to consider:

1. Timestamp changes between CI machine images could cause Bazel to rebuild everything--some investigation will be needed into Bazel build stability

2. We use homebrew currently to install ffmpeg & its dependencies for capturing videos for emulator tests--there's not a clear way to upload/download Homebrew artifacts without just reinstalling them (or if it will actually be a performance improvement)

3. We need to handle the case of many quick, successive commits (especially after the initial PR commit and before that commit's slower preparation workflows have completed)

[BenHenning]

@vinitamurthi any thoughts?

[BenHenning]

/cc @miaboloix in case you have any thoughts

[vinitamurthi]

Overall this looks good to me. I think the emulator image specifically will be cached by the GitHub cache action itself. Not so sure about whether we can build Bazel incrementally with the way the cache works today.

[vinitamurthi]

I also like the idea of step 3 but we'd have to make a script to do that as far as I understand. We could even use that script in the pre-push hooks and run tests just for the affected files

[BenHenning]

Thanks. To be clear, I'm assuming we need to use artifacts as a cache to achieve the goals of this issue, rather than using the existing cache since that doesn't seem to share things across job instances.

[BenHenning]

bazelbuild/bazel#8521 & https://groups.google.com/g/bazel-discuss/c/I9udqWIcEdI/m/iczVgWLOBQAJ describe approaches for doing "dirty target" detection, but it's a bit tricky to do. #1904 is showing that remote caching is not sufficient:

1. Sometimes remote caches inexplicably aren't resulting in many cache hits
2. Remote caching must be made inaccessible to forks

[BenHenning]

Also, per #1904 we're attempting to use remote caching as a simpler way to share dependencies across builds. However, it seems to cache ~20% of the total actions, so there's still a lot of local computation happening (and it seems rather slow, especially for resource processing & data-binding). I think there are fundamental slow parts of the Android build pipeline that we're paying multiplicatively across all of our individual test targets that we only pay once in Gradle builds. This seems to be a significant enough chunk of the overall build time that it's leading to a 3-4x slower test runtime on Bazel over Gradle. This might just be something we have to live with in the beginning until we can:

1. Fragment the codebase and use affected target computations to limit the number of tests each PR runs (though we still need to consider cases when all tests need to run, anyway, e.g. due to changes in the root parts of the build graph)
2. Figure out ways to improve caching (from my early analysis, there are a few resource/info actions done as part of data-binding that are not deterministic and this ruins caching for critical parts of the Android build pipeline)
3. Find ways to enable remote caching for forks since otherwise the builds are going to suffer drastically, maybe by sharing caches across build nodes as proposed above (though we need to keep in mind caches are multiple gigabytes and tens of thousands of files, so this may be infeasible with out-of-the-box actions options)
4. Look into ways to make data-binding faster, or faster data-binding alternatives (maybe Jetpack Compose?). This needs more investigation.
5. If all else fails, introduce remote build execution. This is a last resort since it's expected that this will be much more expensive than the current remote caching we have enabled.

[BenHenning]

So digging into the build reproducibility issue a bit more, it seems like the following two artifacts are causing issues:

• class-info.zip
• layout-info.zip

Both of these are generated for databinding purposes. It seems like they both are either empty or contain some basic JSON or processed XML files. I'm not totally sure if those files are reproducible, but they use structured data and don't seem to contain anything time or random based so they can probably be made to be reproducible if they aren't already. The key problem here seems to be in the zip files themselves: creating consecutive zip files will never result in the same file (https://tanzu.vmware.com/content/blog/barriers-to-deterministic-reproducible-zip-files for context). This actually doesn't seem solvable, so I may need to convince folks to use an alternate structure or not use compression at all (at least for Bazel).

[BenHenning]

Aha, digging into layout-info.zip a bit more, I'm finding out that the main difference is the generated layout info XML files include absolute references to the original (copied) layout file. This, unfortunately, includes a PID or other identifier that changes for each build within the path itself, ruining reproducibility.

[BenHenning]

The class-info.zip differences seem to be tied to just being a zip file that changes (the contents are otherwise the same, as is the file size).

[BenHenning]

I'm reaching out to the databinding team to better understand what our options are here before I delve into the Bazel side to figure out how to improve build reproducibility (and thus, caching).

[BenHenning]

Haven't followed up, but it seems that this is just something that's been missed in the Android build pipeline for Bazel & something we can probably fix ourselves once we have time. Hopefully later in the year. :)

[BenHenning]

While we still need to improve caching for Android actions, per actions/cache#239 it seems like we might be able to get good mileage out of the GitHub actions cache by sharing a cache across all Bazel builds and benefiting greatly from Bazel's incremental building functionality. The cache will grow unbounded until we reach an established threshold, then we'll reset it (which effectively just means on occasion different actions will trigger a rebuild of the cache and take a bit longer to fill in the cache pieces).

Due to Bazel being a hermetic build system, the natural data race in trying to backfill the cache should also be fine (e.g. if multiple actions detect a cache reset and each start from scratch then the last one to upload wins, and all later actions benefit from it).

From local testing, I'm guessing we'll see a 3x-10x performance improvement depending on how expensive uploading/downloading the cache is (which for a 4GB cache might actually take a lot of time). However, speeding up the builds may let us reduce sharding by running blocks of tests in each job rather than having one job per test. In that way we'll share the one-time cost of downloading/uploading the cache, and potentially reduce the number of distinct phases from ~32 (160 tests / 5 concurrent tests) to ~2-3 which may result in tests taking 10-40 minutes to run instead of 8-16 hours.

Note that if we go with this approach we probably will also want to block the test runs on the Bazel build since that will act as an effective "prime cache" mechanism to further reduce potential OOMs when building & running large numbers of tests.

[BenHenning]

Per #3513 we should also consider reverting the SDK pinning occurring in that run or find a way to cache it to avoid the ~1 minute cost to download the SDK (caching is probably not viable since the SDK is quite large).

[BenHenning]

Another potential improvement from the above: we could modify the existing SDK already included in the CI environment by removing the preinstalled build tools & just downloading the ones we need. That would be much faster, but it puts us in a weird semi-hermetic SDK environment which doesn't seem ideal.