Fix #1832 and part of #1831: Add example Espresso emulator test to GitHub CI #1844
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Per https://github.com/marketplace/actions/android-emulator-runner it seems Android emulator tests are supported in GitHub actions, but only when run in Mac OS (due to Linux requiring KVM, and KVM not being broadly available in the machines running GitHub actions). This PR introduces a setup only for SplashActivityTest since we should be adding tests one at a time until we can guarantee broad stability. SplashActivityTest seems pretty reliable, so we can start with this one.
Switch Pixel XL with Nexus 6 and run all tests instead of a specific one to try and debug why they're failing.
Android SDK needs to be set up in the path correctly to build instrumented tests.
Temporarily remove other workflows, and add additional logging options for Gradle.
Add additional logs to investigate the build failure (possibly related to actions/runner-images#60 and DroidKaigi/conference-app-2020#710).
Introduce script to simplify the workflow setup & avoid pitfalls from quote escaping.
…f github.com:oppia/oppia-android into add-splash-activity-test-as-first-emulator-ci-action
Try different action for setting up Android SDK and add additional debug steps.
Add missing command to set up platform tools to make ADB available.
Use Java 8 since it may be required for the Android SDK manager (and for emulator tests--we don't actually use Java 9 for these).
Remove original setup-android action.
BenHenning
added a commit
to BenHenning/action-android
that referenced
this pull request
Sep 16, 2020
Clarify the purpose of each step, and add a missing step that's needed for ADB to work correctly (``sdkmanager platform-tools``) per https://github.com/Malinskiy/action-android/blob/5c20fa7/.github/workflows/test-macos.yml#L16 and Malinskiy#22 (comment). I noticed this when trying to get oppia/oppia-android#1844 to work.
Limit Mac OS version to 10.15 since that's the version we're starting with, and try to run the emulator script directly to see if the environment variables are being set correctly.
Remove sudo to see if it's causing the environment variables to reset.
support for video recordings of running tests. This needs to be cleaned up in a subsequent commit. Note that this script makes use of an ./emulator_tests working directory that was also added to .gitignore.
tests script (+ add documentation).
…f github.com:oppia/oppia-android into add-splash-activity-test-as-first-emulator-ci-action
The initial whitelist is just SplashActivityTest.
during script development.
Fix environment variable export in a way that's compatible with sh.
unnecessary ADB step. Update run_emulator_tests to fail overall if any tests fail. Temporarily run only one test in the only active test suite (SplashActivityTest) to speed up the CI run for debugging.
Debug whether GitHub is defining an array env variable correctly.
Add parsing code for AVD device settings.
Use correct delimiter for device settings lists.
Fix syntax error. Remove parentheses from device parameters since it's only parsing based on spaces.
Add debugging for dev settings vars.
Fix device settings array variable reference.
BenHenning
changed the title
Also, some clean up & small fixes. Added more useful list of devices that we can test on.
…f github.com:oppia/oppia-android into add-splash-activity-test-as-first-emulator-ci-action + post-merge fixes. Conflicts: .github/workflows/main.yml
Specifically: remove debug lines, remove StateFragmentTest, start Gradle build in parallel, and add proper support for multiple devices by segmenting test output to its own directory.
And other miscellaneous documentation improvements.
SDK 29 has some occasional mul-overflow errors, so switching to SDK 28 for now for hopefully better stability.
Also, add timeout so that wait_for_emulator doesn't hang.
Install coreutils instead of timeout (for timeout functionality).
Re-add platforms SDK install.
Quiet sdkmanager output. Try again to use SDK 29 since 28 seems to unexpectedly have pathing issues.
Redirect stdout to /dev/null for sdkmanager since it probably works a bit better than using grep.
Attempt to run SplashActivityTest on all support devices at once.
Log build output in case something fails & upload it.
Fix upload artifacts naming & pathing, and turn off fail-fast for matrix expansions of jobs.
Set access token for cancellation action.
Only block on the Gradle build if it isn't already running.
Malinskiy
pushed a commit
to Malinskiy/action-android
that referenced
this pull request
Nov 22, 2020
Clarify the purpose of each step, and add a missing step that's needed for ADB to work correctly (``sdkmanager platform-tools``) per https://github.com/Malinskiy/action-android/blob/5c20fa7/.github/workflows/test-macos.yml#L16 and #22 (comment). I noticed this when trying to get oppia/oppia-android#1844 to work.
|
From talking to folks this week, I'm thinking about repurposing this PR to focus on just running a few of the essential test suites, specifically:
This is far from comprehensive, but it at least introduces some CI coverage for key scenarios to offset the very long test runs. I'll probably wait until #1904 is merged, first (due to both PRs significantly impacting CI load). |
BenHenning
added
the
PR don't merge - NEEDS UPDATE
|
Due to lack of time to work on this, and the desire to use a cloud platform for Espresso tests, closing this for now (though it still may be a nice reference). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix #1832
Fix part of #1831 (since only one test is being added).
Per https://github.com/marketplace/actions/android-emulator-runner it seems Android emulator tests are supported in GitHub actions, but only when run in Mac OS (due to Linux requiring KVM, and KVM not being broadly available in the machines running GitHub actions). Note that this PR introduces custom emulator management rather than using an existing action since this lets us better parallelize tasks, and lets us have predefined device profiles to run on.
This PR introduces a setup only for SplashActivityTest since we should be adding tests one at a time until we can guarantee broad stability. SplashActivityTest seems pretty reliable, so we can start with this one. That being said, this was tested with multiple test suites (see sample runs below). Features of the new job includes:
Note that the screen recording functionality was a bit tricky to implement. It currently works by sequentially logging up to 10 back-to-back videos using
adb shell screenrecord(which can only record at most 3 minutes), then pieces them together in one video. I spent some time trying to get this to work using an in-app test fixture, but doing so requires a rooted device & further effort. For an idea on what that work looked like, see: 5368910.Some other miscellaneous things to note:
develop(that way we can at least revert PRs if they break something on Espresso after getting merged in).ReactiveCircus/android-emulator-runner#90 is tracking how to use an environmental variable for defining the test suite instead of needing to specify it in-line. I decided to custom manage the emulator & AVD setup, instead, since it provides more flexibility.
Sample runs: