diff --git a/docs/contributors/code/react-native/README.md b/docs/contributors/code/react-native/README.md index f5b8fc79aba865..10df7268ddddb6 100644 --- a/docs/contributors/code/react-native/README.md +++ b/docs/contributors/code/react-native/README.md @@ -20,8 +20,8 @@ Also, the mobile client is packaged and released via the [official WordPress app If you encounter a failed Android/iOS test on your pull request, we recommend the following steps: -1. Re-running the failed GitHub Action job ([guide for how to re-run](https://docs.github.com/en/actions/configuring-and-managing-workflows/managing-a-workflow-run#viewing-your-workflow-history)) - This should fix failed tests the majority of the time. Cases where you need to re-run tests for a pass should go down in the near future as flakiness in tests is actively being worked on. See the following GitHub issue for updated info on known failures: https://github.com/WordPress/gutenberg/issues/23949 -2. You can check if the test is failing locally by following the steps to run the E2E test on your machine from the [mobile getting started guide](/docs/contributors/code/react-native/getting-started-react-native.md#ui-tests), with even more relevant info in the [relevant directory README.md](https://github.com/WordPress/gutenberg/tree/HEAD/packages/react-native-editor/__device-tests__#running-the-tests-locally) +1. Re-running the failed GitHub Action job ([guide for how to re-run](https://docs.github.com/en/actions/configuring-and-managing-workflows/managing-a-workflow-run#viewing-your-workflow-history)) - This should fix failed tests the majority of the time. +2. You can check if the test is failing locally by following the steps to run the E2E test on your machine from the [E2E testing documentation](/packages/react-native-editor/__device-tests__/README.md). 3. In addition to reading the logs from the E2E test, you can download a video recording from the Artifacts section of the GitHub job that may have additional useful information. 4. Check if any changes in your PR would require corresponding changes to `.native.js` versions of files. 5. Lastly, if you're stuck on a failing mobile test, feel free to reach out to contributors on Slack in the #mobile or #core-editor chats in the WordPress Core Slack, [free to join](https://make.wordpress.org/chat/). @@ -32,4 +32,4 @@ Follow the instructions in [Native mobile testing](/docs/contributors/code/react ## Internationalization (i18n) -Further information about this topic can be found in the [React Native Internationalization Guide](/docs/contributors/code/react-native/internationalization-guide.md). \ No newline at end of file +Further information about this topic can be found in the [React Native Internationalization Guide](/docs/contributors/code/react-native/internationalization-guide.md). diff --git a/docs/contributors/code/react-native/getting-started-react-native.md b/docs/contributors/code/react-native/getting-started-react-native.md index 53d5f7eee5a124..b014ed0f31f008 100644 --- a/docs/contributors/code/react-native/getting-started-react-native.md +++ b/docs/contributors/code/react-native/getting-started-react-native.md @@ -141,34 +141,8 @@ Then, open `chrome://inspect` in Chrome to attach the debugger (look into the "R This project is set up to use [jest](https://facebook.github.io/jest/) for tests. You can configure whatever testing strategy you like, but jest works out of the box. Create test files in directories called `__tests__` or with the `.test.js` extension to have the files loaded by jest. See an example test [here](https://github.com/WordPress/gutenberg/blob/HEAD/packages/react-native-editor/src/test/api-fetch-setup.test.js). The [jest documentation](https://facebook.github.io/jest/docs/en/getting-started.html) is also a wonderful resource, as is the [React Native testing tutorial](https://facebook.github.io/jest/docs/en/tutorial-react-native.html). -## UI Tests +## End-to-End Tests -This repository uses Appium to run UI tests. The tests live in `__device-tests__` and are written using Appium to run tests against simulators and real devices. To run these you'll need to check off a few things: +In addition to unit tests, the Gutenberg for Mobile Apps (GMA) project relies upon end-to-end (E2E) tests to automate testing critical flows in an environment similar to that of an end user. We generally prefer unit tests due to their speed and ease of maintenance. However, assertions that require OS-level features (e.g. complex gestures, text selection) or visual regression testing (e.g. dark mode, contrast levels) we use E2E tests. -- When running the tests, you'll need to ensure the Metro bundler (`npm run native start`) is not running. -- [Appium CLI](https://github.com/appium/appium/blob/1.x/docs/en/about-appium/getting-started.md) installed and available globally. We also recommend using [appium-doctor](https://github.com/appium/appium-doctor) to ensure all of Appium's dependencies are good to go. You don't have to worry about starting the server yourself, the tests handle starting the server on port 4723, just be sure that the port is free or feel free to change the port number in the test file. -- For iOS a simulator should automatically launch but for Android you'll need to have an emulator _with at least platform version 8.0_ fired up and running. - -Then, to run the UI tests on iOS: - -```sh -npm run native test:e2e:ios:local -``` - -and for Android: - -```sh -npm run native test:e2e:android:local -``` - -To run a single test instead of the entire suite, use `npm run native device-tests:local`. Here's an example that runs only `gutenberg-editor-paragraph.test.js`: - -```sh -npm run native test:e2e:android:local gutenberg-editor-paragraph.test.js -``` - -Note: You might experience problems that seem to be related to the tests starting the Appium server, e.g. errors that say `Connection Refused`, `Connection Reset` or `The requested environment is not available`. If so, you can manually start the Appium server via `npm run native appium:start`, and (optionally) comment out related code in the `beforeAll` and `afterAll` block. - -For a more detailed outline of the UI tests and how to get started writing one, please visit the [UI Test documentation](/packages/react-native-editor/__device-tests__/README.md) and our [contributing guide](/packages/react-native-editor/__device-tests__/CONTRIBUTING.md). - -You might want to use Visual Studio Code as an editor. The project includes the configuration needed to use the above codestyle and linting tools automatically. +The E2E tests are found in the [`packages/react-native-editor/__device-tests__`](/packages/react-native-editor/__device-tests__) directory. Additional documentation on running and contributing to these tests can be found in the [tests directory](/packages/react-native-editor/__device-tests__#readme). diff --git a/packages/react-native-editor/__device-tests__/CONTRIBUTING.md b/packages/react-native-editor/__device-tests__/CONTRIBUTING.md deleted file mode 100644 index eba49df6949838..00000000000000 --- a/packages/react-native-editor/__device-tests__/CONTRIBUTING.md +++ /dev/null @@ -1,44 +0,0 @@ -Writing a UI test? Great! šŸ˜¬ This guide is here to help fill in some of the blanks of how the tests are written now and how you can add a new one. - -You can find our on-device UI tests in the `__device-tests__` folder and that's where all of the code for that really lives. -The test suite follows a sort of [Page Object Pattern](https://webdriver.io/docs/pageobjects.html), the `__device-tests__/pages/editor-page.js` manages all interactions with the pages and the `__device-tests__/gutenberg-editor.test.js` actually uses the functions made available via the Page Object `EditorPage` to drive the test cases. At the time of writing this, all the tests live there but as the suite gets large it might be better to manage different classes of tests in different files. - -So what does the process for writing a test look like? Here are some steps that I hope can help make this easier, - -### First, define the scenario - -- What are the actions that need to take place here? Walk through the scenario and manually to have an idea of what the test steps will need to do, the elements you'll need to interact with and how you're going to need to interact with them. I found it helps to properly define the steps taken in the scenario and the different user interactions that are needed to accomplish it. - -You'd just add a new scenario to the test file as well that would look something like, - -```javscript -it( 'should be able to do something', async () => { - // Code to do something... -} ); -``` - -That first parameter in the block above is where you'd put a short description of the scenario while the next parameter is the code you'd like to execute. - -### Second, figure out how to find the elements - -- The UI tests rely on locator strategies to identify elements... There's a number of locator strategies available to use and [this blog post](https://saucelabs.com/resources/blog/advanced-locator-strategies) describes in a little more detail what a few of these are and how to use them. You'll need to start thinking about what locator strategy you'll need to use to find the elements you need if it isn't already available. -- The preferred strategy is the accessibility identifier and in a lot of cases this might not be possible and you'll have to resort to other less robust alternatives such as XPath. - -There's a few tools you have available to figure out what you need. - -For Android, you can fire up the app and then within Android Studio select `Tools -> Layout Inspector` which will then open up a `.li` file which you can then use to inspect various areas of the app. - -For iOS, you can also fire up and use the accessibility inspector, which is an app that should come available on your OSX machine. From there you can choose the process running your simulator and inspect various areas of the app. - -Alternative for both of these platforms and for an interface to simulate the commands I'd recommend [Appium Inspector](https://github.com/appium/appium-inspector/releases), a great tool for inspecting the view hierarchy and interacting with elements on screen as your test would. In order to connect the Appium Inspector, you'll need to start the Appium server manually by running `npm run native appium:start` and then configure Appium Inspector with the appropriate capabilities. You can find the capabilities in `__device-tests__/helpers/caps.js`. - -Using one or a combination of these tools will make it much easier to identify what locator strategy you're going to use or which elements need accessibility identifiers to ease the search process without affecting VoiceOver features. - -### Finally, once you've figured out how you're going to find the elements - -- You'll write any functions needed to interact with the page in the `EditorPage` page object and then call those interactions within the test. The code you'll need to write to actually do the finding will use a combination of - -- Appium's spec https://github.com/appium/appium/blob/1.x/docs/en/about-appium/intro.md which you can find examples of a variety of functions under the commands tab -- WebDriver I/O Appium protocols https://webdriver.io/docs/api/appium.html which provides examples and descriptions of what those look like. - -It takes some getting used to but looking at the existing code should be helpful in identifying common commands that it'd help to be familiar with. diff --git a/packages/react-native-editor/__device-tests__/README.md b/packages/react-native-editor/__device-tests__/README.md index 81a2d4d794cb47..db649a448f684a 100644 --- a/packages/react-native-editor/__device-tests__/README.md +++ b/packages/react-native-editor/__device-tests__/README.md @@ -1,84 +1,120 @@ -# Overview +# Gutenberg for Mobile Apps E2E Tests -We use [appium](http://appium.io/) combined with [SauceLabs](https://saucelabs.com/) as an on-device testing solution for covering writing flows using Gutenberg blocks. +The Gutenberg for Mobile Apps (GMA) project maintains a suite of automated end-to-end (E2E) tests that uses [Appium](https://appium.io/docs/en/2.1/) to facilitate UI automation. The E2E tests run on iOS simulators and Android emulators to simulate an environment similar to that of an end user. This document provides an overview for running these tests on your local development computer. -Appium is built on the idea that testing native apps shouldn't require including an SDK or recompiling your app. And that you should be able to use your preferred test practices, frameworks, and tools. Appium is an open source project and has made design and tool decisions to encourage a vibrant contributing community. +## Setup -SauceLabs is a cloud hosting platform that provides access to a variety of simulators, emulators and real devices. +Before setting up Appium, the required iOS and Android dependencies must be installed. -## Getting set up to run the tests +> **Note** +> The required dependencies change overtime. We do our best to update the scripts documented below, but it is best to review the [Appium capabilities](https://github.com/WordPress/gutenberg/blob/trunk/packages/react-native-editor/__device-tests__/helpers/caps.js) configuration to identify the currently required `deviceName` and `platformVersion` for each of the iOS and Android platforms. -### Emulators & Simulators +### iOS -> **Note** -> -> Visual regression tests that rely upon screenshots require specific devices and OS versions for Android and iOS, respectively, otherwise the tests will fail due to subtle OS differences. To run or update visual regression tests, install the emulators/simulators listed in the test suite [configuration files](https://github.com/WordPress/gutenberg/blob/trunk/packages/react-native-editor/__device-tests__/helpers/caps.js#L30-L31). +- Complete the [React Native Getting Started](https://reactnative.dev/docs/environment-setup) guide, which covers installing and setting up Xcode. +- Open [Xcode settings](https://developer.apple.com/documentation/xcode/installing-additional-simulator-runtimes#Install-and-manage-Simulator-runtimes-in-settings) to install the iOS 16.2 simulator runtime. +- Create the required simulators by running the follow scripts in your shell: + - `xcrun simctl create "iPhone 13" "iPhone 13" "com.apple.CoreSimulator.SimRuntime.iOS-16-2"` + - `xcrun simctl create "iPad Pro (9.7-inch)" "iPad Pro (9.7-inch)" "com.apple.CoreSimulator.SimRuntime.iOS-16-2"` + +### Android + +- Complete the [React Native Getting Started](https://reactnative.dev/docs/environment-setup) guide, which covers installing and setting up Android Studio and the Android SDK. +- Open Android Studio and [create an emulator](https://developer.android.com/studio/run/managing-avds) for a Pixel 3 XL running Android 11.0 with the ā€œEnable Device Frameā€ option disabled. + +### Appium + +The GMA project provides a script to set up the testing environment, installing the necessary Appium dependencies. + +```shell +npm run native test:e2e:setup +``` + +## Running Tests + +The process for running the E2E tests differ in subtle, important ways for iOS and Android respectively. Before running E2E tests, if you previously ran `npm run native start:reset`, **ensure the Metro development server is no longer running.** + +### iOS + +The following script will launch the correct iOS simulator and run all of the E2E tests. + +```shell +npm run native test:e2e:ios:local +``` -**iOS:** If you have completed the [React Native Getting Started](https://reactnative.dev/docs/environment-setup) guide you should already have Xcode installed and the simulators set up. +### Android -**Android:** If you have completed the [React Native Getting Started](https://reactnative.dev/docs/environment-setup) guide you should already have Android Studio installed and the Android SDK installed. You'll also need to have the [Android Emulator](https://developer.android.com/studio/run/emulator) installed and set up. The emulator needs to be running prior to running the tests. +Before running E2E tests on Android, launch the emulator using the following script. -### Physical Devices +```shell +emulator -avd Pixel_3_XL_API_30 -noaudio +``` -**iOS:** Currently unsupported. +The following script will run all of the E2E tests. -**Android:** Connect your device to your computer via USB and [enable USB debugging](https://developer.android.com/studio/debug/dev-options). +```shell +npm run native test:e2e:android:local +``` -## Running the tests locally +## Filtering Test Runs -TL;DR: to run the tests locally ensure metro isn't running and then run `npm run native test:e2e:ios:local` and `npm run native test:e2e:android:local` for the desired platform. +By default, the E2E test scripts run all tests in the suite. While this is helpful for verifying all tests pass, the entire test suite takes a long time to complete. Running all tests is not very effective for local development. -Those commands include the process to build a testable version of the app with these steps: +You can filter which test runs by one of two ways: -1. Create the JS bundle via `test:e2e:bundle:(ios|android)` -1. Compile the native app code via `test:e2e:build-app:(ios|android)` -1. Call the test runner via `device-tests:local` +- Passing a file name argument to the CLI script. +- Leveraging Jestā€™s interactive watch mode. -Once the JS bundle and native app code are created they can be re-used by the test runner `device-tests:local` in subsequent runs. While writing tests: +```shell +# Run a single test file on iOS +npm run native test:e2e:ios:local gutenberg-editor-paragraph.test.js -- If you only changed the native app code you can run only `test:e2e:build-app:(ios|android)` followed by `device-tests:local` -- If you only changed JS app code you can run only `test:e2e:bundle:(ios|android)` followed by `device-tests:local` -- If you didn't change native or JS app code but only have modified e2e tests under `__device-tests__` you only need to re-rerun `device-tests:local` which uses the pre-built native and JS app code -- If it's the case you don't want to run the - full suite and want to run a specific file or files you can run `TEST_RN_PLATFORM=android npm run native device-tests ` where the pattern can just be the file name. +# Enable watch mode on iOS +npm run native test:e2e:ios:local -- --watch +``` -### Debugging +## Speeding Up Test Runs -There's a debug variant of the command: `device-tests:debug`, which starts a Node.js process that listens for a debugging client. You can use any inspector client at that point to attach and add breakpoints. More information about that can be found here: https://nodejs.org/en/docs/guides/debugging-getting-started/ +The `native test:e2e:(android|ios):local` script performs several steps via associated npm scripts. -You can also write `debugger;` in the JS code in any line to add a breakpoint. +- Create the JavaScript bundle: `test:e2e:bundle:(ios|android)` +- Compile the app: `test:e2e:build-app:(ios|android)` +- Launch the test runner: `device-tests:local` -### Starting the Appium Server +While we must run all of these at least once to produce a testable app, it is often not necessary to run them each multiple times over while modifying or writing tests. To speed up multiple test runs, you can invoke the individual scripts as needed. -One of the Caveats to using Appium is the need for the Appium server to be running to interact with the Simulator or Device through Webdriver, as a result the appium server will need to be started before running the tests. +If you only modified the native app code, you can run `test:e2e:build-app:(ios|android)` followed by `device-tests:local`. +If you only modified JavaScript app code, you can run `test:e2e:bundle:(ios|android)` followed by `device-tests:local`. +If you only modified E2E tests code, you can run `device-tests:local`. -To make the entire process easier in the `beforeAll` block of the tests an Appium instance is fired up on a default port of 4723. If you already have something running on that port and would rather not stop that you can change the port within the code that starts that up. At the moment that port number is referenced from the config located at `__device-tests__/helpers/serverConfigs.js`. +By default `device-tests:local` runs tests for Android. To run tests on iOS, you can prefix the script with the `TEST_RN_PLATFORM` environment variable. -The process is killed in the `afterAll` block but at the time of writing this there's a small chance some errors might cause it not to get there so it might be best to kill the process yourself if you think something is up. The server output when running the tests are written to `appium-out.log`, this can provide useful information when debugging the issues with the tests. +```shell +TEST_RN_PLATFORM=ios npm run native device-tests:local +``` -If the `beforeAll` and `afterAll` functionality is not working correctly, you can start the Appium server manually by running `npm run native appium:start`. +## Debugging Tests -### WebDriver capabilities +Much like other development servers, values outputted to the console via `console.log` should display in the Jest test runner server log. -Appium uses a config object that contains `capabilities` to define how it will connect to a simulator or device, this object is currently located in `__device-tests__/helpers/caps.js` and then referenced when firing up the driver. There are two values that I think are important to know and that's +Occasionally, it is helpful to inspect breakpoints during the execution of a test.The GMA project includes a `device-tests:debug` script which sets the `--inspect` flag required for attaching an inspector. Additional details on attaching an inspector can be found in Node.jsā€™ [Debugging Guide](https://nodejs.org/en/docs/guides/debugging-getting-started). -- `platformVersion` which is the platform version of a connected adb device. e.g `9.0` for Android or `12.2` for iOS. The version used here is upper bounded by the max allowed on CI but feel free to change this value locally as needed. -- `app` which is the absolute path to the `.app` or `.apk` file or the path relative to the **Appium root**. It's important to note that when using the relative paths it's not to the project folder but to the appium server, since by default we start up appium in the project root when running the paths appear relative to the root but if you were using another instance of the Appium server the relative path would need to come from there. +## Writing Tests -A full spec on the capabilities can be found [here](https://github.com/appium/appium/blob/1.x/docs/en/writing-running-appium/caps.md). If you'd like to change configurations like -what port appium runs on or what device or emulator the tests should be executed on that file would be where you'd like to make that update. +Jest is the test runner for the E2E test suite. Reviewing the [Jestā€™s Introduction](https://jestjs.io/docs/getting-started) documentation provides an overview of writing JavaScript-based tests for Jest. -## The run process +The E2E tests utilize WebdriverIO as the Appium driver, which is an interface that allows Appium to automate a particular platform. The WebdriverIO documentation for [Appium](https://webdriver.io/docs/api/appium/) and the [driver object](https://webdriver.io/docs/api/browser) provides useful information for automating interactions with mobile apps. -At the moment when running locally, the app attempts to fire up an appium server and then connects to it via webdriver. Then +Additionally, the projectā€™s E2E tests leverage a [Page Object](https://webdriver.io/docs/pageobjects/) pattern. The `editor-page.js` file defines methods managing all interactions with page UI. Tests themselves reside in separate files. -- on Android, a debug version of the app is bundled, built, and used. -- on iOS a release version is bundled built and used. +Expanding the Page Object to interact with specific UI elements often requires determining a [locator strategy](https://saucelabs.com/resources/blog/advanced-locator-strategies) for the targeted UI element. Generally, an accessibility (a11y) identifier is preferred over an Xpath selector, which are slower and considered implementation details. -**It's important to ensure that **metro is not running.** This would cause the value of the `__DEV__` variable to be true and load up the sample blocks.** +Determining the correct selector query is often made easier by leveraging a tool to inspect the properties of UI elements. -After the build is complete, an appium server is fired up on port 4723 and the device tests are ran on the connected device/simulator. +- [**Appium Inspector**](https://github.com/appium/appium-inspector#readme) ā€“ cross-platform tool for inspecting UI elements and their properties. +- [**Android Studio Layout Inspector**](https://developer.android.com/studio/debug/layout-inspector) ā€“ Android tool allowing you to debug the layout of your app by showing a view hierarchy and allowing you to inspect the properties of each view. +- [**Xcode Accessibility Inspector**](https://developer.apple.com/documentation/accessibility/integrating_accessibility_into_your_app#4154486) ā€“ Xcode tool that displays all accessibility information for an element. ---- +## Continuous Integration -To read more about writing your own tests please read the [contributing guide](https://github.com/WordPress/gutenberg/blob/HEAD/packages/react-native-editor/__device-tests__/CONTRIBUTING.md). +In addition to running the E2E tests on local computers, the continuous integration (CI) server runs some of these tests on every Pull Request. Configuration for this can be found in the [.github configuration directory](/.github/workflows).