[RNMobile] Fix RN iOS E2E tests failure related to environment not available #35879
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fluiddot
added
[Type] Bug
|
Size Change: +13.6 kB (+1%) Total Size: 1.1 MB
ℹ️ View Unchanged
|
|
The first test run failed due to a different error than the original one: I'm going to retry the test to verify if happens sporadically. |
|
Unexpectedly, removing the |
Plot twist: Forcing Job: https://github.com/WordPress/gutenberg/runs/3976900145?check_suite_focus=true |
2 tasks
|
The PR has been re-opened to do a second round of tests, in case recent changes in |
Second roundResults:
Looks like the issue is still happening as per the third run 😞 , I'll investigate further and check the Appium logs in case there's any clue that I overlooked. |
fluiddot
changed the title
|
Looks like using Xcode 13.0 helped to make the workflow more stable ( |
After triggering more runs, I found out that the stability went down compared to the initial tests I did, so I'm not sure about applying the fix and re-enable the E2E tests. For now, I'm going to change the PR back to draft and try new approaches like using a different simulator device. @geriux @twstokes sorry for the ping in the PR, I'd appreciate it if you could help me on this by providing fresh ideas or insights about anything I might be overlooking, thanks 🙇 ! |
|
👋 @fluiddot - I will gladly help look into this. Do you think there's any value in trying to bump to Xcode 13.1? |
@twstokes thanks for the help 🙇 !
Yeah, we could try that although I'm not sure if it's related to the Xcode version. It's still a mystery for me why some runs work and others do not, I compared the Appium logs of succeeded and failed runs and I couldn't find any clue that helped me out to find the culprit. The only clue we have so far is the following log: Appium log - failed runAppium log - success runFrom the error The log right before the error is the open command for the Simulator: And the last error relates the failure with it: So I'm wondering if the executable that is failing the Simulator app itself, although this would be quite weird 🤔. |
|
I tried in the last run just opening the simulator within a new step and failed, so the error is actually related to the Simulator app. Now I’m wondering what could be causing that weird issue of not being able to execute if the MacOS image dcoumentation states that it’s available 🤔 .
|
@twstokes Thanks for running these tests 🙇 ! Seems like the |
|
Sounds good @fluiddot. I thought this was interesting in the It may be useful for us to troubleshoot what that process entails. For instance, can that file be written to while the sim is running? Do those preferences still work with Xcode 13 sims? |
That's interesting, at this point I think we could try to debug the code from Appium that runs the simulator somehow. I did a quick check on the repos and found the lines related to this:
|
|
|
|
After testing the command that opens the Simulator app in a new repo with a basic workflow, I verified that the issue is related to the macOS image used in the I opened the following ticket actions/runner-images#4545 as a follow-up for this issue in the GitHub actions repo. At this point, I don't think we'll be able to provide a solid workaround for re-enabling the iOS E2E tests, so we'll be following up on the issue in case someone can provide some insights about the potential culprit of the error. |
fluiddot
commented
Nov 24, 2021
| @@ -4,7 +4,7 @@ const ios = { | |||
| os: 'iOS', | |||
| deviceOrientation: 'portrait', | |||
| automationName: 'XCUITest', | |||
| appiumVersion: '1.20.1', // Sauce Labs requires appiumVersion to be specified. | |||
| appiumVersion: '1.22.0', // Sauce Labs requires appiumVersion to be specified. | |||
There was a problem hiding this comment.
I noticed that on GB-mobile we didn't update Appium to 1.22.0, so this change might affect the E2E tests in that repo. For this reason, I've opened a PR to update it.
There was a problem hiding this comment.
Looks like we need to make further changes in the GB-mobile E2E tests to use this Appium version, as I triggered a run and failed with the following error (reference):
Appium 1.22.0 does not support iOS 14.5. Please check our platform configurator (https://saucelabs.com/docs/platforms)
I don't want to block this PR due to this change, so I'm going to revert it and address it in a separate PR.
There was a problem hiding this comment.
Change reverted in this commit.
|
Heads up that the solution applied here with the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This is a follow-up of #35754, #35770, and #35829 which fixed an issue of a deprecated Xcode version but introduced flakiness in the iOS E2E tests.
NOTE: This PR also re-enables the "React Native E2E Tests (iOS)" workflow that was disabled in #35844.
First approach ❌
After testing different combinations in a testing branch (#35842), we narrowed down the issue and relate it to the generation of the WDA (Web Driver Agent) required to run the E2E tests on iOS. The fix basically consists in stop using a prebuilt version of WDA and letting Appium take care of the building process, which is the default behavior (reference).
Result: Although some of the runs succeeded others did not (3 success of 7 runs - 43%), in the following section you can check the results.
Click here to see results
Run 1: succeed 🟢
Run 2: failed ❌
[elementByAccessibilityId("Switch To Visual")] Error response status: 7, , NoSuchElement - An element could not be located on the page using the given search parameters. Selenium error: An element could not be located on the page using the given search parameters.Run 3: failed ❌
E2E setup exception: Error: [init({"browserName":"","platformName":"iOS","os":"iOS","deviceOrientation":"portrait","automationName":"XCUITest","appiumVersion":"1.20.1","app":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/GutenbergDemo/Build/Products/Release-iphonesimulator/GutenbergDemo.app","processArguments":{"args":["uitesting"]},"deviceName":"iPhone 11","wdaLaunchTimeout":240000,"usePrebuiltWDA":false,"platformVersion":"14.5","derivedDataPath":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/WDA"})] The environment you requested was unavailable.Run 4: failed ❌
E2E setup exception: Error: [init({"browserName":"","platformName":"iOS","os":"iOS","deviceOrientation":"portrait","automationName":"XCUITest","appiumVersion":"1.20.1","app":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/GutenbergDemo/Build/Products/Release-iphonesimulator/GutenbergDemo.app","processArguments":{"args":["uitesting"]},"deviceName":"iPhone 11","wdaLaunchTimeout":240000,"usePrebuiltWDA":false,"platformVersion":"14.5","derivedDataPath":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/WDA"})] The environment you requested was unavailable.Run 5: succeed 🟢
Run 6: succeed 🟢
Run 7: failed ❌
E2E setup exception: Error: [init({"browserName":"","platformName":"iOS","os":"iOS","deviceOrientation":"portrait","automationName":"XCUITest","appiumVersion":"1.20.1","app":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/GutenbergDemo/Build/Products/Release-iphonesimulator/GutenbergDemo.app","processArguments":{"args":["uitesting"]},"deviceName":"iPhone 11","wdaLaunchTimeout":240000,"usePrebuiltWDA":false,"platformVersion":"14.5","derivedDataPath":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/WDA"})] The environment you requested was unavailable.Conclusion
I tried to relate the failure with the presence of the build cache, the version of the virtual environment, or the use of different simulator instances. However, none of these seemed to be really related as all produced failures. For this reason, although this approach worked in some runs, I decided to not continue with it as it doesn't look reliable.
Second approach ❌
Following the previous workaround, I tried to use a newer version of Xcode (Xcode 13.0) in case the failure was related to the Xcode version.
Result: After triggering 10 runs, three failed so this approach looks a bit more reliable than the previous (7 success of 10 runs - 70%), in the following section you can check the results.
Click here to see results
Run 1: failed ❌
E2E setup exception: Error: [init({"browserName":"","platformName":"iOS","os":"iOS","deviceOrientation":"portrait","automationName":"XCUITest","appiumVersion":"1.20.1","app":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/GutenbergDemo/Build/Products/Release-iphonesimulator/GutenbergDemo.app","processArguments":{"args":["uitesting"]},"deviceName":"iPhone 11","wdaLaunchTimeout":240000,"usePrebuiltWDA":false,"platformVersion":"15.0","derivedDataPath":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/WDA"})] The environment you requested was unavailable.Run 2: succeed 🟢
Run 3: succeed 🟢
Run 4: succeed 🟢
Run 5: succeed 🟢
Run 6: succeed 🟢
Run 7: succeed 🟢
Run 8: succeed 🟢
Run 9: failed ❌
E2E setup exception: Error: [init({"browserName":"","platformName":"iOS","os":"iOS","deviceOrientation":"portrait","automationName":"XCUITest","appiumVersion":"1.20.1","app":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/GutenbergDemo/Build/Products/Release-iphonesimulator/GutenbergDemo.app","processArguments":{"args":["uitesting"]},"deviceName":"iPhone 11","wdaLaunchTimeout":240000,"usePrebuiltWDA":false,"platformVersion":"15.0","derivedDataPath":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/WDA"})] The environment you requested was unavailable.Run 10: failed ❌
2E setup exception: Error: [init({"browserName":"","platformName":"iOS","os":"iOS","deviceOrientation":"portrait","automationName":"XCUITest","appiumVersion":"1.20.1","app":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/GutenbergDemo/Build/Products/Release-iphonesimulator/GutenbergDemo.app","processArguments":{"args":["uitesting"]},"deviceName":"iPhone 11","wdaLaunchTimeout":240000,"usePrebuiltWDA":false,"platformVersion":"15.0","derivedDataPath":"/Users/runner/work/gutenberg/gutenberg/packages/react-native-editor/ios/build/WDA"})] The environment you requested was unavailable.Conclusion
Although I couldn't manage to find the culprit of the failure, looks like using Xcode 13.0 makes the workflow a bit more reliable.
For this reason, I think we could continue with this approach and observe in Gutenberg PRs how it goes.However, I think it's not stable enough to continue with this approach so we should try new ideas for this 😞 .Third approach 🟢
After narrowing down the issue and identifying the command that opens the Simulator as the culprit (reference), we added a new step to prevent the error
The executable is missingas it was referenced in this comment.Result: After triggering 10 runs, all of them succeeded 🎊.
Click here to see results
Run 1: succeed 🟢
Run 2: succeed 🟢
Run 3: succeed 🟢
Run 4: succeed 🟢
Run 5: succeed 🟢
Run 6: succeed 🟢
Run 7: succeed 🟢
Run 8: succeed 🟢
Run 9: succeed 🟢
Run 10: succeed 🟢
Conclusion
Adding a step for executing the command
/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -f /Applications/Xcode_${{ matrix.xcode }}.app/Contents/Developer/Applications/Simulator.appfixed the problem of opening the Simulator app, so this approach looks solid to be used 🥳 .How has this been tested?
npm run native test:e2e:ios:local -- gutenberg-editor-initial-htmlReact Native E2E Tests (iOS)workflow succeeds.Screenshots
N/A
Types of changes
Bug fix.
Checklist:
*.native.jsfiles for terms that need renaming or removal).