Consider app crash tests as non-fatal if they pass on retry #456
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ravimandala
force-pushed
the
feature-retry-app-crash-tests
branch
from
89dcc66 to
f7159fc
Compare
ravimandala
changed the title
ravimandala
force-pushed
the
feature-retry-app-crash-tests
branch
from
f7159fc to
2c97953
Compare
Introducing a flag `retry-app-crash-tests` to retry tests that might have caused app crashes. When this flag is turned on, the execution will not fail if all app crash tests pass on retry. The retries will honor the `error-retries` count, restarts the sim before retrying the crashed test and makes the crash logs available, like before.
ravimandala
force-pushed
the
feature-retry-app-crash-tests
branch
from
2c97953 to
ad30ab4
Compare
ob
reviewed
Aug 12, 2020
Oscar, I agree a few flags might have gotten complicated over time and may even have deviated from the original purpose. So, I am going to take the latter route to simplify the flag set and deprecate some of them. Created issue #457 for further discussion and follow-up on this. |
ob
approved these changes
Aug 12, 2020
Sure. Merging this now. |
ravimandala
added a commit
that referenced
this pull request
Oct 2, 2020
Introducing a flag `retry-app-crash-tests` to retry tests that might have caused app crashes. When this flag is turned on, the execution will not fail if all app crash tests pass on retry. The retries will honor the `error-retries` count, restarts the sim before retrying the crashed test and makes the crash logs available, like before.
AhmedEid
added a commit
to tinyspeck/bluepill
that referenced
this pull request
Nov 18, 2020
* master: Reference libXCTestBundleInject.dylib only if it exists (MobileNativeFoundation#460) Record video using simctl (MobileNativeFoundation#441) Xcode 12.0 support (MobileNativeFoundation#458) Retry app crash tests and consider then non-fatal if they pass (MobileNativeFoundation#456) Do not retry crashed tests (MobileNativeFoundation#443) Fixing minor issues and doing some minor refactoring/clean-up (MobileNativeFoundation#444) Xcode 11.5 support (MobileNativeFoundation#447) Xcode 11.4 support (MobileNativeFoundation#446) Add a flag to disable Xcode version check failure (MobileNativeFoundation#436) Support sidecar applications (MobileNativeFoundation#438) Added a few more tests to mock test failure scenarios and fixed a few final Exit Status issues (MobileNativeFoundation#430) make retry checks more resilient to prevent infinite retries (MobileNativeFoundation#432) Xcode 11.3 support (MobileNativeFoundation#426) Supporting tests and app hosts in subfolders (MobileNativeFoundation#419) # Conflicts: # bp/src/SimulatorHelper.m # bp/tests/SimulatorHelperTests.m
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introducing a flag
retry-app-crash-teststo retry tests that. might have caused app crashes. When this flag is turned on, the execution will not fail if all app crash tests pass on retry.The retries will honor the
error-retriescount, restarts the sim before retrying the crashed test and makes the crash logs available, like before.Ref: #455
\cc @ob @chenxiao0228