Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bumps webdriverio to latest, adds isRetryableError utility, reduces test flakiness #89

Merged
merged 7 commits into from
Dec 9, 2024

Conversation

mhassan1
Copy link
Collaborator

@mhassan1 mhassan1 commented Dec 3, 2024

This PR bumps webdriverio to latest, which may help with CI flakiness. It also adds an isRetryableError utility that will retry jobs on the following errors:

  1. Failed to create session
Error: Failed to create session.
WebDriverError: This operation was aborted when running "https://hub-cloud.browserstack.com/wd/hub/session" with method "POST" and args "{"capabilities":{"alwaysMatch":{"bstack:options":{"sessionName":"build:null_2024-12-02T23:30:01.135Z","projectName":"polyfill-library","local":true,"video":true,"debug":true,"consoleLogs":"errors","networkLogs":true},"browserName":"chrome","browserVersion":"102.0","webSocketUrl":true},"firstMatch":[{}]}}"
    at startWebDriverSession (file:///home/runner/work/polyfill-library/polyfill-library/node_modules/webdriver/build/index.js:853:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async _WebDriver.newSession (file:///home/runner/work/polyfill-library/polyfill-library/node_modules/webdriver/build/index.js:1547:41)
    at async remote (file:///home/runner/work/polyfill-library/polyfill-library/node_modules/webdriverio/build/index.js:7228:20)
    at async TestJob.run (/home/runner/work/polyfill-library/polyfill-library/test/polyfills/test-job.js:102:19)
  1. An unknown server-side error occurred while processing the command
unknown error: WebDriverError: An unknown server-side error occurred while processing the command. when running "window" with method "DELETE"
    at FetchRequest._request (file:///home/runner/work/polyfill-library/polyfill-library/node_modules/webdriver/build/node.js:1652:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Browser.wrapCommandFn (file:///home/runner/work/polyfill-library/polyfill-library/node_modules/@wdio/utils/build/index.js:884:23)
    at async TestJob.run (/home/runner/work/polyfill-library/polyfill-library/test/polyfills/test-job.js:175:4)

It also includes #90, which further reduces test flakiness.

@mhassan1 mhassan1 closed this Dec 3, 2024
@mhassan1 mhassan1 deleted the webdriverio-retry branch December 3, 2024 20:09
@mhassan1 mhassan1 restored the webdriverio-retry branch December 3, 2024 20:10
@mhassan1 mhassan1 reopened this Dec 3, 2024
@mhassan1 mhassan1 requested a review from romainmenke December 3, 2024 21:30
@mhassan1
Copy link
Collaborator Author

mhassan1 commented Dec 3, 2024

@romainmenke I didn't see any Failed to create session. error logs in the run on this PR, so maybe the webdriverio upgrade solved that, or maybe we just got lucky 😄. Anyway, this PR should be safe to merge.

@romainmenke
Copy link
Member

Thank you for spending time on this @mhassan1 🙇

Should we also retry on this error:

https://github.com/mrhenry/polyfill-library/actions/runs/12148384546/job/33876735290

WebDriverError: An unknown server-side error occurred while processing the command. when running "window" with method "DELETE"

Or possibly extending the sharding of tests to IE 10 and IE 11?
I think the window becomes unresponsive at some point when running all the tests with also all the polyfills loaded.

It is mostly important that we test each feature individually with every other polyfill present. Not so much that we test all features in one go.

@mhassan1 mhassan1 changed the title Bumps webdriverio to latest, adds retry logic on failure to create browserstack session Bumps webdriverio to latest, adds isRetryableError utility Dec 3, 2024
@mhassan1
Copy link
Collaborator Author

mhassan1 commented Dec 3, 2024

@romainmenke I've created an isRetryableError utility, and I've added that error to the list.

@romainmenke
Copy link
Member

Nice!

https://github.com/mrhenry/polyfill-library/actions/runs/12153971340/job/33892964196

Maybe this one too?

WebDriverError: The operation was aborted due to timeout when running "execute/sync" with method "POST"

@mhassan1
Copy link
Collaborator Author

mhassan1 commented Dec 4, 2024

Maybe this one too?

Added.

- better naming for test runs
- shard ie/10
Copy link
Member

@romainmenke romainmenke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Thank you for working on this 🙇

@mhassan1 mhassan1 changed the title Bumps webdriverio to latest, adds isRetryableError utility Bumps webdriverio to latest, adds isRetryableError utility, reduces test flakiness Dec 9, 2024
@mhassan1 mhassan1 merged commit 1d5a270 into main Dec 9, 2024
17 checks passed
@mhassan1 mhassan1 deleted the webdriverio-retry branch December 9, 2024 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants