Issues with Sauce Labs - testing #737
Comments
|
I saw that too with my latest push on a PR. Annoying |
|
Seems to have stabilized since 5270bcf . I wonder if the "reundant search cycles" were causing instability/timing issues... Or maybe just coincidence: something fixed at Travis. Let's wait and see. |
|
Hmm, of course the PR shouldn't have been failing if the PR itself fixed anything.... |
|
I spoke too soon. It just fell over again on the Polyfill PR and I had to re-run. |
|
I have to restart tests almost every other time. I wonder if we need to start looking for another solution? I don't have experience, but there must be something that runs natively on GitHub? Maybe we could start to build up new tests with a newer ZIM, while having the old ones still running until the new ones are as feature-complete... |
|
I saw that too. |
|
Maybe we could look into (slowly) trying out some new testing framework. Jest (Unit), possibly in combination with Puppeteer (UI), look like popular frameworks where a lot of help is available. See https://jestjs.io/ and https://jestjs.io/docs/puppeteer. Just based on quick search, no proper research. They could run alongside our existing Unit Tests (for non-regression), but we could eventually stop using Sauce Labs for UI Tests (these are the ones that fail 50% of the time due to socket resets, our Unit Tests are fine). |
|
Can you post a recent failing build and possible copy some of its output onto the issue here for later reference? (given GH Actions logs are quite ephemeral) The question would be what specific socket this is about. Eg from which exact software package and version, and to which host name. |
|
Thanks @Krinkle. One that was typical was this one: https://github.com/kiwix/kiwix-js/actions/runs/3335388590/jobs/5519308638 The salient errors are pasted below. The errors seem to correspond to this one: https://docs.saucelabs.com/dev/error-messages/index.html#the-sauce-labs-virtual-machine-failed-to-start-the-browser-or-device, and it is suggested there that it is either an incompatible version of Selenium, or else a request for an incompatible platform/OS/browser combination. I tried many things in #895, all of them failing on recent versions of Firefox, which seemed really counterintuitive given that we can run Firefox 45 with no error... Then when I "gave up" and decided just to disable the latest Firefox on SauceLabs (because we run in headless on GitHub actions anyway),I ran into a second layer of errors. Will summarise those in next post. |
|
Second layer of errors emerged with end-to-end tests after I disabled the latest Firefox. Seems to be I suspect that a dependabot upgrade changed something about the configuration, but at this stage I no longer know what. What I do know is that we're running a very old version of nightwatch with lots of security issues. When I tried upgrading that, it made no difference to these errors. |
|
Looking at our commit log, it's easy to see where the tests began to fail systematically: https://github.com/kiwix/kiwix-js/commits/master, specifically the last "good" commit was 9a6e6f7. The next commit after that didn't touch the test configuration though, it seemed to fail on its own, making me suspect a temporary issue at SauceLabs. |
With almost every commit on almost any PR I make with Kiwix JS, I have to re-run the cross-browser end-to-end tests, because they fail first time. They will usually pass second time, occasionally third time. When I look, it's usually a socket reset that's the problem.
Is this likely to be an issue with Sauce Labs, or an issue with some race condition in Kiwix JS? In the past, with some PRs there has been a race condition that is clearly the fault of new code in Kiwix JS (though it's sometimes taken weeks to find it). However, most of the time the errors occur for no apparent reason.
The text was updated successfully, but these errors were encountered: