Improving Browser QA

[nhunzaker]

We manually test a lot of behavior that can not be captured by JSDOM. Change events, number input quirks, event pooling, error handling, etc.

Despite this test suite, comprehensive unit tests, alpha, beta, and release candidates, a number of bugs have slipped through. Particularly around change events.

I want to explore what it would look like to reduce the burden of manual browser QA when possible. Particularly around things like change events, which hook into JavaScript DOM APIs and should be automatable.

I've filed this issue for myself to explore the following areas (though I'd be happy for others to pitch in 😄):

• Deploying the DOM Test Fixtures to a public URL to distribute manual QA and run automated tests against it
• Investigate BrowserStack or Sauce Labs as possible browser automation options
• Deploy a nightly or edge build of React with an accompanying JSFiddle to make it easier to determine if a fix works on master without needing to know how to setup React locally.
• With automation in place, what extremely laborious (but comprehensive) tests could we run to help fight regressions?

---

I'm probably missing something, but this is where I'm going to start. I'll post follow-up comments as I learn stuff.

[nhunzaker]

cc @aweary and @jquense

[aweary]

• Deploying a build of the fixutres for PRs would be really helpful. We (inadvertently) already have this feature with the docs site, so it should be possible 😀

• It would be nice if the fixtures app had JSFiddle like functionality. That way we could quickly test out DOM changes locally without having to write a new test case.

Deploy a nightly or edge build of React with an accompanying JSFiddle to make it easier to determine if a fix works on master without needing to know how to setup React locally

@zpao used to host a site that deployed builds from master, I'll try to find it when I get home.

[sophiebits]

http://react.zpao.com/ (still builds from our CI)

[gaearon]

I think @sophiebits also suggested having contractors run the QA tests in different browsers. Not sure how hard it would be.

[jquense]

I think that having a few, focused selenium tests via sauces labs or the like for particularly fraught areas is a a reasonable idea. Yes they can be flakey but if they can at least provide some canary-in-the-coalmine early warnings that can inform where to spend time doing some manual QA. The unit-test nature of these tests too should reduce the flakey-ness, plus we get some insight when we break peoples selenium workflows.

CircleCI's v2 workflows feature also makes it easier to set up flows that aren't required or need manual approval.

[jquense]

if it helps also i have some experience with saucelabs/browserstack I've found SL to be less flakey across browsers and done a bunch of e2e testing with protractor and react with better than expected results. glad to help out here :)

[aweary]

I also want to call back to #9301. If we have a definitive list of the browsers we need to test in, we could make sure that we're not wasting time testing in unsupported browsers.

[nhunzaker]

I've started to work with NightwatchJS and BrowserStack Automate. It is very easy to run a test against all of the browsers we cover.

This flow would involve:

1. Automated deployment to an integration URL whenever a pull request is sent out
2. A hook to expose that URL to a pull request
3. Run the automated tests against the URL, providing it as environment variable

Here's some code:
https://github.com/nhunzaker/nightwatch-test

I'm able to run an automated test in Safari, Chrome, Firefox, IE, and Edge. It's pretty neat. BrowserStack even records the test output.

This isn't cheap though. If we went this route, we would need an Automate or Automate Mobile subscription. This is $1000-1500 for a yearly subscription.

Still. I think this a promising way to go. We could automate testing against ~30 browsers.

[jquense]

@nhunzaker that's awesome I was just thinking today i wanted to make some time to work on this. maybe can chat a bit on discord (or whatever)? I fairly recently hooked up selenium and e2e tests for a work app and have some thoughts and experiences that might be useful here, about which stack to use and browserstack vs saucelabs

[klamping]

I'd like to try helping out this cause. I still need to figure out what part I'd like to work on first, but I figure I'd start by saying hi here.

FYI, Sauce Labs offers a free plan for open-source projects, so you wouldn't need to worry about a monthly/yearly license. All you have to do is reach out to their customer support.
[Edit: BrowserStack also has free plans for open-source projects]

On that note, Netlify is a static-site server (similar to github pages) and may be useful for hosting the fixtures pages. They also have a free open-source plan: https://www.netlify.com/open-source/

/cc @giltayar

[klamping]

@nhunzaker fyi, I copied your NightwatchJS example and modified it to use WebdriverIO. You can check it out here:
https://github.com/klamping/wdio-react-example

Benefits of WebdriverIO over NightwatchJS are that WebdriverIO provide built-in services to connect to tools like Browserstack or Chromedriver, and it also provides more flexibility in how you write your tests.

Would love to get your thoughts on this direction, as I have some time to keep plugging away at writing some browser automation for the DOM fixtures.

[klamping]

So I've been doing some work on that test repo and wanted to provide a quick update.

First off, I changed the repo name so that it now lives here: https://github.com/klamping/react-webdriverio-tests

I've also added a few more tests and plan on adding many more to come.

Using CircleCI and Zapier, it's "integrated" with the React PRs to run the tests I do have on the PR code and print out a report for each PR.

I have a temporary issues page which stores the links to the PR, report and CircleCI job. You can check it out here: klamping/react-webdriverio-tests#1

Right now, the tests are running in Firefox and Chrome on the CircleCI box. I'd love to get it integrated with BrowserStack or Sauce Labs, but don't have an account.

Any feedback or suggested direction would be much appreciated. I'm hoping this can help provide a good proof-of-concept (or even some real value) in to how automated browser testing works.

[christian-bromann]

@nhunzaker I am happy to power up your open source account at Sauce Labs for such beautiful open source projects like React 😉

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution.

[stale]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!