Chromeless and Testing

[jbolda]

As much of our code runs client side, some of the errors we see are more subtle or opaque until you build or visit a page. I think it would be helpful in maintaining the myriad of examples and starters by including browser based tests within our test suite. Graphcool just put out a library called Chromeless to run chrome headless locally/on AWS Lamda. We could diff screenshots and check for errors in the console. This might also be useful to add to our CI as we only gatsby build and might miss console errors.

Thoughts or anyone interested in creating a PR?

https://github.com/graphcool/chromeless

[KyleAMathews]

Great thoughts!

Loading the example sites and logging errors + taking screenshots (and comparing with old ones) makes a ton of sense.

Making sure Gatsby's runtime is rock solid is really important so this would be great.

[dan-weaver]

this sounds sorta cool. Anyone started working on this yet?

[sebastienfi]

@dan-weaver Not that I know of. Are you digging in ?

[dan-weaver]

Looking into chromeless a bit. Would we want to actually run these in Lambda or just on the normal ci server?

[KyleAMathews]

This will get a lot easier with #3242

[m-allanson]

See also #4790

[sebastienfi]

@jbolda Why not using Chromium and Puppeteer for that?

Here is some code to get anyone started:

const puppeteer = require("puppeteer");
const { imgDiff } = require("img-diff-js");

(async () => {
  const browser = await puppeteer.launch(); // Here a bunch of options can be set like `headless` and `executablePath`
  const page = await browser.newPage();
  await page.goto("https://example.com"); // `waitUntil: 'load'` seems required for a Gatsby site.

  // We can emulate screen dimension
  await page.setViewport({ width: 1600, height: 1100 });
  // ... and device type
  await page.emulateMedia("screen");

  // Take a screenshot and diff it
  await page.screenshot({ path: "actual.png" }).then(() =>
    imgDiff({
      actualFilename: "actual.png",
      expectedFilename: "expected.png",
      diffFilename: "diff.png"
    }).then(result => console.log(result))
  );

  // Listen to a console message containing error
  page.on(
    "console",
    msg =>
      msg.toLowerInvariant().contains("error")
        ? console.log("PAGE LOG ERROR:", msg.text())
        : ""
  );

  await browser.close();
})();

[jbolda]

@sebastienfi because it wasn't published when I posted this :)

I think at this point puppeteer makes more sense, especially that we can include it in the jest testing. The CI would maybe have different considerations, but that still seems in flux.

[jbolda]

Closing this in favor of tracking in #4361 and #4790. @tsriram seems to be doing some great work there.