Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add visual regression testing with cypress #1339

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
node_modules/
35 changes: 35 additions & 0 deletions .github/workflows/cypress.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Cypress

on:
pull_request:
push:
branches:
- master
- stable*

env:
APP_NAME: nextcloud-vue

jobs:
jest:
runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
node-version: ['12.x']

name: Runner ${{ matrix.containers }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No parallelism anymore (Wasn't needed because we have 1 test, but how would we add more?)

Copy link
Contributor Author

@raimund-schluessler raimund-schluessler Aug 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we gain much here. The whole Cypress action runs for 4:00 min. The cypress run took 35 s, of which running the actual AppSideber.visual.js test took 10 s. We would need to add a lot of tests to really save time.

Edit: But feel free to add it again, of course.


steps:
- uses: actions/checkout@v2

- name: Set up node ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}

- name: Cypress run
env:
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
run: npm run cypress:record
Comment on lines +32 to +35
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not fond, I think that complicates the setup a lot.
Why did we need to load this inside a docker container?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we need to load this inside a docker container?

Because the tests now pass, and failed before 😉

The tiny differences we saw with the font rendering are caused by running the tests on different machines. Between them, maybe the browser, the display settings, the available fonts or whatever else differs slightly, leading to tiny differences in the rendering which make the tests fail.

Of course, we could increase the difference threshold to make the tests pass. But this is not really an option as it would lead to false positives.

The difference in this image is 0.1623 and is not a regression (it's the difference between running locally on my machine and inside the docker container):
AppSidebar vue - starred_null - compact_false - header_none - secondary_none-diff

The difference in this image is 4 times lower with 0.0406 and it clearly is a regression (I intentionally changed the padding of the close button):
AppSidebar vue - starred_null - compact_false - header_none - secondary_none-diff

So, running the visual tests in the very same environment ensured by a docker container seems like the only option to fix this. If you have another idea, I am open to check it out.

I think that complicates the setup a lot.

It sure is complicated, but I got the impression, so is visual regression testing in general 🙈

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cypress actually have dockerized github actions as well.
We could have use those?

I'm just not fond of having to also maintain a Docker setup there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with using this github action docker setup, but we have to as well locally run cypress within the container. How would you do that then?

6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,9 @@ coverage
# Docs compiled build
styleguide/build/
styleguide/index.html

# Cypress files
cypress/videos
cypress/screenshots
cypress/snapshots/actual
cypress/snapshots/diff
6 changes: 6 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
FROM cypress/base:14.7.0
RUN mkdir /app
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY .eslintrc.js .stylelintrc.js babel.config.js cypress.json webpack.common.js webpack.dev.js .stylelintignore ./
15 changes: 15 additions & 0 deletions cypress.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"projectId": "3paxvy",
"viewportWidth": 1920,
"viewportHeight": 1080,
"defaultCommandTimeout": 6000,
"experimentalComponentTesting": true,
"componentFolder": "tests/visual",
"nodeVersion": "system",
"env": {
"failSilently": false,
"type": "actual"
},
"screenshotsFolder": "cypress/snapshots/actual",
"trashAssetsBeforeRuns": true
}
31 changes: 31 additions & 0 deletions cypress/plugins/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
// ***********************************************************
// This example plugins/index.js can be used to load plugins
//
// You can change the location of this file or turn off loading
// the plugins file with the 'pluginsFile' configuration option.
//
// You can read more here:
// https://on.cypress.io/plugins-guide
// ***********************************************************

// This function is called when a project is opened or re-opened (e.g. due to
// the project's config changing)

const getCompareSnapshotsPlugin = require('cypress-visual-regression/dist/plugin')

const webpack = require('@cypress/webpack-preprocessor')

const webpackOptions = require('../../webpack.dev.js')
webpackOptions.externals = {}

const options = {
// send in the options from your webpack.config.js, so it works the same
// as your app's code
webpackOptions,
watchOptions: {},
}

module.exports = (on, config) => {
getCompareSnapshotsPlugin(on)
on('file:preprocessor', webpack(options))
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 3 additions & 0 deletions cypress/support/commands.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
const compareSnapshotCommand = require('cypress-visual-regression/dist/command')

compareSnapshotCommand()
17 changes: 17 additions & 0 deletions cypress/support/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// ***********************************************************
// This example support/index.js is processed and
// loaded automatically before your test files.
//
// This is a great place to put global configuration and
// behavior that modifies Cypress.
//
// You can change the location of this file or turn off
// automatically serving support files with the
// 'supportFile' configuration option.
//
// You can read more here:
// https://on.cypress.io/configuration
// ***********************************************************

import 'cypress-vue-unit-test/dist/support'
import './commands'
13 changes: 13 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
version: '3'
services:
visual:
build:
context: .
volumes:
- "./cypress:/app/cypress"
- "./src:/app/src"
- "./styleguide:/app/styleguide"
- "./tests:/app/tests"
- "./l10n:/app/l10n"
environment:
- CYPRESS_RECORD_KEY
Loading