-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add visual regression testing with cypress #1339
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
node_modules/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
name: Cypress | ||
|
||
on: | ||
pull_request: | ||
push: | ||
branches: | ||
- master | ||
- stable* | ||
|
||
env: | ||
APP_NAME: nextcloud-vue | ||
|
||
jobs: | ||
jest: | ||
runs-on: ubuntu-latest | ||
|
||
strategy: | ||
fail-fast: false | ||
matrix: | ||
node-version: ['12.x'] | ||
|
||
name: Runner ${{ matrix.containers }} | ||
|
||
steps: | ||
- uses: actions/checkout@v2 | ||
|
||
- name: Set up node ${{ matrix.node-version }} | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: ${{ matrix.node-version }} | ||
|
||
- name: Cypress run | ||
env: | ||
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }} | ||
run: npm run cypress:record | ||
Comment on lines
+32
to
+35
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not fond, I think that complicates the setup a lot. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Because the tests now pass, and failed before 😉 The tiny differences we saw with the font rendering are caused by running the tests on different machines. Between them, maybe the browser, the display settings, the available fonts or whatever else differs slightly, leading to tiny differences in the rendering which make the tests fail. Of course, we could increase the difference threshold to make the tests pass. But this is not really an option as it would lead to false positives. The difference in this image is 0.1623 and is not a regression (it's the difference between running locally on my machine and inside the docker container): The difference in this image is 4 times lower with 0.0406 and it clearly is a regression (I intentionally changed the padding of the close button): So, running the visual tests in the very same environment ensured by a docker container seems like the only option to fix this. If you have another idea, I am open to check it out.
It sure is complicated, but I got the impression, so is visual regression testing in general 🙈 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Cypress actually have dockerized github actions as well. I'm just not fond of having to also maintain a Docker setup there. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am fine with using this github action docker setup, but we have to as well locally run cypress within the container. How would you do that then? |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
FROM cypress/base:14.7.0 | ||
RUN mkdir /app | ||
WORKDIR /app | ||
COPY package.json package-lock.json ./ | ||
RUN npm ci | ||
COPY .eslintrc.js .stylelintrc.js babel.config.js cypress.json webpack.common.js webpack.dev.js .stylelintignore ./ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
{ | ||
"projectId": "3paxvy", | ||
"viewportWidth": 1920, | ||
"viewportHeight": 1080, | ||
"defaultCommandTimeout": 6000, | ||
"experimentalComponentTesting": true, | ||
"componentFolder": "tests/visual", | ||
"nodeVersion": "system", | ||
"env": { | ||
"failSilently": false, | ||
"type": "actual" | ||
}, | ||
"screenshotsFolder": "cypress/snapshots/actual", | ||
"trashAssetsBeforeRuns": true | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
// *********************************************************** | ||
// This example plugins/index.js can be used to load plugins | ||
// | ||
// You can change the location of this file or turn off loading | ||
// the plugins file with the 'pluginsFile' configuration option. | ||
// | ||
// You can read more here: | ||
// https://on.cypress.io/plugins-guide | ||
// *********************************************************** | ||
|
||
// This function is called when a project is opened or re-opened (e.g. due to | ||
// the project's config changing) | ||
|
||
const getCompareSnapshotsPlugin = require('cypress-visual-regression/dist/plugin') | ||
|
||
const webpack = require('@cypress/webpack-preprocessor') | ||
|
||
const webpackOptions = require('../../webpack.dev.js') | ||
webpackOptions.externals = {} | ||
|
||
const options = { | ||
// send in the options from your webpack.config.js, so it works the same | ||
// as your app's code | ||
webpackOptions, | ||
watchOptions: {}, | ||
} | ||
|
||
module.exports = (on, config) => { | ||
getCompareSnapshotsPlugin(on) | ||
on('file:preprocessor', webpack(options)) | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
const compareSnapshotCommand = require('cypress-visual-regression/dist/command') | ||
|
||
compareSnapshotCommand() |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
// *********************************************************** | ||
// This example support/index.js is processed and | ||
// loaded automatically before your test files. | ||
// | ||
// This is a great place to put global configuration and | ||
// behavior that modifies Cypress. | ||
// | ||
// You can change the location of this file or turn off | ||
// automatically serving support files with the | ||
// 'supportFile' configuration option. | ||
// | ||
// You can read more here: | ||
// https://on.cypress.io/configuration | ||
// *********************************************************** | ||
|
||
import 'cypress-vue-unit-test/dist/support' | ||
import './commands' |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
version: '3' | ||
services: | ||
visual: | ||
build: | ||
context: . | ||
volumes: | ||
- "./cypress:/app/cypress" | ||
- "./src:/app/src" | ||
- "./styleguide:/app/styleguide" | ||
- "./tests:/app/tests" | ||
- "./l10n:/app/l10n" | ||
environment: | ||
- CYPRESS_RECORD_KEY |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No parallelism anymore (Wasn't needed because we have 1 test, but how would we add more?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we gain much here. The whole
Cypress
action runs for 4:00 min. The cypress run took 35 s, of which running the actual AppSideber.visual.js test took 10 s. We would need to add a lot of tests to really save time.Edit: But feel free to add it again, of course.