Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Visual Testing: Cypress + Percy #419

Merged
merged 13 commits into from
Aug 19, 2024
Merged

Adding Visual Testing: Cypress + Percy #419

merged 13 commits into from
Aug 19, 2024

Conversation

lee00678
Copy link
Collaborator

I'm setting the trigger for visual testing to happen when merge to main for now. I think we could also have Percy runs on a weekly basis just in case.

  • Installed Cypress and Percy for visual testing, built into our CI pipeline, triggers on merge to main
  • Added a random seed generator in our dev environment so the test can be consistent.

@lee00678 lee00678 requested a review from reb-dev July 24, 2024 22:18
Copy link
Contributor

@rokotyan rokotyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Can you please also provide some steps how to set it up locally?

package.json Outdated Show resolved Hide resolved
@@ -0,0 +1,37 @@
/// <reference types="cypress" />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this file at all?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It turns out we do need this file. Without it Cypress can't resolve /commands. See here

packages/dev/src/utils/data.ts Outdated Show resolved Hide resolved
packages/dev/src/utils/data.ts Outdated Show resolved Hide resolved
packages/dev/cypress/fixtures/example.json Outdated Show resolved Hide resolved
packages/dev/cypress/e2e/unovis.cy.ts Outdated Show resolved Hide resolved

it('Basic Annotation', () => {
cy.contains('Basic Annotations').click()
cy.wait(3000)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a specific reason to wait 3000 ms?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a specific reason to wait 3000 ms?

Some page has animations. This way we wait till everything is fully rendered before taking the screenshot.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add duration as a query string argument to the pages? I think this way you can speed up the testing significantly.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add duration as a query string argument to the pages? I think this way you can speed up the testing significantly.

What do you mean? Add something like ?duration=3000 to the url?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More like ?duration=0 to render charts without animation

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More like ?duration=0 to render charts without animation

That's a good idea.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More like ?duration=0 to render charts without animation

See a POC here

packages/dev/cypress.config.ts Outdated Show resolved Hide resolved
cy.percySnapshot('Stacked Bar with Labels', { scope: '.exampleViewer' })
})
})

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can try generating this file automatically based on the example files. Technically they have the required information, like the description text.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can try generating this file automatically based on the example files. Technically they have the required information, like the description text.

I suppose we could, but some page require additional settings. For example the API Endpoints Tree requires a bigger minHeight, we'd have to manually set things like that for specific test cases. Do you have a workaround for cases like this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can have a map of additional settings in the testing file to apply on the go if needed. I can explain more if you want, just let me know.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean to have an array or load a separate file which contains

testingArray = [
  {name: 'Single Container', duration:3000, additionalParams: {width: [1300]}},
  {name: 'Sankey', duration:3000},  ...
]

And iterate through the array inside the unovis.cy.ts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at unovis/packages/dev/src/examples/index.ts, I was thinking about something similar to generate this test file automatically. You'll need to "compile" it using Webpack but I think it will be a reasonable trade-off.

So the script will:

  • Batch import all the index.tsx files in the examples folder;
  • You'll be able to get the required information from the imports (titles, ...);
  • You can define additional testing configuration in the unovis.cy.ts file and match the corresponding examples by name or path. Or you can define it in the index.tsx files of the examples themsevles.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rokotyan I will give it a try. Is this PR getting too big? Should we have a separate PR for this and the duration params?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Up to you. I would do it here, but I don't see any problem with merging this PR first and having another one

@lee00678
Copy link
Collaborator Author

This is great! Can you please also provide some steps how to set it up locally?

Running this locally is a little tricky. You'd have to have a Percy account and a token to be able to do that. See this. It is possible to have a free personal account, which is what I did when I was testing this out. You need to create a project and it will generate a PERCY_TOKEN. You will then set it as part of your local env.

@lee00678 lee00678 force-pushed the qian/feature/cypress branch from 82eeb5e to 9d567db Compare July 25, 2024 18:56
reb-dev
reb-dev previously approved these changes Jul 26, 2024
@lee00678 lee00678 force-pushed the qian/feature/cypress branch from 9d567db to 86f9ed0 Compare August 1, 2024 17:28
@lee00678 lee00678 force-pushed the qian/feature/cypress branch 2 times, most recently from 6f2198b to 7780999 Compare August 1, 2024 22:49
@lee00678 lee00678 marked this pull request as draft August 2, 2024 17:23
@lee00678 lee00678 marked this pull request as ready for review August 2, 2024 19:06
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rokotyan @reb-dev Updating this as a proof of concept to see this is the right approach. I will update the rest of the files if this looks good.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I think I would try doing it in packages/dev/src/components/ExampleViewer/index.tsx and passing as a prop, or defining a dedicated React hook that can be imported to avoid code duplication.

@lee00678 lee00678 force-pushed the qian/feature/cypress branch from 0c8331e to 2ead8ea Compare August 7, 2024 20:22
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rokotyan @reb-dev updated by passing duration as props.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reb-dev the latest push should fully implement the duration props for all dev files. I put the urls in a separate file, we can fully implement the autogen for testing urls in a separate PR.

@lee00678 lee00678 force-pushed the qian/feature/cypress branch from 29a2648 to feddac8 Compare August 12, 2024 22:23
reb-dev
reb-dev previously approved these changes Aug 15, 2024
@sumitkumar25
Copy link
Collaborator

@lee00678 dont enable percy run on merge to master. it will eat up the screenshot quota.

  1. i would recommend a weekly run.
  2. We should check if we can have a percy run before a new release version is published.

@lee00678
Copy link
Collaborator Author

@lee00678 dont enable percy run on merge to master. it will eat up the screenshot quota.

  1. i would recommend a weekly run.
  2. We should check if we can have a percy run before a new release version is published.

It has a weekly run. Removed the trigger for merge to main.

@lee00678 lee00678 merged commit d20f8d9 into main Aug 19, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants