Performance Regression in 3.x

[arahansen]

Current behavior:

Running 13 spec files in [email protected] takes 152 seconds (including test compilation and execution)

Desired behavior:

Running 13 spec files in [email protected] takes 79 seconds (including test compilation and execution)

Steps to reproduce:

My guess is that this regression is related to Cypress now running tests in isolation. There appears to be a lot of setup and teardown time between specs running. Is there anyway around this? Cypress taking twice as long to run the same test suite is surprising.

I'm not sure I can justify upgrading to 3.0 with this issue. It at least will make me think long and hard whether I can justify adding another test when it will be a significant impact to my CI run times.

UPDATE: My tests currently do an afterEach(() => { cy.screenshot() }) removing this cuts down execution time to 110 seconds.

Versions

[email protected]

[brian-mann]

We're aware of the regression. It's a combination of things, some of which we have already fixed.

There's an open PR fixing the cy.screenshot performance issues here: #1864

Opening + closing the browser will add a bit of extra time, but it shouldn't be as drastic as what you're seeing, maybe only 1-3 seconds per additional spec file. It is actually possible for us to use this time to begin preemptively transpiling the next spec file - since transpilation time is honestly the most expensive thing when the tests first run, and typically only gets started once the browser is open and Cypress is ready to start running tests. Doing this in the background ahead of time would likely yield back the browser opening penalty. I will open a separate issue for this.

We noticed the same performance drop in our own internal tests but haven't yet spent the time to identify the root cause other than what I mentioned. Even tests on a single spec are taking longer.

[arahansen]

Thanks for the update @brian-mann. Happy to provide more diagnostics if it would help!

I'll keep an eye out for other possible offenders too.

[chrislopresto]

We're seeing a similar performance regression from Cypress 2.1 to Cypress 3.1:

• 72 spec files
• Representative Cypress 2.1 run: 317 seconds
• Representative Cypress 3.1 run (no parallelization): 670 seconds

This would indicate a thoroughly unscientific, approximate ~5 second penalty per spec file. 🙂

With Cypress 3's built-in parallelization (6 workers appears to be our sweet spot), we can improve the time of the overall test suite run to 96 seconds, so that's fantastic. For better or worse that shifts our conversation to evaluating among a few options:

• Hold steady on Cypress 2
• Combine tests into fewer files. Presumably help unparallelized build time on free plan, hurt parallelized build time on paid plan.
• Upgrade to paid plan to use Cypress 3's built-in parallelization
• Use our own parallelization scheme on Cypress 3 free plan

We're happy to provide diagnostics and help dig in for troubleshooting as well. Cheers!

[jennifer-shehane]

We fixed many performance issues recently. Please update to our latest version and see if these issues still persist. Comment here if so and we will reopen.

Also - cypress run is expected to be slower than the Run All Specs button in `cypress open.

Explanation of why it is slower (this is expected behavior)

cypress run relaunches the browser in between each spec file while 'Run All Tests' button does not. There is an issue to address this discrepancy here: #1586 This would account for the difference in running time.

If this is the case, then we will close this issue and suggest you add your thoughts and 👍 to this issue: #2951

[tomitrescak]

Is it possible to NOT open the browser on each spec file via some flag ?

[jennifer-shehane]

@tomitrescak No there isn't a flag. The browser relaunches in between each spec file so there is not any discrepancy between how runs run when parallelizaed (which would have each specfile in isolation completely) versus not.