Proposal: Record open devTools in screenshots and video during cypress run

[chrisbreiding]

With these changes, it will no longer be possible to have opened the dev tools for electron in interactive mode and then have them automatically opened during a run. Also, it's never been possible to explicitly tells other browsers to open their dev tools.

Add a flag (--dev-tools or --open-dev-tools) that will open the browser dev tools on launch so a debugger in test or app code can hit and pause the run for debugging purposes.

[brian-mann]

Chrome already has a flag to do this. So does electron. I suppose we could normalize this into a CLI argument perhaps - but users could already do this via the plugins API and modifying the launch args for chrome + for electron.

[brian-mann]

• https://peter.sh/experiments/chromium-command-line-switches/#auto-open-devtools-for-tabs
• https://electronjs.org/docs/api/browser-window#new-browserwindowoptions

[jennifer-shehane]

Electron says that the option is devTools: true by default. But, Cypress actually overrides this in our options when opening Electron here:

https://github.com/cypress-io/cypress/blob/issue-4310/packages/server/lib/gui/windows.coffee#L142

WebPreferences logged at launch include no devTools

{ 
  partition: null,
  chromeWebSecurity: true,
  nodeIntegration: false,
  backgroundThrottling: false 
}

Even so, when the option is true is only allows you to later call BrowserWindow.webContents.openDevTools(), which we are not doing (unless the user's previous state had the devTools open.)

So, I don't think this will work with Electron without some changes to Cypress.

Workaround for Chrome

Auto opening devTools within Chrome does work today with the following code in your cypress/plugins.js file.

You can run this with cypress open, cypress run --browser=chrome --headed and will see the DevTools auto open during the run.

Since there is no video recording done with the chrome browser, using this flag for cypress run --browser=chrome would be useless. Furthermore, when screenshots are taken via cy.screenshot() or a failure screenshot, these do not include the DevTools within the screenshot.

module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome' || browser.name === 'chromium' || browser.name === 'canary') {
      args.push('--auto-open-devtools-for-tabs')

      return args
    }
  })
}

[geralfonso]

There is no workaround for electron?

[dwelle]

@geralfonso There is, by setting args.devTools = true (which is an option custom to Cypress) --- Electron-browser only.

on('before:browser:launch', (browser = {}, args) => {
  if (browser.name === 'chrome') {
    args = args.concat('--auto-open-devtools-for-tabs');
  } else if (browser.name === 'electron') {
    // Cypress-specific option (state)
    args.devTools = true;
  }
  return args;
});

[Mark-J-Lawrence]

I could really do with this when run'ing headless with electron. At the moment a test is failing during a Jenkins build and it would have been helpful to force the devTools to open so I could see the console errors in the screenshot.

I've tried @dwelle suggestion, but it doesn't work on 3.8.1. devTools = true is definitely being sent though as I can see it in the DEBUG output.

UPDATE: I've worked around it by using the suggested workarounds to print all console output to stdout. Now I can see the browser errors

[mikhaildzarasovvineti]

@mjlawrence83 can you post a link with that workaround? Thanks

[FFdhorkin]

@geralfonso There is, by setting args.devTools = true (which is an option custom to Cypress) --- Electron-browser only.

on('before:browser:launch', (browser = {}, args) => {
  if (browser.name === 'chrome') {
    args = args.concat('--auto-open-devtools-for-tabs');
  } else if (browser.name === 'electron') {
    // Cypress-specific option (state)
    args.devTools = true;
  }
  return args;
});

This is not quite right (or it's out of date), based on my testing. Here's what I have:

    on('before:browser:launch', (browser = {}, launchOptions) => {
        if (browser.family === 'chromium' && browser.name !== 'electron') {
            launchOptions.args.push('--auto-open-devtools-for-tabs');
        } else if (browser.family === 'firefox') {
            launchOptions.args.push('-devtools');
        } else if (browser.name === 'electron') {
            launchOptions.preferences.devTools = true;
        }

        return launchOptions;
    });

[nukboda]

Please can you tell me in which file I should update ,this commands. Can you provide with entire example with file name and file contents fully.

[Nesay-hub]

Please can you tell me in which file I should update ,this commands. Can you provide with entire example with file name and file contents fully.

[rbabayoff]

For electron, launchOptions.preferences.devTools = true only allows the opening of dev tools, doesn't actually open it.

To open it, the documentation states that you need to call BrowserWindow.webContents.openDevTools(). BrowserWindow is an electron object. How do I get access to it from my tests, if at all possible?

@mjlawrence83 , I have the same problem as you - a test that fails only in ci. 2 days on it already trying figure out how to open devtools or redirect output. How did you end up printing all console output to stdout?

Cypress team - on behalf of myself and I'm sure many many others (saw quite a few related issues), it is very frustrating that easy access to console output to debug ci only problems is not part of the cypress core.

[jennifer-shehane]

This is now possible in all supported browsers with the workaround below. See https://on.cypress.io/browser-launch-api#Modify-browser-launch-arguments

Closing as resolved.

// plugins/index.js
module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, launchOptions) => {
    // `args` is an array of all the arguments that will
    // be passed to browsers when it launches
    console.log(launchOptions.args) // print all current args

    if (browser.family === 'chromium' && browser.name !== 'electron') {
      // auto open devtools
      launchOptions.args.push('--auto-open-devtools-for-tabs')
    }

    if (browser.family === 'firefox') {
      // auto open devtools
      launchOptions.args.push('-devtools')
    }

    if (browser.name === 'electron') {
      // auto open devtools
      launchOptions.preferences.devTools = true
    }

    // whatever you return here becomes the launchOptions
    return launchOptions
  })
}

[mellis481]

@jennifer-shehane This only appears to work with the Cypress test runner, not when you run Cypress headlessly (as @mjlawrence83 and @rbabayoff mentioned above). IMO automatically opening devtools during cypress open is a slight convenience whereas opening devtools while running Cypress headlessly would be of enormous benefit in situations where the cy.log'ed error isn't helpful and one needs to troubleshoot an AOT error.

Please advise.

[kpittman-securus]

@jennifer-shehane we have the same use case as @mellis481 and @mjlawrence83, the logged Cypress error is basically useless in some cases for us, especially in async code. If we perform several API calls to set up our environment, sometimes we will just get a message that says basically just "500". We can't use cy.request because we need to use multiple results and we don't want to nest cy.request.thens. The stack traces are unhelpful, they don't point to the offending line of code, but instead point to some plumbing code like XHR. It would be so useful to just open the devtools and have them appear in the screenshots and recordings when running headless on our CI builds. I mean, it would be nicer if we could just have the error and stacktrace in the Cypress logs, but that seems unobtainable.

So basically, our team has the same need as @mellis481 and @mjlawrence83, and we are looking for advice as well.

[Crysknight]

For electron, launchOptions.preferences.devTools = true only allows the opening of dev tools, doesn't actually open it.

To open it, the documentation states that you need to call BrowserWindow.webContents.openDevTools(). BrowserWindow is an electron object. How do I get access to it from my tests, if at all possible?

+1 to that. Is there any way right now to access BrowserWindow from Cypress related code @jennifer-shehane? Thank you in advance

[jennifer-shehane]

So, this option works when:

• Running via cypress open
• Running via cypress run with any headed browser

So you can see the devTools open while it runs. However, the video and screenshots are not capturing the devTools. So I think that is the more specific issue now, that the devTools panel needs to be captured in screenshots and videos so that it can be reviewed.

As for printing the error that occurs in your devTools, this should be possible, similar to the https://github.com/flotwig/cypress-log-to-output plugin. If you create a task to console.log, you can print anything to the stdout. There's a few issues concerning this: #700 #300

[kpittman-securus]

For reference, this is what I was referring to when doing multiple async calls in a cy.wrap(null).then(async () => {...}) function during our before.

The stacktrace reads:

Error: Request failed with status code 400
    at createError (http://34.229.239.122:32770/__cypress/tests?p=src/smokeTests/smokeTests.spec.ts:897:15)
    at settle (http://34.229.239.122:32770/__cypress/tests?p=src/smokeTests/smokeTests.spec.ts:1172:12)
    at XMLHttpRequest.handleLoad (http://34.229.239.122:32770/__cypress/tests?p=src/smokeTests/smokeTests.spec.ts:364:7)

Which is unhelpful as well.

The various Cypress logging plugins we have found have trouble with reporting during before hooks, and the cypress-har-generator plugin would have been perfect - except that it doesn't support Electron, which we use exclusively during CI runs.

If that screenshot and video could have had the devtools open, it would simplify our efforts to find the root cause of the actual 400 errors. Yes, we can examine the application logs from the container that was running during the tests, but it's a lot more effort than just getting a meaningful report of the error from the test itself.

[WORMSS]

@jennifer-shehane is there any update on recording devTools during a headless cypress run?
It seems it's the only way to get any meaningful information on an intermittent (possibly cors) problem I have been trying to track down for 3 days, and the server team won't even look at it until I can prove the server/load-balancer is causing the problem?

[chuckienorton]

This is now possible in all supported browsers with the workaround below. See https://on.cypress.io/browser-launch-api#Modify-browser-launch-arguments

Closing as resolved.

// plugins/index.js
module.exports = (on, config) => {

Hey Cypress team!!!

I would also love the ability to view devTools console logs from CI. Weirdly, while the above code does open devTools from the cypress open commend (locally), it actually HIDES the screenshots/recordings in cypress dashboard.

Here is a screenshot from the cypress dashboard before trying to open dev tools.

Here is after applying the above code to the plugins.js file.

It actually removed the portion of the image with the browser + devTools, only showing the cypress console on left now. Very strange.

This would be super helpful since it's complicated to perfectly test github actions locally.

Thanks in advance!

[ericcmmi]

Found this after several days of trying to debug a ci only issue inside a container. Life is too short for this.

[JohnnyDevNull]

Push... running in the same problem here. We need to take a look at the log's in CI run, but that seems not easy to handle
Does anyone have a stable solution for this?

[ccannawright]

would also like to see the dev console logs in videos for failures only seen running tests in CI - even if there's just a way to log those - that would be fine as well

[dannyshaw]

Or somehow dump the entire console log to a file we can inspect/upload as a CI artefact?

[Dinika]

+1 to this. We are looking forward to this feature as well. Any updates on this?

[anselmbradford]

I added the example from https://docs.cypress.io/api/plugins/browser-launch-api#Open-devtools-by-default and merely adding it as shown crashes Cypress with Error: The Test Runner unexpectedly exited via a exit event with signal SIGSEGV

[BloodyRain2k]

I've tried the same as Ansel and while my Cypress at least didn't crash, it didn't really work either.

While Electron and Edge opened the dev tools on it's own as expected in Open mode, it was completely pointless for Runner mode.
Because as Chuckie showed in his screenshots, Cypress just cuts everything off instead. Videos included.
It also seems to throw the dimensions off massively, as the "mainwindow" area of the Runner is crushed so much that 1280x800 on a 2560x1440 display ends up at 13%.
A seemingly "massive" Cypress log on the left, a tiny sliver of page on the right, and the entire space where the console presumively was missing.

What I wonder about though, is whenever or not Cypress is just capturing the wrong area of the window, or if Edge is actively interferring on it's own for "security reasons".
Like how Firefox doesn't allow the page screenshots to be done on for example Tampermonkey's tab.

Electron on the other hand does not even care anymore in Runner mode and refuses to open the dev tools.
This does preserve the resolution though.

And while it's nice not having to open dev tools each start of Cypress Open, this sadly doesn't do what it was actually intended for: include the console in failure screenshots...

Edit: out of curiousity I've tried to enable screenshotOnRunFailure in Open mode, to see if it changes anything about the screenshot behavior.
Instead I've found out that the Node events before:run and before:spec do NOT get called in Open mode.
After that I've also found out that screenshotOnRunFailure seems to be ignored in Open mode anyways.
At least did I not get it working after shoving it right at the beginning of a test, next to a piece of code I knew to get executed.

[mellis481]

I just upgraded to Cypress 13 and it seems like this ticket can now be closed because what is called out can be accomplished with Test Replay. 🎉

[jennifer-shehane]

Closing this issue as we don't intend to extend the behavior of opening devtools during screenshots and videos in the future.

Much of where Cypress is headed is in capturing the DOM, events, and debugging logs and allowing those to be replayed via Test Replay. You can see console logs, including errors that printed during your run and inspect them in your browsers devtools using it.

Screenshots and videos were a great step for debugging beyond obtuse error messages of the past, but much of what it offers is flawed in its own ways, giving a limited view and lacking the real time exploration that Test Replay offers. Thanks for all of your feedback on this issue. 🙏🏻

[FFdhorkin]

Closing this issue as we don't intend to extend the behavior of opening devtools during screenshots and videos in the future.

Much of where Cypress is headed is in capturing the DOM, events, and debugging logs and allowing those to be replayed via Test Replay. You can see console logs, including errors that printed during your run and inspect them in your browsers devtools using it.

Screenshots and videos were a great step for debugging beyond obtuse error messages of the past, but much of what it offers is flawed in its own ways, giving a limited view and lacking the real time exploration that Test Replay offers. Thanks for all of your feedback on this issue. 🙏🏻

Frankly, this is a bit disappointing to hear, but not surprising (given Cypress is a business). Cypress Cloud is cool, and Test Replay looks like useful functionality, but realistically, it's a paid feature, and the vast majority of people using Cypress are not paid customers.

[BloodyRain2k]

Frankly, this is a bit disappointing to hear, but not surprising (given Cypress is a business). Cypress Cloud is cool, and Test Replay looks like useful functionality, but realistically, it's a paid feature, and the vast majority of people using Cypress are not paid customers.

Not only that but I personally also don't see why I even need to involve the cloud when I have everything happening on my local workstation. If you're already running testing and building in the cloud, whatever. At that point it even makes sense. But for a local project?
The main used for me would be debugging tests, and the reason I often can't use "open mode" for that is Cypress running the browser (doesn't matter which) in a way that it literally kills itself when the memory usage reaches 4 GB, no damns given about the station still having 6-8 GB unused memory free at that point...
And having to use the cloud for that? That just rubs me the wrong way on principle...
I'm really hoping that there's gonna be an "offline replay" feature down the road.