add option to fail tests that print errors and warnings

[harrykao]

Do you want to request a feature or report a bug?

Feature.

What is the current behavior?

Tests pass even if they print errors and warnings to the console.

What is the expected behavior?

It would be nice if we could force the test to fail.

Our test suite prints lots of things like this:

    console.warn node_modules/moment/moment.js:293
      Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non RFC2822/ISO date formats are discouraged and will be removed in an upcoming major release. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.

    console.error node_modules/fbjs/lib/warning.js:33
      Warning: Each child in an array or iterator should have a unique "key" prop.

(node:15601) UnhandledPromiseRejectionWarning: SyntaxError: The string did not match the expected pattern.

    console.error node_modules/fbjs/lib/warning.js:33
      Warning: `value` prop on `input` should not be null. Consider using an empty string to clear the component or `undefined` for uncontrolled components.

These are all legitimate errors that we should fix, yet our automated tests pass. If Jest could be made to fail, that would encourage developers to keep the tests clean.

[apaatsio]

This can already be achieved with a couple lines of code.

const spy = jest.spyOn(global.console, 'error')
// ...
expect(spy).not.toHaveBeenCalled();

[rickhanlonii]

@apaatsio is correct - the way to do it is something like that example

[chrisstevensap]

@apaatsio how would you suggest someone implement this so that it runs across all test suites without having to include it in each individual test suite?

[SimenB]

setupFiles, you can throw instead of assert

[lukeapage]

To get it working I've ended up with this ugly thing

beforeEach(() => {
    isConsoleWarningOrError = false;
    const originalError = global.console.error;
    jest.spyOn(global.console, 'error').mockImplementation((...args) => {
        isConsoleWarningOrError = true;
        originalError(...args);
    });
    const originalWarn = global.console.warn;
    jest.spyOn(global.console, 'warn').mockImplementation((...args) => {
        isConsoleWarningOrError = true;
        originalWarn(...args);
    });
});

afterEach(() => {

    if (isConsoleWarningOrError) {
        throw new Error('Console warnings and errors are not allowed');
    }
});

requirements:

1. still log the warning or error like it does currently
2. don't add a new stack trace to it
3. run automatically - don't have to add to every spec
4. still allow tests to mock console and add their own implementations and assert number of calls (this is why I can't just expect it to be called - our own logging code mocks console and tests the fn calls)

[lukeapage]

Note: I've tried lots of simpler ideas and they all fell down somehow.

[stevula]

Is there a preferred way to do this globally that doesn't involve adding a beforeEach/All or afterEach/All to every test file? I want all tests to fail if any console.error is called.

I'm looking at the solution from here, which seems to do the job:

let error = console.error

console.error = function (message) {
  error.apply(console, arguments) // keep default behaviour
  throw (message instanceof Error ? message : new Error(message))
}

[siddharthkp]

@stevula For what it's worth, that's exactly what works for me

[rickhanlonii]

Yup, that's the preferred way to do that globally

[akx]

Thanks, @stevula – that's close, but the problem is that the %s interpolation supported by the console isn't supported by thrown messages:

Error: Warning: Received `%s` for a non-boolean attribute `%s`.

If you want to write it to the DOM, pass a string instead: %s="%s" or %s={value.toString()}.%s

You can see the interpolated message too, but it's a little cumbersome. :)

[valgussev]

@akx as an option you can iterate through all the arguments and replace all possible substitutions strings with values, i.e.

console.error = (error, ...args) => {
  let errorMessage = typeof error === 'string' ? error : error.message;
  args.forEach(argument => {
    errorMessage = errorMessage.replace(/%(s|d|i|o|O)/, argument);
  });
  throw new Error(errorMessage);
};

[icaropires]

Is there a preferred way to do this globally that doesn't involve adding a beforeEach/All or afterEach/All to every test file? I want all tests to fail if any console.error is called.

I'm looking at the solution from here, which seems to do the job:

let error = console.error

console.error = function (message) {
  error.apply(console, arguments) // keep default behaviour
  throw (message instanceof Error ? message : new Error(message))
}

When I tried this option I got an error on executing because of Create React App:

So I wrote a bash script to for manually looking for consoles and warnings (it can be found here).

Later I discovered that options not supported by React Create App on jest configurations can be used via the command line. So one could run:

npm test -- --setupFiles './tests/jest.overrides.js'

instead of changing package.json, and the solution pointed by @stevula would work.

[scotthovestadt]

@icaropires If you want to disallow console.error (etc), I'd suggest doing it as a lint rule.

[icaropires]

@scotthovestadt Thanks for the suggestion. I did put it as a lint rule for my team's code. But in case of warnings, consoles were being called from inside my libraries:

This picture from my CI shows that.

[sdushay]

@lukeapage Thank you for contributing that solution (#6121 (comment)) -- just a note about it:

You're going to get Maximum Call Stack Size Exceeded when you call originalError/originalWarning, because you're still spying on the parent global.console object, so it recurses.

May I suggest either using console.log in their place if you want the whole stack trace, or just letting jest spit out the original error/warning to stdout, and throwing the error as usual?

See below for revised code:

let isConsoleWarningOrError;

beforeEach(() => {
  isConsoleWarningOrError = false;
  jest.spyOn(global.console, 'error').mockImplementation((...args) => {
    isConsoleWarningOrError = true;
    // Optional: I've found that jest spits out the errors anyways
    console.log(...args);
  });
  jest.spyOn(global.console, 'warn').mockImplementation((...args) => {
    isConsoleWarningOrError = true;
    // Optional: I've found that jest spits out the errors anyways
    console.log(...args);
  });
});

afterEach(() => {
  if (isConsoleWarningOrError) {
    throw new Error('Console warnings and errors are not allowed');
  }
});

[lukeapage]

@sdushay it does not recourse because once you grab a reference locally it is immune to spying. That’s why it’s calling original. The code above or a slight variation on it runs and works for us.

[sdushay]

@lukeapage You're right, I don't know what was blowing our call stack.

[greypants]

Here's the variant I landed on, using Node's Util.format function to preserve console string formatting via arguments in the error output:

// setupTests.js
import { format } from "util";

const error = global.console.error;

global.console.error = function(...args) {
  error(...args);
  throw new Error(format(...args));
};

cc @akx, re:

Error: Warning: Received %s for a non-boolean attribute %s.

[axelboc]

@greypants' version does not work for me with async tests. I'm trying to detect errors logged by React Testing Library - e.g. when a render is triggered after a test has completed. Throwing in console.error does not fail the async test that logs the error ... but fails the tests that follow it!

The version below, adapted from @sdushay, works nicely. It fails any async test that logs an error or warning to the console without affecting the others. Note that it is also compatible with ESLint's no-console rule and Jest's resetMocks option.

let consoleHasErrorOrWarning = false;
const { error, warn } = console;

global.console.error = (...args) => {
  consoleHasErrorOrWarning = true;
  error(...args);
};
global.console.warn = (...args) => {
  consoleHasErrorOrWarning = true;
  warn(...args);
};

afterEach(() => {
  if (consoleHasErrorOrWarning) {
    consoleHasErrorOrWarning = false;
    throw new Error('Console has error or warning');
  }
});

[ValentinH]

We also needed this and having a working version both for sync and async tests was not trivial. Especially if you don't want to lose the stacktrace, have a nice output and not have weird behaviors.

We ended up using some code from React's repository and I made it compatible with plain Jest : https://github.com/ricardo-ch/jest-fail-on-console

[vmuresanu]

I have also stumbled uppon the maximum callstack problem referenced when using the #6121 (comment) solution. But to be honest I didn't understand the point of calling error(...args); or error.apply(console, arguments). Hence why upon removing this line, and ended with:

console.error = function(...args) {
  throw new Error(format(...args));
};

console.warn = function(...args) {
  throw new Error(format(...args));
};

This was the result prior to overriding console warns:

And this is the result with above code:

The initial point was to fail a test if it throws internally console.warns or console.errors. And this works for me. I haven't tried the async part yet (as mentioned in #6121 (comment)), but for now it works just fine and also it preserved the original stack trace.

[donaldpipowitch]

Building up on @axelboc's variant:

// check if console was used
let usedConsole = false;
['log', 'error', 'warn'].forEach((key) => {
  const originalFn = console[key];
  console[key] = (...args) => {
    usedConsole = true;
    originalFn(...args);
  };
});

// check if a test failed
// see https://stackoverflow.com/a/62557472/1337972
jasmine.getEnv().addReporter({
  specStarted: (result) => (jasmine.currentTest = result),
});

afterEach(() => {
  // if test hasn't failed yet, but console was used, we let it fail
  if (usedConsole && !jasmine.currentTest.failedExpectations.length) {
    usedConsole = false;
    throw `To keep the unit test output readable you should remove all usages of \`console\`.
They mostly refer to fixable errors, warnings and deprecations or forgotten debugging statements.

If your test _relies_ on \`console\` you should mock it:

\`\`\`
const errorSpy = jest.spyOn(console, 'error');
errorSpy.mockImplementation(() => {});
errorSpy.mockRestore();
\`\`\`
`;
  }
});

[LucasBrandt]

If you are using jest-environment-jsdom, the solutions here might not work for you. None of them worked for our project.

We are using jest-environment-jsdom along with jest-environment-jsdom-global to enable manipulating the jsdom object globally for our tests. This code worked well enough - tests now fail when there is a console error, and we can see the logged error - but I'm sure there are potential improvements.

In jest config:
setupFiles: ["<rootdir>/test/setup/jsdom.js"

in jsdom.js:

import { format } from "util";

const { JSDOM, VirtualConsole } = require("jsdom");
const virtualConsole = new VirtualConsole();

global.console.error = function(...args) {
  throw new Error(format(...args));
};

virtualConsole.sendTo(global.console);

// You might not need the html tags here, we were already using them in our project so I've included it here
const jsdom = new JSDOM("<!doctype html><html><body></body></html>", {
  virtualConsole
});

export default jsdom;

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.