Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail test even after it has passed #1330

Open
novemberborn opened this issue Apr 1, 2017 · 4 comments
Open

Fail test even after it has passed #1330

novemberborn opened this issue Apr 1, 2017 · 4 comments
Labels

Comments

@novemberborn
Copy link
Member

09b23e0 made it an error if an assertion is added after a test has finished. This can occur if the test finished earlier than the user intended. Once a test has passed though we cannot update its state in the reporters.

The feedback in 09b23e0 could only be provided to users if the test was waiting for a pending assertion. That behavior itself is confusing, which means that when #1327 is fixed we won't be able to provide any feedback at all.

If a test has passed, but then later an assertion is added before other tests have finished, we should change the test state to a failure and update the reporters. The mini reporter should work well, since it hides previously passing tests. The verbose reporter would have a duplicate entry for the test but perhaps that's OK. I'm not sure whether it's acceptable to add a duplicate entry in the TAP reporter. The alternative is buffering the output for each test file which is not ideal.

@sindresorhus
Copy link
Member

The verbose reporter would have a duplicate entry for the test but perhaps that's OK.

That's ok.

@novemberborn
Copy link
Member Author

This also applies to callback tests ending more than once.

@novemberborn
Copy link
Member Author

Note that this will still be best-effort. The test worker will exit once all tests have completed, even if there is a setTimeout() scheduled which double-ends a callback test, or makes an extraneous assertion.

@novemberborn
Copy link
Member Author

With #1776, we don't write test results until the test file has finished executing. So we'll be able to fail a test even if it seemed to pass at first.

@novemberborn novemberborn added the blocked waiting on something else to be resolved first label Apr 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants