Ensure code excerpt is from test file, even if assertion failure occurs elsewhere

[billyjanitsch]

Description

When you import and use a test macro from another file and an assertion in that macro fails, the AVA output doesn't include the file path of the failed test -- only the path of the macro. This makes debugging difficult, especially if the test has a macro-generated title.

Error Message & Stack Trace

  1 failed

  1 + 2 = 4

  /Users/billy/work/ava-macro/test/_macros.js:2

   1: export function fooMacro(t, a, b, expected) {
   2:   t.is(a + b, expected)                      
   3: }                                            

  Difference:

  - 3
  + 4
npm ERR! Test failed.  See above for more details.

Command-Line Arguments

npm test

Relevant Links

See this minimal reproduction repo.

Environment

node 8.9.4
npm 5.7.1
ava 1.0.0-beta.3

There is a $60.00 open bounty on this issue. Add more on Issuehunt.

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on Issuehunt to raise funds.

[oantoro]

as far as I know, ava extract first line of the error thrown, where this line is the location where the throw happened. I think extracting the calling graph of the error message started from the test file will be useful.

[novemberborn]

There's two things here we could do better. The code excerpt is based on where the assertion lives. I think we can keep that but if it's not from the test file we should present it differently.

If the assertion's stack trace contains call sites from multiple files, or multiple call sites from the test file, we should print it in its entirety. That way you should be able to trace it back to the actual test declaration.

What do you think @billyjanitsch?

We have a related issue at #867. That's a compile-time enhancement however. It'd be an enhancement but I think it's different from what I'm proposing here.

Since this affects the reporters it's blocked by #1722 which makes a lot of changes.

[billyjanitsch]

Yeah, I agree that it's always useful to show the assertion excerpt.

Do we always know where the test file is? If so, I feel like the stack trace should always include everything from the assertion up to the test (which would normally be just the assertion and the test, except in the case of macros).

I'm not sure that it's sufficient to only do this if the trace spans multiple files, because it would still be difficult to debug a failing assertion in a macro with an automatically generated title in the same file.

[novemberborn]

Do we always know where the test file is? If so, I feel like the stack trace should always include everything from the assertion up to the test (which would normally be just the assertion and the test, except in the case of macros).

I'm not sure that it's sufficient to only do this if the trace spans multiple files, because it would still be difficult to debug a failing assertion in a macro with an automatically generated title in the same file.

Yes. We filter out call sites from AVA itself, so async stack traces aside we have all the relevant call sites. Indeed it's not sufficient to check if the trace spans multiple files, which is why I suggested checking if there are multiple call sites from a single file. Does that make sense?

[billyjanitsch]

Indeed it's not sufficient to check if the trace spans multiple files, which is why I suggested checking if there are multiple call sites from a single file. Does that make sense?

Sorry, I had misread what you wrote! Yes, I agree completely.

[IssueHuntBot]

@IssueHunt has funded $60.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[novemberborn]

FWIW I'm not sure if this is still an issue.

[novemberborn]

Both due to the age of this issue, and the state of our reporters, I've decided to roll this into #2501.