Warnings are not being shown in junit output

[kylesykes]

Update from Stoplight/Phil: Let's go with the "Convert anything that meets the fail-severity to an error in the case of JUnit" option. It makes sense.

We use the junit output to show errors in Gitlab's Merge Requests, and it's worked great until now. After upgrading to v5, we were trying to setup our builds to fail when spectral catches an error, but would also like to show warnings. However, it appears that the junit output doesn't display warnings, but other formatters do (checked with stylish and json, but haven't rigorously tested all the others).

With stylish formatter, warnings show:

With junit formatter, no warnings shown:

I'd be happy to try and cook up an example if it's needed, but my hope is that it was just either a) an oversight on our part, b) something that's easily spotted to fix or c) intentional but undocumented behavior.

Let me know if you need more information!
Thanks!

[m-denton]

Not sure if this fits here or not but it would also be nice to have the names of these errors appended with a timestamp or other unique identifier (simple number incrementation). This is requested as when loading the junit output into tools like Gitlab CI, if there are multiple juinit entries with the same name, only one of those messages will show in the console. For example, see the following (redacted most of the messages so mismatch on the failure count):

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite package="org.spectral" time="0" tests="103" errors="0" failures="103" name="<classname>.yaml">
<testcase time="0" name="org.spectral.oas3-valid-content-schema-example" classname="<classname>"><failure message="<message> (oas3-valid-content-schema-example)></failure></testcase>
<testcase time="0" name="org.spectral.oas3-valid-content-schema-example" classname="<classname>"><failure message="<message> (oas3-valid-content-schema-example)></failure></testcase>
<testcase time="0" name="org.spectral.oas3-schema" classname="<classname>"><failure message="<message> (oas3-schema) at path #/components/responses/GetPartyTypesOk]]></failure></testcase>
<testcase time="0" name="org.spectral.oas3-valid-schema-example" classname="<classname>"><failure message="<message> (oas3-valid-schema-example)></failure></testcase>
<testcase time="0" name="org.spectral.oas3-valid-schema-example" classname="<classname>"><failure message="<message> (oas3-valid-schema-example)></failure></testcase>
<testcase time="0" name="org.spectral.oas3-valid-schema-example" classname="<classname>"><failure message="<message> (oas3-valid-schema-example)></failure></testcase>
</testsuite>
</testsuites>

Produces this:

Notice it shows 3 failed/error test results.. when in reality, there was 1 true error, then 5 warnings. Since the names are the same in the junit output, Gitlab is bucketing them into the same message which can be deceiving when you may be expecting more warnings/errors.

[P0lip]

If I am not mistaken we did it on purpose in case of JUnit, as JUnit has no way to render other messages than errors or failures.
Initially we used to include all diagnostics.

@philsturgeon do you still recall that issue?

[philsturgeon]

Yeah JUnit does not support warnings only errors, so if you're using JUnit that's all your gonna get. We either had to leave warnings out, or upgrade all warnings to errors, and I chose the former because changing the semantics of what a warning is sounds more confusing than some warnings not showing up.

[kylesykes]

Perfect. We thought that might be the case, but was just confirming. Thanks all!

[philsturgeon]

Thanks @kylesykes! :D

[rhurling]

Hi, I think this issue was somewhat correct and I didn't want to open a new issue for this.
I think the problem is that the JUnit formatter only outputs errors and ignores the --fail-severity (as was planned?)

So even if I want the JUnit formatter to output warnings I can't do that via spectral

[philsturgeon]

@rhurling it's more... that you can't output warnings in JUnit so you can't use JUnit, not that you can't use Spectral. Unless we want to revisit the decision to drop warnings instead of converting warnings to errors for the sake of JUnit seeing them?

If we did that, would it be ok to just convert warnings but not converting info or hint? Or should we convert anything that meets the fail-severity to an error in the case of JUnit?

[rhurling]

@philsturgeon I know that JUnit can't output warnings. But according to the --fail-severity argument shouldn't spectral output output everything that would cause a non-zero exit code as failure in the JUnit formatter?

Which would then work correctly and also should not break anything, since the argument would need to be set by the user to "enable" JUnit to output more than just errors as failures (as far as i understand JUnit works in a boolean success / failure way and I would like the ability to set what would be counted as failure).

I basically would not care about the difference between warning / error when using JUnit and manually setting --fail-severity=warn.

For example if I want to count warnings and errors as failure then I would use the following command:
npx @stoplight/spectral lint --fail-severity=warn --format=junit --output=openapi-spec-report.xml ./openapi-spec.yaml

Edit after more carefully reading the questions:

Or should we convert anything that meets the fail-severity to an error in the case of JUnit?

Yes.

[fgabolde]

I'd like to add a +1 on "convert anything that meets the fail-severity to an error in the case of JUnit". I was upgrading a CI linter job to report warnings and it was pretty surprising that the test started failing (the spectral process returns 1, after all) without any new failures in the report.

I understand JUnit is not a perfect fit for linter output since they typically have a severity that is not supported by JUnit, but lots of tools work best with JUnit reports. With the proposed change, it would at least be possible to report lower severity issues, even imperfectly, whereas now they are completely lost.

Also, I'd like to report that the documentation for the CLI's --format parameter is very sparse:

--format, -f    formatter to use for outputting results    [string] [default: "stylish"]

We only know what the default value is, not what possible values are allowed.

[philsturgeon]

convert anything that meets the fail-severity to an error in the case of JUnit

Okie doke, that can work. I'm not sure we actually had a fail severity at the time, so this might have saved us from this confusing situation.

[P0lip]

Hey @kylesykes @rhurling @fgabolde
We've introduced a fix over here #1375. Could any of you verify whether the implemented solution meets your expectations?

[fgabolde]

Sure, will do first thing tomorrow.

[rhurling]

The solution works for me and outputs warnings via JUnit as desired (when setting --fail-severity=warn)

[fgabolde]

It works for me too.

Also thanks for the extra output in --help!