contest: Add "critical" status?

[matttbe]

I understand that the goal is first to validate pending patches, than finding new bugs and showing the current state of net-next/net, but I think it might still be good to easily identify kernel crashes and warnings from the UI.

If I'm not mistaken, currently if a kernel panic is detected, the test will be marked as failed, like any issues we could have with the test. If it is possible to mark the tests that caused kernel oops and warnings, it might help identifying problematic patches quicker (and possibly bugs that were not introduced by the new patches). Could be fixed by introducing a new "critical" status?

I guess people having access to the server can easily grep in the logs to find crashes and warnings. I didn't find a way to list critical issues from the UI.

[kuba-moo]

The other option is to add the crash info to result JSON. We already try to generate a "fingerprint" of the crash to be able to filter, so we could easily add an array of test name + crash fingerprint. UI can either cross-correlate that to render "critical" status or display crashes as a separate widget. WDYT?

[matttbe]

Nice, even better :-)

(I forgot that with this CI, everything is controlled, so it is "easy" to share more data, and display more useful items ; you are not limited by a specific interface, e.g. GitHub Actions Summary view :) )

[kuba-moo]

:) In this case the "do it yourself" tradeoff is to our benefit :)
Coincidentally I was also thinking about a JSON extension to re-run failing tests, and report "retry" results. Make investigating easier for the maintainer ("can this be a flake or is the failure persistent"). But I can't quite connect the dots on how to present that one, and we'd need more info about runtime deadlines to avoid retrying crazy long tests in net and net/forwarding.
As always feel free to hack on any of this, the crashes are all the way down at number 10 on my todo list :)

[matttbe]

Good idea for the "retry" results. I don't know how to present that. Maybe by using another colour in "flakes"? Maybe in "contest" and "status", the status could be "unstable" instead of "fail", and a link to the re-run?

Similarly, maybe it is not needed to display a separate widget with the crash, just a link to the logs with the crash? I don't know how to present the crash fingerprint.