consider disallowing skip_report: false for non-blocking jobs

[BenTheElder]

I think at least currently these jobs have largely been a source of confusion rather than a useful signal. Jobs can always still be checked via gubernator / deck / test-grid without having their status on GitHub. We should consider updating the config presubmits to disallow non-blocking jobs being set to report to GitHub.

Previously brought up here: #4913 (comment)

/area prow

[0xmichalis]

cc @stevekuznetsov

[stevekuznetsov]

Generally I think the comment should be re-structured ... see #4602

If we create a comment that has clear messaging on "these are the tests that are currently valid for your PR, tested against your latest HEAD, please look at Gubernator for historical data" I believe you are correct @BenTheElder that only blocking tests should publish statuses

[BenTheElder]

I think besides the bot messages, I've been seeing developers concerned with getting a full set of "green checks" for their PRs and that showing jobs they don't need one for has only caused confusion and frustration.
We can easily fix that by simply not reporting them to GitHub and enforcing that with a presubmit, but I'm not sure if maybe there might be some valid reasons behind our currently allowing them to be reported on GitHub instead of just gubernator / deck / testgrid.

[stevekuznetsov]

Yeah, GitHub has this first-class concept of status for commits which is a natural place for tests to report but it is 100% conflated with "what needs to happen for merge". The status for the submit-queue (or tide, or whatever) is a LOT more complex (have these labels, don't have those, have this milestone, etc) so I agree that using the "green checkmarks" to message to developers things that they need to merge is more appropriate and less confusing than the current approach, as long as they still have somewhere obvious to look at test results.

[BenTheElder]

Discussed today at sig-testing meeting, we intend to make this reality. I've had to fix a few other things on the way but I'll have a PR to make this happen shortly and then we should notify the broader community.

Also @stevekuznetsov brought up some more good points about improving the test result bot comments which we should follow up with at some point.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[BenTheElder]

we cleaned this up a bit but we're still not forcing it, this won't make as much sense with tide anyhow.

/close