proposal: testing: verify failure (rather than skip)

[lavalamp]

I want to write many tests. Because I run CI, I want them all to pass all the time. Because I have not implemented all of my features, some of them actually fail: I want to skip them. Because over time I want to fix my features, I want to be notified when formerly failing tests begin passing so I can stop skipping them.

There is no good way to do this with the go testing package. I think this is a much better thing to do than just skip as long as the failures are deterministic.

My first thought is to do something like in #16502:

// ShouldFail is meant to be used as: `defer ShouldFail(t)`; it allows the
// test to still be run even when we expect it to fail.
func ShouldFail(t *testing.T) {
        if t.Failed() {
                t.Skip("this test is known to currently be broken")
        } else {
                t.Error("expected a failure, but test exited successfully")
        }
}

This doesn't work, because Skip doesn't un-fail a test, which is reasonable.

My second thought was to write a wrapper for *testing.T which overrides implementation for Error, Fatal, etc methods. This is not good because it's viral: I can't pass the type into functions that were expecting *testing.T as a parameter, because testing.T is type struct.

I can think of a number of ways to solve the problem if I can change the testing.T type:

• Add a "CancelFailure()" method that just clears the failure bit.
• Add a top-level "ShouldFail()" method instead of making people write it.

I can think of more invasive changes, too, but they're not worth mentioning due to the cost of change.

[ianlancetaylor]

Dup of #25951?

[lavalamp]

Definitely related, but I'd be very unhappy with a command-line-flag based solution? It's hard to tell what the actual proposal is there.

[ianlancetaylor]

If this is the same problem as #25951, then let's close it in favor of that one, because that one is on hold for testing v2.

[lavalamp]

I guess that means it'll be considered for v2? If there's no chance of changing v1.x then I guess that's fine.