net/http: TestServerCancelsReadHeaderTimeoutWhenIdle failing frequently since 2022-09-05

[bcmills]

greplogs -l -e 'FAIL: TestServerCancelsReadHeaderTimeoutWhenIdle'
2022-09-06T15:48:16-07b19bf/windows-amd64-race
2022-09-06T15:44:27-6fcd9b4/dragonfly-amd64-622
2022-09-06T15:44:25-c761409/darwin-amd64-11_0
2022-09-06T14:44:23-a60a3dc/linux-amd64-unified
2022-09-06T11:14:09-1c50484/windows-amd64-newcc-race
2022-09-05T08:12:37-67e6542/windows-amd64-race
2022-09-05T08:08:18-4ad55cd/openbsd-arm64-jsing
2022-09-05T08:08:18-4ad55cd/solaris-amd64-oraclerel
2022-09-05T08:08:09-02700e5/linux-386-softfloat
2022-09-05T08:07:56-7d83d32/openbsd-386-68
2022-09-05T08:07:47-357b922/linux-ppc64-buildlet

(attn @neild)

[bcmills]

This is the regression test added for #54784 (CC @hawkw).

[aclements]

Just to add some detail: this test was added 2022-09-05 in CL 426895. So this test has been flaky from the start.

[hawkw]

Interesting...the test exercises behavior related to timeouts, is it possible it's flaky on CI for that reasons?

[hawkw]

It looks like in all the failures linked in the issue, the test fails on the first request, before waiting for the idle timeout: https://github.com/hawkw/go/blob/09332743ad6d5a9eb1137adaade2810c583d38ca/src/net/http/serve_test.go#L5878-L5880

Sending this initial request is necessary to trigger the incorrect behavior prior to #54784, but isn't actually testing the change itself. So, this isn't a regression, but an issue with the test itself.

This test passes for me locally in the following environment:

$ uname -a
Linux noctis 5.18.19 #1-NixOS SMP PREEMPT_DYNAMIC Sun Aug 21 13:18:56 UTC 2022 x86_64 GNU/Linux

so I wonder if there's something specific to CI environment that's causing it to be flaky? As a first-time contributor, I'm not super familiar with the environment these tests are run in, so any details that effect tests like this would be very helpful!

[mknyszek]

I came upon this recently in a trybot (https://storage.googleapis.com/go-build-log/3a262583/openbsd-amd64-70_38a0ba2b.log) and it blocked an auto-submit.

I too tried to reproduce this locally with stress and was unable to. Any objections to skipping this test on the platforms we've seen the failures on?

[bcmills]

The failures don't seem to be platform-specific.

I suggest that we either revert the CL that introduced the test (which can be resent and merged once the test is fixed), or skip the entire test always using testenv.SkipFlaky until it can be fixed.

[bcmills]

@hawkw, generally any test that has specific timeout behavior will be flaky on CI machines, because they tend to be much more heavily loaded that developer workstations.

I took a look at the test and left some more specific comments on https://go.dev/cl/426895.

[toothrot]

2022-09-09T20:29:05-54182ff/linux-riscv64-jsing
2022-09-09T20:25:35-59ba97b/linux-amd64-race
2022-09-09T18:42:57-b244b89/linux-mips-rtrk
and others

[gopherbot]

Change https://go.dev/cl/430955 mentions this issue: net/http: deflake TestServerCancelsReadHeaderTimeoutWhenIdle

[bcmills]

I got tired of restarting TryBots for this, so I mailed https://go.dev/cl/430955 with a fix.