cmd/go/internal/lockedfile: spurious EDEADLK failures on AIX and Solaris

[bcmills]

From https://build.golang.org/log/f268986231f8f43781dacb0f5fe56772143ca24b (on aix-ppc64):

go test proxy running at GOPROXY=http://127.0.0.1:34444/mod
--- FAIL: TestScript (0.00s)
    --- FAIL: TestScript/mod_concurrent (1.51s)
        script_test.go:191: 
            # Concurrent builds should succeed, even if they need to download modules. (0.924s)
            > go build ./x &
            > go build ./y
            [stderr]
            go: can't lock version list lockfile: Lock $WORK/gopath/pkg/mod/cache/download/golang.org/x/text/@v/list.lock: deadlock condition if locked
            [exit status 1 (core dumped)]
            FAIL: testdata/script/mod_concurrent.txt:5: unexpected command failure
            
FAIL
FAIL	cmd/go	106.612s

I suspect that this is actually an AIX kernel bug (namely, being overly pessimistic about the deadlock properties of file locks in a multi-threaded program), but perhaps we should consider some sort of workaround.

CC @Helflym @jayconrod

[Helflym]

Yeah, I'm already aware of this bug. I know that it can happen sometimes on AIX 7.2 TL0.
However, I think that's the first time I'm seeing it on AIX 7.2 TL2 (the AIX version on the builder)...

I've made a old workaround a few months back (cf CL 152718). It could be updated and merged if we need a quick workaround while we're trying to find a true fix (if any).

[bradfitz]

Approximately the same thing on Illumos:

https://build.golang.org/log/2b8036635d51414fe4f6f3421dee2e27bc2a6577

It returns deadlock situation detected/avoided, which is error EDEADLK.

/cc @jclulow

[bcmills]

2019-11-13T18:03:37-e762378/illumos-amd64
2019-11-13T14:12:20-54cf776/illumos-amd64
2019-11-12T22:30:48-995ade8/illumos-amd64
2019-11-12T01:09:06-5b17b65/illumos-amd64
2019-11-11T17:44:21-275a7be/illumos-amd64
2019-11-02T22:02:43-3873e54/illumos-amd64
2019-11-02T21:51:14-a3ffb0d/illumos-amd64
2019-11-02T18:30:37-35cfe05/aix-ppc64
2019-11-02T05:36:43-971ec87/illumos-amd64
2019-10-22T18:26:35-3db6d46/illumos-amd64
2019-10-16T23:14:14-3b003c3/aix-ppc64
2019-10-16T06:29:51-03bb3e9/aix-ppc64
2019-10-08T20:05:42-d7dd91a/aix-ppc64
2019-09-02T15:31:19-060669a/aix-ppc64
2019-07-30T02:52:57-fbb819e/aix-ppc64
2019-06-27T19:58:49-f1a8ca3/aix-ppc64
2019-04-30T17:45:16-1fd1408/aix-ppc64
2019-02-25T23:01:48-05e77d4/solaris-amd64-oraclerel

[bcmills]

Illumos, at least, seems to have an flock implementation now, and the other platforms that provide flock do not seem to exhibit EDEADLK flakes.

So probably the ideal solution on that platform is to implement syscall.Flock (filed as #35618) and adjust the tags for lockedfile/internal/filelock to use the flock variant on illumos.

[gopherbot]

Change https://golang.org/cl/222277 mentions this issue: cmd/go/internal/modload: use a retry loop to suppress EDEADLK on AIX and Solaris

[bcmills]

Amusingly, the regression test I added closely parallels this nearly-19-year-old bug report!
https://bugzilla.mozilla.org/show_bug.cgi?id=62457#c5