testutil.CheckDeepEqual: fatal error: checkptr: unsafe pointer arithmetic

[tstromberg]

make test on my machine has several tests that fail with the same panic message, all from uses of go-cmp.

For example:

goroutine 106 [running]:
runtime.throw(0x2c04a2b, 0x23)
	/usr/local/go/src/runtime/panic.go:1112 +0x72 fp=0xc000728fa0 sp=0xc000728f70 pc=0x1074352
runtime.checkptrArithmetic(0xc00071aa58, 0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/checkptr.go:41 +0xb5 fp=0xc000728fd0 sp=0xc000728fa0 pc=0x1045e15
github.com/google/go-cmp/cmp.retrieveUnexportedField(0x2af6740, 0xc00071aa50, 0x199, 0x28cfc48, 0xa, 0x299d9a3, 0x43, 0x2ec82e0, 0x2b03040, 0x0, ...)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/export_unsafe.go:22 +0x7d fp=0xc000729030 sp=0xc000728fd0 pc=0x27689ad
github.com/google/go-cmp/cmp.StructField.Values(0xc00064e000, 0x0, 0x0, 0x2ec82e0, 0x199, 0x199, 0x8a55008)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/path.go:190 +0x187 fp=0xc0007291e8 sp=0xc000729030 pc=0x276c507
github.com/google/go-cmp/cmp.(*valueNode).PushStep(0xc000701a20, 0x2e90660, 0xc00064e000, 0x1072626)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/report_value.go:54 +0x5d fp=0xc000729340 sp=0xc0007291e8 pc=0x278058d
github.com/google/go-cmp/cmp.(*defaultReporter).PushStep(0xc000702910, 0x2e90660, 0xc00064e000)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/report.go:24 +0x6e fp=0xc000729380 sp=0xc000729340 pc=0x276dede
github.com/google/go-cmp/cmp.(*state).compareAny(0xc000704c00, 0x2e90660, 0xc00064e000)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/compare.go:214 +0x22d fp=0xc0007294f0 sp=0xc000729380 pc=0x2760c3d
github.com/google/go-cmp/cmp.(*state).compareStruct(0xc000704c00, 0x2ec82e0, 0x2af6740, 0x2af6740, 0xc00071aa50, 0x199, 0x2af6740, 0xc00071a960, 0x199)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/compare.go:383 +0x548 fp=0xc0007296b0 sp=0xc0007294f0 pc=0x2764338
github.com/google/go-cmp/cmp.(*state).compareAny(0xc000704c00, 0x2e905a0, 0xc00070c900)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/compare.go:252 +0x1a58 fp=0xc000729820 sp=0xc0007296b0 pc=0x2762468
github.com/google/go-cmp/cmp.(*state).comparePtr(0xc000704c00, 0x2ec82e0, 0x2b12040, 0x2b12040, 0xc00071aa50, 0x16, 0x2b12040, 0xc00071a960, 0x16)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/compare.go:513 +0x526 fp=0xc0007298b0 sp=0xc000729820 pc=0x2766f26
github.com/google/go-cmp/cmp.(*state).compareAny(0xc000704c00, 0x2e836e0, 0xc00070c8c0)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/compare.go:258 +0x18e6 fp=0xc000729a20 sp=0xc0007298b0 pc=0x27622f6
github.com/google/go-cmp/cmp.Equal(0x2b12040, 0xc00071aa50, 0x2b12040, 0xc00071a960, 0xc00070aae0, 0x2, 0x2, 0x11cb2b2)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/compare.go:107 +0x406 fp=0xc000729b68 sp=0xc000729a20 pc=0x275f606
github.com/google/go-cmp/cmp.Diff(0x2b12040, 0xc00071aa50, 0x2b12040, 0xc00071a960, 0xc000702900, 0x1, 0x1, 0x0, 0x0)
	/Users/tstromberg/src/skaffold/vendor/github.com/google/go-cmp/cmp/compare.go:127 +0x1ce fp=0xc000729c08 sp=0xc000729b68 pc=0x275feae
github.com/GoogleContainerTools/skaffold/testutil.CheckDeepEqual(0xc00001c480, 0x2b12040, 0xc00071a960, 0x2b12040, 0xc00071aa50, 0xc000702900, 0x1, 0x1)
	/Users/tstromberg/src/skaffold/testutil/util.go:254 +0x9e fp=0xc000729c80 sp=0xc000729c08 pc=0x27c462e
github.com/GoogleContainerTools/skaffold/testutil.(*T).CheckDeepEqual(0xc00070aaa0, 0x2b12040, 0xc00071a960, 0x2b12040, 0xc00071aa50, 0xc000702900, 0x1, 0x1)
	/Users/tstromberg/src/skaffold/testutil/util.go:133 +0xbd fp=0xc000729cd0 sp=0xc000729c80 pc=0x27c2e1d
github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/cluster.TestNewBuilder.func1(0xc00070aaa0)
	/Users/tstromberg/src/skaffold/pkg/skaffold/build/cluster/types_test.go:102 +0x35c fp=0xc000729e48 sp=0xc000729cd0 pc=0x28be24c
github.com/GoogleContainerTools/skaffold/testutil.Run.func1(0xc00001c480)
	/Users/tstromberg/src/skaffold/testutil/util.go:238 +0xed fp=0xc000729ed0 sp=0xc000729e48 pc=0x27c5c3d
testing.tRunner(0xc00001c480, 0xc0007028d0)
	/usr/local/go/src/testing/testing.go:993 +0x1ec fp=0xc000729fd0 sp=0xc000729ed0 pc=0x11cc33c
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc000729fd8 sp=0xc000729fd0 pc=0x10a8751
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1044 +0x661

This is on macOS 10.14.6, go 1.14rc1, and Docker is not installed.

[tstromberg]

Here are the tests that panic when make test is run:

FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/cluster	2.250s
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/local	1.470s
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/debug	2.224s
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/deploy	3.582s
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes	1.770s
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes/portforward	2.945s
FAIL	github.com/GoogleContainerTools/skaffold/pkg/skaffold/trigger	0.415s

Curiously, if I run just tests individually, they do not panic:

ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/cluster	(cached)
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/local	0.288s
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/debug	0.364s
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/deploy	0.496s
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes	0.336s
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes/portforward	1.207s
ok  	github.com/GoogleContainerTools/skaffold/pkg/skaffold/trigger	0.215s

This reeks of global state and abstraction leakage. My instinct says that there are two bugs: one within CheckDeepEqual / go-cmp, and one in a specific test being executed that poisons the other tests.

[tstromberg]

I've narrowed it down to the hidden use of -race when make test is used.