cmd/go: [email protected]+incompatible currently allows Semantic Import Versioning to be optional

[thepudds]

This is a spinout from "Factor 2" from #31543 ("cmd/go: creating v2+ modules has lower success rate than it could").

Compared to Go 1.11 and 1.12, current tip / 1.13 might have increased the modes of interaction that work with v2+ modules that have not adopted Semantic Import Versioning.

What version of Go are you using (go version)?

go 1.12.6

Does this issue reproduce with the latest release?

Yes, including tip as of yesterday (devel +bc27b64d Tue Jun 18 03:41:59 2019 +0000)

What did you do?

Observe people accidentally creating and using modules that have v2+ semver tags but that have not adopted Semantic Import Versioning.

For example, if there is a module example.com/foo that:

• has a go.mod file (that is, it has adopted modules)
• has a v2.0.0 semver tag as its latest tag
• and its module line reads module example.com/foo (without the in theory required /v2)

then this still works for a module-based consumer, even though the module is in a "bad" state:

go get example.com.com/[email protected]+incompatible

This means people can and do create usable v2+ modules that did not adopt Semantic Import Versioning, and consumers can and do consume those "bad" modules.

However, other things such as upgrades or go get example.com.com/foo@latest do not work as expected, which leads to confusion.

What did you expect to see?

Some type of warning when consuming a v2+ module that has not adopted Semantic Import Versioning (or perhaps an error in the future after the ecosystem has more time to adapt and more tools exist to help prevent or catch mistakes).

What did you see instead?

Here is a concrete example. This uses github.com/pierrec/lz4:

• v2.2.3 has an incorrect go.mod file missing /v2 in the module statement.
• v2.0.6 seems to be when the "bad" go.mod was first introduced.
• v2.0.5 is the tag immediately before the "bad" go.mod was introduced.

First, here is Go 1.12 successfully building a consumer importing the "bad" v2.0.6, as well as this shows a subsequent upgrade not working in 1.12.

cd $(mktemp -d)
export GOPATH=$(mktemp -d)
go mod init tempmod
echo 'require github.com/pierrec/lz4 v2.0.6+incompatible' >> go.mod
cat <<EOF > consumer.go
package consumer
import _ "github.com/pierrec/lz4"
EOF

# build works
go build
go list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.0.6+incompatible

# attempting to upgrade via 'go get -u' does not upgrade lz4
go get -u
go list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.0.6+incompatible

# attempting to upgrade via '@latest' downgrades
go get github.com/pierrec/lz4@latest
go list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.0.5+incompatible

Second, tip is also able to build using the "bad" v2.0.6, but now tip allows upgrading.

cd $(mktemp -d)
export GOPATH=$(mktemp -d)
gotip mod init tempmod
echo 'require github.com/pierrec/lz4 v2.0.6+incompatible' >> go.mod
cat <<EOF > consumer.go
package consumer
import _ "github.com/pierrec/lz4"
EOF

# build works
gotip build
gotip list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.0.6+incompatible

# upgrade now actually does upgrade to a newer "bad" version
gotip get github.com/pierrec/lz4@latest
gotip list -m github.com/pierrec/lz4
# outputs:
# github.com/pierrec/lz4 v2.2.3+incompatible

head -1 $(gotip list -f '{{.Dir}}' github.com/pierrec/lz4)/go.mod
# outputs:
# module github.com/pierrec/lz4

[thepudds]

https://golang.org/cl/181881 might help with this.

[jayconrod]

Hmm, this is unfortunate. In direct mode, it seems we correctly don't list these versions:

$ export GOPROXY=direct
$ go list -m -versions github.com/pierrec/lz4
github.com/pierrec/lz4 v1.0.1 v2.0.1+incompatible v2.0.2+incompatible v2.0.3+incompatible v2.0.4+incompatible v2.0.5+incompatible

However, the proxy knows about them.

$ export GOPROXY=https://proxy.golang.org,direct
$ go list -m -versions github.com/pierrec/lz4
github.com/pierrec/lz4 v1.0.1 v2.0.1+incompatible v2.0.2+incompatible v2.0.3+incompatible v2.0.4+incompatible v2.0.5+incompatible v2.0.6+incompatible v2.0.7+incompatible v2.1.1+incompatible v2.1.2+incompatible v2.2.3+incompatible

And in your example, the Go command validation doesn't catch the problem with these versions, even with GOPROXY=direct. So I think what happened is someone manually wrote in these versions to a go.mod. The proxy fetched them, they passed validation, and the proxy serves them as normal versions thereafter.

Patching in https://golang.org/cl/181881, it looks like a go.mod with a version like this will break the build in direct mode, since it's now validated.

We should figure out how many modules have invalid requirements like this.

[jayconrod]

cc @bcmills

[bcmills]

Ugh. Agreed, we need to fix this and we need to figure out how big the crater will be.

[zikaeroh]

I think another example of this is github.com/go-redis/redis, where its v6.15.3+incompatible is the first with a go.mod. I actually used tip just the other day to do an upgrade from 6.15.2 that 1.12 was unable to perform, thinking it was a bug that had been fixed in #30634 or similar. I guess I'm guilty of doing something I shouldn't have, but I didn't perform it by hand either.

EDIT: On a second look, setting GOPROXY=https://proxy.golang.org and doing an update the usual way on 1.12 also upgrades me to 6.15.3+incompatible, while direct does not.

[bcmills]

On go1.12.6 I get the following results:

example.com$ go1.12.6 mod download -json github.com/go-redis/[email protected]
go: finding github.com/go-redis/redis v6.15.3
{
        "Path": "github.com/go-redis/redis",
        "Version": "v0.0.0-20190503082931-75795aa4236d",
        "Info": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v0.0.0-20190503082931-75795aa4236d.info",
        "GoMod": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v0.0.0-20190503082931-75795aa4236d.mod",
        "Zip": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v0.0.0-20190503082931-75795aa4236d.zip",
        "Dir": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/github.com/go-redis/[email protected]",
        "Sum": "h1:x7dvutgTPxsHoeUboXlOp9fEUMdIRO4OJV5ly1LYE40=",
        "GoModSum": "h1:W2YCLaZryXHirdd9QqwkiVUxCQsrx8SbLq9Uqk7JS7A="
}

example.com$ go clean -modcache
example.com$ go1.12.6 mod download -json github.com/go-redis/[email protected]+incompatible
go: finding github.com/go-redis/redis v6.15.3+incompatible
{
        "Path": "github.com/go-redis/redis",
        "Version": "v6.15.3+incompatible",
        "Info": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v6.15.3+incompatible.info",
        "GoMod": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v6.15.3+incompatible.mod",
        "Zip": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/cache/download/github.com/go-redis/redis/@v/v6.15.3+incompatible.zip",
        "Dir": "/tmp/tmp.dmSFtGAmfo/_gopath/pkg/mod/github.com/go-redis/[email protected]+incompatible",
        "Sum": "h1:NZ0O90AhLSvSrvLZ/S9h7D4kl1mW2PrKyxL7MyBKO2g=",
        "GoModSum": "h1:W2YCLaZryXHirdd9QqwkiVUxCQsrx8SbLq9Uqk7JS7A="
}

That first result is the intended behavior; the second is this bug.

[gopherbot]

Change https://golang.org/cl/181881 mentions this issue: cmd/go: validate pseudo-versions against module paths and revision metadata

[bcmills]

Two more examples of modules that now fail validation:

• github.com/cenkalti/backoff@latest (see v2.2.0 cannot be used with Go modules cenkalti/backoff#76)
• gotest.tools@latest (go modules compatibility gotestyourself/gotest.tools#156)