cmd/go: "go install" ignores vendor folder

[as]

The go install command ignores my vendor folder. This behavior differs from go get.

Below is a command line utility with a vendor folder.

go install github.com/as/[email protected]

When run, the output suggests it is downloading existing content redundantly.

The go.mod, go.sum, and vendor folder exist and are configured. This behavior is surprising and prevents easy packaging of versioned command line tools using go modules, especially in a locked down environment where packages are hosted on an air gapped or heavily firewalled network.

; docker build - <<EOF
FROM golang:1.17-alpine
RUN go install github.com/as/[email protected]
ENTRYPOINT ["/bin/sh"]
EOF

What did you expect to see?

Step 4/5 : RUN go install github.com/as/[email protected]
go: downloading github.com/as/ccp v0.0.1

What did you see instead?

Step 4/5 : RUN go install github.com/as/[email protected]
go: downloading github.com/as/ccp v0.0.1
go: downloading google.golang.org/genproto v0.0.0-20210624174822-c5cf32407d0a
go: downloading google.golang.org/api v0.49.0
...
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading golang.org/x/oauth2 v0.0.0-20210615190721-d04028783cf1

/cc @bcmills

[theckman]

That feels like this is an oversight, with the official Modules proxy being unable to guarantee that all packages it cached will remain cached forever. Considering that, vendoring is the only mechanism we have for 100% reproducible builds, with it being especially important for tools that choose to be distributed this way and not precompiled and distributed via the OS's package manager.

[jayconrod]

This is working as intended, though it looks like it needs to be documented better. Neither https://golang.org/ref/mod#go-install nor go help install mention that vendor directories are not used. They should.

go install cmd@version is meant to be a simple replacement for go get that works in most situations. In particular, when go install is used with a version suffix, no module is considered the main module, which means replace and exclude directives are not applied (actually it's an error if they're present in the module providing the packages named on the command line), and the vendor directory is not used. Additionally, the vendor directory can't be used because it's not present in module zip files. #40276 is the proposal where this was discussed. See the rationale and FAQs sections there.

I understand that doesn't really solve your problem though. I'd suggest one of the following solutions:

• Use go install cmd instead of go install cmd@version. The form without the @version suffix still works and will still use the vendor directory from the main module. You'll need to checkout the module from a VCS though, and if it's not a module you develop, you'll need to mirror the VCS somewhere locally.
• Run a local proxy behind your firewall. This could be something as simple as downloading dependencies ahead of time into a module cache, then using the module cache as a proxy with a file:// URL. Or something like Athens.

[gopherbot]

Change https://golang.org/cl/349590 mentions this issue: _content/ref/mod: document 'go install cmd@version' ignores vendor

[gopherbot]

Change https://golang.org/cl/349591 mentions this issue: cmd/go: document 'go install cmd@version' ignores vendor directories

[theckman]

@jayconrod can you point to the document where that was discussed? That's the distribution command for many applications, that cannot be what we intend. :(

[mvdan]

@theckman see "Why can't vendor directories be used?" and the other mentions of module zips in #40276

[jayconrod]

@theckman As @mvdan pointed out, this is mentioned in #40276. We definitely thought about it. Please read through the design doc and the discussion there.

Supporting vendor directories would require a completely different implementation than what any other module-aware subcommand does, including go get, which go install cmd@version was meant to replace. We decided not to go in that direction.

[vsapronov]

@as I needed the same to work in my project. Saw this issue created by you.
My situation is even more complicated since I have replace directives in my go.mod of the module that I want to install.
I just went to the option of releasing my tool that I want to be go-installable on the side - in the different repo.
And I have my tool that is just "localizing" dependencies code, including vendored dependencies. So the output of my tool is just a self-contained pile of code ready to be placed into "release" repository. The tool is here: https://github.com/specgen-io/goven/blob/main/README.md