cmd/go: mod download doesn't allow insecure download

[lizihuai]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

GO111MODULE=on go mod download

If possible, provide a recipe for reproducing the error.

error:

go: k8s.io/[email protected]+incompatible: unrecognized import path "k8s.io/client-go" (https fetch: Get https://k8s.io/client-go?go-get=1: x509: certificate signed by unknown authority)
go: k8s.io/[email protected]: unrecognized import path "k8s.io/api" (https fetch: Get https://k8s.io/api?go-get=1: x509: certificate signed by unknown authority)

A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

What did you see instead?

[oiooj]

@lizihuai Could you run this command and log it.

echo | openssl s_client -showcerts -servername k8s.io -connect k8s.io:443 2>/dev/null | openssl x509 -inform pem -noout -text

It seems that you have no letsencrypt's CA.

[lizihuai]

@oiooj
I run the command is failure,error message
unable to load certificate
140308543207312:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:707:Expecting: TRUSTED CERTIFICATE

Explain that I have no CA?

Why do I need a CA when I use go mod?

[lizihuai]

@oiooj but I use curl -kv https://k8s.io:443 is success

[fraenkel]

@lizihuai Drop the -k and the failure should occur.

[lizihuai]

@fraenkel Drop the -k is failure . how can I do ? Why can I access other https URLs? I can't access this.

[fraenkel]

Most OSes have added the LetsEncrypt CA as a trusted CA.
But you might be able to get away with a go get -insecure instead. I am not sure if go mod download is missing the -insecure flag.

[agnivade]

That is the exact issue - go mod download doesn't have an insecure flag.

go help mod download
usage: go mod download [-dir] [-json] [modules]

/cc @rsc @bcmills

[lizihuai]

@fraenkel
use go get insecure can download k8s.io.but the other problem can't download
https://gopkg.in/yaml.v2?go-get=1
@agnivade GO 1.11 can fix this problem ?
Why is there such a problem, what is the cause of this problem? thanks

[gcstang]

I have the same issue for a self-generated cert on my private repository when using
go mod tidy
go mod download
etc...

[agnivade]

Why is there such a problem, what is the cause of this problem? thanks

It is how https works. The client has to verify the cert returned by the server. We can explicitly choose to ignore validating the cert. go get has an option to allow that, go mod download doesn't. Hence the issue.

[gopherbot]

Change https://golang.org/cl/135735 mentions this issue: cmd/go/internal/modcmd: mod download allow insecure download

[gcstang]

I see in the recent change the flag has been added will this also allow it to pick up the setting in GO_FLAGS like other commands can?

i.e. GOFLAGS=-insecure

[bcmills]

CC @FiloSottile

[bcmills]

If the problem is that go get is missing some needed root certificate, I would rather we add a mechanism to tell go get about the missing certificate rather than ignore all of the ones it already has.

[bcmills]

(Or, to put it another way: I would prefer that we deprecate the -insecure flag rather than expanding its reach.)

[gcstang]

I'm good either way, as long as I can add a self generated certificate for it to work.

[fearful-symmetry]

Seconding this. My work has a number of internal gitlab instances with self-signed certs. An -insecure flag would be very helpful.

[sgoodrow]

We are dealing with this problem as well. An -insecure flag would be very helpful.

[FiloSottile]

How many people would need the full -insecure shotgun, and why, as opposed to an environment variable to point to a private CA (cough) or self-signed certificate to trust?

[myitcv]

I guess an alternative here would be to have a GOPROXY implementation that handles this.

[daqingshu]

How many people would need the full -insecure shotgun, and why, as opposed to an environment variable to point to a private CA (cough) or self-signed certificate to trust?

example1: in some company, use private Gitlab env, just use http and ssh, no https

[cvigo]

I vote for this.

I filed a related issue this weekend #29059

[daqingshu]

but they would have to deal with the certificates pain

Setting up a private CA is practically a breeze with mkcert or various other tools out there. The entire world is moving to https. There are various web features (like SWs) which are not available on http at all.

What @bcmills and @FiloSottile are trying to understand is why folks are unable to move to https. And so far the answer is that "certificate pain".

I may have minimal experience on this, so @FiloSottile can chime in, but installing a private CA takes no time at all. And for external servers, you have letsencrypt. Although I agree that one has to still ramp up on these tools and in some corporate environments, there are levels of red tape. So it is not always a one command install.

A tough call.

I can't convince IT (a private gitlab cluster with hundreds of servers deployed in different cities, I don't know what the actual risk on the production environment), just as I can't convince you.
glide can help me
go mod can't
Sorry, I shut up.

[cvigo]

When you have hundreds of certificates (=> a few expiring every single day) and thousands of developers, then the breeze becomes a hurricane.

But if these certificates automatically update as required (which is the case with many LetsEncrypt integrations), then the cost is ~ zero. And with a bit of trivial scripting, it would be easy to automate the updating of non-private server certificates (via DNS challenge mode). And that's all before we consider wildcard certificates which might even make much of this discussion moot.

Good in theory. Never seen it in practice. Just as an example, I have 7 different private CA's installed and trusted in my laptop. Not sure who manages each. Security dept. sometimes answer "that one is not mine, try with XXX dept.". Not a definition of breeze

[bitfield]

Seems to me this boils down to "Our Gitlab server has an invalid TLS certificate, so please disable security for every Go user in the world so we don't have to fix our certificate".

[cvigo]

Who has said that?

[samart]

We have mcafee web gateway in the middle which I think is causing this error and making go mod unusable. To get certs added to that infra (if that is a fix) would mean many discussion with clueless security teams that would then want to know how golang works and then likely create more roadblocks for us. Is an insecure option possible with go mod?

[bitfield]

I vote to close this as "works as designed".

[samart]

How about allowing an optional, non default —insecure flag? For those in environments where go mod won’t work. Seems like it would help some and hurt no one.

[gcstang]

I have an alternative solution but I can understand the dilemma some of these developers find themselves in.

Maybe I don't understand why this is such an issue to add a flag to give support to the community that needs it in order to continue down the path that was given. Make this an optional flag or at least supply a fix that will allow those that need it as a way to progress that is realistic.

I've never seen such push back on something that would be so simple to address.
Is this push back just due to not wanting to do something or just to get developers of the Go project riled up? At this point it just seems antagonistic which is just the opposite of what I've seen and experienced in the Go community, I hope this simple request can be resolved in a friendlier manner.

[dawnhammond]

All I wanted for Christmas was the --insecure flag.

[bitfield]

From the discussion on the CL:

it sounds like most of the cases where this would be useful are cases where go get is missing a root certificate for the requested domain.

I'd prefer that we address that case directly rather than adding a bypass flag: we don't really want folks to put -insecure in their GOFLAGS and end up downloading and running untrusted code.

Nothing to do with antagonism or being unfriendly; it's a security issue. Using HTTPS links without actually checking the certificates gives a false sense of security. If you can't set up a verifiable TLS cert chain, you should be using a non-HTTPS URL, to make it clear that this is not a secure connection.

I've never seen such push back on something that would be so simple to address.

Then perhaps there's a good reason.

[samart]

When not on the corporate network it works fine. When on the corporate network, we have mcafee web gateway in the middle which .. i think ... needs some cert installed or adds something to the chain.. What cert I don’t know. That’s my guess.

But this is huge issue for go development in my team and at my company. We are completely stuck whilst this back and forth happens.

@bitfield I don’t understand why an option would be so bad. You don’t have to use it and neither does anyone else for whom it’s working fine. At least we can code. Otherwise we have to change programming languages? That’s extreme.

note, the errors we are getting are on golang.org, etc, downloading crypto packages, etc (not k8s.io as in the original post).

[FiloSottile]

A global flag defeating all security is uncomfortable because users will naturally use it and go on with their day instead of reporting the specific reason they need it, which we might be able to address safely. Remember that modules support is still in experimental state, so now is the time to learn. For example:

• if we are failing to recognize an installed root CA, like in crypto/x509: root_cgo_darwin and root_nocgo_darwin omit some system certs #24652, we should fix that
• if users need to use external root CAs more easily, a proposal to add an environment variable to add them to the pool without installing them might make sense
• if internal insecure modules (http:// or git://) are widespread, we might consider a bypass list in a config file the same way we will have an exclude list for internal modules that should not be fetched from the mirror; this critically only defeats security for some modules, maybe the ones that are fetched over a VPN anyway; or maybe this need will be addressed by a GOPROXY implementation that will have much better flexibility in authentication methods than we could ever provide in the go tool.

If any of the above seem to match your use case (or not), please open a specific issue with as many details as possible of what you are trying to do and in what environment (not just the solution you want) and tag me and @bcmills. I am closing this issue to encourage per-scenario discussion, not because there are no valid use cases here.

(If instead you do indeed need to defeat all security because of some security appliance, sounds like you should bring that up as an issue with your vendor as soon as possible.)

[berniedurfee-ge]

Disappointed to see this request being closed. Many large enterprises don't manage certificates ideally for a variety of reasons. Very often developers in those environments don't have enough authority or influence to change how those enterprises manage certificates.

I understand the reluctance to introduce a flag allowing users to disable part of TLS is not ideal, but reality is often not ideal. This is precisely why most tools include a -insecure flag of some sort to allow those in less than ideal environments to use the tool. Not including this flag in an effort to make the world safer is a noble ideal, but ultimately just makes life more difficult for those developers who don't have the ability to change their environments.

Hopefully this decision is reconsidered and the -insecure flag is added to go mod vendor. Tools like this should be focused on delivering functionality in any environment. Again, it's noble to take advantage of the leverage you have as tool maintainers to make the world a better place. However, it's also okay to allow institutions and users to decide for themselves how they want to manage risk.

[samart]

@berniedurfee-ge We should write to the head of the golang project at google. Low level engineers dont see the big picture.

[chinglinwen]

I don't know go mod download before, I just using go get most of the time

Is go mod download serve for different purpose than go get

If that so, maybe there's different constraint or assumption?

[chiefy]

We have our own CA and must go through a corporate proxy. Unless I want to install our CA on every docker image I pull, I usually use an -insecure flag. The fact that go mod download doesn't support this is not cool. Now in order to run go mod download I have to disconnect from our VPN, run the command and then get back onto our VPN which is a waste of at least 2-3 minutes each time.

[andesli]

need -insecure

[bcmills]

@andesli, see http://golang.org/wiki/NoPlusOne.

(Also note that this issue is closed, and we have no process in place to follow up on closed issues.)

[FiloSottile]

This issue is closed, but as I said we do want to address the use case, even if not with a blunt -insecure flag. The third case in my comment is being expanded on in #32966. If that solution doesn't work for anyone, please do open a new issue.

[Haiyung]

I vote for this.

I filed a related issue this weekend #29059

vote +1

[ianlancetaylor]

@Haiyung This issue is closed. Also, please see https://golang.org/wiki/NoPlusOne. Thanks.