bluemix doesn't work behind a corporate proxy on OSX

[campbellr]

At work we have a corporate firewall that intercepts all https traffic, which requires installing a custom CA root certificate.

There is a bug (golang/go#16532) in older versions of Go that causes it to only search the "System Root" keychain for certs (not the "System" keychain, which is where our mitm certs are installed).

As far as I can tell, bluemix must be built with an older version of Go without the fix, because it is ignoring the corporate root certificate, resulting in most bluemix commands failing with a certificate signed by unknown authority error.

Rebuilding against Go 1.9+ should fix the issue (I've confirmed this with a simple example program that uses http.Get, which fails on Go 1.8, but succeeds when built with Go 1.9).

Any chance that a new release could be made?

For reference, I'm running the latest version of bluemix:

$ bluemix --version
bluemix version 0.6.4+41cb1aa9-2017-12-19T04:00:09+00:00

UPDATE: Turns out the real problem was in the container-registry plugin, which (at the time) was built against Go 1.8

[archerzz]

@campbellr 0.6.4 is built against go 1.9.2...

@weieigao Is it related to NTLM?

[campbellr]

@archerzz Hm, interesting.

I also ran bluemix with GODEBUG=x509roots=1 and it certainly seems to be ignoring my certificates.

Here's the output of bx cr images:

$ GODEBUG=x509roots=1 BLUEMIX_TRACE=true bx cr images
Located cached latest version: 0.1.256 @ 2018-01-10T12:02:04-07:00
Last cache check 162919.  Threshold is 86400
calling plugin list endpoint: https://plugins.ng.bluemix.net/bx/list/container-registry
crypto/x509: 4 certs have a trust policy
crypto/x509: ran security verify-cert 0 times
latest version is 0.0.0
Listing images...


REQUEST: [2018-01-12T09:17:23-07:00]
GET /api/v1/images?vulnerabilities=true HTTP/1.1
Host: registry.ng.bluemix.net
Accept: application/json
Accept-Language:
Account: XXXXXXXXXXXXXXXXXXXXXX
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
Organization: XXXXXXXXXXXXXXXXXXXX
User-Agent: Bluemix-CR-CLI/0.1.256 (darwin amd64)
X-Region: us-south


Request failed, err object: Get https://registry.ng.bluemix.net/api/v1/images?vulnerabilities=true: x509: certificate signed by unknown authority

Here's the sample program i used to narrow the problem down to the Go version:

package main

import (
	"io"
	"os"

	bluemix "github.com/IBM-Bluemix/bluemix-go"
	"github.com/IBM-Bluemix/bluemix-go/http"
)

func main() {
	c := &bluemix.Config{} . // ignore the fact that this is missing fields, it's not necessary to trigger the certificate error
	client := http.NewHTTPClient(c)
	resp, err := client.Get("https://registry.ng.bluemix.net/api/v1/images")
	if err != nil {
		panic(err)
	}

	io.Copy(os.Stdout, resp.Body)
}

And running it:

yancampbell@m0164576000:~$ CGO_ENABLED=0 GODEBUG=x509roots=1 /usr/local/opt/[email protected]/bin/go run main.go
crypto/x509: 4 certs have a trust policy
crypto/x509: ran security verify-cert 0 times
panic: Get https://registry.ng.bluemix.net/api/v1/images: x509: certificate signed by unknown authority

goroutine 1 [running]:
main.main()
	/Users/ryancampbell/main.go:16 +0x10c
exit status 2
ryancampbell@m0164576000:~$ CGO_ENABLED=0 GODEBUG=x509roots=1 /usr/local/opt/[email protected]/bin/go run main.go
crypto/x509: 4 certs have a trust policy
crypto/x509: verify-cert approved XXXXX JSS Built-in Certificate Authority
crypto/x509: verify-cert approved XXX-APP-PRD108-CA
crypto/x509: verify-cert approved MS-P-PKI01-CA
crypto/x509: verify-cert approved MS-P-PKI01-CA
crypto/x509: verify-cert approved xxxx-cidc-ise3.xxx.com
crypto/x509: verify-cert approved MS-P-PKI01-CA
crypto/x509: verify-cert approved xxxx-xxx-xxx.xx.com
crypto/x509: ran security verify-cert 7 times
{
    "code": "CRG0003E",
    "message": "Your request did not include an IBM Cloud bearer token. Set the Authorization header of your request to a valid IBM Cloud or IAM bearer token. Run 'bx iam oauth-tokens' to get your token.",
    "request-id": "1549-1515774035.205-4537217"
}

[campbellr]

@archerzz it seems you're right about the Go version though. Digging through the output of strings $(which bluemix) seems to indicate that it is indeed built against Go 1.9.2.

After thinking about it for a bit, I suspect the problem is actually in the container-registry plugin which seems to be built against 1.8.3

Is this still the right place to report the problem?

[boyang9527]

no problem, I have forwarded this issue to the container-registry plugin team

[campbellr]

Thanks! The latest version of the container-registry plugin seems to work.

[campbellr]

Unfortunately, it seems like the container-service plugin has the same issue though.

[stiggity]

Thank you for alerting us to this problem. I've provided Ryan with a pre-release version of the container-service plugin and he's verified that using that plugin he is no longer seeing the problem. I'll update our build to use 1.9.2 and run it through testing.

[stiggity]

If you run

bx plugin update container-service -r stage

This will update you to build 439, which is built on 1.9.2.