EKS authentication requires newer client-go

[ghost]

Getting the following error when trying to use kubeseal with EKS.

$ kubeseal --fetch-cert
panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

I am able to interact with the cluster normally using kubectl.

$ kubectl get pods -n kube-system
NAME                                         READY     STATUS    RESTARTS   AGE
aws-node-68vfm                               1/1       Running   0          6d
aws-node-hl9ww                               1/1       Running   1          6d
aws-node-nhkwb                               1/1       Running   1          6d
fluentd-78l2j                                1/1       Running   0          6d
fluentd-kfdn4                                1/1       Running   0          6d
fluentd-wfj48                                1/1       Running   0          6d
kube-dns-7cc87d595-fbmxq                     3/3       Running   0          6d
kube-proxy-fkpsw                             1/1       Running   0          6d
kube-proxy-lbwvz                             1/1       Running   0          6d
kube-proxy-qqs2p                             1/1       Running   0          6d
sealed-secrets-controller-5bc5488bb4-79blg   1/1       Running   0          22m

[anguslees]

Interesting, thanks for the report.

As background explanation: --fetch-cert requires reading the public-key portion of the sealed-secrets master Secret in kube-system somehow. Accessing the Secret directly works, but is a bit scary security-wise (since the private-key is there too), so instead I serve just the public-key portion via HTTP from the sealed-secrets controller, and then kubeseal fetches it via a proxy through the k8s apiserver. This is why it requires services/proxy in kube-system.

So, your options are:

• Retry with a k8s user with greater access. "system:anonymous" sounds like you are using an unauthenticated user of some sort.
• If you can read the logs from the sealed-secrets-controller pod (kubectl -n kube-system logs sealed-secrets-controller-5bc5488bb4-79blg in your example above), then the public key is written to the log on startup(*). I expect system:anonymous will not grant sufficient access for this either.
• Get to the http://sealed-secrets-controller.kube-system:8080/v1/cert.pem in-cluster URL some other way, perhaps from a pod running in the cluster.

(*) The public key (via any method) is the text between and including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----

It doesn't matter which user you use to get the public key, it's the same public key for all users of that cluster. So typically the workflow would be: admin installs sealed-secret controller, admin runs kubeseal --fetch-cert, admin publishes cert via some other means (intranet webpage, git repo, etc), regular users use cert.

For this bug:

It would be nice if we could make this easier for future users of EKS. I haven't used EKS - how did you get these credentials to the EKS cluster? Is it going to be typical for EKS users to use system:anonymous in this way? If so, we might need to explicitly add a more liberal services/proxy RBAC rule for the controller http service (it is meant to be ~widely accessible).

[ghost]

My kubeconfig is setup using eksctl:

eksctl utils write-kubeconfig my-cluster --profile my-profile

Relevant code is here: https://github.com/weaveworks/eksctl/blob/master/cmd/eksctl/utils.go#L85-L164

EKS requires aws-iam-authenticator to auth with k8s. It was formerly called heptio-authenticator-aws and is vendored in eksutil here: https://github.com/weaveworks/eksctl/tree/master/vendor/github.com/heptio/authenticator/cmd/heptio-authenticator-aws.

[ghost]

I'm able to view logs for the controller pod and get the certificate that way. I'll try updating RBAC tomorrow.

[ghost]

My temporary solution:

kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')

[anguslees]

Huh, I think I originally misunderstood the issue here. This isn't about RBAC, since we should not allow system:anonymous to access the service (there's nothing actually wrong with this since the pubkey is supposed to be public, but system:anonymous represents API requests that have not provided any authentication token and probably should not have any access to the cluster at all). As @jramos last comment shows, if we provide a valid token, then we're no longer system:anonymous and everything works.

So the issue here is that the API request that kubeseal makes does not include the AWS bearer token. The fix appears to require updating to a version of client-go that includes the "exec" auth plugin, which is alpha in 1.10 / beta in k8s 1.11.

Meanwhile, the correct workaround is the explicit --token arg from the previous comment.

[ghost]

👍 thanks for the explanation.

[stromvirvel]

just for information:

This issue obviously also pops up when you're trying to seal a secret:

kubeseal  < secrets.yaml
Please enter Username: Please enter Password: **panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9

I wasted a lot of time to find out what the problem is here, until I found this issue. Maybe you could mention the workaround in the README?

[admssa]

Hi guys, do you have any ETA for updating client-go lib and rebuild?
Or maybe you have some beta binaries?
And one more thing - can I set controller IP manually? It tries to connect directly to pod.

admssa$ kubeseal --fetch-cert --token $(aws-iam-authenticator token -i cluster-name  | jq -r '.status.token')

panic: Error fetching certificate: an error on the server ("Error: 'dial tcp 10.237.72.52:8080: getsockopt: connection timed out'\nTrying to reach: 'http://10.237.72.52:8080/v1/cert.pem'") has prevented the request from succeeding (get services http:sealed-secrets-controller:)

goroutine 1 [running]:

[anguslees]

can I set controller IP manually?

No. As currently written, kubeseal talks to the controller via the apiserver "proxy" to a known service (by namespace+name).

You only need this for fetching the public key though - you can run kubeseal --fetch-cert once, and then pass the certificate file to later invocations of kubeseal --cert. If you do this, then kubeseal doesn't talk to the controller at all during sealing. This is the recommended approach for automated pipelines (since it removes the need to give your robots authenticated k8s API access).

[JessieAMorris]

It looks like this has been resolved with #126. I just tested on an EKS cluster and it is working correctly for me.

[NeoTech]

I found this and tried the "hack" i keep crashing tho for an AWS EKS cluster.

> kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
panic: Error fetching certificate: Unauthorized

goroutine 1 [running]:
main.main()
	/home/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9

[morganchristiansson]

I'm getting same error as @NeoTech

EDIT: Managed workaround. Use local secret: kubeseal --cert sealed-secrets-key.yaml --fetch-cert

EDIT2: Still not working. output is supposed to be a pem file but I'm getting yaml output. But my workaround to use entirely local workflow is valid just --fetch-cert doesn't work correctly

[damascenorakuten]

Same issue here:

panic: Error fetching certificate: Unauthorized

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9

[dhanvi]

kubectl port-forward sealed-secrets-controller-xxxxx 8080:8080 and then
curl -O localhost:8080/v1/cert.pem is doing the trick for me! from then on use the local file to seal secrets

[alejandrox1]

In case anyone else comes across this issue, besides all the tremendously useful answers above, here is a practical summary of what to look for in order to get this going:
So #126 seems to have worked, however, this is NOT included in the v0.7.0 release of kubeseal (see commit history). So to have kubeseal --fetch-cert work, you may need to clone the repo and make kubeseal.

The second catch, is that kubeseal --fetch-cert uses a Kubernetes service to communicate with the sealed-secrets controller and get the cert from /v1/cert.pem (see comment right above this one). If for some reason you get an Error fetching certificate: services "sealed-secret-controller" not found then your service may be named something else (i.e., sealed-secrets).

So in short, you may have to clone the repo, make kubeseal, use that executable instead of the one in the release and make sure you are using the right service name.
In my case, I used the official Helm chart to deploy sealed-secrets and ended up with this:

$ kubectl get svc -n kube-system
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kube-dns         ClusterIP   172.20.0.10     <none>        53/UDP,53/TCP   15h
sealed-secrets   ClusterIP   172.20.241.81   <none>        8080/TCP        14h
tiller-deploy    ClusterIP   172.20.235.94   <none>        44134/TCP       14h

So I had to do

$ ./kubeseal --fetch-cert --controller-name sealed-secrets
-----BEGIN CERTIFICATE-----
...

to get the cert.

[alejandrox1]

As an extra comment, I got so involved in debugging this that I forgot what the original issue is:
Originally, if I wanted to do seal a secret, I got the following

$ kubeseal  <mysecret.json >mysealedsecret.json
panic: Error fetching certificate: services "http:sealed-secrets:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9

Again, by cloning and compiling the kubeseal binary myself (again, this inlcudes the bump to v0.7.0 of client-go) I was the able to seal a secret as normal:

$ ./kubeseal --controller-name sealed-secrets <mysecret.json >mysealedsecret.json

[mbelang]

@alejandrox1 When do we expect an official release?

[jgrabenstein]

Would be pretty cool if there could be a new release cut with #126 included to resolve the issues of fetching the cert w/kubectl proxy. This also affects encrypting secrets (must download cert locally- and --fetch-cert is broken too). Last release was over 1 year ago. Cloning the repo and running make kubeseal does not work due to missing go dependencies.

For reference, this is the error I received:
panic: Error fetching certificate: an error on the server ("Error: 'dial tcp 10.65.1.2:8080: i/o timeout'\nTrying to reach: 'http://10.65.1.2:8080/v1/cert.pem'") has prevented the request from succeeding (get services http:sealed-secrets-controller:)

I'm now writing a workaround into my wrapper to use port-forwarding to fetch the cert, save it locally, use it for encryption, then remove the cert, instead of just being able to run kubeseal <mysecret.json >mysealedsecret.json

This did work as expected when I was using a cluster with Public IP's. Now that I am using a cluster with Private IP's, it is not, and I have to do a couple extra steps.

[alejandrox1]

@mbelang I don't know 😅 I'm not actually a maintainer of this project.

[alejandrox1]

@jgrabenstein do you use kubeseal the same way you use kubectl (or any other Kubernetes client)?
Could you provide more details about how your cluster is setup? Source code if possible to try and replicate it.

[burnjake]

Getting this when running on EKS Kubernetes version 1.12.

kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
panic: Error fetching certificate: Unauthorized

goroutine 1 [running]:
main.main()
	/private/tmp/kubeseal-20181221-52005-1n89slr/sealed-secrets-0.7.0/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3ad

Does anyone have an update?

[JessieAMorris]

If you use the git repo from master it works. They haven't done a release in a long time. Is this project still being maintained?

[jgrabenstein]

@jgrabenstein do you use kubeseal the same way you use kubectl (or any other Kubernetes client)?
Could you provide more details about how your cluster is setup? Source code if possible to try and replicate it.

These are regional, private (no external/public IP's) GKE clusters on a shared VPC.
It appears that running simple commands like kubeseal <mysecret.json >mysealedsecret.json do not work with private clusters (see the error I reported above).

To work around that, I use kubectl port-forward to make the controller's port available locally. Then i can download the cert from the controller. Next you will encrypt with kubeseal, pointing it to the cert you just downloaded: kubeseal --cert /path/to/public-cert $RAW_SECRET > $SEALED_SECRET. You can then apply the sealed secret as you normally would.

Not sure my company would appreciate me sharing the source code for my wrapper.. If people here think it would help them I could possibly see if we could contribute it.

I think ultimately this project needs some decision/leadership around its future - this is something that could be fixed in kubeseal, even if it cannibalizes my wrapper's method.

[mkmik]

going to cut a new release soon

[jwenz723]

Due to the fact that I have Weave CNI installed in my EKS cluster I seem to have broken the ability for kubectl proxy to be able to communicate to pods in my cluster. So if I run kubectl proxy and then browse to http://localhost:8001/api/v1/namespaces/kube-system/services/sealed-secrets-controller/proxy/healthz I get the following response:

Error: 'Address is not allowed'
Trying to reach: 'http://10.32.0.3:8080/healthz'

It seems like kubeseal --fetch-cert is relying on this functionality to work. The error from kubeseal I get is:

$ kubeseal --fetch-cert
panic: Error fetching certificate: the server is currently unable to handle the request (get services http:sealed-secrets-controller:)

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:340 +0x5c6

Currently the work around that I have gotten to work is:

$ kubectl port-forward -n kube-system svc/sealed-secrets-controller 8080
$ curl -LJO http://localhost:8080/v1/cert.pem

This can also be done in a single shell using something like the following. This was useful to me as I am trying to retrieve the public key within a CI pipeline.

# inspired by https://stackoverflow.com/a/54931055
kubectl port-forward -n kube-system svc/sealed-secrets-controller 8080 &
proxy_pid=$!
echo $proxy_pid

until curl -fsSL http://localhost:8080/healthz > /dev/null; do
    echo "waiting for kubectl port-forward" >&2
    sleep 2
done

curl -LJO http://localhost:8080/v1/cert.pem

function cleanup {
    echo "killing kubectl proxy" >&2
    kill $proxy_pid
}
trap cleanup EXIT

Just posting this here in case it is useful to anyone else.

[mkmik]

Would it make sense if the sealedsecret controller posted the public key into a configmap?