Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data resource for Service Account fails to find default token #1104

Closed
servo1x opened this issue Jan 4, 2021 · 20 comments · Fixed by #1634
Closed

Data resource for Service Account fails to find default token #1104

servo1x opened this issue Jan 4, 2021 · 20 comments · Fixed by #1634

Comments

@servo1x
Copy link

servo1x commented Jan 4, 2021

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v0.13.3
Kubernetes provider version: 1.13.2
Kubernetes version: 1.16.11

Affected Resource(s)

  • data.kubernetes_service_account

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.
data "kubernetes_service_account" "test_sa" {
  metadata {
    name      = "test"
    namespace = "test"
  }
}

Debug Output

2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Response Details:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ RESPONSE ]--------------------------------------
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: HTTP/2.0 200 OK
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Length: 326
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Audit-Id: ...
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Cache-Control: no-cache, private
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Type: application/json
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Date: Mon, 04 Jan 2021 13:34:13 GMT
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "kind": "ServiceAccount",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "apiVersion": "v1",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "metadata": {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "name": "test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "selfLink": "/api/v1/namespaces/test/serviceaccounts/test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "uid": "1ce457f7-276e-4579-a7df-ab489ae1c9cc",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "resourceVersion": "544195689",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "creationTimestamp": "2020-06-05T05:51:48Z"
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "secrets": [
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "name": "test-token-ncwqf"
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   }
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  ]
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: }
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.702-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Request Details:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ REQUEST ]---------------------------------------
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: GET /api/v1/namespaces/test/secrets/test-token-ncwqf HTTP/1.1
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Host: k8s
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: User-Agent: HashiCorp/1.0 Terraform/0.13.3
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Accept: application/json, */*
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Authorization: Bearer ...
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Accept-Encoding: gzip
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Response Details:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ RESPONSE ]--------------------------------------
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: HTTP/2.0 200 OK
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Length: 3113
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Audit-Id: ...
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Cache-Control: no-cache, private
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Type: application/json
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Date: Mon, 04 Jan 2021 13:34:13 GMT
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "kind": "Secret",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "apiVersion": "v1",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "metadata": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "name": "test-token-ncwqf",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "test",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "selfLink": "/api/v1/namespaces/test/secrets/test-token-ncwqf",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "uid": "baef6c8c-e549-4962-9e3c-eb0a9de64e6c",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "resourceVersion": "544195687",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "creationTimestamp": "2020-10-04T00:57:08Z",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "annotations": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "kubernetes.io/service-account.name": "test",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "kubernetes.io/service-account.uid": "1ce457f7-276e-4579-a7df-ab489ae1c9cc"
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   }
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "data": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "ca.crt": "...",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "...",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "token": "..."
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "type": "kubernetes.io/service-account-token"
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: }
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.856-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Skipping test-token-ncwqf as it wasn't created at the same time as the service account
2021/01/04 05:34:13 [ERROR] eval: *terraform.evalReadDataRefresh, err: Failed to discover the default service account token: Unable to find any service accounts tokens which could have been the default one
2021/01/04 05:34:13 [ERROR] eval: *terraform.EvalSequence, err: Failed to discover the default service account token: Unable to find any service accounts tokens which could have been the default one

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. Create a Service Account
  2. Rotate Service Account Default Token
  3. terraform apply apply fails because of differing creation timestamps

Expected Behavior

A valid service token should be found regardless of creation timestamp drift.

Actual Behavior

When a service account default token gets rotated, the new secret has a different timestamp and Terraform is unable to find the default token.

References

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment
@servo1x servo1x added the bug label Jan 4, 2021
@amsgeodis
Copy link

@servo1x Have you found any workaround for this issue?

@amsgeodis
Copy link

amsgeodis commented Feb 11, 2021

The problem is with the 3 seconds check performed in the below line.

if secret.CreationTimestamp.Sub(sa.CreationTimestamp.Time) > (3 * time.Second) {

Can we have it increased to a bigger limit, may be 60 seconds? If we have mutation webhooks, the secret creation can take longer.

amsgeodis added a commit to amsgeodis/terraform-provider-kubernetes that referenced this issue Feb 11, 2021
Increased service_account and secret creation_time difference check to a bigger limit, 60 seconds; instead of 3 seconds. If we have mutation webhooks, the secret creation can take longer.
hashicorp#1104
amsgeodis added a commit to amsgeodis/terraform-provider-kubernetes that referenced this issue Feb 11, 2021
Increased service_account and secret creation_time difference check to a bigger limit, 60 seconds; instead of 3 seconds. If we have mutation webhooks, the secret creation can take longer.
hashicorp#1104
@wojtek-oledzki
Copy link

@wjam, @alexsomesan why do we have that time check in the first place (#377)?

@wjam
Copy link
Contributor

wjam commented Mar 3, 2021

@wojtek-oledzki
Copy link

Why not use the secrets list from the sa object itself?
My token got rotated and is month older then the sa.

@wjam
Copy link
Contributor

wjam commented Mar 3, 2021

Then that secret would not be the default one as defined by the documentation.

Name of the default secret, containing service account token, created & managed by the service
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/service_account#default_secret_name

@wjam
Copy link
Contributor

wjam commented Mar 3, 2021

Note that the PR you linked to was just for adding support to import a service account and nothing more

@wojtek-oledzki
Copy link

But why do we use findDefaultServiceAccount at all when we have getServiceAccountDefaultSecret that returns, what it looks like to be the sa secret (default token)

@wjam
Copy link
Contributor

wjam commented Mar 3, 2021

Because import has to deal with the fact that there may be many secrets associated with the service account and it needs to discover which the 'default' one was - which the create operation has already defined as the secret that was created alongside the service account.

I would argue that the default_secret_name attribute should be removed and the secret list also be computed, but that would be a breaking change and I've not contributed to this provider in years.

@amsgeodis
Copy link

Note that the PR you linked to was just for adding support to import a service account and nothing more

@wjam Doesn't it impact datasource?

@wjam
Copy link
Contributor

wjam commented Mar 3, 2021

No, PR #377 is about adding support for importing service accounts.

@amsgeodis
Copy link

Thanks.

We have found the following snippet failing.
data "kubernetes_service_account" "vault_injector" { metadata { name = var.k8s_vault_sa namespace = var.k8s_vault_namespace } }
.. with the error log Unable to find any service accounts tokens which could have been the default one. And the code/method modified is the only place we have found that message.

return "", fmt.Errorf("Unable to find any service accounts tokens which could have been the default one")

@wjam
Copy link
Contributor

wjam commented Mar 3, 2021

Okay. But remember, the PR I raised didn't touch this - review the PR and you can see that it doesn't touch the data source. Adding the service account data source was PR #731.

@amsgeodis
Copy link

Got it. I had another PR raised 20 days ago to mitigate it. #1165

@avishnyakov
Copy link

Team, will this work be moved?

Basic sa account lookup fails. The whole data source kubernetes_service_account does not seem to work.

data "kubernetes_service_account" "test_sa_deploy" {
  metadata {
    namespace = "test"
    name = "test-account"
  }
}

Error: Failed to discover the default service account token: Unable to find any service accounts tokens which could have been the default one

@wilsncc
Copy link

wilsncc commented Sep 1, 2021

I am also stuck with this. Seems like the motivation for 3 seconds was not entirely random, but I also really need this PR merged so I can get past this issue.

@avishnyakov
Copy link

Thinking about null provider with local-exec to get shell -> kubectl up and running. That should work, ideally.

@erSitzt
Copy link

erSitzt commented Oct 11, 2021

Thx @rossdotpink because of that link i checked my nodes for clock drift... and indeed one of them was off just enough to cause the problem.
Easy fix for me with my current issue... but i still think this should be more than 3 seconds... or at least a parameter, if time is still needed to get the correct token... which still seems a little "hacky" to me 🤷

@Kyslik
Copy link

Kyslik commented Dec 7, 2021

We are hitting similar issue; we create ~200 namespaces and their own service accounts at once; this creates queue and 3 seconds isn't enough.

Usage:

data "kubernetes_service_account" "this" {
  metadata {
    namespace = kubernetes_service_account.this.metadata.0.namespace
    name      = kubernetes_service_account.this.metadata.0.name
  }
}

data "kubernetes_secret" "this" {
  metadata {
    namespace = kubernetes_service_account.this.metadata.0.namespace
    name      = data.kubernetes_service_account.this.default_secret_name
  }
}

With error:

Error: Failed to discover the default service account token: Unable to find any service accounts tokens which could have been the default one

A workaround is to manually delete SA and let TF re-create it again - without the "queue"...

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants