terraform AWS EKS 'destroy' failed module.eks.kubernetes_config_map.aws_auth[0] Error: Unauthorized

[alex-beyond-minds]

Description

at terraform destroy execution it's failing on module.eks.kubernetes_config_map.aws_auth[0]
with error:
Error: Unauthorized

Versions

• Terraform: 1.0.9

• Provider(s): kubernetes 2.6.1

• Module: EKS 17.22.0

Reproduction

Steps to reproduce the behavior:
Are you using workspaces?: yes
Have you cleared the local cache (see Notice section above)?: yes
List steps in order that led up to the issue you encountered:
create AWS EKS cluster with custom users, run terraform destroy.

Code Snippet to Reproduce

module "eks" {
version = "17.22.0"
source = "terraform-aws-modules/eks/aws"
cluster_name = var.cluster_name
cluster_version = "1.21"
.
.
.
map_roles = var.map_roles
map_users = var.map_users
}

Expected behavior

Destroy removes EKS cluster

Actual behavior

Destroy is failing on deleting aws_auth configmap

[vasilinamatov]

I have the same issue with same module version and same cluster version.

[daroga0002]

Unuthorized means you have something wrong with your kubernetes provider as terraform is not able to rach EKS api

[alex-beyond-minds]

Correct, but this is out of my control as all what I have done is added users throug map_users param. and seems by the time aws_auth to be removed, cluster is gone.

[alex-beyond-minds]

I think similar issue has been discovered, but not fixed in the latest release #1162

[daroga0002]

maybe this is related to #1658, please try pin kubernetes provider to lower version

[alex-beyond-minds]

Well, creating is working fine for me. And yes, I tried several versions of kubernetes provider, it didn't help. But thank you for advice.

[alex-beyond-minds]

@daroga0002 just in case re-verified with 2.5.1 version with exact same result.

[daroga0002]

dows from host you are able to curl a api endpoint?

[alex-beyond-minds]

this is actually terraform cloud, and yes it can at normal circumstance. At destroy time, cluster seems get removed first, so aws_auth is left behind untill it's too late and cluster is gone. Seems aws_auth is not something we can control from tf files point of view - this is eks module internals.

[alex-beyond-minds]

@daroga0002 and this is easy to reproduce - create aws EKS cluster through terraform(with additional users and or roles) and try to remove it

[daroga0002]

I am doing this constantly and no issue observed. Only thing which I have now in my mind is that you maybe corrupting a aws_auth configmap by wrong syntax or etc.

[alex-beyond-minds]

@daroga0002 have you tried to add users through the terraform at your testing?

[daroga0002]

yup, running this on prod. Also our example examples/managed_node_groups/main.tf is adding some user entries into aws auth and no issue observed during creation or destroy

[alex-beyond-minds]

Hm. In my case it happens every time I destroy env. What versions are you running it at? (also I manage to use kubectl with my AWS account, so assuming configmap is fine)

[daroga0002]

What versions are you running it at?

I didnt had issue with that, newest version is working fine for me (but also multiple earlier)

[daroga0002]

for your information I tested this now again and in general it worked for me.

Steps which I done:

terraform init
terraform apply 
terraform destroy

[alex-beyond-minds]

Great! What versions have you used for your testing?

[daroga0002]

module v17.22.0
terraform 1.0.9

[alex-beyond-minds]

thank you! This doesn't make sense now as I'm running on exact same versions and aws_auth is always failed to remove. Have you use additional users through map_users ? What is included in your destroy plan?

[daroga0002]

aws_auth cm was included in destroy plan and was destroyed.

In general if you require help I ask to paste here full working setup which you using to replicate this problem as without this it will not bring us anywhere

[alex-beyond-minds]

Roles and users maps (shrinked actual list is bigger):

variable "map_roles" {
  description = "Additional IAM roles to add to the aws-auth configmap."
  type = list(object({
    rolearn  = string
    username = string
    groups   = list(string)
  }))

  default = [
    {
      rolearn  = "arn:aws:iam::1122334455:role/admin"
      username = "admin"
      groups   = ["system:masters"]
    },
    {
      rolearn  = "arn:aws:iam::1122334455:role/dev"
      username = "dev_user"
      groups   = []
    },
  ]
}

//
variable "map_users" {
  description = "Additional IAM users to add to the aws-auth configmap."
  type = list(object({
    userarn  = string
    username = string
    groups   = list(string)
  }))

  default = [
    {
      userarn  = "arn:aws:iam::1122334455:user/user1"
      username = "user1"
      groups   = ["system:masters"]
    },
    {
      userarn  = "arn:aws:iam::1122334455:user/eks-user2"
      username = "user2"
      groups   = ["system:masters"]
    },
    
  ]
}

Is anything else on top of it will be helpful?

[daroga0002]

• module configuration which you are using
• terraform plan
• failing terraform destroy log

[alex-beyond-minds]

Here is module to call eks module in the beginning of the thread:

module "eks" {
  source          = "./modules/eks"
  cluster_name    = "${var.BRANCH}-${var.cluster_name_suffix}"
  region          = var.region
  tags            = local.common_tags
  eks_version     = var.eks_version
  vpc_id          = module.vpc.vpc_id
  private_subnets = module.vpc.private_subnets
  public_subnets  = module.vpc.public_subnets
  vpc_cidr_block  = module.vpc.vpc_cidr_block
  map_roles       = var.map_roles
  map_users       = var.map_users
  aws_profile     = var.aws_profile
  relative_path   = var.relative_path
  account_id      = var.account_id
}

[alex-beyond-minds]

plan.log
apply_failed.txt
and plan and failed apply

[alex-beyond-minds]

thank you for looking into it

[alex-beyond-minds]

Hi @daroga0002

is anything in provided info causing on issue with my code?

BR

[daroga0002]

I reviewed logs suspecting this is because token to EKS expiring. From start to executing aws_auth resource there is more than 20 minutes.

Please try add to kubernetes provider something like:
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#exec-plugins

[alex-beyond-minds]

I have in my config at this moment:

provider "kubernetes" {
  host  = data.aws_eks_cluster.cluster.endpoint
  token = module.eks.eks_token
  #var.eks_token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  # version = "2.5.1"
  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    command     = "aws-iam-authenticator"
    args        = ["token", "-i", data.aws_eks_cluster.cluster.name]
    env = {
      name : "AWS_PROFILE"
      value : var.aws_profile
    }
  }
}

[alex-beyond-minds]

This is unlikely related with expiration. In case I remove roles and users parameters, I can install and destroy environment with no issues.

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

This issue was automatically closed because of stale in 10 days

[antonbabenko]

This issue has been resolved in version 18.0.0 🎉

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.