error waiting for KMS Key - timeout while waiting for state to become 'TRUE' #8020

Klyde-Moradeyo · 2021-12-10T15:27:13Z

Receiving the following error after attempting to follow the EKS Terraform quick start template

│ Error: error waiting for KMS Key (eea6239e-40cc-408a-a412-fa1ece308c15) policy propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 5m0s)
│
│   with module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0],
│   on .terraform\modules\eks-jx\modules\vault\main.tf line 98, in resource "aws_kms_key" "kms_vault_unseal":
│   98: resource "aws_kms_key" "kms_vault_unseal" {

I've seen the resolution to this in this thread here, which says I should downgrade the aws provider to 3.52. Although upon attempting to do that I get the following error regarding constraints.

│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ hashicorp/aws: locked provider registry.terraform.io/hashicorp/aws 3.52.0
│ does not match configured version constraint 1.52.0, >= 2.23.0, >= 2.53.0,
│ >= 2.70.0, >= 3.56.0, < 4.0.0; must use terraform init -upgrade to allow
│ selection of new versions

Terraform Version
v1.1.0

Registry Versions

+ provider registry.terraform.io/hashicorp/aws v3.69.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.4.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.7.1
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

My Variables.tf File contains:

// ----------------------------------------------------------------------------
// Optional Variables
// ----------------------------------------------------------------------------
variable "region" {
  description = "AWS region code for creating resources."
  type        = string
  default     = "eu-west-2"
}

variable "profile" {
  description = "Profile stored in aws config or credentials file"
  type        = string
  default     = "default"
}

variable "cluster_version" {
  description = "Kubernetes version to use for the EKS cluster."
  type        = string
  default     =  "1.21"
}

variable "vault_user" {
  description = "The AWS IAM Username whose credentials will be used to authenticate the Vault pods against AWS"
  type        = string
  default     = "Administrator"
}

variable "cluster_name" {
  description = "Name of the Kubernetes cluster to create"
  type        = string
  default     = "jx-infrastructure"
}

variable "force_destroy" {
  description = "Flag to determine whether storage buckets get forcefully destroyed. If set to false, empty the bucket first in the aws s3 console, else terraform destroy will fail with BucketNotEmpty error"
  type        = bool
  default     = false
}

variable "is_jx2" {
  default     = false
  type        = bool
  description = "Flag to specify if jx2 related resources need to be created"
}

variable "jx_git_url" {
  description = "URL for the Jenins X cluster git repository"
  type        = string
  default     = "" //  I've set to my Jenins X cluster git repo
}

variable "jx_bot_username" {
  description = "Bot username used to interact with the Jenkins X cluster git repository"
  type        = string
  default     = ""//  I've set the default to bot username
}

variable "jx_bot_token" {
  description = "Bot token used to interact with the Jenkins X cluster git repository"
  type        = string
  default     = ""  // I've set the default to my Bot Token
}

variable "nginx_chart_version" {
  type        = string
  description = "nginx chart version"
  default     = "3.12.0"
}

variable "install_kuberhealthy" {
  description = "Flag to specify if kuberhealthy operator should be installed"
  type        = bool
  default     = true
}

The text was updated successfully, but these errors were encountered:

ankitm123 · 2021-12-11T18:41:54Z

The reason u cannot downgrade to 3.52.0 is that the underlying eks module requires >= 3.56.0.
It seems it can be fixed in higher version of the aws provider: hashicorp/terraform-provider-aws#21225

ankitm123 · 2021-12-11T21:53:41Z

I just tested with the latest module version, and dont see this error.
For vault_user, I have:

Case 1

variable "vault_user" {
  type    = string
  default = ""
}

so the module creates an user jenkins-x-vault,

Case 2 (your case)

variable "vault_user" {
  type    = string
  default = "jx-user"
}

In this case, the module does not create the iam user, and uses an existing user supplied by the end user.

This is the output from terraform version (I do use v1.1.0 for tf):

Terraform v1.1.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.69.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.4.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.7.1
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

So one thing to check is the iam policies associated with Administrator iam user?
Also, the user that is doing the terraform apply, does it have kms:PutKeyPolicy?

And if there are there some guard rails in ur AWS account preventing the policy to be attached to the KMS key?

Klyde-Moradeyo · 2021-12-15T18:08:08Z

Thanks @ankitm123, your suggestion resolved the issue.

Currently cannot enter the Deverloper namespace which is the next step in the quick start.

klyde@machine (main): $ jx ns jx
error: namespaces "jx" not found

I have the jx CLI installed

$ jx
Jenkins X 3.x command line

General:
  dashboard View the Jenkins X Pipelines Dashboard
  namespace View or change the current namespace context in the current Kubernetes cluster
  ui Views the Jenkins X UI (octant)
  upgrade Upgrades resources
  version Displays the version of this command
  add Adds one or more resources
  get Display one or more resources
  create Create one or more resources
  start Starts a resource
  stop Stops a resource
  import alias for: jx import

Other Commands:
  help Help about any command

Usage:
  jx [flags] [options]
Use "jx <command> --help" for more information about a given command.

ankitm123 · 2021-12-18T05:45:52Z

namespaces "jx" not found

Most likely the boot job did not run. Did you install the jx-git-operator chart? what does helm list -a -A show?
check the resources in the jx-git-operator namespace ...

Klyde-Moradeyo · 2021-12-23T15:51:06Z

helm list -a -A returns:

'helm' is not recognized as an internal or external command,
operable program or batch file.

I'm able to access the jx-git-operator namespace

ankitm123 · 2021-12-24T02:41:36Z

That means helm is not installed. This is something that u can install for diagnostic purposes.
It has nothing to do with Jenkins X (https://helm.sh/docs/intro/install/)

I'm able to access the jx-git-operator namespace

What do u see in that namespace? What about output from jx admin log?
Did ur terraform apply get completed successfully?

Klyde-Moradeyo · 2021-12-28T16:06:23Z

Ah I see, I miss read. I thought jx quickstart would install helm for me aswell lol.
Anywas I've installed helm and terraform apply returns the following:

╷
│ Warning: Helm release "jx-git-operator" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│
│   with module.eks-jx.module.cluster.helm_release.jx-git-operator[0],
│   on .terraform\modules\eks-jx\modules\cluster\charts.tf line 1, in resource "helm_release" "jx-git-operator":
│    1: resource "helm_release" "jx-git-operator" {
│
╵
╷
│ Error: timed out waiting for the condition
│
│   with module.eks-jx.module.cluster.helm_release.jx-git-operator[0],
│   on .terraform\modules\eks-jx\modules\cluster\charts.tf line 1, in resource "helm_release" "jx-git-operator":
│    1: resource "helm_release" "jx-git-operator" {
│
╵

helm list -a -A returns:

NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
cluster-checks          kuberhealthy    1               2021-12-24 14:37:25.4136212 +0000 UTC   deployed        jx-kh-check-0.0.78      0.0.78
health-checks-install   jx-git-operator 1               2021-12-24 14:37:25.3313531 +0000 UTC   deployed        jx-kh-check-0.0.78      0.0.78
jx-git-operator         jx-git-operator 1               2021-12-24 14:36:19.2718193 +0000 UTC   failed          jx-git-operator-0.0.194 0.0.88
kuberhealthy            kuberhealthy    1               2021-12-24 14:35:35.7836248 +0000 UTC   deployed        kuberhealthy-64         v2.4.1
vault-instance          jx-vault        1               2021-12-24 14:37:26.4632703 +0000 UTC   deployed        vault-instance-1.0.24   1.0.24
vault-operator          jx-vault        1               2021-12-24 14:35:41.3081319 +0000 UTC   deployed        vault-operator-1.14.3   1.14.3

jx ns jx returns:
error: namespaces "jx" not found

I can still acecss the jx-git-operator namespace.

jx admin log returns:

waiting for the Git Operator to be ready in namespace jx-git-operator...
pod jx-git-operator-7bc44fc4c-kvdc5 has status Running

WARNING: the git operator pod has failed but will restart
to view the log of the failed git operator pod run: kubectl logs -n jx-git-operator jx-git-operator-7bc44fc4c-kvdc5

Looking into the pods log files via kubectl logs -n jx-git-operator jx-git-operator-7bc44fc4c-kvdc5:

no custom git initialisation scripts
using poll duration 20s
ERROR: failed to list repositories: failed to create repo.Repository: failed to create git URL from url [email protected]:SurreyFYP/jx-Cluster.git username: [my git user] password [my password] failed to parse git URL [email protected]:SurreyFYP/jx-Cluster.git: parse "[email protected]:SurreyFYP/jx-Cluster.git": first path segment in URL cannot contain colon

The issue was that I had used a wrong git ssh rather than the url

Thanks for your help!

ankitm123 · 2021-12-28T16:20:44Z

I thought jx quickstart would install helm for me aswell lol.

That would be too much magic 🪄 😁

The issue was that I had used a wrong git ssh rather than the url

yes we dont support ssh urls, only https url, hence we ask users to supply access tokens, I am going to work on adding ssh support soon.

Klyde-Moradeyo · 2021-12-28T21:25:35Z

yes we dont support ssh urls, only https url, hence we ask users to supply access tokens, I am going to work on adding ssh support soon.

noice

Ran into another issue regarding webhook lighthouse-hmac-token after running jx import project.
I'm going to create another thread going into details

Klyde-Moradeyo · 2021-12-28T22:40:30Z

new issue here: #8031

ankitm123 self-assigned this Dec 11, 2021

Klyde-Moradeyo closed this as completed Dec 28, 2021

Klyde-Moradeyo mentioned this issue Dec 28, 2021

could not find lighthouse hmac token lighthouse-hmac-token in namespace jx #8031

Closed

Klyde-Moradeyo mentioned this issue Jan 6, 2022

error waiting for KMS Key - timeout while waiting for state to become 'TRUE' #8038

Closed

homer6 mentioned this issue Apr 12, 2022

Fresh install fails due to KMS key creation failure. jx3-gitops-repositories/jx3-terraform-eks#33

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error waiting for KMS Key - timeout while waiting for state to become 'TRUE' #8020

error waiting for KMS Key - timeout while waiting for state to become 'TRUE' #8020

Klyde-Moradeyo commented Dec 10, 2021

ankitm123 commented Dec 11, 2021

ankitm123 commented Dec 11, 2021 •

edited

Loading

Klyde-Moradeyo commented Dec 15, 2021

ankitm123 commented Dec 18, 2021 •

edited

Loading

Klyde-Moradeyo commented Dec 23, 2021 •

edited

Loading

ankitm123 commented Dec 24, 2021 •

edited

Loading

Klyde-Moradeyo commented Dec 28, 2021

ankitm123 commented Dec 28, 2021

Klyde-Moradeyo commented Dec 28, 2021

Klyde-Moradeyo commented Dec 28, 2021

error waiting for KMS Key - timeout while waiting for state to become 'TRUE' #8020

error waiting for KMS Key - timeout while waiting for state to become 'TRUE' #8020

Comments

Klyde-Moradeyo commented Dec 10, 2021

ankitm123 commented Dec 11, 2021

ankitm123 commented Dec 11, 2021 • edited Loading

Klyde-Moradeyo commented Dec 15, 2021

ankitm123 commented Dec 18, 2021 • edited Loading

Klyde-Moradeyo commented Dec 23, 2021 • edited Loading

ankitm123 commented Dec 24, 2021 • edited Loading

Klyde-Moradeyo commented Dec 28, 2021

ankitm123 commented Dec 28, 2021

Klyde-Moradeyo commented Dec 28, 2021

Klyde-Moradeyo commented Dec 28, 2021

ankitm123 commented Dec 11, 2021 •

edited

Loading

ankitm123 commented Dec 18, 2021 •

edited

Loading

Klyde-Moradeyo commented Dec 23, 2021 •

edited

Loading

ankitm123 commented Dec 24, 2021 •

edited

Loading