Updates for use of CDP Terraform Provider

README.md

@@ -54,34 +54,17 @@ If you no longer need the infrastructure that’s provisioned by the Terraform m
 terraform destroy
 ```
 
-## External dependencies
+## Dependencies
 
-The module includes the option to discover the cross account Ids and to run the CDP deployment using external tools.
+To set up CDP via deployment automation using this guide, the following dependencies must be installed in your local environment:
 
-To utilize these options extra requirements are needed - Python 3, Ansible 2.12, the CDP CLI, the [jq utility](https://stedolan.github.io/jq/download/) and a number of support Python libraries and Ansible collections.
+* Terraform can be installed by following the instructions at https://developer.hashicorp.com/terraform/downloads
 
-A summary of the install and configuration steps for these additional requirements is given below.
-We recommend these steps be performed within an Python virtual environment.
+Configure Terraform Provider for AWS or Azure
 
-```bash
-# Install jq as per instructions at https://stedolan.github.io/jq/download/
-# Example for MacOS using homebew shown below
-brew install jq
-
-# Install the Ansible core Python package
-pip install ansible-core==2.12.10 jmespath==1.0.1
-
-# Install cdpy, a Pythonic wrapper for Cloudera CDP CLI. This in turn installs the CDP CLI.
-pip install git+https://github.com/cloudera-labs/cdpy@main#egg=cdpy
-
-# Install the cloudera.cloud Ansible Collection
-ansible-galaxy collection install git+https://github.com/cloudera-labs/cloudera.cloud.git,devel
-
-# Install the community.general Ansible Collection
-ansible-galaxy collection install community.general:==5.5.0
-
-# Configure cdp with CDP API access key ID and private key
-cdp configure
-```
+* Configure the Terraform Provider for CDP with access key ID and private key by dowloading or creating a CDP configuation file.
+  * See the [CDP documentation for steps to Generate the API access key](https://docs.cloudera.com/cdp-public-cloud/cloud/cli/topics/mc-cli-generating-an-api-access-key.html).
 
-NOTE - See the [CDP documentation for steps to Generate the API access key](https://docs.cloudera.com/cdp-public-cloud/cloud/cli/topics/mc-cli-generating-an-api-access-key.html) required in the `cdp configure` command above.
+* To create resources in the Cloud Provider, access credentials or service account are needed for authentication.
+  * For **AWS** access keys are required to be able to create the Cloud resources via the Terraform aws provider. See the [AWS Terraform Provider Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration).
+  * For **Azure**, authentication with the Azure subscription is required. There are a number of ways to do this outlined in the [Azure Terraform Provider Documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure).

modules/terraform-cdp-aws-pre-reqs/data.tf

@@ -70,16 +70,3 @@ data "http" "datalake_backup_policy_doc" {
 data "http" "datalake_restore_policy_doc" {
   url = "https://raw.githubusercontent.com/hortonworks/cloudbreak/master/cloud-aws-cloudformation/src/main/resources/definitions/aws-datalake-restore-policy.json"
 }
-
-# Use the cdp cli to determin the 
-data "external" "cdpcli" {
-
-  count = var.lookup_cdp_account_ids == true ? 1 : 0
-
-  program = ["bash", "${path.module}/run_cdp_get_cred_prereqs.sh"]
-  query = {
-    infra_type  = var.infra_type
-    cdp_profile = var.cdp_profile
-    cdp_region  = var.cdp_control_plane_region
-  }
-}

modules/terraform-cdp-aws-pre-reqs/defaults.tf

@@ -111,7 +111,7 @@ locals {
     replace(
       replace(
       data.http.ranger_audit_s3_policy_doc.response_body, "$${ARN_PARTITION}", "aws"),
-    "$${STORAGE_LOCATION_BASE}", "${local.data_storage.data_storage_bucket}${local.storage_suffix}"),
+    "$${STORAGE_LOCATION_BASE}", "${local.data_storage.data_storage_bucket}${local.storage_suffix}/${replace(local.data_storage.data_storage_object, "/", "")}"),
   "$${DATALAKE_BUCKET}", "${local.data_storage.data_storage_bucket}${local.storage_suffix}")
 
   # ...then assign either input or downloaded policy doc to var used in resource
@@ -125,7 +125,7 @@ locals {
   datalake_admin_s3_policy_doc_processed = replace(
     replace(
     data.http.datalake_admin_s3_policy_doc.response_body, "$${ARN_PARTITION}", "aws"),
-  "$${STORAGE_LOCATION_BASE}", "${local.data_storage.data_storage_bucket}${local.storage_suffix}")
+  "$${STORAGE_LOCATION_BASE}", "${local.data_storage.data_storage_bucket}${local.storage_suffix}/${replace(local.data_storage.data_storage_object, "/", "")}")
 
   # ...then assign either input or downloaded policy doc to var used in resource
   datalake_admin_s3_policy_doc = coalesce(var.datalake_admin_s3_policy_doc, local.datalake_admin_s3_policy_doc_processed)
@@ -172,10 +172,6 @@ locals {
   # ------- Roles -------
   xaccount_role_name = coalesce(var.xaccount_role_name, "${var.env_prefix}-xaccount-role")
 
-  xaccount_account_id = coalesce(var.xaccount_account_id, var.lookup_cdp_account_ids ? data.external.cdpcli[0].result.account_id : null)
-
-  xaccount_external_id = coalesce(var.xaccount_external_id, var.lookup_cdp_account_ids ? data.external.cdpcli[0].result.external_id : null)
-
   idbroker_role_name = coalesce(var.idbroker_role_name, "${var.env_prefix}-idbroker-role")
 
   log_role_name = coalesce(var.log_role_name, "${var.env_prefix}-logs-role")

modules/terraform-cdp-aws-pre-reqs/examples/ex01-minimal_inputs/main.tf

@@ -27,4 +27,20 @@ module "ex01_minimal_inputs" {
 
   ingress_extra_cidrs_and_ports = var.ingress_extra_cidrs_and_ports
 
+  # Using CDP TF Provider cred pre-reqs data source for values of xaccount account_id and external_id
+  xaccount_account_id  = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.account_id
+  xaccount_external_id = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.external_id
+
+}
+
+# Use the CDP Terraform Provider to find the xaccount account and external ids
+terraform {
+  required_providers {
+    cdp = {
+      source  = "cloudera/cdp"
+      version = "0.1.3-pre"
+    }
+  }
 }
+
+data "cdp_environments_aws_credential_prerequisites" "cdp_prereqs" {}

modules/terraform-cdp-aws-pre-reqs/examples/ex02-existing-vpc/main.tf

@@ -27,6 +27,10 @@ module "ex02_existing_vpc" {
 
   ingress_extra_cidrs_and_ports = var.ingress_extra_cidrs_and_ports
 
+  # Using CDP TF Provider cred pre-reqs data source for values of xaccount account_id and external_id
+  xaccount_account_id  = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.account_id
+  xaccount_external_id = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.external_id
+
   create_vpc             = var.create_vpc
   cdp_vpc_id             = aws_vpc.cdp_vpc.id
   cdp_public_subnet_ids  = values(aws_subnet.cdp_public_subnets)[*].id
@@ -41,3 +45,15 @@ module "ex02_existing_vpc" {
   ]
 
 }
+
+# Use the CDP Terraform Provider to find the xaccount account and external ids
+terraform {
+  required_providers {
+    cdp = {
+      source  = "cloudera/cdp"
+      version = "0.1.3-pre"
+    }
+  }
+}
+
+data "cdp_environments_aws_credential_prerequisites" "cdp_prereqs" {}

modules/terraform-cdp-aws-pre-reqs/examples/ex03-create-keypair/main.tf

@@ -47,4 +47,20 @@ module "ex01_create_keypair" {
 
   ingress_extra_cidrs_and_ports = var.ingress_extra_cidrs_and_ports
 
+  # Using CDP TF Provider cred pre-reqs data source for values of xaccount account_id and external_id
+  xaccount_account_id  = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.account_id
+  xaccount_external_id = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.external_id
+
+}
+
+# Use the CDP Terraform Provider to find the xaccount account and external ids
+terraform {
+  required_providers {
+    cdp = {
+      source  = "cloudera/cdp"
+      version = "0.1.3-pre"
+    }
+  }
 }
+
+data "cdp_environments_aws_credential_prerequisites" "cdp_prereqs" {}

modules/terraform-cdp-aws-pre-reqs/main.tf

@@ -282,14 +282,14 @@ data "aws_iam_policy_document" "cdp_xaccount_role_policy_doc" {
 
     principals {
       type        = "AWS"
-      identifiers = ["arn:aws:iam::${local.xaccount_account_id}:root"]
+      identifiers = ["arn:aws:iam::${var.xaccount_account_id}:root"]
     }
 
     condition {
       test     = "StringEquals"
       variable = "sts:ExternalId"
 
-      values = [local.xaccount_external_id]
+      values = [var.xaccount_external_id]
     }
   }
 }
@@ -310,6 +310,13 @@ resource "aws_iam_role_policy_attachment" "cdp_xaccount_role_attach" {
   policy_arn = aws_iam_policy.cdp_xaccount_policy.arn
 }
 
+# Wait for propagation of IAM xaccount role.
+# Required for CDP credential
+resource "time_sleep" "iam_propagation" {
+  depends_on      = [aws_iam_role.cdp_xaccount_role]
+  create_duration = "45s"
+}
+
 # ------- AWS Service Roles - CDP IDBroker -------
 # First create the Assume role policy document
 data "aws_iam_policy_document" "cdp_idbroker_role_policy_doc" {

modules/terraform-cdp-aws-pre-reqs/modules/vpc/provider.tf

@@ -16,7 +16,7 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 4.0"
+      version = "4.67.0"
     }
   }
 

modules/terraform-cdp-aws-pre-reqs/outputs.tf

@@ -18,19 +18,6 @@ output "tags" {
   description = "Tags associated with the environment and its resources"
 }
 
-# CDP settings
-output "cdp_profile" {
-  value = var.cdp_profile
-
-  description = "Profile for CDP credentials"
-}
-
-output "cdp_control_plane_region" {
-  value = var.cdp_control_plane_region
-
-  description = "CDP Control Plane region"
-}
-
 # CSP settings
 output "aws_region" {
   value = var.aws_region

modules/terraform-cdp-aws-pre-reqs/provider.tf

@@ -16,20 +16,20 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 4.0"
+      version = "4.67.0"
     }
     http = {
       source  = "hashicorp/http"
       version = "3.2.1"
     }
-    external = {
-      source  = "hashicorp/external"
-      version = "2.3.1"
-    }
     random = {
       source  = "hashicorp/random"
       version = "3.4.3"
     }
+    time = {
+      source  = "hashicorp/time"
+      version = "0.9.1"
+    }
   }
 
   required_version = "> 1.3.0"

modules/terraform-cdp-aws-pre-reqs/variables.tf

@@ -52,21 +52,13 @@ variable "env_prefix" {
 }
 
 # ------- CDP Environment Deployment -------
-variable "cdp_profile" {
-  type        = string
-  description = "Profile for CDP credentials"
-
-  # Profile is default unless explicitly specified
-  default = "default"
-}
-
-variable "cdp_control_plane_region" {
-  type        = string
-  description = "CDP Control Plane Region"
+# variable "cdp_control_plane_region" {
+#   type        = string
+#   description = "CDP Control Plane Region"
 
-  # Region is us-west-1 unless explicitly specified
-  default = "us-west-1"
-}
+#   # Region is us-west-1 unless explicitly specified
+#   default = "us-west-1"
+# }
 
 variable "deployment_template" {
   type = string
@@ -79,14 +71,6 @@ variable "deployment_template" {
   }
 }
 
-variable "lookup_cdp_account_ids" {
-  type = bool
-
-  description = "Auto lookup CDP Account and External ID using CDP CLI commands. If false then the xaccount_account_id and xaccount_external_id input variables need to be specified"
-
-  default = true
-}
-
 # variable "enable_raz" {
 #   type = bool
 
@@ -348,14 +332,12 @@ variable "xaccount_account_id" {
   type        = string
   description = "Account ID of the cross account"
 
-  default = null
 }
 
 variable "xaccount_external_id" {
   type        = string
   description = "External ID of the cross account"
 
-  default = null
 }
 
 # IDBroker service role

modules/terraform-cdp-deploy/README.md

@@ -18,8 +18,7 @@ In each directory an example `terraform.tfvars.sample` values file is included t
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | > 1.3.0 |
-| <a name="requirement_local"></a> [local](#requirement\_local) | 2.2.3 |
-| <a name="requirement_null"></a> [null](#requirement\_null) | 3.2.1 |
+| <a name="requirement_cdp"></a> [cdp](#requirement\_cdp) | 0.1.3-pre |
 
 ## Providers
 
@@ -73,8 +72,6 @@ No resources.
 | <a name="input_azure_xaccount_app_pword"></a> [azure\_xaccount\_app\_pword](#input\_azure\_xaccount\_app\_pword) | Password for the Azure AD Cross Account Application. Required for CDP deployment on Azure. | `string` | `null` | no |
 | <a name="input_azure_xaccount_app_uuid"></a> [azure\_xaccount\_app\_uuid](#input\_azure\_xaccount\_app\_uuid) | UUID for the Azure AD Cross Account Application. Required for CDP deployment on Azure. | `string` | `null` | no |
 | <a name="input_cdp_admin_group_name"></a> [cdp\_admin\_group\_name](#input\_cdp\_admin\_group\_name) | Name of the CDP IAM Admin Group associated with the environment. Defaults to '<env\_prefix>-cdp-admin-group' if not specified. | `string` | `null` | no |
-| <a name="input_cdp_control_plane_region"></a> [cdp\_control\_plane\_region](#input\_cdp\_control\_plane\_region) | CDP Control Plane Region | `string` | `"us-west-1"` | no |
-| <a name="input_cdp_profile"></a> [cdp\_profile](#input\_cdp\_profile) | Profile for CDP credentials | `string` | `"default"` | no |
 | <a name="input_cdp_user_group_name"></a> [cdp\_user\_group\_name](#input\_cdp\_user\_group\_name) | Name of the CDP IAM User Group associated with the environment. Defaults to '<env\_prefix>-cdp-user-group' if not specified. | `string` | `null` | no |
 | <a name="input_cdp_xacccount_credential_name"></a> [cdp\_xacccount\_credential\_name](#input\_cdp\_xacccount\_credential\_name) | Name of the CDP Cross Account Credential. Defaults to '<env\_prefix>-xaccount-cred' if not specified. | `string` | `null` | no |
 | <a name="input_datalake_name"></a> [datalake\_name](#input\_datalake\_name) | Name of the CDP datalake. Defaults to '<env\_prefix>-<aw\|az\|gc\|>-dl' if not specified. | `string` | `null` | no |