Skip to content

Latest commit

 

History

History
175 lines (156 loc) · 24.3 KB

README.md

File metadata and controls

175 lines (156 loc) · 24.3 KB

Terraform Module for CDP Prerequisites on AWS

This module contains resource files and example variable definition files for creation of the pre-requisite AWS cloud resources required for Cloudera Data Platform (CDP) Public Cloud.

Usage

The examples directory has example AWS Cloud Service Provider deployments for different scenarios:

  • ex01-minimal-inputs uses the minimum set of inputs for the module.

  • ex02-existing-vpc creates a VPC and subnets outside of the module and passes this as an additional input. CDP deployment then uses these network assets.

  • ex03-create-keypair creates the AWS EC2 Keypair in the module caller and passes this as an additional input.

In each directory an example terraform.tfvars.sample values file is included to show input variable values.

Requirements

Name Version
terraform >= 1.3.0
aws >= 5.30
http 3.2.1
random 3.4.3
time 0.9.1

Providers

Name Version
aws >= 5.30
random 3.4.3

Modules

Name Source Version
aws_cdp_cred_permissions ../terraform-aws-cred-permissions n/a
aws_cdp_permissions ../terraform-aws-permissions n/a
aws_cdp_vpc ../terraform-aws-vpc n/a

Resources

Name Type
aws_kms_alias.cdp_kms_alias resource
aws_kms_key.cdp_kms_key resource
aws_s3_bucket.cdp_storage_locations resource
aws_s3_bucket_public_access_block.cdp_storage_locations resource
aws_s3_bucket_server_side_encryption_configuration.cdp_storage_location_kms resource
aws_s3_bucket_versioning.cdp_storage_location_versioning resource
aws_s3_object.cdp_backup_storage_object resource
aws_s3_object.cdp_log_storage_object resource
aws_security_group.cdp_default_sg resource
aws_security_group.cdp_endpoint_sg resource
aws_security_group.cdp_knox_sg resource
aws_security_group_rule.cdp_default_sg_egress resource
aws_security_group_rule.cdp_default_sg_ingress resource
aws_security_group_rule.cdp_default_sg_ingress_self resource
aws_security_group_rule.cdp_endpoint_ingress_self resource
aws_security_group_rule.cdp_endpoint_sg_egress resource
aws_security_group_rule.cdp_endpoint_sg_ingress resource
aws_security_group_rule.cdp_knox_sg_egress resource
aws_security_group_rule.cdp_knox_sg_ingress resource
aws_security_group_rule.cdp_knox_sg_ingress_self resource
aws_vpc_endpoint.gateway_endpoints resource
aws_vpc_endpoint.interface_endpoints resource
random_id.bucket_suffix resource
aws_vpc_endpoint_service.gateway_endpoints data source
aws_vpc_endpoint_service.interface_endpoints data source

Inputs

Name Description Type Default Required
backup_bucket_access_policy_doc Contents of Backup Bucket Access Data Access Policy string n/a yes
data_bucket_access_policy_doc Data Bucket Access Data Access Policy string n/a yes
datalake_admin_s3_policy_doc Contents of Datalake Admin S3 Data Access Policy string n/a yes
datalake_backup_policy_doc Contents of Datalake Backup Data Access Policy string n/a yes
datalake_restore_policy_doc Contents of Datalake Restore Data Access Policy string n/a yes
deployment_template Deployment Pattern to use for Cloud resources and CDP string n/a yes
env_prefix Shorthand name for the environment. Used in resource descriptions string n/a yes
idbroker_policy_doc Contents of IDBroker Assumer Policy Document. string n/a yes
log_bucket_access_policy_doc Contents of Log Bucket Access Data Access Policy string n/a yes
log_data_access_policy_doc Contents of Log Data Access Policy string n/a yes
ranger_audit_s3_policy_doc Contents of Ranger S3 Audit Data Access Policy string n/a yes
xaccount_account_id Account ID of the cross account string n/a yes
xaccount_account_policy_doc Contents of cross acount policy document string n/a yes
xaccount_external_id External ID of the cross account string n/a yes
agent_source_tag Tag to identify deployment source map(any) 
{
  "agent_source": "tf-cdp-module"
}
 no
aws_region Region which Cloud resources will be created string null no
backup_bucket_access_policy_name Backup Bucket Access Data Access Policy Name string null no
backup_storage Optional Backup location for CDP environment. If not provided follow the data_storage variable 
object({
    backup_storage_bucket = string
    backup_storage_object = string
  })
 null no
cdp_default_sg_egress_cidrs List of egress CIDR blocks for CDP Default Security Group Egress rule list(string) 
[
  "0.0.0.0/0"
]
 no
cdp_endpoint_sg_egress_cidrs List of egress CIDR blocks for VPC Endpoint Security Group Egress rule list(string) 
[
  "0.0.0.0/0"
]
 no
cdp_knox_sg_egress_cidrs List of egress CIDR blocks for CDP Knox Security Group Egress rule list(string) 
[
  "0.0.0.0/0"
]
 no
cdp_private_subnet_ids List of private subnet ids. Required if create_vpc is false. list(any) null no
cdp_public_subnet_ids List of public subnet ids. Required if create_vpc is false. list(any) null no
cdp_vpc_id VPC ID for CDP environment. Required if create_vpc is false. string null no
create_vpc Flag to specify if the VPC should be created bool true no
create_vpc_endpoints Flag to specify if VPC Endpoints should be created bool true no
data_bucket_access_policy_name Data Bucket Access Data Access Policy Name string null no
data_storage Data storage locations for CDP environment 
object({
    data_storage_bucket = string
    data_storage_object = string
  })
 null no
datalake_admin_role_name Datalake Admin role Name string null no
datalake_admin_s3_policy_name Datalake Admin S3 Data Access Policy Name string null no
datalake_backup_policy_name Datalake backup Data Access Policy Name string null no
datalake_restore_policy_name Datalake restore Data Access Policy Name string null no
enable_bucket_versioning Flag to enable versioning of S3 buckets. bool true no
enable_kms_bucket_encryption Flag to create AWS KMS for encryption of S3 buckets. Currently disabled as further settings needed for successful CDP deployment. bool false no
env_tags Tags applied to provised resources map(any) null no
existing_xaccount_role_name Name of existing CDP Cross Account Role. If set then no Cross Account policy or role resources are created. string null no
idbroker_policy_name IDBroker Policy name string null no
idbroker_role_name IDBroker service role Name string null no
ingress_extra_cidrs_and_ports List of extra CIDR blocks and ports to include in Security Group Ingress rules 
object({
    cidrs = list(string)
    ports = list(number)
  })
 
{
  "cidrs": [],
  "ports": []
}
 no
log_bucket_access_policy_name Log Bucket Access Data Access Policy Name string null no
log_data_access_policy_name Log Data Access Policy Name string null no
log_role_name Log service role Name string null no
log_storage Optional log locations for CDP environment. If not provided follow the data_storage variable 
object({
    log_storage_bucket = string
    log_storage_object = string
  })
 null no
private_cidr_range Size of each private subnet. Required if create_vpc is true. Number of subnets will be automatically selected to match on the number of Availability Zones in the selected AWS region. (Depending on the selected deployment pattern, one subnet will be created per region.) number 19 no
private_network_extensions Enable creation of resources for connectivity to CDP Control Plane (public subnet and NAT Gateway) for Private Deployment. Only relevant for private deployment template bool true no
public_cidr_range Size of each public subnet. Required if create_vpc is true. Number of subnets will be automatically selected to match on the number of Availability Zones in the selected AWS region. (Depending on the selected deployment pattern, one subnet will be created per region.) number 24 no
random_id_for_bucket Create a random suffix for the bucket names bool true no
ranger_audit_role_name Ranger Audit role Name string null no
ranger_audit_s3_policy_name Ranger S3 Audit Data Access Policy Name string null no
security_group_default_name Default Security Group for CDP environment string null no
security_group_endpoint_name Security Group for VPC Endpoints string null no
security_group_knox_name Knox Security Group for CDP environment string null no
vpc_cidr VPC CIDR Block. Required if create_vpc is true. string "10.10.0.0/16" no
vpc_endpoint_gateway_services List of AWS services used for VPC Gateway Endpoints list(string) 
[
  "s3"
]
 no
vpc_endpoint_interface_services List of AWS services used for VPC Interface Endpoints list(string) 
[
  "sts",
  "rds",
  "elasticloadbalancing",
  "elasticfilesystem",
  "eks",
  "ecr.dkr",
  "ecr.api",
  "ec2",
  "cloudformation",
  "autoscaling"
]
 no
vpc_name Name of the VPC. Defaults to <env_prefix>-net if not specified string null no
vpc_private_inbound_acl_rules Private subnets inbound network ACLs. Exposes default value of VPC module variable to allow for overriding. Only used when create_vpc is true. list(map(string)) 
[
  {
    "cidr_block": "0.0.0.0/0",
    "from_port": 0,
    "protocol": "-1",
    "rule_action": "allow",
    "rule_number": 100,
    "to_port": 0
  }
]
 no
vpc_private_outbound_acl_rules Private subnets outbound network ACLs. Exposes default value of VPC module variable to allow for overriding. Only used when create_vpc is true. list(map(string)) 
[
  {
    "cidr_block": "0.0.0.0/0",
    "from_port": 0,
    "protocol": "-1",
    "rule_action": "allow",
    "rule_number": 100,
    "to_port": 0
  }
]
 no
vpc_public_inbound_acl_rules Inbound network ACLs for Public subnets. Exposes default value of VPC module variable to allow for overriding. Only used when create_vpc is true. list(map(string)) 
[
  {
    "cidr_block": "0.0.0.0/0",
    "from_port": 0,
    "protocol": "-1",
    "rule_action": "allow",
    "rule_number": 100,
    "to_port": 0
  }
]
 no
vpc_public_outbound_acl_rules Public subnets outbound network ACLs. Exposes default value of VPC module variable to allow for overriding. Only used when create_vpc is true. list(map(string)) 
[
  {
    "cidr_block": "0.0.0.0/0",
    "from_port": 0,
    "protocol": "-1",
    "rule_action": "allow",
    "rule_number": 100,
    "to_port": 0
  }
]
 no
vpc_public_subnets_map_public_ip_on_launch Auto-assign public IP on launch for instances created in Public Subnets. Exposes default value of VPC module variable to allow for overriding. Only used when create_vpc is true. bool true no
xaccount_policy_name Cross Account Policy name string null no
xaccount_role_name Cross account Assume role Name string null no

Outputs

Name Description
aws_backup_storage_bucket AWS backup storage bucket
aws_backup_storage_location AWS backup storage location
aws_backup_storage_object AWS backup storage object
aws_data_storage_bucket AWS data storage bucket
aws_data_storage_location AWS data storage location
aws_data_storage_object AWS data storage object
aws_datalake_admin_role_arn Datalake Admin role ARN
aws_datalake_admin_role_name Datalake Admin role Name
aws_default_route_table_id AWS default route table ID
aws_idbroker_instance_profile_arn IDBroker instance profile ARN
aws_idbroker_role_name IDBroker role Name
aws_log_instance_profile_arn Log instance profile ARN
aws_log_role_name Log role Name
aws_log_storage_bucket AWS log storage bucket
aws_log_storage_location AWS log storage location
aws_log_storage_object AWS log storage object
aws_private_route_table_ids AWS private route table IDs
aws_private_subnet_ids AWS private subnet IDs
aws_public_route_table_ids AWS public route table IDs
aws_public_subnet_ids AWS public subnet IDs
aws_ranger_audit_role_arn Ranger Audit role ARN
aws_ranger_audit_role_name Ranger Audit role Name
aws_region Cloud provider region of the Environment
aws_security_group_default_id AWS security group id for default CDP SG
aws_security_group_knox_id AWS security group id for Knox CDP SG
aws_vpc_id AWS VPC ID
aws_vpc_subnets List of subnets associated with the CDP VPC
aws_xaccount_role_arn Cross Account role ARN
aws_xaccount_role_name Cross Account role name
tags Tags associated with the environment and its resources