Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Adding missing extra_fqdn support for operator that was supported previously #197

Merged
merged 7 commits into from
Apr 17, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,8 @@ Upgrades must be executed in step-wise fashion from one version to the next. You
| <a name="input_acm_certificate_arn"></a> [acm\_certificate\_arn](#input\_acm\_certificate\_arn) | The ARN of an existing ACM certificate. | `string` | `null` | no |
| <a name="input_allowed_inbound_cidr"></a> [allowed\_inbound\_cidr](#input\_allowed\_inbound\_cidr) | CIDRs allowed to access wandb-server. | `list(string)` | n/a | yes |
| <a name="input_allowed_inbound_ipv6_cidr"></a> [allowed\_inbound\_ipv6\_cidr](#input\_allowed\_inbound\_ipv6\_cidr) | CIDRs allowed to access wandb-server. | `list(string)` | n/a | yes |
| <a name="input_app_wandb_env"></a> [app\_wandb\_env](#input\_app\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
| <a name="input_aws_loadbalancer_controller_tags"></a> [aws\_loadbalancer\_controller\_tags](#input\_aws\_loadbalancer\_controller\_tags) | (Optional) A map of AWS tags to apply to all resources managed by the load balancer controller | `map(string)` | `{}` | no |
| <a name="input_bucket_kms_key_arn"></a> [bucket\_kms\_key\_arn](#input\_bucket\_kms\_key\_arn) | The Amazon Resource Name of the KMS key with which S3 storage bucket objects will be encrypted. | `string` | `""` | no |
| <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | n/a | `string` | `""` | no |
| <a name="input_create_bucket"></a> [create\_bucket](#input\_create\_bucket) | ######################################### External Bucket # ######################################### Most users will not need these settings. They are ment for users who want a bucket and sqs that are in a different account. | `bool` | `true` | no |
Expand All @@ -177,14 +179,17 @@ Upgrades must be executed in step-wise fashion from one version to the next. You
| <a name="input_enable_dummy_dns"></a> [enable\_dummy\_dns](#input\_enable\_dummy\_dns) | Boolean indicating whether or not to enable dummy DNS for the old alb | `bool` | `false` | no |
| <a name="input_enable_operator_alb"></a> [enable\_operator\_alb](#input\_enable\_operator\_alb) | Boolean indicating whether to use operatore ALB (true) or not (false). | `bool` | `false` | no |
| <a name="input_external_dns"></a> [external\_dns](#input\_external\_dns) | Using external DNS. A `subdomain` must also be specified if this value is true. | `bool` | `false` | no |
| <a name="input_extra_fqdn"></a> [extra\_fqdn](#input\_extra\_fqdn) | n/a | `list(string)` | `[]` | no |
| <a name="input_extra_fqdn"></a> [extra\_fqdn](#input\_extra\_fqdn) | Additional fqdn's must be in the same hosted zone as `domain_name`. | `list(string)` | `[]` | no |
| <a name="input_kms_key_alias"></a> [kms\_key\_alias](#input\_kms\_key\_alias) | KMS key alias for AWS KMS Customer managed key. | `string` | `null` | no |
| <a name="input_kms_key_deletion_window"></a> [kms\_key\_deletion\_window](#input\_kms\_key\_deletion\_window) | Duration in days to destroy the key after it is deleted. Must be between 7 and 30 days. | `number` | `7` | no |
| <a name="input_kms_key_policy"></a> [kms\_key\_policy](#input\_kms\_key\_policy) | The policy that will define the permissions for the kms key. | `string` | `""` | no |
| <a name="input_kubernetes_alb_internet_facing"></a> [kubernetes\_alb\_internet\_facing](#input\_kubernetes\_alb\_internet\_facing) | Indicates whether or not the ALB controlled by the Amazon ALB ingress controller is internet-facing or internal. | `bool` | `true` | no |
| <a name="input_kubernetes_alb_subnets"></a> [kubernetes\_alb\_subnets](#input\_kubernetes\_alb\_subnets) | List of subnet ID's the ALB will use for ingress traffic. | `list(string)` | `[]` | no |
| <a name="input_kubernetes_instance_types"></a> [kubernetes\_instance\_types](#input\_kubernetes\_instance\_types) | EC2 Instance type for primary node group. | `list(string)` | <pre>[<br> "m5.large"<br>]</pre> | no |
| <a name="input_kubernetes_map_accounts"></a> [kubernetes\_map\_accounts](#input\_kubernetes\_map\_accounts) | Additional AWS account numbers to add to the aws-auth configmap. | `list(string)` | `[]` | no |
| <a name="input_kubernetes_map_roles"></a> [kubernetes\_map\_roles](#input\_kubernetes\_map\_roles) | Additional IAM roles to add to the aws-auth configmap. | <pre>list(object({<br> rolearn = string<br> username = string<br> groups = list(string)<br> }))</pre> | `[]` | no |
| <a name="input_kubernetes_map_users"></a> [kubernetes\_map\_users](#input\_kubernetes\_map\_users) | Additional IAM users to add to the aws-auth configmap. | <pre>list(object({<br> userarn = string<br> username = string<br> groups = list(string)<br> }))</pre> | `[]` | no |
| <a name="input_kubernetes_node_count"></a> [kubernetes\_node\_count](#input\_kubernetes\_node\_count) | Number of nodes | `number` | `2` | no |
| <a name="input_kubernetes_public_access"></a> [kubernetes\_public\_access](#input\_kubernetes\_public\_access) | Indicates whether or not the Amazon EKS public API server endpoint is enabled. | `bool` | `false` | no |
| <a name="input_kubernetes_public_access_cidrs"></a> [kubernetes\_public\_access\_cidrs](#input\_kubernetes\_public\_access\_cidrs) | List of CIDR blocks which can access the Amazon EKS public API server endpoint. | `list(string)` | `[]` | no |
| <a name="input_license"></a> [license](#input\_license) | Weights & Biases license key. | `string` | n/a | yes |
Expand All @@ -193,17 +198,25 @@ Upgrades must be executed in step-wise fashion from one version to the next. You
| <a name="input_network_database_subnet_cidrs"></a> [network\_database\_subnet\_cidrs](#input\_network\_database\_subnet\_cidrs) | List of private subnet CIDR ranges to create in VPC. | `list(string)` | <pre>[<br> "10.10.20.0/24",<br> "10.10.21.0/24"<br>]</pre> | no |
| <a name="input_network_database_subnets"></a> [network\_database\_subnets](#input\_network\_database\_subnets) | A list of the identities of the database subnetworks in which resources will be deployed. | `list(string)` | `[]` | no |
| <a name="input_network_elasticache_subnet_cidrs"></a> [network\_elasticache\_subnet\_cidrs](#input\_network\_elasticache\_subnet\_cidrs) | List of private subnet CIDR ranges to create in VPC. | `list(string)` | <pre>[<br> "10.10.30.0/24",<br> "10.10.31.0/24"<br>]</pre> | no |
| <a name="input_network_elasticache_subnets"></a> [network\_elasticache\_subnets](#input\_network\_elasticache\_subnets) | A list of the identities of the subnetworks in which elasticache resources will be deployed. | `list(string)` | `[]` | no |
| <a name="input_network_id"></a> [network\_id](#input\_network\_id) | The identity of the VPC in which resources will be deployed. | `string` | `""` | no |
| <a name="input_network_private_subnet_cidrs"></a> [network\_private\_subnet\_cidrs](#input\_network\_private\_subnet\_cidrs) | List of private subnet CIDR ranges to create in VPC. | `list(string)` | <pre>[<br> "10.10.10.0/24",<br> "10.10.11.0/24"<br>]</pre> | no |
| <a name="input_network_private_subnets"></a> [network\_private\_subnets](#input\_network\_private\_subnets) | A list of the identities of the private subnetworks in which resources will be deployed. | `list(string)` | `[]` | no |
| <a name="input_network_public_subnet_cidrs"></a> [network\_public\_subnet\_cidrs](#input\_network\_public\_subnet\_cidrs) | List of private subnet CIDR ranges to create in VPC. | `list(string)` | <pre>[<br> "10.10.0.0/24",<br> "10.10.1.0/24"<br>]</pre> | no |
| <a name="input_network_public_subnets"></a> [network\_public\_subnets](#input\_network\_public\_subnets) | A list of the identities of the public subnetworks in which resources will be deployed. | `list(string)` | `[]` | no |
| <a name="input_other_wandb_env"></a> [other\_wandb\_env](#input\_other\_wandb\_env) | Extra environment variables for W&B | `map(any)` | `{}` | no |
| <a name="input_parquet_wandb_env"></a> [parquet\_wandb\_env](#input\_parquet\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
| <a name="input_private_link_allowed_account_ids"></a> [private\_link\_allowed\_account\_ids](#input\_private\_link\_allowed\_account\_ids) | List of AWS account IDs allowed to access the VPC Endpoint Service | `list(string)` | `[]` | no |
| <a name="input_public_access"></a> [public\_access](#input\_public\_access) | Is this instance accessable a public domain. | `bool` | `false` | no |
| <a name="input_size"></a> [size](#input\_size) | Deployment size | `string` | `null` | no |
| <a name="input_ssl_policy"></a> [ssl\_policy](#input\_ssl\_policy) | SSL policy to use on ALB listener | `string` | `"ELBSecurityPolicy-FS-1-2-Res-2020-10"` | no |
| <a name="input_subdomain"></a> [subdomain](#input\_subdomain) | Subdomain for accessing the Weights & Biases UI. Default creates record at Route53 Route. | `string` | `null` | no |
| <a name="input_system_reserved_cpu_millicores"></a> [system\_reserved\_cpu\_millicores](#input\_system\_reserved\_cpu\_millicores) | (Optional) The amount of 'system-reserved' CPU millicores to pass to the kubelet. For example: 100. A value of -1 disables the flag. | `number` | `70` | no |
| <a name="input_system_reserved_ephemeral_megabytes"></a> [system\_reserved\_ephemeral\_megabytes](#input\_system\_reserved\_ephemeral\_megabytes) | (Optional) The amount of 'system-reserved' ephemeral storage in megabytes to pass to the kubelet. For example: 1000. A value of -1 disables the flag. | `number` | `750` | no |
| <a name="input_system_reserved_memory_megabytes"></a> [system\_reserved\_memory\_megabytes](#input\_system\_reserved\_memory\_megabytes) | (Optional) The amount of 'system-reserved' memory in megabytes to pass to the kubelet. For example: 100. A value of -1 disables the flag. | `number` | `100` | no |
| <a name="input_system_reserved_pid"></a> [system\_reserved\_pid](#input\_system\_reserved\_pid) | (Optional) The amount of 'system-reserved' process ids [pid] to pass to the kubelet. For example: 1000. A value of -1 disables the flag. | `number` | `500` | no |
| <a name="input_use_internal_queue"></a> [use\_internal\_queue](#input\_use\_internal\_queue) | n/a | `bool` | `false` | no |
| <a name="input_weave_wandb_env"></a> [weave\_wandb\_env](#input\_weave\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
| <a name="input_zone_id"></a> [zone\_id](#input\_zone\_id) | Domain for creating the Weights & Biases subdomain on. | `string` | n/a | yes |

## Outputs
Expand All @@ -216,14 +229,19 @@ Upgrades must be executed in step-wise fashion from one version to the next. You
| <a name="output_cluster_id"></a> [cluster\_id](#output\_cluster\_id) | n/a |
| <a name="output_cluster_node_role"></a> [cluster\_node\_role](#output\_cluster\_node\_role) | n/a |
| <a name="output_database_connection_string"></a> [database\_connection\_string](#output\_database\_connection\_string) | n/a |
| <a name="output_database_instance_type"></a> [database\_instance\_type](#output\_database\_instance\_type) | n/a |
| <a name="output_database_password"></a> [database\_password](#output\_database\_password) | n/a |
| <a name="output_database_username"></a> [database\_username](#output\_database\_username) | n/a |
| <a name="output_eks_node_count"></a> [eks\_node\_count](#output\_eks\_node\_count) | n/a |
| <a name="output_eks_node_instance_type"></a> [eks\_node\_instance\_type](#output\_eks\_node\_instance\_type) | n/a |
| <a name="output_elasticache_connection_string"></a> [elasticache\_connection\_string](#output\_elasticache\_connection\_string) | n/a |
| <a name="output_internal_app_port"></a> [internal\_app\_port](#output\_internal\_app\_port) | n/a |
| <a name="output_kms_key_arn"></a> [kms\_key\_arn](#output\_kms\_key\_arn) | The Amazon Resource Name of the KMS key used to encrypt data at rest. |
| <a name="output_network_id"></a> [network\_id](#output\_network\_id) | The identity of the VPC in which resources are deployed. |
| <a name="output_network_private_subnets"></a> [network\_private\_subnets](#output\_network\_private\_subnets) | The identities of the private subnetworks deployed within the VPC. |
| <a name="output_network_public_subnets"></a> [network\_public\_subnets](#output\_network\_public\_subnets) | The identities of the public subnetworks deployed within the VPC. |
| <a name="output_redis_instance_type"></a> [redis\_instance\_type](#output\_redis\_instance\_type) | n/a |
| <a name="output_standardized_size"></a> [standardized\_size](#output\_standardized\_size) | n/a |
| <a name="output_url"></a> [url](#output\_url) | The URL to the W&B application |

<!-- END_TF_DOCS -->
Expand Down
20 changes: 16 additions & 4 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,8 @@ locals {
module "app_eks" {
source = "./modules/app_eks"

fqdn = local.domain_filter
fqdn = local.domain_filter
subject_alternative_names = var.enable_dummy_dns ? var.extra_fqdn : []

namespace = var.namespace
kms_key_arn = local.kms_key_arn
Expand Down Expand Up @@ -152,6 +153,11 @@ module "app_eks" {
aws_loadbalancer_controller_tags = var.aws_loadbalancer_controller_tags
}

locals {
full_fqdn = var.enable_dummy_dns ? "old-${local.fqdn}" : local.fqdn
extra_fqdn = var.enable_dummy_dns ? [for fqdn in var.extra_fqdn : "old-${fqdn}"] : var.extra_fqdn
}

module "app_lb" {
source = "./modules/app_lb"

Expand All @@ -160,8 +166,8 @@ module "app_lb" {
acm_certificate_arn = local.acm_certificate_arn
zone_id = var.zone_id

fqdn = var.enable_dummy_dns ? "old.${local.fqdn}" : local.fqdn
extra_fqdn = var.extra_fqdn
fqdn = local.full_fqdn
extra_fqdn = local.extra_fqdn
allowed_inbound_cidr = var.allowed_inbound_cidr
allowed_inbound_ipv6_cidr = var.allowed_inbound_ipv6_cidr
target_port = local.internal_app_port
Expand Down Expand Up @@ -270,13 +276,19 @@ module "wandb" {
"alb.ingress.kubernetes.io/inbound-cidrs" = <<-EOF
${join("\\,", var.allowed_inbound_cidr)}
EOF
"external-dns.alpha.kubernetes.io/hostname" = var.enable_operator_alb ? local.fqdn : ""
"external-dns.alpha.kubernetes.io/ingress-hostname-source" = "annotation-only"
"alb.ingress.kubernetes.io/scheme" = var.kubernetes_alb_internet_facing ? "internet-facing" : "internal"
"alb.ingress.kubernetes.io/target-type" = "ip"
"alb.ingress.kubernetes.io/listen-ports" = "[{\\\"HTTPS\\\": 443}]"
"alb.ingress.kubernetes.io/certificate-arn" = local.acm_certificate_arn
},
length(var.extra_fqdn) > 0 && var.enable_dummy_dns ? {
"external-dns.alpha.kubernetes.io/hostname" = <<-EOF
${local.fqdn}\,${join("\\,", var.extra_fqdn)}\,${local.fqdn}
EOF
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I use the local.fqdn twice is because of a bug with external dns not reading this form of the annotation correctly:

      external-dns.alpha.kubernetes.io/hostname: |
        smooth-operator.sandbox-aws.wandb.ml,rough-operator.sandbox-aws.wandb.ml

In order to get around this, I added ${local.fqdn} to the end which works just perfectly. 😢

time="2024-03-29T21:23:33Z" level=debug msg="Endpoints generated from ingress: default/wandb: [smooth-operator.sandbox-aws.wandb.ml 0 IN CNAME  smooth-operator-alb-k8s-388333795.us-east-2.elb.amazonaws.com [] rough-operator.sandbox-aws.wandb.ml 0 IN CNAME  smooth-operator-alb-k8s-388333795.us-east-2.elb.amazonaws.com [] smooth-operator.sandbox-aws.wandb.ml\n 0 IN CNAME  smooth-operator-alb-k8s-388333795.us-east-2.elb.amazonaws.com []]"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sure doesn't look perfect to me with a duplicated entry in the endpoints generated. Is there something goofy we're doing with writing out these annotations that it ends up in an incompatible format or is it just good ol' externaldns madness?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

External DNS madness. I am actually creating an issue now in https://github.com/kubernetes-sigs/external-dns/issues. Will comment when I create it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

} : {
"external-dns.alpha.kubernetes.io/hostname" = var.enable_operator_alb ? local.fqdn : ""
},
length(var.kubernetes_alb_subnets) > 0 ? {
"alb.ingress.kubernetes.io/subnets" = <<-EOF
${join("\\,", var.kubernetes_alb_subnets)}
Expand Down
20 changes: 15 additions & 5 deletions modules/app_eks/external_dns/external_dns.tf
Original file line number Diff line number Diff line change
@@ -1,8 +1,15 @@
locals {
domain_filters = concat(
[{ name = "domainFilters[0]", value = var.fqdn }],
[for san in var.subject_alternative_names : { name = "domainFilters[${index(var.subject_alternative_names, san) + 1}]", value = san }]
)
}

resource "helm_release" "external_dns" {
name = "external-dns"
namespace = "kube-system"
chart = "external-dns"
version = "1.13.1"
version = "1.14.1"
repository = "https://kubernetes-sigs.github.io/external-dns"

set {
Expand All @@ -20,11 +27,14 @@ resource "helm_release" "external_dns" {
value = "external-dns"
}

set {
name = "domainFilters[0]"
value = var.fqdn
}
dynamic "set" {
for_each = local.domain_filters

content {
name = set.value.name
value = set.value.value
}
}
set {
name = "policy"
value = "sync"
Expand Down
4 changes: 4 additions & 0 deletions modules/app_eks/external_dns/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,7 @@ variable "oidc_provider" {
variable "fqdn" {
type = string
}

variable "subject_alternative_names" {
type = list(string)
}
2 changes: 1 addition & 1 deletion modules/app_eks/lb_controller/controller.tf
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ resource "helm_release" "aws_load_balancer_controller" {
repository = "https://aws.github.io/eks-charts"
chart = "aws-load-balancer-controller"
namespace = "kube-system"
version = "1.6.2"
version = "1.7.2"

set {
name = "clusterName"
Expand Down
4 changes: 3 additions & 1 deletion modules/app_eks/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,9 @@ module "external_dns" {

namespace = var.namespace
oidc_provider = aws_iam_openid_connect_provider.eks
fqdn = var.fqdn

fqdn = var.fqdn
subject_alternative_names = var.subject_alternative_names

depends_on = [module.eks]
}
4 changes: 4 additions & 0 deletions modules/app_eks/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ variable "fqdn" {
type = string
}

variable "subject_alternative_names" {
type = list(string)
}

variable "bucket_sqs_queue_arn" {
default = ""
type = string
Expand Down
5 changes: 3 additions & 2 deletions variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,9 @@ variable "enable_operator_alb" {
}

variable "extra_fqdn" {
type = list(string)
default = []
type = list(string)
description = "Additional fqdn's must be in the same hosted zone as `domain_name`."
default = []
}

##########################################
Expand Down
Loading