[Issue 963] Moves ECS services into private subnets #1014

coilysiren · 2024-01-12T01:16:01Z

Summary

Fixes #963

This PR will be split into 3 major stages:

Stage 1: Create private subnets

The existing networking infrastructure is in its "default" (AWS provided) state. This state does not meet the standards of our ideal network layer design in a few ways. But the most notable one is: the default networking infrastructure lacks private subnets. You cannot remove the public IP addresses from the ECS instances without a first creating private subnets. So I'm requesting code review for Stage 1, backfilling the private subnets into the default VPC. This also includes attaching the VPC endpoints onto those private subnets.

Deploying this will require me to deploy the subnets from my branch, before I merge. I've talked about this with @aplybeah. Essentially, I need to (temporarily) ignore "only deploy from main" rule in order to resolve #963 in a timely manner. The alternative would be for me to take several days to deploy an entirely new networking layer from scratch, which feels like too much work for this task / our timeline.

Due to the above, there will be a long running (eg. for a few days) terraform state diff on the infra/networks/ folder.

Stage 2: Creating a "staging" site inside the private subnets

When I create the private subnets, I won't be able to be sure that they are totally working for the sake of deploying our application. There's a chance this could be a breaking change from the point of view of the application. So it requires testing, specifically testing in the form of creating a 3rd simpler ECS application. I'm calling this application "staging", since "dev" and "prod" are already in use. I'll deploy this application to the private subnets to confirm that the application is working. I'll then keep the application around so that we can use it for other purposes later.

Stage 3: Move all sites into the private subnets

After confirming that staging works, I'll make sure the terraform configuration for dev and prod are using the new subnets as well. I'll deploy to dev and prod after I merge. This will not change the load balancer URL, but it might cause a few minutes of downtime. I expect that any potential downtime will be 5 minutes at most, enough time for the new app containers to start up.

Testing

Deploy the networking layer changes:

$ cd infra/networks/
$ terraform apply

Deploy the frontend application (with the latest build as of Jan 16th)

$ cd infra/frontend/service
$ terraform init -backend-config=staging.s3.tfbackend -reconfigure
$ terraform apply \
    -var="environment_name=staging" \
    -var="image_tag=2698a01eca3a7373ae4ddeec78271108f50234fd"

Deploy the database

$ cd infra/api/database
$ terraform init -backend-config=staging.s3.tfbackend -reconfigure
$ terraform apply \
    -var="environment_name=staging"

Deploy the backend application (with the latest build as of Jan 16th)

$ cd infra/api/service
$ terraform init -backend-config=staging.s3.tfbackend -reconfigure
$ terraform apply \
    -var="environment_name=staging" \
    -var="image_tag=2698a01eca3a7373ae4ddeec78271108f50234fd"

Future Work

In the future, I would like to move to the network architecture described in the Nava infra template. That design specifies a separate network for every environment. This would be an improvement over our current design, where there is one network for all environments. The fact that there's only 1 network is what makes testing stage 1 so complicated.

This PR includes a new staging app, but does not (yet) add that staging app to Github Actions. I intend to do that at a later date!

jamesbursa

Looks good!

infra/networks/subnet-backfill.tf

Co-authored-by: James Bursa <[email protected]>

coilysiren · 2024-01-16T21:28:33Z

infra/frontend/service/main.tf

+    name   = "tag:subnet_type"
+    values = ["private"]


Changes from deploying in every subnet, to only deploying into the private subnets

coilysiren · 2024-01-16T21:29:08Z

infra/modules/service/main.tf

-    # TODO(https://github.com/navapbc/template-infra/issues/152) set assign_public_ip = false after using private subnets
-    # checkov:skip=CKV_AWS_333:Switch to using private subnets
-    assign_public_ip = true
+    assign_public_ip = false


This line is the whole reason I'm here

coilysiren · 2024-01-16T21:29:35Z

infra/networks/main.tf

+  # S3 and DynamoDB use Gateway VPC endpoints. All other services use Interface VPC endpoints
+  interface_vpc_endpoints = toset([for aws_service in local.aws_service_integrations : aws_service if !contains(["s3", "dynamodb"], aws_service)])
+  gateway_vpc_endpoints   = toset([for aws_service in local.aws_service_integrations : aws_service if contains(["s3", "dynamodb"], aws_service)])


This pattern was pulled from the Nava template: https://github.com/navapbc/template-infra/

coilysiren · 2024-01-16T21:31:26Z

infra/networks/main.tf

 resource "aws_vpc_endpoint" "aws_service" {
  for_each = local.aws_service_integrations

  vpc_id              = data.aws_vpc.default.id
  service_name        = "com.amazonaws.${data.aws_region.current.name}.${each.key}"
  vpc_endpoint_type   = "Interface"
  security_group_ids  = [aws_security_group.aws_services[0].id]
-  subnet_ids          = data.aws_subnets.default.ids
+  subnet_ids          = [for subnet in aws_subnet.backfill_private : subnet.id]


See: https://betagrantsgov.slack.com/archives/C05TSL64VUH/p1705438485101009

I needed to deploy the VPC endpoints exclusively into the private subnets

coilysiren · 2024-01-16T22:38:20Z

infra/modules/service/variables.tf

@@ -70,9 +70,20 @@ variable "vpc_id" {
  description = "Uniquely identifies the VPC."
 }

-variable "subnet_ids" {
+variable "public_subnet_ids" {


A lot of these changes were all pulled from the Nava template. Here's the diff between our version and main:

navapbc/template-infra@fe5c7cd...main

coilysiren · 2024-01-17T20:40:43Z

infra/modules/database/role_manager/role_manager.py

@@ -2,9 +2,14 @@
 import itertools
 from operator import itemgetter
 import os
+import json


via @chouinar and https://github.com/navapbc/template-infra/blame/7f43aaa9f1ee630e4d674d3197bbcaf134bafc19/infra/modules/database/role_manager/role_manager.py#L5

coilysiren · 2024-01-17T20:42:23Z

I'm going to merge as-is since I've got two approvals! I'll watch the dev deploy to make sure it works 🙏🏼

Backfill private subnets

766db2d

coilysiren changed the title ~~Backfill private subnets~~ [Issue 963] Moves ECS services into private subnets Jan 12, 2024

checkov

563b26d

github-actions bot added terraform infra labels Jan 12, 2024

coilysiren added 3 commits January 12, 2024 12:40

Merge remote-tracking branch 'origin/main' into backfill-subnets

325b8e6

docs

43babf2

docs

43fde08

coilysiren self-assigned this Jan 12, 2024

coilysiren marked this pull request as ready for review January 12, 2024 22:02

coilysiren requested review from acouch, aplybeah and jamesbursa January 12, 2024 22:03

jamesbursa approved these changes Jan 16, 2024

View reviewed changes

infra/networks/subnet-backfill.tf Outdated Show resolved Hide resolved

infra/networks/subnet-backfill.tf Outdated Show resolved Hide resolved

infra/networks/subnet-backfill.tf Outdated Show resolved Hide resolved

infra/networks/subnet-backfill.tf Outdated Show resolved Hide resolved

coilysiren and others added 9 commits January 16, 2024 09:25

Update infra/networks/subnet-backfill.tf

175950c

Co-authored-by: James Bursa <[email protected]>

Update infra/networks/subnet-backfill.tf

d8dabe7

Co-authored-by: James Bursa <[email protected]>

Update infra/networks/subnet-backfill.tf

c92157a

Co-authored-by: James Bursa <[email protected]>

Update infra/networks/subnet-backfill.tf

046898c

Co-authored-by: James Bursa <[email protected]>

Stage 2 changes

3d9074d

move subnets

62ac8c7

security group ingress

d460cce

staging deploy

595df4f

docs

e730721

coilysiren commented Jan 16, 2024

View reviewed changes

pull in latest nava template changes

4434228

coilysiren commented Jan 16, 2024

View reviewed changes

use staging sendy

530f1bd

add staging backend

f0ae706

aplybeah approved these changes Jan 17, 2024

View reviewed changes

fix role manager lambda

e3998c2

github-actions bot added the python label Jan 17, 2024

coilysiren commented Jan 17, 2024

View reviewed changes

coilysiren merged commit 174d6e8 into main Jan 17, 2024
9 checks passed

coilysiren deleted the backfill-subnets branch January 17, 2024 20:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Issue 963] Moves ECS services into private subnets #1014

[Issue 963] Moves ECS services into private subnets #1014

coilysiren commented Jan 12, 2024 •

edited

Loading

jamesbursa left a comment

coilysiren Jan 16, 2024

coilysiren Jan 16, 2024

coilysiren Jan 16, 2024

coilysiren Jan 16, 2024

coilysiren Jan 16, 2024 •

edited

Loading

coilysiren Jan 17, 2024

coilysiren commented Jan 17, 2024

[Issue 963] Moves ECS services into private subnets #1014

[Issue 963] Moves ECS services into private subnets #1014

Conversation

coilysiren commented Jan 12, 2024 • edited Loading

Summary

Stage 1: Create private subnets

Stage 2: Creating a "staging" site inside the private subnets

Stage 3: Move all sites into the private subnets

Testing

Future Work

jamesbursa left a comment

Choose a reason for hiding this comment

coilysiren Jan 16, 2024

Choose a reason for hiding this comment

coilysiren Jan 16, 2024

Choose a reason for hiding this comment

coilysiren Jan 16, 2024

Choose a reason for hiding this comment

coilysiren Jan 16, 2024

Choose a reason for hiding this comment

coilysiren Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

coilysiren Jan 17, 2024

Choose a reason for hiding this comment

coilysiren commented Jan 17, 2024

coilysiren commented Jan 12, 2024 •

edited

Loading

coilysiren Jan 16, 2024 •

edited

Loading