Converting W&B Endpoint from Public to Private

[flamarion]

Objective:

Provide a solution for converting a publicly exposed W&B endpoint deployed using the W&B AWS Terraform Module to a private endpoint within the customer’s cloud environment. The conversion must ensure minimal disruption to existing resources (DB, Bucket...) and no Terraform state drift after the change.

Current Behavior:

• Deployments using the W&B AWS Terraform Module expose the W&B endpoint publicly by default.
• Example with Route 53 Public DNS
• Example with External Public DNS

Requirements:

1. Enable Conversion to Private Endpoint:

• Provide a mechanism to reconfigure the W&B endpoint from public to private without disrupting the existing resources.

2. Assumptions for the Customer Environment:

• Private DNS Zone is available.
• Private Network is set up to support an internal load balancer (preferably an ALB).
• All clients are properly configured to access W&B through the private endpoint.

3. Use SSL/TLS with ACM Certificates (to be confirmed):

• Ensure the internal ALB terminates SSL/TLS using a certificate issued by AWS Certificate Manager (ACM).
  • Create a new one or using an exiting.
• Requirements for the certificate:
  • It must match the private DNS name of the W&B endpoint.
  • It must be issued and available in the same AWS region as the ALB.
  • Ensure the DNS configuration properly resolves the private DNS name to the ALB’s internal IP address.
  • Traffic between the clients and the internal ALB must be encrypted using the ACM certificate.

4. Avoid Terraform State Drift:

• If the implementation uses only Terraform, ensure that the changes are managed entirely through Terraform to avoid manual configurations.
• If external tools/scripts are needed, they must prevent Terraform from reverting changes after the execution.
• Scripts should be developed in Golang if applicable.

5. Flexibility:

• Preference is given to a Terraform-only solution.
• If a Terraform-only solution is not feasible, provide a well-documented procedure that includes external scripts.

6. Testing:

• Test the conversion process thoroughly in an isolated environment.
• Validate that the endpoint becomes private, the ACM certificate is applied correctly, and all clients can access the endpoint without disruptions.

Deliverables:

1. Solution Implementation:

• A Terraform-only implementation OR
• A procedure detailing the conversion process with necessary scripts (preferably in Golang).

2. Documentation:

• Clear and detailed documentation on how to execute the conversion.
• Include prerequisites, step-by-step instructions, and expected outcomes.

3. Testing Instructions (goot to have):

• Steps to validate the solution in a test environment.
• Verification checklist to ensure the endpoint is private, SSL/TLS is configured with the ACM certificate, and the system is fully functional.

4. Avoid Terraform Drift:

• Ensure that all changes made by the solution remain consistent with the Terraform state.

References:

• W&B AWS Terraform Module Documentation
• W&B Deployment Documentation
• Examples:
  • Public DNS with Route53
  • Public DNS with External