Crush

(description)

Architecture:

Authentication Service

User Service

S3

Match Service

Match Generation Engine

SQS + Lambda Consumer

• Generated matches are released bi-monthly at 5 pm, with traffic being highly concentrated at that time. To handle this, while ensuring the guarantees and restrictions of matching, selecting new matches is decoupled with SQS and searching for users is handled via Opensearch (see below).

Database

• PostgreSQL 15, hosted on RDS.
• Transaction isolation level: Read Committed
• Schema: see ./database/rds_schema.sql

Opensearch

• Uses change data capture (CDC) via DMS to keep the Opensearch index eventually consistent with RDS. Avoids unnecessary expenses of 2PC, while still handling 'bursty' search load at scale.

Deploying:

• Install dependencies:

  • Terraform

  • AWS CLI

  • kubernetes (kubectl)

TO-DO

MVP:

• Terraform IaC for AWS

  • VPC
  • EKS
  • RDS
  • S3
  • CloudFront
  • Route53
  • ACM
  • Opensearch, with CDC using DMS
  • Lambda
  • SQS

• Authentication Service, using Google OAuth 2.0

• User Service

  • CRUD with DB
  • Signed S3 URL generation
  • Opensearch integration

• Match Service

  • Match generation engine
  • Read from DB
  • Decoupled Create/Update matches through SQS
  • Lambda SQS consumer

• Github Actions CI/CD pipelines (Build, Test, Deploy)

Planned Infrastructure:

• Additional unit tests and more comprehensive integration tests
• Integrate Logrus for structured logging
• Security/Permissions:
  • IAM role/PostgresQL user with more fine-grained permissions for each service
• Upgrade to Postgres 17 for additional logical replication features (need DMS support; currently not supported)

Planned Features:

• User notification on match via email/text using SNS
• Extend to graduate students at Yale

Motivation