Backend Infrastructure next steps (EPIC)

[batpad]

We successfully migrated the GO deploy to a Kubernetes based backend on AKS with CI running using Azure Pipelines: #1531

Creating this ticket with further actions and next steps here. Some of these are smaller, more immediate tasks, and some of these are more long-term with a view to expanding the infrastructure setup to include other GO-adjacent applications like the Risk Module, as well as other projects like Alert Hub and other experimental and student projects.

More short-term tasks:

• Setup SSH tunnel to database to allow database access for external users
• Setup resource configuration such that we can have different resource allocations for staging and production
• Improve documentation for new deploy setup, add guidelines on creating releases, etc. under new system
• Setup Sentry within the cluster
• Upgrade Kubernetes version from 1.25.5 to (at least 1.26.6 or better to) 1.27.7
• Remove unused or redundant env variables (e.g. DJANGO_DB_HOST + RESOURCES_DB_SERVER)
• Get rid of log noise (e.g. WARNING - Queue is full. Dropping telemetry)

Medium-term:

• Move deploy folder in go-api to its own repository, separating out management of the cluster and deployment of individual applications
• Setup CPU / Memory utilization based auto-scaling on the GO API
• Improve handling of secrets and configuration variables to make it easier for individual applications to manage secrets and configuration
• Create consistent pattern to deploy go-api, risk module, and other applications
• Create clear documentation to run and test all the k8s code locally
• Add Schema JSON for values configuration to make validation of configuration options better
• Add basic tests for Helm chart to increase confidence while deploying applications

Longer-term:

• Setup a playground cluster to allow more experimental projects to use as a testing space for deployments and ideas
• Have comprehensive documentation on best practices for deploying an app to the "GO Ecosystem", including setting up CI / CD
• Investigate and setup comprehensive monitoring and logging tools across clusters, manage permissions so that we can give well scoped permissions to people to monitor their own applications

@tovari I can break these out into separate more detailed tickets as we figure prioritization.

cc @szabozoltan69 @geohacker - please add anything here that I might have missed!

+cc @sunu

[batpad]

+cc @k9845 - there are some dependencies here that ideally we'd get through before doing the Risk Module deploy into the cluster, so we can do it in a more "proper way", but again, let's discuss how best to proceed - there maybe an intermediary solution if we want to hasten up the migration of Risk Module onto the GO cluster.

[szabozoltan69]

About #2086 : Recently all our GO infrastructure is in Azure, including the ifrcgoacr container registry. Why don't we use this container registry instead of introducing a new one? We can have multiple repos in it.

[sunu]

@szabozoltan69 - replied to your comment in that thread: #2086 (comment)