📖 Airflow 3.0 - SecretsManager for User Secrets

[jhpyke]

User Story

As a user of the new airflow environment
I need to be able to store secrets in a consistent manner
So that there's a known, well understood method for me to create a secret for my airflow job that can be consumed at runtime.

Value / Purpose

Current mechanisms for achieving this are very ad hoc, with users essentially having to directly request with us the creation and population of secrets in the data-production account. This ticket looks at creating a user driven setup where they can request the creation of new secrets, and be given a way of populating those underlying values.

Useful Contacts

No response

User Types

No response

Hypothesis

If we... allow users to create secrets easily
Then... they will be less likely to do weird janky ways of passing secret values around

Proposal

• Users creating an Airflow Role that requests secrets manager access should be able to request secrets within an area we provide (e.g. `airflow//<dag_name>/*)
• When creating these secrets, they should be able to specify alpha_users who should be able to set the value of those secrets
• Provide those alpha_users a mechanism for setting secret values that doesn't require us to intervene directly (control panel page? Script?)

Additional Information

No response

Definition of Done

• Users can define secrets to be stored in analytical-platform-compute
• Users are able to update secrets they've defined
• Secrets can be associated with the DAGs they belong to