[Delivery metrics 2.0] Data quality checks

[widal001]

Description

Add data quality checks to catch issues with our ETL pipeline or data platform, so that we can address issues early rather than having them "leak" into the user facing dashboards. Consider adopting a tool like great expectations to assist with this.

Questions

• Which tool(s) should we use for data quality monitoring? Options include:
  • Great expectations
  • Monte carlo
• At which stage in the process should we set up these checks? For example:
  • After export but before load
  • After load into the DB
• What should we do when a data quality check fails? For example:
  • Notify the team via slack
  • Halt execution
  • Write failing fields to a dead letter queue

Probability of success and key risks

• Probability of success: 80%
• Key risks
  • Adding data quality monitoring increases the overall complexity of the application
  • Adding data quality monitoring results in too many false positives, blocking downstream metrics

Date of delivery (estimated)

Sprint 2.5 (Mar 18, 2024)

Level of effort

L