Add auxiliary etl query table

[maxalbert]

Closes #988

I have:

• Formatted any Python files with black
• Brought the branch up to date with master
• Added any relevant Github labels
• Added tests for any new additions
• Added or updated any relevant documentation
• Added an Architectural Decision Record (ADR), if appropriate
• Added an MPLv2 License Header if appropriate
• Updated the Changelog

Description

This PR adds a table etl.post_etl_queries to record the outcome of queries run as part of the regular ETL process. I'm happy to tweak the name and structure of the table if we think something else is more appropriate, suggestions are welcome.

For the time being I have kept the data types for all columns (except cdr_date) as TEXT. Going forward one could imagine using enums for cdr_type and in particular for type_of_query_or_check (so that we enforce consistency in this column, which would help with querying the table). In the future we may also be more specific with the data type of the outcome column (e.g. use int/float/boolean/text depending on the actual query it records) but until the details become clearer and we split this auxiliary table into separate ones (probably as part of #640) this seemed to be the most generically useful approach.

[greenape]

Can we record the date of the check as well?

[maxalbert]

Good point. 👍

[maxalbert]

I've made it a TIMESTAMPTZ rather than a DATE column because there is no harm in having the more precise info about when the check was run.

[codecov]

Codecov Report

Merging #993 into master will decrease coverage by 6.63%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #993      +/-   ##
==========================================
- Coverage     100%   93.36%   -6.64%     
==========================================
  Files           1      137     +136     
  Lines          92     6892    +6800     
  Branches        7      699     +692     
==========================================
+ Hits           92     6435    +6343     
- Misses          0      332     +332     
- Partials        0      125     +125

Flag	Coverage Δ
#flowapi_unit_tests	77.27% <ø> (?)
#flowauth_unit_tests	94.86% <ø> (?)
#flowclient_unit_tests	82.29% <ø> (?)
#flowetl_unit_tests	100% <ø> (?)
#flowkit_jwt_generator_unit_tests	100% <ø> (ø)	⬆️
#flowmachine_unit_tests	91.06% <ø> (?)
#integration_tests	60.16% <ø> (?)

Impacted Files	Coverage Δ
...e/server/query_schemas/joined_spatial_aggregate.py	100% <0%> (ø)
...achine/core/server/query_schemas/daily_location.py	100% <0%> (ø)
...e/flowmachine/features/raster/raster_statistics.py	87.87% <0%> (ø)
...lowmachine/flowmachine/features/spatial/circles.py	100% <0%> (ø)
flowmachine/flowmachine/core/custom_query.py	100% <0%> (ø)
flowmachine/flowmachine/core/server/server.py	87.34% <0%> (ø)
...hine/features/location/unique_subscriber_counts.py	100% <0%> (ø)
...ne/core/server/query_schemas/radius_of_gyration.py	100% <0%> (ø)
...owmachine/features/subscriber/subscriber_degree.py	100% <0%> (ø)
flowapi/flowapi/main.py	96.1% <0%> (ø)
... and 126 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e247fc...87b6ed3. Read the comment docs.

[codecov]

Codecov Report

Merging #993 into master will decrease coverage by 6.63%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #993      +/-   ##
==========================================
- Coverage     100%   93.36%   -6.64%     
==========================================
  Files           1      137     +136     
  Lines          92     6892    +6800     
  Branches        7      699     +692     
==========================================
+ Hits           92     6435    +6343     
- Misses          0      332     +332     
- Partials        0      125     +125

Flag	Coverage Δ
#flowapi_unit_tests	77.27% <ø> (?)
#flowauth_unit_tests	94.86% <ø> (?)
#flowclient_unit_tests	82.29% <ø> (?)
#flowetl_unit_tests	100% <ø> (?)
#flowkit_jwt_generator_unit_tests	100% <ø> (ø)	⬆️
#flowmachine_unit_tests	91.06% <ø> (?)
#integration_tests	60.16% <ø> (?)

Impacted Files	Coverage Δ
...e/server/query_schemas/joined_spatial_aggregate.py	100% <0%> (ø)
...achine/core/server/query_schemas/daily_location.py	100% <0%> (ø)
...e/flowmachine/features/raster/raster_statistics.py	87.87% <0%> (ø)
...lowmachine/flowmachine/features/spatial/circles.py	100% <0%> (ø)
flowmachine/flowmachine/core/custom_query.py	100% <0%> (ø)
flowmachine/flowmachine/core/server/server.py	87.34% <0%> (ø)
...hine/features/location/unique_subscriber_counts.py	100% <0%> (ø)
...ne/core/server/query_schemas/radius_of_gyration.py	100% <0%> (ø)
...owmachine/features/subscriber/subscriber_degree.py	100% <0%> (ø)
flowapi/flowapi/main.py	96.1% <0%> (ø)
... and 126 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e247fc...87b6ed3. Read the comment docs.