You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As part of the daily ingestion process one typically wants to run some initial quality checks or calculate informative metrics. Simple examples: how many call events happened on that date? Are there any cell_id values in this date's CDR data which are not present in the infrastructure.cells table (and if so how many)? Etc.
In the longer run this will be supported by a set of supplementary tables (to be added as part of #640) that record the results of standard queries (or standard pre-aggregations) which are run as part of the ingestion process.
In the immediate short term it will be useful to have a single auxiliary table where the results of semi-ad-hoc queries can be recorded in a slightly less structured and more informal way but which still allows quick querying of results. Suggestion for the structure of this table:
As part of the daily ingestion process one typically wants to run some initial quality checks or calculate informative metrics. Simple examples: how many call events happened on that date? Are there any
cell_id
values in this date's CDR data which are not present in theinfrastructure.cells
table (and if so how many)? Etc.In the longer run this will be supported by a set of supplementary tables (to be added as part of #640) that record the results of standard queries (or standard pre-aggregations) which are run as part of the ingestion process.
In the immediate short term it will be useful to have a single auxiliary table where the results of semi-ad-hoc queries can be recorded in a slightly less structured and more informal way but which still allows quick querying of results. Suggestion for the structure of this table:
Such a table should be added to flowdb, e.g. as
etl.daily_queries
(any suggestions for a better name?).The text was updated successfully, but these errors were encountered: