Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow running informative queries as part of etl process #989

Closed
maxalbert opened this issue Jun 27, 2019 · 0 comments · Fixed by #1051
Closed

Allow running informative queries as part of etl process #989

maxalbert opened this issue Jun 27, 2019 · 0 comments · Fixed by #1051
Assignees
@danwilliams
Labels
enhancement New feature or request FlowETL

Comments

@maxalbert
Copy link
Contributor

FlowETL should allow running supplementary queries on top of the daily ingestion process (see #988). In the future we will want to make this more sophisticated, but in the short run it would be useful to run a series of simple queries whose results are recorded in the table suggested in #988. This should probably happen in a "post-load" step in the etl DAG.
@maxalbert maxalbert added enhancement New feature or request FlowETL labels Jun 27, 2019
danwilliams added a commit that referenced this issue Jul 3, 2019
@danwilliams
Added initial ETLPostQueryOutcome model (#989)
ab4002c
danwilliams added a commit that referenced this issue Jul 3, 2019
@danwilliams
Added id as a primary key to the post_etl_queries table (#989)
4695dca 
Added id because there needs to be a primary key for SQLAlchemy to map
the table.
danwilliams added a commit that referenced this issue Jul 3, 2019
@danwilliams
Added tests for the ETLPostQueryOutcome model (#989)
d19d11a
danwilliams added a commit that referenced this issue Jul 3, 2019
@danwilliams
Added initial postload task (#989)
33acd5e 
This will be responsible for running the queries and record the results.
danwilliams added a commit that referenced this issue Jul 4, 2019
@danwilliams
Added initial postload queries structure and logic for running (#989)
943d04e
danwilliams added a commit that referenced this issue Jul 4, 2019
@danwilliams
Updated query mapping to CDR type and removed useless comments (#989)
41794e1
danwilliams added a commit that referenced this issue Jul 4, 2019
@danwilliams
Added sample num_total_calls function for the CALLS cdr_type (#989)
b435167
danwilliams added a commit that referenced this issue Jul 4, 2019
@danwilliams
Added logic to run queries mapped to the CDR type, record outcome (#989)
7393f99
danwilliams added a commit that referenced this issue Jul 8, 2019
@danwilliams
Updated code styling (#989)
22f8472
danwilliams added a commit that referenced this issue Jul 8, 2019
@danwilliams
Added tests for the postetl query logic (#989)
49b5576
danwilliams added a commit that referenced this issue Jul 8, 2019
@danwilliams
Fixed test_dags_present test fail (#989)
69d16f1
@danwilliams danwilliams mentioned this issue Jul 8, 2019
Merged
8 tasks
@danwilliams danwilliams self-assigned this Jul 11, 2019
danwilliams added a commit that referenced this issue Jul 12, 2019
@danwilliams
Rephrased source from file to data in the etl model (#989)
e7ecec8
danwilliams added a commit that referenced this issue Jul 12, 2019
@danwilliams
Uppercased the SQL keywords (#989)
e845fbc
danwilliams added a commit that referenced this issue Jul 12, 2019
@danwilliams
Added exception text matching in the postetl tests (#989)
9d40326
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danwilliams danwilliams

Labels
enhancement New feature or request FlowETL
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Tasks/989 allow postetl informative queries Flowminder/FlowKit
2 participants
@danwilliams @maxalbert