Add new dbt example DAG #9
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… to permanently store the store_failures tables generated by dbt test.
Issues with BigQuery ID and dbt schema led to dropping the custom test that was designed to fail, and designing the month test to fail instead. The custom test for whatever reason was trying to use the schema.project_id as the project ID and BigQuery did NOT like that. Not sure what the fix is. Additionally, the DAG is completed by adding a BigQuery copy table operator. This copies a single table written by dbt test on failures to a permament path. This is a not-so-great, but workable, solution until AIP-42 gets merged and we can do some real dynamic task generation based on XCOM return values. In preparation for that time, BigQueryGetDatasetTablesOperator is left in commented-out.
Renamed DAGs and cleaned up bigquery version to not use certain imports and operators until AIP-42 is released. Added a snowflake version of the DAG which can also use the FFMC test. Runs dbt run then dbt test, with 2 tests designed to fail. The outputs are then loaded to a permanent table in the original schema. Modifications to the directory structure to support these two separate DAGs as their own dbt projects, as they work slightly differently.
New DAG and dbt project were created and modified to allow use of Redshift as a backend to permanently store the store_failures tables generated by dbt on test failures.
| ) | ||
|
|
||
| """ | ||
| Run dbt test suite |
There was a problem hiding this comment.
Only feedback here is that from the perspective of a new user (me), it's not clear reading this DAG what exactly the dbt test command is doing and what it's testing for. I don't know if that'd be obvious with someone with more dbt experience, but just wanted to call that out
There was a problem hiding this comment.
Gotcha. So the dbt test command runs the test suite which is specified under include/dbt/[project]/models/[model]/schema.yml. I can add this to the docstring.
virajmparekh
approved these changes
Nov 29, 2021
josh-fell
reviewed
Nov 29, 2021
josh-fell
reviewed
Nov 30, 2021
| limitations in dynamic task mapping, where needed values like 'source_table' | ||
| cannot be retrieved from Variables or other backend sources. | ||
|
|
||
| One is given as an example. |
There was a problem hiding this comment.
Is this comment needed here and in copy_store_failures_snowflake.py?
josh-fell
approved these changes
Nov 30, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add new dbt example DAG and necessary updates for dbt to showcase how to permanently store the store_failures tables generated by dbt test. The DAG comes from a question from the data quality webinar:
From dbt PR #2593: