Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make flux check optional in FlowETL dags #3603

Closed
jc-harrison opened this issue Feb 5, 2021 · 0 comments · Fixed by #3604
Closed

Make flux check optional in FlowETL dags #3603

jc-harrison opened this issue Feb 5, 2021 · 0 comments · Fixed by #3604
Labels

Comments

@jc-harrison
Copy link
Member

There are some situations where the flux check task in a FlowETL dag is unnecessary (e.g. a two-stage ingestion process where data from a multiple-day file are ingested into a staging table, and then daily dag runs extract from that staging table - a flux check is needed in the first stage, but the second-stage daily runs do not need to check for flux again). Since the table flux check can be quite slow, create_dag should provide an option to disable this check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant