Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Airflow update #4838

Closed
wants to merge 37 commits into from
Closed

Airflow update #4838

wants to merge 37 commits into from

Conversation

Thingus
Copy link
Contributor

@Thingus Thingus commented Jan 25, 2022 •
edited
Loading

Closes #3400

I have:

  • Formatted any Python files with black
  • Brought the branch up to date with master
  • Added any relevant Github labels
  • Added tests for any new additions
  • Added or updated any relevant documentation
  • Added an Architectural Decision Record (ADR), if appropriate
  • Added an MPLv2 License Header if appropriate
  • Updated the Changelog

Description

Updates Airflow to version 2

Changes made:

  • Replace pendulum.Interval with pendulum.Duration
  • Removed or renamed any depreciated functions + decorations (esp. default decorators + third-party operators)
  • Unpinned sqlalchemy
  • Fixing tests to handle paraams returning airflow objects instead of str
  • Pinning Pandas at 1.2 (Pandas 1.4.0 needs sqlalchemy >=1.4.0, but airflow needs <1.4.0)
  • Removing redundant folder in qa_checks + adjusting jinja_env test fixture to match

Thingus added 18 commits January 24, 2022 15:01
@Thingus
Changing airflow imports
59c5591
@Thingus
Updating Pipfile in prep for airflow 2.0
73afd65
@Thingus
*Unpinning sqlalchemy*
618bca3
@Thingus
Setting template_undefined in DAG creation
b9a5069
@Thingus
Unpinned Airflow, undated pipfile and ran airflow db upgrade
af89ed6
@Thingus
Pendulum replacing Interval with Duration
8b641fc
@Thingus
Pendulum replacing Interval with Duration
77d89bf
@Thingus
More airflow import fixing
81e4372
@Thingus
Change in how you get the dag context
6c5bd3a
@Thingus
Find-test now passing (may remove extra layer in qa_check folder)
bdaebfe
@Thingus
params.get now retusn a PAram object - tests modified
e0eae46
@Thingus
Pinning dev version of Pandas to older version
880f705 
Pandas 1.4.0 needs sqlalchemy >=1.4.0, but airflow needs <1.4.0
@Thingus
Removing + renaming depreciated aiflow functions
c5c3190
@Thingus
Changing params to params.dump() in tests
aff6a55
@Thingus
Removed extra layer of qa_checks
08ea83c
@Thingus
Make internals of qa_check sets
dfcb97f 
Airflow does not like multiples of operator IDs
@Thingus
Adjust jinja_env fixture to account for move
6de4647
@Thingus
Marking depreciation test as xfail for now
1c32655
@Thingus Thingus requested a review from jc-harrison January 25, 2022 20:23
@Thingus Thingus added dependencies Pull requests that update a dependency file FlowETL maintenance labels Jan 25, 2022
@cypress
Copy link

cypress bot commented Jan 25, 2022 •
edited
Loading


Test summary

43 • 0 • 0 • 0 • Flakiness 0

Run details
Project FlowAuth
Status Passed
Commit 493f8b8
Started Jan 31, 2022 10:20 AM
Ended Jan 31, 2022 10:28 AM
Duration 07:50 💡
OS Linux Debian - 10.5
Browser Electron 94

View run in Cypress Dashboard ➡️

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@Thingus
Copy link
Contributor Author

Thingus commented Mar 10, 2022

Closing as addressed in #4940

@Thingus Thingus closed this Mar 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jc-harrison jc-harrison Awaiting requested review from jc-harrison

Assignees
No one assigned
Labels
dependencies Pull requests that update a dependency file FlowETL maintenance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Upgrade flowetl to airflow 2
1 participant
@Thingus