AIP-72: Task SDK support for on_task_instance_* listeners, make OpenLineage compatible #45294
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:Executors-core
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
4 times, most recently
from
f07a78e to
81b886b
Compare
|
@mobuchowski - can you please rebase that one -> we found and issue with @jscheffl with the new caching scheme - fixed in #45347 that would run "main" version of the tests. I am asking in all affected PRs to rebase. |
potiuk
force-pushed
the
tasksdk-call-listeners
branch
from
81b886b to
15fd95a
Compare
|
Actually - I rebased it now. |
|
I will rebase very soon as I'm working on some of the test failures anyway 🙂 |
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
7 times, most recently
from
35a94e2 to
86297af
Compare
mobuchowski
requested review from
ephraimbuddy,
pierrejeambrun,
jedcunningham,
XD-DENG,
ashb,
o-nikolas and
hussein-awala
as code owners
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
from
86297af to
6d88389
Compare
mobuchowski
changed the title
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
4 times, most recently
from
c2e2165 to
305fff0
Compare
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
8 times, most recently
from
3dc5ae4 to
39172c3
Compare
|
@mobuchowski You had also mentioned that you would be raising another PR once #45732 was merged. Since that is already merged, is that follow-up PR already in progress. Thank you! |
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
from
39172c3 to
0705a85
Compare
… TaskSDK, make OpenLineage provider support Airflow 3's listener interface Signed-off-by: Maciej Obuchowski <[email protected]>
mobuchowski
force-pushed
the
tasksdk-call-listeners
branch
from
0705a85 to
61da140
Compare
|
Hey @vikramkoka - rebased this PR and fixed conflicts, it should be good to merge. Haven't started on followup to #45732 but will try to find some time next week. I don't think we need to hold up this PR for that, especially since rebasing 2000 lines PR on fast moving target takes significant amount of time. |
uranusjr
reviewed
Feb 11, 2025
uranusjr
approved these changes
Feb 11, 2025
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Feb 12, 2025
So far cleaning airflow installation only happened in canary runs and it caused some PRs not failing when they should - for exmaple the apache#45294 was green when it should fail because uuid6 package was not removed before installing old version of Airlfow. Cleaning airflow installation is fast with uv so we should be ok with running it always for compatibility tests.
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Feb 12, 2025
So far cleaning airflow installation only happened in canary runs and it caused some PRs not failing when they should - for exmaple the apache#45294 was green when it should fail because uuid6 package was not removed before installing old version of Airlfow. Cleaning airflow installation is fast with uv so we should be ok with running it always for compatibility tests.
potiuk
added a commit
that referenced
this pull request
Feb 12, 2025
…46693) So far cleaning airflow installation only happened in canary runs and it caused some PRs not failing when they should - for exmaple the #45294 was green when it should fail because uuid6 package was not removed before installing old version of Airlfow. Cleaning airflow installation is fast with uv so we should be ok with running it always for compatibility tests.
ambika-garg
pushed a commit
to ambika-garg/airflow
that referenced
this pull request
Feb 17, 2025
… TaskSDK, make OpenLineage provider support Airflow 3's listener interface (apache#45294) Signed-off-by: Maciej Obuchowski <[email protected]>
ambika-garg
pushed a commit
to ambika-garg/airflow
that referenced
this pull request
Feb 17, 2025
…pache#46693) So far cleaning airflow installation only happened in canary runs and it caused some PRs not failing when they should - for exmaple the apache#45294 was green when it should fail because uuid6 package was not removed before installing old version of Airlfow. Cleaning airflow installation is fast with uv so we should be ok with running it always for compatibility tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With AIP-72, there is no access to the database session from the worker process, and the runtime objects have some differences to the db models. This PR contains three commits that deal with that situation:
on_task_instance_*listeners interface to AIP-72: dropssessionargument and makestask_instanceargument an instance ofRuntimeTaskInstanceclass, not database modelSome followup work:
Activityto make logging better visible from UI, and distinct from task logsRuntimeTaskInstance)closes #45423