Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AIP-84 Handle backfill for DAGs with when depends_on_past is True #45731

Merged
merged 18 commits into from
Jan 22, 2025
Merged

AIP-84 Handle backfill for DAGs with when depends_on_past is True #45731

merged 18 commits into from
Jan 22, 2025

Conversation

vatsrahul1001
Copy link
Collaborator

@vatsrahul1001 vatsrahul1001 commented Jan 17, 2025 •
edited
Loading

Currently below two scenarios results in HTTP500. This PR handles it.

  1. When depends_on_past = True is set for a Task and reprocess_behavior is "none"
    Before:
    image

    After:
    image

  2. When depends_on_past = True is set for a Task and run_backwards=True
    Before:
    image

    After:
    image

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@vatsrahul1001 vatsrahul1001 marked this pull request as draft January 17, 2025 06:33
@vatsrahul1001 vatsrahul1001 marked this pull request as ready for review January 17, 2025 08:11
pierrejeambrun
pierrejeambrun reviewed Jan 20, 2025
View reviewed changes
Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Just a few nitpicks.

@pierrejeambrun
Copy link
Member

cc: @dstandish

@pierrejeambrun
Copy link
Member

Don't hesitate to resolve comments that you addressed. Also I think a couple of them are marked as done but I don't see the related change yet. (maybe you have some local unpushed code ?)

pierrejeambrun
pierrejeambrun approved these changes Jan 21, 2025
View reviewed changes
Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small improvement. Can merge after.

pierrejeambrun
pierrejeambrun requested changes Jan 22, 2025
View reviewed changes
Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to not merge by mistake.

@pierrejeambrun pierrejeambrun changed the title Handle backfill for DAGs with when depends_on_past is True AIP-84 Handle backfill for DAGs with when depends_on_past is True Jan 22, 2025
@pierrejeambrun pierrejeambrun added the AIP-84 Modern Rest API label Jan 22, 2025
@pierrejeambrun pierrejeambrun merged commit 66b1712 into apache:main Jan 22, 2025
45 checks passed
@pierrejeambrun pierrejeambrun deleted the backfill-handle-depends-on-past branch January 22, 2025 13:47
dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
@vatsrahul1001 @dauinh
AIP-84 Handle backfill for DAGs with when depends_on_past is True (ap…
c9925da 
…ache#45731)

* backfill handle depends on past

* fixing backfill model tests

* Implementing review comments

* fix test

* uncomment test commneted incorrectly

* implement review comments

* handle exceptions better

* fixing exceptin handling and tests

* remove 422 from router
@utkarsharma2 utkarsharma2 added the type:new-feature Changelog: New Features label Jan 27, 2025
@utkarsharma2 utkarsharma2 added this to the Airflow 3.0.0 milestone Jan 27, 2025
gpathak128 pushed a commit to gpathak128/airflow that referenced this pull request Jan 29, 2025
@vatsrahul1001
AIP-84 Handle backfill for DAGs with when depends_on_past is True (ap…
6ab692a 
…ache#45731)

* backfill handle depends on past

* fixing backfill model tests

* Implementing review comments

* fix test

* uncomment test commneted incorrectly

* implement review comments

* handle exceptions better

* fixing exceptin handling and tests

* remove 422 from router
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
@vatsrahul1001 @got686-yandex
AIP-84 Handle backfill for DAGs with when depends_on_past is True (ap…
5a4d2a1 
…ache#45731)

* backfill handle depends on past

* fixing backfill model tests

* Implementing review comments

* fix test

* uncomment test commneted incorrectly

* implement review comments

* handle exceptions better

* fixing exceptin handling and tests

* remove 422 from router
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
@vatsrahul1001 @niklasr22
AIP-84 Handle backfill for DAGs with when depends_on_past is True (ap…
5ca3689 
…ache#45731)

* backfill handle depends on past

* fixing backfill model tests

* Implementing review comments

* fix test

* uncomment test commneted incorrectly

* implement review comments

* handle exceptions better

* fixing exceptin handling and tests

* remove 422 from router
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pierrejeambrun pierrejeambrun pierrejeambrun approved these changes

@dstandish dstandish dstandish approved these changes

@jason810496 jason810496 jason810496 approved these changes

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@bbovenzi bbovenzi Awaiting requested review from bbovenzi

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton

@jscheffl jscheffl Awaiting requested review from jscheffl

Assignees
No one assigned
Labels
AIP-84 Modern Rest API type:new-feature Changelog: New Features
Projects
None yet
Milestone
Airflow 3.0.0
Development

Successfully merging this pull request may close these issues.
5 participants
@vatsrahul1001 @pierrejeambrun @dstandish @jason810496 @utkarsharma2