Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for timeout to BatchOperator #45660

Merged

Conversation

nrobinson-intelycare
Copy link
Contributor

Redo of #45619 to avoid merge commit


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@nrobinson-intelycare
Copy link
Contributor Author

@o-nikolas @vincbeck @eladkal

Anything I'm missing blocking this from being merged?

@vincbeck
Copy link
Contributor

@o-nikolas @vincbeck @eladkal

Anything I'm missing blocking this from being merged?

Just curious, have you tested it manually when submit_job_timeout is not provided? In that case timeout is passed to the the api submit_job with value {"attemptDurationSeconds": None}. I am wondering it Boto3 is okay with that

@nrobinson-intelycare
Copy link
Contributor Author

nrobinson-intelycare commented Jan 15, 2025

@o-nikolas @vincbeck @eladkal
Anything I'm missing blocking this from being merged?

Just curious, have you tested it manually when submit_job_timeout is not provided? In that case timeout is passed to the the api submit_job with value {"attemptDurationSeconds": None}. I am wondering it Boto3 is okay with that

I've changed it to only add timeout to the args dict if submit_job_timeout is not None:

https://github.com/apache/airflow/pull/45660/files#diff-730ff9a9f5255ee440584231fc8733730c53a6a1a3fa3c2edd993c9d5157f219R322

@vincbeck
Copy link
Contributor

@o-nikolas @vincbeck @eladkal
Anything I'm missing blocking this from being merged?

Just curious, have you tested it manually when submit_job_timeout is not provided? In that case timeout is passed to the the api submit_job with value {"attemptDurationSeconds": None}. I am wondering it Boto3 is okay with that

I've changed it to only add timeout to the args dict if submit_job_timeout is not None:

https://github.com/apache/airflow/pull/45660/files#diff-730ff9a9f5255ee440584231fc8733730c53a6a1a3fa3c2edd993c9d5157f219R322

I missed that! LGTM :)

@o-nikolas o-nikolas merged commit caa401d into apache:main Jan 17, 2025
61 checks passed
Copy link

boring-cyborg bot commented Jan 17, 2025

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@nrobinson-intelycare
Copy link
Contributor Author

@o-nikolas @vincbeck

Any idea when a new release of apache-airflow-providers-amazon will be made? Looking forward to using this.

@vincbeck
Copy link
Contributor

Usually releases are done every 2 weeks but @eladkal should be able to give you more details about the next wave.

@eladkal
Copy link
Contributor

eladkal commented Jan 22, 2025

Will have to wait after Celery release.
I don't have time to parallel releases. Sorry.

@nrobinson-intelycare
Copy link
Contributor Author

Will have to wait after Celery release. I don't have time to parallel releases. Sorry.

No worries, thanks!

dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
An execution timeout for the submit_job api call can now be passed through the operator to the boto3 call.
@nrobinson-intelycare
Copy link
Contributor Author

@eladkal Any chance of a release this week? Thanks.

got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
An execution timeout for the submit_job api call can now be passed through the operator to the boto3 call.
@nrobinson-intelycare
Copy link
Contributor Author

@eladkal @vincbeck Any plans for making a release soon?

niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
An execution timeout for the submit_job api call can now be passed through the operator to the boto3 call.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:amazon AWS/Amazon - related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants