Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Break up task edge registration #238

Merged
merged 10 commits into from
Apr 6, 2021
Merged

Conversation

zangell44
Copy link
Contributor

@zangell44 zangell44 commented Apr 5, 2021

Summary

This PR batches Task and Edge creation when creating a Flow.

Importance

For very large flows, we're hitting a limit on Postgres query size when trying to create the full Flow model. See this Slack thread for a full discussion https://prefect-community.slack.com/archives/C014Z8DPDSR/p1617128789186300.

I think eventually we'll want to do this batching on the client side, but this PR along with an update to speed up flow validation will allow registration of fairly large flows for the time being.

Checklist

This PR:

  • adds new tests (if appropriate)
  • adds a change file in the changes/ directory (if appropriate)

@zangell44 zangell44 requested review from zanieb and cicdw April 5, 2021 16:45
@zangell44 zangell44 requested a review from jlowin as a code owner April 5, 2021 16:45
@github-actions github-actions bot added the API label Apr 5, 2021
Copy link
Contributor

@zanieb zanieb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good sleuthing!

src/prefect_server/api/flows.py Outdated Show resolved Hide resolved
src/prefect_server/utilities/utils.py Outdated Show resolved Hide resolved
tests/api/test_flows.py Outdated Show resolved Hide resolved
tests/api/test_flows.py Outdated Show resolved Hide resolved
changes/pr238.yaml Outdated Show resolved Hide resolved
Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work digging into the subtleties here both client and server side!

src/prefect_server/api/flows.py Outdated Show resolved Hide resolved
src/prefect_server/api/flows.py Outdated Show resolved Hide resolved
tests/api/test_flows.py Outdated Show resolved Hide resolved
@codecov-io
Copy link

codecov-io commented Apr 6, 2021

Codecov Report

Merging #238 (9b452d4) into master (23cc8e7) will increase coverage by 0.08%.
The diff coverage is 100.00%.

@zangell44 zangell44 requested review from cicdw and zanieb April 6, 2021 01:25
@zangell44
Copy link
Contributor Author

All feedback has been addressed, albeit likely imperfectly

Would be good to get another set of eyes on the exception handling here https://github.com/PrefectHQ/server/pull/238/files#diff-23db4a75ea24edcd454c5f66ca3e1bf042862bc3713786a8924acd5e06cc256dR320. Slightly confused on what the best pattern for this would be, but logging the error and raising a custom exception from the original seemed somewhat reasonable

Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some really minor comments, overall this is looking good to me

src/prefect_server/api/flows.py Outdated Show resolved Hide resolved
tests/api/test_flows.py Show resolved Hide resolved
@zangell44 zangell44 requested review from cicdw and zanieb April 6, 2021 19:52
@zanieb zanieb merged commit 2b7c996 into master Apr 6, 2021
@zanieb zanieb deleted the break-up-task-edge-registration branch April 6, 2021 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants