-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support to .remote for client retries #2403
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rculbertson
force-pushed
the
ryan/retries
branch
from
October 25, 2024 01:09
9dcc47f
to
7d32de2
Compare
rculbertson
changed the title
[wip] Add support for client retries
[wip] Add support to .remote for client retries
Oct 25, 2024
rculbertson
force-pushed
the
ryan/retries
branch
2 times, most recently
from
October 28, 2024 19:17
b9d4ac7
to
0e4c9e1
Compare
rculbertson
force-pushed
the
ryan/retries
branch
3 times, most recently
from
November 6, 2024 18:11
eba18f5
to
e4d7fb8
Compare
gongy
reviewed
Nov 6, 2024
rculbertson
force-pushed
the
ryan/retries
branch
7 times, most recently
from
November 13, 2024 20:17
f7d8b2b
to
e09464e
Compare
rculbertson
force-pushed
the
ryan/retries
branch
from
November 14, 2024 01:44
9429b04
to
bc1cdd8
Compare
rculbertson
changed the title
[wip] Add support to .remote for client retries
Add support to .remote for client retries
Nov 14, 2024
rculbertson
force-pushed
the
ryan/retries
branch
3 times, most recently
from
November 14, 2024 22:34
1a637d8
to
5b2bc03
Compare
freider
reviewed
Nov 21, 2024
rculbertson
force-pushed
the
ryan/retries
branch
from
November 21, 2024 20:08
5b2bc03
to
918bdb5
Compare
rculbertson
force-pushed
the
ryan/retries
branch
from
November 25, 2024 22:28
315be36
to
853368b
Compare
rculbertson
force-pushed
the
ryan/retries
branch
from
November 26, 2024 18:24
853368b
to
9d5207e
Compare
2 tasks
gongy
approved these changes
Nov 26, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for taking a while. This is fantastic work @rculbertson and @rohansingh.
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes SVC-179.
Adds support for retrying failures in client rather than server. This functionality is disabled by default, but can be enabled by setting the env var
MODAL_CLIENT_RETRIES
. Hiding it behind this feature flag allows us to merge the changes into main, so we can test it in production with high frequency monitoring, before exposing it to users.The basic flow is this:
FunctionMap
, which returns retry policy, a JWT for the function call, and a JWT per pipelined inputFunctionPutInputs
, which returns JWT per input.FunctionGetOutputs
, which returns outputa. If failure and a retry policy is specified: client calls new
RetryInput
endpoint passing input id, input payload, and JWTs from earlier for authentication. This will loop according to policyb. If failure and no retry policy is specified: exception is raised
c. If success, return output to caller.