Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: retryStrategy.Limit is now read properly for backoff strategy. Fixes #9170. #9213

Merged
merged 6 commits into from
Aug 1, 2022

Conversation

dpadhiar
Copy link
Member

@dpadhiar dpadhiar commented Jul 22, 2022

Signed-off-by: Dillen Padhiar [email protected]

Fixes #9170

Previously, this if statement always evaluated to false if you set a Limit for RetryStrategy as the retryStrategy.Limit.IntVal always equaled 0. The appropriate way to get the integer value from a intstr.IntOrString is to use IntValue which returns an int we cast to int32.

if time.Now().Before(waitingDeadline) && retryStrategy.Limit != nil && int32(len(node.Children)) <= 
    int32(retryStrategy.Limit.IntValue()) {
    woc.requeueAfter(timeToWait)
    retryMessage := fmt.Sprintf("Backoff for %s", humanize.Duration(timeToWait))
    return woc.markNodePhase(node.Name, node.Phase, retryMessage), false, nil
}

@dpadhiar dpadhiar marked this pull request as ready for review July 25, 2022 21:20
Then().
ExpectWorkflow(func(t *testing.T, _ *metav1.ObjectMeta, status *wfv1.WorkflowStatus) {
assert.Equal(t, wfv1.WorkflowPhase("Failed"), status.Phase)
assert.Equal(t, "Max duration limit exceeded", status.Message)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to validate the number of retries since the issue is "string" limit does not get interpret correctly

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, we could check the length of child nodes at the end of the run which should be 5 based on the backoff time.

@@ -898,7 +898,7 @@ func (woc *wfOperationCtx) processNodeRetries(node *wfv1.NodeStatus, retryStrate
}

// See if we have waited past the deadline
if time.Now().Before(waitingDeadline) && retryStrategy.Limit != nil && int32(len(node.Children)) <= retryStrategy.Limit.IntVal {
if time.Now().Before(waitingDeadline) && retryStrategy.Limit != nil && int32(len(node.Children)) <= int32(retryStrategy.Limit.IntValue()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@dpadhiar dpadhiar requested a review from tczhao July 27, 2022 16:03
@sarabala1979 sarabala1979 self-assigned this Jul 27, 2022
templates:
- name: main
retryStrategy:
limit: 10
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've pointed this out last time, since the error is with the intstring conversion maybe we should test both '10' and 10

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add a separate workflow to test this, good idea.

@dpadhiar dpadhiar requested a review from tczhao July 28, 2022 23:19
@sarabala1979 sarabala1979 merged commit 2d1758f into argoproj:master Aug 1, 2022
@sarabala1979 sarabala1979 mentioned this pull request Aug 2, 2022
51 tasks
terrytangyuan pushed a commit that referenced this pull request Aug 2, 2022
…ixes #9170. (#9213)

* fix: retryStrategy.Limit is now read properly

Signed-off-by: Dillen Padhiar <[email protected]>

* test: add test case for backoff

Signed-off-by: Dillen Padhiar <[email protected]>

* chore: empty commit

Signed-off-by: Dillen Padhiar <[email protected]>

* test: update test case to check for certain amount of retries

Signed-off-by: Dillen Padhiar <[email protected]>

* test: added second case where limit is a string

Signed-off-by: Dillen Padhiar <[email protected]>
@tanken
Copy link

tanken commented Oct 27, 2022

am on v3.3.9 but backoff doesnt seem to apply any delays between retries

juchaosong pushed a commit to juchaosong/argo-workflows that referenced this pull request Nov 3, 2022
…ixes argoproj#9170. (argoproj#9213)

* fix: retryStrategy.Limit is now read properly

Signed-off-by: Dillen Padhiar <[email protected]>

* test: add test case for backoff

Signed-off-by: Dillen Padhiar <[email protected]>

* chore: empty commit

Signed-off-by: Dillen Padhiar <[email protected]>

* test: update test case to check for certain amount of retries

Signed-off-by: Dillen Padhiar <[email protected]>

* test: added second case where limit is a string

Signed-off-by: Dillen Padhiar <[email protected]>
Signed-off-by: juchao <[email protected]>
reddymh pushed a commit to reddymh/argo-workflows that referenced this pull request Jan 2, 2023
…ixes argoproj#9170. (argoproj#9213)

* fix: retryStrategy.Limit is now read properly

Signed-off-by: Dillen Padhiar <[email protected]>

* test: add test case for backoff

Signed-off-by: Dillen Padhiar <[email protected]>

* chore: empty commit

Signed-off-by: Dillen Padhiar <[email protected]>

* test: update test case to check for certain amount of retries

Signed-off-by: Dillen Padhiar <[email protected]>

* test: added second case where limit is a string

Signed-off-by: Dillen Padhiar <[email protected]>
Signed-off-by: Reddy <[email protected]>
@agilgur5 agilgur5 added the area/retryStrategy Template-level retryStrategy label Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/retryStrategy Template-level retryStrategy
Projects
None yet
Development

Successfully merging this pull request may close these issues.

backoff under retryStrategy does not work as expected for version 3.3.8
5 participants