[SDP-1397] Circle Payouts pt2: error handling & retries #491

marcelosalloum · 2024-12-11T00:12:05Z

What

Errors in the POST /recipient endpoint are not being properly handled. What;s missing?

When POST /recipient fails with ErrCircleRecipientCreationFailedTooManyTimes, we update the payments table with the failed status explaining the reason.
When a payment that failed due to recipient creation issues is retried, the recipient is reset with status=NULL and sync_attempts=0.
The method ensureRecipientIsReady was split into different sub-methods for better modularity and clarity, as suggested in [SDP-1397] Circle payouts pt1 #486 (comment).

Tested with scheduled jobs
Tested with Kafka

Why

To wrap up https://stellarorg.atlassian.net/browse/SDP-1397

Checklist

PR Structure

This PR has a reasonably narrow scope (if not, break it down into smaller PRs).
This PR title and description are clear enough for anyone to review it.
This PR does not mix refactoring changes with feature changes (split into two PRs otherwise).

Thoroughness

This PR adds tests for the new functionality or fixes.
This PR contains the link to the Jira ticket it addresses.

Configs and Secrets

No new CONFIG variables are required -OR- the new required ones were added to the helmchart's values.yaml file.
No new CONFIG variables are required -OR- the new required ones were added to the deployments (pr-preview, dev, demo, prd).
No new SECRETS variables are required -OR- the new required ones were mentioned in the helmchart's values.yaml file.
No new SECRETS variables are required -OR- the new required ones were added to the deployments (pr-preview secrets, dev secrets, demo secrets, prd secrets).

Release

This is not a breaking change.
This is ready for production.. If your PR is not ready for production, please consider opening additional complementary PRs using this one as the base. Only merge this into develop or main after it's ready for production!

Deployment

Does the deployment work after merging?

stellar-jenkins · 2024-12-11T00:16:07Z

stellar-disbursement-platform-backend-preview is available here:
SDP: https://sdp-backend-pr491.previews.kube001.services.stellar-ops.com/health
AP: https://sdp-ap-pr491.previews.kube001.services.stellar-ops.com/health
Frontend: https://sdp-backend-dashboard-pr491.previews.kube001.services.stellar-ops.com

stellar-jenkins · 2024-12-12T21:17:51Z

stellar-disbursement-platform-backend-preview is available here:
SDP: https://sdp-backend-pr491.previews.kube001.services.stellar-ops.com/health
AP: https://sdp-ap-pr491.previews.kube001.services.stellar-ops.com/health
Frontend: https://sdp-backend-dashboard-pr491.previews.kube001.services.stellar-ops.com

stellar-jenkins · 2024-12-12T22:54:10Z

stellar-disbursement-platform-backend-preview is available here:
SDP: https://sdp-backend-pr491.previews.kube001.services.stellar-ops.com/health
AP: https://sdp-ap-pr491.previews.kube001.services.stellar-ops.com/health
Frontend: https://sdp-backend-dashboard-pr491.previews.kube001.services.stellar-ops.com

stellar-jenkins · 2024-12-13T00:34:23Z

stellar-disbursement-platform-backend-preview is available here:
SDP: https://sdp-backend-pr491.previews.kube001.services.stellar-ops.com/health
AP: https://sdp-ap-pr491.previews.kube001.services.stellar-ops.com/health
Frontend: https://sdp-backend-dashboard-pr491.previews.kube001.services.stellar-ops.com

stellar-jenkins · 2024-12-13T00:37:54Z

stellar-disbursement-platform-backend-preview is available here:
SDP: https://sdp-backend-pr491.previews.kube001.services.stellar-ops.com/health
AP: https://sdp-ap-pr491.previews.kube001.services.stellar-ops.com/health
Frontend: https://sdp-backend-dashboard-pr491.previews.kube001.services.stellar-ops.com

marwen-abid

LGTM! This is great!

marwen-abid · 2024-12-13T00:58:06Z

internal/serve/httphandler/payments_handler.go

+			_, err = p.Models.CircleRecipient.ResetRecipientsForRetryIfNeeded(ctx, dbTx, reqBody.PaymentIDs...)
+			if err != nil {
+				return nil, fmt.Errorf("resetting circle recipients for retry if needed: %w", err)
+			}


nit: Totally optional as it's not an expensive operation. But you can also add a check to only do this operation for Circle distribution accounts.

distAccount, err := p.DistributionAccountResolver.DistributionAccountFromContext(ctx) if err != nil { return nil, fmt.Errorf("resolving distribution account: %w", err) } if distAccount.IsCircle() { _, err = p.Models.CircleRecipient.ResetRecipientsForRetryIfNeeded(ctx, dbTx, reqBody.PaymentIDs...) if err != nil { return nil, fmt.Errorf("resetting circle recipients for retry if needed: %w", err) } }

Updated in 83469c5.

internal/services/paymentdispatchers/circle_payment_dispatcher.go

…tore the error in the response_body column.

stellar-jenkins · 2024-12-13T18:26:06Z

stellar-disbursement-platform-backend-preview is available here:
SDP: https://sdp-backend-pr491.previews.kube001.services.stellar-ops.com/health
AP: https://sdp-ap-pr491.previews.kube001.services.stellar-ops.com/health
Frontend: https://sdp-backend-dashboard-pr491.previews.kube001.services.stellar-ops.com

stellar-jenkins · 2024-12-13T18:56:48Z

stellar-disbursement-platform-backend-preview is available here:
SDP: https://sdp-backend-pr491.previews.kube001.services.stellar-ops.com/health
AP: https://sdp-ap-pr491.previews.kube001.services.stellar-ops.com/health
Frontend: https://sdp-backend-dashboard-pr491.previews.kube001.services.stellar-ops.com

stellar-jenkins · 2024-12-13T18:57:50Z

Something went wrong with PR preview build please check

stellar-jenkins · 2024-12-13T19:09:47Z

stellar-disbursement-platform-backend-preview is available here:
SDP: https://sdp-backend-pr491.previews.kube001.services.stellar-ops.com/health
AP: https://sdp-ap-pr491.previews.kube001.services.stellar-ops.com/health
Frontend: https://sdp-backend-dashboard-pr491.previews.kube001.services.stellar-ops.com

marcelosalloum self-assigned this Dec 11, 2024

marcelosalloum temporarily deployed to Anchor Integration Tests December 11, 2024 00:12 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 11, 2024 00:12 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Anchor Integration Tests December 12, 2024 21:14 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 12, 2024 21:14 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Anchor Integration Tests December 12, 2024 22:51 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 12, 2024 22:51 — with GitHub Actions Inactive

marcelosalloum force-pushed the feat/SDP-1397-pt2-retry-recipients branch from a1d17ec to e43fafa Compare December 13, 2024 00:31

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 13, 2024 00:31 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Anchor Integration Tests December 13, 2024 00:31 — with GitHub Actions Inactive

marcelosalloum added 7 commits December 12, 2024 16:33

Start handling ErrCircleRecipientCreationFailedTooManyTimes

83d9510

Update tests to account for the latest code changes.

6856f14

Add unit tests for ensureRecipientIsReadyWithRetry.

29e0157

Slightly refactor sendPaymentsToCircle.

cb47f20

Fix unit tests.

105e3e8

Split logic from ensureRecipientIsReady as suggested in #486 (comment)

3f5bed8

Handle CircleRecipients for payment retries.

f15f82b

marcelosalloum force-pushed the feat/SDP-1397-pt2-retry-recipients branch from e43fafa to f15f82b Compare December 13, 2024 00:34

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 13, 2024 00:34 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Anchor Integration Tests December 13, 2024 00:34 — with GitHub Actions Inactive

marcelosalloum marked this pull request as ready for review December 13, 2024 00:40

marcelosalloum requested a review from marwen-abid December 13, 2024 00:45

marcelosalloum changed the title ~~[SDP-1397] Circle Payouts error handling~~ [SDP-1397] Circle Payouts pt2: error handling & retries Dec 13, 2024

marwen-abid approved these changes Dec 13, 2024

View reviewed changes

Create a constant for the initial backoff duration.

cbc5c18

marcelosalloum added 2 commits December 13, 2024 10:15

Remove TODO.

67f1b4d

When PostRecipient fails inside the submitRecipientToCircle method, s…

5722deb

…tore the error in the response_body column.

marcelosalloum temporarily deployed to Anchor Integration Tests December 13, 2024 18:23 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 13, 2024 18:23 — with GitHub Actions Inactive

Only call ResetRecipientsForRetryIfNeeded for Circle tenants.

83469c5

marcelosalloum temporarily deployed to Anchor Integration Tests December 13, 2024 18:52 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 13, 2024 18:52 — with GitHub Actions Inactive

Create a constant for the initial backoff duration. [2]

7ea368e

marcelosalloum temporarily deployed to Anchor Integration Tests December 13, 2024 18:53 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 13, 2024 18:53 — with GitHub Actions Inactive

Empty commit to trigger CI

a26c537

marcelosalloum temporarily deployed to Anchor Integration Tests December 13, 2024 19:07 — with GitHub Actions Inactive

marcelosalloum temporarily deployed to Receiver Registration - E2E Integration Tests (Stellar) December 13, 2024 19:07 — with GitHub Actions Inactive

marcelosalloum merged commit f9e2520 into develop Dec 13, 2024
13 checks passed

marcelosalloum deleted the feat/SDP-1397-pt2-retry-recipients branch December 13, 2024 19:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SDP-1397] Circle Payouts pt2: error handling & retries #491

[SDP-1397] Circle Payouts pt2: error handling & retries #491

marcelosalloum commented Dec 11, 2024 •

edited

Loading

stellar-jenkins commented Dec 11, 2024

stellar-jenkins commented Dec 12, 2024

stellar-jenkins commented Dec 12, 2024

stellar-jenkins commented Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

marwen-abid left a comment

marwen-abid Dec 13, 2024

marcelosalloum Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

[SDP-1397] Circle Payouts pt2: error handling & retries #491

[SDP-1397] Circle Payouts pt2: error handling & retries #491

Conversation

marcelosalloum commented Dec 11, 2024 • edited Loading

What

Why

Checklist

PR Structure

Thoroughness

Configs and Secrets

Release

Deployment

stellar-jenkins commented Dec 11, 2024

stellar-jenkins commented Dec 12, 2024

stellar-jenkins commented Dec 12, 2024

stellar-jenkins commented Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

marwen-abid left a comment

Choose a reason for hiding this comment

marwen-abid Dec 13, 2024

Choose a reason for hiding this comment

marcelosalloum Dec 13, 2024

Choose a reason for hiding this comment

stellar-jenkins commented Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

stellar-jenkins commented Dec 13, 2024

marcelosalloum commented Dec 11, 2024 •

edited

Loading