Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backupccl: retry restore's AdminSplit on ephemeral errors #85333

Closed
wants to merge 1 commit into from

Conversation

msbutler
Copy link
Collaborator

Informs #84635,#84162

Release note: none

@msbutler msbutler self-assigned this Jul 29, 2022
@msbutler msbutler requested review from a team and rhu713 July 29, 2022 17:31
@cockroach-teamcity
Copy link
Member

This change is Reviewable

if err == nil {
break
}
err := s.db.AdminSplit(ctx, newSplitKey, expirationTime)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth moving this retry logic one level deeper? Maybe we want to ask KV but don't we have a lot of callers outside restore that use db.AdminSplit as well? import, index backfills etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or atleast auditing all the places we call db.AdminSplit and deciding whether to switch them to this wrapper method.

Copy link
Collaborator Author

@msbutler msbutler Jul 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I don't see why the caller needs to handle these replication level ephemeral retries. Lemme ask kv what they think in slack.

@msbutler msbutler added T-kv KV Team and removed T-kv KV Team labels Jul 29, 2022
Copy link
Contributor

@rhu713 rhu713 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

}
err := s.db.AdminSplit(ctx, newSplitKey, expirationTime)

if !(kvserver.IsRetriableReplicationChangeError(err)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need to add IsLeaseTransferRejectedBecauseTargetMayNeedSnapshotError() here or inside some lower level e.g. IsRetriableReplicationChangeError() as this error is what #84891 is failing with.

@msbutler
Copy link
Collaborator Author

msbutler commented Aug 1, 2022

Closing, as kvserver will handle this internally db2d653

@msbutler msbutler closed this Aug 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants