Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backup: fix retry of fine-grained backup #43252

Merged
merged 5 commits into from
Apr 21, 2023

Conversation

YuJuncen
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #43236

Problem Summary:
We are using the tikv.Backoffer to implement retry. Which isn't so customizable hence we cannot fully control the printed error messages and back off duration.

What is changed and how it works?

This PR added a new structure RetryWithBackoffer to adapt the retry logic with the TiDB backoffer logic, at the same time it has enhanced the error message.
This PR will also increase the max retry time of fine-grained backup. The reason is simple: given it begins the fine-grained backup, there must be some problems in the cluster. We need to be more patient.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Fixed a bug that may cause BR report false "resolve lock timeout" error.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Apr 20, 2023

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Leavrth
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 20, 2023
@YuJuncen YuJuncen changed the title backup: fix fine-grained retry backup: fix retry of fine-grained backup Apr 20, 2023
@YuJuncen YuJuncen requested review from 3pointer and Leavrth April 20, 2023 09:52
@YuJuncen YuJuncen requested a review from overvenus April 21, 2023 09:29
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 21, 2023

// TotalSleepInMS returns the total sleeped time in ms.
func (r *RetryWithBackoffer) TotalSleepInMS() int {
return r.totalBackoff + r.bo.GetTotalSleep()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the r.bo is always forked, and only the forked bos are updated something. So is r.bo.GetTotalSleep always 0?

Copy link
Contributor Author

@YuJuncen YuJuncen Apr 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah indeed, but the origin logic is this. I'm not sure why it was implemented as this(and have no idea about how to improve that), so I just inherited that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway unlocking locks has a bounded backoff time, which is the TTL of the lock. I think omit the retry time it cost won't be really harmful, hopefully...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sole difference between the original logic is that resolve lock may fail due to the accumulated sleep time reaches the sleep time limit. I think that would be fine to eliminate that fail case.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 21, 2023
@YuJuncen
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 886bc2b

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 21, 2023
@ti-chi-bot ti-chi-bot merged commit 1a94e5d into pingcap:master Apr 21, 2023
@YuJuncen YuJuncen added needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. labels Apr 22, 2023
@YuJuncen
Copy link
Contributor Author

/run-cherry-picker

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.1: #43302.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-5.3: #43303.

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 22, 2023
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-5.4: #43304.

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 22, 2023
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #43305.

ti-chi-bot bot pushed a commit that referenced this pull request Apr 23, 2023
* cherry-pick 1a94e5d

Signed-off-by: hillium <[email protected]>

* run gofmt

Signed-off-by: hillium <[email protected]>

* fix a typo

Signed-off-by: hillium <[email protected]>

---------

Signed-off-by: hillium <[email protected]>
@YuJuncen YuJuncen added the needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. label Apr 28, 2023
@YuJuncen
Copy link
Contributor Author

/run-cherry-picker

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #43487.

ti-chi-bot bot pushed a commit that referenced this pull request May 8, 2023
YuJuncen added a commit to ti-chi-bot/tidb that referenced this pull request May 23, 2023
ti-chi-bot bot pushed a commit that referenced this pull request May 23, 2023
ti-chi-bot bot pushed a commit that referenced this pull request Jun 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

backup: the error message of fine-grained backup is misleading
4 participants