Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lightning: split backoff is not long enough, and fail when EBS BR disabled region split #49517

Closed
mittalrishabh opened this issue Dec 16, 2023 · 1 comment · Fixed by #49518
Closed

Lightning: split backoff is not long enough, and fail when EBS BR disabled region split #49517

mittalrishabh opened this issue Dec 16, 2023 · 1 comment · Fixed by #49518
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/lightning This issue is related to Lightning of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@mittalrishabh
Copy link
Contributor

mittalrishabh commented Dec 16, 2023 •
edited by D3Hunter
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. Run EBS BR while lightning is running. It might fail lightning if BR suspends region split for more than 1 - 2 min.

2. What did you expect to see? (Required)

Lightning to not fail if region split are suspended because of EBS BR

3. What did you see instead (Required)

Lightning failed

4. What is your TiDB version? (Required)

6.5
@mittalrishabh mittalrishabh added the type/bug The issue is confirmed as a bug. label Dec 16, 2023
@mittalrishabh mittalrishabh mentioned this issue Dec 16, 2023
Merged
13 tasks
@aytrack aytrack added severity/major component/lightning This issue is related to Lightning of TiDB. labels Dec 18, 2023
@D3Hunter
Copy link
Contributor 

failed to split: Err(Error(Kv(Error(Other(\\\"[src/server/raftkv/raft_extension.rs:128]: oneshot canceled\\\")))))\" "]

the error happens when doing EBS based backup, in which lightning don't support it

@D3Hunter D3Hunter changed the title Lightning:backoff if split and scatter fails Lightning: split backoff is not long enough, and fail when EBS BR disabled region split Dec 19, 2023
@lance6716 lance6716 added may-affects-7.5 affects-6.5 This bug affects the 6.5.x(LTS) versions. and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 labels Dec 19, 2023
ti-chi-bot bot pushed a commit that referenced this issue Jan 8, 2024
@mittalrishabh
Lightning: increase backoff if split fails (#49518)
2a564d4 
close #49517
@ti-chi-bot ti-chi-bot mentioned this issue Jan 8, 2024
Merged
13 tasks
AilinKid pushed a commit to AilinKid/tidb that referenced this issue Jan 17, 2024
@mittalrishabh @AilinKid
Lightning: increase backoff if split fails (pingcap#49518)
846fb38 
close pingcap#49517
ti-chi-bot bot pushed a commit that referenced this issue Jan 25, 2024
@ti-chi-bot
Lightning: increase backoff if split fails (#49518) (#50164)
6f6386a 
close #49517
@ti-chi-bot ti-chi-bot mentioned this issue Mar 20, 2024
Merged
13 tasks
@lance6716 lance6716 added affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. labels May 10, 2024
ti-chi-bot bot pushed a commit that referenced this issue May 10, 2024
@ti-chi-bot
Lightning: increase backoff if split fails (#49518) (#51929)
5678c38 
close #49517
@ti-chi-bot ti-chi-bot mentioned this issue Oct 28, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 11, 2024
@ti-chi-bot
Lightning: increase backoff if split fails (#49518) (#56874)
884b46e 
close #49517
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/lightning This issue is related to Lightning of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Lightning: increase backoff if split fails
4 participants
@lance6716 @D3Hunter @aytrack @mittalrishabh