Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

for large import task, lightning needs to improve region split/scatter/check #43510

Closed
Tracked by #43740
lance6716 opened this issue May 4, 2023 · 0 comments · Fixed by #43807
Closed
Tracked by #43740

for large import task, lightning needs to improve region split/scatter/check #43510

lance6716 opened this issue May 4, 2023 · 0 comments · Fixed by #43807
Labels
affects-7.1 This bug affects the 7.1.x(LTS) versions. type/enhancement The issue or PR belongs to an enhancement.

Comments

@lance6716
Copy link
Contributor

Enhancement

we meet a problem that for a 10 lightning parallel import task, region actions are slow so lightning exits with error.

there are two cases at least

  • scattering region timed out and lightning continues to import. And due to region are imbalanced some TiKV nodes run out of disk
  • checking region continuous timed out with message "region endKey not equal to next region startKey" and lightning exited
@lance6716 lance6716 added the type/enhancement The issue or PR belongs to an enhancement. label May 4, 2023
@lance6716 lance6716 mentioned this issue May 4, 2023
Closed
12 tasks
This was referenced May 12, 2023
Open
Merged
@ti-chi-bot ti-chi-bot bot closed this as completed in 6e5604f May 17, 2023
@lance6716 lance6716 added the affects-7.1 This bug affects the 7.1.x(LTS) versions. label May 17, 2023
@ti-chi-bot ti-chi-bot mentioned this issue May 17, 2023
Merged
12 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue May 17, 2023
@lance6716 @ti-chi-bot
lightning: improve region actions for huge region number (pingcap#43807)
48d3b78 
close pingcap#43510
ti-chi-bot bot pushed a commit that referenced this issue May 18, 2023
@ti-chi-bot
lightning: improve region actions for huge region number (#43807) (#4…
0b57a18 
…3920)

close #43510
@GMHDBJD GMHDBJD mentioned this issue Jun 25, 2023
Merged
16 tasks
@lance6716 lance6716 mentioned this issue Aug 18, 2023
Closed
@D3Hunter D3Hunter mentioned this issue Aug 24, 2023
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-7.1 This bug affects the 7.1.x(LTS) versions. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

lightning: improve region actions for huge region number lance6716/tidb
1 participant
@lance6716