Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store/tikv: pipe the split and scatter task #23357

Closed
wants to merge 3 commits into from

Conversation

nolouch
Copy link
Member

@nolouch nolouch commented Mar 16, 2021

Signed-off-by: nolouch [email protected]

What problem does this PR solve?

close #22969

Problem Summary:
Splitting and Scattering at the same time will cause multiple conflicts on the same region.
image

What is changed and how it works?

pipeline all splits task and scatter tasks

Release note

  • Make the time consuming of split table more stable
I have run multiple times  to split 4000 regions, and the time is stale about `1m30s`
2021/03/17 17:25:57 split single sql result: [map[SCATTER_FINISH_RATIO:1 TOTAL_SPLIT_REGION:2001]]
2021/03/17 17:25:57 split all sql run total cost: 1m28.908304306s
2021/03/17 17:25:57 split all sql run success!!!

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 16, 2021
@github-actions github-actions bot added the sig/sql-infra SIG: SQL Infra label Mar 16, 2021
@nolouch nolouch requested a review from crazycs520 March 16, 2021 09:57
Comment on lines 209 to 213
for _, r := range spResp.Regions {
tid := int64(0)
if tableID != nil {
tid = *tableID
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for _, r := range spResp.Regions {
tid := int64(0)
if tableID != nil {
tid = *tableID
}
tid := int64(0)
if tableID != nil {
tid = *tableID
}
for _, r := range spResp.Regions {

Comment on lines 238 to 241
tid := int64(0)
if tableID != nil {
tid = *tableID
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
tid := int64(0)
if tableID != nil {
tid = *tableID
}

tid = *tableID
}
spResp := batchResp.resp.Resp.(*kvrpcpb.SplitRegionResponse)
for _, r := range spResp.Regions {
if err = s.scatterRegion(bo, r.Id, tableID); err == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's still using

_, err := s.pdClient.ScatterRegions(bo.ctx, []uint64{regionID}, opts...)
the batch API to scatter regions one by one. Which may make the scatter not work?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean #22788 .
OK, I'm off topic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current region scatter may not work in a specific case which described in tikv/pd#3422 , I think it's not related to the api.

@ti-chi-bot
Copy link
Member

@nolouch: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 4, 2021
@nolouch nolouch closed this Jul 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-5.0 needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. sig/sql-infra SIG: SQL Infra size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make the time consuming of split table more stable
5 participants