Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schedule: one operator only occupy one limit #3820

Merged
merged 20 commits into from
Jul 2, 2021

Conversation

bufferflies
Copy link
Contributor

@bufferflies bufferflies commented Jun 29, 2021

Signed-off-by: tongjian [email protected]

What problem does this PR solve?

fix: #3807
#3778

What is changed and how it works?

operator controller's count will occupy the highest priority OpKind even if the operator has many OpKind.

the relation between limit and scheduler is blow:
image

red mark need to be changed

Check List

Tests

  • Unit test

Code changes

Side effects

Related changes

Release note

Reduce the conflict due to multiple scheduler running in same time

@ti-chi-bot ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jun 29, 2021
@ti-chi-bot ti-chi-bot requested review from rleungx and Yisaer June 29, 2021 10:18
@codecov
Copy link

codecov bot commented Jun 29, 2021

Codecov Report

Merging #3820 (bf1f8fc) into master (193bdde) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3820      +/-   ##
==========================================
- Coverage   75.17%   75.14%   -0.03%     
==========================================
  Files         245      245              
  Lines       24630    24628       -2     
==========================================
- Hits        18515    18507       -8     
- Misses       4500     4501       +1     
- Partials     1615     1620       +5     
Flag Coverage Δ
unittests 75.14% <100.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
server/schedule/operator/kind.go 100.00% <ø> (ø)
server/schedule/operator/operator.go 96.55% <100.00%> (+0.02%) ⬆️
server/schedule/operator_controller.go 85.91% <100.00%> (+0.72%) ⬆️
server/schedulers/balance_region.go 88.07% <100.00%> (ø)
server/id/id.go 76.19% <0.00%> (-4.77%) ⬇️
server/election/leadership.go 85.88% <0.00%> (-1.18%) ⬇️
server/member/member.go 68.81% <0.00%> (-1.08%) ⬇️
server/tso/allocator_manager.go 78.96% <0.00%> (-1.01%) ⬇️
server/core/storage.go 67.68% <0.00%> (-0.77%) ⬇️
server/grpc_service.go 47.66% <0.00%> (-0.69%) ⬇️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 193bdde...bf1f8fc. Read the comment docs.

@bufferflies
Copy link
Contributor Author

/cc @HunDunDM @lhy1024

@ti-chi-bot ti-chi-bot requested review from HunDunDM and lhy1024 June 30, 2021 06:43
Copy link
Member

@HunDunDM HunDunDM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OperatorCount can return oc.counts[mask] directly now?

server/schedule/operator/operator.go Outdated Show resolved Hide resolved
Signed-off-by: bufferflies <[email protected]>
@ti-chi-bot ti-chi-bot merged commit e058927 into tikv:master Jul 2, 2021
@nolouch nolouch added type/cherry-pick-for-release-5.0 The PR belongs to release-5.0 cherry pick. type/cherry-pick-for-release-4.0 The PR belongs to release-4.0 cherry pick. type/cherry-pick-for-release-5.1 The PR belongs to release-5.1 cherry pick. labels Jul 2, 2021
@nolouch
Copy link
Contributor

nolouch commented Jul 2, 2021

/run-cherry-picker

@bufferflies bufferflies deleted the featuer/OperatorDecouple branch July 6, 2021 07:50
@HunDunDM HunDunDM added needs-cherry-pick-release-4.0 The PR needs to cherry pick to release-4.0 branch. needs-cherry-pick-release-5.0 The PR needs to cherry pick to release-5.0 branch. needs-cherry-pick-release-5.1 Type: Need cherry pick to release-5.1 and removed type/cherry-pick-for-release-4.0 The PR belongs to release-4.0 cherry pick. type/cherry-pick-for-release-5.0 The PR belongs to release-5.0 cherry pick. type/cherry-pick-for-release-5.1 The PR belongs to release-5.1 cherry pick. labels Jul 9, 2021
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jul 9, 2021
* use O(1) to get WMA

Signed-off-by: tongjian <[email protected]>

* add sonar file ignore code duplicated
Signed-off-by: tongjian <[email protected]>

Signed-off-by: buffer <[email protected]>

* one operator should occupy one limit

Signed-off-by: bufferflies <[email protected]>

* remove no need change

Signed-off-by: bufferflies <[email protected]>

* move scheduler OpKind to operator.go

Signed-off-by: bufferflies <[email protected]>

* replace opkind to get op producer by using LowBit

Signed-off-by: bufferflies <[email protected]>

* pass uint test

Signed-off-by: bufferflies <[email protected]>

* replace chinese link

Signed-off-by: bufferflies <[email protected]>

* OperatorCount use O(1)

Signed-off-by: bufferflies <[email protected]>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3856.

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jul 9, 2021
* use O(1) to get WMA

Signed-off-by: tongjian <[email protected]>

* add sonar file ignore code duplicated
Signed-off-by: tongjian <[email protected]>

Signed-off-by: buffer <[email protected]>

* one operator should occupy one limit

Signed-off-by: bufferflies <[email protected]>

* remove no need change

Signed-off-by: bufferflies <[email protected]>

* move scheduler OpKind to operator.go

Signed-off-by: bufferflies <[email protected]>

* replace opkind to get op producer by using LowBit

Signed-off-by: bufferflies <[email protected]>

* pass uint test

Signed-off-by: bufferflies <[email protected]>

* replace chinese link

Signed-off-by: bufferflies <[email protected]>

* OperatorCount use O(1)

Signed-off-by: bufferflies <[email protected]>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jul 9, 2021
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3857.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3858.

bufferflies added a commit to bufferflies/pd that referenced this pull request Jul 9, 2021
* use O(1) to get WMA

Signed-off-by: tongjian <[email protected]>

* add sonar file ignore code duplicated
Signed-off-by: tongjian <[email protected]>

Signed-off-by: buffer <[email protected]>

* one operator should occupy one limit

Signed-off-by: bufferflies <[email protected]>

* remove no need change

Signed-off-by: bufferflies <[email protected]>

* move scheduler OpKind to operator.go

Signed-off-by: bufferflies <[email protected]>

* replace opkind to get op producer by using LowBit

Signed-off-by: bufferflies <[email protected]>

* pass uint test

Signed-off-by: bufferflies <[email protected]>

* replace chinese link

Signed-off-by: bufferflies <[email protected]>

* OperatorCount use O(1)

Signed-off-by: bufferflies <[email protected]>
ti-chi-bot added a commit that referenced this pull request Jul 15, 2021
* This is an automated cherry-pick of #3820

Signed-off-by: ti-chi-bot <[email protected]>

* fix conflicts

Signed-off-by: nolouch <[email protected]>

* fix test

Signed-off-by: nolouch <[email protected]>

Co-authored-by: buffer <[email protected]>
Co-authored-by: nolouch <[email protected]>
ti-chi-bot added a commit that referenced this pull request Jul 22, 2021
* schedule:  one operator only occupy one limit (#3820)

* use O(1) to get WMA

Signed-off-by: tongjian <[email protected]>

* add sonar file ignore code duplicated
Signed-off-by: tongjian <[email protected]>

Signed-off-by: buffer <[email protected]>

* one operator should occupy one limit

Signed-off-by: bufferflies <[email protected]>

* remove no need change

Signed-off-by: bufferflies <[email protected]>

* move scheduler OpKind to operator.go

Signed-off-by: bufferflies <[email protected]>

* replace opkind to get op producer by using LowBit

Signed-off-by: bufferflies <[email protected]>

* pass uint test

Signed-off-by: bufferflies <[email protected]>

* replace chinese link

Signed-off-by: bufferflies <[email protected]>

* OperatorCount use O(1)

Signed-off-by: bufferflies <[email protected]>

* fix test

Signed-off-by: nolouch <[email protected]>

Co-authored-by: buffer <[email protected]>
Co-authored-by: nolouch <[email protected]>
Co-authored-by: 混沌DM <[email protected]>
ti-chi-bot added a commit that referenced this pull request Aug 13, 2021
* schedule:  one operator only occupy one limit (#3820)

* use O(1) to get WMA

Signed-off-by: tongjian <[email protected]>

* add sonar file ignore code duplicated
Signed-off-by: tongjian <[email protected]>

Signed-off-by: buffer <[email protected]>

* one operator should occupy one limit

Signed-off-by: bufferflies <[email protected]>

* remove no need change

Signed-off-by: bufferflies <[email protected]>

* move scheduler OpKind to operator.go

Signed-off-by: bufferflies <[email protected]>

* replace opkind to get op producer by using LowBit

Signed-off-by: bufferflies <[email protected]>

* pass uint test

Signed-off-by: bufferflies <[email protected]>

* replace chinese link

Signed-off-by: bufferflies <[email protected]>

* OperatorCount use O(1)

Signed-off-by: bufferflies <[email protected]>

* opAdmin place first

Signed-off-by: bufferflies <[email protected]>

* pass unit test

Signed-off-by: bufferflies <[email protected]>

* pass unit test

Signed-off-by: bufferflies <[email protected]>

* pass unit test

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: buffer <[email protected]>
Co-authored-by: 混沌DM <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-4.0 The PR needs to cherry pick to release-4.0 branch. needs-cherry-pick-release-5.0 The PR needs to cherry pick to release-5.0 branch. needs-cherry-pick-release-5.1 Type: Need cherry pick to release-5.1 release-note Denotes a PR that will be considered when it comes time to generate release notes. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

merge operator blocks the balance leader operations
6 participants