Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: use MPMCQueue in UnionBlockInputStream and SharedQueryBlockInputStream #4418

Merged
merged 14 commits into from
Mar 30, 2022

Conversation

fuzhe1989
Copy link
Contributor

@fuzhe1989 fuzhe1989 commented Mar 24, 2022

Signed-off-by: fuzhe1989 [email protected]

What problem does this PR solve?

Issue Number: close #4419

Problem Summary:

What is changed and how it works?

What we could acheive from MPMCQueue:

  1. Better performance.
  2. Easier error handling.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

Signed-off-by: fuzhe1989 <[email protected]>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Mar 24, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • bestwoody
  • windtalker

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed do-not-merge/needs-linked-issue labels Mar 24, 2022
@fuzhe1989
Copy link
Contributor Author

/run-all-tests

@fuzhe1989 fuzhe1989 requested a review from fzhedu March 24, 2022 09:56
@sre-bot
Copy link
Collaborator

sre-bot commented Mar 24, 2022

Coverage for changed files

Filename                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ParallelInputsProcessor.h              89                89     0.00%          17                17     0.00%         153               153     0.00%          46                46     0.00%
SharedQueryBlockInputStream.h          45                45     0.00%          12                12     0.00%          90                90     0.00%          20                20     0.00%
UnionBlockInputStream.h               102               102     0.00%          24                24     0.00%         135               135     0.00%          52                52     0.00%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                 236               236     0.00%          53                53     0.00%         378               378     0.00%         118               118     0.00%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17044      9571             43.85%    191811  97018        49.42%

full coverage report (for internal network access only)

Signed-off-by: fuzhe1989 <[email protected]>
@fuzhe1989
Copy link
Contributor Author

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 24, 2022

Coverage for changed files

Filename                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ParallelInputsProcessor.h              89                89     0.00%          17                17     0.00%         153               153     0.00%          46                46     0.00%
SharedQueryBlockInputStream.h          46                46     0.00%          12                12     0.00%          91                91     0.00%          20                20     0.00%
UnionBlockInputStream.h               102               102     0.00%          24                24     0.00%         135               135     0.00%          52                52     0.00%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                 237               237     0.00%          53                53     0.00%         379               379     0.00%         118               118     0.00%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17044      9571             43.85%    191812  96990        49.43%

full coverage report (for internal network access only)

@fuzhe1989 fuzhe1989 requested a review from bestwoody March 24, 2022 13:00
@fuzhe1989
Copy link
Contributor Author

/run-all-tests

1 similar comment
@fuzhe1989
Copy link
Contributor Author

/run-all-tests

@fuzhe1989
Copy link
Contributor Author

/rebuild

@fuzhe1989
Copy link
Contributor Author

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 25, 2022

Coverage for changed files

Filename                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ParallelInputsProcessor.h              89                89     0.00%          17                17     0.00%         153               153     0.00%          46                46     0.00%
SharedQueryBlockInputStream.h          46                46     0.00%          12                12     0.00%          91                91     0.00%          20                20     0.00%
UnionBlockInputStream.h               102               102     0.00%          24                24     0.00%         135               135     0.00%          52                52     0.00%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                 237               237     0.00%          53                53     0.00%         379               379     0.00%         118               118     0.00%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17044      9571             43.85%    191812  97033        49.41%

full coverage report (for internal network access only)

@fzhedu
Copy link
Contributor

fzhedu commented Mar 25, 2022

can we add some tests?

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 29, 2022
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 29, 2022
Signed-off-by: fuzhe1989 <[email protected]>
if (isCancelled() || read_suffixed)
return {};
} while (!queue.tryPop(block, try_action_millisecionds));
if (!isCancelled())
Copy link
Contributor

@bestwoody bestwoody Mar 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if !exception_msg.empty() && !isCancelled() happens?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's dangerous to visit exception_msg before waitThread. don't worry about the blocking, waitThread will be called only when the fetchBlocks itself finishes the queue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment, I'm not worry about blocking. My point is that exception_msg may not be empty and !isCanceled is false, then we miss the throw Exception(exception_msg); in waitThread();

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the inputstream is canceled, then the exception_msg doesn't matter, we can safely ignore it.

Copy link
Contributor

@bestwoody bestwoody Mar 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure... @windtalker PTAL

fuzhe1989 added 3 commits March 30, 2022 12:55
@ti-chi-bot ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 30, 2022
Signed-off-by: fuzhe1989 <[email protected]>
@fuzhe1989
Copy link
Contributor Author

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 30, 2022

Coverage for changed files

Filename                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ParallelInputsProcessor.h              89                89     0.00%          17                17     0.00%         153               153     0.00%          46                46     0.00%
SharedQueryBlockInputStream.h          51                51     0.00%          12                12     0.00%          99                99     0.00%          24                24     0.00%
UnionBlockInputStream.h               104               104     0.00%          24                24     0.00%         133               133     0.00%          48                48     0.00%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                 244               244     0.00%          53                53     0.00%         385               385     0.00%         118               118     0.00%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17069      9589             43.82%    192079  97194        49.40%

full coverage report (for internal network access only)

Copy link
Contributor

@bestwoody bestwoody left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For rest of concerns, PTAL

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 30, 2022
Copy link
Contributor

@windtalker windtalker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 30, 2022
@fuzhe1989
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@fuzhe1989: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 7a1bf90

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 30, 2022
@ti-chi-bot
Copy link
Member

@fuzhe1989: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 30, 2022

Coverage for changed files

Filename                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ParallelInputsProcessor.h              89                89     0.00%          17                17     0.00%         153               153     0.00%          46                46     0.00%
SharedQueryBlockInputStream.h          51                51     0.00%          12                12     0.00%          99                99     0.00%          24                24     0.00%
UnionBlockInputStream.h               104               104     0.00%          24                24     0.00%         133               133     0.00%          48                48     0.00%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                 244               244     0.00%          53                53     0.00%         385               385     0.00%         118               118     0.00%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17069      9589             43.82%    192079  97196        49.40%

full coverage report (for internal network access only)

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 30, 2022

Coverage for changed files

Filename                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ParallelInputsProcessor.h              89                89     0.00%          17                17     0.00%         153               153     0.00%          46                46     0.00%
SharedQueryBlockInputStream.h          51                51     0.00%          12                12     0.00%          99                99     0.00%          24                24     0.00%
UnionBlockInputStream.h               104               104     0.00%          24                24     0.00%         133               133     0.00%          48                48     0.00%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                 244               244     0.00%          53                53     0.00%         385               385     0.00%         118               118     0.00%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17069      9589             43.82%    192115  97210        49.40%

full coverage report (for internal network access only)

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 30, 2022

Coverage for changed files

Filename                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ParallelInputsProcessor.h              89                89     0.00%          17                17     0.00%         153               153     0.00%          46                46     0.00%
SharedQueryBlockInputStream.h          51                51     0.00%          12                12     0.00%          99                99     0.00%          24                24     0.00%
UnionBlockInputStream.h               104               104     0.00%          24                24     0.00%         133               133     0.00%          48                48     0.00%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                 244               244     0.00%          53                53     0.00%         385               385     0.00%         118               118     0.00%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17070      9590             43.82%    192118  97205        49.40%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot merged commit cf792d1 into pingcap:master Mar 30, 2022
@fuzhe1989 fuzhe1989 deleted the fuzhe/use_mpmc_queue branch March 31, 2022 02:52
@SeaRise SeaRise mentioned this pull request Apr 12, 2022
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Replace ConcurrentBoundedQueue by MPMCQueue
6 participants