Migrate table scan related code to `DAGStorageInterpreter` #4783

SeaRise · 2022-04-27T10:26:04Z

What problem does this PR solve?

Issue Number: ref #4118

Problem Summary:

What is changed and how it works?

remove DAGQueryBlock used in DAGStorageInterpreter
move executeRemoteQuery, executeCastAfterTableScan and executePushedDownFilter from DAGQueryBlockInterpreter to DAGStorageInterpreter

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-04-27T10:26:06Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang
ywqzzy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

SeaRise · 2022-04-27T10:37:40Z

/run-integration-test

SeaRise · 2022-04-27T11:29:04Z

/run-integration-test

dbms/src/Flash/Coprocessor/DAGStorageInterpreter.h

dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp

dbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp

dbms/src/Flash/Coprocessor/RemoteRequest.cpp

dbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp

Co-authored-by: yanweiqi <[email protected]>

dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp

dbms/src/Flash/Coprocessor/RemoteRequest.cpp

Co-authored-by: JaySon <[email protected]>

ywqzzy

LGTM

JaySon-Huang

LGTM

Signed-off-by: JaySon-Huang <[email protected]>

JaySon-Huang · 2022-05-05T11:53:20Z

/merge

ti-chi-bot · 2022-05-05T11:53:21Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-05-05T11:53:22Z

This pull request has been accepted and is ready to merge.

Commit hash: d3facf8

ti-chi-bot · 2022-05-05T11:53:35Z

@SeaRise: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-05-05T12:19:19Z

Coverage for changed files

Filename                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DAGQueryBlockInterpreter.cpp         285               268     5.96%          39                28    28.21%         656               585    10.82%         214               207     3.27%
DAGQueryBlockInterpreter.h             2                 0   100.00%           2                 0   100.00%           2                 0   100.00%           0                 0         -
DAGStorageInterpreter.cpp            420               420     0.00%          32                32     0.00%         861               861     0.00%         282               282     0.00%
InterpreterUtils.cpp                  32                25    21.88%           4                 2    50.00%          47                36    23.40%          24                20    16.67%
PushDownFilter.cpp                    15                15     0.00%           3                 3     0.00%          33                33     0.00%          12                12     0.00%
PushDownFilter.h                       1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
RemoteRequest.cpp                     31                31     0.00%           3                 3     0.00%          80                80     0.00%          22                22     0.00%
RemoteRequest.h                        1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
TiDBTableScan.cpp                     31                31     0.00%           2                 2     0.00%          51                51     0.00%          24                24     0.00%
TiDBTableScan.h                        6                 6     0.00%           6                 6     0.00%          18                18     0.00%           0                 0         -
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                824               798     3.16%          93                78    16.13%        1750              1666     4.80%         578               567     1.90%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18076      10153            43.83%    200504  101030       49.61%

full coverage report (for internal network access only)

SeaRise added 3 commits April 27, 2022 17:39

move codes

efeed2d

fix and replace fmt

e321901

fix

7d3ccf8

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 27, 2022

ywqzzy self-requested a review April 27, 2022 10:33

update

e742296

SeaRise added 3 commits April 27, 2022 18:54

fix

30fc69d

fix

955eb07

update

80cd379

JaySon-Huang self-requested a review April 27, 2022 11:36

JaySon-Huang mentioned this pull request Apr 27, 2022

Refine some comments about learner read #4784

Merged

12 tasks

SeaRise requested a review from windtalker April 27, 2022 15:00

ywqzzy reviewed Apr 29, 2022

View reviewed changes

dbms/src/Flash/Coprocessor/DAGStorageInterpreter.h Outdated Show resolved Hide resolved

dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp Outdated Show resolved Hide resolved

dbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp Show resolved Hide resolved

ywqzzy self-requested a review April 29, 2022 03:44

SeaRise added 3 commits April 29, 2022 12:51

address comments

f73d6f9

f

794795f

fix

44f2aec

ywqzzy reviewed May 5, 2022

View reviewed changes

dbms/src/Flash/Coprocessor/RemoteRequest.cpp Outdated Show resolved Hide resolved

dbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp Outdated Show resolved Hide resolved

SeaRise and others added 2 commits May 5, 2022 16:21

Update dbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp

7f12002

Co-authored-by: yanweiqi <[email protected]>

address comments

0a751e5

JaySon-Huang reviewed May 5, 2022

View reviewed changes

dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp Outdated Show resolved Hide resolved

dbms/src/Flash/Coprocessor/RemoteRequest.cpp Outdated Show resolved Hide resolved

SeaRise and others added 2 commits May 5, 2022 16:45

Update dbms/src/Flash/Coprocessor/RemoteRequest.cpp

0e11afb

Co-authored-by: JaySon <[email protected]>

Update dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp

14f738a

Co-authored-by: JaySon <[email protected]>

ywqzzy approved these changes May 5, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 5, 2022

JaySon-Huang approved these changes May 5, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 5, 2022

Fix build

d3facf8

Signed-off-by: JaySon-Huang <[email protected]>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 5, 2022

Merge branch 'master' into move_table_scan_to_stroage_interpreter

b2719cf

ti-chi-bot merged commit e10c6ed into pingcap:master May 5, 2022

SeaRise mentioned this pull request May 11, 2022

fix tispark query err: for PushDownFilter, conditions and executor_id should both be empty or neither should be empty #4864

Merged

12 tasks

SeaRise deleted the move_table_scan_to_stroage_interpreter branch September 26, 2022 05:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate table scan related code to `DAGStorageInterpreter` #4783

Migrate table scan related code to `DAGStorageInterpreter` #4783

SeaRise commented Apr 27, 2022

ti-chi-bot commented Apr 27, 2022 •

edited

Loading

SeaRise commented Apr 27, 2022

SeaRise commented Apr 27, 2022

ywqzzy left a comment

JaySon-Huang left a comment

JaySon-Huang commented May 5, 2022

ti-chi-bot commented May 5, 2022

ti-chi-bot commented May 5, 2022

ti-chi-bot commented May 5, 2022

sre-bot commented May 5, 2022

Migrate table scan related code to DAGStorageInterpreter #4783

Migrate table scan related code to DAGStorageInterpreter #4783

Conversation

SeaRise commented Apr 27, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Apr 27, 2022 • edited Loading

SeaRise commented Apr 27, 2022

SeaRise commented Apr 27, 2022

ywqzzy left a comment

Choose a reason for hiding this comment

JaySon-Huang left a comment

Choose a reason for hiding this comment

JaySon-Huang commented May 5, 2022

ti-chi-bot commented May 5, 2022

ti-chi-bot commented May 5, 2022

ti-chi-bot commented May 5, 2022

sre-bot commented May 5, 2022

Coverage for changed files

Coverage summary

Migrate table scan related code to `DAGStorageInterpreter` #4783

Migrate table scan related code to `DAGStorageInterpreter` #4783

ti-chi-bot commented Apr 27, 2022 •

edited

Loading