Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

planner, executor: support inline projection for TopN | tidb-test=pr/2464 #58500

Merged
merged 18 commits into from
Jan 6, 2025
Merged

planner, executor: support inline projection for TopN | tidb-test=pr/2464 #58500

merged 18 commits into from
Jan 6, 2025

Conversation

EricZequan
Copy link
Contributor

@EricZequan EricZequan commented Dec 24, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #54245

Problem Summary:

What is changed and how it works?

Prune the TopN's schema to avoid no-needs column transmission by implementing inline projection for TopN.

The implementation is very similar to #19900

To learn more about inline projection, see: #14428

First, we modified the topN structure logicalSchemaProducer so that TopN can have its own schema information. In pkg/planner/core/task.go:(PhysicalTopN) getPushedDownTopN, we prune the schema of TopN with the columns required by the upper-level plan to avoid passing unnecessary columns.

Then, in the executor part, we added columnIdxsUsedByChild to TopN to mark the columns that actually need to be processed. In the Next method, the required column results are recorded in req according to this mark.

VectorDBBench test result :
QPS improvement: 1M-768D--14%, 500K-1536D--30%

before:
1M-768D---"results": [{"metrics": {"max_load_count": 0, "load_duration": 0.0, "qps": 72.7114, "serial_latency_p99": 0.1342, "recall": 0.9012}
500K-1536D---"results": [{"metrics": {"max_load_count": 0, "load_duration": 0.0, "qps": 61.2001, "serial_latency_p99": 0.1865, "recall": 0.9335}

after:
1M-768D---"results": [{"metrics": {"max_load_count": 0, "load_duration": 0.0, "qps": 82.7748, "serial_latency_p99": 0.1297, "recall": 0.8865}
500K-1536D---"results": [{"metrics": {"max_load_count": 0, "load_duration": 0.0, "qps": 79.219, "serial_latency_p99": 0.1503, "recall": 0.9171}

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@EricZequan
prune topn plan
abf7ec8 
Signed-off-by: “EricZequan” <[email protected]>
@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed do-not-merge/needs-linked-issue sig/planner SIG: Planner size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 24, 2024
@tiprow tiprow
Copy link

tiprow bot commented Dec 24, 2024

Hi @EricZequan. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@EricZequan
update
7656bb6 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
fix integration test case
24829de 
Signed-off-by: “EricZequan” <[email protected]>
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 24, 2024
@EricZequan
update
bf29607 
Signed-off-by: “EricZequan” <[email protected]>
EricZequan
EricZequan commented Dec 24, 2024
View reviewed changes
pkg/planner/core/task.go
Comment on lines 920 to 925
// └─Byitem: vec_distance(vec, '[1,2,3]')
// └─Schema: id, vec
//
// New: DataSource(id, vec) -> Projection(id, vec->dis) -> TopN(by dis) -> Projection(id)
// └─Byitem: dis
// └─Schema: id, dis
//
// Note that for plan now, TopN has its own schema and does not use the schema of children.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@breezewish PTAL~

pkg/planner/core/casetest/vectorsearch/vector_index_test.go
@@ -413,3 +411,36 @@ func TestVectorSearchWithPKForceTiKV(t *testing.T) {
require.Equal(t, output[i].Warn, testdata.ConvertSQLWarnToStrings(tk.Session().GetSessionVars().StmtCtx.GetWarnings()))
}
}

func TestVectorSearchHeavyFunction(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@breezewish PTAL~

@EricZequan
update
3fd9494 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
Copy link
Contributor Author

/retest

@tiprow tiprow
Copy link

tiprow bot commented Dec 24, 2024

@EricZequan: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@EricZequan
update
f0258a6 
Signed-off-by: “EricZequan” <[email protected]>
@breezewish
Copy link
Member

/ok-to-test

@ti-chi-bot ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Dec 24, 2024
@EricZequan
update
81f7021 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
fix ci
43447b5 
Signed-off-by: “EricZequan” <[email protected]>
@codecov Codecov
Copy link

codecov bot commented Dec 25, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 68.46847% with 35 lines in your changes missing coverage. Please review.

Project coverage is 73.8020%. Comparing base (068b9a8) to head (6375d6a).
Report is 23 commits behind head on master.

Additional details and impacted files 
@@               Coverage Diff                @@
##             master     #58500        +/-   ##
================================================
+ Coverage   73.1093%   73.8020%   +0.6927%     
================================================
  Files          1676       1681         +5     
  Lines        463369     470270      +6901     
================================================
+ Hits         338766     347069      +8303     
+ Misses       103786     102229      -1557     
- Partials      20817      20972       +155
Flag Coverage Δ
integration 43.2516% <56.7567%> (?)
unit 72.5028% <68.4684%> (+0.2018%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.6910% <ø> (ø)
parser ∅ <ø> (∅)
br 45.7486% <ø> (-0.0628%) ⬇️

@EricZequan
fix ci
c34665b 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
Copy link
Contributor Author

/test unit-test

@tiprow tiprow
Copy link

tiprow bot commented Dec 25, 2024

@EricZequan: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test fast_test_tiprow
  • /test fast_test_tiprow_ddlargsv1
  • /test tidb_parser_test

Use /test all to run the following jobs that were automatically triggered:

  • fast_test_tiprow
  • tidb_parser_test

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@EricZequan
fix ci
8c6addf 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
fix ci
031b875 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
Copy link
Contributor Author

/retest

@EricZequan
fix ci
debe5cc 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
Merge remote-tracking branch 'upstream/master' into prune-topN
871a6e4 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
fix ci
81f26b9 
Signed-off-by: “EricZequan” <[email protected]>
@EricZequan
fix ci
e9faed6 
Signed-off-by: “EricZequan” <[email protected]>
@winoros winoros changed the title planner: Prune topN schema for avoiding no-need column exchange planner, executor: support inline projection for TopN Dec 31, 2024
@EricZequan EricZequan changed the title planner, executor: support inline projection for TopN planner, executor: support inline projection for TopN | tidb-test=pr/2464 Dec 31, 2024
@EricZequan
Copy link
Contributor Author

/retest

@EricZequan
Copy link
Contributor Author

/retest

@EricZequan
update
e14fe59 
Signed-off-by: “EricZequan” <[email protected]>
xzhangxian1008
xzhangxian1008 approved these changes Jan 3, 2025
View reviewed changes
Copy link
Contributor

@xzhangxian1008 xzhangxian1008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

pkg/executor/sortexec/topn.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 3, 2025
@EricZequan
Copy link
Contributor Author

/retest

@EricZequan
update
6375d6a 
Signed-off-by: “EricZequan” <[email protected]>
@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Jan 6, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AilinKid, xzhangxian1008

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 6, 2025
@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Jan 6, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-01-03 02:20:49.757497163 +0000 UTC m=+662585.113501702: ☑️ agreed by xzhangxian1008.
  • 2025-01-06 03:26:35.918971116 +0000 UTC m=+151339.207802821: ☑️ agreed by AilinKid.

@ti-chi-bot ti-chi-bot bot merged commit 911d5a1 into pingcap:master Jan 6, 2025
24 checks passed
@hawkingrei hawkingrei mentioned this pull request Jan 7, 2025
Open
This was referenced Jan 8, 2025
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xzhangxian1008 xzhangxian1008 xzhangxian1008 approved these changes

@AilinKid AilinKid AilinKid approved these changes

Assignees
No one assigned
Labels
approved lgtm ok-to-test Indicates a PR is ready to be tested. release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@EricZequan @breezewish @AilinKid @xzhangxian1008