Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix inappropriate heuristic method of estimating out-of-range values #19200

Merged
merged 3 commits into from
Aug 14, 2020

Conversation

qw4990
Copy link
Contributor

@qw4990 qw4990 commented Aug 14, 2020

cherry pick for #18543


What problem does this PR solve?

Issue Number: close #18461

Problem Summary: If the estimated value is out of range, an inappropriate heuristic rule sel = 1/NDV*(modifyRows/totalRows) is used, which may cause unexpected low sel when a few rows are modified.

What is changed and how it works?

Change this rule to:

func outOfRangeEQSelectivity(ndv, modifyRows, totalRows int64) float64 {
	if modifyRows == 0 {
		return 0 // it must be 0 since the histogram contains the whole data
	}
	if ndv < outOfRangeBetweenRate {
		ndv = outOfRangeBetweenRate // avoid inaccurate selectivity caused by small NDV
	}
	selectivity := 1 / float64(ndv) // TODO: After extracting TopN from histograms, we can minus the TopN fraction here.
	if selectivity*float64(totalRows) > float64(modifyRows) {
		selectivity = float64(modifyRows) / float64(totalRows)
	}
	return selectivity
}

Check List

Tests

  • Unit test

Release note

  • planner: fix the inappropriate heuristic rule to estimate the EQ selectivity when out of range

@qw4990
Copy link
Contributor Author

qw4990 commented Aug 14, 2020

/run-all-tests

@qw4990 qw4990 requested review from eurekaka and winoros August 14, 2020 07:12
Copy link
Member

@winoros winoros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 14, 2020
Copy link
Contributor

@eurekaka eurekaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 14, 2020
@eurekaka
Copy link
Contributor

/merge

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 14, 2020
@ti-srebot
Copy link
Contributor

/run-all-tests

@ti-srebot
Copy link
Contributor

@qw4990 merge failed.

@qw4990
Copy link
Contributor Author

qw4990 commented Aug 14, 2020

/run-sqllogic-test-1

@qw4990 qw4990 merged commit 6446961 into pingcap:release-3.0 Aug 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/statistics priority/release-blocker This issue blocks a release. Please solve it ASAP. sig/planner SIG: Planner status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/3.0-cherry-pick
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants