*: improve performance for BuildHistAndTopN

[hawkingrei]

What problem does this PR solve?

Issue Number: close #49180

Problem Summary:

What changed and how does it work?

Now, Analyze will take most of the time on the BuildHistAndTopN. BuildHistAndTopN has many unnecessary sort and compare with low NDV. so we can merge the same data then sort and compare. it will have better performance.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

➜  tidb git:(improve_against)  go test -benchmem -run=^$ -bench ^BenchmarkBuildHistAndTopNWithLowNDV github.com/pingcap/tidb/pkg/statistics --benchtime=30s
goos: linux
goarch: amd64
pkg: github.com/pingcap/tidb/pkg/statistics
cpu: AMD Ryzen 7 7735HS with Radeon Graphics
BenchmarkBuildHistAndTopNWithLowNDV/true-16         	     483	  74791981 ns/op	59844620 B/op	 1002092 allocs/op
BenchmarkBuildHistAndTopNWithLowNDV/false-16        	     477	  75674610 ns/op	59844684 B/op	 1002092 allocs/op
PASS
ok  	github.com/pingcap/tidb/pkg/statistics	90.061s

• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from hawkingrei. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

Attention: Patch coverage is 86.82171% with 34 lines in your changes missing coverage. Please review.

Project coverage is 57.0459%. Comparing base (542907f) to head (ae68073).
Report is 2296 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #48902         +/-   ##
=================================================
- Coverage   72.0440%   57.0459%   -14.9981%     
=================================================
  Files          1452       1671        +219     
  Lines        347472     652500     +305028     
=================================================
+ Hits         250333     372225     +121892     
- Misses        76776     254801     +178025     
- Partials      20363      25474       +5111     

Flag	Coverage Δ
integration	41.3288% <39.1472%> (?)
unit	70.1822% <86.8217%> (-1.8619%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	54.0269% <ø> (-2.2860%)	⬇️
parser	∅ <ø> (∅)
br	59.7614% <ø> (+7.6339%)	⬆️

[hawkingrei]

/retest

[hawkingrei]

/retest

[time-and-fate]

1. Please at least keep the comments valid and up-to-date after you copied them from elsewhere.
2. If you need to copy ~80% of an existing function, I suggest we try to find some methods to avoid the large duplication.
3. From your benchmark result, I think we are trading space for time instead of a pure improvement. So what's the reason behind this change, and have we checked the risk?

[hawkingrei]

1. Please at least keep the comments valid and up-to-date after you copied them from elsewhere.

   2. If you need to copy ~80% of an existing function, I suggest we try to find some methods to avoid the large duplication.

   3. From your benchmark result, I think we are trading space for time instead of a pure improvement. So what's the reason behind this change, and have we checked the risk?

I have done the refactor to merge the same code and reduce memory allocate

Now, the new one is better than the old one.

[time-and-fate]

I think there is little performance difference in the new benchmark result.

[xhebox]

Suggested change

	highNDVMode bool,
	lowNDVMode bool,

Why it is high NDV?

[ti-chi-bot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ti-chi-bot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ti-chi-bot]

@hawkingrei: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
idc-jenkins-ci-tidb/build	ae68073	link	true	/test build
idc-jenkins-ci-tidb/unit-test	ae68073	link	true	/test unit-test
idc-jenkins-ci-tidb/check_dev_2	ae68073	link	true	/test check-dev2
pull-lightning-integration-test	ae68073	link	true	/test pull-lightning-integration-test
pull-integration-e2e-test	ae68073	link	true	/test pull-integration-e2e-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.