Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

planner: simplify StatsHandler and break it down into multiple smaller and simpler sub-modules #46905

Closed
qw4990 opened this issue Sep 12, 2023 · 3 comments
Closed

planner: simplify StatsHandler and break it down into multiple smaller and simpler sub-modules #46905

qw4990 opened this issue Sep 12, 2023 · 3 comments
Assignees
@hawkingrei @qw4990
Labels
affects-7.5 This bug affects the 7.5.x(LTS) versions. component/statistics report/customer Customers have encountered this bug. sig/planner SIG: Planner type/enhancement The issue or PR belongs to an enhancement.

Comments

@qw4990
Copy link
Contributor

qw4990 commented Sep 12, 2023

Enhancement

As you can see from the diagram below, the StatsHandler(statistics/handle/handle.go:Handle) structure is too complex.
Actually, there are multiple features(modules) decoupled in it, which makes it hard to maintain and evolve.
And these features rely on memory to synchronize information, which brings some risks like whether to lock or not when accessing some fields, which methods be thread-safe, how to synchronize information with other nodes.

To make it easier to maintain, we'd better break it down since obviously maintaining multiple smaller and simpler modules is safer than maintaining a big and complex module.
And only allow them to synchronize information through the storage to avoid consensus problems and thread-safe issues.

image
@qw4990 qw4990 added type/enhancement The issue or PR belongs to an enhancement. sig/planner SIG: Planner component/statistics labels Sep 12, 2023
@hawkingrei hawkingrei mentioned this issue Sep 13, 2023
Merged
12 tasks
@qw4990 qw4990 mentioned this issue Sep 13, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 13, 2023
@hawkingrei
*: unite stats cache's GET (#46939)
2c4e8dc 
ref #46905
ti-chi-bot bot pushed a commit that referenced this issue Sep 14, 2023
@qw4990
planner: refactor to make colStatsUsageMap thread-safe (#46949)
a7617f3 
ref #46905
@hawkingrei hawkingrei mentioned this issue Sep 14, 2023
Merged
12 tasks
@qw4990 qw4990 mentioned this issue Sep 14, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 14, 2023
@qw4990
planner: refactor to make tableStatsDelta thread-safe (#46977)
322104f 
ref #46905
ti-chi-bot bot pushed a commit that referenced this issue Sep 14, 2023
@hawkingrei
handle: add global status handler (#46974)
e890aed 
ref #46905
@qw4990 qw4990 mentioned this issue Sep 15, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 18, 2023
@qw4990
planner: use the session pool to execute SQLs in statshandler (#47007)
bf173eb 
ref #46905
@qw4990 qw4990 mentioned this issue Sep 18, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 18, 2023
@qw4990
planner: create a separate unified function to update variables for s…
4cc6c65 
…tatistics sctx (#47051)

ref #46905
@qw4990 qw4990 mentioned this issue Sep 18, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 19, 2023
@qw4990
planner: use the session pool to execute SQLs in statshandler (#47065)
fdff1e6 
ref #46905
@qw4990 qw4990 mentioned this issue Sep 19, 2023
Merged
13 tasks
@qw4990
Copy link
Contributor Author

qw4990 commented Sep 19, 2023 •
edited
Loading

Handle.mu should be removed, see #47051:

fe247aa0-da36-4d27-bb15-2ec1a85c8c98

qw4990 added a commit to qw4990/tidb that referenced this issue Sep 19, 2023
@qw4990
planner: remove StatsHandler.mu (pingcap#47082)
8f0b9d5 
ref pingcap#46905
@hawkingrei hawkingrei mentioned this issue Sep 20, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 20, 2023
@hawkingrei
statistics: simple globalstats (#47129)
b20237f 
ref #46905
@qw4990 qw4990 mentioned this issue Sep 21, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 22, 2023
@qw4990
planner: move index_usage code into a separate package (#47163)
8fcbcd9 
ref #46905
@qw4990 qw4990 mentioned this issue Sep 22, 2023
Merged
13 tasks
@hawkingrei hawkingrei mentioned this issue Sep 22, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 22, 2023
@qw4990
planner: move auto_analyze code into a separate package (#47194)
47bc019 
ref #46905
ti-chi-bot bot pushed a commit that referenced this issue Sep 22, 2023
@hawkingrei
statistics: remove useless partition stats cache (#47201)
db4ebd7 
ref #46905
@ti-chi-bot ti-chi-bot mentioned this issue Oct 18, 2023
Merged
13 tasks
@qw4990 qw4990 mentioned this issue Oct 18, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 18, 2023
@qw4990
planner: move more methods from StatsHandle to its sub-packages (#47749)
ac1f0d9 
ref #46905
@ti-chi-bot ti-chi-bot mentioned this issue Oct 18, 2023
Closed
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 19, 2023
@qw4990
planner: move more methods from StatsHandle to its sub-packages (#47760)
6dea081 
ref #46905
@ti-chi-bot ti-chi-bot mentioned this issue Oct 19, 2023
Closed
13 tasks
This was referenced Oct 19, 2023
Closed
Merged
ti-chi-bot bot pushed a commit that referenced this issue Oct 20, 2023
@ti-chi-bot
planner: move more methods from StatsHandle to its sub-packages (#47739…
690ffed 
…) (#47751)

ref #46905
@ti-chi-bot ti-chi-bot mentioned this issue Oct 20, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 20, 2023
@qw4990
planner: unify the way to handle txn in statshandle (#47859)
36160d1 
ref #46905
@ti-chi-bot ti-chi-bot mentioned this issue Oct 20, 2023
Merged
13 tasks
@hawkingrei hawkingrei mentioned this issue Oct 23, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 23, 2023
@hawkingrei
statistics: move MergePartTopN2GlobalTopN into handle/globalstats (#4…
98b6c43 
…7901)

ref #46905
@ti-chi-bot ti-chi-bot mentioned this issue Oct 23, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 23, 2023
@ti-chi-bot
planner: unify the way to handle txn in statshandle (#47859) (#47889)
4e79bcb 
ref #46905
ti-chi-bot bot pushed a commit that referenced this issue Oct 23, 2023
@ti-chi-bot
planner: move more methods from StatsHandle to its sub-packages (#47749…
e44dccd 
…) (#47872)

ref #46905
ti-chi-bot bot pushed a commit that referenced this issue Oct 23, 2023
@ti-chi-bot
statistics: move MergePartTopN2GlobalTopN into handle/globalstats (#4…
639f3f2 
…7901) (#47911)

ref #46905
@hawkingrei
Copy link
Member

I think it has been completed.

wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@hawkingrei @wuhuizuo
statistics: move some functions into global stats (pingcap#47669)
5e1d6dd 
ref pingcap#46905
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@qw4990 @wuhuizuo
planner: introduce an interface for StatsHandle to simplify the code (
e121793 
pingcap#47670)

ref pingcap#46905
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@qw4990 @wuhuizuo
planner: move more methods from StatsHandle to its sub-packages (ping…
ed79924 
…cap#47679)

ref pingcap#46905
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@qw4990 @wuhuizuo
planner: move more methods from StatsHandle to its sub-packages (ping…
b73f77c 
…cap#47696)

ref pingcap#46905
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@qw4990 @wuhuizuo
planner: move more methods from StatsHandle to its sub-packages (ping…
49845b2 
…cap#47739)

ref pingcap#46905
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@qw4990 @wuhuizuo
planner: move more methods from StatsHandle to its sub-packages (ping…
5a30584 
…cap#47749)

ref pingcap#46905
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@qw4990 @wuhuizuo
planner: move more methods from StatsHandle to its sub-packages (ping…
fc51cc7 
…cap#47760)

ref pingcap#46905
@qw4990 qw4990 mentioned this issue Apr 28, 2024
Closed
@seiya-annie
Copy link

/found customer

@ti-chi-bot ti-chi-bot bot added the report/customer Customers have encountered this bug. label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hawkingrei hawkingrei

@qw4990 qw4990

Labels
affects-7.5 This bug affects the 7.5.x(LTS) versions. component/statistics report/customer Customers have encountered this bug. sig/planner SIG: Planner type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hawkingrei @qw4990 @seiya-annie @ti-chi-bot