Export PageStorage V3 config into global config #4502

jiaqizho · 2022-03-30T06:18:38Z

What problem does this PR solve?

Issue Number: ref #3594

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-03-30T06:18:39Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Lloyd-Pottiger
flowbehappy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

dbms/src/Storages/Page/PageStorage.h

jiaqizho · 2022-03-31T05:50:50Z

need added needSave to config.

CLAassistant · 2022-04-01T05:55:19Z

All committers have signed the CLA.

CLAassistant · 2022-04-01T05:56:03Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

jiaqizho · 2022-04-06T05:19:17Z

/run-all-tests

sre-bot · 2022-04-06T05:47:57Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Interpreters/Settings.h                                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Server/tests/gtest_server_config.cpp                   2591               390    84.95%          10                 0   100.00%         465                 3    99.35%         802               398    50.37%
Storages/DeltaMerge/StoragePool.cpp                      67                14    79.10%          17                 6    64.71%         150                37    75.33%          34                12    64.71%
Storages/Page/ConfigSettings.cpp                         10                 0   100.00%           2                 0   100.00%          26                 0   100.00%           8                 4    50.00%
Storages/Page/PageDefines.h                               7                 2    71.43%           7                 2    71.43%          13                 4    69.23%           0                 0         -
Storages/Page/PageStorage.h                              28                 9    67.86%          28                 9    67.86%         103                35    66.02%           0                 0         -
Storages/Page/V3/PageDirectoryFactory.cpp                39                 5    87.18%           4                 0   100.00%         117                15    87.18%          46                 5    89.13%
Storages/Page/V3/PageDirectoryFactory.h                   2                 0   100.00%           2                 0   100.00%           8                 0   100.00%           0                 0         -
Storages/Page/V3/PageStorageImpl.cpp                     87                25    71.26%          21                 7    66.67%         204                65    68.14%          46                24    47.83%
Storages/Page/V3/PageStorageImpl.h                        8                 2    75.00%           8                 2    75.00%          20                 2    90.00%           0                 0         -
Storages/Page/V3/WAL/WALReader.cpp                       72                10    86.11%           9                 0   100.00%         173                15    91.33%          54                14    74.07%
Storages/Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Storages/Page/V3/WALStore.cpp                            83                48    42.17%           8                 1    87.50%         138                43    68.84%          38                21    44.74%
Storages/Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Storages/Page/V3/tests/gtest_page_directory.cpp        8833              1575    82.17%          50                 0   100.00%        1641                 7    99.57%        2774              1405    49.35%
Storages/Page/V3/tests/gtest_wal_store.cpp             1948               260    86.65%          12                 0   100.00%         465                13    97.20%         646               314    51.39%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                 13789              2343    83.01%         184                28    84.78%        3546               246    93.06%        4454              2199    50.63%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17013      9513             44.08%    191770  96418        49.72%

full coverage report (for internal network access only)

jiaqizho · 2022-04-06T06:46:33Z

/run-all-tests

sre-bot · 2022-04-06T07:15:17Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Interpreters/Settings.h                                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Server/tests/gtest_server_config.cpp                   2726               403    85.22%          10                 0   100.00%         479                 3    99.37%         844               419    50.36%
Storages/DeltaMerge/StoragePool.cpp                      67                14    79.10%          17                 6    64.71%         151                37    75.50%          34                12    64.71%
Storages/Page/ConfigSettings.cpp                         10                 0   100.00%           2                 0   100.00%          27                 0   100.00%           8                 4    50.00%
Storages/Page/PageDefines.h                               7                 2    71.43%           7                 2    71.43%          13                 4    69.23%           0                 0         -
Storages/Page/PageStorage.h                              28                 8    71.43%          28                 8    71.43%         105                22    79.05%           0                 0         -
Storages/Page/V2/gc/DataCompactor.cpp                   216                84    61.11%          10                 0   100.00%         371               109    70.62%         140                70    50.00%
Storages/Page/V3/PageDirectory.cpp                      456                64    85.96%          31                 2    93.55%        1009               183    81.86%         374                83    77.81%
Storages/Page/V3/PageDirectory.h                         25                 6    76.00%          25                 6    76.00%         104                21    79.81%           0                 0         -
Storages/Page/V3/PageDirectoryFactory.cpp                39                 5    87.18%           4                 0   100.00%         117                15    87.18%          46                 5    89.13%
Storages/Page/V3/PageDirectoryFactory.h                   2                 0   100.00%           2                 0   100.00%           8                 0   100.00%           0                 0         -
Storages/Page/V3/PageStorageImpl.cpp                     96                26    72.92%          21                 7    66.67%         205                65    68.29%          50                26    48.00%
Storages/Page/V3/PageStorageImpl.h                        8                 2    75.00%           8                 2    75.00%          21                 2    90.48%           0                 0         -
Storages/Page/V3/WAL/WALReader.cpp                       72                10    86.11%           9                 0   100.00%         173                15    91.33%          54                14    74.07%
Storages/Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Storages/Page/V3/WALStore.cpp                            83                48    42.17%           8                 1    87.50%         138                43    68.84%          38                21    44.74%
Storages/Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Storages/Page/V3/tests/gtest_page_directory.cpp        8833              1575    82.17%          50                 0   100.00%        1641                 7    99.57%        2774              1405    49.35%
Storages/Page/V3/tests/gtest_wal_store.cpp             1948               260    86.65%          12                 0   100.00%         465                13    97.20%         646               314    51.39%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                 14630              2510    82.84%         250                35    86.00%        5050               546    89.19%        5014              2375    52.63%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17013      9512             44.09%    191802  96411        49.73%

full coverage report (for internal network access only)

dbms/src/Storages/Page/V3/WAL/WALReader.cpp

jiaqizho · 2022-04-29T04:10:36Z

/merge

ti-chi-bot · 2022-04-29T04:10:37Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-04-29T04:10:38Z

This pull request has been accepted and is ready to merge.

Commit hash: 61d74bf

ti-chi-bot · 2022-04-29T04:10:51Z

@jiaqizho: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

yibin87 · 2022-04-29T06:19:28Z

Unit test failed due to unstable gtest, just rerun and ignore it.

jiaqizho · 2022-04-29T06:33:03Z

/merge

ti-chi-bot · 2022-04-29T06:33:04Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

jiaqizho · 2022-04-29T06:33:40Z

Unit test failed due to unstable gtest, just rerun and ignore it.

@yibin87 what should I do now?

sre-bot · 2022-04-29T06:49:26Z

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Interpreters/Settings.h                                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Server/tests/gtest_server_config.cpp                   2051               338    83.52%          10                 0   100.00%         409                 3    99.27%         634               314    50.47%
Storages/DeltaMerge/StoragePool.cpp                      67                14    79.10%          17                 6    64.71%         146                37    74.66%          34                12    64.71%
Storages/Page/ConfigSettings.cpp                          6                 0   100.00%           2                 0   100.00%          18                 0   100.00%           4                 2    50.00%
Storages/Page/PageDefines.h                               8                 2    75.00%           8                 2    75.00%          14                 4    71.43%           0                 0         -
Storages/Page/PageStorage.h                              28                 8    71.43%          28                 8    71.43%         105                22    79.05%           0                 0         -
Storages/Page/V2/gc/DataCompactor.cpp                   216                84    61.11%          10                 0   100.00%         371               109    70.62%         140                70    50.00%
Storages/Page/V3/PageDirectory.cpp                      504                78    84.52%          31                 2    93.55%        1063               187    82.41%         394                94    76.14%
Storages/Page/V3/PageDirectory.h                         24                 4    83.33%          24                 4    83.33%         101                13    87.13%           0                 0         -
Storages/Page/V3/PageDirectoryFactory.cpp                39                 5    87.18%           4                 0   100.00%         117                15    87.18%          46                 5    89.13%
Storages/Page/V3/PageDirectoryFactory.h                   2                 0   100.00%           2                 0   100.00%           8                 0   100.00%           0                 0         -
Storages/Page/V3/PageStorageImpl.cpp                    114                36    68.42%          21                 7    66.67%         243                75    69.14%          58                32    44.83%
Storages/Page/V3/PageStorageImpl.h                        8                 2    75.00%           8                 2    75.00%          21                 2    90.48%           0                 0         -
Storages/Page/V3/WAL/WALReader.cpp                       72                10    86.11%           9                 0   100.00%         173                15    91.33%          54                14    74.07%
Storages/Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Storages/Page/V3/WALStore.cpp                            81                46    43.21%           9                 2    77.78%         150                55    63.33%          36                19    47.22%
Storages/Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Storages/Page/V3/tests/gtest_page_directory.cpp        8833              1575    82.17%          50                 0   100.00%        1646                 7    99.57%        2774              1405    49.35%
Storages/Page/V3/tests/gtest_wal_store.cpp             1948               260    86.65%          12                 0   100.00%         465                13    97.20%         646               314    51.39%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                 14015              2465    82.41%         251                34    86.45%        5073               564    88.88%        4826              2283    52.69%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18058      10148            43.80%    200136  100960       49.55%

full coverage report (for internal network access only)

Export PageStorage V3 config into global config

6708428

jiaqizho requested a review from JaySon-Huang March 30, 2022 06:18

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 30, 2022

jiaqizho changed the title ~~Export PageStorage V3 config into global config~~ [WIP]Export PageStorage V3 config into global config Mar 30, 2022

ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 30, 2022

jiaqizho commented Mar 30, 2022

View reviewed changes

dbms/src/Storages/Page/PageStorage.h Show resolved Hide resolved

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 31, 2022

merge to master

2006d7a

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 6, 2022

export to dt

13bc1df

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 6, 2022

change need save

5cd4af7

jiaqizho changed the title ~~[WIP]Export PageStorage V3 config into global config~~ Export PageStorage V3 config into global config Apr 6, 2022

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 6, 2022

jiaqizho requested review from lidezhu and CalvinNeo April 7, 2022 08:30

only export dt_storage_blob_heavy_gc_valid_rate to upper layer

61d74bf

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 28, 2022

jiaqizho requested a review from Lloyd-Pottiger April 28, 2022 04:46

Lloyd-Pottiger reviewed Apr 28, 2022

View reviewed changes

dbms/src/Storages/Page/V3/WAL/WALReader.cpp Show resolved Hide resolved

flowbehappy self-requested a review April 29, 2022 02:54

flowbehappy approved these changes Apr 29, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 29, 2022

Lloyd-Pottiger approved these changes Apr 29, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 29, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 29, 2022

Merge branch 'master' into export-v3-config

e43a8a3

ti-chi-bot merged commit 63cae97 into pingcap:master Apr 29, 2022

jiaqizho deleted the export-v3-config branch April 29, 2022 07:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Export PageStorage V3 config into global config #4502

Export PageStorage V3 config into global config #4502

jiaqizho commented Mar 30, 2022

ti-chi-bot commented Mar 30, 2022 •

edited

Loading

jiaqizho commented Mar 31, 2022

CLAassistant commented Apr 1, 2022 •

edited

Loading

CLAassistant commented Apr 1, 2022

jiaqizho commented Apr 6, 2022

sre-bot commented Apr 6, 2022

jiaqizho commented Apr 6, 2022

sre-bot commented Apr 6, 2022

jiaqizho commented Apr 29, 2022

ti-chi-bot commented Apr 29, 2022

ti-chi-bot commented Apr 29, 2022

ti-chi-bot commented Apr 29, 2022

yibin87 commented Apr 29, 2022

jiaqizho commented Apr 29, 2022

ti-chi-bot commented Apr 29, 2022

jiaqizho commented Apr 29, 2022

sre-bot commented Apr 29, 2022

Export PageStorage V3 config into global config #4502

Export PageStorage V3 config into global config #4502

Conversation

jiaqizho commented Mar 30, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Mar 30, 2022 • edited Loading

jiaqizho commented Mar 31, 2022

CLAassistant commented Apr 1, 2022 • edited Loading

CLAassistant commented Apr 1, 2022

jiaqizho commented Apr 6, 2022

sre-bot commented Apr 6, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Apr 6, 2022

sre-bot commented Apr 6, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Apr 29, 2022

ti-chi-bot commented Apr 29, 2022

ti-chi-bot commented Apr 29, 2022

ti-chi-bot commented Apr 29, 2022

yibin87 commented Apr 29, 2022

jiaqizho commented Apr 29, 2022

ti-chi-bot commented Apr 29, 2022

jiaqizho commented Apr 29, 2022

sre-bot commented Apr 29, 2022

Coverage for changed files

Coverage summary

ti-chi-bot commented Mar 30, 2022 •

edited

Loading

CLAassistant commented Apr 1, 2022 •

edited

Loading