Complete lightning env initlize procdeure #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
license-eye has totally checked 3528 files.
| Valid | Invalid | Ignored | Fixed |
|---|---|---|---|
| 1579 | 9 | 1940 | 0 |
Click to see the invalid file list
- ddl/addindex/index.go
- ddl/index_lightning.go
- ddl/lightning/backend.go
- ddl/lightning/engine.go
- ddl/lightning/engine_mgr.go
- ddl/lightning/kv.go
- ddl/lightning/mem_mgr.go
- ddl/lightning/metrics.go
- table/tables/util/util.go
Benjamin2037
pushed a commit
that referenced
this pull request
Jun 1, 2022
create sysvar TiDBFastDDL (#3) * Change TiDBFastDDL to Global scope. * TiDBFastDDL Variable usage 1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job. 2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process. Complete lightning env initlize procdeure (#4) 1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test uniform lightning related errmessage text to lightning_error file. (#5) complete memory track module work. refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. Add log for lightning processing. refactor some code` complete memory track module work. (#7) complete user cancel and exception part logic implement * complete memory track module work. * refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. * Add log for lightning processing. refactor some code` restore logic first stage adjust metric of ddl process progress value for lightning solution refine import cycle set up disk quota refine code and add some ut and ft. Add config paramemter TiDBlightningSortPath for setting sort parth for add index. fix mem reclaim problems Add log infromation combine the optimizes: 1, prundecode 2, json expression Add some log information
Benjamin2037
pushed a commit
that referenced
this pull request
Jun 1, 2022
create sysvar TiDBFastDDL (#3) * Change TiDBFastDDL to Global scope. * TiDBFastDDL Variable usage 1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job. 2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process. Complete lightning env initlize procdeure (#4) 1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test uniform lightning related errmessage text to lightning_error file. (#5) complete memory track module work. refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. Add log for lightning processing. refactor some code` complete memory track module work. (#7) complete user cancel and exception part logic implement * complete memory track module work. * refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. * Add log for lightning processing. refactor some code` restore logic first stage adjust metric of ddl process progress value for lightning solution refine import cycle set up disk quota refine code and add some ut and ft. Add config paramemter TiDBlightningSortPath for setting sort parth for add index. fix mem reclaim problems Add log infromation combine the optimizes: 1, prundecode 2, json expression Add some log information
Benjamin2037
added a commit
that referenced
this pull request
Jun 1, 2022
create sysvar TiDBFastDDL (#3) * Change TiDBFastDDL to Global scope. * TiDBFastDDL Variable usage 1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job. 2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process. Complete lightning env initlize procdeure (#4) 1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test uniform lightning related errmessage text to lightning_error file. (#5) complete memory track module work. refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. Add log for lightning processing. refactor some code` complete memory track module work. (#7) complete user cancel and exception part logic implement * complete memory track module work. * refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. * Add log for lightning processing. refactor some code` restore logic first stage adjust metric of ddl process progress value for lightning solution refine import cycle set up disk quota refine code and add some ut and ft. Add config paramemter TiDBlightningSortPath for setting sort parth for add index. fix mem reclaim problems Add log infromation combine the optimizes: 1, prundecode 2, json expression Add some log information
Benjamin2037
added a commit
that referenced
this pull request
Jun 15, 2022
create sysvar TiDBFastDDL (#3) * Change TiDBFastDDL to Global scope. * TiDBFastDDL Variable usage 1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job. 2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process. Complete lightning env initlize procdeure (#4) 1. Get max open file limition for lightning, default 1024 2. Set light sorted path： 2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning 2.2 otherwise set path to DataDir + "lightning" Check whether the lightning sorted path is exist and a dir, if yes then keep use it. otherwise create new one. 3, Set the lightning dir quota, default 10G Add one unit test uniform lightning related errmessage text to lightning_error file. (#5) complete memory track module work. refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. Add log for lightning processing. refactor some code` complete memory track module work. (#7) complete user cancel and exception part logic implement * complete memory track module work. * refacter memmory manager to resource manager for expand to control concurrent base available cpu core number. Finsih integrate with lightning concurrency process logic. * Add log for lightning processing. refactor some code` restore logic first stage adjust metric of ddl process progress value for lightning solution refine import cycle set up disk quota refine code and add some ut and ft. Add config paramemter TiDBlightningSortPath for setting sort parth for add index. fix mem reclaim problems Add log infromation combine the optimizes: 1, prundecode 2, json expression Add some log information
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning
2.2 otherwise set path to DataDir + "lightning"
Check whether the lightning sorted path is exist and a dir, if yes then keep use it.
otherwise create new one.
3, Set the lightning dir quota, default 10G
Add one unit test
What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
What is changed and how it works?
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.