Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLASH-386] DeltaMerge DDL support #190

Merged
merged 36 commits into from
Aug 28, 2019
Merged

[FLASH-386] DeltaMerge DDL support #190

merged 36 commits into from
Aug 28, 2019

Conversation

JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Aug 21, 2019 •
edited by flowbehappy
Loading

Design doc:
https://docs.google.com/document/d/1m2wqvr7QEsv7w4uRGlWzgX8IEb5nuV2iwH_sJSIP2qY/edit#

  • DM create table statement now accept TiDB::TableInfo as optional second param
  • Support for applying alter on DM tables. Details are in the design doc
    • Add Column
    • Drop Column
    • Modify Column DataType (including nullable <-> not null)
    • Rename Table
    • Rename Column
    • Create/Modify Column with non 'zero' default value
  • DM readChunkData now support read chunk column data as column_defines.
    If there are different between column_defines and chunk.columns, it will do cast / fill missing values as need
  • Add some test cases in tests/delta_merge

TODO:

  • Rename column name (FLASH-452)
  • Create/alter column with default value, which may change to a 'CAST' expr in AlterCommand::validate, will throw exception (FLASH-453)

@JaySon-Huang
cast DataType while reading from PageStorage
8d6925b
@JaySon-Huang
add isLossyCast function
003982b
@JaySon-Huang
alter for StroageDeltaMerge, WIP
2684327
@JaySon-Huang
add Alter for StorageDeltaMerge
1cbc939
@JaySon-Huang
add Alter for StorageDeltaMerge
469812e
@JaySon-Huang
add TableInfo in StroageDeltaMerge
d7c3145
@JaySon-Huang
rename table for StorageDeltaMerge
51bde2d
@JaySon-Huang
fix bug: the TableInfo from TiDB
e777080
@JaySon-Huang
add comments for DeltaMerge flush && cache
24ce541
@JaySon-Huang
more faster(?) way to cast mismatch datatype
d4cb785
@JaySon-Huang
support cast for numeric type null/not null
98bbd37
@JaySon-Huang
support for other data type just change null / not null
3ce7811
@JaySon-Huang
isLossyCast -> isSupportedDataTypeCast
45393ad
@JaySon-Huang
isSupportedDataTypeCast add decimal detect
c667ac7
@JaySon-Huang
rename function
3b6b419
@JaySon-Huang
fix compile errors in gtests
0179ff6
@JaySon-Huang
small fix
16c88c5
@JaySon-Huang
fix broken tests
4d5532a
@JaySon-Huang
support new column with non-zero default value
d2bbdc8
@JaySon-Huang
remove unused code
3169cbc
@JaySon-Huang
fix compile error in CI
536d0ae
@JaySon-Huang
fix bug in table rename
6e2da1a
@JaySon-Huang
small fix
8b5c4f7
@JaySon-Huang
minor fix
b2c1822
@JaySon-Huang
refine cast function in chunk
60ea794
@JaySon-Huang
update DeltaMergeStore's segments within lock
9a3a06f
@JaySon-Huang
[WIP]Add test cases for default value ddl. Still has bugs
bb2c552
JaySon-Huang
JaySon-Huang commented Aug 27, 2019
View reviewed changes
dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp
}
}

void DeltaMergeStore::flushCache(const Context & db_context)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flowbehappy please take a look at here, should we acquire for a lock here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DeltaMergeStore::flushCache does not need any lock. Because It does not update segments structure. But Segment:: flushCache do need a std::unique_lock lock(read_write_mutex) to synchronize between read threads on the Segment objects.

@JaySon-Huang
Copy link
Contributor Author

/run-integration-tests

@JaySon-Huang
Copy link
Contributor Author

/run-integration-tests

@JaySon-Huang JaySon-Huang merged commit e67dbf7 into pingcap:DeltaMergeEngine Aug 28, 2019
@JaySon-Huang JaySon-Huang deleted the dm-ddl branch August 28, 2019 06:37
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Sep 5, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (pingcap#190)
ebef65f 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Sep 12, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (pingcap#190)
534a017 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Sep 18, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (pingcap#190)
6b4d7af 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Sep 23, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (pingcap#190)
06ec850 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit that referenced this pull request Sep 26, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (#190)
848eb4e 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Oct 17, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (pingcap#190)
3584d68 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Oct 18, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (pingcap#190)
4d5f222 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit that referenced this pull request Oct 22, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (#190)
9ba1627 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Oct 23, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (pingcap#190)
5c690a9 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Oct 30, 2019
@JaySon-Huang
[FLASH-386] DeltaMerge DDL support (pingcap#190)
80b3e76 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Nov 1, 2019
@JaySon-Huang @zanmato1984
[FLASH-386] DeltaMerge DDL support (pingcap#190)
83f2e73 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Nov 1, 2019
@JaySon-Huang @zanmato1984
[FLASH-386] DeltaMerge DDL support (pingcap#190)
e6a12be 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Nov 5, 2019
@JaySon-Huang @zanmato1984
[FLASH-386] DeltaMerge DDL support (pingcap#190)
c281052 
* cast DataType while reading from PageStorage

* add isLossyCast function

* alter for StroageDeltaMerge, WIP

* add Alter for StorageDeltaMerge

* add Alter for StorageDeltaMerge

* add TableInfo in StroageDeltaMerge

* rename table for StorageDeltaMerge

* fix bug: the TableInfo from TiDB

* add comments for DeltaMerge flush && cache

* more faster(?) way to cast mismatch datatype

* support cast for numeric type null/not null

* support for other data type just change null / not null

* isLossyCast -> isSupportedDataTypeCast

* isSupportedDataTypeCast add decimal detect

* rename function

* fix compile errors in gtests

* small fix

* fix broken tests

* support new column with non-zero default value

* remove unused code

* fix compile error in CI

* fix bug in table rename

* small fix

* minor fix

* refine cast function in chunk

* update DeltaMergeStore's segments within lock

* [WIP]Add test cases for default value ddl. Still has bugs

* fix bugs after rebasing to latest master

* add some TODO marks

* fix compile error in gtests && remove unused comments

* fix broken gtests

* flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply

* use TypeIndex instead of typeid_cast

* clean up data after tests

* address comment
JaySon-Huang pushed a commit that referenced this pull request Aug 6, 2024
@breezewish @JaySon-Huang
storage: Add vector search metrics (#190)
4acd841 
Signed-off-by: Wish <[email protected]>
Lloyd-Pottiger pushed a commit to Lloyd-Pottiger/tiflash that referenced this pull request Aug 26, 2024
@breezewish @Lloyd-Pottiger
storage: Add vector search metrics (pingcap#190)
900efa4 
Signed-off-by: Wish <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@flowbehappy flowbehappy Awaiting requested review from flowbehappy

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JaySon-Huang @flowbehappy