Pre-handle SSTFiles to DTFiles when applying snapshots #1439

JaySon-Huang · 2021-02-19T11:20:14Z

What problem does this PR solve?

Issue Number: a part of #1347

Problem Summary: Pre-decode SSTFiles and save committed data to DTFiles, ingest those DTFiles to Delta-Tree Storage engine.

What is changed and how it works?

Add raft.snapshot.method in configuration file to control the method of applying snapshots
- "block" decode snapshot data as a block
- "file1" decode snapshot data as DTFiles (directory mode) [By default]
- "file2" decode snapshot data as DTFiles (single file mode)
Add a stream SSTFilesToBlockInputStream to decode snapshot data into blocks
Add a stream BoundedSSTFilesToBlockInputStream to bound the blocks read from SSTFilesToBlockInputStream by column _tidb_rowid
Add a stream SSTFilesToDTFilesOutputStream that accept BoundedSSTFilesToBlockInputStream as input stream to write blocks into DTFiles
Fix Ingesting DTFile to DeltaTree storage make it unable to read data #1787 by initializeSubFileStatIfNeeded inside DMFilePackFilter::init
Make STORAGE_FORMAT_CURRENT to be STORAGE_FORMAT_V2 by default to support ingest DTFile into DT engine

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test

Side effects

Performance regression
- Consumes more CPU
- Consumes less MEM
Breaking backward compatibility
- Downgrade to 4.0.x is not support

Release note

Apply Region snapshots by ingesting files to greatly reduce the memory usage

JaySon-Huang · 2021-03-16T13:27:29Z

dbms/src/Storages/FormatVersion.h

@@ -77,7 +77,7 @@ inline static const StorageFormatVersion STORAGE_FORMAT_V2 = StorageFormatVersio
    .page = PageFormat::V2,
 };

-inline StorageFormatVersion STORAGE_FORMAT_CURRENT = STORAGE_FORMAT_V1;
+inline StorageFormatVersion STORAGE_FORMAT_CURRENT = STORAGE_FORMAT_V2;


Note that this PR will enable STORAGE_FORMAT_V2 by default

dbms/src/Storages/Transaction/ApplySnapshot.cpp

JaySon-Huang · 2021-03-18T13:20:52Z

/run-all-tests

JaySon-Huang · 2021-03-22T13:54:36Z

/run-all-tests

JaySon-Huang · 2021-03-23T04:15:59Z

/run-all-tests

JaySon-Huang · 2021-03-26T15:59:31Z

/run-all-tests

Signed-off-by: JaySon-Huang <[email protected]>

JaySon-Huang · 2021-04-29T13:33:11Z

/run-all-tests

JaySon-Huang · 2021-04-29T14:53:28Z

/run-all-tests

Signed-off-by: JaySon-Huang <[email protected]>

flowbehappy

LGTM

JaySon-Huang · 2021-04-30T08:40:36Z

/run-all-tests

ti-srebot · 2021-05-07T02:56:00Z

cherry pick to release-5.0 in PR #1867

* Add `raft.snapshot.method` in configuration file to control the method of applying snapshots - "block" decode snapshot data as a block - "file1" decode snapshot data as DTFiles (directory mode) [**By default**] * Add a stream `SSTFilesToBlockInputStream` to decode snapshot data into blocks * Add a stream `BoundedSSTFilesToBlockInputStream` to bound the blocks read from SSTFilesToBlockInputStream by column `_tidb_rowid` * Add a stream `SSTFilesToDTFilesOutputStream` that accept `BoundedSSTFilesToBlockInputStream` as input stream to write blocks into DTFiles * Make `STORAGE_FORMAT_CURRENT` to be `STORAGE_FORMAT_V2` by default to support ingest DTFile into DT engine * Fix the bug that the block decoded from SSTFile is not sorted by primary key and version (#1888) * Fix panic with feature compaction filter with DTFile * Fix ingest write amplification after snapshot apply optimization (#1913) Co-authored-by: Zhigao Tong <[email protected]> Signed-off-by: JaySon-Huang <[email protected]>

…ap#1439) (pingcap#1867)" This reverts commit c947fd6. Conflicts: dbms/src/Common/FailPoint.cpp dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h dbms/src/Storages/DeltaMerge/File/DMFileWriter.h dbms/src/Storages/DeltaMerge/ReorganizeBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp dbms/src/Storages/DeltaMerge/Segment.cpp dbms/src/Storages/DeltaMerge/StableValueSpace.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_file.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h dbms/src/Storages/Transaction/ApplySnapshot.cpp dbms/src/Storages/Transaction/PartitionStreams.cpp tests/delta-merge-test/raft/schema/drop_on_restart.test

Signed-off-by: ti-srebot <[email protected]> Co-authored-by: Flowyi <[email protected]> cherry pick pingcap#1821 to release-5.0 (pingcap#1827) Signed-off-by: ti-srebot <[email protected]> Co-authored-by: xufei <[email protected]> Remove ReplicatedMergeTree family (pingcap#1805) (pingcap#1826) cherry pick pingcap#1835 to release-5.0 (pingcap#1840) Signed-off-by: ti-srebot <[email protected]> Co-authored-by: xufei <[email protected]> Use file path as cache key (pingcap#1852) (pingcap#1862) Lazily initializing DeltaMergeStore (pingcap#1423) (pingcap#1751) (pingcap#1868) Add background gc check thread for DeltaTree storage (pingcap#1742) (pingcap#1828) * cherry pick pingcap#1742 to release-5.0 Signed-off-by: ti-srebot <[email protected]> * fix conflict Co-authored-by: lidezhu <[email protected]> Fix calculate stable property (pingcap#1839) (pingcap#1878) * cherry pick pingcap#1839 to release-5.0 Signed-off-by: ti-srebot <[email protected]> * fix compile and cherry pick a small fix Co-authored-by: lidezhu <[email protected]> Fix client-c mistake setting current_ts to 0 when resolving lock for async commit (pingcap#1856) (pingcap#1870) support constant folding in dbg coprocessor (pingcap#1881) (pingcap#1884) Optimize & Reduce time cost about ci test (pingcap#1849) (pingcap#1894) * reduce time cost about ci test by parallel * add `-DNO_WERROR=ON` to cmake config for release-darwin build * Fix tidb_ghpr_tics_test fail (pingcap#1895) Signed-off-by: Zhigao Tong <[email protected]> Fix panic with feature `compaction filter` in release-5.0 (pingcap#1891) Init store in background task. (pingcap#1843,pingcap#1896) (pingcap#1874) Push down filter on timestamp type column to storage layer (pingcap#1875) (pingcap#1906) * cherry pick pingcap#1875 to release-5.0 Signed-off-by: ti-srebot <[email protected]> * remove irrevalant code Co-authored-by: lidezhu <[email protected]> Fix ExchangeSender: remove duplicated write stream operation (pingcap#1379) (pingcap#1883) cherry pick pingcap#1917 to release-5.0 (pingcap#1923) Signed-off-by: ti-srebot <[email protected]> Co-authored-by: Fu Zhe <[email protected]> Pre-handle SSTFiles to DTFiles when applying snapshots (pingcap#1439) (pingcap#1867) * Add `raft.snapshot.method` in configuration file to control the method of applying snapshots - "block" decode snapshot data as a block - "file1" decode snapshot data as DTFiles (directory mode) [**By default**] * Add a stream `SSTFilesToBlockInputStream` to decode snapshot data into blocks * Add a stream `BoundedSSTFilesToBlockInputStream` to bound the blocks read from SSTFilesToBlockInputStream by column `_tidb_rowid` * Add a stream `SSTFilesToDTFilesOutputStream` that accept `BoundedSSTFilesToBlockInputStream` as input stream to write blocks into DTFiles * Make `STORAGE_FORMAT_CURRENT` to be `STORAGE_FORMAT_V2` by default to support ingest DTFile into DT engine * Fix the bug that the block decoded from SSTFile is not sorted by primary key and version (pingcap#1888) * Fix panic with feature compaction filter with DTFile * Fix ingest write amplification after snapshot apply optimization (pingcap#1913) Co-authored-by: Zhigao Tong <[email protected]> Signed-off-by: JaySon-Huang <[email protected]> Cast int/real as real (pingcap#1911) (pingcap#1928) Fix the problem that old dmfile is not removed atomically (pingcap#1918) (pingcap#1925) Eliminate the table lock between reading,writing and DDL operators for TiFlash (pingcap#1736) (pingcap#1858) * Eliminate the table lock between reading,writing and DDL operators for TiFlash (pingcap#1736) 1. Port the latest `RWLock` for `IStorage` from Clickhouse. "phase-fair rwlocks have been shown to provide more predictable timings compared to task-fair rwlocks for certain workloads (for relatively low share of writes, <25%). Brandenburg B, Anderson J, (2010)" 2. Refactor the lock model for `IStorage` to eliminate the lock between reading, writing, and DDL operations. 3. Acquire table lock by QueryID/threadName instead of function name 4. Tombstone the database after receiving a "drop" DDL from TiDB. Physically drop the databases / tables after gc safe point. 5. Remove some outdated tests under `tests/mutable-test/txn_schema` (all those tests are ported into `tests/delta-merge-test/raft/schema`) * Mark FIXME for conflict codes Conflicts: dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp Signed-off-by: JaySon-Huang <[email protected]> cached tics code incase of clone same code repeatedly (pingcap#1994) (pingcap#2001) Add row & bytes threshold config for CompactLog (pingcap#1997) (pingcap#2005) * cherry pick pingcap#1997 to release-5.0 Signed-off-by: ti-srebot <[email protected]> Co-authored-by: Zhigao Tong <[email protected]> Fix the bug that incomplete write batches are not truncated (pingcap#1934) (pingcap#2003) Revert "Push down filter on timestamp type column to storage layer (pingcap#1875) (pingcap#1906)" (pingcap#2010) This reverts commit 630b612. Revert Lazily Init Store (pingcap#2011) * Revert "Init store in background task. (pingcap#1843,pingcap#1896) (pingcap#1874)" This reverts commit 0882461. Conflicts: dbms/src/Storages/StorageDeltaMerge.cpp * format code * Revert "Lazily initializing DeltaMergeStore (pingcap#1423) (pingcap#1751) (pingcap#1868)" This reverts commit bbce050. Conflicts: dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h Co-authored-by: JinheLin <[email protected]> Co-authored-by: Flowyi <[email protected]> Revert "background gc thread" (pingcap#2015) * Revert "Fix calculate stable property (pingcap#1839) (pingcap#1878)" This reverts commit 6a9eb58. * Revert "Add background gc check thread for DeltaTree storage (pingcap#1742) (pingcap#1828)" This reverts commit cbbbd09. * remove code * format code * format code * fix test compile * fix comment Fix the potential concurrency problem when clone the shared delta index (pingcap#2030) (pingcap#2033) * cherry pick pingcap#2030 to release-5.0 Signed-off-by: ti-srebot <[email protected]> * Update dbms/src/Storages/DeltaMerge/DeltaIndex.h Co-authored-by: JaySon <[email protected]> Co-authored-by: lidezhu <[email protected]> Co-authored-by: JaySon <[email protected]> Fix bug for unexpected deleted ingest file (pingcap#2047) (pingcap#2048) * Only enable GC for DTFiles after they get applied to all segments. * Refine some loggings Signed-off-by: JaySon-Huang <[email protected]> Fix cast int as time (pingcap#1788) (pingcap#1893) cherry pick pingcap#1903 to release-5.0 (pingcap#1915) Signed-off-by: ti-srebot <[email protected]> Fix ingesting file may add ref page to deleted page (pingcap#2054) (pingcap#2055) 1. Put the ingest file id into `storage_pool.data` before ingesting them into segments 2. Add ref pages to the ingest files for each segment 3. Delete the original page id after all Signed-off-by: JaySon-Huang <[email protected]> Fix compile error on Mac Clang 12.0.5 (pingcap#2058) (pingcap#2070) cherry pick pingcap#2017 pingcap#2020 pingcap#2051 to release-5.0 (pingcap#2074) Revert "cherry pick pingcap#2017 pingcap#2020 pingcap#2051 to release-5.0 (pingcap#2074)" This reverts commit 8e0712c. Revert "Fix compile error on Mac Clang 12.0.5 (pingcap#2058) (pingcap#2070)" This reverts commit 3c31b92. Revert "Fix ingesting file may add ref page to deleted page (pingcap#2054) (pingcap#2055)" This reverts commit c773d61. Revert "Fix bug for unexpected deleted ingest file (pingcap#2047) (pingcap#2048)" This reverts commit b329618. Revert "Pre-handle SSTFiles to DTFiles when applying snapshots (pingcap#1439) (pingcap#1867)" This reverts commit c947fd6. Conflicts: dbms/src/Common/FailPoint.cpp dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h dbms/src/Storages/DeltaMerge/File/DMFileWriter.h dbms/src/Storages/DeltaMerge/ReorganizeBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp dbms/src/Storages/DeltaMerge/Segment.cpp dbms/src/Storages/DeltaMerge/StableValueSpace.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_file.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h dbms/src/Storages/Transaction/ApplySnapshot.cpp dbms/src/Storages/Transaction/PartitionStreams.cpp tests/delta-merge-test/raft/schema/drop_on_restart.test Fix comment Signed-off-by: JaySon-Huang <[email protected]>

* Revert "cherry pick #2017 #2020 #2051 to release-5.0 (#2074)" * Revert "Fix compile error on Mac Clang 12.0.5 (#2058) (#2070)" * Revert "Fix ingesting file may add ref page to deleted page (#2054) (#2055)" * Revert "Fix bug for unexpected deleted ingest file (#2047) (#2048)" * Revert "Pre-handle SSTFiles to DTFiles when applying snapshots (#1439) (#1867)" This reverts commit 8e0712c , 3c31b92 , c773d61 , b329618 , c947fd6 Conflicts: dbms/src/Common/FailPoint.cpp dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h dbms/src/Storages/DeltaMerge/File/DMFileWriter.h dbms/src/Storages/DeltaMerge/ReorganizeBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp dbms/src/Storages/DeltaMerge/Segment.cpp dbms/src/Storages/DeltaMerge/StableValueSpace.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_file.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h dbms/src/Storages/Transaction/ApplySnapshot.cpp dbms/src/Storages/Transaction/PartitionStreams.cpp tests/delta-merge-test/raft/schema/drop_on_restart.test Signed-off-by: JaySon-Huang <[email protected]>

JaySon-Huang requested review from flowbehappy and solotzg February 19, 2021 11:20

JaySon-Huang mentioned this pull request Feb 20, 2021

Refactor some methods for applying snapshots #1444

Merged

ti-srebot mentioned this pull request Feb 20, 2021

Refactor some methods for applying snapshots (#1444) #1446

Merged

JaySon-Huang force-pushed the optimize-handle-sst branch 3 times, most recently from c3392bf to 9b1b03d Compare February 24, 2021 12:14

JaySon-Huang mentioned this pull request Feb 25, 2021

Refine interface and tests of DTFile #1479

Merged

JaySon-Huang force-pushed the optimize-handle-sst branch from 9b1b03d to 4064ad2 Compare February 26, 2021 08:44

JaySon-Huang force-pushed the optimize-handle-sst branch from b4f3b17 to cf53289 Compare March 8, 2021 07:29

JaySon-Huang self-assigned this Mar 8, 2021

JaySon-Huang force-pushed the optimize-handle-sst branch from cf53289 to 9634e48 Compare March 15, 2021 12:16

JaySon-Huang changed the title ~~[DNM] Pre-handle SSTFiles to DTFiles when applying snapshots~~ Pre-handle SSTFiles to DTFiles when applying snapshots Mar 16, 2021

JaySon-Huang commented Mar 16, 2021

View reviewed changes

JaySon-Huang mentioned this pull request Mar 18, 2021

Refactor TiFlashRaftConfig / Define main entry point for gtests_dbms #1583

Merged

This was referenced Mar 18, 2021

Refactor TiFlashRaftConfig / Define main entry point for gtests_dbms (#1583) #1584

Merged

Refactor TiFlashRaftConfig / Define main entry point for gtests_dbms (#1583) #1585

Merged

JaySon-Huang force-pushed the optimize-handle-sst branch from 9a7086b to 24aff54 Compare March 18, 2021 11:17

JaySon-Huang commented Mar 18, 2021

View reviewed changes

dbms/src/Storages/Transaction/ApplySnapshot.cpp Outdated Show resolved Hide resolved

JaySon-Huang added the type/new-feature Issue or PR for new feature label Mar 18, 2021

JaySon-Huang force-pushed the optimize-handle-sst branch from a2475c1 to 4ac3476 Compare March 22, 2021 12:48

This comment has been minimized.

Sign in to view

JaySon-Huang force-pushed the optimize-handle-sst branch 3 times, most recently from 9de1f06 to 3e57877 Compare March 26, 2021 10:05

JaySon-Huang mentioned this pull request Mar 26, 2021

Remove useless StorageDebugging & move mock applying snapshot functions together #1666

Merged

JaySon-Huang added 2 commits April 29, 2021 17:36

Fix bug for decoding schema && schema align

e08de31

Signed-off-by: JaySon-Huang <[email protected]>

Add more test case for ingest file with an older schema

65795d1

Signed-off-by: JaySon-Huang <[email protected]>

JaySon-Huang and others added 4 commits April 30, 2021 04:01

Persist block property for ingested DTFiles

5e5d31a

Signed-off-by: JaySon-Huang <[email protected]>

Remove useless code

258a985

Signed-off-by: JaySon-Huang <[email protected]>

Merge branch 'master' into optimize-handle-sst

a9049d5

Cleanup code

1c3eaac

Signed-off-by: JaySon-Huang <[email protected]>

flowbehappy approved these changes Apr 30, 2021

View reviewed changes

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 30, 2021

JaySon-Huang merged commit 401eb6d into pingcap:master Apr 30, 2021

JaySon-Huang deleted the optimize-handle-sst branch April 30, 2021 09:37

JaySon-Huang mentioned this pull request May 6, 2021

Fix clang warning -Wpessimizing-move and -Wunused-private-field #1866

Merged

flowbehappy added the needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 label May 7, 2021

flowbehappy added this to the v5.0.2 milestone May 7, 2021

ti-srebot mentioned this pull request May 7, 2021

Pre-handle SSTFiles to DTFiles when applying snapshots (#1439) #1867

Merged

JaySon-Huang added type/enhancement The issue or PR belongs to an enhancement. and removed type/new-feature Issue or PR for new feature labels May 13, 2021

JaySon-Huang mentioned this pull request May 17, 2021

Fix ingest write amplification after snapshot apply optimization #1913

Merged

JaySon-Huang removed this from the v5.0.2 milestone May 27, 2021

JaySon-Huang mentioned this pull request Jun 1, 2021

DTFile ingested may be deleted by accident #2040

Closed

JaySon-Huang mentioned this pull request Jun 8, 2021

Check the diff between revert_snapshot and pick need commit JaySon-Huang/tiflash#8

Merged

JaySon-Huang mentioned this pull request Jun 9, 2021

The ingesting sst files is broken while running schrodinger-test/bank #2118

Closed

JaySon-Huang mentioned this pull request Dec 19, 2024

*: Remove RegionPtrWithBlock #9735

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pre-handle SSTFiles to DTFiles when applying snapshots #1439

Pre-handle SSTFiles to DTFiles when applying snapshots #1439

JaySon-Huang commented Feb 19, 2021 •

edited

Loading

JaySon-Huang Mar 16, 2021

JaySon-Huang commented Mar 18, 2021

JaySon-Huang commented Mar 22, 2021

JaySon-Huang commented Mar 23, 2021

This comment has been minimized.

JaySon-Huang commented Mar 26, 2021

JaySon-Huang commented Apr 29, 2021

JaySon-Huang commented Apr 29, 2021

flowbehappy left a comment

JaySon-Huang commented Apr 30, 2021

ti-srebot commented May 7, 2021

Pre-handle SSTFiles to DTFiles when applying snapshots #1439

Pre-handle SSTFiles to DTFiles when applying snapshots #1439

Conversation

JaySon-Huang commented Feb 19, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

JaySon-Huang Mar 16, 2021

Choose a reason for hiding this comment

JaySon-Huang commented Mar 18, 2021

JaySon-Huang commented Mar 22, 2021

JaySon-Huang commented Mar 23, 2021

This comment has been minimized.

JaySon-Huang commented Mar 26, 2021

JaySon-Huang commented Apr 29, 2021

JaySon-Huang commented Apr 29, 2021

flowbehappy left a comment

Choose a reason for hiding this comment

JaySon-Huang commented Apr 30, 2021

ti-srebot commented May 7, 2021

JaySon-Huang commented Feb 19, 2021 •

edited

Loading