[Enhancement] Support move truncated old data to recycle bin #43107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
TPC-H: Total hot run time: 41673 ms |
TPC-DS: Total hot run time: 196664 ms |
ClickBench: Total hot run time: 32.42 s |
Vallishp
force-pushed
the
truncate_recycle
branch
from
0b29c92 to
cd197f8
Compare
|
run buildall |
TPC-H: Total hot run time: 41504 ms |
TPC-DS: Total hot run time: 196727 ms |
ClickBench: Total hot run time: 32.88 s |
Vallishp
force-pushed
the
truncate_recycle
branch
from
cd197f8 to
dffdb0d
Compare
|
run buildall |
TPC-H: Total hot run time: 41109 ms |
TPC-DS: Total hot run time: 195175 ms |
ClickBench: Total hot run time: 32.05 s |
dataroaring
reviewed
Nov 2, 2024
| } | ||
|
|
||
| public TableRef getTblRef() { | ||
| return tblRef; | ||
| } | ||
|
|
||
| public boolean isForceDrop() { | ||
| return forceDrop; | ||
| } |
Vallishp
force-pushed
the
truncate_recycle
branch
from
dffdb0d to
3946ee4
Compare
|
run buildall |
TPC-H: Total hot run time: 41369 ms |
TPC-DS: Total hot run time: 196468 ms |
ClickBench: Total hot run time: 32.11 s |
dataroaring
requested changes
Nov 9, 2024
6 tasks
yujun777
reviewed
Nov 11, 2024
| @@ -1196,6 +1196,75 @@ public Partition dropPartition(long dbId, String partitionName, boolean isForceD | |||
| return dropPartition(dbId, partitionName, isForceDrop, !isForceDrop); | |||
| } | |||
|
|
|||
| public Partition dropPartitionForTruncate(long dbId, boolean isForceDrop, | |||
There was a problem hiding this comment.
OlapTable has a function dropPartition, its code seem much like this, can refactor code to reuse them ?
There was a problem hiding this comment.
thank you. i will refactor it
yujun777
reviewed
Nov 11, 2024
| // 2. If "ifForceDrop" is true, the partition will be dropped immediately | ||
| Partition partition = recyclePartitionParam.partition; | ||
| if (partition != null) { | ||
| idToPartition.remove(partition.getId()); |
There was a problem hiding this comment.
need add nameToPartition.remove(partitionName), just like OlapTable.dropPartition function.
so refactor dropPartition and dropPartitionForTruncate to reuse code;
There was a problem hiding this comment.
replacePartition had already removed old partition from nameToPartition and added new partition created by truncate process. so it should not be called in truncate process.
yujun777
reviewed
Nov 11, 2024
| truncateTableInternal(olapTable, info.getPartitions(), info.isEntireTable()); | ||
|
|
||
| // add tablet to inverted index | ||
| TabletInvertedIndex invertedIndex = Env.getCurrentInvertedIndex(); |
There was a problem hiding this comment.
this code is to add tablets meta of the new partitions to TabletInvertIndex.
why delete this code
There was a problem hiding this comment.
this code is to add tablets meta of the new partitions to TabletInvertIndex. why delete this code
Thanks. its adding the tabletInvertIndex for info.getPartitions() , which is old partitions. i think now it is not necessary to added.
i have tested like,
truncate table, restart FE , and check table content and recover table from recycle bin.
Any thing extra need to test?
There was a problem hiding this comment.
after truncating table, restart fe， have you check TabletInvertIndex contains all new tablets of this Table ?
In my options, line 3786 ~ 3806 should not delete. It's used to add new partitions's tablet meta into TabletInvertIndex.
You can see function InternalCatalog.replayAddPartition, it's doing the same things, for replaying the new partition, it should put its tablet meta into TabletInvertIndex.
Vallishp
force-pushed
the
truncate_recycle
branch
from
3946ee4 to
8a2e347
Compare
|
run buildall |
1 similar comment
|
run buildall |
Vallishp
force-pushed
the
truncate_recycle
branch
from
14c8dd7 to
7addadd
Compare
|
run buildall |
TPC-H: Total hot run time: 39821 ms |
TPC-DS: Total hot run time: 196996 ms |
ClickBench: Total hot run time: 33.3 s |
Vallishp
force-pushed
the
truncate_recycle
branch
from
7addadd to
e783858
Compare
|
run buildall |
TPC-H: Total hot run time: 39925 ms |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
TPC-DS: Total hot run time: 191583 ms |
ClickBench: Total hot run time: 32.55 s |
|
run p0 |
1 similar comment
|
run p0 |
Vallishp
force-pushed
the
truncate_recycle
branch
from
d916a66 to
65b69a8
Compare
|
run buildall |
github-actions
bot
removed
the
approved
TPC-H: Total hot run time: 39678 ms |
TPC-DS: Total hot run time: 198474 ms |
ClickBench: Total hot run time: 33.04 s |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Added a new option for truncate table force.
if force is used then data will be deleted immediately.
if no force then data will be moved to recycle bin and use gets a chance to recover later if needed.
Issue Number: close #43104
Related PR: #xxx
NA
Problem Summary:
Check List (For Committer)
Test
mysql> use test;
Database changed
mysql> select * from test1;
+------+------+------------+
| id | name | da |
+------+------+------------+
| 1 | a | 2022-01-02 |
| 1 | a | 2022-01-02 |
+------+------+------------+
2 rows in set (0.03 sec)
mysql> show catalog recycle bin;
Empty set (0.02 sec)
mysql> truncate table test1 force;
Query OK, 0 rows affected (0.23 sec)
mysql> show catalog recycle bin;
Empty set (0.00 sec)
mysql> select * from test5;
+------+------+------------+
| id | name | da |
+------+------+------------+
| 2 | a | 2023-01-02 |
| 1 | a | 2022-01-02 |
| 1 | a | 2022-01-02 |
+------+------+------------+
3 rows in set (0.04 sec)
mysql> truncate table test5;
Query OK, 0 rows affected (0.10 sec)
mysql> show catalog recycle bin;
+-----------+------+-------+---------+-------------+---------------------+-----------+----------------+
| Type | Name | DbId | TableId | PartitionId | DropTime | DataSize | RemoteDataSize |
+-----------+------+-------+---------+-------------+---------------------+-----------+----------------+
| Partition | p3 | 10021 | 12023 | 12020 | 2024-11-01 13:07:25 | 1.307 KB | 0.000 |
| Partition | p4 | 10021 | 12023 | 12021 | 2024-11-01 13:07:25 | 667.000 B | 0.000 |
+-----------+------+-------+---------+-------------+---------------------+-----------+----------------+
2 rows in set (0.00 sec)
Behavior changed:
After Truncate table, older version delete data directly. newer version old data will be moved to recycle bin
Does this need documentation?
Release note
None
Check List (For Reviewer who merge this PR)