Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: RFC for TTL tables #39264

Merged
merged 43 commits into from
Mar 17, 2023
Merged
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
1c327a1
ttl: add rfc for ttl
lcwangchao Nov 18, 2022
6cb2c91
update
lcwangchao Nov 20, 2022
91f78ac
update
lcwangchao Nov 20, 2022
02c30ac
update
lcwangchao Nov 20, 2022
e201649
update
lcwangchao Nov 20, 2022
25850fa
update
lcwangchao Nov 20, 2022
56f9da4
update
lcwangchao Nov 20, 2022
cb6b2d3
add issues
lcwangchao Nov 21, 2022
dbeeba5
Merge branch 'master' into ttl_doc
lcwangchao Nov 21, 2022
d2a9605
update
lcwangchao Nov 21, 2022
3868907
Merge branch 'ttl_doc' of github.com:lcwangchao/tidb into ttl_doc
lcwangchao Nov 21, 2022
bfe2ed7
update
lcwangchao Nov 21, 2022
4867da9
add sysvar `tidb_ttl_enable_instance_worker`
lcwangchao Nov 21, 2022
ce3fb33
Update docs/design/2022-11-17-ttl-table.md
lcwangchao Nov 22, 2022
501a330
Update docs/design/2022-11-17-ttl-table.md
lcwangchao Nov 22, 2022
c75ff9e
Update docs/design/2022-11-17-ttl-table.md
lcwangchao Nov 22, 2022
18fe079
update
lcwangchao Nov 22, 2022
e19989e
Update docs/design/2022-11-17-ttl-table.md
lcwangchao Nov 22, 2022
3511a2b
update
lcwangchao Nov 22, 2022
fa73868
Merge branch 'master' into ttl_doc
lcwangchao Nov 22, 2022
51b87b7
update remove ttl
lcwangchao Nov 22, 2022
29e3726
update document
lcwangchao Nov 23, 2022
d4196fa
update future works
lcwangchao Nov 25, 2022
49b388a
update table
lcwangchao Nov 25, 2022
a0137d3
update
lcwangchao Nov 25, 2022
c2f27da
update timezone
lcwangchao Nov 25, 2022
2d094f2
update
lcwangchao Nov 25, 2022
7200043
Update docs/design/2022-11-17-ttl-table.md
lcwangchao Nov 30, 2022
9301a17
Update docs/design/2022-11-17-ttl-table.md
lcwangchao Nov 30, 2022
3a759ff
update
lcwangchao Nov 30, 2022
4b5aa55
update
lcwangchao Dec 1, 2022
1e6585c
Merge branch 'master' into ttl_doc
bb7133 Dec 1, 2022
630b1e6
update doc
lcwangchao Dec 2, 2022
2020fac
Merge branch 'ttl_doc' of github.com:lcwangchao/tidb into ttl_doc
lcwangchao Dec 2, 2022
a290eea
add test plans
lcwangchao Dec 5, 2022
c0cbec0
update future works
lcwangchao Dec 6, 2022
ed0c3f8
update
lcwangchao Dec 6, 2022
41b45fb
update
lcwangchao Dec 8, 2022
4a0cf1e
update
lcwangchao Dec 12, 2022
f9b0515
update
lcwangchao Dec 13, 2022
88b0077
Merge branch 'master' into ttl_doc
lcwangchao Mar 17, 2023
a202c34
update
lcwangchao Mar 17, 2023
87c3465
update
lcwangchao Mar 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update
lcwangchao committed Dec 12, 2022

Verified

This commit was signed with the committer’s verified signature.
commit 4a0cf1e8b071700a46d62a2b01e809445864a272
8 changes: 4 additions & 4 deletions docs/design/2022-11-17-ttl-table.md
Original file line number Diff line number Diff line change
@@ -48,7 +48,7 @@ The following example shows how to create a TTL table. The column `create_at` is
CREATE TABLE t1 (
id int PRIMARY KEY,
created_at TIMESTAMP
) TTL = `created_at` + INTERVAL 3 MONTHS;
) TTL = `created_at` + INTERVAL 3 MONTH;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we pass -3 MONTH here?

btw, seem here should be "3 MONTHS", plural.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, I prefer Spanner's TTL Clause https://cloud.google.com/spanner/docs/ttl/working-with-ttl#postgresql
creating a TTL "ON" a column may be more straightforward.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we pass -3 MONTH here?

It is allowed currently, but meaningless in most times. If one row is created using the real time, it will be deleted after next job scheduled, so it equals to setting the expire interval to a very small value. If user sets a time in one row with a past time, it may be some different, but I don't know any scene requires it. Maybe we can disallow user to setting it a a negative value or just leave it alone because it will not bring some big problem.

btw, seem here should be "3 MONTHS", plural.

We are using an expression to parse the duration, so, it supports both 3 MONTH and 3 MONTHS. I can update it plural in doc to make it more natural

btw, I prefer Spanner's TTL Clause https://cloud.google.com/spanner/docs/ttl/working-with-ttl#postgresql
creating a TTL "ON" a column may be more straightforward.

I do not have a strong preference, PTAL @SunRunAway

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, I prefer Spanner's TTL Clause https://cloud.google.com/spanner/docs/ttl/working-with-ttl#postgresql
creating a TTL "ON" a column may be more straightforward.

I prefer keeping it as an expression syntax instead of a particular syntax because there may be more syntax extensions in the future if you don't have a strong opinion.

```

We can use another `TTL_ENABLE` option to disable/enable the TTL job for the table. For example:
@@ -57,7 +57,7 @@ We can use another `TTL_ENABLE` option to disable/enable the TTL job for the tab
CREATE TABLE t1 (
id int PRIMARY KEY,
created_at TIMESTAMP
) TTL = `created_at` + INTERVAL 3 MONTHS TTL_ENABLE = 'OFF';
) TTL = `created_at` + INTERVAL 3 MONTH TTL_ENABLE = 'OFF';
```

The above table will not delete expired rows automatically because `TTL_ENABLE` is set to `OFF`. When the `TTL_ENABLE` is omitted, it uses value `ON` by default.
@@ -68,15 +68,15 @@ To make it compatible with mysql, TTL options also support the comment format. F
CREATE TABLE t1 (
id int PRIMARY KEY,
created_at TIMESTAMP
) /*T![ttl] TTL = `created_at` + INTERVAL 3 MONTHS TTL_ENABLE = 'OFF'*/;
) /*T![ttl] TTL = `created_at` + INTERVAL 3 MONTH TTL_ENABLE = 'OFF'*/;
```

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the doc.
Just to confirm one thing: this will work even for virtual column?
Imagine if we have an existing table like below with a lot of data, and created_at is unix timestamp in nanosecond.
CREATE TABLE t1 (
id int PRIMARY KEY,
created_at long
)

We dont want to modify existing schema to add a 'DateTime' or 'Date' field, which may cause a lot of existing data to be rewritten.
So I just want to confirm below way can work as expected:
ALTER TABLE t1 ADD column timestamp_ttl bigint AS (FROM_UNIXTIME(created_at/1000000000)) VIRTUAL;
ALTER TABLE t1 ADD TTL = 'timestamp_ttl + INTERVAL 180 DAYS';

If so, can you please help document that (how to enable that on existing tables using long in a light way(without rewriting data))?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it supports generated column with some limitations. Because currently generated column can not be pushed down, so it make take more performance, we'll optimize it in the future. It is documented: https://github.com/pingcap/tidb/pull/39264/files#diff-3199b62f6b171f846c9c0c1670428817d339e0e2ee2b2a35822df8f532c51e58R202

Copy link

@Yanhan0507 Yanhan0507 Dec 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick response!
Just curious how is pushdown involved here, isn't this just some micro batch scan and delete statements?
If there is indeed a performance overhead even for this scan+delete approach, is it possible to just support column with the type of LONG (unix timestamp)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lcwangchao
Could you add a section about "generated column" in this document to elaborate clearly?
We can talk about the compatibility about the generated column and the solution to optimize the generated column calculation.


#### Alter a Table with TTL

We can alter an exist table with TTL options, for example:

```sql
ALTER TABLE t1 TTL = `created_at` + INTERVAL 3 MONTHS;
ALTER TABLE t1 TTL = `created_at` + INTERVAL 3 MONTH;
```

We should allow to update the existed TTL options. When it is updated, the running background job for this table should stop or restart according to the newest settings.