*: add global variable tidb_slow_log_masking to control masking slow log query #17637

crazycs520 · 2020-06-03T10:43:01Z

Signed-off-by: crazycs [email protected]

What problem does this PR solve?

Issue Number: close #17463

This PR adds global variable tidb_slow_log_masking to control masking slow log query.

Such as below:

Enable the tidb_slow_log_masking first.

 set @@global.tidb_slow_log_masking=1;

Then, when TiDB record the slow log, it will masking the user data of SQL,
just like SQL normalize, such as below:

# Time: 2020-06-03T18:28:39.886171+08:00
...
...
...
select count ( ? ) , sum ( query_time ) , digest from information_schema . cluster_slow_query where time >= ? and time < ? group by digest order by sum ( query_time ) desc;

The original SQL is:

select count(*), sum(query_time), digest from information_schema.CLUSTER_SLOW_QUERY where time >= '2020-03-05 20:55:00' and time < '2020-06-05 20:57:00' group by digest order by sum(query_time) desc;

What is changed and how it works?

Related changes

PR to update pingcap/docs/pingcap/docs-cn:

Check List

Tests

Unit test
Manual test

Side effects

No

Release note

add global variable tidb_slow_log_masking to control masking slow log query.

…log query Signed-off-by: crazycs <[email protected]>

codecov · 2020-06-03T10:50:23Z

Codecov Report

Merging #17637 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #17637   +/-   ##
===========================================
  Coverage   79.4969%   79.4969%           
===========================================
  Files           524        524           
  Lines        141413     141413           
===========================================
  Hits         112419     112419           
  Misses        19930      19930           
  Partials       9064       9064

Signed-off-by: crazycs <[email protected]>

crazycs520 · 2020-06-03T14:23:00Z

/run-all-tests

bb7133

LGTM

wjhuang2016

LGTM

bb7133 · 2020-06-04T07:34:34Z

/run-all-tests

jackysp · 2020-06-04T07:39:55Z

/merge

sre-bot · 2020-06-04T07:39:58Z

Sorry @jackysp, you don't have permission to trigger auto merge event on this branch.

bb7133 · 2020-06-04T07:54:58Z

Data Race: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/tidb_ghpr_unit_test/detail/tidb_ghpr_unit_test/38689/pipeline

bb7133 · 2020-06-04T07:55:10Z

/run-unit-test /run-integration-br-test

AndreMouche · 2020-06-04T07:57:38Z

/merge

sre-bot · 2020-06-04T07:57:41Z

Sorry @AndreMouche, you don't have permission to trigger auto merge event on this branch. You are not a committer for this part

jackysp · 2020-06-04T08:02:46Z

/merge

sre-bot · 2020-06-04T08:02:48Z

Sorry @jackysp, you don't have permission to trigger auto merge event on this branch. You are not a committer for this part

jackysp · 2020-06-04T08:03:33Z

/merge

sre-bot · 2020-06-04T08:03:35Z

Sorry @jackysp, you don't have permission to trigger auto merge event on this branch. You are not a committer for this part

jackysp · 2020-06-04T08:05:28Z

/merge

sre-bot · 2020-06-04T08:05:30Z

Sorry @jackysp, you don't have permission to trigger auto merge event on this branch. You are not a committer for this part

jackysp · 2020-06-04T08:06:24Z

/merge

sre-bot · 2020-06-04T08:06:46Z

/run-all-tests

sre-bot · 2020-06-04T08:07:36Z

Sorry @jackysp, you don't have permission to trigger auto merge event on this branch. You are not a committer for this part

sre-bot · 2020-06-04T08:09:17Z

@crazycs520 merge failed.

AndreMouche · 2020-06-04T08:17:22Z

/merge

sre-bot · 2020-06-04T08:17:24Z

Sorry @AndreMouche, you don't have permission to trigger auto merge event on this branch. You are not a committer for this part

bb7133 · 2020-06-04T08:20:48Z

/run-common-test

breezewish · 2020-06-04T09:15:17Z

sessionctx/variable/tidb_vars.go

@@ -399,6 +399,9 @@ const (

 	// TiDBEnableClusteredIndex indicates if clustered index feature is enabled.
 	TiDBEnableClusteredIndex = "tidb_enable_clustered_index"
+
+	// TiDBSlowLogMasking indicates that whether masking the query data when log slow query.


Does it only effective for parameterized queries?

What's your mean of parameterized queries?

Actually the masking SQL is also used to generate the SQL digest.

Oh, I mean prepared statements. For non prepared statements, will they have effect?

yes, It also works on non-prepared statement.

bb7133 · 2020-06-04T09:51:44Z

/run-all-tests

crazycs520 · 2020-06-04T10:16:11Z

/integration-ddl-test

bb7133 · 2020-06-04T10:52:57Z

/run-integration-ddl-test

breezewish · 2020-06-04T14:48:58Z

What will happen to the parameters appears in the plan? This can also be a source of leaking data.

breezewish · 2020-06-04T14:49:32Z

I also opened a related issue for the statement part, which will encounter similar problems and needs some design: #17690

Signed-off-by: sre-bot <[email protected]>

sre-bot · 2020-06-05T01:16:50Z

cherry pick to release-4.0 in PR #17694

…log query (#17637) (#17694) * cherry pick #17637 to release-4.0 Signed-off-by: sre-bot <[email protected]> * fix conflict Signed-off-by: crazycs <[email protected]> Co-authored-by: crazycs <[email protected]> Co-authored-by: Lynn <[email protected]>

*: add global variable tidb_slow_log_masking to control masking slow …

595e287

…log query Signed-off-by: crazycs <[email protected]>

crazycs520 added the type/enhancement The issue or PR belongs to an enhancement. label Jun 3, 2020

github-actions bot added the sig/execution SIG execution label Jun 3, 2020

crazycs520 added 2 commits June 3, 2020 22:11

add test

05c73e8

Signed-off-by: crazycs <[email protected]>

Merge branch 'master' into slow-log-masking

af0b987

crazycs520 added 2 commits June 3, 2020 22:58

Merge branch 'master' into slow-log-masking

aaec4d9

Merge branch 'master' into slow-log-masking

76f31ae

bb7133 reviewed Jun 4, 2020

View reviewed changes

bb7133 added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 4, 2020

wjhuang2016 approved these changes Jun 4, 2020

View reviewed changes

crazycs520 added the needs-cherry-pick-4.0 label Jun 4, 2020

bb7133 added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 4, 2020

jackysp removed the sig/execution SIG execution label Jun 4, 2020

jackysp closed this Jun 4, 2020

jackysp reopened this Jun 4, 2020

jackysp added the sig/transaction SIG:Transaction label Jun 4, 2020

sre-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 4, 2020

jackysp removed the sig/transaction SIG:Transaction label Jun 4, 2020

jackysp added status/can-merge Indicates a PR has been approved by a committer. and removed status/can-merge Indicates a PR has been approved by a committer. labels Jun 4, 2020

github-actions bot added the sig/execution SIG execution label Jun 4, 2020

breezewish reviewed Jun 4, 2020

View reviewed changes

bb7133 and others added 2 commits June 4, 2020 17:18

Merge branch 'master' into slow-log-masking

ec4dac6

Merge branch 'master' into slow-log-masking

9bdd384

AndreMouche removed the sig/execution SIG execution label Jun 4, 2020

github-actions bot added the sig/execution SIG execution label Jun 4, 2020

Merge branch 'master' into slow-log-masking

857bd48

crazycs520 merged commit 35e2d3a into pingcap:master Jun 5, 2020

sre-bot pushed a commit to sre-bot/tidb that referenced this pull request Jun 5, 2020

cherry pick pingcap#17637 to release-4.0

78ede75

Signed-off-by: sre-bot <[email protected]>

sre-bot mentioned this pull request Jun 5, 2020

*: add global variable tidb_slow_log_masking to control masking slow log query (#17637) #17694

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add global variable tidb_slow_log_masking to control masking slow log query #17637

*: add global variable tidb_slow_log_masking to control masking slow log query #17637

crazycs520 commented Jun 3, 2020 •

edited by bb7133

Loading

codecov bot commented Jun 3, 2020 •

edited

Loading

crazycs520 commented Jun 3, 2020

bb7133 left a comment

wjhuang2016 left a comment

bb7133 commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

bb7133 commented Jun 4, 2020

bb7133 commented Jun 4, 2020

AndreMouche commented Jun 4, 2020

sre-bot commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

sre-bot commented Jun 4, 2020

sre-bot commented Jun 4, 2020

AndreMouche commented Jun 4, 2020

sre-bot commented Jun 4, 2020

bb7133 commented Jun 4, 2020

breezewish Jun 4, 2020

crazycs520 Jun 4, 2020

breezewish Jun 4, 2020

crazycs520 Jun 5, 2020

bb7133 commented Jun 4, 2020

crazycs520 commented Jun 4, 2020

bb7133 commented Jun 4, 2020

breezewish commented Jun 4, 2020

breezewish commented Jun 4, 2020

sre-bot commented Jun 5, 2020

*: add global variable tidb_slow_log_masking to control masking slow log query #17637

*: add global variable tidb_slow_log_masking to control masking slow log query #17637

Conversation

crazycs520 commented Jun 3, 2020 • edited by bb7133 Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

codecov bot commented Jun 3, 2020 • edited Loading

Codecov Report

crazycs520 commented Jun 3, 2020

bb7133 left a comment

Choose a reason for hiding this comment

wjhuang2016 left a comment

Choose a reason for hiding this comment

bb7133 commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

bb7133 commented Jun 4, 2020

bb7133 commented Jun 4, 2020

AndreMouche commented Jun 4, 2020

sre-bot commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

jackysp commented Jun 4, 2020

sre-bot commented Jun 4, 2020

sre-bot commented Jun 4, 2020

sre-bot commented Jun 4, 2020

AndreMouche commented Jun 4, 2020

sre-bot commented Jun 4, 2020

bb7133 commented Jun 4, 2020

breezewish Jun 4, 2020

Choose a reason for hiding this comment

crazycs520 Jun 4, 2020

Choose a reason for hiding this comment

breezewish Jun 4, 2020

Choose a reason for hiding this comment

crazycs520 Jun 5, 2020

Choose a reason for hiding this comment

bb7133 commented Jun 4, 2020

crazycs520 commented Jun 4, 2020

bb7133 commented Jun 4, 2020

breezewish commented Jun 4, 2020

breezewish commented Jun 4, 2020

sre-bot commented Jun 5, 2020

crazycs520 commented Jun 3, 2020 •

edited by bb7133

Loading

codecov bot commented Jun 3, 2020 •

edited

Loading