Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

domain: stop schema validator to prohibit DML executing when tidb lost session of etcd #8441

Merged
merged 14 commits into from
Nov 28, 2018
Merged

domain: stop schema validator to prohibit DML executing when tidb lost session of etcd #8441

merged 14 commits into from
Nov 28, 2018

Conversation

winkyao
Copy link
Contributor

@winkyao winkyao commented Nov 25, 2018

What problem does this PR solve?

The etcd is responsible for schema synchronization, we should ensure there is at most two diffrent schema version in the TiDB cluster, to make the data/schema be consistent.

If we lost connection/session to etcd, the cluster will treats this TiDB as a down instance, and etcd will remove the key of /tidb/ddl/all_schema_versions/tidb-id.

Say the schema version now is 1, the owner is changing the schema version to 2, it will not wait for this down TiDB syncing the schema, then continue to change the TiDB schema to version 3. Unfortunately, this down TiDB schema version will still be version 1. And version 1 is not consistent to version 3.

So we need to stop the schema validator to prohibit the DML executing when we lost session of etcd.

This PR revert partial changes of #7810


This change is Reviewable

@winkyao winkyao added type/bugfix This PR fixes a bug. priority/release-blocker This issue blocks a release. Please solve it ASAP. component/DDL-need-LGT3 labels Nov 25, 2018
@winkyao
Copy link
Contributor Author

winkyao commented Nov 25, 2018

@tiancaiamao
Copy link
Contributor

tiancaiamao commented Nov 25, 2018

it will not wait for this down TiDB syncing the schema, then continue to change the TiDB schema to version 3

A schema change should meet either of the two conditions:

  1. There is no network partition, all peers agree with a consistent cluster state
  2. Wait a lease since last change, to guarantee that a schema won't change twice within a lease

If none of the condition is satisfied, how could it change to a newer version 3 ?

So we need to stop the schema validator to prohibit the DML executing when we lost session of etcd.

We do not need to stop the schema validator, instead, we should stop the version change in the owner. @winkyao

@shenli
Copy link
Member

shenli commented Nov 25, 2018

@winkyao Please add a test case.

@winkyao
Copy link
Contributor Author

winkyao commented Nov 26, 2018

I'll add test cases.

@winkyao
Copy link
Contributor Author

winkyao commented Nov 26, 2018

@tiancaiamao It is another issue, we can fix the issue you mentioned later, but not his PR.

ddl/fail_db_test.go Outdated Show resolved Hide resolved
domain/domain.go Outdated Show resolved Hide resolved
domain/domain.go Outdated Show resolved Hide resolved
domain/schema_validator.go Show resolved Hide resolved
domain/schema_validator.go Show resolved Hide resolved
session/session.go Outdated Show resolved Hide resolved
@tiancaiamao
Copy link
Contributor

LGTM

@tiancaiamao tiancaiamao added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 26, 2018
@winkyao
Copy link
Contributor Author

winkyao commented Nov 27, 2018

@crazycs520 @zimulala PTAL

@winkyao
Copy link
Contributor Author

winkyao commented Nov 27, 2018

/run-all-tests

@winkyao
Copy link
Contributor Author

winkyao commented Nov 27, 2018

/run-common-test tidb-test=pr/668

@winkyao
Copy link
Contributor Author

winkyao commented Nov 27, 2018

/run-integration-common-test tidb-test=pr/668

Copy link
Contributor

@crazycs520 crazycs520 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@crazycs520 crazycs520 added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 27, 2018
Copy link
Contributor

@ciscoxll ciscoxll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@winkyao winkyao added require-LGT3 Indicates that the PR requires three LGTM. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Nov 28, 2018
@winkyao winkyao merged commit 419c881 into pingcap:master Nov 28, 2018
@winkyao winkyao deleted the fix_schema_validator branch November 28, 2018 06:05
zz-jason pushed a commit that referenced this pull request Nov 28, 2018
zz-jason pushed a commit that referenced this pull request Nov 28, 2018
@you06 you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority/release-blocker This issue blocks a release. Please solve it ASAP. require-LGT3 Indicates that the PR requires three LGTM. sig/sql-infra SIG: SQL Infra type/bugfix This PR fixes a bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants