security: Encrypt region boundary keys, Part 4 - KMS #3141

yiwu-arbug · 2020-11-03T00:28:57Z

What problem does this PR solve?

This is part 4 for adding TDE support to PD. pingcap/tidb#18262 This PR supports AWS KMS as master key.

What is changed and how it works?

The previous PR in the series support storing master key as a file. This is only suitable for testing. For production this is insecure. This PR supports using AWS KMS as master key. KMS is a service holds an encryption key (called a CMK), and provide API to encrypt and decrypt data using the encryption key, without exposing the key itself. We only use two of its interfaces:

  GenerateDataKey() (plaintextKey, ciphertextKey),
  Decrypt(ciphertextKey) plaintextKey,

GenerateDataKey() generates an encryption key from the CMK, returning both the plaintext key, and the key after encrypted by the CMK (the ciphertext key). Decrypt() can then used to decrypt the ciphertext key. We could use the KMS data key as our data key, but on PD restart we would need to decrypt the data keys one by one, and AWS KMS doesn't provide a batch API. Instead we use the KMS data key as our master key. So here's another layer of envelope encryption.

KMS CMK (hold by KMS, never exposed to us)
↓ encrypts
KMS data key (a.k.a PD master key)
↓ encrypts
PD data key

When PD persist data keys (after key rotation):

Call GenerateDataKey to obtain a new KMS data key (plaintext and ciphertext key pair), use it to encrypt the PD data keys
Store the encrypted data keys to etcd, along with the ciphertext key returned by KMS. discard the plaintext key.

When PD restart, or is notified of change to the keys through watcher:

load the encrypted data keys from etcd, which contains the ciphertext key.
call Decrypt to restore plaintext KMS data key from the ciphertext key.
decrypt PD data keys using the plaintext key.

This PR only support AWS KMS, but in the future we need to support other KMS services (e.g. GCP KMS). The vender field in encryptionpb.MasterKey is used for forward compatibility - we will fail if the field is not "AWS".

Check List

Tests

New unit test
Manual test with sysbench on a tidb cluster, with KMS key.

Related changes

part 1 security: Encrypt region boundary keys, Part 1 - helper functions #2931, part 2 security: Encrypt region boundary keys, Part 2 - server changes #2976, part 3 security: Encrypt region boundary keys, Part 3 - key manager #3060

Release note

No release note

Signed-off-by: Yi Wu <[email protected]>

HunDunDM · 2020-11-09T08:22:42Z

/run-all-tests

Yisaer

LGTM

Yisaer · 2020-11-12T08:04:13Z

/merge

ti-chi-bot · 2020-11-12T08:04:14Z

@Yisaer: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the tidb-community-bots/prow-config repository.

ti-chi-bot · 2020-11-12T08:04:16Z

@Yisaer: adding 'status/can-merge' to this PR must have 2 LGTMs

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the tidb-community-bots/prow-config repository.

Yisaer

LGTM

Yisaer · 2020-11-12T08:08:10Z

/merge

ti-chi-bot · 2020-11-12T08:08:10Z

@Yisaer: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the tidb-community-bots/prow-config repository.

ti-chi-bot · 2020-11-12T08:08:13Z

Can merge label has been added.

Git tree hash: 7480d5d

yiwu-arbug · 2020-11-12T20:42:22Z

/merge

ti-chi-bot · 2020-11-12T20:42:23Z

@yiwu-arbug: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the tidb-community-bots/prow-config repository.

ti-chi-bot · 2020-11-12T20:42:23Z

@yiwu-arbug: you cannot merge your own PR.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the tidb-community-bots/prow-config repository.

HunDunDM · 2020-11-13T01:56:43Z

/run-integration-ddl-test

HunDunDM · 2020-11-13T01:56:51Z

/merge

ti-chi-bot · 2020-11-13T01:56:52Z

@HunDunDM: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the tidb-community-bots/prow-config repository.

HunDunDM · 2020-11-13T02:18:19Z

/run-integration-lightning-test

Yi Wu added 30 commits September 17, 2020 07:43

Encrypt region boundary keys, Part 2 - server changes

20c682d

Signed-off-by: Yi Wu <[email protected]>

update errno and config template

b6f7ba0

Signed-off-by: Yi Wu <[email protected]>

fix typo

0bf0304

Signed-off-by: Yi Wu <[email protected]>

fix tests

23954db

Signed-off-by: Yi Wu <[email protected]>

fix tests

2bad0b9

Signed-off-by: Yi Wu <[email protected]>

fix tests

f88d26a

Signed-off-by: Yi Wu <[email protected]>

fix tests

6f6d93a

Signed-off-by: Yi Wu <[email protected]>

address comment in #2931

928de26

Signed-off-by: Yi Wu <[email protected]>

fix loadRegion

9b6bcfd

Signed-off-by: Yi Wu <[email protected]>

rename encryption_key_manager package

bdd5de4

Signed-off-by: Yi Wu <[email protected]>

Merge branch 'master' into enc_server

bbdb8ef

fix lint

90de5ef

Signed-off-by: Yi Wu <[email protected]>

fix lint

c16ecd8

Signed-off-by: Yi Wu <[email protected]>

fix comments

2428248

Signed-off-by: Yi Wu <[email protected]>

fix comment

ce83a42

Signed-off-by: Yi Wu <[email protected]>

use option pattern

073f13e

Signed-off-by: Yi Wu <[email protected]>

move loadRegion and saveRegion

c35f948

Signed-off-by: Yi Wu <[email protected]>

Merge branch 'master' into enc_server

06d8963

revert changes

551b3f9

Signed-off-by: Yi Wu <[email protected]>

fix doc

7051407

Signed-off-by: Yi Wu <[email protected]>

address comment

7201c37

Signed-off-by: Yi Wu <[email protected]>

Merge branch 'master' into enc_server

c9eb341

Merge remote-tracking branch 'origin/master' into enc_server

39f147e

Merge branch 'master' into enc_server

12f03c4

Merge branch 'master' into enc_server

f951278

key manager

ee2e0cd

Signed-off-by: Yi Wu <[email protected]>

add test and refactor

9e753b4

Signed-off-by: Yi Wu <[email protected]>

make EncryptionKeysPath a const

a6d1359

Signed-off-by: Yi Wu <[email protected]>

fix region_crypter key manager nil check

103a612

Signed-off-by: Yi Wu <[email protected]>

save conflict test

e9d810d

Signed-off-by: Yi Wu <[email protected]>

Yi Wu added 5 commits November 9, 2020 13:34

address comment

9a6020f

Signed-off-by: Yi Wu <[email protected]>

Merge branch 'master' into enc_kms

5a69760

fix error type

0321a80

Signed-off-by: Yi Wu <[email protected]>

fix test

312bb48

Signed-off-by: Yi Wu <[email protected]>

Merge branch 'master' into enc_kms

e90b370

HunDunDM approved these changes Nov 9, 2020

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 9, 2020

Merge branch 'master' into enc_kms

7480d5d

Yisaer reviewed Nov 12, 2020

View reviewed changes

Yisaer approved these changes Nov 12, 2020

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 12, 2020

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 12, 2020

Merge branch 'master' into enc_kms

f1d3380

ti-chi-bot merged commit 3c62563 into tikv:master Nov 13, 2020

yiwu-arbug deleted the enc_kms branch November 13, 2020 07:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

security: Encrypt region boundary keys, Part 4 - KMS #3141

security: Encrypt region boundary keys, Part 4 - KMS #3141

yiwu-arbug commented Nov 3, 2020 •

edited

Loading

HunDunDM commented Nov 9, 2020

Yisaer left a comment

Yisaer commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

Yisaer left a comment

Yisaer commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

yiwu-arbug commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

HunDunDM commented Nov 13, 2020

HunDunDM commented Nov 13, 2020

ti-chi-bot commented Nov 13, 2020

HunDunDM commented Nov 13, 2020

security: Encrypt region boundary keys, Part 4 - KMS #3141

security: Encrypt region boundary keys, Part 4 - KMS #3141

Conversation

yiwu-arbug commented Nov 3, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

HunDunDM commented Nov 9, 2020

Yisaer left a comment

Choose a reason for hiding this comment

Yisaer commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

Yisaer left a comment

Choose a reason for hiding this comment

Yisaer commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

yiwu-arbug commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

ti-chi-bot commented Nov 12, 2020

HunDunDM commented Nov 13, 2020

HunDunDM commented Nov 13, 2020

ti-chi-bot commented Nov 13, 2020

HunDunDM commented Nov 13, 2020

yiwu-arbug commented Nov 3, 2020 •

edited

Loading