cdc: Data may be lost during leader transfer

[zeminzhou]

Bug Report

1. Describe the bug

In TiKV, when a leader transfer occurs, the old leader may send the last resolved ts, which may be larger than the new leader's first causal timestamp after transfer.

Details as follow(from log):

1. At 20:04:06.105, leader transfer from A to B, now B's causal ts is 435080671018356250.
2. At 20:04:06.106, A as a leader broadcasts resolved ts(435080671018356650).
3. At 20:04:06.116, B becomes leader, appends ts(435080671018356350) for new key.
4. At 20:04:06.132, cdc receives resolved ts(435080671018356650) from A.
5. At 20:04:06.225, cdc start new request to B with start ts(435080671018356650).

So B don't send the key that is with ts(435080671018356350) to cdc.

2. Minimal reproduce step (Required)

1. start cdc
2. put 5000000 key/value pairs into src tikv
3. get checksum from src tikv and dst tikv
4. if checksums are same, repeat step2-3.

3. What did you see instead (Required)

1. checksums are not same

4. What did you expect to see? (Required)

1. checksums are not always same

[zeminzhou]

cc@pingyu @haojinming

[pingyu]

Some additional information:

1. A causal timestamp flush in node B is invoked after B became the "Candidate" (see here, and here), before step 3.

2. Node A will check leadership by check_leader RPC request before step 2. See here.

We may conclude that the node A & B do not get an agreement on who is the leader. So maybe we should flush causal timestamp after the moment node B "really" become the Leader (other than Candidate). Or use another method to check leader in node A.