Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

br: support checkSum during stream restore dml kv-events #35

Merged
merged 2 commits into from
Mar 29, 2022

Conversation

joccau
Copy link
Collaborator

@joccau joccau commented Mar 21, 2022

What is changed and how it works?

Issue Number: Close pingcap/tidb#33400

What's Changed:

  • Support checkSum in file level when stream restore DML kv-events.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • PR to update pingcap/tidb-ansible:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM
  • Breaking backward compatibility

Release note

None

@joccau joccau force-pushed the br-stream-develop branch from 4812885 to 55c743a Compare March 23, 2022 09:39
Copy link
Owner

@3pointer 3pointer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@YuJuncen YuJuncen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

},
|bytes| Ok(bytes.to_vec()),
)?;
if !expected_s.eq(&cal_sha256) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if !expected_s.eq(&cal_sha256) {
if expected_s != cal_sha256 {

Copy link
Collaborator Author

@joccau joccau Mar 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!x.eq(&y) equals x != y

@joccau joccau merged commit 0f084d3 into 3pointer:br-stream Mar 29, 2022
@joccau joccau deleted the br-stream-develop branch April 18, 2022 09:10
YuJuncen added a commit that referenced this pull request Apr 26, 2022
…h `br-stream`:

- backup-stream: update kvproto to master [by @3pointer] (#48)

- Added retry for initial scanning and some metrics [by @YuJuncen] (#47)

- Added a new observer type 'pitr' [by @YuJuncen] (#46)

- backup-stream: fix some bugs in log backup [by @3pointer] (#45)

- Refactor resolver [by @YuJuncen] (#44)

- report error to PD server [by @YuJuncen] (#43)

- br: support pause/resume stream task [by @joccau] (#42)

- Adapt error code for `endpoint::Error` and implement the contextual error [by @YuJuncen] (#41)

- set uuid for header to prevent raftstore merging [by @YuJuncen] (#40)

- fix the upload part of S3 storage [by @YuJuncen] (#39)

- Use min ts of mem lock [by @YuJuncen] (#38)

- Fix size leakage and build [by @YuJuncen] (#37)

- eliminate the block call in ticker [by @YuJuncen] (#36)

- br: support checkSum during stream restore dml kv-events [by @joccau] (#35)

- refine br-stream to backup-stream [by @3pointer] (#34)

- Allow local storage support directory and partition the log files by table [by @YuJuncen] (#33)

- br-stream: added store error to the store [by @YuJuncen] (#32)

- Scan on Leader Change [by @YuJuncen] (#31)

- use local thread pool for downloading [by @YuJuncen] (#30)

- display the error when failed to get snapshot [by @YuJuncen] (#29)

- br-stream: remove duplicate entry in apply kv file [by @3pointer] (#28)

- don't hold lock during flushing [by @YuJuncen] (#27)

- fix listener on follower if region changed [by @YuJuncen] (#26)

- Update Service GC Safe Point after Flushing [by @YuJuncen] (#25)

- br-stream: use raft router to apply kv files for sst_importer [by @3pointer] (#24)

- added integration test framework [by @YuJuncen] (#22)

- br-stream: add restore ts to filter data out of range. [by @3pointer] (#20)

- *: batch write to temp file [by @YuJuncen] (#19)

- omit coping in EventIterator, use write batch for apply [by @YuJuncen] (#18)

- br: don't flush to externStorage periodically when have empty kv-record [by @joccau] (#17)

- added SegmentMap to replace the BTreeMap [by @YuJuncen] (#16)

- added resolved ts related metrics [by @YuJuncen] (#15)

- Added resolved timestamp uploading [by @YuJuncen] (#14)

- add flush tick [by @3pointer] (#13)

- implement stream restore for tikv side. [by @3pointer] (#12)

- added metrics [by @YuJuncen] (#11)

- update kvproto [by @YuJuncen] (#9)

- Initial scanning && Error reporting [by @YuJuncen] (#8)

- resolve conflict [by @3pointer] (#7)

- br: backup stream: support flushing temp files to ExternalStorage [by @joccau] (#6)

- encoder: move encoder to a independent mod [by @3pointer] (#5)

- br-stream: don't clone the key & value in encode_event() [by @kennytm] (#4)

- br: backup stream: Modify log print format [by @joccau] (#3)

- br-stream: reduce the lock of tables; added some metrics [by @YuJuncen] (#2)

For more details of these commits, please check the origin feature branch at https://github.com/3pointer/tikv/tree/br-stream.

Signed-off-by: Yu Juncen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

br: support checksum in file level during stream restore
3 participants