br/restore: use a more relax check for changefeeds when restoring #53121

YuJuncen · 2024-05-09T04:07:43Z

What problem does this PR solve?

Issue Number: close #53131

Problem Summary:
In the current version, we will reject to restore once there is any changefeed.
This is too strict and not pretty friendly to the end user.

What changed and how does it work?

For now, we will only reject to restore when the changefeed:

Have a checkpoint earlier than the backup time for this restore.
Is not failed.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Create such a changefeed:

function run
  changefeedctl create --sink-uri=blackhole:// --no-confirm -c $argv[1] --start-ts (pdctl service-gc-safepoint | jq .gc_safe_point)
  changefeedctl pause -c $argv[1]
end
run test

Backup the cluster with:

./br backup full -s (s3path tpcc-10)

Then, restore the backup, should fail:

root@xanadu ~/op# ./br restore full -s (s3path tpcc-10)
Detail BR log in /tmp/br.log.2024-05-09T14.17.38+0800
[2024/05/09 14:17:42.087 +08:00] [INFO] [collector.go:77] ["Full Restore failed summary"] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0]
Error: found CDC changefeed(s): cluster/namespace: default/default changefeed(s): [test], please remove changefeed(s) before restore

Resume the changefeed and wait it proceed.

changefeedctl resume -c test

5.Now the BR passed the CDC check (BR still fail due to we are trying to restore to the origin cluster, that is expected.)

root@xanadu ~/op# ./br restore full -s (s3path tpcc-10)
Detail BR log in /tmp/br.log.2024-05-09T14.23.18+0800
[2024/05/09 14:23:22.117 +08:00] [INFO] [collector.go:77] ["Full Restore failed summary"] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0]
#######################################################################
# the target cluster is not fresh, cannot restore.
# you can drop existing databases and tables and start restore again
#######################################################################
Error: user db/tables: test, warehouse, district, customer, history, new_order, orders, order_line, stock, item: [BR:Restore:ErrRestoreNotFreshCluster]cluster is not fresh

No need to test
- I checked and no code files have been changed.

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Now, restore from an archive old enough won't be conflict with TiCDC changefeeds.

Signed-off-by: Yu Juncen <[email protected]>

…cdc-br

Signed-off-by: Yu Juncen <[email protected]>

tiprow · 2024-05-09T04:07:58Z

Hi @YuJuncen. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-05-09T04:27:40Z

Codecov Report

Attention: Patch coverage is 69.34307% with 42 lines in your changes are missing coverage. Please review.

Project coverage is 75.5631%. Comparing base (f304f04) to head (cbbd432).
Report is 56 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #53121        +/-   ##
================================================
+ Coverage   72.4716%   75.5631%   +3.0915%     
================================================
  Files          1491       1519        +28     
  Lines        429012     449567     +20555     
================================================
+ Hits         310912     339707     +28795     
+ Misses        98878      88682     -10196     
- Partials      19222      21178      +1956

Flag	Coverage Δ
integration	`53.5457% <56.9343%> (?)`
unit	`71.2557% <63.9705%> (-0.1084%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9957% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.1969% <46.6666%> (+19.7534%)`	⬆️

Signed-off-by: Yu Juncen <[email protected]>

… functions Signed-off-by: Yu Juncen <[email protected]>

sdojjy · 2024-05-09T06:26:34Z

pkg/util/cdcutil/cdc.go

+	var changefeedMsgBuf strings.Builder
+	changefeedMsgBuf.WriteString("found CDC changefeed(s): ")
+	for clusterID, changefeedID := range s.changefeeds {
+		changefeedMsgBuf.WriteString("cluster/namespace: ")


namespace is a part of changefeed ID

Here I'm going to try keep the old output format. Some of test cases rely on it.

sdojjy · 2024-05-09T06:38:02Z

pkg/util/cdcutil/cdc.go

+	}
+	switch info.State {
+	// https://docs.pingcap.com/zh/tidb/stable/ticdc-changefeed-overview
+	case "error", "failed":


error state still block gc, it's the same with warning

sdojjy · 2024-05-09T06:40:51Z

pkg/util/cdcutil/cdc.go

+	if err := json.Unmarshal(infoResp.Kvs[0].Value, &info); err != nil {
+		return 0, errors.Trace(err)
+	}
+	switch info.State {


ignore changefeeds in finished state

Signed-off-by: Yu Juncen <[email protected]>

ti-chi-bot · 2024-05-13T03:19:09Z

[LGTM Timeline notifier]

Timeline:

2024-05-09 06:46:27.023852027 +0000 UTC m=+1117340.780987604: ☑️ agreed by sdojjy.
2024-05-13 03:19:08.814865858 +0000 UTC m=+1450502.572001428: ☑️ agreed by Leavrth.

ti-chi-bot · 2024-05-15T12:28:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, Leavrth, sdojjy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [D3Hunter,Leavrth,sdojjy]
~~br/OWNERS~~ [Leavrth]
~~lightning/OWNERS~~ [D3Hunter]
~~pkg/executor/importer/OWNERS~~ [D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ngcap#53121) close pingcap#53131

YuJuncen added 6 commits May 8, 2024 10:34

first version

911ca60

Signed-off-by: Yu Juncen <[email protected]>

added test cases

c0b1585

Signed-off-by: Yu Juncen <[email protected]>

make clippy happy

be727da

Signed-off-by: Yu Juncen <[email protected]>

Merge branch 'master' of https://github.com/pingcap/tidb into compat-…

35e3a29

…cdc-br

resolve conflicts

5d85495

Signed-off-by: Yu Juncen <[email protected]>

Add check for incompatible TiCDC changefeeds before restoring

2eb39cb

Signed-off-by: Yu Juncen <[email protected]>

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 9, 2024

YuJuncen added 2 commits May 9, 2024 13:20

make clippy happy

f3167fe

Signed-off-by: Yu Juncen <[email protected]>

Fix error handling in fetchCheckpointTSFromStatus and checkpointTSFor…

0ad5998

… functions Signed-off-by: Yu Juncen <[email protected]>

sdojjy reviewed May 9, 2024

View reviewed changes

handing state by comment

cbbd432

Signed-off-by: Yu Juncen <[email protected]>

sdojjy approved these changes May 9, 2024

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. and removed do-not-merge/needs-linked-issue labels May 9, 2024

Leavrth approved these changes May 13, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels May 13, 2024

D3Hunter approved these changes May 15, 2024

View reviewed changes

ti-chi-bot bot added the approved label May 15, 2024

ti-chi-bot bot merged commit b9b330a into pingcap:master May 15, 2024
37 checks passed

terry1purcell pushed a commit to terry1purcell/tidb that referenced this pull request May 17, 2024

br/restore: use a more relax check for changefeeds when restoring (pi…

a60de17

…ngcap#53121) close pingcap#53131

RidRisR pushed a commit to RidRisR/tidb that referenced this pull request May 23, 2024

br/restore: use a more relax check for changefeeds when restoring (pi…

3adf8e9

…ngcap#53121) close pingcap#53131

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

br/restore: use a more relax check for changefeeds when restoring #53121

br/restore: use a more relax check for changefeeds when restoring #53121

YuJuncen commented May 9, 2024 •

edited

Loading

tiprow bot commented May 9, 2024

codecov bot commented May 9, 2024 •

edited

Loading

sdojjy May 9, 2024

YuJuncen May 9, 2024

sdojjy May 9, 2024

sdojjy May 9, 2024

ti-chi-bot bot commented May 13, 2024

ti-chi-bot bot commented May 15, 2024

br/restore: use a more relax check for changefeeds when restoring #53121

br/restore: use a more relax check for changefeeds when restoring #53121

Conversation

YuJuncen commented May 9, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented May 9, 2024

codecov bot commented May 9, 2024 • edited Loading

Codecov Report

sdojjy May 9, 2024

Choose a reason for hiding this comment

YuJuncen May 9, 2024

Choose a reason for hiding this comment

sdojjy May 9, 2024

Choose a reason for hiding this comment

sdojjy May 9, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented May 13, 2024

[LGTM Timeline notifier]

ti-chi-bot bot commented May 15, 2024

YuJuncen commented May 9, 2024 •

edited

Loading

codecov bot commented May 9, 2024 •

edited

Loading