sink(ticdc): split RowChangeEvent if unique key is updated #9437
Conversation
ti-chi-bot
bot
added
release-note-none
sdojjy
changed the title
ti-chi-bot
bot
added
the
do-not-merge/work-in-progress
cdc/model/sink.go
Outdated
| } | ||
|
|
||
| func (e RowChangedEvents) Less(i, j int) bool { | ||
| return len(e[i].Columns)-len(e[i].PreColumns) < |
There was a problem hiding this comment.
This is not easy to understand directly, please add some comments about how the condition comes up.
There was a problem hiding this comment.
changed to another implementation
| @@ -719,6 +748,115 @@ func (t *SingleTableTxn) GetCommitTs() uint64 { | |||
| return t.CommitTs | |||
| } | |||
|
|
|||
| // TrySplitAndSortUpdateEvent split update events if unique key is updated | |||
| func (t *SingleTableTxn) TrySplitAndSortUpdateEvent() error { | |||
| if len(t.Rows) < 2 { | |||
There was a problem hiding this comment.
is it possible there is only an update event, shall we split it here ?
There was a problem hiding this comment.
no, it's not needed to split a single update event.
There was a problem hiding this comment.
What happens if this update event has unique key columns changed? I think this update event also should be splitted
There was a problem hiding this comment.
This method is almost identical to the convertRowChangedEvents method.
After the enable-old-value is removed, this 2 method can be merged into one.
There was a problem hiding this comment.
this pr focuses on the duplicated key case, the pre-condition is more than two update events emitted.
There was a problem hiding this comment.
convertRowChangedEvents checks "enable-old-value" and handle key.
|
/retest-required |
1 similar comment
|
/retest-required |
…f-handle-key-is-changefeed # Conflicts: # cdc/sink/dmlsink/mq/mq_dml_sink.go
…ed-if-handle-key-is-changefeed
|
/test all |
ti-chi-bot
bot
removed
the
size/XL
ti-chi-bot
bot
added
the
size/XL
ti-chi-bot
bot
added
needs-1-more-lgtm
| for i := range updateEvent.Columns { | ||
| col := updateEvent.Columns[i] | ||
| preCol := updateEvent.PreColumns[i] | ||
| if col != nil && (col.Flag.IsUniqueKey() || col.Flag.IsHandleKey()) && |
There was a problem hiding this comment.
what does this condition " col.Flag.IsHandleKey()) && preCol != nil && (preCol.Flag.IsUniqueKey()" mean?
There was a problem hiding this comment.
here we check if the column is a part of the handle key or a unique key, if so we should check if the value has been changed,
for the condition " col.Flag.IsHandleKey()) && preCol != nil && (preCol.Flag.IsUniqueKey()" I think it's safe to split the row because we can always split an update to delete + insert.
…ed-if-handle-key-is-changefeed
|
/retest-required |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: asddongmen, nongfushanquan The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
|
In response to a cherrypick label: new pull request created to branch |
Signed-off-by: ti-chi-bot <[email protected]>
|
In response to a cherrypick label: new pull request created to branch |
Signed-off-by: ti-chi-bot <[email protected]>
What problem does this PR solve?
Issue Number: close #9430
What is changed and how it works?
split and sort RowChangeEvent if unique key is updated
split the update to delete and insert, then sort the whole txn with the order delete>update>insert
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note