VReplication Copy Phase: Parallelize Bulk Inserts #10828

rohit-nayak-ps · 2022-07-25T20:06:09Z

Description

This PR adds a new experimental vttablet flag --vreplication-parallel-insert-workers=[integer] that enables parallel bulk inserts during the copy phase of VReplication. It is disabled (1) by default.

This PR also adds new VReplication stats:

VReplicationTableCopyTimings: similar to PhaseTimings{Copy}, but provides per-table-per-stream observability and updates throughout the copy phase, instead of at the end of the copy phase.
VReplicationTableCopyRowCounts: counts rows copied per-table-per-stream throughout the copy phase.

Approach

When parallel inserts are not enabled, VReplication behaves the same way as before this PR, with the exception that the new stats are used.

When setting --vreplication-parallel-insert-workers > 1 the bulk inserts — each
executed as a single transaction from the vstream packet contents — may happen in parallel and
out-of-order, but the commits of those transactions are still serialized in order.

Other aspects of the VReplication copy-phase logic are preserved:

All statements executed when processing a vstream packet occur within a single MySQL transaction.
Writes to _vt.copy_state always follow their corresponding inserts from within the vstream packet.
The final commit for the vstream packet always follows the corresponding write to _vt.copy_state.
The vstream packets are committed in the order seen in the stream. So for any PK1 and PK2, the write to _vt.copy_state and commit steps (steps 2 and 3 above) for PK1 will both precede the _vt.copy_state write and commit steps of PK2.

Other phases (catchup, fast-forward, "running") are not changed by this PR.

Checklist

Should this PR be backported? (no)
Tests were added
Documentation: feat: document vttablet --vreplication-parallel-insert-workers website#1196

Impacted Areas in Vitess

Components that this PR will affect:

vitess-bot · 2022-07-25T20:06:11Z

maxenglander · 2022-08-02T17:15:49Z

With @mattlord help I have been benchmarking the performance of this PR alongside #10788. Both PRs improve performance between 10-200%, depending on factors like CPU load on the data source.

Because this PR came first, already has undergone some code review, and is simpler than #10788, I'm going to add more work to this PR.

Completely avoid async path when experimental flag not present
Get E2E tests to pass without experimental flag
Get unit tests to pass without experimental flag
Get E2E tests to pass with experimental flag
Get unit tests to pass with experimental flag
Add more unit tests with experimental flag
Add more E2E tests with experimental flag

go/vt/vttablet/tabletmanager/vreplication/engine.go

go/vt/vttablet/tabletmanager/vreplication/shard_sorter.go

go/vt/vttablet/tabletmanager/vreplication/vplayer.go

go/vt/vttablet/tabletmanager/vreplication/vreplicator.go

go/vt/vttablet/tabletmanager/vreplication/engine.go

go/vt/vttablet/tabletmanager/vreplication/vreplicator.go

go/vt/vttablet/tabletmanager/vreplication/vcopier.go

go/flags/endtoend/vttablet.txt

go/vt/vttablet/tabletmanager/vreplication/vcopier.go

maxenglander · 2022-10-20T20:30:37Z

Have squashed all the commits. Was too annoying to try to fix the commit that was missing DCO with rebase. Previous commit history preserved over here just in case.

…ovements This PR adds a new experimental flag `--vreplication-parallel-insert-workers=[integer]` that enables parallel bulk inserts during the copy phase of VReplication. It is not set by default. This PR also adds three new VReplication stats: - `VReplicationTableCopyTimings` is similar to `PhaseTimings{Copy}`, but provides per-table-per-stream observability and updates throughout the copy phase, instead of at the end of the copy phase. - `VReplicationTableCopyRowCounts` counts rows copied per-table-per-stream throughout the copy phase. - `VReplicationBulkInsertConcurrency` gauges the configured number of concurrent insertion workers. Co-authored-by: Rohit Nayak <[email protected]> Signed-off-by: Max Englander <[email protected]>

mattlord

Once again, great work on this! I only had some minor nits/questions/suggestions. We can huddle on those and get this merged. ❤️

doc/releasenotes/16_0_0_summary.md

go/test/endtoend/vreplication/resharding_workflows_v2_test.go

go/test/endtoend/vreplication/vreplication_test.go

mattlord · 2022-10-21T22:11:00Z

go/vt/binlog/binlogplayer/binlog_player.go

+	bps.TableCopyRowCounts = stats.NewCountersWithSingleLabel("", "", "Table", "")
+	bps.TableCopyTimings = stats.NewTimings("", "", "Table")


Do you have a sample of what this looks like in the /debug/vars output?

Here's a sample:

"VReplicationTableCopyRowCounts": {"src.-.copy.1.data": 8388608},

go/vt/vttablet/tabletmanager/vreplication/vcopier.go

mattlord · 2022-10-25T00:24:04Z

go/vt/vttablet/tabletmanager/vreplication/vreplicator.go

-func (vr *vreplicator) readSettings(ctx context.Context) (settings binlogplayer.VRSettings, numTablesToCopy int64, err error) {
-	settings, err = binlogplayer.ReadVRSettings(vr.dbClient, vr.id)
+// Same as readSettings, but stores some of the results on this vr.
+func (vr *vreplicator) loadSettings(ctx context.Context, dbClient *vdbClient) (settings binlogplayer.VRSettings, numTablesToCopy int64, err error) {


I don't see the need for these two function changes, what am I missing?

I want to be able to re-use logic in vreplicator.go for setting things like setting and unsettingsql_mode and disabling FK checks. I want to use that logic in each new parallel MySQL conn that's created in vcopier.go. The current vreplicator.go code makes that a little difficult because it changes state in vreplicator.

To address that I move the logic I wanted to re-use into a stateless fn readSettings, and move the stateful stuff into loadSettings.

Signed-off-by: Max Englander <[email protected]>

rohit-nayak-ps mentioned this pull request Jul 25, 2022

Do Not Merge: VReplication Copy Phase: Parallelize bulk insert to check for performance improvements #7728

Closed

8 tasks

maxenglander force-pushed the rn-vrep-concurrent-copy branch 5 times, most recently from 8d13a17 to b6d6298 Compare August 8, 2022 20:19

maxenglander force-pushed the rn-vrep-concurrent-copy branch 4 times, most recently from a877a99 to a7da4da Compare August 15, 2022 23:08

maxenglander reviewed Aug 15, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/engine.go Outdated Show resolved Hide resolved

maxenglander reviewed Aug 15, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/shard_sorter.go Outdated Show resolved Hide resolved

maxenglander reviewed Aug 15, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vplayer.go Outdated Show resolved Hide resolved

maxenglander reviewed Aug 15, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vreplicator.go Outdated Show resolved Hide resolved

maxenglander force-pushed the rn-vrep-concurrent-copy branch 3 times, most recently from 2a41fd9 to 9d90fe6 Compare August 16, 2022 05:23

maxenglander reviewed Aug 16, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/engine.go Outdated Show resolved Hide resolved

maxenglander reviewed Aug 16, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vreplicator.go Outdated Show resolved Hide resolved

mattlord added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication Benchmark me Add label to PR to run benchmarks labels Aug 16, 2022

maxenglander force-pushed the rn-vrep-concurrent-copy branch 3 times, most recently from ce597c0 to 1a7bbb5 Compare August 18, 2022 05:33

mattlord changed the title ~~Do Not Merge: VReplication Copy Phase: Parallelize bulk insert to check for performance improvements~~ VReplication Copy Phase: Parallelize bulk insert to check for performance improvements Aug 18, 2022

mattlord marked this pull request as ready for review August 18, 2022 15:01

mattlord self-requested a review as a code owner August 18, 2022 15:01

maxenglander force-pushed the rn-vrep-concurrent-copy branch from 4facf7b to fb29459 Compare September 8, 2022 16:33

maxenglander reviewed Sep 8, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vcopier.go Outdated Show resolved Hide resolved

maxenglander reviewed Sep 8, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vcopier.go Outdated Show resolved Hide resolved

maxenglander reviewed Sep 8, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vcopier.go Outdated Show resolved Hide resolved

maxenglander reviewed Sep 8, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vcopier.go Show resolved Hide resolved

mattlord reviewed Sep 9, 2022

View reviewed changes

go/flags/endtoend/vttablet.txt Outdated Show resolved Hide resolved

mattlord reviewed Sep 11, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vcopier.go Outdated Show resolved Hide resolved

maxenglander requested a review from mattlord September 12, 2022 21:03

mattlord added the release notes (needs details) This PR needs to be listed in the release notes in a dedicated section (deprecation notice, etc...) label Sep 19, 2022

maxenglander mentioned this pull request Sep 28, 2022

updates to _vt.copy_state contribute to replica lag #11385

Closed

rohit-nayak-ps commented Oct 9, 2022

View reviewed changes

go/vt/vttablet/tabletmanager/vreplication/vcopier.go Show resolved Hide resolved

rohit-nayak-ps removed the release notes (needs details) This PR needs to be listed in the release notes in a dedicated section (deprecation notice, etc...) label Oct 9, 2022

maxenglander requested review from GuptaManan100, frouioui and deepthi as code owners October 20, 2022 18:49

maxenglander force-pushed the rn-vrep-concurrent-copy branch from 3aea5b5 to 5d8bde8 Compare October 20, 2022 20:26

maxenglander force-pushed the rn-vrep-concurrent-copy branch from 5d8bde8 to 41a4233 Compare October 20, 2022 21:24

mattlord reviewed Oct 25, 2022

View reviewed changes

address pr comments

135e79e

Signed-off-by: Max Englander <[email protected]>

maxenglander requested a review from mattlord October 25, 2022 19:10

mattlord approved these changes Oct 26, 2022

View reviewed changes

maxenglander mentioned this pull request Oct 26, 2022

feat: document vttablet --vreplication-parallel-insert-workers vitessio/website#1196

Merged

mattlord changed the title ~~VReplication Copy Phase: Parallelize bulk insert to check for performance improvements~~ VReplication Copy Phase: Parallelize Bulk Inserts Oct 26, 2022

mattlord merged commit c5eb7a6 into vitessio:main Oct 26, 2022

mattlord deleted the rn-vrep-concurrent-copy branch October 26, 2022 14:37

This was referenced Feb 13, 2023

Bug Report: regression, VReplication errors lost their type, converted to errorString #12326

Closed

(re)Formalize SQLError in VReplication, add underlying wrap/unwrap functionality #12327

Merged

rohit-nayak-ps mentioned this pull request Sep 5, 2023

MoveTables: allow copying all tables in a single atomic copy phase cycle #13137

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VReplication Copy Phase: Parallelize Bulk Inserts #10828

VReplication Copy Phase: Parallelize Bulk Inserts #10828

rohit-nayak-ps commented Jul 25, 2022 •

edited by mattlord

Loading

vitess-bot bot commented Jul 25, 2022 •

edited by mattlord

Loading

maxenglander commented Aug 2, 2022 •

edited

Loading

maxenglander commented Oct 20, 2022 •

edited

Loading

mattlord left a comment

mattlord Oct 21, 2022

maxenglander Oct 25, 2022

mattlord Oct 25, 2022

maxenglander Oct 25, 2022 •

edited

Loading

		bps.TableCopyRowCounts = stats.NewCountersWithSingleLabel("", "", "Table", "")
		bps.TableCopyTimings = stats.NewTimings("", "", "Table")

VReplication Copy Phase: Parallelize Bulk Inserts #10828

VReplication Copy Phase: Parallelize Bulk Inserts #10828

Conversation

rohit-nayak-ps commented Jul 25, 2022 • edited by mattlord Loading

Description

Approach

Checklist

Impacted Areas in Vitess

vitess-bot bot commented Jul 25, 2022 • edited by mattlord Loading

Review Checklist

General

Bug fixes

Non-trivial changes

New/Existing features

Backward compatibility

maxenglander commented Aug 2, 2022 • edited Loading

maxenglander commented Oct 20, 2022 • edited Loading

mattlord left a comment

Choose a reason for hiding this comment

mattlord Oct 21, 2022

Choose a reason for hiding this comment

maxenglander Oct 25, 2022

Choose a reason for hiding this comment

mattlord Oct 25, 2022

Choose a reason for hiding this comment

maxenglander Oct 25, 2022 • edited Loading

Choose a reason for hiding this comment

rohit-nayak-ps commented Jul 25, 2022 •

edited by mattlord

Loading

vitess-bot bot commented Jul 25, 2022 •

edited by mattlord

Loading

maxenglander commented Aug 2, 2022 •

edited

Loading

maxenglander commented Oct 20, 2022 •

edited

Loading

maxenglander Oct 25, 2022 •

edited

Loading