Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

analyzer/fast-sync: Reduce DB contention by writing updates into a writeahead log #567

Merged
merged 5 commits into from
Nov 21, 2023

Conversation

mitjat
Copy link
Contributor

@mitjat mitjat commented Nov 16, 2023

The task

There is a handful of UPDATE queries that are limiting the speed of fast-sync indexing:

  • Queries that update last_mutate_round in evm_tokens, evm_token_balances
  • The query that updates the epoch range (`UPDATE chain.epochs ...`)

For the first, we see many reports of deadlocks. For the latter, GCP's query insights lists it as a query with outsized "lock wait time", meaning it doesn't quite produce a deadlock, but it's waiting for the row to become available for writing, which slows the whole tx down (and likely increases deadlocks elsewhere).

For all of these, the approach is the same: Create a dedicated fast-sync temp table; then replace operations that are currently UPDATEs with an INSERT into the temp table; then in FinalizeFastSync(), aggregate the temp table into UPDATEs in the primary tables.

This PR

Running the fast-sync analyzer with almost any parallelism (as low as parallelism=5) caused significant DB lock contention and consequently slowdowns.
This PR mostly removes the contention by replacing table UPDATEs with INSERTs into new temporary tables in a new updates_todo schema. Then after fast-sync, those temp tables get merged back into the "real" tables.

Testing: I locally modified the e2e_regression config to use fast-sync for the majority of the scanned range. Confirmed there were no diffs to the expected output.

@@ -92,16 +92,6 @@ var (
ON CONFLICT (analyzer, height) DO UPDATE SET locked_time = excluded.locked_time
RETURNING height`

// IsAnyBlockInRangeProcessedByFastSync = `
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated obsolete code

@mitjat mitjat force-pushed the mitjat/less-dead-reckon branch from 9b54b39 to 1b48f64 Compare November 16, 2023 22:26
Copy link
Collaborator

@Andrew7234 Andrew7234 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, thanks!

analyzer/consensus/consensus.go Outdated Show resolved Hide resolved
analyzer/queries/queries.go Outdated Show resolved Hide resolved
analyzer/queries/queries.go Outdated Show resolved Hide resolved
@mitjat mitjat force-pushed the mitjat/less-dead-reckon branch from 0514168 to 873ced8 Compare November 21, 2023 22:50
@mitjat mitjat enabled auto-merge November 21, 2023 22:51
@mitjat mitjat merged commit 33248c2 into main Nov 21, 2023
@mitjat mitjat deleted the mitjat/less-dead-reckon branch November 21, 2023 22:55
@csillag csillag mentioned this pull request Dec 20, 2024
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants