Added reversed for proof-of-stake syncing #3092

Giulio2002 · 2021-12-05T16:52:01Z

No description provided.

Giulio2002 · 2021-12-10T01:02:46Z

Note: this is just an experimental version, it does not make use of request skeleton and it can roughly download 4000 blocks/second... treat this as being a mere prototype, it was able to sync all the way down from block 13,665,338 in 57 minutes roughly, all subsequent stages are executed correctly.

AlexeyAkhunov · 2021-12-10T07:05:36Z

Sorry, I introduced some conflicts for your with my simplification PR. Once you resolved, I will review this one again

eth/stagedsync/stage_headers.go

AlexeyAkhunov · 2021-12-11T23:07:36Z

eth/stagedsync/stage_headers.go

+	var req headerdownload.HeaderRequest
+	for !stopped {
+		sentToPeer := false
+		for !sentToPeer {


Sorry, it is this loop that will be mostly tight

That is one of the reason in the forward sync I limited number of iterations to 64, and also inserted select clause with timer, with ctx.Done() and also with printing progress. I recommend you do it also here

if i used a 30 seconds timer i would get 6.8 blk/sec, what i could do is sentToPeer==false ? then i sleep for 30 milliseconds, by doing it i dont implode the CPU.

turbo/stages/headerdownload/header_algos.go

eth/stagedsync/stage_headers.go

AlexeyAkhunov

That's probably enough commenting for now, I think we can start with that. But there is still room for improvements :)

yperbasis · 2021-12-12T11:51:45Z

cmd/sentry/download/downloader.go

-				canRequestMore = canRequestMore || requestMore
-				if len(penalties) > 0 {
-					cs.Penalize(ctx, penalties)
+			if cs.Hd.GetBackwards() {


I'm a bit confused about this backwards flag. Firstly, my understanding is that the existing (pre-Merge) downloader downloads headers mostly (besides skeleton requests) backwards, towards the genesis. So "backwards" meaning PoS sync is a bit misleading. Secondly, why is it all or nothing mode? Couldn't exclusively backwards downloading of PoS blocks run simultaneously with the old skeleton+backwards algorithm for pre-Merge blocks? To sum up, to my mind "GetBackwards" is a misleading name.

I changed it to POSSync, the flag indicates when we are doing sync for proof-of-stake or not

So does it mean that we can't download PoS and PoW headers simultaneously?

we can, if we assume that the block supplied by the beacon chain is correct, we sync backwards until we find a block whose parenthash is present in our database, this can be past the terminal block. if such thing happen, we unwind

Hmm. What about a node that wants to sync from scratch some time after the Merge? It doesn't have any PoW headers in the DB yet.

Okay, i think that headersForward means that headers are inserted in "forward" order, if you were remove preverified hashes, you would have to consensus check in forward order(genesis to head), however in our case we process the headers in downward order we start from head and go to genesis

If you think that headersFoward is wrong/misleading that can be refactored in another PR, it is not in the scope of this one i believe

Hmm. I see the following problem with inserting PoS headers into the DB in the downwards direction. Previously we had a single number for header stage's progress: the highest header inserted into the DB. Now there are essentially two numbers: the highest PoW header & the lowest PoS header that are inserted into the DB. So what happens when a node is restarted while there's a gap between the two? It doesn't fit into our current notion of a single number describing stage progress.
I'm wondering why do we have to insert PoS headers into the DB in the downwards direction anyway? To my mind it's simpler to tweak the current algo, have a long PoS segment growing downwards in memory, and then insert it into the DB when it touches the terminal PoW block (or a pre-verified PoS block after the Merge). @AlexeyAkhunov what do you think?

We need to insert them backwards to check their validity, we need to check if previous block parenthash is equal to current block hash and also if we need to unwind

But we can verify parent hashes in memory, before committing to the DB, and grow in memory a long PoS segment from the Beacon chain tip downwards until we touch a PoW block in the DB. That's what the current algo does: it grows segments in memory downwards.

yperbasis

There is some git merge/rebase issue in the PR: I see unrelated stuff like ParliaConfig

yperbasis

As discussed, let's rename HeadersDownward & HeadersForward to something like HeadersPoS & HeadersPoW.

Giulio2002 marked this pull request as draft December 5, 2021 16:52

yperbasis added the proof-of-stake label Dec 7, 2021

reverse sync slow but working

5ca7263

Giulio2002 force-pushed the reverse-download branch from 0fa980c to 5ca7263 Compare December 7, 2021 15:38

yperbasis mentioned this pull request Dec 7, 2021

[TheMerge] Kintsugi tracking #3078

Closed

12 tasks

Giulio2002 added 9 commits December 9, 2021 00:43

progress

a30cde0

p

0e08e99

Merge remote-tracking branch 'origin/devel' into optimistic-sync

c1ef4eb

close request

3b133e6

backwards refactoring

7d357ae

cleanup 1

cc2ddcf

added RequestAssembler

9f39c16

remove trash code and spaghetti

7c2d6c5

efficient

3568d3c

Giulio2002 force-pushed the reverse-download branch from 4713232 to 3568d3c Compare December 9, 2021 15:52

Giulio2002 added 9 commits December 9, 2021 22:06

fix

6a19eeb

refactor

aff42e7

tf

d9d6470

refact

5a80bdf

final refactoring

0ecde3b

headers forward restored

c8cfb55

Merge remote-tracking branch 'origin/devel' into reverse-download

456560a

test fixed

15e90f8

make CI happy

ebfd90b

Giulio2002 marked this pull request as ready for review December 10, 2021 00:59

Giulio2002 requested review from enriavil1 and yperbasis December 10, 2021 00:59

resolved

eb62c97

Giulio2002 added 2 commits December 11, 2021 23:05

removed debug config

850ff37

avoid pointers

c62845c

AlexeyAkhunov reviewed Dec 11, 2021

View reviewed changes

eth/stagedsync/stage_headers.go Show resolved Hide resolved

AlexeyAkhunov reviewed Dec 11, 2021

View reviewed changes

turbo/stages/headerdownload/header_algos.go Outdated Show resolved Hide resolved

AlexeyAkhunov reviewed Dec 11, 2021

View reviewed changes

eth/stagedsync/stage_headers.go Outdated Show resolved Hide resolved

added sleep

b0a74d5

AlexeyAkhunov reviewed Dec 12, 2021

View reviewed changes

eth/stagedsync/stage_headers.go Outdated Show resolved Hide resolved

use of channels

9af084a

Giulio2002 force-pushed the reverse-download branch from 09a0ab3 to 9af084a Compare December 12, 2021 10:25

sleeping

2186e3e

Giulio2002 force-pushed the reverse-download branch from f99b813 to 2186e3e Compare December 12, 2021 10:28

AlexeyAkhunov approved these changes Dec 12, 2021

View reviewed changes

added logs for ETL

58a21a7

Giulio2002 force-pushed the reverse-download branch from 2cdff6f to 58a21a7 Compare December 12, 2021 11:12

yperbasis reviewed Dec 12, 2021

View reviewed changes

added more cleanup

daafe78

Giulio2002 force-pushed the reverse-download branch from 629e335 to daafe78 Compare December 12, 2021 12:06

Giulio2002 added 5 commits December 12, 2021 13:08

correct glacier

2cd664d

some refactoring

2d52f79

maxRequests

0527723

tweaks

7f9e9a8

config.go

c88561e

yperbasis requested changes Dec 13, 2021

View reviewed changes

config conflicts

1f8d95b

yperbasis approved these changes Dec 13, 2021

View reviewed changes

renamed functions

2e00bfd

Giulio2002 merged commit 23b3c1d into devel Dec 13, 2021

Giulio2002 deleted the reverse-download branch December 13, 2021 16:46

VBulikov mentioned this pull request Feb 5, 2025

Proof of stake #13695

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added reversed for proof-of-stake syncing #3092

Added reversed for proof-of-stake syncing #3092

Giulio2002 commented Dec 5, 2021

Giulio2002 commented Dec 10, 2021

AlexeyAkhunov commented Dec 10, 2021

AlexeyAkhunov Dec 11, 2021

AlexeyAkhunov Dec 11, 2021

Giulio2002 Dec 12, 2021 •

edited

Loading

AlexeyAkhunov left a comment

yperbasis Dec 12, 2021 •

edited

Loading

Giulio2002 Dec 12, 2021

yperbasis Dec 12, 2021

Giulio2002 Dec 12, 2021

yperbasis Dec 12, 2021

Giulio2002 Dec 13, 2021

Giulio2002 Dec 13, 2021

yperbasis Dec 13, 2021 •

edited

Loading

Giulio2002 Dec 13, 2021 •

edited

Loading

yperbasis Dec 13, 2021

yperbasis left a comment

yperbasis left a comment

Added reversed for proof-of-stake syncing #3092

Added reversed for proof-of-stake syncing #3092

Conversation

Giulio2002 commented Dec 5, 2021

Giulio2002 commented Dec 10, 2021

AlexeyAkhunov commented Dec 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Giulio2002 Dec 12, 2021 • edited Loading

Choose a reason for hiding this comment

AlexeyAkhunov left a comment

Choose a reason for hiding this comment

yperbasis Dec 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yperbasis Dec 13, 2021 • edited Loading

Choose a reason for hiding this comment

Giulio2002 Dec 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yperbasis left a comment

Choose a reason for hiding this comment

yperbasis left a comment

Choose a reason for hiding this comment

Giulio2002 Dec 12, 2021 •

edited

Loading

yperbasis Dec 12, 2021 •

edited

Loading

yperbasis Dec 13, 2021 •

edited

Loading

Giulio2002 Dec 13, 2021 •

edited

Loading