fixes for performing the overlay transition #203

gballet · 2023-05-05T18:01:49Z

No description provided.

gballet · 2023-05-05T18:03:53Z

core/state/database.go

 		if err != nil {
 			return nil, err
 		}
-		return trie.NewTransitionTree(mpt.(*trie.SecureTrie), vkt.(*trie.VerkleTrie)), nil
+		return trie.NewTransitionTree(mpt.(*trie.SecureTrie), vkt.(*trie.VerkleTrie), true, fmt.Sprintf("%x %x %x", stateRoot, addrHash, root)), nil


The true parameter specifies that this is a storage tree (see below). The fmt.Sprintf stuff was for debugging, it should not be committed.

gballet · 2023-05-05T18:04:35Z

core/state/database.go

@@ -205,11 +205,11 @@ func (fdb *ForkingDB) OpenStorageTrie(stateRoot, addrHash, root common.Hash, sel
 	if fdb.started && err == nil {
 		// Return a "storage trie" that is an adapter between the storge MPT
 		// and the unique verkle tree.
-		vkt, err := fdb.VerkleDB.OpenStorageTrie(stateRoot, addrHash, fdb.translatedRoots[root], self)
+		vkt, err := fdb.VerkleDB.OpenStorageTrie(stateRoot, addrHash, fdb.translatedRoots[root], self.(*trie.TransitionTrie).Overlay())


I had to add an extra parameter to OpenStorageTrie so that the single verkle tree can be reused, see implementation of OpenStorageTrie

gballet · 2023-05-05T18:05:25Z

core/state/database.go

@@ -193,7 +193,7 @@ func (fdb *ForkingDB) CopyTrie(t Trie) Trie {

 	if fdb.started {
 		overlay := fdb.VerkleDB.CopyTrie(t)
-		return trie.NewTransitionTree(mpt.(*trie.SecureTrie), overlay.(*trie.VerkleTrie))
+		return trie.NewTransitionTree(mpt.(*trie.SecureTrie), overlay.(*trie.VerkleTrie), false /* XXX */, "copied")


the false is incorrect, there should be an accessor for whether or not the tree is a storage tree, see implementation below.

gballet · 2023-05-05T18:06:05Z

core/state_processor.go

-			for count < maxMovedCount {
-				if accIt.Next() {
+			for count < maxMovedCount && accIt.Next() {
+				count++ // count increase for the account itself


count++ was not in the right location

gballet · 2023-05-05T18:06:21Z

core/state_processor.go


 			// if less than maxCount slots were moved, move to the next account
-			for count < maxMovedCount {
-				if accIt.Next() {
+			for count < maxMovedCount && accIt.Next() {


just a code simplification

gballet · 2023-05-05T18:07:13Z

core/state_processor.go

@@ -150,12 +155,25 @@ func (p *StateProcessor) Process(block *types.Block, statedb *state.StateDB, cfg
 					}

 					if acc.HasStorage() {
+					stIt, err := statedb.Snaps().StorageIterator(mpt.Hash(), accIt.Hash(), common.Hash{})


the storage iterator was opened only once, for the last processed account. It needs to be opened each time an account has storage.

gballet · 2023-05-05T18:29:03Z

core/state_processor.go

+							value     []byte   // slot value after RLP decoding
+							safeValue [32]byte // 32-byte aligned value
+						)
+						if err := rlp.DecodeBytes(stIt.Slot(), &value); err != nil {


in the MPT, the data is serialized using RLP. It must be deserialized before it is inserted.

gballet · 2023-05-05T18:32:19Z

trie/transition.go

+	}
+	// the value will come as RLP, decode it so that the
+	// interface is consistent.
+	_, content, _, err := rlp.Split(rlpval)


in the MPT, the data is serialized using RLP. The verkle tree does not do this, so to keep it consistent, deserialize here.

gballet · 2023-05-05T18:34:57Z

trie/transition.go

 	"github.com/gballet/go-verkle"
 )

 type TransitionTrie struct {
 	overlay *VerkleTrie
 	base    *SecureTrie
+	storage bool


The calling code differentiates between account tries and storage tries. Even though the verkle tree is a single tree, there are multiple transition trees, one for the (unique overlay tree, unique account trie) combination, and one for each of the (unique overlay tree, storage trie for a given account) combination.

As a result, this flag is added to specify if the current TransitionTree corresponds to the account trie or any of the storage tries.

One thing that this flag should be used for, is to not Commit when set, otherwise the tree will be committed over and over again, every time an account's storage is updated. This will kill performance, so Commit should be a no-op unless storage == false.

Two questions/points:

storage is only used in the NewTransitionTree, so technically "not used". Maybe your comment was telling something that we still have to do? (Just checking we're aware that, for now is something unused)

Your comment made me realize that in state_processor.go we never re-set tt each time the MPT-tree changes. For example, state_processor.go L158 is using mpt.Hash() and mpt was only assigned once doing tt.Base().. but at that point we might already jump to another tree, right? Feels like we should reopen tt when we jump to another account?

you are correct storage isn't used, I should remove it

the MPT tree becomes read only, so we shouldn't have to update it. I'm probably missing your point here.

Reg the second point, dismiss it. I've been trying to double-check (since I said this three weeks ago 😆 ) if I was referring to something else, but doesn't seem convincing to me. So I think this is fine!

gballet · 2023-05-05T18:36:09Z

trie/verkle.go

@@ -68,7 +68,7 @@ func (trie *VerkleTrie) GetKey(key []byte) []byte {
 // not be modified by the caller. If a node was not found in the database, a
 // trie.MissingNodeError is returned.
 func (trie *VerkleTrie) TryGet(addr, key []byte) ([]byte, error) {
-	pointEval := trie.pointCache.GetTreeKeyHeader(key)
+	pointEval := trie.pointCache.GetTreeKeyHeader(addr)


that's what you get when you copy/paste, it was using the slot number instead of the address.

jsign · 2023-05-07T20:10:19Z

trie/transition.go

 }

-func NewTransitionTree(base *SecureTrie, overlay *VerkleTrie) *TransitionTrie {
+func NewTransitionTree(base *SecureTrie, overlay *VerkleTrie, st bool, sig string) *TransitionTrie {


Does the fact that sig is an unused parameter something to be considered? (Or maybe planning already something else?)

it should have been removed, I will push a fix to this effect in a few minutes

jsign · 2023-05-07T20:13:38Z

trie/transition.go

+	_, content, _, err := rlp.Split(rlpval)
+	if err != nil || len(content) == 0 {
+		fmt.Println(err)
+		return nil, err


Maybe we should split the case when len(content) == 0?
In that case, printing and returning err would only print/return nil which can be confusing.

So maybe if len(content) == 0 do some errors.New(...).

that was a debug thing, so I removed it. Do you still think that I should split both cases?

Do you still think that I should split both cases?

No, actually I think this is fine. I thought the intention was to return a non-nil error if len(content)==0, but that's not actually the case. (Maybe my confusion is part of the split suggestion :))

jsign · 2023-05-07T20:26:04Z

core/state_processor.go

@@ -128,10 +129,14 @@ func (p *StateProcessor) Process(block *types.Block, statedb *state.StateDB, cfg
 				slotnr := rawdb.ReadPreimage(statedb.Database().DiskDB(), stIt.Hash())
 				mkv.addStorageSlot(addr, slotnr, stIt.Slot())
 			}
+			if count == maxMovedCount {
+				stIt.Release()


Should we in L110 do some kind of defer accIt.Release() too?

jsign · 2023-05-07T20:28:59Z

core/state_processor.go

@@ -128,10 +129,14 @@ func (p *StateProcessor) Process(block *types.Block, statedb *state.StateDB, cfg
 				slotnr := rawdb.ReadPreimage(statedb.Database().DiskDB(), stIt.Hash())
 				mkv.addStorageSlot(addr, slotnr, stIt.Slot())
 			}
+			if count == maxMovedCount {


I think I mentioned something some days ago that we forgot to do; skipping the "first" position.

If LastAccHash is an EOA, then it should be skipped (since AccountIterator would repeat again the migration logic)

If LastAccHash isn't an EOA, then we should skip the first result of the stIt to not double-migrate the last migrated storage slot.

I have entirely rewritten this part, this should be pretty clear (let me know if it isn't / you think there's a bug)

jsign · 2023-05-07T20:31:59Z

core/state_processor.go

+							safeValue [32]byte // 32-byte aligned value
+						)
+						if err := rlp.DecodeBytes(stIt.Slot(), &value); err != nil {
+							return nil, nil, 0, fmt.Errorf("error decoding bytes %x: %w", stIt.Slot(), err)


Should we stIt.Release() here before returning?

called defer like you suggested

jsign · 2023-05-07T20:37:10Z

core/state_processor.go

@@ -166,7 +184,6 @@ func (p *StateProcessor) Process(block *types.Block, statedb *state.StateDB, cfg
 			} else {
 				// Update the pointers in the forking db
 				fdb.LastAccHash = accIt.Hash()


Uhm, isn't this line suffering from a side-effect caused by L182?
While trying to detect if we finished the conversion, we called actIt.Next(). If there's still work, it will return true and move the iterator one place forward. And when executing L186, accIt.Hash() will save as the "last processed" the "next" one instead of the last processed one. That doesn't sound right if the last processed one still had storage-slots to migrate.

jsign · 2023-05-07T20:47:50Z

trie/transition.go

 	"github.com/gballet/go-verkle"
 )

 type TransitionTrie struct {
 	overlay *VerkleTrie
 	base    *SecureTrie
+	storage bool


Two questions/points:

storage is only used in the NewTransitionTree, so technically "not used". Maybe your comment was telling something that we still have to do? (Just checking we're aware that, for now is something unused)

Your comment made me realize that in state_processor.go we never re-set tt each time the MPT-tree changes. For example, state_processor.go L158 is using mpt.Hash() and mpt was only assigned once doing tt.Base().. but at that point we might already jump to another tree, right? Feels like we should reopen tt when we jump to another account?

jsign

Leaving some comments.

jsign · 2023-05-26T20:24:01Z

core/state/database.go

-	LastAccHash     common.Hash                 // hash of the last translated account
-	LastSlotHash    common.Hash                 // hash of the last translated storage slot
+	started, ended      bool
+	translatedRoots     map[common.Hash]common.Hash // hash of the translated root, for opening\


jsign · 2023-05-26T20:31:34Z

core/state/database.go

@@ -205,11 +213,13 @@ func (fdb *ForkingDB) OpenStorageTrie(stateRoot, addrHash, root common.Hash, sel
 	if fdb.started && err == nil {
 		// Return a "storage trie" that is an adapter between the storge MPT
 		// and the unique verkle tree.
+		fdb.translatedRootsLock.RLock()


Why do we need a mutex?
Is this related to the prefetcher? (but that shouldn't be "on" in VKT mode?)
Maybe it will become apparent later when I continue with the review.

we need a mutex because the map is accessed from the processor and the state prefetcher, that is on (not the trie prefetecher, that is off).

jsign · 2023-05-26T20:45:15Z

core/state/state_object.go

@@ -221,11 +222,15 @@ func (s *stateObject) GetCommittedState(db Database, key common.Hash) common.Has
 	}
 	// If the snapshot is unavailable or reading from it fails, load from the database.
 	if s.db.snap == nil || err != nil {
+		var tr = s.getTrie(db)


Can we move this line inside the if at L227?

jsign · 2023-05-26T20:51:41Z

core/state/state_object.go

+			v, err = tr.TryGet(s.address[:], key.Bytes())
+			copy(value[:], v)
+		} else {
+			enc, err = s.getTrie(db).TryGet(s.address[:], key.Bytes())


OK, despite this looks correct it's a bit weird-ish at first sight, because:

L230: We do the copy(value[:], v) even if err != nil

L232: Doesn't have a symmetrical copy to value.

L237: is actually capturing the err != nil for both cases.

L242: is decoding if len(enc)>0 which is detecting we entered the if-branch in L232 or we come from the snapshot, setting the value that we want.

To be clear, the logic is correct and it's trying to be a bit smart into "merging" the potential RLP decoding in a single place, considering you can come from different places. And eagerly doing that L230 copy to "accommodate" into that.

Probably we can come up with a better way to describe this logic that doesn't look that weird. Feels like this code is waiting to be a bug the next time someone adds some extra line of code around here.

To be honest, I don't have a good suggestion now. We can think about it when cleaning up all the work maybe.
</rant :) >

this temporary code, I have a geth PR awaiting a merge that should make all this simpler

jsign · 2023-05-26T20:52:43Z

core/state/state_object.go

 	var tr Trie
-	if s.db.trie.IsVerkle() {
-		tr = s.db.trie
-	} else {
-		tr = s.getTrie(db)
-	}
+	tr = s.getTrie(db)


Nit: merge both L330 and L331 in tr := s.getTrie(db).

yes it should be simplified like this

jsign · 2023-05-26T21:49:50Z

core/state_processor.go

+						// move the storage to the next value, so that if the max
+						// leaf count is reached, the loop interrupted, and resumed
+						// at the next block, `stIt.Next()` will check fot the presence
+						// of the next slot, and mark the whole storage as "processed"
+						// if no such slot can be found.
+						for x := 31; x >= 0; x-- {
+							if fdb.LastSlotHash[x] < 255 {
+								break
+							}
+							fdb.LastSlotHash[x] = 0
+
+							// This means that 0xff....ff will circle back to 0x00..00
+							// but it's not a problem because the next call to stIt.Next()
+							// will update the processing marker, so the iterator will
+							// not be used again.
+						}


I'm a bit confused by this with the comment move the storage to the next value, so let me put some examples:

If fdb.LastSlotHash = 0x00...ff00, this is leaving fdb.LastSlotHash to the same value. Shouldn't the result be 0x00...ff01?

If fdb.LastSlotHash = 0x00...ff this will set fdb.LastSlotHash = 0x00...00 (i.e: zero). So looks like it is decreasing the number not moving to next value.

What I understand you're trying to do here is to always leave LastSlotHash in the right place if reached the max count. That makes sense, but I don't see how this code is doing that. If I'm not understanding something incorrectly, wouldn't be easier to transform the common.Hash to a big-int and do +1 mod 256 ?

Also, if this is the case LastSlotHash isn't really the last migrated slot, but more like the "next" one; isn't it?

yeah the name is confusing, also that's indeed what I'm trying to do: +1 mod 256, but wiothout using big.Int :D

and indeed, this code is broken. Well, not completely: in the worst case, some values are converted twice, but it's still incorrect.

jsign · 2023-05-26T21:52:17Z

core/state_processor.go

-			for count < maxMovedCount && accIt.Next() {
-				count++ // count increase for the account itself
+			for accIt.Next() && count < maxMovedCount {
+				fdb.LastAccHash = accIt.Hash()


Wouldn't a rename from LastAccHash to CurrAccHash be more correct?
fdb.LastAccHash is pointing at the current account that should be continued to be processed.

If LastAccHash sounds to me more like "the last fully migrated account". Also, the database.go comment seems to indicate that's the case when it isn't:
LastAccHash common.Hash // hash of the last translated account

jsign · 2023-05-26T21:58:56Z

core/state_processor.go

+				// XXX There is an undhandler corner case here: if an account's
+				// storage is processed, but the account itself is not, and this
+				// account is the last one, then the transition will be marked
+				// as complete, but missing the last account.


Can we simplify the counting and not count the account as an extra counting?
Yes, it won't be 100% precise the limit, but it would remove at least some corner cases.

If the limit end up been 1k or 10k, doing +1 doesn't sound like a big deal, and we'll always be sure that if fdb.StorageProcessed == true then the account was also migrated too.

No strong opinions about this, just saying that maybe if we can allow that definition (because we're still inventing rules now?), maybe that can simplify and avoid weird corner cases that might never trigger in tests (it's a very subtle corner case) and suddenly we wake up in mainnet with being unlucky in some EL client not handling this correctly.

Or keep the current precise counting and put a TODO somewhere "Please, add this case to a spec test-vector"

the only problem I see with this approach is that the code can be up to 24kb, which means 700+ leaves, but yeah I'll revert that if it works for me.

jsign · 2023-05-26T22:14:00Z

core/state_processor.go

+
+					// reset storage iterator marker for next account
+					fdb.LastSlotHash = common.Hash{}
+				}
 			}


I think there's a corner case that isn't correctly handled here; let me explain to double-check with you.

Let's suppose we're at L206 and fdb.StorageProcessed == true and count < maxMovedCount. OK, we finished with the storage slots, and we also have room to migrate the account. We migrate the account so count++.

We'll jump again to the main loop at L125 doing for accIt.Next() && count < maxMovedCount, but suppose that count < maxMovedCount == false. i.e the last account that we fully migrated fitted perfectly into the limit.

In this situation, fdb.LastAccHash is still that fully migrated account... so in the next block looks like we'll migrate it again. This sounds like a similar situation that (I think) you wanted to solve for storage-slots.

right, initially I was doing like you said: not checking for count < maxMovedCount. I even have a comment to that effect. So you are correct that we need something similar, and I'm starting to see the wisdom in what you said before: revert and add a TODO.

jsign · 2023-05-26T22:21:26Z

trie/verkle.go

+	if acc.Nonce == 0 && len(values) > 10 && len(values[10]) > 0 && bytes.Equal(values[4], zero[:]) {
+		// WORKAROUND: detect if this account has been deleted: if the
+		// 10th entry is 0, it means that the account has been deleted
+		// at least once, and if the code keccak is also zero, then it
+		// means that the account is currently deleted.
+		// Detecting a deleted account is important because otherwise,
+		// some strange bugs occur where an account is returned with a
+		// negative balance, which can be quite tricky to debug.


Is this 10th element a new thing that isn't in the spec?
Looking there, vector item at position 10 doesn't seem to have any meaning.

yes, this is why I can detect that an account has been deleted and then return nil if that is the case. This will otherwise trigger the following bug: if an account is deleted from an EVM point of view, in verkle it's overwritten with 0s. But if then a contract tries to read that deleted account, it will find "something": an account with 0s everywhere. So we need to find some sort of marker that an account has been deleted. Because value 10 is unused, seing it set to 0 was used as a marker.

Now there's a few problems with this (and in fact, it caused a bug at block 1251415): if the account was deleted then recreated, it would think the account was still deleted since value 10 was set. A better indicator is to check the code hash, if it's 0s then this is an empty account.

A better indicator is to check the code hash, if it's 0s then this is an empty account.

Cool, never old to (ab)use a sentinel value there, considering the empty code hash is not zero. :P

Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification

Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB

* overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]>

Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (#137) trie: avoid endianness conversion in GetTreeKey (#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <[email protected]> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <[email protected]> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <[email protected]> Signed-off-by: Ignacio Hagopian <[email protected]> upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138) updating ci to use self-hosted machine (#143) fix: storage offset in non-header group + reuse of value buffer (#145) dedup call to ChunkifyCode, same as replay branch (#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (#157) fix REVERT in state processor test execution (#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (#159) fix code chunk key calculation and storage key calculation (#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <[email protected]> fix: infinite loop when calling extcodecopy on empty code (#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (#174) fix: read-touch the code size and Keccak of the origin (#175) List of changes for converting a sepolia database (#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> trie/utils: fix potential overflow (#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <[email protected]> * revert formatter changes Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> trie/utils: fix incorrect bigint assignment (#193) Signed-off-by: Ignacio Hagopian <[email protected]> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (#192) fixes to use the latest go-verkle@master (#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (#204) fix gas accounting issue in state_processor_test.go (#207) update go-verkle not to use StatelessNode anymore (#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (#205) add missing error checks for the root node type (#214) implement OpenStorageTrie for verkle trees (#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (#217) implement some heretofore unimplemented iterator methods (#219) params: move verkle params to their own file (#228) fix: proper number of chunk evals (#215) overlay transition (#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> review changes remove replay-specific code

Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (#137) trie: avoid endianness conversion in GetTreeKey (#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <[email protected]> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <[email protected]> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <[email protected]> Signed-off-by: Ignacio Hagopian <[email protected]> upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138) updating ci to use self-hosted machine (#143) fix: storage offset in non-header group + reuse of value buffer (#145) dedup call to ChunkifyCode, same as replay branch (#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (#157) fix REVERT in state processor test execution (#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (#159) fix code chunk key calculation and storage key calculation (#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <[email protected]> fix: infinite loop when calling extcodecopy on empty code (#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (#174) fix: read-touch the code size and Keccak of the origin (#175) List of changes for converting a sepolia database (#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> trie/utils: fix potential overflow (#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <[email protected]> * revert formatter changes Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> trie/utils: fix incorrect bigint assignment (#193) Signed-off-by: Ignacio Hagopian <[email protected]> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (#192) fixes to use the latest go-verkle@master (#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (#204) fix gas accounting issue in state_processor_test.go (#207) update go-verkle not to use StatelessNode anymore (#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (#205) add missing error checks for the root node type (#214) implement OpenStorageTrie for verkle trees (#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (#217) implement some heretofore unimplemented iterator methods (#219) params: move verkle params to their own file (#228) fix: proper number of chunk evals (#215) overlay transition (#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> review changes remove replay-specific code move UpdateContractCode to updateStateObject fix: generated rlp decoders fix: activate Shanghai in test add withdrawals to witness fix the first half of verkle test fix: open an empty storage trie after transition activate verkle on IsVerkle, not IsCancun deactivate snapshot for tests enable proof of stake in verkle test save last MPT root for transition remove unnecessary snapshot Cap in flush fix test: include EIP-3860 implement missing odrDatabase function fix incorrect equality condition is HasAccount fixes to replay ~500 blocks fix to replay more blocks fix preimage issue in conversion code cleanup: remove a lot of TODOs & friends more code cleanup fix: alignment of values whose len is < 32 New access-witness module (#235) Use plain addresses/slot numbers instead of hashing them in the witness remove unused map param in MakeVerkleMultiProof (#261) core/state: rewrite a new optimized keyValueMigrator (#256) * trie/utils: add helper to calculate code tree indices * core/state: rewrite optimized version of keyValueMigrator Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: remove uint256 allocs (#257) Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> core: move overlay conversion code to its own file (#266) add pre-pbss rebase branches to CI (#270) * add post-pbss rebase branches to CI * fix go version in CI * fix linter issues * upgrade go version to 1.21.1 to avoid github deploy error quell zero-tree message port kaustinen code on top of shapella-rebased branch activate proof generation on fork + remove code dups use go-verkle's post-state API to verify proofs (#262) use prague as the verkle activation fork (#263) upgrade to latest go-ipa activate verkle transition in "miner" (#265) fix: do not force cancunTime upon verkle activation workaround: do not use root translation in replay workaround: deactivate overlay transition for now fixes from trying to get the devnet to work (#267) this line was left out from the previous commit upgrade to go-verkle with fixed newvalue serialization fix: ensure point cache isn't nil in copy (#268) fix: dependency cycle in tests (#269) upgrade to latest go-verkle fix: write trie preimage data to db (#274) fix: zero-root in produced block + sync (#275) upgrade go-ipa fix build fix typo include review feedback add switch to add proofs to blocks (#278) add fee recipient to witness (#279) touch all fields in withdrawal account header (#277) fix: add ProofInBlocks to chain config (#280) remove StateDB as an extra param to Prepare (#281) fix: need commitment in tests (#282) fix linter message fix: a couple of CI issues fix more CI complaints achieving perfection workaround: disable check for root presence (#283) fix: activate verkle at genesis (#284) fix: ensure read-only values are resolved in post trie (#285) upgrade deps to get proof generation absence/presence bugfix fix: make sure AccessWitness isn't nil when calling RPC methods (#287) fix: incorrect access copy (#288) fix: return serialized root in state root, not its mapping to a scalar field (#289) add workaround to use --override.prague (#292) Shanghai time workaround (#294) * add workaround to use --override.prague * fix typo in comments * fix: enable overrides at genesis time remove pre-state tree warmup (#293) * remove pre-state tree warmup Signed-off-by: Ignacio Hagopian <[email protected]> * update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>

Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (#137) trie: avoid endianness conversion in GetTreeKey (#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <[email protected]> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <[email protected]> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <[email protected]> Signed-off-by: Ignacio Hagopian <[email protected]> upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138) updating ci to use self-hosted machine (#143) fix: storage offset in non-header group + reuse of value buffer (#145) dedup call to ChunkifyCode, same as replay branch (#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (#157) fix REVERT in state processor test execution (#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (#159) fix code chunk key calculation and storage key calculation (#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <[email protected]> fix: infinite loop when calling extcodecopy on empty code (#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (#174) fix: read-touch the code size and Keccak of the origin (#175) List of changes for converting a sepolia database (#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> trie/utils: fix potential overflow (#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <[email protected]> * revert formatter changes Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> trie/utils: fix incorrect bigint assignment (#193) Signed-off-by: Ignacio Hagopian <[email protected]> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (#192) fixes to use the latest go-verkle@master (#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (#204) fix gas accounting issue in state_processor_test.go (#207) update go-verkle not to use StatelessNode anymore (#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (#205) add missing error checks for the root node type (#214) implement OpenStorageTrie for verkle trees (#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (#217) implement some heretofore unimplemented iterator methods (#219) params: move verkle params to their own file (#228) fix: proper number of chunk evals (#215) overlay transition (#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> review changes remove replay-specific code

fixes for performing the overlay transition

4c4205b

gballet requested a review from holiman as a code owner May 5, 2023 18:01

gballet commented May 5, 2023

View reviewed changes

jsign reviewed May 7, 2023

View reviewed changes

fixes for the full replay

426ae7c

jsign reviewed May 26, 2023

View reviewed changes

gballet added 5 commits June 1, 2023 14:39

fix: deletion-and-recreation of EoA

95ccc05

fixes to replay 2M+ blocks

b10b176

upgrade to go-verkle@master

249cd18

fix: proper number of chunk evals

0d2bd5d

rewrite conversion loop to fix known issues

f75311a

gballet force-pushed the overlay/fix/complete-transition branch from 38b8ddb to f75311a Compare June 7, 2023 11:56

gballet merged commit b49cef4 into overlay-implementation Jun 7, 2023

fixes for performing the overlay transition #203

fixes for performing the overlay transition #203

Conversation

gballet commented May 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gballet May 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign May 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign May 7, 2023 • edited Loading

Choose a reason for hiding this comment

jsign May 7, 2023 • edited Loading

Choose a reason for hiding this comment

jsign left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gballet May 5, 2023 •

edited

Loading

jsign May 7, 2023 •

edited

Loading

jsign May 26, 2023 •

edited

Loading

jsign May 7, 2023 •

edited

Loading

jsign May 7, 2023 •

edited

Loading