Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e3: history no auto-increment #7097

Merged
merged 144 commits into from
Mar 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
144 commits
Select commit Hold shift + click to select a range
449b5d2
save
AskAlexSharov Feb 23, 2023
986ccdb
save
AskAlexSharov Feb 23, 2023
a1a6512
save
AskAlexSharov Feb 23, 2023
00da255
save
AskAlexSharov Feb 23, 2023
9d2306b
save
AskAlexSharov Feb 23, 2023
b93c6ba
save
AskAlexSharov Feb 23, 2023
dafe41e
save
AskAlexSharov Feb 23, 2023
15eca45
save
AskAlexSharov Feb 24, 2023
d640ad9
save
AskAlexSharov Feb 24, 2023
a1cb6e9
save
AskAlexSharov Feb 24, 2023
27de45a
save
AskAlexSharov Feb 24, 2023
6329526
save
AskAlexSharov Feb 24, 2023
017e876
save
AskAlexSharov Feb 25, 2023
4d4b6cc
save
AskAlexSharov Feb 26, 2023
96224e9
Merge branch 'devel' into etl_empty_val
AskAlexSharov Feb 27, 2023
a9bad89
save
AskAlexSharov Feb 27, 2023
3b7a46f
save
AskAlexSharov Feb 27, 2023
d895781
Merge branch 'devel' into etl_empty_val
AskAlexSharov Feb 28, 2023
b13604d
save
AskAlexSharov Feb 28, 2023
63b20c4
save
AskAlexSharov Feb 28, 2023
fa79bcd
save
AskAlexSharov Feb 28, 2023
8604d4a
save
AskAlexSharov Feb 28, 2023
fb168cb
save
AskAlexSharov Feb 28, 2023
a6cc8c9
save
AskAlexSharov Feb 28, 2023
4706bb8
save
AskAlexSharov Feb 28, 2023
6e4f35c
save
AskAlexSharov Feb 28, 2023
c793783
Merge branch 'e3_call_tracer_reset' into e3_call_tracer_reset2
AskAlexSharov Feb 28, 2023
062f93e
save
AskAlexSharov Feb 28, 2023
6ba2448
save
AskAlexSharov Feb 28, 2023
316110a
save
AskAlexSharov Feb 28, 2023
b26f113
save
AskAlexSharov Feb 28, 2023
11570cd
save
AskAlexSharov Feb 28, 2023
40a4336
save
AskAlexSharov Feb 28, 2023
f9f00ca
save
AskAlexSharov Feb 28, 2023
ecefb12
save
AskAlexSharov Feb 28, 2023
70965f8
save
AskAlexSharov Feb 28, 2023
89753f7
save
AskAlexSharov Feb 28, 2023
5383706
save
AskAlexSharov Feb 28, 2023
f62e956
save
AskAlexSharov Feb 28, 2023
49a0b66
save
AskAlexSharov Feb 28, 2023
877375a
save
AskAlexSharov Feb 28, 2023
fa6a6f2
save
AskAlexSharov Feb 28, 2023
b20ccdd
save
AskAlexSharov Feb 28, 2023
9638894
save
AskAlexSharov Feb 28, 2023
db8ac57
save
AskAlexSharov Feb 28, 2023
c9c9a3d
save
AskAlexSharov Feb 28, 2023
adb7a82
save
AskAlexSharov Feb 28, 2023
4dbc240
save
AskAlexSharov Feb 28, 2023
2f54fbb
save
AskAlexSharov Feb 28, 2023
19cc32d
save
AskAlexSharov Feb 28, 2023
c664c26
save
AskAlexSharov Feb 28, 2023
c52fd98
save
AskAlexSharov Feb 28, 2023
7349408
save
AskAlexSharov Feb 28, 2023
dcf5d4c
save
AskAlexSharov Feb 28, 2023
48d359f
Merge branch 'devel' into e3_kvlist_string
AskAlexSharov Feb 28, 2023
be73c64
save
AskAlexSharov Mar 1, 2023
36892d4
Merge branch 'devel' into e3_kvlist_string
AskAlexSharov Mar 1, 2023
27792fe
Merge branch 'devel' into e3_kvlist_string
AskAlexSharov Mar 1, 2023
f57c3f6
save
AskAlexSharov Mar 1, 2023
a11be3a
Merge branch 'devel' into e3_kvlist_string
AskAlexSharov Mar 2, 2023
db5950c
save
AskAlexSharov Mar 2, 2023
0b5b66a
save
AskAlexSharov Mar 2, 2023
76a9b32
save
AskAlexSharov Mar 2, 2023
25bd4b8
save
AskAlexSharov Mar 2, 2023
f662b36
save
AskAlexSharov Mar 2, 2023
6fde10c
save
AskAlexSharov Mar 2, 2023
3d814ce
save
AskAlexSharov Mar 2, 2023
4eff532
save
AskAlexSharov Mar 2, 2023
cadb6d3
Merge branch 'devel' into e3_map4
AskAlexSharov Mar 2, 2023
0f9415b
save
AskAlexSharov Mar 2, 2023
b89e3f5
save
AskAlexSharov Mar 2, 2023
a575681
save
AskAlexSharov Mar 2, 2023
d5879a9
save
AskAlexSharov Mar 2, 2023
eb9f41a
save
AskAlexSharov Mar 2, 2023
9769086
save
AskAlexSharov Mar 2, 2023
791f819
save
AskAlexSharov Mar 2, 2023
cae410e
save
AskAlexSharov Mar 2, 2023
ff08168
save
AskAlexSharov Mar 2, 2023
2ad9bb7
Merge branch 'e3_map4_faster_apply4' into e3_map5
AskAlexSharov Mar 2, 2023
d95dddb
save
AskAlexSharov Mar 3, 2023
e1d5fb7
Merge branch 'devel' into e3_map6
AskAlexSharov Mar 3, 2023
9374b1d
save
AskAlexSharov Mar 3, 2023
d0fdad5
Merge branch 'devel' into e3_map4_faster_apply2
AskAlexSharov Mar 3, 2023
a22cf07
Merge branch 'e3_map4_faster_apply2' into e3_map6
AskAlexSharov Mar 3, 2023
f6dca39
save
AskAlexSharov Mar 3, 2023
bd00e7a
save
AskAlexSharov Mar 3, 2023
1268a8a
save
AskAlexSharov Mar 3, 2023
80295a3
save
AskAlexSharov Mar 3, 2023
6443f4d
save
AskAlexSharov Mar 3, 2023
be3c12a
save
AskAlexSharov Mar 3, 2023
6dda70f
save
AskAlexSharov Mar 4, 2023
d45e885
Merge branch 'etl_empty_val' into e3_agg2
AskAlexSharov Mar 4, 2023
46b41ab
save
AskAlexSharov Mar 6, 2023
1d995c8
Merge branch 'devel' into e3_map6
AskAlexSharov Mar 6, 2023
0af411a
Merge branch 'etl_empty_val' into e3_agg2
AskAlexSharov Mar 6, 2023
737bdca
Merge branch 'e3_map6' into e3_agg2
AskAlexSharov Mar 6, 2023
ff657e2
save
AskAlexSharov Mar 6, 2023
fb77dea
save
AskAlexSharov Mar 6, 2023
2e7da84
save
AskAlexSharov Mar 6, 2023
4eb266b
save
AskAlexSharov Mar 6, 2023
cb02270
Merge branch 'devel' into e3_history_no_id
AskAlexSharov Mar 6, 2023
df80ceb
save
AskAlexSharov Mar 6, 2023
31de149
save
AskAlexSharov Mar 7, 2023
e83d77f
save
AskAlexSharov Mar 7, 2023
ecaa06a
save
AskAlexSharov Mar 7, 2023
e8d37f7
save
AskAlexSharov Mar 7, 2023
b4214a8
save
AskAlexSharov Mar 7, 2023
aeb6d6b
save
AskAlexSharov Mar 7, 2023
9401674
Merge branch 'devel' into e3_history_no_id
AskAlexSharov Mar 7, 2023
75f13f8
save
AskAlexSharov Mar 7, 2023
2777ad9
save
AskAlexSharov Mar 8, 2023
3dac0cf
save
AskAlexSharov Mar 8, 2023
1bb28fc
save
AskAlexSharov Mar 8, 2023
5e9ec4f
save
AskAlexSharov Mar 8, 2023
b1fd3ee
save
AskAlexSharov Mar 8, 2023
df21c83
remove NextBatch method
AskAlexSharov Mar 9, 2023
2bd82bb
better iter dup
AskAlexSharov Mar 9, 2023
a48bb25
save
AskAlexSharov Mar 9, 2023
10e4d9b
save
AskAlexSharov Mar 9, 2023
abcc6d0
save
AskAlexSharov Mar 9, 2023
5b8fa31
save
AskAlexSharov Mar 11, 2023
8921789
go mod tidy
AskAlexSharov Mar 13, 2023
1004d05
save
AskAlexSharov Mar 13, 2023
af44fd6
save
AskAlexSharov Mar 13, 2023
0694e1e
save
AskAlexSharov Mar 13, 2023
e90a599
save
AskAlexSharov Mar 13, 2023
9d1da8f
save
AskAlexSharov Mar 13, 2023
db69b96
save
AskAlexSharov Mar 13, 2023
c71ad85
save
AskAlexSharov Mar 13, 2023
78854d4
save
AskAlexSharov Mar 14, 2023
3583552
save
AskAlexSharov Mar 14, 2023
5cf3df9
save
AskAlexSharov Mar 14, 2023
779141a
save
AskAlexSharov Mar 14, 2023
a321181
save
AskAlexSharov Mar 14, 2023
fc2b3d7
save
AskAlexSharov Mar 14, 2023
d164892
save
AskAlexSharov Mar 14, 2023
22a30ed
save
AskAlexSharov Mar 14, 2023
1ec49ef
save
AskAlexSharov Mar 14, 2023
5a5bf1b
save
AskAlexSharov Mar 14, 2023
f10fb6d
Merge branch 'devel' into e3_history_no_id_dup
AskAlexSharov Mar 15, 2023
2a7b373
save
AskAlexSharov Mar 15, 2023
196f87c
save
AskAlexSharov Mar 15, 2023
f3f9980
save
AskAlexSharov Mar 15, 2023
399cc65
save
AskAlexSharov Mar 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions cl/cltypes/attestations_test.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package cltypes_test

import (
_ "embed"
"testing"

libcommon "github.com/ledgerwatch/erigon-lib/common"
Expand All @@ -10,8 +11,6 @@ import (
"github.com/ledgerwatch/erigon/cl/cltypes"
"github.com/ledgerwatch/erigon/cl/utils"
"github.com/ledgerwatch/erigon/common"

_ "embed"
)

var testAttData = &cltypes.AttestationData{
Expand Down
3 changes: 2 additions & 1 deletion cl/utils/crypto.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,10 @@
package utils

import (
"github.com/minio/sha256-simd"
"hash"
"sync"

"github.com/minio/sha256-simd"
)

type HashFunc func(data []byte, extras ...[]byte) [32]byte
Expand Down
1 change: 1 addition & 0 deletions cmd/erigon-cl/core/state/state.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package state

import (
"encoding/binary"

"github.com/minio/sha256-simd"

lru2 "github.com/hashicorp/golang-lru/v2"
Expand Down
3 changes: 2 additions & 1 deletion core/vm/contracts.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ package vm
import (
"encoding/binary"
"errors"
"github.com/minio/sha256-simd"
"math/big"

"github.com/minio/sha256-simd"

"github.com/holiman/uint256"
"github.com/ledgerwatch/erigon-lib/chain"
libcommon "github.com/ledgerwatch/erigon-lib/common"
Expand Down
3 changes: 2 additions & 1 deletion crypto/ecies/ecies_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,12 @@ import (
"encoding/hex"
"flag"
"fmt"
"github.com/minio/sha256-simd"
"math/big"
"os"
"testing"

"github.com/minio/sha256-simd"

"github.com/ledgerwatch/erigon/crypto"
)

Expand Down
3 changes: 2 additions & 1 deletion crypto/ecies/params.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,10 @@ import (
"crypto/elliptic"
"crypto/sha512"
"fmt"
"github.com/minio/sha256-simd"
"hash"

"github.com/minio/sha256-simd"

ethcrypto "github.com/ledgerwatch/erigon/crypto"
)

Expand Down
13 changes: 7 additions & 6 deletions eth/stagedsync/exec3.go
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,6 @@ func ExecV3(ctx context.Context,

applyWorker.ResetTx(tx)

notifyReceived := func() { rwsReceiveCond.Signal() }
var t time.Time
var lastBlockNum uint64
drainF := func(txTask *exec22.TxTask) (added int64) {
Expand Down Expand Up @@ -259,7 +258,7 @@ func ExecV3(ctx context.Context,
processedResultSize, processedTxNum, conflicts, processedBlockNum, err := func() (processedResultSize int64, processedTxNum, conflicts, processedBlockNum uint64, err error) {
rwsLock.Lock()
defer rwsLock.Unlock()
return processResultQueue(rws, outputTxNum.Load(), rs, agg, tx, triggerCount, notifyReceived, applyWorker)
return processResultQueue(rws, outputTxNum.Load(), rs, agg, tx, triggerCount, rwsReceiveCond, applyWorker)
}()
if err != nil {
return err
Expand Down Expand Up @@ -329,7 +328,7 @@ func ExecV3(ctx context.Context,
case <-pruneEvery.C:
if rs.SizeEstimate() < commitThreshold {
if agg.CanPrune(tx) {
if err = agg.Prune(ctx, ethconfig.HistoryV3AggregationStep/10); err != nil { // prune part of retired data, before commit
if err = agg.Prune(ctx, ethconfig.HistoryV3AggregationStep*10); err != nil { // prune part of retired data, before commit
return err
}
} else {
Expand Down Expand Up @@ -365,7 +364,7 @@ func ExecV3(ctx context.Context,
}
}
applyWorker.ResetTx(tx)
processedResultSize, processedTxNum, conflicts, processedBlockNum, err := processResultQueue(rws, outputTxNum.Load(), rs, agg, tx, triggerCount, func() {}, applyWorker)
processedResultSize, processedTxNum, conflicts, processedBlockNum, err := processResultQueue(rws, outputTxNum.Load(), rs, agg, tx, triggerCount, nil, applyWorker)
if err != nil {
return err
}
Expand Down Expand Up @@ -769,7 +768,7 @@ func blockWithSenders(db kv.RoDB, tx kv.Tx, blockReader services.BlockReader, bl
return b, nil
}

func processResultQueue(rws *exec22.TxTaskQueue, outputTxNumIn uint64, rs *state.StateV3, agg *state2.AggregatorV3, applyTx kv.Tx, triggerCount *atomic2.Uint64, onSuccess func(), applyWorker *exec3.Worker) (resultSize int64, outputTxNum, conflicts, processedBlockNum uint64, err error) {
func processResultQueue(rws *exec22.TxTaskQueue, outputTxNumIn uint64, rs *state.StateV3, agg *state2.AggregatorV3, applyTx kv.Tx, triggerCount *atomic2.Uint64, rwsCond *sync.Cond, applyWorker *exec3.Worker) (resultSize int64, outputTxNum, conflicts, processedBlockNum uint64, err error) {
var i int
outputTxNum = outputTxNumIn
for rws.Len() > 0 && (*rws)[0].TxNum == outputTxNum {
Expand Down Expand Up @@ -797,7 +796,9 @@ func processResultQueue(rws *exec22.TxTaskQueue, outputTxNumIn uint64, rs *state
}
triggerCount.Add(rs.CommitTxNum(txTask.Sender, txTask.TxNum))
outputTxNum++
onSuccess()
if rwsCond != nil {
rwsCond.Signal()
}
if err := rs.ApplyHistory(txTask, agg); err != nil {
return resultSize, outputTxNum, conflicts, processedBlockNum, fmt.Errorf("StateV3.Apply: %w", err)
}
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/ledgerwatch/erigon
go 1.18

require (
github.com/ledgerwatch/erigon-lib v0.0.0-20230315063413-59238cf44dab
github.com/ledgerwatch/erigon-lib v0.0.0-20230315064748-e143f7756f24
github.com/ledgerwatch/erigon-snapshot v1.1.1-0.20230306083105-1391330d62a3
github.com/ledgerwatch/log/v3 v3.7.0
github.com/ledgerwatch/secp256k1 v1.0.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -517,8 +517,8 @@ github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v0.0.0-20170224010052-a616ab194758 h1:0D5M2HQSGD3PYPwICLl+/9oulQauOuETfgFvhBDffs0=
github.com/leanovate/gopter v0.2.9 h1:fQjYxZaynp97ozCzfOyOuAGOU4aU/z37zf/tOujFk7c=
github.com/leanovate/gopter v0.2.9/go.mod h1:U2L/78B+KVFIx2VmW6onHJQzXtFb+p5y3y2Sh+Jxxv8=
github.com/ledgerwatch/erigon-lib v0.0.0-20230315063413-59238cf44dab h1:P8FIZDukSMLAmjAMitj59ca6jKsxHIKUt1EH9m2izgI=
github.com/ledgerwatch/erigon-lib v0.0.0-20230315063413-59238cf44dab/go.mod h1:HbqSyXJh/Xd7a5WQScXOhAOTFyCuFu7OLJQfpCsYMnc=
github.com/ledgerwatch/erigon-lib v0.0.0-20230315064748-e143f7756f24 h1:vMymvQA0ZdaHzWgsfBwRmY3plXxfFhjckF4lLSskfFU=
github.com/ledgerwatch/erigon-lib v0.0.0-20230315064748-e143f7756f24/go.mod h1:HbqSyXJh/Xd7a5WQScXOhAOTFyCuFu7OLJQfpCsYMnc=
github.com/ledgerwatch/erigon-snapshot v1.1.1-0.20230306083105-1391330d62a3 h1:tfzawK1gIIgRjVZeANXOr0Ziu+kqCIBuKMe0TXfl5Aw=
github.com/ledgerwatch/erigon-snapshot v1.1.1-0.20230306083105-1391330d62a3/go.mod h1:3AuPxZc85jkehh/HA9h8gabv5MSi3kb/ddtzBsTVJFo=
github.com/ledgerwatch/log/v3 v3.7.0 h1:aFPEZdwZx4jzA3+/Pf8wNDN5tCI0cIolq/kfvgcM+og=
Expand Down
3 changes: 2 additions & 1 deletion p2p/discover/v5wire/crypto_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,12 @@ import (
"bytes"
"crypto/ecdsa"
"crypto/elliptic"
"github.com/minio/sha256-simd"
"reflect"
"strings"
"testing"

"github.com/minio/sha256-simd"

"github.com/ledgerwatch/erigon/common/hexutil"
"github.com/ledgerwatch/erigon/crypto"
"github.com/ledgerwatch/erigon/p2p/enode"
Expand Down
3 changes: 2 additions & 1 deletion p2p/discover/v5wire/encoding.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,10 @@ import (
"encoding/binary"
"errors"
"fmt"
"github.com/minio/sha256-simd"
"hash"

"github.com/minio/sha256-simd"

"github.com/ledgerwatch/erigon/common/mclock"
"github.com/ledgerwatch/erigon/p2p/enode"
"github.com/ledgerwatch/erigon/p2p/enr"
Expand Down
3 changes: 2 additions & 1 deletion p2p/server_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ import (
"context"
"crypto/ecdsa"
"errors"
"github.com/minio/sha256-simd"
"io"
"math/rand"
"net"
Expand All @@ -29,6 +28,8 @@ import (
"testing"
"time"

"github.com/minio/sha256-simd"

"github.com/ledgerwatch/erigon/crypto"
"github.com/ledgerwatch/erigon/p2p/enode"
"github.com/ledgerwatch/erigon/p2p/enr"
Expand Down
3 changes: 2 additions & 1 deletion turbo/trie/vtree/verkle_utils_test.go
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
package vtree

import (
"github.com/minio/sha256-simd"
"math/big"
"math/rand"
"testing"

"github.com/minio/sha256-simd"
)

func BenchmarkPedersenHash(b *testing.B) {
Expand Down