Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EIP-4844: Archive hot blobs and prune old blobs #4862

Merged
merged 3 commits into from
Dec 7, 2022
Merged

EIP-4844: Archive hot blobs and prune old blobs #4862

merged 3 commits into from
Dec 7, 2022

Conversation

dapplion
Copy link
Contributor

@dapplion dapplion commented Dec 6, 2022

Motivation

Description

  • On finalization, move cannonical blobs to archive, delete others
  • After retention period, prune old blobs. Simple logic that even on restarts will prune any range of old blocks.

@dapplion dapplion requested a review from a team as a code owner December 6, 2022 09:11
@dapplion dapplion force-pushed the dapplion/eip-4844-archive-blobs branch from ac9b177 to f4a33b0 Compare December 6, 2022 09:18
@github-actions
Copy link
Contributor

github-actions bot commented Dec 6, 2022 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: e0b4d2a Previous: 8bd86ae Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.4854 ms/op 2.6234 ms/op 0.95
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 83.129 us/op 90.403 us/op 0.92
BLS verify - blst-native 1.8585 ms/op 2.2314 ms/op 0.83
BLS verifyMultipleSignatures 3 - blst-native 3.8190 ms/op 4.5751 ms/op 0.83
BLS verifyMultipleSignatures 8 - blst-native 8.2181 ms/op 9.8666 ms/op 0.83
BLS verifyMultipleSignatures 32 - blst-native 29.783 ms/op 35.737 ms/op 0.83
BLS aggregatePubkeys 32 - blst-native 39.907 us/op 47.312 us/op 0.84
BLS aggregatePubkeys 128 - blst-native 153.63 us/op 184.13 us/op 0.83
getAttestationsForBlock 99.902 ms/op 117.06 ms/op 0.85
isKnown best case - 1 super set check 427.00 ns/op 517.00 ns/op 0.83
isKnown normal case - 2 super set checks 415.00 ns/op 508.00 ns/op 0.82
isKnown worse case - 16 super set checks 417.00 ns/op 488.00 ns/op 0.85
CheckpointStateCache - add get delete 9.5830 us/op 10.960 us/op 0.87
validate gossip signedAggregateAndProof - struct 4.2810 ms/op 5.1781 ms/op 0.83
validate gossip attestation - struct 2.0434 ms/op 2.4388 ms/op 0.84
pickEth1Vote - no votes 2.1332 ms/op 2.6257 ms/op 0.81
pickEth1Vote - max votes 23.745 ms/op 26.311 ms/op 0.90
pickEth1Vote - Eth1Data hashTreeRoot value x2048 12.070 ms/op 13.319 ms/op 0.91
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 23.028 ms/op 24.742 ms/op 0.93
pickEth1Vote - Eth1Data fastSerialize value x2048 1.6213 ms/op 2.0047 ms/op 0.81
pickEth1Vote - Eth1Data fastSerialize tree x2048 16.584 ms/op 17.263 ms/op 0.96
bytes32 toHexString 1.2670 us/op 1.3340 us/op 0.95
bytes32 Buffer.toString(hex) 747.00 ns/op 871.00 ns/op 0.86
bytes32 Buffer.toString(hex) from Uint8Array 999.00 ns/op 1.1450 us/op 0.87
bytes32 Buffer.toString(hex) + 0x 743.00 ns/op 854.00 ns/op 0.87
Object access 1 prop 0.41500 ns/op 0.44800 ns/op 0.93
Map access 1 prop 0.29500 ns/op 0.34900 ns/op 0.85
Object get x1000 18.797 ns/op 21.411 ns/op 0.88
Map get x1000 1.0100 ns/op 1.2310 ns/op 0.82
Object set x1000 134.77 ns/op 147.17 ns/op 0.92
Map set x1000 81.357 ns/op 91.541 ns/op 0.89
Return object 10000 times 0.36470 ns/op 0.43940 ns/op 0.83
Throw Error 10000 times 5.9491 us/op 7.1288 us/op 0.83
fastMsgIdFn sha256 / 200 bytes 4.3460 us/op 5.0470 us/op 0.86
fastMsgIdFn h32 xxhash / 200 bytes 580.00 ns/op 671.00 ns/op 0.86
fastMsgIdFn h64 xxhash / 200 bytes 728.00 ns/op 814.00 ns/op 0.89
fastMsgIdFn sha256 / 1000 bytes 13.376 us/op 15.677 us/op 0.85
fastMsgIdFn h32 xxhash / 1000 bytes 753.00 ns/op 845.00 ns/op 0.89
fastMsgIdFn h64 xxhash / 1000 bytes 905.00 ns/op 945.00 ns/op 0.96
fastMsgIdFn sha256 / 10000 bytes 114.11 us/op 134.60 us/op 0.85
fastMsgIdFn h32 xxhash / 10000 bytes 2.4480 us/op 2.8140 us/op 0.87
fastMsgIdFn h64 xxhash / 10000 bytes 1.7760 us/op 2.0850 us/op 0.85
enrSubnets - fastDeserialize 64 bits 3.2330 us/op 3.4270 us/op 0.94
enrSubnets - ssz BitVector 64 bits 824.00 ns/op 954.00 ns/op 0.86
enrSubnets - fastDeserialize 4 bits 431.00 ns/op 473.00 ns/op 0.91
enrSubnets - ssz BitVector 4 bits 823.00 ns/op 955.00 ns/op 0.86
prioritizePeers score -10:0 att 32-0.1 sync 2-0 113.73 us/op 120.79 us/op 0.94
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 124.75 us/op 146.41 us/op 0.85
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 235.96 us/op 264.56 us/op 0.89
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 499.40 us/op 557.93 us/op 0.90
prioritizePeers score 0:0 att 64-1 sync 4-1 469.43 us/op 547.82 us/op 0.86
RateTracker 1000000 limit, 1 obj count per request 206.27 ns/op 233.32 ns/op 0.88
RateTracker 1000000 limit, 2 obj count per request 156.10 ns/op 176.60 ns/op 0.88
RateTracker 1000000 limit, 4 obj count per request 132.07 ns/op 150.48 ns/op 0.88
RateTracker 1000000 limit, 8 obj count per request 119.14 ns/op 142.15 ns/op 0.84
RateTracker with prune 6.0800 us/op 6.2470 us/op 0.97
array of 16000 items push then shift 2.8437 us/op 3.3983 us/op 0.84
LinkedList of 16000 items push then shift 19.280 ns/op 20.224 ns/op 0.95
array of 16000 items push then pop 255.39 ns/op 287.37 ns/op 0.89
LinkedList of 16000 items push then pop 17.379 ns/op 19.423 ns/op 0.89
array of 24000 items push then shift 3.9854 us/op 4.7562 us/op 0.84
LinkedList of 24000 items push then shift 25.121 ns/op 23.689 ns/op 1.06
array of 24000 items push then pop 232.61 ns/op 250.24 ns/op 0.93
LinkedList of 24000 items push then pop 20.165 ns/op 20.770 ns/op 0.97
intersect bitArray bitLen 8 11.762 ns/op 13.738 ns/op 0.86
intersect array and set length 8 209.21 ns/op 212.04 ns/op 0.99
intersect bitArray bitLen 128 61.873 ns/op 86.422 ns/op 0.72
intersect array and set length 128 2.4993 us/op 2.7575 us/op 0.91
Buffer.concat 32 items 1.9780 ns/op 2.2500 ns/op 0.88
pass gossip attestations to forkchoice per slot 7.2227 ms/op 4.8611 ms/op 1.49
computeDeltas 6.0760 ms/op 7.7544 ms/op 0.78
computeProposerBoostScoreFromBalances 922.26 us/op 1.0944 ms/op 0.84
altair processAttestation - 250000 vs - 7PWei normalcase 4.7380 ms/op 5.1498 ms/op 0.92
altair processAttestation - 250000 vs - 7PWei worstcase 7.1499 ms/op 7.7890 ms/op 0.92
altair processAttestation - setStatus - 1/6 committees join 228.79 us/op 261.03 us/op 0.88
altair processAttestation - setStatus - 1/3 committees join 417.41 us/op 496.56 us/op 0.84
altair processAttestation - setStatus - 1/2 committees join 592.53 us/op 700.10 us/op 0.85
altair processAttestation - setStatus - 2/3 committees join 767.16 us/op 894.90 us/op 0.86
altair processAttestation - setStatus - 4/5 committees join 1.0496 ms/op 1.2531 ms/op 0.84
altair processAttestation - setStatus - 100% committees join 1.2580 ms/op 1.5770 ms/op 0.80
altair processBlock - 250000 vs - 7PWei normalcase 30.722 ms/op 33.209 ms/op 0.93
altair processBlock - 250000 vs - 7PWei normalcase hashState 39.513 ms/op 46.242 ms/op 0.85
altair processBlock - 250000 vs - 7PWei worstcase 97.067 ms/op 102.12 ms/op 0.95
altair processBlock - 250000 vs - 7PWei worstcase hashState 103.08 ms/op 122.70 ms/op 0.84
phase0 processBlock - 250000 vs - 7PWei normalcase 4.3134 ms/op 4.4956 ms/op 0.96
phase0 processBlock - 250000 vs - 7PWei worstcase 48.813 ms/op 55.640 ms/op 0.88
altair processEth1Data - 250000 vs - 7PWei normalcase 1.1345 ms/op 1.2280 ms/op 0.92
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 8.2400 us/op 13.495 us/op 0.61
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 26.929 us/op 34.775 us/op 0.77
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 11.909 us/op 16.737 us/op 0.71
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 8.6110 us/op 14.119 us/op 0.61
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 105.43 us/op 144.44 us/op 0.73
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.0843 ms/op 1.2754 ms/op 0.85
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 141069 16.022 ms/op 16.945 ms/op 0.95
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 250000 27.353 ms/op 31.065 ms/op 0.88
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 250000 92.230 ms/op 100.88 ms/op 0.91
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 250000 48.014 ms/op 57.816 ms/op 0.83
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 250000 137.85 ms/op 152.02 ms/op 0.91
Tree 40 250000 create 1.0830 s/op 901.00 ms/op 1.20
Tree 40 250000 get(125000) 298.54 ns/op 346.12 ns/op 0.86
Tree 40 250000 set(125000) 3.8166 us/op 3.2597 us/op 1.17
Tree 40 250000 toArray() 36.431 ms/op 38.082 ms/op 0.96
Tree 40 250000 iterate all - toArray() + loop 36.250 ms/op 40.246 ms/op 0.90
Tree 40 250000 iterate all - get(i) 118.27 ms/op 141.87 ms/op 0.83
MutableVector 250000 create 20.546 ms/op 19.511 ms/op 1.05
MutableVector 250000 get(125000) 13.347 ns/op 17.734 ns/op 0.75
MutableVector 250000 set(125000) 836.97 ns/op 828.52 ns/op 1.01
MutableVector 250000 toArray() 8.1423 ms/op 9.3215 ms/op 0.87
MutableVector 250000 iterate all - toArray() + loop 8.3336 ms/op 9.4999 ms/op 0.88
MutableVector 250000 iterate all - get(i) 3.2811 ms/op 4.2143 ms/op 0.78
Array 250000 create 7.5650 ms/op 9.3148 ms/op 0.81
Array 250000 clone - spread 4.3121 ms/op 3.2779 ms/op 1.32
Array 250000 get(125000) 1.9030 ns/op 1.3580 ns/op 1.40
Array 250000 set(125000) 1.8830 ns/op 1.3580 ns/op 1.39
Array 250000 iterate all - loop 170.35 us/op 201.46 us/op 0.85
effectiveBalanceIncrements clone Uint8Array 300000 112.75 us/op 87.844 us/op 1.28
effectiveBalanceIncrements clone MutableVector 300000 1.5860 us/op 878.00 ns/op 1.81
effectiveBalanceIncrements rw all Uint8Array 300000 252.58 us/op 305.18 us/op 0.83
effectiveBalanceIncrements rw all MutableVector 300000 288.35 ms/op 202.53 ms/op 1.42
phase0 afterProcessEpoch - 250000 vs - 7PWei 185.63 ms/op 218.61 ms/op 0.85
phase0 beforeProcessEpoch - 250000 vs - 7PWei 105.48 ms/op 82.111 ms/op 1.28
altair processEpoch - mainnet_e81889 523.16 ms/op 704.90 ms/op 0.74
mainnet_e81889 - altair beforeProcessEpoch 156.39 ms/op 167.84 ms/op 0.93
mainnet_e81889 - altair processJustificationAndFinalization 20.011 us/op 34.511 us/op 0.58
mainnet_e81889 - altair processInactivityUpdates 12.393 ms/op 12.748 ms/op 0.97
mainnet_e81889 - altair processRewardsAndPenalties 99.893 ms/op 112.53 ms/op 0.89
mainnet_e81889 - altair processRegistryUpdates 3.9340 us/op 7.7220 us/op 0.51
mainnet_e81889 - altair processSlashings 656.00 ns/op 2.3270 us/op 0.28
mainnet_e81889 - altair processEth1DataReset 840.00 ns/op 2.1390 us/op 0.39
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.2704 ms/op 2.9597 ms/op 0.77
mainnet_e81889 - altair processSlashingsReset 5.1940 us/op 11.410 us/op 0.46
mainnet_e81889 - altair processRandaoMixesReset 5.6370 us/op 13.285 us/op 0.42
mainnet_e81889 - altair processHistoricalRootsUpdate 899.00 ns/op 2.0230 us/op 0.44
mainnet_e81889 - altair processParticipationFlagUpdates 2.7650 us/op 6.9960 us/op 0.40
mainnet_e81889 - altair processSyncCommitteeUpdates 904.00 ns/op 1.8880 us/op 0.48
mainnet_e81889 - altair afterProcessEpoch 194.59 ms/op 239.58 ms/op 0.81
phase0 processEpoch - mainnet_e58758 560.26 ms/op 644.34 ms/op 0.87
mainnet_e58758 - phase0 beforeProcessEpoch 259.86 ms/op 279.53 ms/op 0.93
mainnet_e58758 - phase0 processJustificationAndFinalization 21.515 us/op 40.256 us/op 0.53
mainnet_e58758 - phase0 processRewardsAndPenalties 124.44 ms/op 159.41 ms/op 0.78
mainnet_e58758 - phase0 processRegistryUpdates 10.422 us/op 20.158 us/op 0.52
mainnet_e58758 - phase0 processSlashings 772.00 ns/op 2.2690 us/op 0.34
mainnet_e58758 - phase0 processEth1DataReset 739.00 ns/op 2.1160 us/op 0.35
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.7197 ms/op 2.7524 ms/op 0.99
mainnet_e58758 - phase0 processSlashingsReset 4.8280 us/op 10.135 us/op 0.48
mainnet_e58758 - phase0 processRandaoMixesReset 5.7770 us/op 12.999 us/op 0.44
mainnet_e58758 - phase0 processHistoricalRootsUpdate 916.00 ns/op 2.3270 us/op 0.39
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.7960 us/op 12.313 us/op 0.39
mainnet_e58758 - phase0 afterProcessEpoch 158.73 ms/op 193.23 ms/op 0.82
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.5751 ms/op 3.1016 ms/op 0.83
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.5912 ms/op 4.2604 ms/op 0.84
altair processInactivityUpdates - 250000 normalcase 46.217 ms/op 49.922 ms/op 0.93
altair processInactivityUpdates - 250000 worstcase 55.204 ms/op 58.116 ms/op 0.95
phase0 processRegistryUpdates - 250000 normalcase 8.7120 us/op 15.471 us/op 0.56
phase0 processRegistryUpdates - 250000 badcase_full_deposits 446.61 us/op 539.56 us/op 0.83
phase0 processRegistryUpdates - 250000 worstcase 0.5 235.66 ms/op 262.47 ms/op 0.90
altair processRewardsAndPenalties - 250000 normalcase 132.21 ms/op 158.45 ms/op 0.83
altair processRewardsAndPenalties - 250000 worstcase 87.872 ms/op 103.41 ms/op 0.85
phase0 getAttestationDeltas - 250000 normalcase 15.680 ms/op 16.632 ms/op 0.94
phase0 getAttestationDeltas - 250000 worstcase 15.796 ms/op 16.998 ms/op 0.93
phase0 processSlashings - 250000 worstcase 6.1331 ms/op 7.3086 ms/op 0.84
altair processSyncCommitteeUpdates - 250000 302.32 ms/op 343.40 ms/op 0.88
BeaconState.hashTreeRoot - No change 491.00 ns/op 590.00 ns/op 0.83
BeaconState.hashTreeRoot - 1 full validator 63.489 us/op 74.537 us/op 0.85
BeaconState.hashTreeRoot - 32 full validator 716.65 us/op 1.0716 ms/op 0.67
BeaconState.hashTreeRoot - 512 full validator 7.6841 ms/op 7.6261 ms/op 1.01
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 81.521 us/op 98.348 us/op 0.83
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.2650 ms/op 1.4423 ms/op 0.88
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.205 ms/op 19.505 ms/op 0.88
BeaconState.hashTreeRoot - 1 balances 63.537 us/op 75.313 us/op 0.84
BeaconState.hashTreeRoot - 32 balances 640.86 us/op 715.12 us/op 0.90
BeaconState.hashTreeRoot - 512 balances 6.2851 ms/op 7.2329 ms/op 0.87
BeaconState.hashTreeRoot - 250000 balances 91.358 ms/op 105.33 ms/op 0.87
aggregationBits - 2048 els - zipIndexesInBitList 42.904 us/op 41.744 us/op 1.03
regular array get 100000 times 67.464 us/op 80.933 us/op 0.83
wrappedArray get 100000 times 67.436 us/op 80.940 us/op 0.83
arrayWithProxy get 100000 times 27.383 ms/op 34.059 ms/op 0.80
ssz.Root.equals 498.00 ns/op 614.00 ns/op 0.81
byteArrayEquals 483.00 ns/op 597.00 ns/op 0.81
shuffle list - 16384 els 11.042 ms/op 13.304 ms/op 0.83
shuffle list - 250000 els 162.44 ms/op 195.30 ms/op 0.83
processSlot - 1 slots 12.925 us/op 17.889 us/op 0.72
processSlot - 32 slots 1.8327 ms/op 2.0698 ms/op 0.89
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 436.20 us/op 444.45 us/op 0.98
getCommitteeAssignments - req 1 vs - 250000 vc 5.3687 ms/op 6.3891 ms/op 0.84
getCommitteeAssignments - req 100 vs - 250000 vc 7.3520 ms/op 8.9699 ms/op 0.82
getCommitteeAssignments - req 1000 vs - 250000 vc 7.8875 ms/op 9.4843 ms/op 0.83
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 9.7600 ns/op 11.820 ns/op 0.83
state getBlockRootAtSlot - 250000 vs - 7PWei 1.3360 us/op 1.4787 us/op 0.90
computeProposers - vc 250000 18.321 ms/op 21.445 ms/op 0.85
computeEpochShuffling - vc 250000 166.63 ms/op 204.84 ms/op 0.81
getNextSyncCommittee - vc 250000 302.28 ms/op 344.86 ms/op 0.88

by benchmarkbot/action

@dapplion dapplion added the spec-deneb 🐡 Issues targeting the Deneb spec version label Dec 6, 2022
wemeetagain
wemeetagain reviewed Dec 6, 2022
View reviewed changes
packages/beacon-node/src/node/nodejs.ts
@@ -149,6 +149,9 @@ export class BeaconNode {

// start db if not already started
await db.start();
// Prune hot db repos
// TODO: Should this call be awaited?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably so unless we can be sure the pruning is done by the time we're syncing.

twoeths
twoeths previously approved these changes Dec 7, 2022
View reviewed changes
@twoeths twoeths merged commit 17c4d85 into unstable Dec 7, 2022
@twoeths twoeths deleted the dapplion/eip-4844-archive-blobs branch December 7, 2022 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain left review comments

@twoeths twoeths twoeths approved these changes

Assignees
No one assigned
Labels
spec-deneb 🐡 Issues targeting the Deneb spec version
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dapplion @wemeetagain @twoeths