Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Layered txpool by default and txpool options hoverhaul #5772

Merged
merged 10 commits into from
Sep 11, 2023

Conversation

fab-10
Copy link
Contributor

@fab-10 fab-10 commented Aug 18, 2023

PR description

This PR started with the goal to switch to the new layered by default, adding the possibility to select the old implementation if needed, introducing a new option --tx-pool that can take the values layered (default) and legacy, all the previously unstable options for the layered pool --tx-pool-layer-max-capacity, --tx-pool-max-prioritized, --tx-pool-max-future-by-sender are not stable.
Options specific to each implementation have been grouped, also in the help section, and validation has been added to fail in case options for a specific implementation are misused with the other implementation.

While implementing this, I found that the txpool options were a bit messy, stable and unstable options mixed, many options cluttering BesuCommand and the presence of boilerplate (and error prone) code, so I decided to use this PR to also review, clean and improve the code:

  • to move all the txpool options away from BesuCommand and split them correctly between stable and unstable,
  • made getCLIOptions to dynamically generate its output, via reflection, avoiding to manually add, and then maintain, any option, that is error prone. Most of the changes are related to this feature, that inspect by reflection the options in the class. Since these classes are only used at startup or during unit tests the performance of the reflection should not be an issue
  • made the custom converters to return directly the high level type instead of the raw type, that seems the right way to use them.
  • fixed some issues that happens when using the layered txpool by default in tests.

If the work done for txpool options, is found to be effective, then I can apply the same to other groups of options to clean and uniform them.

@github-actions
Copy link

github-actions bot commented Aug 18, 2023

  • I thought about documentation and added the doc-change-required label to this PR if updates are required.
  • I thought about the changelog and included a changelog update if required.
  • If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

@fab-10 fab-10 force-pushed the txpool-impl-option branch 3 times, most recently from 1d72074 to 01eac16 Compare August 21, 2023 13:23
return format(value.toUInt256());
}

public static String format(final Object value) {

Check notice

Code scanning / CodeQL

Confusing overloading of methods

Method OptionParser.format(..) could be confused with overloaded method [format](1), since dispatch depends on static types. Method OptionParser.format(..) could be confused with overloaded method [format](2), since dispatch depends on static types.
@fab-10 fab-10 force-pushed the txpool-impl-option branch 2 times, most recently from 3b887b6 to ce7fd67 Compare August 22, 2023 11:09
@fab-10 fab-10 changed the title Txpool options refactoring Option to select txpool implementation and txpool options hoverhaul Aug 22, 2023
@fab-10 fab-10 changed the title Option to select txpool implementation and txpool options hoverhaul Layered txpool by default and txpool options hoverhaul Aug 22, 2023
@fab-10 fab-10 self-assigned this Aug 22, 2023
@fab-10 fab-10 added the doc-change-required Indicates an issue or PR that requires doc to be updated label Aug 22, 2023
@fab-10 fab-10 force-pushed the txpool-impl-option branch from ce7fd67 to 1f5a858 Compare August 22, 2023 15:39
@fab-10 fab-10 marked this pull request as ready for review August 22, 2023 15:40
@fab-10 fab-10 force-pushed the txpool-impl-option branch from a0181ed to ead47c4 Compare August 23, 2023 09:47
Copy link
Contributor

@macfarla macfarla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# Conflicts:
#	CHANGELOG.md
@fab-10 fab-10 enabled auto-merge (squash) September 11, 2023 09:49
@fab-10 fab-10 merged commit 25c2065 into hyperledger:main Sep 11, 2023
delehef pushed a commit to Consensys/linea-besu that referenced this pull request Sep 12, 2023
@alexandratran alexandratran removed the doc-change-required Indicates an issue or PR that requires doc to be updated label Sep 12, 2023
siladu pushed a commit to siladu/besu that referenced this pull request Sep 20, 2023
---

Drop Kotti Network support (ETC) (hyperledger#5816)

Signed-off-by: Diego López León <[email protected]>

fix ForkId if there are no Forks and the starting timestamp is not 0 (hyperledger#5819)

Signed-off-by: Stefan <[email protected]>

enforce that BlobTransactions have at least one blob (hyperledger#5826)

* enforce that BlobTransactions have at least one blob

Signed-off-by: Stefan <[email protected]>
Signed-off-by: Stefan Pingel <[email protected]>

Do not create ignorable segments on `revert storage-variables` (hyperledger#5830)

* fix the bug that creates the ignorable chain pruner segment, add rocks exception parsing to RocksDBColumnarKeyValueStorage subclasses
* parse rocksdb error for unprintable column family id's

Signed-off-by: garyschulte <[email protected]>

add versioned hashes and number of blobs to toString() (hyperledger#5831)

Signed-off-by: Stefan <[email protected]>

add parent beacon block root to payload id calculation (hyperledger#5843)

Signed-off-by: Stefan <[email protected]>

bump version to 23.7.3-SNAPSHOT (hyperledger#5854)

Signed-off-by: Daniel Lehrner <[email protected]>

set the beacon root address to the correct value (hyperledger#5853)

Signed-off-by: Stefan <[email protected]>

docs(readme): fix broken link to installation of binaries page (hyperledger#5859)

Fixes hyperledger#5858

Signed-off-by: Peter Somogyvari <[email protected]>

Update RocksDB version from 8.0.0 to 8.3.2 (hyperledger#5832)

Signed-off-by: Ameziane H <[email protected]>

Co-authored-by: Sally MacFarlane <[email protected]>

use non-deprecated authenticate methods (hyperledger#5852)

Signed-off-by: Sally MacFarlane <[email protected]>

move to Hyperledger shared runners for current github actions (hyperledger#5860)

Signed-off-by: garyschulte <[email protected]>

Add range tracing with worldstate (hyperledger#5844)

Implement a method to trace a range of blocks and have access to the worldstate before and after the tracing

Signed-off-by: Karim TAAM <[email protected]>

Layered txpool by default and txpool options hoverhaul (hyperledger#5772)

Signed-off-by: Fabio Di Fabio <[email protected]>

Fix issue 5824 - Duplicate key errors in EthScheduler-Transactions (hyperledger#5857)

Fix issue 5824 - Duplicate key errors in EthScheduler-Transactions

Signed-off-by: Ameziane H <[email protected]>

updated gradle verification metadata (hyperledger#5870)

* removed old artefacts [skip ci]

* works with compileTestJava

* restored metadata needed for codeQL and trusted-artifacts block for javadoc/sources

Signed-off-by: Sally MacFarlane <[email protected]>

---------

Signed-off-by: Sally MacFarlane <[email protected]>

[4844] Add encodingContext to TransactionEncoder and TransactionDecoder (hyperledger#5820)

* Add decode type to TransactionDecoder

* Refactoring TransactionDecoder

* Invert methods order

* Use Transaction encoder instead of writeTo

* Move enter and leave list to inner method as pr suggestion

* Size calculation should use opaque bytes instead of rlp

---------

Signed-off-by: Gabriel-Trintinalia <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>

Signed-off-by: Stefan <[email protected]>

payload attributes: fix wrong warning and fail if beacon root is available before cancun (hyperledger#5872)

Signed-off-by: Stefan <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>

Merge MutableAccount and EVMAccount (hyperledger#5863)

Merge MutableAccount and EVMAccount functionalities by removing
EVMAccount, all calls to getMutable, and the WrappedEVMAccount that was
wrapping non-EVMAccounts in a mutable fashion.  Instead, use a
MutableAccount in all cases an EVMAccount would have been used.  This
also tends to reduce a level of layering in many places.

Signed-off-by: Danno Ferrin <[email protected]>

Add world context to transaction tracing API (hyperledger#5836)

* Add world context to transaction tracing API

Signed-off-by: Franklin Delehelle <[email protected]>

* Update changelog with PR ID

Signed-off-by: Franklin Delehelle <[email protected]>

* Add the Transaction to traceEndTransaction

Signed-off-by: Franklin Delehelle <[email protected]>

* Rebase on main

Signed-off-by: Franklin Delehelle <[email protected]>

* Add receipt-linked information to the transaction tracer

Signed-off-by: Franklin Delehelle <[email protected]>

* added test

Signed-off-by: Daniel Lehrner <[email protected]>

---------

Signed-off-by: Franklin Delehelle <[email protected]>
Signed-off-by: Daniel Lehrner <[email protected]>
Co-authored-by: Daniel Lehrner <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>

Bonsai based reference test worldstate (hyperledger#5686)

* create a bonsai based reference test worldstate
  -> getOrCreate in BonsaiWorldStateUpdateAccumulator - do not throw if we discover an empty account in a non-null BonsaiValue<Account>
  -> add curentStateRoot to t8n
  -> storageEntriesFrom and streamAccounts implemented in BonsaiWorldStateKeyValueStorage
  -> add endKey version of streamFromKey
* bonsai fix for self-destruct and create2 at the same address and same block

Signed-off-by: garyschulte <[email protected]>
Signed-off-by: Karim TAAM <[email protected]>
Co-authored-by: Karim TAAM <[email protected]>

Don't start BFT mining coordinators until initial sync has completed (hyperledger#5861)

* Don't start BFT mining coordinators until initial sync has completed

Signed-off-by: Matthew Whitehead <[email protected]>

* Fix unit tests

Signed-off-by: Matthew Whitehead <[email protected]>

* Fix 'enable' logic

Signed-off-by: Matthew Whitehead <[email protected]>

---------

Signed-off-by: Matthew Whitehead <[email protected]>

display only peers ready for requets on ethstats (hyperledger#5880)

* display only ready for requets peers in ethstats

Signed-off-by: Karim TAAM <[email protected]>

* cast to int

Signed-off-by: Sally MacFarlane <[email protected]>

---------

Signed-off-by: Karim TAAM <[email protected]>
Signed-off-by: Sally MacFarlane <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>
Co-authored-by: Stefan Pingel <[email protected]>

[MINOR] test RLP used for encode/decode blob tx should contain to field (hyperledger#5883)

* validate to field on encode/decode for blob tx

Signed-off-by: Sally MacFarlane <[email protected]>

* revert decode/encode checks - tis done later in tx validation

Signed-off-by: Sally MacFarlane <[email protected]>

---------

Signed-off-by: Sally MacFarlane <[email protected]>

Fix: correctly convert percentage options in TOML configuration file (hyperledger#5886)

Signed-off-by: Fabio Di Fabio <[email protected]>

EIP7516 - Add BlobBaseFee opcode to Cancun EVM (hyperledger#5884)

Signed-off-by: Gabriel-Trintinalia <[email protected]>

Fix snapsync heal (hyperledger#5838)

Signed-off-by: Karim TAAM <[email protected]>

Upgrade besu-native (hyperledger#5893)

Upgrade besu-native to 0.8.2

Signed-off-by: Danno Ferrin <[email protected]>

Tune G1GC to reduce Besu memory footprint (hyperledger#5879)

Signed-off-by: Fabio Di Fabio <[email protected]>

Add updated storage to evmtool json trace (hyperledger#5892)

Add the EIP-3155 "storage" option to the standard tracer, with the
caveat only updated storage is logged.

Signed-off-by: Danno Ferrin <[email protected]>

Update holesky with fixed extraData, genesis time, shanghaiTime (hyperledger#5890)

Signed-off-by: Simon Dudley <[email protected]>

[CHANGELOG] removed duplicated line (hyperledger#5904)

* removed duplicated line [skip ci]

Signed-off-by: Sally MacFarlane <[email protected]>

* fixed spelling on Holesky

Signed-off-by: Sally MacFarlane <[email protected]>

---------

Signed-off-by: Sally MacFarlane <[email protected]>
garyschulte pushed a commit to garyschulte/besu that referenced this pull request Sep 20, 2023
garyschulte pushed a commit to garyschulte/besu that referenced this pull request Sep 20, 2023
siladu pushed a commit to siladu/besu that referenced this pull request Sep 20, 2023
siladu pushed a commit to siladu/besu that referenced this pull request Sep 20, 2023
siladu added a commit that referenced this pull request Sep 20, 2023
burn-in candidate for 23.7.3 from main sha 6dc10a9..eef40bd

https://github.com/hyperledger/besu/compare/6dc10a9..eef40bd

---

* Drop Kotti Network support (ETC) (#5816)

Signed-off-by: Diego López León <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* fix ForkId if there are no Forks and the starting timestamp is not 0 (#5819)

Signed-off-by: Stefan <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* enforce that BlobTransactions have at least one blob (#5826)

* enforce that BlobTransactions have at least one blob

Signed-off-by: Stefan <[email protected]>
Signed-off-by: Stefan Pingel <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Do not create ignorable segments on `revert storage-variables` (#5830)

* fix the bug that creates the ignorable chain pruner segment, add rocks exception parsing to RocksDBColumnarKeyValueStorage subclasses
* parse rocksdb error for unprintable column family id's

Signed-off-by: garyschulte <[email protected]>

* add versioned hashes and number of blobs to toString() (#5831)

Signed-off-by: Stefan <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* add parent beacon block root to payload id calculation (#5843)

Signed-off-by: Stefan <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* bump version to 23.7.3-SNAPSHOT (#5854)

Signed-off-by: Daniel Lehrner <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* set the beacon root address to the correct value (#5853)

Signed-off-by: Stefan <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* docs(readme): fix broken link to installation of binaries page (#5859)

Fixes #5858

Signed-off-by: Peter Somogyvari <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Update RocksDB version from 8.0.0 to 8.3.2 (#5832)

Signed-off-by: Ameziane H <[email protected]>

Co-authored-by: Sally MacFarlane <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* use non-deprecated authenticate methods (#5852)

Signed-off-by: Sally MacFarlane <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* move to Hyperledger shared runners for current github actions (#5860)

Signed-off-by: garyschulte <[email protected]>

* Add range tracing with worldstate (#5844)

Implement a method to trace a range of blocks and have access to the worldstate before and after the tracing

Signed-off-by: Karim TAAM <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Layered txpool by default and txpool options hoverhaul (#5772)

Signed-off-by: Fabio Di Fabio <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Fix issue 5824 - Duplicate key errors in EthScheduler-Transactions (#5857)

Fix issue 5824 - Duplicate key errors in EthScheduler-Transactions

Signed-off-by: Ameziane H <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* updated gradle verification metadata (#5870)

* removed old artefacts [skip ci]

* works with compileTestJava

* restored metadata needed for codeQL and trusted-artifacts block for javadoc/sources

Signed-off-by: Sally MacFarlane <[email protected]>

---------

Signed-off-by: Sally MacFarlane <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* [4844] Add encodingContext to TransactionEncoder and TransactionDecoder (#5820)

* Add decode type to TransactionDecoder

* Refactoring TransactionDecoder

* Invert methods order

* Use Transaction encoder instead of writeTo

* Move enter and leave list to inner method as pr suggestion

* Size calculation should use opaque bytes instead of rlp

---------

Signed-off-by: Gabriel-Trintinalia <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* #5868: fix beacon root address and modulus for devnet 9 (#5871)

Signed-off-by: Stefan <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* payload attributes: fix wrong warning and fail if beacon root is available before cancun (#5872)

Signed-off-by: Stefan <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Merge MutableAccount and EVMAccount (#5863)

Merge MutableAccount and EVMAccount functionalities by removing
EVMAccount, all calls to getMutable, and the WrappedEVMAccount that was
wrapping non-EVMAccounts in a mutable fashion.  Instead, use a
MutableAccount in all cases an EVMAccount would have been used.  This
also tends to reduce a level of layering in many places.

Signed-off-by: Danno Ferrin <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Add world context to transaction tracing API (#5836)

* Add world context to transaction tracing API

Signed-off-by: Franklin Delehelle <[email protected]>

* Update changelog with PR ID

Signed-off-by: Franklin Delehelle <[email protected]>

* Add the Transaction to traceEndTransaction

Signed-off-by: Franklin Delehelle <[email protected]>

* Rebase on main

Signed-off-by: Franklin Delehelle <[email protected]>

* Add receipt-linked information to the transaction tracer

Signed-off-by: Franklin Delehelle <[email protected]>

* added test

Signed-off-by: Daniel Lehrner <[email protected]>

---------

Signed-off-by: Franklin Delehelle <[email protected]>
Signed-off-by: Daniel Lehrner <[email protected]>
Co-authored-by: Daniel Lehrner <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Bonsai based reference test worldstate (#5686)

* create a bonsai based reference test worldstate
  -> getOrCreate in BonsaiWorldStateUpdateAccumulator - do not throw if we discover an empty account in a non-null BonsaiValue<Account>
  -> add curentStateRoot to t8n
  -> storageEntriesFrom and streamAccounts implemented in BonsaiWorldStateKeyValueStorage
  -> add endKey version of streamFromKey
* bonsai fix for self-destruct and create2 at the same address and same block

Signed-off-by: garyschulte <[email protected]>
Signed-off-by: Karim TAAM <[email protected]>
Co-authored-by: Karim TAAM <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Don't start BFT mining coordinators until initial sync has completed (#5861)

* Don't start BFT mining coordinators until initial sync has completed

Signed-off-by: Matthew Whitehead <[email protected]>

* Fix unit tests

Signed-off-by: Matthew Whitehead <[email protected]>

* Fix 'enable' logic

Signed-off-by: Matthew Whitehead <[email protected]>

---------

Signed-off-by: Matthew Whitehead <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* display only peers ready for requets on ethstats (#5880)

* display only ready for requets peers in ethstats

Signed-off-by: Karim TAAM <[email protected]>

* cast to int

Signed-off-by: Sally MacFarlane <[email protected]>

---------

Signed-off-by: Karim TAAM <[email protected]>
Signed-off-by: Sally MacFarlane <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>
Co-authored-by: Stefan Pingel <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* [MINOR] test RLP used for encode/decode blob tx should contain to field (#5883)

* validate to field on encode/decode for blob tx

Signed-off-by: Sally MacFarlane <[email protected]>

* revert decode/encode checks - tis done later in tx validation

Signed-off-by: Sally MacFarlane <[email protected]>

---------

Signed-off-by: Sally MacFarlane <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Fix: correctly convert percentage options in TOML configuration file (#5886)

Signed-off-by: Fabio Di Fabio <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* EIP7516 - Add BlobBaseFee opcode to Cancun EVM (#5884)

Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Fix snapsync heal (#5838)

Signed-off-by: Karim TAAM <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Upgrade besu-native (#5893)

Upgrade besu-native to 0.8.2

Signed-off-by: Danno Ferrin <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Tune G1GC to reduce Besu memory footprint (#5879)

Signed-off-by: Fabio Di Fabio <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Add updated storage to evmtool json trace (#5892)

Add the EIP-3155 "storage" option to the standard tracer, with the
caveat only updated storage is logged.

Signed-off-by: Danno Ferrin <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* Update holesky with fixed extraData, genesis time, shanghaiTime (#5890)

Signed-off-by: Simon Dudley <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* [CHANGELOG] removed duplicated line (#5904)

* removed duplicated line [skip ci]

Signed-off-by: Sally MacFarlane <[email protected]>

* fixed spelling on Holesky

Signed-off-by: Sally MacFarlane <[email protected]>

---------

Signed-off-by: Sally MacFarlane <[email protected]>
Signed-off-by: garyschulte <[email protected]>

* add 23.7.2 release SHAs and bump to 23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-23.7.3-RC version

Signed-off-by: garyschulte <[email protected]>

---------

Signed-off-by: Diego López León <[email protected]>
Signed-off-by: garyschulte <[email protected]>
Signed-off-by: Stefan <[email protected]>
Signed-off-by: Stefan Pingel <[email protected]>
Signed-off-by: Daniel Lehrner <[email protected]>
Signed-off-by: Peter Somogyvari <[email protected]>
Signed-off-by: Sally MacFarlane <[email protected]>
Signed-off-by: Karim TAAM <[email protected]>
Signed-off-by: Fabio Di Fabio <[email protected]>
Signed-off-by: Ameziane H <[email protected]>
Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: Danno Ferrin <[email protected]>
Signed-off-by: Franklin Delehelle <[email protected]>
Signed-off-by: Matthew Whitehead <[email protected]>
Signed-off-by: Simon Dudley <[email protected]>
Co-authored-by: Diego López León <[email protected]>
Co-authored-by: Stefan Pingel <[email protected]>
Co-authored-by: Daniel Lehrner <[email protected]>
Co-authored-by: Peter Somogyvari <[email protected]>
Co-authored-by: ahamlat <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>
Co-authored-by: matkt <[email protected]>
Co-authored-by: Fabio Di Fabio <[email protected]>
Co-authored-by: Gabriel-Trintinalia <[email protected]>
Co-authored-by: Danno Ferrin <[email protected]>
Co-authored-by: delehef <[email protected]>
Co-authored-by: Matt Whitehead <[email protected]>
Co-authored-by: Simon Dudley <[email protected]>
@fab-10 fab-10 deleted the txpool-impl-option branch October 17, 2023 08:05
@fab-10 fab-10 mentioned this pull request Oct 17, 2023
eum602 pushed a commit to lacchain/besu that referenced this pull request Nov 3, 2023
NickSneo pushed a commit to NickSneo/besu that referenced this pull request Nov 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants