From 6d9c8b134bf5145ce7171865f00347ac93c7c483 Mon Sep 17 00:00:00 2001 From: Jason Frame Date: Fri, 12 Jan 2024 18:40:47 +1000 Subject: [PATCH] Merge Besu main (#34) * mark deleted slot during clear storage step (#6305) Signed-off-by: Karim Taam Co-authored-by: garyschulte * made directory structure of tests match source; fixed one typo (#6337) Signed-off-by: Sally MacFarlane * migrate controller tests to junit 5 (#6338) Signed-off-by: Sally MacFarlane * add new forkids for testnets, update forkid test to Junit5, no longer need named network specific trusted setups (#6322) Signed-off-by: jflo * Fix trielog shipping issue during self destruct (#6340) * fix trielog shipping issue Signed-off-by: Karim Taam * bump gradle properties version and adjust changelog to match release (#6347) Signed-off-by: garyschulte * finalized cancun spec (#6351) * finalized cancun spec Signed-off-by: jflo * finalized cancun spec Signed-off-by: jflo --------- Signed-off-by: jflo * Optimize RocksDB WAL file (#6328) Signed-off-by: Fabio Di Fabio * Make RPC reason settable, pass execution failure reason in RPC error message (#6343) * Make RPC reason settable, pass execution failure reason in RPC error message Signed-off-by: Matthew Whitehead * Update unit tests Signed-off-by: Matthew Whitehead * Update tests Signed-off-by: Matthew Whitehead * Update change log Signed-off-by: Matthew Whitehead * Update integration tests Signed-off-by: Matthew Whitehead --------- Signed-off-by: Matthew Whitehead * TestWatcher junit5 (#6339) * TestWatcher junit5 * add test class and method name to context * moved the testwatcher junit5 function to a new junit5 superclass * one qbft test to junit5 superclass Signed-off-by: Sally MacFarlane --------- Signed-off-by: Sally MacFarlane Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> * Migrate BFT tests to junit 5 (#6350) * bft tests to junit 5 * base class for pki extend AcceptanceTestBaseJunit5 * try/catch in case of empty optionals * fixed parameterization method Signed-off-by: Sally MacFarlane --------- Signed-off-by: Sally MacFarlane * fixing on selfdestruct (#6359) Signed-off-by: Karim Taam * migrate clique tests fully to junit5 (#6362) * migrate clique tests fully to junit5 Signed-off-by: Sally MacFarlane --------- Signed-off-by: Sally MacFarlane * fixed link to logging docs (#6366) Signed-off-by: Sally MacFarlane * Move logging to RunnerBuilder (#6367) Signed-off-by: Gabriel-Trintinalia * Use synchronized call to access the chain head block in `eth_estimateGas` (#6345) * Use synchronized call to access the chain head block in estimateGas() Signed-off-by: Matthew Whitehead * Add error log entries when throwing internal error from estimateGas() Signed-off-by: Matthew Whitehead * Update unit tests Signed-off-by: Matthew Whitehead * Update changelog Signed-off-by: Matthew Whitehead --------- Signed-off-by: Matthew Whitehead * Add --X-trie-log subcommand (#6303) * Add x-trie-log subcommand for one-off trie log backlog prune Signed-off-by: Simon Dudley Signed-off-by: Gabriel Fukushima --------- Signed-off-by: Simon Dudley Signed-off-by: Gabriel Fukushima Co-authored-by: Simon Dudley * fix typos (#6368) Signed-off-by: vuittont60 <81072379+vuittont60@users.noreply.github.com> * Added alias --sync-min-peers for --fast-sync-min-peers (#6372) * sync-min-peers as an alias * added unit tests Signed-off-by: Sally MacFarlane --------- Signed-off-by: Sally MacFarlane * Fix: Fallback to getName when canonicalName is null in BlockHeaderValidator DEBUG log (#6332) * fallback to simple name when canonical name is null * use getName instead of getSimpleName to include the package name Signed-off-by: Manoj P R --------- Signed-off-by: Manoj P R Co-authored-by: Sally MacFarlane * fix: use UID 1000 for besu user (#6358) (#6360) The openjdk-latest Docker image is using UID 1001 for besu, because its base image ubuntu:23.10 now contains a default "ubuntu" user with UID 1000. (This UID change causes the besu user with UID 1001 to not have access to files created for past versions with UID 1000.) We now remove the default ubuntu user and explicitly use UID 1000 when creating the besu user. Signed-off-by: Hal Blackburn * Ignore generated files when running the spdx license check (#6379) Signed-off-by: Meredith Baxter * full sync - don't fail startup if sync-min-peers specified (#6373) Signed-off-by: Sally MacFarlane * Copy also computed fields, when doing a Transaction detached copy (#6329) Signed-off-by: Fabio Di Fabio * Disable txpool when not in sync (#6302) Signed-off-by: Fabio Di Fabio * Bump to nex release snapshot 24.1.1 (#6383) * release next snapshot 24.1.1 Signed-off-by: garyschulte * move recent changelog items to 24.1.1-SNAPSHOT Signed-off-by: garyschulte --------- Signed-off-by: garyschulte * Correct Tangerine Whistle definition in Fluent EVM APIs. (#6382) The fluent API incorrectly added the code size limit in Tangerine Whistle instead of first adding it in Spurious Dragon. Signed-off-by: Danno Ferrin * [MINOR] Fix pki tests condition check on mac (#6387) Signed-off-by: Gabriel-Trintinalia * Upgrade dependencies (#6377) * Bump com.github.oshi:oshi-core to 6.4.10 Signed-off-by: Fabio Di Fabio * Bump com.github.tomakehurst to org.wiremock 3.3.1 Signed-off-by: Fabio Di Fabio * Bump com.google.auto.service:auto-service to 1.1.1 Signed-off-by: Fabio Di Fabio * Bump com.google.dagger group to 2.50 Signed-off-by: Fabio Di Fabio * Bump com.graphql-java:graphql-java to 21.3 Signed-off-by: Fabio Di Fabio * Bump com.splunk.logging:splunk-library-javalogging to 1.11.8 Signed-off-by: Fabio Di Fabio * Bump com.squareup.okhttp3:okhttp to 4.12.0 Signed-off-by: Fabio Di Fabio * Bump commons-io:commons-io to 2.15.1 Signed-off-by: Fabio Di Fabio * Bump dnsjava:dnsjava to 3.5.3 Signed-off-by: Fabio Di Fabio * Bump info.picocli group to 4.7.5 Signed-off-by: Fabio Di Fabio * Bump io.grpc group to 1.60.1 Signed-off-by: Fabio Di Fabio * Bump io.kubernetes:client-java to 18.0.1 Signed-off-by: Fabio Di Fabio * Bump io.netty group to 4.1.104.Final Signed-off-by: Fabio Di Fabio * Bump net.java.dev.jna:jna to 5.14.0 Signed-off-by: Fabio Di Fabio * Bump org.apache.commons:commons-compress to 1.25.0 Signed-off-by: Fabio Di Fabio * Bump org.apache.commons:commons-lang3 to 3.14.0 Signed-off-by: Fabio Di Fabio * Bump org.apache.commons:commons-text to 1.11.0 Signed-off-by: Fabio Di Fabio * Bump org.apache.logging.log4j group to 2.22.1 Signed-off-by: Fabio Di Fabio * Redorder io.tmio group Signed-off-by: Fabio Di Fabio * Bump org.assertj:assertj-core to 3.25.1 Signed-off-by: Fabio Di Fabio * Bump org.bouncycastle group to 1.77 Signed-off-by: Fabio Di Fabio * Bump org.fusesource.jansi:jansi to 2.4.1 Signed-off-by: Fabio Di Fabio * Bump org.immutables group 2.10.0 Signed-off-by: Fabio Di Fabio * Bump org.java-websocket:Java-WebSocket to 1.5.5 Signed-off-by: Fabio Di Fabio * Bump org.jetbrains.kotlin:kotlin-stdlib to 1.9.22 Signed-off-by: Fabio Di Fabio * Bump org.junit.jupiter group to 5.10.1 Signed-off-by: Fabio Di Fabio * Bump org.jupnp group to 2.7.1 Signed-off-by: Fabio Di Fabio * Bump org.rocksdb:rocksdbjni to 8.9.1 Signed-off-by: Fabio Di Fabio * Bump org.slf4j group to 2.0.10 Signed-off-by: Fabio Di Fabio * Bump org.springframework.security:spring-security-crypto to 6.2.1 Signed-off-by: Fabio Di Fabio * Bump org.testcontainers:testcontainers to 1.19.3 Signed-off-by: Fabio Di Fabio * Bump org.web3j group to 4.10.3 Signed-off-by: Fabio Di Fabio * Bump org.xerial.snappy:snappy-java to 1.1.10.5 Signed-off-by: Fabio Di Fabio * Regenerate gradle verification metadata Signed-off-by: Fabio Di Fabio * Update commons-codec:commons-codec to 1.16.0 Signed-off-by: Fabio Di Fabio * Update org.junit.vintage:junit-vintage-engine to 5.10.1 Signed-off-by: Fabio Di Fabio * Update CHANGELOG Signed-off-by: Fabio Di Fabio --------- Signed-off-by: Fabio Di Fabio * add a fallback for docker detection on Mac (#6356) Signed-off-by: garyschulte * Fix test flackyness of acceptanceTestsPermissioning (#6384) Signed-off-by: Fabio Di Fabio Signed-off-by: Sally MacFarlane Co-authored-by: Sally MacFarlane * Upgrade `com.fasterxml.jackson` dependencies (#6378) Signed-off-by: Fabio Di Fabio * Use mining beneficiary from protocol spec in TraceServiceImpl (#6390) * use mining beneficiary from protocol spec Signed-off-by: Daniel Lehrner --------- Signed-off-by: Daniel Lehrner Co-authored-by: Sally MacFarlane * Update verification metadata and allowed licenses for Linea-Besu --------- Signed-off-by: Karim Taam Signed-off-by: Sally MacFarlane Signed-off-by: jflo Signed-off-by: garyschulte Signed-off-by: Fabio Di Fabio Signed-off-by: Matthew Whitehead Signed-off-by: Gabriel-Trintinalia Signed-off-by: Simon Dudley Signed-off-by: Gabriel Fukushima Signed-off-by: vuittont60 <81072379+vuittont60@users.noreply.github.com> Signed-off-by: Manoj P R Signed-off-by: Hal Blackburn Signed-off-by: Meredith Baxter Signed-off-by: Danno Ferrin Signed-off-by: Daniel Lehrner Co-authored-by: Karim TAAM Co-authored-by: garyschulte Co-authored-by: Sally MacFarlane Co-authored-by: Justin Florentine Co-authored-by: Fabio Di Fabio Co-authored-by: Matt Whitehead Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: Gabriel-Trintinalia Co-authored-by: Gabriel Fukushima Co-authored-by: Simon Dudley Co-authored-by: vuittont60 <81072379+vuittont60@users.noreply.github.com> Co-authored-by: Manoj P R Co-authored-by: Hal Blackburn Co-authored-by: mbaxter Co-authored-by: Danno Ferrin Co-authored-by: daniellehrner --- CHANGELOG.md | 7 + acceptance-tests/dsl/build.gradle | 2 +- .../dsl/condition/eth/EthConditions.java | 4 + .../condition/eth/SyncingStatusCondition.java | 40 + .../acceptance/dsl/node/cluster/Cluster.java | 7 +- .../eth/EthSyncingTransaction.java | 40 + .../dsl/transaction/eth/EthTransactions.java | 4 + acceptance-tests/tests/build.gradle | 2 +- ...rtContractPermissioningAcceptanceTest.java | 5 + ...ContractPermissioningV2AcceptanceTest.java | 5 + .../simple_permissioning_genesis.json | 2 +- .../simple_permissioning_v2_genesis.json | 2 +- .../org/hyperledger/besu/cli/BesuCommand.java | 18 +- .../besu/cli/DefaultCommandValues.java | 4 +- .../storage/RocksDbUsageHelper.java | 10 +- .../storage/StorageSubCommand.java | 6 +- .../subcommands/storage/TrieLogHelper.java | 361 ++ .../storage/TrieLogSubCommand.java | 147 + .../besu/controller/BesuController.java | 18 +- .../controller/BesuControllerBuilder.java | 3 +- .../besu/services/TraceServiceImpl.java | 2 +- .../hyperledger/besu/cli/BesuCommandTest.java | 27 +- .../storage/TrieLogHelperTest.java | 265 ++ .../besu/services/BesuEventsImplTest.java | 5 +- build.gradle | 1 + docker/openjdk-latest/Dockerfile | 5 +- docs/trace_rpc_apis.md | 2 +- .../EthGetFilterChangesIntegrationTest.java | 3 +- .../EthGetFilterChangesIntegrationTest.java | 3 +- .../besu/ethereum/core/Transaction.java | 45 +- .../mainnet/BlockHeaderValidator.java | 5 +- .../BonsaiWorldStateKeyValueStorage.java | 2 +- .../trie/bonsai/trielog/TrieLogPruner.java | 17 +- .../bonsai/worldview/BonsaiWorldState.java | 1 + .../besu/ethereum/eth/manager/EthPeers.java | 2 +- .../ethereum/eth/manager/EthScheduler.java | 49 + .../eth/transactions/TransactionPool.java | 69 +- .../transactions/TransactionPoolFactory.java | 52 +- .../transactions/TransactionPoolMetrics.java | 37 +- .../eth/manager/EthSchedulerTest.java | 47 + .../ethtaskutils/AbstractMessageTaskTest.java | 6 +- .../ethtaskutils/PeerMessageTaskTest.java | 6 +- .../AbstractTransactionPoolTest.java | 7 +- .../besu/evm/fluent/EVMExecutor.java | 2 +- gradle/allowed-licenses.json | 8 + gradle/license-normalizer-bundle.json | 2 +- gradle/verification-metadata.xml | 2910 ++++++----------- gradle/versions.gradle | 111 +- .../metrics/ReplaceableDoubleSupplier.java | 55 + .../ReplaceableDoubleSupplierTest.java | 36 + .../besu/nat/docker/DockerDetector.java | 2 + .../HardwareKeyStoreFileWrapperTest.java | 25 +- .../besu/testutil/JsonTestParameters.java | 9 +- 53 files changed, 2404 insertions(+), 2101 deletions(-) create mode 100644 acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/eth/SyncingStatusCondition.java create mode 100644 acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/eth/EthSyncingTransaction.java create mode 100644 besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java create mode 100644 besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogSubCommand.java create mode 100644 besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java create mode 100644 metrics/core/src/main/java/org/hyperledger/besu/metrics/ReplaceableDoubleSupplier.java create mode 100644 metrics/core/src/test/java/org/hyperledger/besu/metrics/ReplaceableDoubleSupplierTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index ea4b5f915e9..33a9d9ca350 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Breaking Changes - New `EXECUTION_HALTED` error returned if there is an error executing or simulating a transaction, with the reason for execution being halted. Replaces the generic `INTERNAL_ERROR` return code in certain cases which some applications may be checking for [#6343](https://github.com/hyperledger/besu/pull/6343) +- The Besu Docker images with `openjdk-latest` tags since 23.10.3 were incorrectly using UID 1001 instead of 1000 for the container's `besu` user. The user now uses 1000 again. Containers created from or migrated to images using UID 1001 will need to chown their persistent database files to UID 1000 [#6360](https://github.com/hyperledger/besu/pull/6360) ### Deprecations - Forest pruning (`pruning-enabled` options) is deprecated and will be removed soon. To save disk space consider switching to Bonsai data storage format [#6230](https://github.com/hyperledger/besu/pull/6230) @@ -15,9 +16,15 @@ - Add custom genesis file name to config overview if specified [#6297](https://github.com/hyperledger/besu/pull/6297) - Update Gradle plugins and replace unmaintained License Gradle Plugin with the actively maintained Gradle License Report [#6275](https://github.com/hyperledger/besu/pull/6275) - Optimize RocksDB WAL files, allows for faster restart and a more linear disk space utilization [#6328](https://github.com/hyperledger/besu/pull/6328) +- Disable transaction handling when the node is not in sync, to avoid unnecessary transaction validation work [#6302](https://github.com/hyperledger/besu/pull/6302) +- Upgrade dependencies [#6377](https://github.com/hyperledger/besu/pull/6377) +- Upgrade `com.fasterxml.jackson` dependencies [#6378](https://github.com/hyperledger/besu/pull/6378) ### Bug fixes - INTERNAL_ERROR from `eth_estimateGas` JSON/RPC calls [#6344](https://github.com/hyperledger/besu/issues/6344) +- Fix Besu Docker images with `openjdk-latest` tags since 23.10.3 using UID 1001 instead of 1000 for the `besu` user [#6360](https://github.com/hyperledger/besu/pull/6360) +- Fluent EVM API definition for Tangerine Whistle had incorrect code size validation configured [#6382](https://github.com/hyperledger/besu/pull/6382) +- Correct mining beneficiary for Clique networks in TraceServiceImpl [#6390](https://github.com/hyperledger/besu/pull/6390) ## 23.10.3 diff --git a/acceptance-tests/dsl/build.gradle b/acceptance-tests/dsl/build.gradle index 96a6a1c1bc0..6c9090fe9b9 100644 --- a/acceptance-tests/dsl/build.gradle +++ b/acceptance-tests/dsl/build.gradle @@ -26,7 +26,6 @@ dependencies { implementation project(':testutil') implementation project(':util') - implementation 'com.github.tomakehurst:wiremock-jre8' implementation 'com.google.guava:guava' implementation 'com.google.dagger:dagger' annotationProcessor 'com.google.dagger:dagger-compiler' @@ -45,6 +44,7 @@ dependencies { implementation 'org.web3j:abi' implementation 'org.web3j:besu' implementation 'org.web3j:crypto' + implementation 'org.wiremock:wiremock' implementation 'org.testcontainers:testcontainers' implementation 'org.junit.jupiter:junit-jupiter' diff --git a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/eth/EthConditions.java b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/eth/EthConditions.java index c4ee1347278..d3a86bafa86 100644 --- a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/eth/EthConditions.java +++ b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/eth/EthConditions.java @@ -77,6 +77,10 @@ public Condition miningStatus(final boolean isMining) { return new MiningStatusCondition(transactions.mining(), isMining); } + public Condition syncingStatus(final boolean isSyncing) { + return new SyncingStatusCondition(transactions.syncing(), isSyncing); + } + public Condition expectNewPendingTransactions( final BigInteger filterId, final List transactionHashes) { return new NewPendingTransactionFilterChangesCondition( diff --git a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/eth/SyncingStatusCondition.java b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/eth/SyncingStatusCondition.java new file mode 100644 index 00000000000..5e01a0d4a90 --- /dev/null +++ b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/condition/eth/SyncingStatusCondition.java @@ -0,0 +1,40 @@ +/* + * Copyright Hyperledger Besu Contributors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on + * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + * SPDX-License-Identifier: Apache-2.0 + */ +package org.hyperledger.besu.tests.acceptance.dsl.condition.eth; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.hyperledger.besu.tests.acceptance.dsl.WaitUtils; +import org.hyperledger.besu.tests.acceptance.dsl.condition.Condition; +import org.hyperledger.besu.tests.acceptance.dsl.node.Node; +import org.hyperledger.besu.tests.acceptance.dsl.transaction.eth.EthSyncingTransaction; + +public class SyncingStatusCondition implements Condition { + + private final EthSyncingTransaction transaction; + private final boolean syncingMiningStatus; + + public SyncingStatusCondition( + final EthSyncingTransaction transaction, final boolean syncingStatus) { + this.transaction = transaction; + this.syncingMiningStatus = syncingStatus; + } + + @Override + public void verify(final Node node) { + WaitUtils.waitFor( + 10, () -> assertThat(node.execute(transaction)).isEqualTo(syncingMiningStatus)); + } +} diff --git a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/cluster/Cluster.java b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/cluster/Cluster.java index 16a02874262..b454894e066 100644 --- a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/cluster/Cluster.java +++ b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/cluster/Cluster.java @@ -86,11 +86,8 @@ public void start(final List nodes) { final Optional bootnode = selectAndStartBootnode(nodes); nodes.parallelStream() - .filter( - node -> { - LOG.info("starting non-bootnode {}", node.getName()); - return bootnode.map(boot -> boot != node).orElse(true); - }) + .filter(node -> bootnode.map(boot -> boot != node).orElse(true)) + .peek(node -> LOG.info("starting non-bootnode {}", node.getName())) .forEach(this::startNode); if (clusterConfiguration.isAwaitPeerDiscovery()) { diff --git a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/eth/EthSyncingTransaction.java b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/eth/EthSyncingTransaction.java new file mode 100644 index 00000000000..21b628abef2 --- /dev/null +++ b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/eth/EthSyncingTransaction.java @@ -0,0 +1,40 @@ +/* + * Copyright Hyperledger Besu Contributors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on + * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + * SPDX-License-Identifier: Apache-2.0 + */ +package org.hyperledger.besu.tests.acceptance.dsl.transaction.eth; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.hyperledger.besu.tests.acceptance.dsl.transaction.NodeRequests; +import org.hyperledger.besu.tests.acceptance.dsl.transaction.Transaction; + +import java.io.IOException; + +import org.web3j.protocol.core.methods.response.EthSyncing; + +public class EthSyncingTransaction implements Transaction { + + EthSyncingTransaction() {} + + @Override + public Boolean execute(final NodeRequests node) { + try { + EthSyncing response = node.eth().ethSyncing().send(); + assertThat(response).isNotNull(); + return response.isSyncing(); + } catch (final IOException e) { + throw new RuntimeException(e); + } + } +} diff --git a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/eth/EthTransactions.java b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/eth/EthTransactions.java index 84882b822d0..b8b72052d0b 100644 --- a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/eth/EthTransactions.java +++ b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/transaction/eth/EthTransactions.java @@ -73,6 +73,10 @@ public EthMiningTransaction mining() { return new EthMiningTransaction(); } + public EthSyncingTransaction syncing() { + return new EthSyncingTransaction(); + } + public EthNewPendingTransactionFilterTransaction newPendingTransactionsFilter() { return new EthNewPendingTransactionFilterTransaction(); } diff --git a/acceptance-tests/tests/build.gradle b/acceptance-tests/tests/build.gradle index 464805779c8..b20b96b2735 100644 --- a/acceptance-tests/tests/build.gradle +++ b/acceptance-tests/tests/build.gradle @@ -55,7 +55,6 @@ dependencies { testImplementation project(':testutil') testImplementation project(':util') - testImplementation 'com.github.tomakehurst:wiremock-jre8-standalone' testImplementation 'commons-io:commons-io' testImplementation 'io.grpc:grpc-all' testImplementation 'io.grpc:grpc-core' @@ -84,6 +83,7 @@ dependencies { testImplementation 'org.web3j:abi' testImplementation 'org.web3j:besu' testImplementation 'org.web3j:core' + testImplementation 'org.wiremock:wiremock' testRuntimeOnly 'org.junit.vintage:junit-vintage-engine' } diff --git a/acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningAcceptanceTest.java b/acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningAcceptanceTest.java index c5fbb693043..e9442bc0077 100644 --- a/acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningAcceptanceTest.java +++ b/acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningAcceptanceTest.java @@ -49,6 +49,11 @@ public void setUp() { permissionedNode.verify(admin.addPeer(bootnode)); permissionedNode.verify(admin.addPeer(allowedNode)); + + allowedNode.verify(eth.syncingStatus(false)); + bootnode.verify(eth.syncingStatus(false)); + permissionedNode.verify(eth.syncingStatus(false)); + forbiddenNode.verify(eth.syncingStatus(false)); } @Test diff --git a/acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningV2AcceptanceTest.java b/acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningV2AcceptanceTest.java index 72aab9cb7e4..b04bad5644d 100644 --- a/acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningV2AcceptanceTest.java +++ b/acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningV2AcceptanceTest.java @@ -46,6 +46,11 @@ public void setUp() { permissionedNode.execute(allowNode(permissionedNode)); permissionedNode.verify(connectionIsAllowed(permissionedNode)); + + allowedNode.verify(eth.syncingStatus(false)); + bootnode.verify(eth.syncingStatus(false)); + permissionedNode.verify(eth.syncingStatus(false)); + forbiddenNode.verify(eth.syncingStatus(false)); } @Test diff --git a/acceptance-tests/tests/src/test/resources/permissioning/simple_permissioning_genesis.json b/acceptance-tests/tests/src/test/resources/permissioning/simple_permissioning_genesis.json index 04ee1a4882e..5e8bc92a46a 100644 --- a/acceptance-tests/tests/src/test/resources/permissioning/simple_permissioning_genesis.json +++ b/acceptance-tests/tests/src/test/resources/permissioning/simple_permissioning_genesis.json @@ -4,7 +4,7 @@ "londonBlock": 0, "zeroBaseFee": true, "ethash": { - "fixeddifficulty": 100 + "fixeddifficulty": 500 } }, "nonce": "0x42", diff --git a/acceptance-tests/tests/src/test/resources/permissioning/simple_permissioning_v2_genesis.json b/acceptance-tests/tests/src/test/resources/permissioning/simple_permissioning_v2_genesis.json index b7ec620f310..aab9a35cdd3 100644 --- a/acceptance-tests/tests/src/test/resources/permissioning/simple_permissioning_v2_genesis.json +++ b/acceptance-tests/tests/src/test/resources/permissioning/simple_permissioning_v2_genesis.json @@ -4,7 +4,7 @@ "londonBlock": 0, "zeroBaseFee": true, "ethash": { - "fixeddifficulty": 100 + "fixeddifficulty": 500 } }, "nonce": "0x42", diff --git a/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java b/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java index 9fa41f31354..e359c9dc4e3 100644 --- a/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java +++ b/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java @@ -23,6 +23,7 @@ import static org.hyperledger.besu.cli.config.NetworkName.MAINNET; import static org.hyperledger.besu.cli.util.CommandLineUtils.DEPENDENCY_WARNING_MSG; import static org.hyperledger.besu.cli.util.CommandLineUtils.DEPRECATION_WARNING_MSG; +import static org.hyperledger.besu.cli.util.CommandLineUtils.isOptionSet; import static org.hyperledger.besu.controller.BesuController.DATABASE_PATH; import static org.hyperledger.besu.ethereum.api.graphql.GraphQLConfiguration.DEFAULT_GRAPHQL_HTTP_PORT; import static org.hyperledger.besu.ethereum.api.jsonrpc.JsonRpcConfiguration.DEFAULT_ENGINE_JSON_RPC_PORT; @@ -521,11 +522,11 @@ private InetAddress autoDiscoverDefaultIP() { private SyncMode syncMode = null; @Option( - names = {"--fast-sync-min-peers"}, + names = {"--sync-min-peers", "--fast-sync-min-peers"}, paramLabel = MANDATORY_INTEGER_FORMAT_HELP, description = - "Minimum number of peers required before starting fast sync. Has only effect on PoW networks. (default: ${DEFAULT-VALUE})") - private final Integer fastSyncMinPeerCount = FAST_SYNC_MIN_PEER_COUNT; + "Minimum number of peers required before starting sync. Has effect only on non-PoS networks. (default: ${DEFAULT-VALUE})") + private final Integer syncMinPeerCount = SYNC_MIN_PEER_COUNT; @Option( names = {"--network"}, @@ -2028,11 +2029,10 @@ private void issueOptionWarnings() { "--p2p-port", "--remote-connections-max-percentage")); - CommandLineUtils.failIfOptionDoesntMeetRequirement( - commandLine, - "--fast-sync-min-peers can't be used with FULL sync-mode", - !SyncMode.isFullSync(getDefaultSyncModeIfNotSet()), - singletonList("--fast-sync-min-peers")); + if (SyncMode.isFullSync(getDefaultSyncModeIfNotSet()) + && isOptionSet(commandLine, "--sync-min-peers")) { + logger.warn("--sync-min-peers is ignored in FULL sync-mode"); + } CommandLineUtils.failIfOptionDoesntMeetRequirement( commandLine, @@ -2867,7 +2867,7 @@ private SynchronizerConfiguration buildSyncConfig() { return unstableSynchronizerOptions .toDomainObject() .syncMode(syncMode) - .fastSyncMinimumPeerCount(fastSyncMinPeerCount) + .fastSyncMinimumPeerCount(syncMinPeerCount) .build(); } diff --git a/besu/src/main/java/org/hyperledger/besu/cli/DefaultCommandValues.java b/besu/src/main/java/org/hyperledger/besu/cli/DefaultCommandValues.java index b7d603b8caa..d8644c36de7 100644 --- a/besu/src/main/java/org/hyperledger/besu/cli/DefaultCommandValues.java +++ b/besu/src/main/java/org/hyperledger/besu/cli/DefaultCommandValues.java @@ -66,8 +66,8 @@ public interface DefaultCommandValues { NatMethod DEFAULT_NAT_METHOD = NatMethod.AUTO; /** The constant DEFAULT_JWT_ALGORITHM. */ JwtAlgorithm DEFAULT_JWT_ALGORITHM = JwtAlgorithm.RS256; - /** The constant FAST_SYNC_MIN_PEER_COUNT. */ - int FAST_SYNC_MIN_PEER_COUNT = 5; + /** The constant SYNC_MIN_PEER_COUNT. */ + int SYNC_MIN_PEER_COUNT = 5; /** The constant DEFAULT_MAX_PEERS. */ int DEFAULT_MAX_PEERS = 25; /** The constant DEFAULT_P2P_PEER_LOWER_BOUND. */ diff --git a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/RocksDbUsageHelper.java b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/RocksDbUsageHelper.java index 4bfd9f42951..4a11abcdf02 100644 --- a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/RocksDbUsageHelper.java +++ b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/RocksDbUsageHelper.java @@ -34,17 +34,17 @@ static void printUsageForColumnFamily( final RocksDB rocksdb, final ColumnFamilyHandle cfHandle, final PrintWriter out) throws RocksDBException, NumberFormatException { final String size = rocksdb.getProperty(cfHandle, "rocksdb.estimate-live-data-size"); + final String numberOfKeys = rocksdb.getProperty(cfHandle, "rocksdb.estimate-num-keys"); boolean emptyColumnFamily = false; - if (!size.isEmpty() && !size.isBlank()) { + if (!size.isBlank() && !numberOfKeys.isBlank()) { try { final long sizeLong = Long.parseLong(size); + final long numberOfKeysLong = Long.parseLong(numberOfKeys); final String totalSstFilesSize = rocksdb.getProperty(cfHandle, "rocksdb.total-sst-files-size"); final long totalSstFilesSizeLong = - !totalSstFilesSize.isEmpty() && !totalSstFilesSize.isBlank() - ? Long.parseLong(totalSstFilesSize) - : 0; - if (sizeLong == 0) { + !totalSstFilesSize.isBlank() ? Long.parseLong(totalSstFilesSize) : 0; + if (sizeLong == 0 && numberOfKeysLong == 0) { emptyColumnFamily = true; } diff --git a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/StorageSubCommand.java b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/StorageSubCommand.java index e46ffde2739..0dbfa3d22c5 100644 --- a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/StorageSubCommand.java +++ b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/StorageSubCommand.java @@ -45,7 +45,11 @@ description = "This command provides storage related actions.", mixinStandardHelpOptions = true, versionProvider = VersionProvider.class, - subcommands = {StorageSubCommand.RevertVariablesStorage.class, RocksDbSubCommand.class}) + subcommands = { + StorageSubCommand.RevertVariablesStorage.class, + RocksDbSubCommand.class, + TrieLogSubCommand.class + }) public class StorageSubCommand implements Runnable { /** The constant COMMAND_NAME. */ diff --git a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java new file mode 100644 index 00000000000..52dbe559034 --- /dev/null +++ b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java @@ -0,0 +1,361 @@ +/* + * Copyright contributors to Hyperledger Besu. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on + * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + * SPDX-License-Identifier: Apache-2.0 + */ + +package org.hyperledger.besu.cli.subcommands.storage; + +import static com.google.common.base.Preconditions.checkArgument; +import static org.hyperledger.besu.controller.BesuController.DATABASE_PATH; + +import org.hyperledger.besu.datatypes.Hash; +import org.hyperledger.besu.ethereum.chain.Blockchain; +import org.hyperledger.besu.ethereum.chain.MutableBlockchain; +import org.hyperledger.besu.ethereum.core.BlockHeader; +import org.hyperledger.besu.ethereum.trie.bonsai.storage.BonsaiWorldStateKeyValueStorage; +import org.hyperledger.besu.ethereum.worldstate.DataStorageConfiguration; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.io.ObjectOutputStream; +import java.io.PrintWriter; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.IdentityHashMap; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.tuweni.bytes.Bytes32; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** Helper class for counting and pruning trie logs */ +public class TrieLogHelper { + private static final String TRIE_LOG_FILE = "trieLogsToRetain"; + private static final long BATCH_SIZE = 20_000; + private static final int ROCKSDB_MAX_INSERTS_PER_TRANSACTION = 1000; + private static final Logger LOG = LoggerFactory.getLogger(TrieLogHelper.class); + + static void prune( + final DataStorageConfiguration config, + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + final MutableBlockchain blockchain, + final Path dataDirectoryPath) { + final String batchFileNameBase = + dataDirectoryPath.resolve(DATABASE_PATH).resolve(TRIE_LOG_FILE).toString(); + + validatePruneConfiguration(config); + + final long layersToRetain = config.getUnstable().getBonsaiTrieLogRetentionThreshold(); + + final long chainHeight = blockchain.getChainHeadBlockNumber(); + + final long lastBlockNumberToRetainTrieLogsFor = chainHeight - layersToRetain + 1; + + if (!validPruneRequirements(blockchain, chainHeight, lastBlockNumberToRetainTrieLogsFor)) { + return; + } + + final long numberOfBatches = calculateNumberofBatches(layersToRetain); + + processTrieLogBatches( + rootWorldStateStorage, + blockchain, + chainHeight, + lastBlockNumberToRetainTrieLogsFor, + numberOfBatches, + batchFileNameBase); + + if (rootWorldStateStorage.streamTrieLogKeys(layersToRetain).count() == layersToRetain) { + deleteFiles(batchFileNameBase, numberOfBatches); + LOG.info("Prune ran successfully. Enjoy some disk space back! \uD83D\uDE80"); + } else { + LOG.error("Prune failed. Re-run the subcommand to load the trie logs from file."); + } + } + + private static void processTrieLogBatches( + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + final MutableBlockchain blockchain, + final long chainHeight, + final long lastBlockNumberToRetainTrieLogsFor, + final long numberOfBatches, + final String batchFileNameBase) { + + for (long batchNumber = 1; batchNumber <= numberOfBatches; batchNumber++) { + + final long firstBlockOfBatch = chainHeight - ((batchNumber - 1) * BATCH_SIZE); + + final long lastBlockOfBatch = + Math.max(chainHeight - (batchNumber * BATCH_SIZE), lastBlockNumberToRetainTrieLogsFor); + + final List trieLogKeys = + getTrieLogKeysForBlocks(blockchain, firstBlockOfBatch, lastBlockOfBatch); + + saveTrieLogBatches(batchFileNameBase, rootWorldStateStorage, batchNumber, trieLogKeys); + } + + LOG.info("Clear trie logs..."); + rootWorldStateStorage.clearTrieLog(); + + for (long batchNumber = 1; batchNumber <= numberOfBatches; batchNumber++) { + restoreTrieLogBatches(rootWorldStateStorage, batchNumber, batchFileNameBase); + } + } + + private static void saveTrieLogBatches( + final String batchFileNameBase, + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + final long batchNumber, + final List trieLogKeys) { + + LOG.info("Saving trie logs to retain in file (batch {})...", batchNumber); + + try { + saveTrieLogsInFile(trieLogKeys, rootWorldStateStorage, batchNumber, batchFileNameBase); + } catch (IOException e) { + LOG.error("Error saving trie logs to file: {}", e.getMessage()); + throw new RuntimeException(e); + } + } + + private static void restoreTrieLogBatches( + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + final long batchNumber, + final String batchFileNameBase) { + + try { + LOG.info("Restoring trie logs retained from batch {}...", batchNumber); + recreateTrieLogs(rootWorldStateStorage, batchNumber, batchFileNameBase); + } catch (IOException e) { + LOG.error("Error recreating trie logs from batch {}: {}", batchNumber, e.getMessage()); + throw new RuntimeException(e); + } + } + + private static void deleteFiles(final String batchFileNameBase, final long numberOfBatches) { + + LOG.info("Deleting files..."); + + for (long batchNumber = 1; batchNumber <= numberOfBatches; batchNumber++) { + File file = new File(batchFileNameBase + "-" + batchNumber); + if (file.exists()) { + file.delete(); + } + } + } + + private static List getTrieLogKeysForBlocks( + final MutableBlockchain blockchain, + final long firstBlockOfBatch, + final long lastBlockOfBatch) { + final List trieLogKeys = new ArrayList<>(); + for (long i = firstBlockOfBatch; i >= lastBlockOfBatch; i--) { + final Optional header = blockchain.getBlockHeader(i); + header.ifPresentOrElse( + blockHeader -> trieLogKeys.add(blockHeader.getHash()), + () -> LOG.error("Error retrieving block")); + } + return trieLogKeys; + } + + private static long calculateNumberofBatches(final long layersToRetain) { + return layersToRetain / BATCH_SIZE + ((layersToRetain % BATCH_SIZE == 0) ? 0 : 1); + } + + private static boolean validPruneRequirements( + final MutableBlockchain blockchain, + final long chainHeight, + final long lastBlockNumberToRetainTrieLogsFor) { + if (lastBlockNumberToRetainTrieLogsFor < 0) { + throw new IllegalArgumentException( + "Trying to retain more trie logs than chain length (" + + chainHeight + + "), skipping pruning"); + } + + final Optional finalizedBlockHash = blockchain.getFinalized(); + + if (finalizedBlockHash.isEmpty()) { + throw new RuntimeException("No finalized block present, can't safely run trie log prune"); + } else { + final Hash finalizedHash = finalizedBlockHash.get(); + final Optional finalizedBlockHeader = blockchain.getBlockHeader(finalizedHash); + if (finalizedBlockHeader.isPresent() + && finalizedBlockHeader.get().getNumber() < lastBlockNumberToRetainTrieLogsFor) { + throw new IllegalArgumentException( + "Trying to prune more layers than the finalized block height, skipping pruning"); + } + } + return true; + } + + private static void recreateTrieLogs( + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + final long batchNumber, + final String batchFileNameBase) + throws IOException { + // process in chunk to avoid OOM + + IdentityHashMap trieLogsToRetain = + readTrieLogsFromFile(batchFileNameBase, batchNumber); + final int chunkSize = ROCKSDB_MAX_INSERTS_PER_TRANSACTION; + List keys = new ArrayList<>(trieLogsToRetain.keySet()); + + for (int startIndex = 0; startIndex < keys.size(); startIndex += chunkSize) { + processTransactionChunk(startIndex, chunkSize, keys, trieLogsToRetain, rootWorldStateStorage); + } + } + + private static void processTransactionChunk( + final int startIndex, + final int chunkSize, + final List keys, + final IdentityHashMap trieLogsToRetain, + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage) { + + var updater = rootWorldStateStorage.updater(); + int endIndex = Math.min(startIndex + chunkSize, keys.size()); + + for (int i = startIndex; i < endIndex; i++) { + byte[] key = keys.get(i); + byte[] value = trieLogsToRetain.get(key); + updater.getTrieLogStorageTransaction().put(key, value); + LOG.info("Key({}): {}", i, Bytes32.wrap(key).toShortHexString()); + } + + updater.getTrieLogStorageTransaction().commit(); + } + + private static void validatePruneConfiguration(final DataStorageConfiguration config) { + checkArgument( + config.getUnstable().getBonsaiTrieLogRetentionThreshold() + >= config.getBonsaiMaxLayersToLoad(), + String.format( + "--Xbonsai-trie-log-retention-threshold minimum value is %d", + config.getBonsaiMaxLayersToLoad())); + checkArgument( + config.getUnstable().getBonsaiTrieLogPruningLimit() > 0, + String.format( + "--Xbonsai-trie-log-pruning-limit=%d must be greater than 0", + config.getUnstable().getBonsaiTrieLogPruningLimit())); + checkArgument( + config.getUnstable().getBonsaiTrieLogPruningLimit() + > config.getUnstable().getBonsaiTrieLogRetentionThreshold(), + String.format( + "--Xbonsai-trie-log-pruning-limit=%d must greater than --Xbonsai-trie-log-retention-threshold=%d", + config.getUnstable().getBonsaiTrieLogPruningLimit(), + config.getUnstable().getBonsaiTrieLogRetentionThreshold())); + } + + private static void saveTrieLogsInFile( + final List trieLogsKeys, + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + final long batchNumber, + final String batchFileNameBase) + throws IOException { + + File file = new File(batchFileNameBase + "-" + batchNumber); + if (file.exists()) { + LOG.error("File already exists, skipping file creation"); + return; + } + + try (FileOutputStream fos = new FileOutputStream(file)) { + ObjectOutputStream oos = new ObjectOutputStream(fos); + oos.writeObject(getTrieLogs(trieLogsKeys, rootWorldStateStorage)); + } catch (IOException e) { + LOG.error(e.getMessage()); + throw new RuntimeException(e); + } + } + + @SuppressWarnings("unchecked") + private static IdentityHashMap readTrieLogsFromFile( + final String batchFileNameBase, final long batchNumber) { + + IdentityHashMap trieLogs; + try (FileInputStream fis = new FileInputStream(batchFileNameBase + "-" + batchNumber); + ObjectInputStream ois = new ObjectInputStream(fis)) { + + trieLogs = (IdentityHashMap) ois.readObject(); + + } catch (IOException | ClassNotFoundException e) { + + LOG.error(e.getMessage()); + throw new RuntimeException(e); + } + + return trieLogs; + } + + private static IdentityHashMap getTrieLogs( + final List trieLogKeys, final BonsaiWorldStateKeyValueStorage rootWorldStateStorage) { + IdentityHashMap trieLogsToRetain = new IdentityHashMap<>(); + + LOG.info("Obtaining trielogs from db, this may take a few minutes..."); + trieLogKeys.forEach( + hash -> + rootWorldStateStorage + .getTrieLog(hash) + .ifPresent(trieLog -> trieLogsToRetain.put(hash.toArrayUnsafe(), trieLog))); + return trieLogsToRetain; + } + + static TrieLogCount getCount( + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + final int limit, + final Blockchain blockchain) { + final AtomicInteger total = new AtomicInteger(); + final AtomicInteger canonicalCount = new AtomicInteger(); + final AtomicInteger forkCount = new AtomicInteger(); + final AtomicInteger orphanCount = new AtomicInteger(); + rootWorldStateStorage + .streamTrieLogKeys(limit) + .map(Bytes32::wrap) + .map(Hash::wrap) + .forEach( + hash -> { + total.getAndIncrement(); + blockchain + .getBlockHeader(hash) + .ifPresentOrElse( + (header) -> { + long number = header.getNumber(); + final Optional headerByNumber = + blockchain.getBlockHeader(number); + if (headerByNumber.isPresent() + && headerByNumber.get().getHash().equals(hash)) { + canonicalCount.getAndIncrement(); + } else { + forkCount.getAndIncrement(); + } + }, + orphanCount::getAndIncrement); + }); + + return new TrieLogCount(total.get(), canonicalCount.get(), forkCount.get(), orphanCount.get()); + } + + static void printCount(final PrintWriter out, final TrieLogCount count) { + out.printf( + "trieLog count: %s\n - canonical count: %s\n - fork count: %s\n - orphaned count: %s\n", + count.total, count.canonicalCount, count.forkCount, count.orphanCount); + } + + record TrieLogCount(int total, int canonicalCount, int forkCount, int orphanCount) {} +} diff --git a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogSubCommand.java b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogSubCommand.java new file mode 100644 index 00000000000..bf75fd6eb9a --- /dev/null +++ b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogSubCommand.java @@ -0,0 +1,147 @@ +/* + * Copyright Hyperledger Besu Contributors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on + * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + * SPDX-License-Identifier: Apache-2.0 + */ +package org.hyperledger.besu.cli.subcommands.storage; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; + +import org.hyperledger.besu.cli.util.VersionProvider; +import org.hyperledger.besu.controller.BesuController; +import org.hyperledger.besu.ethereum.chain.MutableBlockchain; +import org.hyperledger.besu.ethereum.storage.StorageProvider; +import org.hyperledger.besu.ethereum.trie.bonsai.storage.BonsaiWorldStateKeyValueStorage; +import org.hyperledger.besu.ethereum.trie.bonsai.trielog.TrieLogPruner; +import org.hyperledger.besu.ethereum.worldstate.DataStorageConfiguration; +import org.hyperledger.besu.ethereum.worldstate.DataStorageFormat; + +import java.io.PrintWriter; +import java.nio.file.Path; +import java.nio.file.Paths; + +import org.apache.logging.log4j.Level; +import org.apache.logging.log4j.core.config.Configurator; +import org.slf4j.LoggerFactory; +import picocli.CommandLine; +import picocli.CommandLine.Command; +import picocli.CommandLine.ParentCommand; + +/** The Trie Log subcommand. */ +@Command( + name = "x-trie-log", + description = "Manipulate trie logs", + mixinStandardHelpOptions = true, + versionProvider = VersionProvider.class, + subcommands = {TrieLogSubCommand.CountTrieLog.class, TrieLogSubCommand.PruneTrieLog.class}) +public class TrieLogSubCommand implements Runnable { + + @SuppressWarnings("UnusedVariable") + @ParentCommand + private static StorageSubCommand parentCommand; + + @SuppressWarnings("unused") + @CommandLine.Spec + private CommandLine.Model.CommandSpec spec; // Picocli injects reference to command spec + + @Override + public void run() { + final PrintWriter out = spec.commandLine().getOut(); + spec.commandLine().usage(out); + } + + private static BesuController createBesuController() { + return parentCommand.parentCommand.buildController(); + } + + @Command( + name = "count", + description = "This command counts all the trie logs", + mixinStandardHelpOptions = true, + versionProvider = VersionProvider.class) + static class CountTrieLog implements Runnable { + + @SuppressWarnings("unused") + @ParentCommand + private TrieLogSubCommand parentCommand; + + @SuppressWarnings("unused") + @CommandLine.Spec + private CommandLine.Model.CommandSpec spec; // Picocli injects reference to command spec + + @Override + public void run() { + TrieLogContext context = getTrieLogContext(); + + final PrintWriter out = spec.commandLine().getOut(); + + out.println("Counting trie logs..."); + TrieLogHelper.printCount( + out, + TrieLogHelper.getCount( + context.rootWorldStateStorage, Integer.MAX_VALUE, context.blockchain)); + } + } + + @Command( + name = "prune", + description = + "This command prunes all trie log layers below the retention threshold, including orphaned trie logs.", + mixinStandardHelpOptions = true, + versionProvider = VersionProvider.class) + static class PruneTrieLog implements Runnable { + + @SuppressWarnings("unused") + @ParentCommand + private TrieLogSubCommand parentCommand; + + @SuppressWarnings("unused") + @CommandLine.Spec + private CommandLine.Model.CommandSpec spec; // Picocli injects reference to command spec + + @Override + public void run() { + TrieLogContext context = getTrieLogContext(); + final Path dataDirectoryPath = + Paths.get( + TrieLogSubCommand.parentCommand.parentCommand.dataDir().toAbsolutePath().toString()); + TrieLogHelper.prune( + context.config(), + context.rootWorldStateStorage(), + context.blockchain(), + dataDirectoryPath); + } + } + + record TrieLogContext( + DataStorageConfiguration config, + BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + MutableBlockchain blockchain) {} + + private static TrieLogContext getTrieLogContext() { + Configurator.setLevel(LoggerFactory.getLogger(TrieLogPruner.class).getName(), Level.DEBUG); + checkNotNull(parentCommand); + BesuController besuController = createBesuController(); + final DataStorageConfiguration config = besuController.getDataStorageConfiguration(); + checkArgument( + DataStorageFormat.BONSAI.equals(config.getDataStorageFormat()), + "Subcommand only works with data-storage-format=BONSAI"); + + final StorageProvider storageProvider = besuController.getStorageProvider(); + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage = + (BonsaiWorldStateKeyValueStorage) + storageProvider.createWorldStateStorage(DataStorageFormat.BONSAI); + final MutableBlockchain blockchain = besuController.getProtocolContext().getBlockchain(); + return new TrieLogContext(config, rootWorldStateStorage, blockchain); + } +} diff --git a/besu/src/main/java/org/hyperledger/besu/controller/BesuController.java b/besu/src/main/java/org/hyperledger/besu/controller/BesuController.java index 254c36d5506..c9d29ac5e66 100644 --- a/besu/src/main/java/org/hyperledger/besu/controller/BesuController.java +++ b/besu/src/main/java/org/hyperledger/besu/controller/BesuController.java @@ -37,6 +37,7 @@ import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule; import org.hyperledger.besu.ethereum.p2p.config.SubProtocolConfiguration; import org.hyperledger.besu.ethereum.storage.StorageProvider; +import org.hyperledger.besu.ethereum.worldstate.DataStorageConfiguration; import java.io.Closeable; import java.io.IOException; @@ -77,6 +78,7 @@ public class BesuController implements java.io.Closeable { private final SyncState syncState; private final EthPeers ethPeers; private final StorageProvider storageProvider; + private final DataStorageConfiguration dataStorageConfiguration; /** * Instantiates a new Besu controller. @@ -96,6 +98,9 @@ public class BesuController implements java.io.Closeable { * @param nodeKey the node key * @param closeables the closeables * @param additionalPluginServices the additional plugin services + * @param ethPeers the eth peers + * @param storageProvider the storage provider + * @param dataStorageConfiguration the data storage configuration */ BesuController( final ProtocolSchedule protocolSchedule, @@ -114,7 +119,8 @@ public class BesuController implements java.io.Closeable { final List closeables, final PluginServiceFactory additionalPluginServices, final EthPeers ethPeers, - final StorageProvider storageProvider) { + final StorageProvider storageProvider, + final DataStorageConfiguration dataStorageConfiguration) { this.protocolSchedule = protocolSchedule; this.protocolContext = protocolContext; this.ethProtocolManager = ethProtocolManager; @@ -132,6 +138,7 @@ public class BesuController implements java.io.Closeable { this.additionalPluginServices = additionalPluginServices; this.ethPeers = ethPeers; this.storageProvider = storageProvider; + this.dataStorageConfiguration = dataStorageConfiguration; } /** @@ -293,6 +300,15 @@ public PluginServiceFactory getAdditionalPluginServices() { return additionalPluginServices; } + /** + * Gets data storage configuration. + * + * @return the data storage configuration + */ + public DataStorageConfiguration getDataStorageConfiguration() { + return dataStorageConfiguration; + } + /** The type Builder. */ public static class Builder { diff --git a/besu/src/main/java/org/hyperledger/besu/controller/BesuControllerBuilder.java b/besu/src/main/java/org/hyperledger/besu/controller/BesuControllerBuilder.java index 92cbf320e84..6bcfea3cad8 100644 --- a/besu/src/main/java/org/hyperledger/besu/controller/BesuControllerBuilder.java +++ b/besu/src/main/java/org/hyperledger/besu/controller/BesuControllerBuilder.java @@ -803,7 +803,8 @@ public BesuController build() { closeables, additionalPluginServices, ethPeers, - storageProvider); + storageProvider, + dataStorageConfiguration); } /** diff --git a/besu/src/main/java/org/hyperledger/besu/services/TraceServiceImpl.java b/besu/src/main/java/org/hyperledger/besu/services/TraceServiceImpl.java index 3ab2961734f..43af4e3866e 100644 --- a/besu/src/main/java/org/hyperledger/besu/services/TraceServiceImpl.java +++ b/besu/src/main/java/org/hyperledger/besu/services/TraceServiceImpl.java @@ -216,7 +216,7 @@ private List trace( worldUpdater, header, transaction, - header.getCoinbase(), + protocolSpec.getMiningBeneficiaryCalculator().calculateBeneficiary(header), tracer, new CachingBlockHashLookup(header, blockchain), false, diff --git a/besu/src/test/java/org/hyperledger/besu/cli/BesuCommandTest.java b/besu/src/test/java/org/hyperledger/besu/cli/BesuCommandTest.java index 3db626287fa..ba0e5b2754a 100644 --- a/besu/src/test/java/org/hyperledger/besu/cli/BesuCommandTest.java +++ b/besu/src/test/java/org/hyperledger/besu/cli/BesuCommandTest.java @@ -1881,6 +1881,30 @@ public void parsesValidFastSyncMinPeersOption() { assertThat(commandErrorOutput.toString(UTF_8)).isEmpty(); } + @Test + public void parsesValidSnapSyncMinPeersOption() { + parseCommand("--sync-mode", "X_SNAP", "--sync-min-peers", "11"); + verify(mockControllerBuilder).synchronizerConfiguration(syncConfigurationCaptor.capture()); + + final SynchronizerConfiguration syncConfig = syncConfigurationCaptor.getValue(); + assertThat(syncConfig.getSyncMode()).isEqualTo(SyncMode.X_SNAP); + assertThat(syncConfig.getFastSyncMinimumPeerCount()).isEqualTo(11); + assertThat(commandOutput.toString(UTF_8)).isEmpty(); + assertThat(commandErrorOutput.toString(UTF_8)).isEmpty(); + } + + @Test + public void parsesValidSyncMinPeersOption() { + parseCommand("--sync-mode", "FAST", "--sync-min-peers", "11"); + verify(mockControllerBuilder).synchronizerConfiguration(syncConfigurationCaptor.capture()); + + final SynchronizerConfiguration syncConfig = syncConfigurationCaptor.getValue(); + assertThat(syncConfig.getSyncMode()).isEqualTo(SyncMode.FAST); + assertThat(syncConfig.getFastSyncMinimumPeerCount()).isEqualTo(11); + assertThat(commandOutput.toString(UTF_8)).isEmpty(); + assertThat(commandErrorOutput.toString(UTF_8)).isEmpty(); + } + @Test public void parsesInvalidFastSyncMinPeersOptionWrongFormatShouldFail() { @@ -5042,8 +5066,7 @@ public void logsSuggestInstallingJemallocWhenEnvVarNotPresent() { @Test public void logWarnIfFastSyncMinPeersUsedWithFullSync() { parseCommand("--sync-mode", "FULL", "--fast-sync-min-peers", "1"); - assertThat(commandErrorOutput.toString(UTF_8)) - .contains("--fast-sync-min-peers can't be used with FULL sync-mode"); + verify(mockLogger).warn("--sync-min-peers is ignored in FULL sync-mode"); } @Test diff --git a/besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java b/besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java new file mode 100644 index 00000000000..82659701b1c --- /dev/null +++ b/besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java @@ -0,0 +1,265 @@ +/* + * Copyright contributors to Hyperledger Besu. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on + * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + * SPDX-License-Identifier: Apache-2.0 + */ + +package org.hyperledger.besu.cli.subcommands.storage; + +import static org.hyperledger.besu.ethereum.worldstate.DataStorageFormat.BONSAI; +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.when; + +import org.hyperledger.besu.datatypes.Hash; +import org.hyperledger.besu.ethereum.chain.MutableBlockchain; +import org.hyperledger.besu.ethereum.core.BlockHeader; +import org.hyperledger.besu.ethereum.core.BlockHeaderTestFixture; +import org.hyperledger.besu.ethereum.core.InMemoryKeyValueStorageProvider; +import org.hyperledger.besu.ethereum.storage.StorageProvider; +import org.hyperledger.besu.ethereum.trie.bonsai.storage.BonsaiWorldStateKeyValueStorage; +import org.hyperledger.besu.ethereum.worldstate.DataStorageConfiguration; +import org.hyperledger.besu.ethereum.worldstate.ImmutableDataStorageConfiguration; +import org.hyperledger.besu.metrics.noop.NoOpMetricsSystem; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Optional; + +import org.apache.tuweni.bytes.Bytes; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +@ExtendWith(MockitoExtension.class) +class TrieLogHelperTest { + + private static final StorageProvider storageProvider = new InMemoryKeyValueStorageProvider(); + private static BonsaiWorldStateKeyValueStorage inMemoryWorldState; + + @Mock private MutableBlockchain blockchain; + + @TempDir static Path dataDir; + + Path test; + static BlockHeader blockHeader1; + static BlockHeader blockHeader2; + static BlockHeader blockHeader3; + static BlockHeader blockHeader4; + static BlockHeader blockHeader5; + + @BeforeAll + public static void setup() throws IOException { + + blockHeader1 = new BlockHeaderTestFixture().number(1).buildHeader(); + blockHeader2 = new BlockHeaderTestFixture().number(2).buildHeader(); + blockHeader3 = new BlockHeaderTestFixture().number(3).buildHeader(); + blockHeader4 = new BlockHeaderTestFixture().number(4).buildHeader(); + blockHeader5 = new BlockHeaderTestFixture().number(5).buildHeader(); + + inMemoryWorldState = + new BonsaiWorldStateKeyValueStorage(storageProvider, new NoOpMetricsSystem()); + + var updater = inMemoryWorldState.updater(); + updater + .getTrieLogStorageTransaction() + .put(blockHeader1.getHash().toArrayUnsafe(), Bytes.fromHexString("0x01").toArrayUnsafe()); + updater + .getTrieLogStorageTransaction() + .put(blockHeader2.getHash().toArrayUnsafe(), Bytes.fromHexString("0x02").toArrayUnsafe()); + updater + .getTrieLogStorageTransaction() + .put(blockHeader3.getHash().toArrayUnsafe(), Bytes.fromHexString("0x03").toArrayUnsafe()); + updater + .getTrieLogStorageTransaction() + .put(blockHeader4.getHash().toArrayUnsafe(), Bytes.fromHexString("0x04").toArrayUnsafe()); + updater + .getTrieLogStorageTransaction() + .put(blockHeader5.getHash().toArrayUnsafe(), Bytes.fromHexString("0x05").toArrayUnsafe()); + updater.getTrieLogStorageTransaction().commit(); + } + + @BeforeEach + void createDirectory() throws IOException { + Files.createDirectories(dataDir.resolve("database")); + } + + @AfterEach + void deleteDirectory() throws IOException { + Files.deleteIfExists(dataDir.resolve("database")); + } + + void mockBlockchainBase() { + when(blockchain.getChainHeadBlockNumber()).thenReturn(5L); + when(blockchain.getFinalized()).thenReturn(Optional.of(blockHeader3.getBlockHash())); + when(blockchain.getBlockHeader(any(Hash.class))).thenReturn(Optional.of(blockHeader3)); + } + + @Test + public void prune() { + + DataStorageConfiguration dataStorageConfiguration = + ImmutableDataStorageConfiguration.builder() + .dataStorageFormat(BONSAI) + .bonsaiMaxLayersToLoad(2L) + .unstable( + ImmutableDataStorageConfiguration.Unstable.builder() + .bonsaiTrieLogRetentionThreshold(3) + .build() + .withBonsaiTrieLogRetentionThreshold(3)) + .build(); + + mockBlockchainBase(); + when(blockchain.getBlockHeader(5)).thenReturn(Optional.of(blockHeader5)); + when(blockchain.getBlockHeader(4)).thenReturn(Optional.of(blockHeader4)); + when(blockchain.getBlockHeader(3)).thenReturn(Optional.of(blockHeader3)); + + // assert trie logs that will be pruned exist before prune call + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader1.getHash()).get(), + Bytes.fromHexString("0x01").toArrayUnsafe()); + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader2.getHash()).get(), + Bytes.fromHexString("0x02").toArrayUnsafe()); + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader3.getHash()).get(), + Bytes.fromHexString("0x03").toArrayUnsafe()); + + TrieLogHelper.prune(dataStorageConfiguration, inMemoryWorldState, blockchain, dataDir); + + // assert pruned trie logs are not in the DB + assertEquals(inMemoryWorldState.getTrieLog(blockHeader1.getHash()), Optional.empty()); + assertEquals(inMemoryWorldState.getTrieLog(blockHeader2.getHash()), Optional.empty()); + + // assert retained trie logs are in the DB + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader3.getHash()).get(), + Bytes.fromHexString("0x03").toArrayUnsafe()); + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader4.getHash()).get(), + Bytes.fromHexString("0x04").toArrayUnsafe()); + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader5.getHash()).get(), + Bytes.fromHexString("0x05").toArrayUnsafe()); + } + + @Test + public void cantPruneIfNoFinalizedIsFound() { + DataStorageConfiguration dataStorageConfiguration = + ImmutableDataStorageConfiguration.builder() + .dataStorageFormat(BONSAI) + .bonsaiMaxLayersToLoad(2L) + .unstable( + ImmutableDataStorageConfiguration.Unstable.builder() + .bonsaiTrieLogRetentionThreshold(2) + .build() + .withBonsaiTrieLogRetentionThreshold(2)) + .build(); + + when(blockchain.getChainHeadBlockNumber()).thenReturn(5L); + when(blockchain.getFinalized()).thenReturn(Optional.empty()); + + assertThrows( + RuntimeException.class, + () -> + TrieLogHelper.prune(dataStorageConfiguration, inMemoryWorldState, blockchain, dataDir)); + } + + @Test + public void cantPruneIfUserRetainsMoreLayerThanExistingChainLength() { + DataStorageConfiguration dataStorageConfiguration = + ImmutableDataStorageConfiguration.builder() + .dataStorageFormat(BONSAI) + .bonsaiMaxLayersToLoad(2L) + .unstable( + ImmutableDataStorageConfiguration.Unstable.builder() + .bonsaiTrieLogRetentionThreshold(10) + .build() + .withBonsaiTrieLogRetentionThreshold(10)) + .build(); + + when(blockchain.getChainHeadBlockNumber()).thenReturn(5L); + + assertThrows( + IllegalArgumentException.class, + () -> + TrieLogHelper.prune(dataStorageConfiguration, inMemoryWorldState, blockchain, dataDir)); + } + + @Test + public void cantPruneIfUserRequiredFurtherThanFinalized() { + + DataStorageConfiguration dataStorageConfiguration = + ImmutableDataStorageConfiguration.builder() + .dataStorageFormat(BONSAI) + .bonsaiMaxLayersToLoad(2L) + .unstable( + ImmutableDataStorageConfiguration.Unstable.builder() + .bonsaiTrieLogRetentionThreshold(2) + .build() + .withBonsaiTrieLogRetentionThreshold(2)) + .build(); + + mockBlockchainBase(); + + assertThrows( + IllegalArgumentException.class, + () -> + TrieLogHelper.prune(dataStorageConfiguration, inMemoryWorldState, blockchain, dataDir)); + } + + @Test + public void exceptionWhileSavingFileStopsPruneProcess() throws IOException { + Files.delete(dataDir.resolve("database")); + + DataStorageConfiguration dataStorageConfiguration = + ImmutableDataStorageConfiguration.builder() + .dataStorageFormat(BONSAI) + .bonsaiMaxLayersToLoad(2L) + .unstable( + ImmutableDataStorageConfiguration.Unstable.builder() + .bonsaiTrieLogRetentionThreshold(2) + .build() + .withBonsaiTrieLogRetentionThreshold(2)) + .build(); + + assertThrows( + RuntimeException.class, + () -> + TrieLogHelper.prune(dataStorageConfiguration, inMemoryWorldState, blockchain, dataDir)); + + // assert all trie logs are still in the DB + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader1.getHash()).get(), + Bytes.fromHexString("0x01").toArrayUnsafe()); + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader2.getHash()).get(), + Bytes.fromHexString("0x02").toArrayUnsafe()); + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader3.getHash()).get(), + Bytes.fromHexString("0x03").toArrayUnsafe()); + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader4.getHash()).get(), + Bytes.fromHexString("0x04").toArrayUnsafe()); + assertArrayEquals( + inMemoryWorldState.getTrieLog(blockHeader5.getHash()).get(), + Bytes.fromHexString("0x05").toArrayUnsafe()); + } +} diff --git a/besu/src/test/java/org/hyperledger/besu/services/BesuEventsImplTest.java b/besu/src/test/java/org/hyperledger/besu/services/BesuEventsImplTest.java index 25db1cb9b49..93cf830faf1 100644 --- a/besu/src/test/java/org/hyperledger/besu/services/BesuEventsImplTest.java +++ b/besu/src/test/java/org/hyperledger/besu/services/BesuEventsImplTest.java @@ -41,7 +41,6 @@ import org.hyperledger.besu.ethereum.eth.manager.EthMessages; import org.hyperledger.besu.ethereum.eth.manager.EthPeer; import org.hyperledger.besu.ethereum.eth.manager.EthPeers; -import org.hyperledger.besu.ethereum.eth.manager.EthScheduler; import org.hyperledger.besu.ethereum.eth.sync.BlockBroadcaster; import org.hyperledger.besu.ethereum.eth.sync.state.SyncState; import org.hyperledger.besu.ethereum.eth.transactions.BlobCache; @@ -64,6 +63,7 @@ import org.hyperledger.besu.plugin.data.PropagatedBlockContext; import org.hyperledger.besu.plugin.data.SyncStatus; import org.hyperledger.besu.services.kvstore.InMemoryKeyValueStorage; +import org.hyperledger.besu.testutil.DeterministicEthScheduler; import org.hyperledger.besu.testutil.TestClock; import java.math.BigInteger; @@ -100,7 +100,6 @@ public class BesuEventsImplTest { @Mock private EthPeers mockEthPeers; @Mock private EthContext mockEthContext; @Mock private EthMessages mockEthMessages; - @Mock private EthScheduler mockEthScheduler; @Mock(answer = Answers.RETURNS_DEEP_STUBS) private TransactionValidatorFactory mockTransactionValidatorFactory; @@ -128,7 +127,7 @@ public void setUp() { when(mockEthContext.getEthMessages()).thenReturn(mockEthMessages); when(mockEthContext.getEthPeers()).thenReturn(mockEthPeers); - when(mockEthContext.getScheduler()).thenReturn(mockEthScheduler); + when(mockEthContext.getScheduler()).thenReturn(new DeterministicEthScheduler()); lenient().when(mockEthPeers.streamAvailablePeers()).thenAnswer(z -> Stream.empty()); when(mockProtocolContext.getBlockchain()).thenReturn(blockchain); lenient().when(mockProtocolContext.getWorldStateArchive()).thenReturn(mockWorldStateArchive); diff --git a/build.gradle b/build.gradle index e30bd9d2ed5..76b6dff3ee6 100644 --- a/build.gradle +++ b/build.gradle @@ -932,6 +932,7 @@ task checkSpdxHeader(type: CheckSpdxHeader) { "(.*${sep}.gradle${sep}.*)", "(.*${sep}.idea${sep}.*)", "(.*${sep}out${sep}.*)", + "(.*${sep}bin${sep}.*)", "(.*${sep}build${sep}.*)", "(.*${sep}src${sep}[^${sep}]+${sep}generated${sep}.*)" ].join("|") diff --git a/docker/openjdk-latest/Dockerfile b/docker/openjdk-latest/Dockerfile index 8202718781e..472223cd108 100644 --- a/docker/openjdk-latest/Dockerfile +++ b/docker/openjdk-latest/Dockerfile @@ -6,7 +6,10 @@ RUN apt-get update && \ apt-get install --no-install-recommends -q --assume-yes openjdk-21-jre-headless=21* libjemalloc-dev=5.* adduser=3* && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* && \ - adduser --disabled-password --gecos "" --home /opt/besu besu && \ + # Ubuntu 23.10 comes with an "ubuntu" user with uid 1000. We need 1000 for besu. + userdel ubuntu 2>/dev/null || true && rm -rf /home/ubuntu && \ + # Ensure we use a stable UID for besu, as file permissions are tied to UIDs. + adduser --uid 1000 --disabled-password --gecos "" --home /opt/besu besu && \ chown besu:besu /opt/besu && \ chmod 0755 /opt/besu diff --git a/docs/trace_rpc_apis.md b/docs/trace_rpc_apis.md index 116c7ea226b..e8db0bcbc5c 100644 --- a/docs/trace_rpc_apis.md +++ b/docs/trace_rpc_apis.md @@ -34,6 +34,6 @@ Besu reports only the actual cost of the precompiled contract call in the ### Out of Gas -Besu reports the operation that causes out fo gas exceptions, including +Besu reports the operation that causes out of gas exceptions, including calculated gas cost. The operation is not executed so no `ex` values are reported. diff --git a/ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/frontier/EthGetFilterChangesIntegrationTest.java b/ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/frontier/EthGetFilterChangesIntegrationTest.java index cff37a0392b..1864246401e 100644 --- a/ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/frontier/EthGetFilterChangesIntegrationTest.java +++ b/ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/frontier/EthGetFilterChangesIntegrationTest.java @@ -18,6 +18,7 @@ import static java.util.Collections.emptyList; import static java.util.stream.Collectors.toList; import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.RETURNS_DEEP_STUBS; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; @@ -107,7 +108,7 @@ public void setUp() { blockchain::getChainHeadHeader); final ProtocolContext protocolContext = executionContext.getProtocolContext(); - EthContext ethContext = mock(EthContext.class); + EthContext ethContext = mock(EthContext.class, RETURNS_DEEP_STUBS); EthPeers ethPeers = mock(EthPeers.class); when(ethContext.getEthPeers()).thenReturn(ethPeers); diff --git a/ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/london/EthGetFilterChangesIntegrationTest.java b/ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/london/EthGetFilterChangesIntegrationTest.java index 0026c56ad84..49a2fbcd87b 100644 --- a/ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/london/EthGetFilterChangesIntegrationTest.java +++ b/ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/london/EthGetFilterChangesIntegrationTest.java @@ -18,6 +18,7 @@ import static java.util.Collections.emptyList; import static java.util.stream.Collectors.toList; import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.RETURNS_DEEP_STUBS; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; @@ -107,7 +108,7 @@ public void setUp() { blockchain::getChainHeadHeader); final ProtocolContext protocolContext = executionContext.getProtocolContext(); - EthContext ethContext = mock(EthContext.class); + EthContext ethContext = mock(EthContext.class, RETURNS_DEEP_STUBS); EthPeers ethPeers = mock(EthPeers.class); when(ethContext.getEthPeers()).thenReturn(ethPeers); diff --git a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/Transaction.java b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/Transaction.java index 79261159498..529661d5098 100644 --- a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/Transaction.java +++ b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/Transaction.java @@ -1007,24 +1007,33 @@ public Transaction detachedCopy() { withCommitments -> blobsWithCommitmentsDetachedCopy(withCommitments, detachedVersionedHashes.get())); - return new Transaction( - true, - transactionType, - nonce, - gasPrice, - maxPriorityFeePerGas, - maxFeePerGas, - maxFeePerBlobGas, - gasLimit, - detachedTo, - value, - signature, - payload.copy(), - detachedAccessList, - sender, - chainId, - detachedVersionedHashes, - detachedBlobsWithCommitments); + final var copiedTx = + new Transaction( + true, + transactionType, + nonce, + gasPrice, + maxPriorityFeePerGas, + maxFeePerGas, + maxFeePerBlobGas, + gasLimit, + detachedTo, + value, + signature, + payload.copy(), + detachedAccessList, + sender, + chainId, + detachedVersionedHashes, + detachedBlobsWithCommitments); + + // copy also the computed fields, to avoid to recompute them + copiedTx.sender = this.sender; + copiedTx.hash = this.hash; + copiedTx.hashNoSignature = this.hashNoSignature; + copiedTx.size = this.size; + + return copiedTx; } private AccessListEntry accessListDetachedCopy(final AccessListEntry accessListEntry) { diff --git a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/BlockHeaderValidator.java b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/BlockHeaderValidator.java index 56b1d4f0dbf..f9e33f267b2 100644 --- a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/BlockHeaderValidator.java +++ b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/BlockHeaderValidator.java @@ -84,7 +84,10 @@ private boolean applyRules( rule -> { boolean worked = rule.validate(header, parent, protocolContext); if (!worked) { - LOG.debug("{} rule failed", rule.innerRuleClass().getCanonicalName()); + String canonicalName = rule.innerRuleClass().getCanonicalName(); + LOG.debug( + "{} rule failed", + canonicalName == null ? rule.innerRuleClass().getName() : canonicalName); } return worked; }); diff --git a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/storage/BonsaiWorldStateKeyValueStorage.java b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/storage/BonsaiWorldStateKeyValueStorage.java index 54089c7e212..97384fd6c12 100644 --- a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/storage/BonsaiWorldStateKeyValueStorage.java +++ b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/storage/BonsaiWorldStateKeyValueStorage.java @@ -204,7 +204,7 @@ public Optional getTrieLog(final Hash blockHash) { return trieLogStorage.get(blockHash.toArrayUnsafe()); } - public Stream streamTrieLogKeys(final int limit) { + public Stream streamTrieLogKeys(final long limit) { return trieLogStorage.streamKeys().limit(limit); } diff --git a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/trielog/TrieLogPruner.java b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/trielog/TrieLogPruner.java index 747a82e1621..b1bf75818e9 100644 --- a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/trielog/TrieLogPruner.java +++ b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/trielog/TrieLogPruner.java @@ -61,33 +61,37 @@ public TrieLogPruner( this.requireFinalizedBlock = requireFinalizedBlock; } - public void initialize() { - preloadQueue(); + public int initialize() { + return preloadQueue(); } - private void preloadQueue() { + private int preloadQueue() { LOG.atInfo() .setMessage("Loading first {} trie logs from database...") .addArgument(loadingLimit) .log(); try (final Stream trieLogKeys = rootWorldStateStorage.streamTrieLogKeys(loadingLimit)) { final AtomicLong count = new AtomicLong(); + final AtomicLong orphansPruned = new AtomicLong(); trieLogKeys.forEach( blockHashAsBytes -> { final Hash blockHash = Hash.wrap(Bytes32.wrap(blockHashAsBytes)); final Optional header = blockchain.getBlockHeader(blockHash); if (header.isPresent()) { - trieLogBlocksAndForksByDescendingBlockNumber.put(header.get().getNumber(), blockHash); + addToPruneQueue(header.get().getNumber(), blockHash); count.getAndIncrement(); } else { // prune orphaned blocks (sometimes created during block production) rootWorldStateStorage.pruneTrieLog(blockHash); + orphansPruned.getAndIncrement(); } }); + LOG.atDebug().log("Pruned {} orphaned trie logs from database...", orphansPruned.intValue()); LOG.atInfo().log("Loaded {} trie logs from database", count); - pruneFromQueue(); + return pruneFromQueue() + orphansPruned.intValue(); } catch (Exception e) { LOG.error("Error loading trie logs from database, nothing pruned", e); + return 0; } } @@ -176,8 +180,9 @@ private NoOpTrieLogPruner( } @Override - public void initialize() { + public int initialize() { // no-op + return -1; } @Override diff --git a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/worldview/BonsaiWorldState.java b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/worldview/BonsaiWorldState.java index 129e3f291f0..548a3a1c414 100644 --- a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/worldview/BonsaiWorldState.java +++ b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/worldview/BonsaiWorldState.java @@ -340,6 +340,7 @@ private void updateAccountStorageState( private void clearStorage( final Optional maybeStateUpdater, final BonsaiWorldStateUpdateAccumulator worldStateUpdater) { + for (final Address address : worldStateUpdater.getStorageToClear()) { // because we are clearing persisted values we need the account root as persisted final BonsaiAccount oldAccount = diff --git a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthPeers.java b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthPeers.java index 1c7da3fd2b6..e28b0a24ad9 100644 --- a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthPeers.java +++ b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthPeers.java @@ -197,7 +197,7 @@ private boolean registerDisconnect( peer.handleDisconnect(); abortPendingRequestsAssignedToDisconnectedPeers(); if (peer.getReputation().getScore() > USEFULL_PEER_SCORE_THRESHOLD) { - LOG.debug("Disonnected USEFULL peer {}", peer); + LOG.debug("Disconnected USEFULL peer {}", peer); } else { LOG.debug("Disconnected EthPeer {}", peer.getShortNodeId()); } diff --git a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthScheduler.java b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthScheduler.java index 43f8d6a4c35..61a01bf54f3 100644 --- a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthScheduler.java +++ b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthScheduler.java @@ -23,9 +23,11 @@ import java.time.Duration; import java.util.Collection; +import java.util.Queue; import java.util.concurrent.CancellationException; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentLinkedDeque; +import java.util.concurrent.ConcurrentLinkedQueue; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Future; @@ -34,6 +36,8 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.locks.ReentrantLock; +import java.util.function.Consumer; import java.util.function.Function; import java.util.function.Supplier; @@ -295,4 +299,49 @@ public void failAfterTimeout(final CompletableFuture promise, final Durat delay, unit); } + + public OrderedProcessor createOrderedProcessor(final Consumer processor) { + return new OrderedProcessor<>(processor); + } + + /** + * This class is a way to execute a set of tasks, one by one, in a strict order, without blocking + * the caller in case there are still previous tasks queued + * + * @param the class of item to be processed + */ + public class OrderedProcessor { + private final Queue blockAddedQueue = new ConcurrentLinkedQueue<>(); + private final ReentrantLock blockAddedLock = new ReentrantLock(); + private final Consumer processor; + + private OrderedProcessor(final Consumer processor) { + this.processor = processor; + } + + public void submit(final ITEM item) { + // add the item to the processing queue + blockAddedQueue.add(item); + + if (blockAddedLock.hasQueuedThreads()) { + // another thread is already waiting to process the queue with our item, there is no need to + // schedule another thread + LOG.trace( + "Block added event queue is already being processed and an already queued thread is present, nothing to do"); + } else { + servicesExecutor.submit( + () -> { + blockAddedLock.lock(); + try { + // now that we have the lock, process as many items as possible + for (ITEM i = blockAddedQueue.poll(); i != null; i = blockAddedQueue.poll()) { + processor.accept(i); + } + } finally { + blockAddedLock.unlock(); + } + }); + } + } + } } diff --git a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPool.java b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPool.java index 5425cf1c383..a3d164105d0 100644 --- a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPool.java +++ b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPool.java @@ -31,6 +31,7 @@ import org.hyperledger.besu.ethereum.core.Transaction; import org.hyperledger.besu.ethereum.eth.manager.EthContext; import org.hyperledger.besu.ethereum.eth.manager.EthPeer; +import org.hyperledger.besu.ethereum.eth.manager.EthScheduler; import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule; import org.hyperledger.besu.ethereum.mainnet.TransactionValidationParams; import org.hyperledger.besu.ethereum.mainnet.TransactionValidator; @@ -61,11 +62,9 @@ import java.util.Map; import java.util.Optional; import java.util.OptionalLong; -import java.util.Queue; import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.ConcurrentLinkedQueue; import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; @@ -107,8 +106,7 @@ public class TransactionPool implements BlockAddedObserver { private volatile OptionalLong subscribeConnectId = OptionalLong.empty(); private final SaveRestoreManager saveRestoreManager = new SaveRestoreManager(); private final Set
localSenders = ConcurrentHashMap.newKeySet(); - private final Lock blockAddedLock = new ReentrantLock(); - private final Queue blockAddedQueue = new ConcurrentLinkedQueue<>(); + private final EthScheduler.OrderedProcessor blockAddedEventOrderedProcessor; public TransactionPool( final Supplier pendingTransactionsSupplier, @@ -130,6 +128,8 @@ public TransactionPool( pluginTransactionValidatorFactory == null ? null : pluginTransactionValidatorFactory.create(); + this.blockAddedEventOrderedProcessor = + ethContext.getScheduler().createOrderedProcessor(this::processBlockAddedEvent); initLogForReplay(); } @@ -322,58 +322,29 @@ public void unsubscribeDroppedTransactions(final long id) { @Override public void onBlockAdded(final BlockAddedEvent event) { if (isPoolEnabled.get()) { - final long started = System.currentTimeMillis(); if (event.getEventType().equals(BlockAddedEvent.EventType.HEAD_ADVANCED) || event.getEventType().equals(BlockAddedEvent.EventType.CHAIN_REORG)) { - // add the event to the processing queue - blockAddedQueue.add(event); - - // we want to process the added block asynchronously, - // but at the same time we must ensure that blocks are processed in order one at time - ethContext - .getScheduler() - .scheduleServiceTask( - () -> { - while (!blockAddedQueue.isEmpty()) { - if (blockAddedLock.tryLock()) { - // no other thread is processing the queue, so start processing it - try { - BlockAddedEvent e = blockAddedQueue.poll(); - // check again since another thread could have stolen our task - if (e != null) { - pendingTransactions.manageBlockAdded( - e.getBlock().getHeader(), - e.getAddedTransactions(), - e.getRemovedTransactions(), - protocolSchedule - .getByBlockHeader(e.getBlock().getHeader()) - .getFeeMarket()); - reAddTransactions(e.getRemovedTransactions()); - LOG.atTrace() - .setMessage("Block added event {} processed in {}ms") - .addArgument(e) - .addArgument(() -> System.currentTimeMillis() - started) - .log(); - } - } finally { - blockAddedLock.unlock(); - } - } else { - try { - // wait a bit before retrying - Thread.sleep(100); - } catch (InterruptedException e) { - throw new RuntimeException(e); - } - } - } - return null; - }); + blockAddedEventOrderedProcessor.submit(event); } } } + private void processBlockAddedEvent(final BlockAddedEvent e) { + final long started = System.currentTimeMillis(); + pendingTransactions.manageBlockAdded( + e.getBlock().getHeader(), + e.getAddedTransactions(), + e.getRemovedTransactions(), + protocolSchedule.getByBlockHeader(e.getBlock().getHeader()).getFeeMarket()); + reAddTransactions(e.getRemovedTransactions()); + LOG.atTrace() + .setMessage("Block added event {} processed in {}ms") + .addArgument(e) + .addArgument(() -> System.currentTimeMillis() - started) + .log(); + } + private void reAddTransactions(final List reAddTransactions) { if (!reAddTransactions.isEmpty()) { // if adding a blob tx, and it is missing its blob, is a re-org and we should restore the blob diff --git a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolFactory.java b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolFactory.java index 97c2a1538fa..253d37a4758 100644 --- a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolFactory.java +++ b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolFactory.java @@ -156,24 +156,62 @@ static TransactionPool createTransactionPool( @Override public void onInitialSyncCompleted() { LOG.info("Enabling transaction handling following initial sync"); - transactionTracker.reset(); - transactionPool.setEnabled(); - transactionsMessageHandler.setEnabled(); - pooledTransactionsMessageHandler.setEnabled(); + enableTransactionHandling( + transactionTracker, + transactionPool, + transactionsMessageHandler, + pooledTransactionsMessageHandler); } @Override public void onInitialSyncRestart() { LOG.info("Disabling transaction handling during re-sync"); - pooledTransactionsMessageHandler.setDisabled(); - transactionsMessageHandler.setDisabled(); - transactionPool.setDisabled(); + disableTransactionHandling( + transactionPool, transactionsMessageHandler, pooledTransactionsMessageHandler); + } + }); + + syncState.subscribeInSync( + isInSync -> { + if (isInSync != transactionPool.isEnabled()) { + if (isInSync) { + LOG.info("Node is in sync, enabling transaction handling"); + enableTransactionHandling( + transactionTracker, + transactionPool, + transactionsMessageHandler, + pooledTransactionsMessageHandler); + } else { + LOG.info("Node out of sync, disabling transaction handling"); + disableTransactionHandling( + transactionPool, transactionsMessageHandler, pooledTransactionsMessageHandler); + } } }); return transactionPool; } + private static void enableTransactionHandling( + final PeerTransactionTracker transactionTracker, + final TransactionPool transactionPool, + final TransactionsMessageHandler transactionsMessageHandler, + final NewPooledTransactionHashesMessageHandler pooledTransactionsMessageHandler) { + transactionTracker.reset(); + transactionPool.setEnabled(); + transactionsMessageHandler.setEnabled(); + pooledTransactionsMessageHandler.setEnabled(); + } + + private static void disableTransactionHandling( + final TransactionPool transactionPool, + final TransactionsMessageHandler transactionsMessageHandler, + final NewPooledTransactionHashesMessageHandler pooledTransactionsMessageHandler) { + transactionPool.setDisabled(); + transactionsMessageHandler.setDisabled(); + pooledTransactionsMessageHandler.setDisabled(); + } + private static void subscribeTransactionHandlers( final ProtocolContext protocolContext, final EthContext ethContext, diff --git a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolMetrics.java b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolMetrics.java index 9ace1819cc1..20657304f0f 100644 --- a/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolMetrics.java +++ b/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolMetrics.java @@ -16,6 +16,7 @@ import org.hyperledger.besu.ethereum.transaction.TransactionInvalidReason; import org.hyperledger.besu.metrics.BesuMetricCategory; +import org.hyperledger.besu.metrics.ReplaceableDoubleSupplier; import org.hyperledger.besu.metrics.RunnableCounter; import org.hyperledger.besu.plugin.services.MetricsSystem; import org.hyperledger.besu.plugin.services.metrics.Counter; @@ -46,6 +47,9 @@ public class TransactionPoolMetrics { private final LabelledMetric expiredMessagesCounter; private final Map expiredMessagesRunnableCounters = new HashMap<>(); private final LabelledMetric alreadySeenTransactionsCounter; + private final Map spaceUsedSuppliers = new HashMap<>(); + private final Map transactionCountSuppliers = new HashMap<>(); + private final Map uniqueSendersSuppliers = new HashMap<>(); public TransactionPoolMetrics(final MetricsSystem metricsSystem) { this.metricsSystem = metricsSystem; @@ -120,17 +124,44 @@ public MetricsSystem getMetricsSystem() { } public void initSpaceUsed(final DoubleSupplier spaceUsedSupplier, final String layer) { - spaceUsed.labels(spaceUsedSupplier, layer); + spaceUsedSuppliers.compute( + layer, + (unused, existingSupplier) -> { + if (existingSupplier == null) { + final var newSupplier = new ReplaceableDoubleSupplier(spaceUsedSupplier); + spaceUsed.labels(newSupplier, layer); + return newSupplier; + } + return existingSupplier.replaceDoubleSupplier(spaceUsedSupplier); + }); } public void initTransactionCount( final DoubleSupplier transactionCountSupplier, final String layer) { - transactionCount.labels(transactionCountSupplier, layer); + transactionCountSuppliers.compute( + layer, + (unused, existingSupplier) -> { + if (existingSupplier == null) { + final var newSupplier = new ReplaceableDoubleSupplier(transactionCountSupplier); + transactionCount.labels(newSupplier, layer); + return newSupplier; + } + return existingSupplier.replaceDoubleSupplier(transactionCountSupplier); + }); } public void initUniqueSenderCount( final DoubleSupplier uniqueSenderCountSupplier, final String layer) { - uniqueSenderCount.labels(uniqueSenderCountSupplier, layer); + uniqueSendersSuppliers.compute( + layer, + (unused, existingSupplier) -> { + if (existingSupplier == null) { + final var newSupplier = new ReplaceableDoubleSupplier(uniqueSenderCountSupplier); + uniqueSenderCount.labels(newSupplier, layer); + return newSupplier; + } + return existingSupplier.replaceDoubleSupplier(uniqueSenderCountSupplier); + }); } public void initExpiredMessagesCounter(final String message) { diff --git a/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/EthSchedulerTest.java b/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/EthSchedulerTest.java index d09f02c6b76..e7797576d64 100644 --- a/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/EthSchedulerTest.java +++ b/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/EthSchedulerTest.java @@ -21,16 +21,24 @@ import static org.mockito.Mockito.times; import static org.mockito.Mockito.verify; +import org.hyperledger.besu.metrics.noop.NoOpMetricsSystem; import org.hyperledger.besu.testutil.DeterministicEthScheduler; import org.hyperledger.besu.testutil.MockExecutorService; import org.hyperledger.besu.testutil.MockScheduledExecutor; import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.Random; import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.Future; import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.function.Consumer; +import java.util.stream.IntStream; +import org.awaitility.Awaitility; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; @@ -209,4 +217,43 @@ public void timeout_cancelsTaskWhenResultIsCancelled() { assertThat(task.isFailed()).isTrue(); assertThat(task.isCancelled()).isTrue(); } + + @Test + public void itemsSubmittedToOrderedProcessorAreProcessedInOrder() throws InterruptedException { + final int numOfItems = 100; + final Random random = new Random(); + final EthScheduler realEthScheduler = new EthScheduler(1, 1, 1, new NoOpMetricsSystem()); + + final List processedStrings = new CopyOnWriteArrayList<>(); + + final Consumer stringProcessor = + s -> { + processedStrings.add(s); + try { + Thread.sleep(random.nextInt(20)); + } catch (InterruptedException e) { + throw new RuntimeException(e); + } + }; + + final var orderProcessor = realEthScheduler.createOrderedProcessor(stringProcessor); + IntStream.range(0, numOfItems) + .mapToObj(String::valueOf) + .forEach( + s -> { + orderProcessor.submit(s); + try { + Thread.sleep(random.nextInt(20)); + } catch (InterruptedException e) { + throw new RuntimeException(e); + } + }); + + final List expectedStrings = new ArrayList<>(numOfItems); + IntStream.range(0, numOfItems).mapToObj(String::valueOf).forEach(expectedStrings::add); + + Awaitility.await().until(() -> processedStrings.size() == numOfItems); + + assertThat(processedStrings).containsExactlyElementsOf(expectedStrings); + } } diff --git a/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/ethtaskutils/AbstractMessageTaskTest.java b/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/ethtaskutils/AbstractMessageTaskTest.java index c5ca2f3d8a7..1538ecc7278 100644 --- a/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/ethtaskutils/AbstractMessageTaskTest.java +++ b/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/ethtaskutils/AbstractMessageTaskTest.java @@ -166,7 +166,7 @@ public void completesWhenPeersAreResponsive() { RespondingEthPeer.blockchainResponder( blockchain, protocolContext.getWorldStateArchive(), transactionPool); final RespondingEthPeer respondingPeer = - EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 1000); + EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 32); // Setup data to be requested and expected response final T requestedData = generateDataToBeRequested(); @@ -190,7 +190,7 @@ public void completesWhenPeersAreResponsive() { @Test public void doesNotCompleteWhenPeersDoNotRespond() { // Setup a unresponsive peer - EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 1000); + EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 32); // Setup data to be requested final T requestedData = generateDataToBeRequested(); @@ -209,7 +209,7 @@ public void doesNotCompleteWhenPeersDoNotRespond() { @Test public void cancel() { // Setup a unresponsive peer - EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 1000); + EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 32); // Setup data to be requested final T requestedData = generateDataToBeRequested(); diff --git a/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/ethtaskutils/PeerMessageTaskTest.java b/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/ethtaskutils/PeerMessageTaskTest.java index 1405fbc24ac..0ed6569387b 100644 --- a/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/ethtaskutils/PeerMessageTaskTest.java +++ b/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/manager/ethtaskutils/PeerMessageTaskTest.java @@ -58,7 +58,7 @@ public void completesWhenPeerReturnsPartialResult() { protocolSchedule, 0.5f); final RespondingEthPeer respondingEthPeer = - EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 1000); + EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 32); // Execute task and wait for response final AtomicReference actualResult = new AtomicReference<>(); @@ -109,7 +109,7 @@ public void completesWhenPeersSendEmptyResponses() { // Setup a unresponsive peer final RespondingEthPeer.Responder responder = RespondingEthPeer.emptyResponder(); final RespondingEthPeer respondingEthPeer = - EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 1000); + EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 32); // Setup data to be requested final T requestedData = generateDataToBeRequested(); @@ -129,7 +129,7 @@ public void recordsTimeoutAgainstPeerWhenTaskTimesOut() { peersDoTimeout.set(true); // Setup a unresponsive peer final RespondingEthPeer respondingEthPeer = - EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 1000); + EthProtocolManagerTestUtil.createPeer(ethProtocolManager, 32); // Setup data to be requested final T requestedData = generateDataToBeRequested(); diff --git a/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/transactions/AbstractTransactionPoolTest.java b/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/transactions/AbstractTransactionPoolTest.java index a87ca36bfb8..c34b4f0906d 100644 --- a/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/transactions/AbstractTransactionPoolTest.java +++ b/ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/transactions/AbstractTransactionPoolTest.java @@ -32,7 +32,6 @@ import static org.mockito.ArgumentMatchers.argThat; import static org.mockito.ArgumentMatchers.eq; import static org.mockito.ArgumentMatchers.nullable; -import static org.mockito.Mockito.doAnswer; import static org.mockito.Mockito.doNothing; import static org.mockito.Mockito.doReturn; import static org.mockito.Mockito.mock; @@ -100,7 +99,6 @@ import java.util.function.BiFunction; import java.util.function.Consumer; import java.util.function.Function; -import java.util.function.Supplier; import java.util.stream.Stream; import org.junit.jupiter.api.BeforeEach; @@ -233,12 +231,9 @@ public void setUp() { ethProtocolManager = EthProtocolManagerTestUtil.create(); ethContext = spy(ethProtocolManager.ethContext()); - final EthScheduler ethScheduler = mock(EthScheduler.class); + final EthScheduler ethScheduler = spy(ethContext.getScheduler()); syncTaskCapture = ArgumentCaptor.forClass(Runnable.class); doNothing().when(ethScheduler).scheduleSyncWorkerTask(syncTaskCapture.capture()); - doAnswer(invocation -> ((Supplier) invocation.getArguments()[0]).get()) - .when(ethScheduler) - .scheduleServiceTask(any(Supplier.class)); doReturn(ethScheduler).when(ethContext).getScheduler(); peerTransactionTracker = new PeerTransactionTracker(); diff --git a/evm/src/main/java/org/hyperledger/besu/evm/fluent/EVMExecutor.java b/evm/src/main/java/org/hyperledger/besu/evm/fluent/EVMExecutor.java index 25ad1a95227..6907257f98b 100644 --- a/evm/src/main/java/org/hyperledger/besu/evm/fluent/EVMExecutor.java +++ b/evm/src/main/java/org/hyperledger/besu/evm/fluent/EVMExecutor.java @@ -222,7 +222,7 @@ public static EVMExecutor tangerineWhistle(final EvmConfiguration evmConfigurati final EVMExecutor executor = new EVMExecutor(MainnetEVMs.tangerineWhistle(evmConfiguration)); executor.precompileContractRegistry = MainnetPrecompiledContracts.frontier(executor.evm.getGasCalculator()); - executor.contractValidationRules = List.of(MaxCodeSizeRule.of(0x6000)); + executor.contractValidationRules = List.of(); executor.initialNonce = 0; return executor; } diff --git a/gradle/allowed-licenses.json b/gradle/allowed-licenses.json index a5d4048b224..2fa9d76635c 100644 --- a/gradle/allowed-licenses.json +++ b/gradle/allowed-licenses.json @@ -4,6 +4,10 @@ "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0" }, + { + "moduleLicense": "Apache License 2.0", + "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0" + }, { "moduleLicense": "BSD Zero Clause License", "moduleLicenseUrl": "https://opensource.org/licenses/0BSD" @@ -60,6 +64,10 @@ "moduleLicense": "Eclipse Public License - v 1.0", "moduleVersion": "4.13.2", "moduleName": "junit:junit" + }, + { + "moduleName": "org.jetbrains.kotlin:kotlin-stdlib-common", + "moduleVersion": "1.9.22" } ] } \ No newline at end of file diff --git a/gradle/license-normalizer-bundle.json b/gradle/license-normalizer-bundle.json index 341b7ab5f62..8c0af450fd3 100644 --- a/gradle/license-normalizer-bundle.json +++ b/gradle/license-normalizer-bundle.json @@ -50,9 +50,9 @@ { "bundleName" : "Apache-2.0", "licenseNamePattern" : ".*Apache License,?( Version)? 2.*" }, { "bundleName" : "Apache-2.0", "licenseUrlPattern" : ".*(www\\.)?opensource\\.org/licenses/Apache-2\\.0.*" }, { "bundleName" : "Apache-2.0", "licenseUrlPattern" : ".*www\\.apache\\.org/licenses/LICENSE-2\\.0.*" }, - { "bundleName" : "LGPL-2.1-only", "licenseUrlPattern" : ".*www\\.gnu\\.org/licenses/old-licenses/lgpl-2\\.1\\.html" }, { "bundleName" : "Apache-2.0", "licenseFileContentPattern" : ".*Apache License,?( Version)? 2.*" }, { "bundleName" : "Apache-1.1", "licenseFileContentPattern" : ".*Apache Software License, Version 1\\.1.*" }, + { "bundleName" : "LGPL-2.1-only", "licenseUrlPattern" : ".*www\\.gnu\\.org/licenses/old-licenses/lgpl-2\\.1\\.html" }, { "bundleName" : "CC0-1.0", "licenseNamePattern" : "CC0(( |-)1(\\.0)?)?" }, { "bundleName" : "CC0-1.0", "licenseUrlPattern" : ".*(www\\.)?creativecommons\\.org/publicdomain/zero/1\\.0/" }, { "bundleName" : "CDDL-1.0", "licenseFileContentPattern" : ".*CDDL.*1\\.0" }, diff --git a/gradle/verification-metadata.xml b/gradle/verification-metadata.xml index 8265a2810c4..aa33426e526 100644 --- a/gradle/verification-metadata.xml +++ b/gradle/verification-metadata.xml @@ -78,9 +78,12 @@ - - - + + + + + + @@ -128,14 +131,9 @@ - - - - - - - - + + + @@ -153,11 +151,6 @@ - - - - - @@ -183,9 +176,9 @@ - - - + + + @@ -208,11 +201,6 @@ - - - - - @@ -229,11 +217,6 @@ - - - - - @@ -242,12 +225,12 @@ - - - + + + - - + + @@ -282,12 +265,12 @@ - - - + + + - - + + @@ -319,75 +302,55 @@ - - - - - - - - - - - + + + - - + + - - - + + + - - + + - - - - - - - - - - - - + + - - - + + + - - - + + + - - + + - - + + - - - + + + - - + + - - - - + + - - - + + + @@ -414,33 +377,33 @@ - - - + + + - - + + - - - + + + - - - + + + - - + + - - - + + + - - + + @@ -523,11 +486,6 @@ - - - - - @@ -584,27 +542,17 @@ - - - - - - - - - - - + + + - - - - + + - - - + + + @@ -615,35 +563,6 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -689,14 +608,6 @@ - - - - - - - - @@ -705,15 +616,10 @@ - - - - - - - - + + + @@ -736,6 +642,14 @@ + + + + + + + + @@ -806,54 +720,36 @@ - - - - - - - - - - - - - - - - - - - + + + - - - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + @@ -864,11 +760,6 @@ - - - - - @@ -909,11 +800,6 @@ - - - - - @@ -922,11 +808,6 @@ - - - - - @@ -947,11 +828,6 @@ - - - - - @@ -960,11 +836,6 @@ - - - - - @@ -1072,9 +943,6 @@ - - - @@ -1281,31 +1149,23 @@ - - - - - - - - - - - + + + - - + + - - - + + + - - + + - - + + @@ -1316,12 +1176,15 @@ - - - + + + + + + - - + + @@ -1329,49 +1192,18 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - - - - - - - - - - + + + + + @@ -1446,17 +1278,6 @@ - - - - - - - - - - - @@ -1465,6 +1286,17 @@ + + + + + + + + + + + @@ -1473,19 +1305,6 @@ - - - - - - - - - - - - - @@ -1494,25 +1313,19 @@ - - - - - - - - - - - + + + - - + + + + @@ -1523,12 +1336,15 @@ - - - + + + + + + - - + + @@ -1539,12 +1355,12 @@ - - - + + + - - + + @@ -1556,6 +1372,9 @@ + + + @@ -1584,15 +1403,10 @@ - - - - - - - - + + + @@ -1610,35 +1424,36 @@ - - - - - - - - + + + - - - - - - - - + + + - - - + + + + + + + + + + + + + + @@ -1649,41 +1464,36 @@ - - - - - - + + + - - - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + @@ -1694,155 +1504,140 @@ - - - + + + - - + + - - - - - - + + + - - - - + + - - - + + + - - + + - - - + + + - - + + - - - - - - + + + - - - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - - - - + + + - - - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + @@ -1853,61 +1648,46 @@ - - - - - - + + + - - - - + + - - - - - - + + + - - - - + + - - - + + + - - - + + + - - + + - - - - - - + + + - - - - + + - - - + + + @@ -1920,12 +1700,12 @@ - - - + + + - - + + @@ -1933,12 +1713,12 @@ - - - + + + - - + + @@ -1949,12 +1729,12 @@ - - - + + + - - + + @@ -1965,20 +1745,20 @@ - - - + + + - - + + - - - + + + - - + + @@ -1990,13 +1770,18 @@ - - - + + + + + + + + @@ -2005,49 +1790,44 @@ - - - - - - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + @@ -2058,28 +1838,28 @@ - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + @@ -2087,12 +1867,12 @@ - - - + + + - - + + @@ -2104,13 +1884,18 @@ - - - + + + + + + + + @@ -2119,17 +1904,12 @@ - - - - - - - - + + + - - + + @@ -2137,9 +1917,9 @@ - - - + + + @@ -2147,22 +1927,12 @@ - - - - - - - - - - - - - + + + - - + + @@ -2170,12 +1940,12 @@ - - - + + + - - + + @@ -2186,23 +1956,23 @@ - - - + + + - - + + - - - + + + - - + + - - + + @@ -2226,12 +1996,12 @@ - - - + + + - - + + @@ -2239,95 +2009,83 @@ - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - + + - - + + - - - - + + - - - + + + - - + + - - + + - - - - - - - + + - - - + + + - - + + - - - + + + - - - - - - - + + - - - + + + - - + + - - - + + + - - + + @@ -2365,14 +2123,6 @@ - - - - - - - - @@ -2428,14 +2178,6 @@ - - - - - - - - @@ -2458,14 +2200,6 @@ - - - - - - - - @@ -2477,14 +2211,6 @@ - - - - - - - - @@ -2507,14 +2233,6 @@ - - - - - - - - @@ -2548,14 +2266,6 @@ - - - - - - - - @@ -2567,14 +2277,6 @@ - - - - - - - - @@ -2586,14 +2288,6 @@ - - - - - - - - @@ -2618,14 +2312,6 @@ - - - - - - - - @@ -2634,14 +2320,6 @@ - - - - - - - - @@ -2750,11 +2428,6 @@ - - - - - @@ -2763,11 +2436,6 @@ - - - - - @@ -2784,11 +2452,6 @@ - - - - - @@ -2805,11 +2468,6 @@ - - - - - @@ -2836,9 +2494,7 @@ - - - + @@ -3003,16 +2659,6 @@ - - - - - - - - - - @@ -3029,11 +2675,6 @@ - - - - - @@ -3042,11 +2683,6 @@ - - - - - @@ -3057,11 +2693,6 @@ - - - - - @@ -3080,11 +2711,6 @@ - - - - - @@ -3119,11 +2745,6 @@ - - - - - @@ -3140,11 +2761,6 @@ - - - - - @@ -3153,11 +2769,6 @@ - - - - - @@ -3171,19 +2782,14 @@ - - - - - - - - + + + - - - + + + @@ -3210,14 +2816,6 @@ - - - - - - - - @@ -3268,12 +2866,12 @@ - - - + + + - - + + @@ -3289,9 +2887,9 @@ - - - + + + @@ -3304,38 +2902,33 @@ - - - - - - - - + + + - - - + + + - - + + - - - + + + - - + + - - - + + + @@ -3343,11 +2936,6 @@ - - - - - @@ -3364,36 +2952,20 @@ - - - - - - - - - - - + + + - - + + - - - + + + - - - - - - - - - - + + @@ -3446,11 +3018,6 @@ - - - - - @@ -3459,16 +3026,6 @@ - - - - - - - - - - @@ -3479,11 +3036,6 @@ - - - - - @@ -3532,14 +3084,6 @@ - - - - - - - - @@ -3549,6 +3093,9 @@ + + + @@ -3570,6 +3117,9 @@ + + + @@ -3600,11 +3150,6 @@ - - - - - @@ -3615,11 +3160,6 @@ - - - - - @@ -3660,35 +3200,14 @@ - - - - - - - - - - - - - - - - + + + - - - - - - - - @@ -3730,9 +3249,9 @@ - - - + + + @@ -3751,38 +3270,38 @@ - - - + + + - - + + - - - + + + - - - + + + - - + + - - - + + + - - + + - - - + + + @@ -3793,112 +3312,51 @@ - - - - - - - - + + + - - - + + + - - - + + + - - - - - - - + + - - - + + + - - - + + + - - - - + + - - - + + + - - + + - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + @@ -3957,22 +3415,12 @@ - - - - - - - - - - - + + + - - - - + + @@ -3996,15 +3444,10 @@ - - - - - - - - + + + @@ -4025,25 +3468,20 @@ - - - - - - - - + + + - - - + + + - - + + @@ -4097,9 +3535,9 @@ - - - + + + @@ -4120,55 +3558,6 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -4198,9 +3587,9 @@ - - - + + + @@ -4259,205 +3648,186 @@ - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - - - - + + + - - - + + + - - + + - - - + + + - - - - - - + + + - - - + + + - - - - + + - - - + + + - - + + - - - + + + - - + + - - - - - - + + + - - - - - - + + + - - - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - - - - + + + - - - + + + - - + + - - - + + + - - - - + + - - + + + + @@ -5195,15 +4565,10 @@ - - - - - - - - + + + @@ -5237,14 +4602,6 @@ - - - - - - - - @@ -5266,11 +4623,6 @@ - - - - - @@ -5387,45 +4739,30 @@ - - - - - + + + - - - - - - - - + + + - - - - - - + + + - - - - - - - - + + + @@ -5473,17 +4810,12 @@ - - - + + + - - - - - - - + + @@ -5539,14 +4871,6 @@ - - - - - - - - @@ -5555,12 +4879,15 @@ - - - + + + + + + - - + + @@ -5581,14 +4908,6 @@ - - - - - - - - @@ -5597,6 +4916,16 @@ + + + + + + + + + + @@ -5605,36 +4934,28 @@ - - - + + + - - + + - - - + + + - - + + - - - + + + - - - - - - - - - - + + @@ -5650,36 +4971,28 @@ - - - - - - - - - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + @@ -5714,14 +5027,6 @@ - - - - - - - - @@ -5851,6 +5156,9 @@ + + + @@ -5858,18 +5166,10 @@ - - - - - - - - - - - + + + @@ -5885,18 +5185,10 @@ - - - - - - - - - - - + + + @@ -5912,18 +5204,10 @@ - - - - - - - - - - - + + + @@ -5931,15 +5215,15 @@ - - - + + + - - + + - - + + @@ -5950,23 +5234,12 @@ - - - - - - + + + - - - - - - - - - - + + @@ -5974,12 +5247,15 @@ - - - + + + + + + - - + + @@ -5990,23 +5266,12 @@ - - - + + + - - - - - - - - - - - - - + + @@ -6022,34 +5287,21 @@ - - - + + + - - - - - - - - - - - - - - - + + + + + - - - @@ -6059,55 +5311,27 @@ - - - - - - - - - - - - - - - - - - - - - + + + - - - - - - - - + + + - - - - - @@ -6126,11 +5350,6 @@ - - - - - @@ -6139,11 +5358,6 @@ - - - - - @@ -6181,11 +5395,6 @@ - - - - - @@ -6202,11 +5411,6 @@ - - - - - @@ -6246,6 +5450,17 @@ + + + + + + + + + + + @@ -6280,12 +5495,12 @@ - - - + + + - - + + @@ -6296,6 +5511,11 @@ + + + + + @@ -6365,17 +5585,12 @@ - - - + + + - - - - - - - + + @@ -6394,35 +5609,25 @@ - - - - - - + + + - - - - + + - - - + + + - - - - - - + + + - - - - + + @@ -6435,14 +5640,9 @@ - - - - - - - - + + + @@ -6476,11 +5676,6 @@ - - - - - @@ -6496,99 +5691,28 @@ - - - - - - - - - - - + + + - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + - - - + + + @@ -6599,6 +5723,17 @@ + + + + + + + + + + + @@ -6607,30 +5742,15 @@ - - - - - - - - - - - + + + - - - - - - - + + - - - - + + @@ -6641,6 +5761,17 @@ + + + + + + + + + + + @@ -6657,17 +5788,15 @@ - - - + + + - - + + - - - - + + @@ -6678,30 +5807,15 @@ - - - - - - - - - - - - - - - - + + + - - + + - - - - + + @@ -6709,6 +5823,17 @@ + + + + + + + + + + + @@ -6717,12 +5842,15 @@ - - - + + + - - + + + + + @@ -6741,12 +5869,15 @@ - - - + + + + + + - - + + @@ -6757,14 +5888,6 @@ - - - - - - - - @@ -6786,11 +5909,6 @@ - - - - - @@ -6799,46 +5917,52 @@ - - - + + + + + + + + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - - + + + - - + + @@ -6860,14 +5984,6 @@ - - - - - - - - diff --git a/gradle/versions.gradle b/gradle/versions.gradle index f239370ab7a..b497109d782 100644 --- a/gradle/versions.gradle +++ b/gradle/versions.gradle @@ -22,7 +22,7 @@ dependencyManagement { entry 'antlr4-runtime' } - dependencySet(group:'com.fasterxml.jackson.core', version:'2.14.2') { + dependencySet(group:'com.fasterxml.jackson.core', version:'2.16.1') { entry 'jackson-databind' entry 'jackson-datatype' entry 'jackson-datatype-jdk8' @@ -30,14 +30,11 @@ dependencyManagement { dependency 'com.github.ben-manes.caffeine:caffeine:3.1.8' - dependencySet(group: 'com.github.tomakehurst', version: '2.35.0') { - entry'wiremock-jre8-standalone' - entry'wiremock-jre8' - } + dependency 'com.github.oshi:oshi-core:6.4.10' - dependency 'com.google.auto.service:auto-service:1.0.1' + dependency 'com.google.auto.service:auto-service:1.1.1' - dependencySet(group: 'com.google.dagger', version: '2.45') { + dependencySet(group: 'com.google.dagger', version: '2.50') { entry'dagger-compiler' entry'dagger' } @@ -51,37 +48,37 @@ dependencyManagement { dependency 'com.google.guava:guava:31.1-jre' - dependency 'com.graphql-java:graphql-java:20.1' + dependency 'com.graphql-java:graphql-java:21.3' - dependency 'com.splunk.logging:splunk-library-javalogging:1.11.5' + dependency 'com.splunk.logging:splunk-library-javalogging:1.11.8' - dependency 'com.squareup.okhttp3:okhttp:4.10.0' + dependency 'com.squareup.okhttp3:okhttp:4.12.0' - dependency 'commons-codec:commons-codec:1.15' + dependency 'commons-codec:commons-codec:1.16.0' - dependency 'commons-io:commons-io:2.11.0' + dependency 'commons-io:commons-io:2.15.1' - dependency 'dnsjava:dnsjava:3.5.2' + dependency 'dnsjava:dnsjava:3.5.3' - dependencySet(group: 'info.picocli', version: '4.7.1') { + dependencySet(group: 'info.picocli', version: '4.7.5') { entry 'picocli' entry 'picocli-codegen' } - dependencySet(group: 'io.grpc', version: '1.59.0') { + dependencySet(group: 'io.grpc', version: '1.60.1') { entry 'grpc-all' entry 'grpc-core' entry 'grpc-netty' entry 'grpc-stub' } - dependency 'io.kubernetes:client-java:18.0.0' + dependency 'io.kubernetes:client-java:18.0.1' - dependency 'io.netty:netty-all:4.1.100.Final' + dependency 'io.netty:netty-all:4.1.104.Final' dependency 'io.netty:netty-tcnative-boringssl-static:2.0.62.Final' - dependency group: 'io.netty', name: 'netty-transport-native-epoll', version:'4.1.100.Final', classifier: 'linux-x86_64' - dependency group: 'io.netty', name: 'netty-transport-native-kqueue', version:'4.1.100.Final', classifier: 'osx-x86_64' - dependency 'io.netty:netty-transport-native-unix-common:4.1.100.Final' + dependency group: 'io.netty', name: 'netty-transport-native-epoll', version:'4.1.104.Final', classifier: 'linux-x86_64' + dependency group: 'io.netty', name: 'netty-transport-native-kqueue', version:'4.1.104.Final', classifier: 'osx-x86_64' + dependency 'io.netty:netty-transport-native-unix-common:4.1.104.Final' dependency 'io.opentelemetry:opentelemetry-api:1.24.0' dependency 'io.opentelemetry:opentelemetry-exporter-otlp:1.24.0' @@ -108,6 +105,20 @@ dependencyManagement { dependency 'io.reactivex.rxjava2:rxjava:2.2.21' + dependencySet(group: 'io.tmio', version: '2.4.2') { + entry 'tuweni-bytes' + entry 'tuweni-config' + entry 'tuweni-concurrent' + entry 'tuweni-crypto' + entry 'tuweni-devp2p' + entry 'tuweni-dns-discovery' + entry 'tuweni-io' + entry 'tuweni-net' + entry 'tuweni-rlp' + entry 'tuweni-toml' + entry 'tuweni-units' + } + dependencySet(group: 'io.vertx', version: '4.3.5') { entry 'vertx-auth-jwt' entry 'vertx-codegen' @@ -121,43 +132,29 @@ dependencyManagement { dependency 'junit:junit:4.13.2' - dependency 'net.java.dev.jna:jna:5.13.0' + dependency 'net.java.dev.jna:jna:5.14.0' - dependency 'org.apache.commons:commons-compress:1.23.0' - dependency 'org.apache.commons:commons-lang3:3.12.0' - dependency 'org.apache.commons:commons-text:1.10.0' + dependency 'org.apache.commons:commons-compress:1.25.0' + dependency 'org.apache.commons:commons-lang3:3.14.0' + dependency 'org.apache.commons:commons-text:1.11.0' - dependencySet(group: 'org.apache.logging.log4j', version: '2.20.0') { + dependencySet(group: 'org.apache.logging.log4j', version: '2.22.1') { entry 'log4j-api' entry 'log4j-core' entry 'log4j-jul' entry 'log4j-slf4j2-impl' } - dependencySet(group: 'io.tmio', version: '2.4.2') { - entry 'tuweni-bytes' - entry 'tuweni-config' - entry 'tuweni-concurrent' - entry 'tuweni-crypto' - entry 'tuweni-devp2p' - entry 'tuweni-dns-discovery' - entry 'tuweni-io' - entry 'tuweni-net' - entry 'tuweni-rlp' - entry 'tuweni-toml' - entry 'tuweni-units' - } - - dependency 'org.assertj:assertj-core:3.24.2' + dependency 'org.assertj:assertj-core:3.25.1' dependency 'org.awaitility:awaitility:4.2.0' - dependencySet(group: 'org.bouncycastle', version: '1.76') { + dependencySet(group: 'org.bouncycastle', version: '1.77') { entry'bcpkix-jdk18on' entry'bcprov-jdk18on' } - dependency 'org.fusesource.jansi:jansi:2.4.0' + dependency 'org.fusesource.jansi:jansi:2.4.1' dependency 'org.openjdk.jol:jol-core:0.17' dependency 'tech.pegasys:jc-kzg-4844:0.8.0' @@ -170,16 +167,16 @@ dependencyManagement { entry 'blake2bf' } - dependencySet(group: 'org.immutables', version: '2.9.3') { + dependencySet(group: 'org.immutables', version: '2.10.0') { entry 'value-annotations' entry 'value' } - dependency 'org.java-websocket:Java-WebSocket:1.5.3' + dependency 'org.java-websocket:Java-WebSocket:1.5.5' - dependency 'org.jetbrains.kotlin:kotlin-stdlib:1.8.10' + dependency 'org.jetbrains.kotlin:kotlin-stdlib:1.9.22' - dependencySet(group: 'org.junit.jupiter', version: '5.8.2') { + dependencySet(group: 'org.junit.jupiter', version: '5.10.1') { entry 'junit-jupiter' entry 'junit-jupiter-api' entry 'junit-jupiter-engine' @@ -188,9 +185,9 @@ dependencyManagement { dependency 'org.junit.platform:junit-platform-runner:1.9.2' - dependency 'org.junit.vintage:junit-vintage-engine:5.9.2' + dependency 'org.junit.vintage:junit-vintage-engine:5.10.1' - dependencySet(group: 'org.jupnp', version:'2.7.0') { + dependencySet(group: 'org.jupnp', version:'2.7.1') { entry 'org.jupnp.support' entry 'org.jupnp' } @@ -207,31 +204,33 @@ dependencyManagement { dependency 'org.owasp.encoder:encoder:1.2.3' - dependency 'org.rocksdb:rocksdbjni:8.3.2' + dependency 'org.rocksdb:rocksdbjni:8.9.1' - dependencySet(group: 'org.slf4j', version:'2.0.7') { + dependencySet(group: 'org.slf4j', version:'2.0.10') { entry 'slf4j-api' entry 'slf4j-nop' } - dependency 'org.springframework.security:spring-security-crypto:6.0.2' + dependency 'org.springframework.security:spring-security-crypto:6.2.1' - dependency 'org.testcontainers:testcontainers:1.17.6' + dependency 'org.testcontainers:testcontainers:1.19.3' dependency 'org.web3j:quorum:4.9.5' - dependencySet(group: 'org.web3j', version: '4.9.7') { + dependencySet(group: 'org.web3j', version: '4.10.3') { entry 'abi' entry 'besu' entry 'core' entry 'crypto' } - dependency 'org.xerial.snappy:snappy-java:1.1.9.1' + dependencySet(group: 'org.wiremock', version: '3.3.1') { + entry 'wiremock' + } + + dependency 'org.xerial.snappy:snappy-java:1.1.10.5' dependency 'org.yaml:snakeyaml:2.0' dependency 'tech.pegasys.discovery:discovery:22.2.0' - - dependency 'com.github.oshi:oshi-core:6.4.1' } } diff --git a/metrics/core/src/main/java/org/hyperledger/besu/metrics/ReplaceableDoubleSupplier.java b/metrics/core/src/main/java/org/hyperledger/besu/metrics/ReplaceableDoubleSupplier.java new file mode 100644 index 00000000000..5c424180ad9 --- /dev/null +++ b/metrics/core/src/main/java/org/hyperledger/besu/metrics/ReplaceableDoubleSupplier.java @@ -0,0 +1,55 @@ +/* + * Copyright Hyperledger Besu Contributors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on + * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + * SPDX-License-Identifier: Apache-2.0 + */ +package org.hyperledger.besu.metrics; + +import java.util.function.DoubleSupplier; + +/** + * This class provides a replaceable double supplier. It allows to replace the current double + * supplier with a new one. + */ +public class ReplaceableDoubleSupplier implements DoubleSupplier { + private DoubleSupplier currentSupplier; + + /** + * Constructs a new ReplaceableDoubleSupplier with the given initial supplier. + * + * @param currentSupplier the initial double supplier + */ + public ReplaceableDoubleSupplier(final DoubleSupplier currentSupplier) { + this.currentSupplier = currentSupplier; + } + + /** + * Gets a double value from the current supplier. + * + * @return the double value supplied by the current supplier + */ + @Override + public double getAsDouble() { + return currentSupplier.getAsDouble(); + } + + /** + * Replaces the current double supplier with a new one. + * + * @param newSupplier the new double supplier + * @return this ReplaceableDoubleSupplier + */ + public ReplaceableDoubleSupplier replaceDoubleSupplier(final DoubleSupplier newSupplier) { + currentSupplier = newSupplier; + return this; + } +} diff --git a/metrics/core/src/test/java/org/hyperledger/besu/metrics/ReplaceableDoubleSupplierTest.java b/metrics/core/src/test/java/org/hyperledger/besu/metrics/ReplaceableDoubleSupplierTest.java new file mode 100644 index 00000000000..a83c32e163f --- /dev/null +++ b/metrics/core/src/test/java/org/hyperledger/besu/metrics/ReplaceableDoubleSupplierTest.java @@ -0,0 +1,36 @@ +/* + * Copyright Hyperledger Besu Contributors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on + * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + * SPDX-License-Identifier: Apache-2.0 + */ +package org.hyperledger.besu.metrics; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Test; + +public class ReplaceableDoubleSupplierTest { + + @Test + public void shouldWorkAsNormalSupplier() { + final var rds = new ReplaceableDoubleSupplier(() -> 1.0); + assertThat(rds.getAsDouble()).isEqualTo(1.0); + } + + @Test + public void shouldReturnValueFromNewSupplierIfReplaced() { + final var rds = new ReplaceableDoubleSupplier(() -> 1.0); + assertThat(rds.getAsDouble()).isEqualTo(1.0); + rds.replaceDoubleSupplier(() -> 2.0); + assertThat(rds.getAsDouble()).isEqualTo(2.0); + } +} diff --git a/nat/src/main/java/org/hyperledger/besu/nat/docker/DockerDetector.java b/nat/src/main/java/org/hyperledger/besu/nat/docker/DockerDetector.java index 5504def9c8d..e7f1cf2328b 100644 --- a/nat/src/main/java/org/hyperledger/besu/nat/docker/DockerDetector.java +++ b/nat/src/main/java/org/hyperledger/besu/nat/docker/DockerDetector.java @@ -33,6 +33,8 @@ public Optional detect() { return stream .filter(line -> line.contains("/docker")) .findFirst() + // fallback to looking for /.dockerenv in case we are running on Docker for Mac + .or(() -> Optional.ofNullable(Files.exists(Paths.get("/.dockerenv")) ? "docker" : null)) .map(__ -> NatMethod.DOCKER); } catch (IOException e) { return Optional.empty(); diff --git a/pki/src/test/java/org/hyperledger/besu/pki/keystore/HardwareKeyStoreFileWrapperTest.java b/pki/src/test/java/org/hyperledger/besu/pki/keystore/HardwareKeyStoreFileWrapperTest.java index 0dde591cf50..31eb03126c5 100644 --- a/pki/src/test/java/org/hyperledger/besu/pki/keystore/HardwareKeyStoreFileWrapperTest.java +++ b/pki/src/test/java/org/hyperledger/besu/pki/keystore/HardwareKeyStoreFileWrapperTest.java @@ -22,11 +22,13 @@ import java.nio.file.Path; import java.security.Provider; import java.security.Security; -import java.util.Arrays; import java.util.Collection; +import java.util.List; import java.util.Optional; import java.util.stream.Stream; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.condition.OS; @@ -37,13 +39,19 @@ public class HardwareKeyStoreFileWrapperTest extends BaseKeyStoreFileWrapperTest private static final String crl = "/keystore/partner1client1/crl.pem"; private static final String configName = "NSScrypto-partner1client1"; private static final String validKeystorePassword = "test123"; + private static KeyStoreWrapperTestParameter keyStoreWrapperTestParameter; - public static Collection data() { - return Arrays.asList( + @BeforeAll + public static void setup() { + keyStoreWrapperTestParameter = new KeyStoreWrapperTestParameter( "HardwareKeyStoreWrapper[PKCS11 keystore/truststore]", true, - CryptoTestUtil.isNSSLibInstalled() ? getHardwareKeyStoreWrapper(configName) : null)); + CryptoTestUtil.isNSSLibInstalled() ? getHardwareKeyStoreWrapper(configName) : null); + } + + public static Collection data() { + return List.of(keyStoreWrapperTestParameter); } private static KeyStoreWrapper getHardwareKeyStoreWrapper(final String cfgName) { @@ -58,11 +66,10 @@ private static KeyStoreWrapper getHardwareKeyStoreWrapper(final String cfgName) .map(provider -> new HardwareKeyStoreWrapper(validKeystorePassword, provider, crlPath)) .orElseGet(() -> new HardwareKeyStoreWrapper(validKeystorePassword, path, crlPath)); } catch (final Exception e) { - if (OS.MAC.isCurrentOs()) { - // nss3 is difficult to setup on mac correctly, don't let it break unit tests for dev - // machines. - System.out.println("Failed to initialize hardware keystore " + e.getLocalizedMessage()); - } + // nss3 is difficult to setup on mac, don't let it break unit tests for dev machines. + Assumptions.assumeFalse( + OS.MAC.isCurrentOs(), + "Failed to initialize hardware keystore: " + e.getLocalizedMessage()); // Not a mac, probably a production build. Full failure. throw new PkiException("Failed to initialize hardware keystore", e); } diff --git a/testutil/src/main/java/org/hyperledger/besu/testutil/JsonTestParameters.java b/testutil/src/main/java/org/hyperledger/besu/testutil/JsonTestParameters.java index fe082c9a32d..93519d1c41d 100644 --- a/testutil/src/main/java/org/hyperledger/besu/testutil/JsonTestParameters.java +++ b/testutil/src/main/java/org/hyperledger/besu/testutil/JsonTestParameters.java @@ -37,6 +37,8 @@ import com.fasterxml.jackson.annotation.JsonCreator; import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.core.JsonFactoryBuilder; +import com.fasterxml.jackson.core.StreamReadConstraints; import com.fasterxml.jackson.databind.JavaType; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.datatype.jdk8.Jdk8Module; @@ -121,7 +123,12 @@ public interface Generator { } private static final ObjectMapper objectMapper = - new ObjectMapper().registerModule(new Jdk8Module()); + new ObjectMapper( + new JsonFactoryBuilder() + .streamReadConstraints( + StreamReadConstraints.builder().maxStringLength(Integer.MAX_VALUE).build()) + .build()) + .registerModule(new Jdk8Module()); // The type to which the json file is directly mapped private final Class jsonFileMappedType;