[ZK Stack] - How to initialise an ecosystem with a local Besu network?

[LimonChillo]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.81.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

I would like to connect my zkstack (v0.1.0-a10c4baa) ecosystem to a custom L1. I tried to connect to a private Besu network (IBFT, running in docker on my local machine) rather than the packaged reth client.

I have updated the wallet adresses at /chains/zk_chain_1/configs/wallets.yaml and /configs/wallets.yaml to pre-funded rich wallets on the Besu network.

After that I started zkstack ecosystem init and it failed executing deploy-scripts/DeployL1.s.sol . Contracts seem to be deployed successfully though, the console output shows execution up until after the updateOwners() task.

Did someone encounter something something similar, or can guide me how I may connect the stack to a custom L1 on Besu?

Thanks a lot everyone!

○ zkstack ecosystem init

┌   ZK Stack CLI
│
◇  Do you want to deploy some test ERC20s?
│  No
│
◇  Do you want to deploy ecosystem contracts? (Not needed if you already have an existing one)
│  Yes
│
◇  What is the RPC URL of the L1 network?
│  http://localhost:21021
│
◇  Do you want to setup observability? (Grafana)
│  Yes
│
●  Initializing ecosystem
│
●  Era observability already setup
│
◇  Installing and building dependencies... done in 28.370514792 secs
│
◓  Deploying ecosystem contracts...                               │
■  Command failed to run
│
■    Status:
│      exit status: 1
│    Stdout:
│      No files changed, compilation skipped
│      EIP-3855 is not supported in one or more of the RPCs used.
│      Unsupported Chain IDs: 1337.
│      Contracts deployed with a Solidity version equal or higher than 0.8.20 might not work properly.
│      For more information, please see https://eips.ethereum.org/EIPS/eip-3855
│      Script ran successfully.
│
│      == Logs ==
│      Deploying L1 contracts
│      Create2Factory deployed at: 0x446e083592D2dFa6661E20097F6898990C3CACE5
│      Multicall3 deployed at: 0x102f00d9f2Dd907Ee5780567e8FaC47006E5a427
│      Verifier deployed at: 0x373116b750D8C0922d3a9b25D44bC4D63EAA14Dd
│      DefaultUpgrade deployed at: 0x2f3e26D9f54dF3F9dcA9BAec60b7592911076627
│      GenesisUpgrade deployed at: 0x29AA208fB9FFd6E384028307E260EC279790cF96
│      ValidatorTimelock deployed at: 0xD57bFE0421c1831C035Ceb6B1738D9Ff429C86BB
│      Governance deployed at: 0xD34B472B1971a5521cEA056d84c566fCb71B24b1
│      ChainAdmin deployed at: 0xFF3d305a882b64A62136242ca8F6933a463D430b
│      Transparent Proxy Admin deployed at: 0x3536Ca51D15f6fc0a76c1f42693F7949b5165F0D
│      Bridgehub Implementation deployed at: 0x913CEdFAa0a859775Cce668D98352545D52D1166
│      Bridgehub Proxy deployed at: 0x72E39d058896933794c64f1ABf086A03E044F8A9
│      BlobVersionedHashRetriever deployed at: 0x12EA1C802e8CE7c0fe6f89e8f0D5f0D60f05dFE4
│      ExecutorFacet deployed at: 0x0a066ab5A867b7E6F23f6c5fF8820E39c6903584
│      AdminFacet deployed at: 0x6163A9fa5beE5eE1FB4AE8af553d72a0a2a63605
│      MailboxFacet deployed at: 0xe9A7a66711319080c7f72056529C5FAa688Ebd30
│      GettersFacet deployed at: 0x63ed4ef3E15c197cfDF47f34232208C3cB1D07FA
│      DiamondInit deployed at: 0xD786d2D30445c2B7Cd9f20c996551dD9681038A9
│      StateTransitionManagerImplementation deployed at: 0x5f2946389BA6Fb66D2AA90698Ee304107270DB26
│      StateTransitionManagerProxy deployed at: 0x27e9A2f6f9Df78bb0B5e2D68d8884804E8D17676
│      StateTransitionManager registered
│      StateTransitionManager set in ValidatorTimelock
│      DiamondProxy deployed at: 0xb41eE62946f71c499563f1191D11fBF8AC1f47E0
│      SharedBridgeImplementation deployed at: 0x6d1685e3792e3C5D5A2B64DCf1d3f87bebe5FDFD
│      SharedBridgeProxy deployed at: 0x725a55d9F58Ba1D2823A3C5b661638E7Cab0c528
│      SharedBridge registered
│      Erc20BridgeImplementation deployed at: 0x276a434a07a4bC8e1f4B17Da88fcA7d446F6AbFB
│      Erc20BridgeProxy deployed at: 0x9863F2ab2B24A13ead07db8531640D70D8AaF957
│      SharedBridge updated with ERC20Bridge address
│      Owners updated
│
│      ## Setting up 1 EVM.
│
│      ==========================
│
│      Chain 1337
│
│      Estimated gas price: 0 gwei
│
│      Estimated total gas used for script: 55854283
│
│      Estimated amount required: 0. ETH
│
│      ==========================
│
│      Transactions saved to:
│      /<PROJECT-PATH>/zksync-era/contracts/l1-contracts/broadcast/DeployL1.s.sol/1337/run-latest.json
│
│      Sensitive values saved to:
│      //<PROJECT-PATH>/zksync-era/contracts/l1-contracts/cache-forge/DeployL1.s.sol/1337/run-latest.json
│
│
│
│    Stderr:
│      Error:
│      Transaction Failure: 0x23c6d323742aae6f05496ff4794a79c02df3955bbee70bfccb430abf40e57ae4
│
│
■  Command failed to run: forge script deploy-scripts/DeployL1.s.sol --legacy --ffi --rpc-url=http://localhost:21021 --slow --private-key=ae6ae8e5ccbfb04590405997ee2d52d2b330726137b875053c36d94e974d162f --broadcast
│  Error:
│  Transaction Failure: 0x23c6d323742aae6f05496ff4794a79c02df3955bbee70bfccb430abf40e57ae4
│
│
▲    0: Command failed to run: forge script deploy-scripts/DeployL1.s.sol --legacy --ffi --rpc-url=http://localhost:21021 --slow --private-key=ae6ae8e5ccbfb04590405997ee2d52d2b330726137b875053c36d94e974d162f --broadcast

Expected Behavior

Stack should be initiated with a custom L1

Current Behavior

See error output above

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[gleenai[bot]]

It seems you are encountering an issue with initializing the ZKsync ecosystem using a custom L1 Besu network. The error message indicates a problem with the deployment script DeployL1.s.sol and specifically mentions a transaction failure.

Here are a few steps and considerations that might help you resolve this issue:

1. Check RPC Compatibility: Ensure that your Besu network's RPC is fully compatible with the Ethereum JSON-RPC specifications required by ZKsync. The error message about EIP-3855 not being supported suggests there might be compatibility issues with the Besu version you are using.

2. Verify Network Configuration: Double-check the network configuration and ensure that the Besu network is correctly set up to handle the types of transactions and contracts being deployed by ZKsync. This includes checking the chain ID and ensuring it matches the expected value in your deployment scripts.

3. Examine Gas Settings: The error log suggests issues related to gas estimation (Estimated gas price: 0 gwei). Verify that your Besu network is configured to provide accurate gas estimates and that it can process transactions with the gas limits being used by the deployment scripts.

4. Review Contract Compatibility: Since the contracts are deployed but the script fails afterward, it might be useful to review the contracts for any compatibility issues with the Besu network, especially concerning features introduced in newer Solidity versions (like EIP-3855).

5. Manual Transaction Submission: Try manually submitting a simple transaction on the network using the same RPC URL to confirm basic connectivity and functionality. This can help isolate the problem to either network configuration or script-specific issues.

6. Logs and Debugging: Increase the logging level of your Besu network and the ZKsync deployment process to capture more detailed error information. This might provide further insights into what exactly is failing.

7. Consult Documentation and Community: Review the ZKsync and Besu documentation for any notes on compatibility or special configuration settings required. You might also want to reach out to the ZKsync developer community or Besu support channels for advice specific to this kind of setup.

If you continue to face issues, providing the detailed logs and configurations might help further diagnose the problem.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[dutterbutter]

Hey @LimonChillo 👋

I do not think anyone has tried to connect to besu before so this is a bit uncharted territory. However, I don't foresee any reason why it should not work but likely some changes may be required when using the cli.

Are you able to successfully run the following forge script forge script deploy-scripts/DeployL1.s.sol --legacy --ffi --rpc-url=http://localhost:21021 --slow --private-key=ae6ae8e5ccbfb04590405997ee2d52d2b330726137b875053c36d94e974d162f --broadcast by itself against your running besu network?

If so, that is a good initial step. Then we need to confirm that the following additional forge scripts can be successfully run against your besu network. Once we have the contracts fully deployed / admins and permissions set then I think that would confirm our ability to use besu as our L1.

[dutterbutter]

I am able to get where you failed, will do some more investigation.

[LimonChillo]

Hi @dutterbutter, just wanted to ask if you were able to get any further with that issue. I could unfortunately not solve it so far, but let me know if I can provide you with any more info.

[dutterbutter]

Apologies on the delay here!

I will re-pick this up next week to further investigate.

[dutterbutter]

Hey there 👋

Apologies for the delay here but I cannot seem to determine the root issue. I do believe its specific to dev besu configuration although I am unsure about what configuration is the issue. Reasoning here is there is no issue against Reth client or any testnet (Sepolia / Holesky). We could further test this by running and syncing a besu node with network set to sepolia and see if the issue persists?

Reviewing the --logging=debug logs when in dev mode:

2025-01-17 15:51:35.010-06:00 | MinerExecutor | DEBUG | BlockTransactionSelector | Transaction pool stats Basefee Prioritized: count: 1, space used: 1139, unique senders: 1, highest priority tx: [score: 127, max fee: 1.00 kwei, curr prio fee: 1.00 kwei, hash: 0x9ead186e5c1d56e167ecb786678ec316c72432b044da7aa28d9a2be78aa40799], lowest priority tx: [score: 127, max fee: 1.00 kwei, curr prio fee: 1.00 kwei, hash: 0x9ead186e5c1d56e167ecb786678ec316c72432b044da7aa28d9a2be78aa40799], next block base fee: 0 wei | Ready: Empty | Sparse: Empty | Dropped: 0
2025-01-17 15:51:35.019-06:00 | MinerExecutor | DEBUG | TransactionSelectionResults | Selection stats: Totals[Evaluated=1, Selected=1, NotSelected=0, Discarded=0]; Detailed[]
2025-01-17 15:51:35.020-06:00 | MinerExecutor | DEBUG | PoWSolver | solving with cpu miner
2025-01-17 15:51:35.219-06:00 | MinerExecutor | DEBUG | DiffBasedWorldState | Persist world state for block Optional[BlockHeader{number=8419, hash=0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa, parentHash=0x74499150243f0c96333a14f91fdf789ae6dde7d37e46a58519c16e9da3c5022e, ommersHash=0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, coinbase=0xfe3b557e8fb62b89f4916b721be55ceb828dbd73, stateRoot=0x57769fc3466ce261bb9614c73c4cfc04d810e39a286718baa90e46efc4640cbf, transactionsRoot=0x28e23ab39f3167d41795c69ac79f853f93de6067f90a82ae78a657a223c74b6d, receiptsRoot=0xa14ea8bbd0cc1cce18fe347a0d98bdfb7a50d482f6d652cb967d578372565a11, logsBloom=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty=0x0000000000000000000000000000000000000000000000000000000000000064, gasLimit=8469493240571404, gasUsed=68131, timestamp=1737150710, extraData=0x, baseFee=0x0000000000000000000000000000000000000000000000000000000000000000, mixHashOrPrevRandao=0x82878f367d5a7c54d82c82a7bc8e42e01bc62efd8503558fbcc3951b1facbdd8, nonce=3833061550173429490, }]
2025-01-17 15:51:35.220-06:00 | MinerExecutor | DEBUG | TrieLogManager | Adding layered world state for 8419 (0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa)
2025-01-17 15:51:35.220-06:00 | MinerExecutor | DEBUG | TrieLogManager | Persisting trie log for block hash 8419 (0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa) and world state root 0x57769fc3466ce261bb9614c73c4cfc04d810e39a286718baa90e46efc4640cbf
2025-01-17 15:51:35.220-06:00 | MinerExecutor | DEBUG | DiffBasedCachedWorldStorageManager | adding layered world state for block 8419 (0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa), state root hash 0x57769fc3466ce261bb9614c73c4cfc04d810e39a286718baa90e46efc4640cbf
2025-01-17 15:51:35.221-06:00 | EthScheduler-Services-0 | DEBUG | AbstractTransactionsLayer | Managing new added block 8419 (0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa)
2025-01-17 15:51:35.221-06:00 | EthScheduler-Services-0 | DEBUG | AbstractTransactionsLayer | Managing new added block 8419 (0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa)
2025-01-17 15:51:35.221-06:00 | EthScheduler-Services-0 | DEBUG | AbstractTransactionsLayer | Managing new added block 8419 (0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa)
2025-01-17 15:51:35.221-06:00 | MinerExecutor | INFO  | BlockMiner | Produced #8,419 / 1 tx / 0 om / 68,131 (0.0%) gas / (0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa) in 1.105s / Timing(started at 2025-01-17T21:51:34.116067Z, protocolWait=888ms, duplicateWorldState=0ms, preTxsSelection=0ms, txsSelection=15ms, blockAssembled=197ms, importingBlock=3ms, notifyListeners=0ms, log=0ms)
2025-01-17 15:51:36.005-06:00 | MinerExecutor | DEBUG | BlockTransactionSelector | Transaction pool stats Basefee Prioritized: count: 1, space used: 4421, unique senders: 1, highest priority tx: [score: 127, max fee: 1.00 kwei, curr prio fee: 1.00 kwei, hash: 0x148cdbad3d7f3524d3cf2045de9462f6d44e0df41ea82e1a42db4678db894a44], lowest priority tx: [score: 127, max fee: 1.00 kwei, curr prio fee: 1.00 kwei, hash: 0x148cdbad3d7f3524d3cf2045de9462f6d44e0df41ea82e1a42db4678db894a44], next block base fee: 0 wei | Ready: Empty | Sparse: Empty | Dropped: 0
2025-01-17 15:51:36.008-06:00 | EthScheduler-BlockCreation-0 | DEBUG | MainnetTransactionProcessor | Transaction 0x148cdbad3d7f3524d3cf2045de9462f6d44e0df41ea82e1a42db4678db894a44 reverted: 0x
2025-01-17 15:51:36.008-06:00 | MinerExecutor | DEBUG | TransactionSelectionResults | Selection stats: Totals[Evaluated=1, Selected=1, NotSelected=0, Discarded=0]; Detailed[]
2025-01-17 15:51:36.009-06:00 | MinerExecutor | DEBUG | PoWSolver | solving with cpu miner
2025-01-17 15:51:36.054-06:00 | MinerExecutor | DEBUG | MainnetTransactionProcessor | Transaction 0x148cdbad3d7f3524d3cf2045de9462f6d44e0df41ea82e1a42db4678db894a44 reverted: 0x
2025-01-17 15:51:36.054-06:00 | MinerExecutor | DEBUG | DiffBasedWorldState | Persist world state for block Optional[BlockHeader{number=8420, hash=0x0ddb2fec901a83fe0c9eb71d80559819fd37d8409732c03e9f5b50955ee359bc, parentHash=0x47caea7aaece68cce8ce6d62ea0d2b91e2a48becfd802493a39505604bc86afa, ommersHash=0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, coinbase=0xfe3b557e8fb62b89f4916b721be55ceb828dbd73, stateRoot=0xd7749d647b4e8612e427e2af98a3917e95a181d21203d968e61ab34223d0f3d3, transactionsRoot=0x8e520089fc500a16c67c7e8d25ced84c8f415c8db464591be3011c3227b344df, receiptsRoot=0xa810707fbe60563a10589d477d1f10540a834cd8fd3914bef5d3df3937b25595, logsBloom=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty=0x0000000000000000000000000000000000000000000000000000000000000064, gasLimit=8469493240571404, gasUsed=1015234, timestamp=1737150711, extraData=0x, baseFee=0x0000000000000000000000000000000000000000000000000000000000000000, mixHashOrPrevRandao=0x8b7105249585dc6812c5be1dcd0703e0ad019d85d251696a67c6d7406458666b, nonce=-145592662891910178, }]
2025-01-17 15:51:36.055-06:00 | MinerExecutor | DEBUG | TrieLogManager | Adding layered world state for 8420 (0x0ddb2fec901a83fe0c9eb71d80559819fd37d8409732c03e9f5b50955ee359bc)
2025-01-17 15:51:36.055-06:00 | MinerExecutor | DEBUG | TrieLogManager | Persisting trie log for block hash 8420 (0x0ddb2fec901a83fe0c9eb71d80559819fd37d8409732c03e9f5b50955ee359bc) and world state root 0xd7749d647b4e8612e427e2af98a3917e95a181d21203d968e61ab34223d0f3d3
2025-01-17 15:51:36.055-06:00 | MinerExecutor | DEBUG | DiffBasedCachedWorldStorageManager | adding layered world state for block 8420 (0x0ddb2fec901a83fe0c9eb71d80559819fd37d8409732c03e9f5b50955ee359bc), state root hash 0xd7749d647b4e8612e427e2af98a3917e95a181d21203d968e61ab34223d0f3d3
2025-01-17 15:51:36.055-06:00 | EthScheduler-Services-0 | DEBUG | AbstractTransactionsLayer | Managing new added block 8420 (0x0ddb2fec901a83fe0c9eb71d80559819fd37d8409732c03e9f5b50955ee359bc)
2025-01-17 15:51:36.055-06:00 | EthScheduler-Services-0 | DEBUG | AbstractTransactionsLayer | Managing new added block 8420 (0x0ddb2fec901a83fe0c9eb71d80559819fd37d8409732c03e9f5b50955ee359bc)
2025-01-17 15:51:36.055-06:00 | EthScheduler-Services-0 | DEBUG | AbstractTransactionsLayer | Managing new added block 8420 (0x0ddb2fec901a83fe0c9eb71d80559819fd37d8409732c03e9f5b50955ee359bc)
2025-01-17 15:51:36.056-06:00 | MinerExecutor | INFO  | BlockMiner | Produced #8,420 / 1 tx / 0 om / 1,015,234 (0.0%) gas / (0x0ddb2fec901a83fe0c9eb71d80559819fd37d8409732c03e9f5b50955ee359bc) in 0.834s / Timing(started at 2025-01-17T21:51:35.221077Z, protocolWait=784ms, duplicateWorldState=0ms, preTxsSelection=0ms, txsSelection=3ms, blockAssembled=43ms, importingBlock=3ms, notifyListeners=0ms, log=0ms)

We see the first transaction 0x9ead186e5c1d56e167ecb786678ec316c72432b044da7aa28d9a2be78aa40799 is successfully added to the tx pool, but when we send the second transaction 0x148cdbad3d7f3524d3cf2045de9462f6d44e0df41ea82e1a42db4678db894a44 it reverts.

If we run with the --skip-simulation flag we observe the following error details:

Error: Failed to send transaction

Context:
- Error #0: Failed to estimate gas for tx
- Error #1: server returned an error response: error code -32000: Execution reverted, data: "0x"

However, reviewing the besu logs with trace it seems to have estimated the gas fine.

Let me know if you have debugged any further or have additional debugging details.

[popzxc]

I'm going to mark this as answered since it appears that the issue is specific to Besu, which is not a target client for ZKsync.
If you will discover the issue and it would require some changes in the ZKsync sequencer codebase to make it work, we will be happy to accept a PR (as long as it doesn't affect compatibility with Reth).

[thecodingshrimp]

Hello all,

tldr: you need to add the "shanghaiTime" field to the genesis.json of besu to be able to deploy the zkStack. The private network setup in docker for besu does not have the shanghai upgrade enabled per default.

I had the same issue when trying to setup zkSynch with a local Besu network. Like @LimonChillo, I used the private network setup in docker.

When executing zkstack ecosystem init I noticed that it was printing a warning saying EIP-3855 is not supported in one or more of the RPCs used. I double checked with the failing transaction by pulling the constructLogs via debug_traceTransaction and it confirmed that some INVALID opcodes were used according to besu. In this case the opcode PUSH0, which was introduced in the shanghai upgrade (EIP-3855).

After I set the shanghaiTime field, I was able to deploy all contracts.

A little side note: I was not able to set the chainId of the l1 chain and therefore the init command failed as long as I didn't set the chainId to 9 in besu/genesis.json. Do you guys have a parameter for that that I have overlooked?