chore: OP modules improvements #11073

varasev · 2024-10-30T10:10:44Z

Fixes #11086, fixes #11101, closes #11077.

Motivation

This PR does the following things:

adds a new socket channel with optimism:new_batch topic and new_optimism_batch event for the frontend (see details below)
replaces optimism_deposits:new_deposits socket topic (and deposits event) with optimism:new_deposits topic (and new_optimism_deposits event) for consistency. The payload remains the same
adds fallback envs to use them when the SystemConfig contract is obsolete: INDEXER_OPTIMISM_L1_PORTAL_CONTRACT, INDEXER_OPTIMISM_L1_START_BLOCK, INDEXER_OPTIMISM_L1_BATCH_INBOX, INDEXER_OPTIMISM_L1_BATCH_SUBMITTER
adds INDEXER_OPTIMISM_L1_ETH_GET_LOGS_RANGE_SIZE and INDEXER_OPTIMISM_L2_ETH_GET_LOGS_RANGE_SIZE env variables to be able to define request limits for eth_getLogs requests (instead of the hardcoded values)
removes INDEXER_OPTIMISM_L1_DEPOSITS_BATCH_SIZE env variable in favor of INDEXER_OPTIMISM_L1_ETH_GET_LOGS_RANGE_SIZE
adds INDEXER_OPTIMISM_BLOCK_DURATION env variable to be able to define block time (which is 2 seconds by default for OP chains)
fixes the Deposits module (now it doesn't use eth_newFilter and eth_getFilterChanges)
adds timeout: :infinity for some db select requests
adds support of transactions without to field in eth_getTransactionByHash RPC response
adds logic to find the last known block after reorg and restart of the instance (only for OP fetcher modules)
hardcodes INDEXER_BEACON_BLOB_FETCHER_* env variables for Indexer.Fetcher.Optimism.TransactionBatch module (see Hardcode INDEXER_BEACON_BLOB_FETCHER_* envs for Indexer.Fetcher.Optimism.TransactionBatch module #11077)

New/updated sockets

https://github.com/blockscout/frontend needs to be updated to read the new socket channel (optimism:new_batch topic and new_optimism_batch event).

The payload for that looks as follows:

"batch": {
  "internal_id": 123,
  "l1_timestamp": "2024-01-29T21:31:35.000000Z",
  "transaction_count": 237
}

Also, optimism_deposits:new_deposits topic was replaced with optimism:new_deposits (and the deposits event with new_optimism_deposits correspondingly). The payload remains the same as before.

New env variables

INDEXER_OPTIMISM_L1_PORTAL_CONTRACT (as fallback when SystemConfig is old)
INDEXER_OPTIMISM_L1_START_BLOCK (as fallback when SystemConfig is old)
INDEXER_OPTIMISM_L1_BATCH_INBOX (as fallback when SystemConfig is old)
INDEXER_OPTIMISM_L1_BATCH_SUBMITTER (as fallback when SystemConfig is old)
INDEXER_OPTIMISM_L1_ETH_GET_LOGS_RANGE_SIZE (optional)
INDEXER_OPTIMISM_L2_ETH_GET_LOGS_RANGE_SIZE (optional)
INDEXER_OPTIMISM_BLOCK_DURATION (optional)

See their descriptions in blockscout/docs#337

Fix for `Indexer.Fetcher.Optimism.Deposit`

Now the module doesn't use eth_newFilter and eth_getFilterChanges JSON RPC requests as they work incorrectly on Nethermind (see NethermindEth/nethermind#7751).

Support transactions without `to` field

Contract-creating transactions don't have the to field in eth_getTransactionByHash response sent by reth RPC node. This fix eliminates the error described in #11086 and #11101.

Finding correct block number to start from after reorg

OP fetchers were improved in the part of reorgs handling: now OP fetcher will try to find nearest block from which it can continue to work after reorg and restarting the instance. Previous implementation just threw an error in logs and stopped the fetcher.

Checklist for your Pull Request (PR)

If I added new functionality, I added tests covering it.
If I fixed a bug, I added a regression test to prevent the bug from silently reappearing again.
I checked whether I should update the docs and did so by submitting a PR to docs repository. Add new OP envs docs#337
If I added/changed/removed ENV var, I submitted a PR to docs repository to update the list of env vars and I updated the version to master in the Version column. If I removed variable, I added it to Deprecated ENV Variables page. After merging docs PR, changes will be reflected in these pages.
If I added new DB indices, I checked, that they are not redundant, with PGHero or other tools.
If I added/removed chain type, I modified the Github CI matrix and PR labels accordingly.

vbaranov

replaces optimism_deposits:new_deposits socket topic (and deposits event) with optimism:new_deposits topic (and new_optimism_deposits event) for consistency. The payload remains the same

This breaking change requires to synchronize releases between backend and frontend which is complicated thing. Can we temporarily keep compatibility with the current version of the frontend? I'd suggest implementing new events and keep old events until frontend will not bind to new ones. Such a multi-step process gives us more flexibility in backend release publishing and more time to find possibly broken functionality.

Seems, web application tests on Filecoin chain type are failed.

apps/indexer/lib/indexer/helper.ex

apps/block_scout_web/lib/block_scout_web/views/api/v2/optimism_view.ex

apps/explorer/lib/explorer/chain/optimism/frame_sequence.ex

apps/indexer/lib/indexer/fetcher/optimism.ex

varasev · 2024-11-11T12:00:43Z

This breaking change requires to synchronize releases between backend and frontend which is complicated thing. Can we temporarily keep compatibility with the current version of the frontend? I'd suggest implementing new events and keep old events until frontend will not bind to new ones. Such a multi-step process gives us more flexibility in backend release publishing and more time to find possibly broken functionality.

Yes, I restored those socket topic/event in c2c52dd

varasev · 2024-11-11T12:55:36Z

2. Seems, web application tests on Filecoin chain type are failed.

I think, this is not related to the current changes, but I'll try to rebase onto master and will see

varasev · 2024-11-15T10:26:53Z

@vbaranov Sorry, I've added one more commit 0e1334c (to fix OP Deposits fetcher due to the bug in Nethermind)

vbaranov · 2024-11-20T09:39:09Z

This breaking change requires to synchronize releases between backend and frontend which is complicated thing. Can we temporarily keep compatibility with the current version of the frontend? I'd suggest implementing new events and keep old events until frontend will not bind to new ones. Such a multi-step process gives us more flexibility in backend release publishing and more time to find possibly broken functionality.

Yes, I restored those socket topic/event in c2c52dd

@varasev shouldn't you keep corresponding channel apps/block_scout_web/lib/block_scout_web/channels/optimism_deposit_channel.ex for compatibility as well?

varasev · 2024-11-20T11:22:40Z

@varasev shouldn't you keep corresponding channel apps/block_scout_web/lib/block_scout_web/channels/optimism_deposit_channel.ex for compatibility as well?

You mean the deposits handler from here? https://github.com/blockscout/blockscout/blob/50da37f33601b1b8192570497a594f6d0bebb17f/apps/block_scout_web/lib/block_scout_web/channels/optimism_deposit_channel.ex

It's here now:

blockscout/apps/block_scout_web/lib/block_scout_web/notifiers/optimism.ex

Lines 27 to 29 in c2c52dd

    
           Endpoint.broadcast("optimism_deposits:new_deposits", "deposits", %{ 
        
             deposits: deposits_count 
        
           })

vbaranov · 2024-11-20T12:03:12Z

Endpoint.broadcast("optimism_deposits:new_deposits", "deposits", %{

@varasev let's mark with todo comments parts of the code which are temporarily added for backward compatibility and conditions when those parts should be removed.

varasev · 2024-11-20T12:14:50Z

@varasev let's mark with todo comments parts of the code which are temporarily added for backward compatibility and conditions when those parts should be removed.

Done in f5b14a4

vbaranov · 2024-11-20T12:37:54Z

@varasev shouldn't you keep corresponding channel apps/block_scout_web/lib/block_scout_web/channels/optimism_deposit_channel.ex for compatibility as well?

You mean the deposits handler from here? https://github.com/blockscout/blockscout/blob/50da37f33601b1b8192570497a594f6d0bebb17f/apps/block_scout_web/lib/block_scout_web/channels/optimism_deposit_channel.ex

It's here now:

blockscout/apps/block_scout_web/lib/block_scout_web/notifiers/optimism.ex

Lines 27 to 29 in c2c52dd

Endpoint.broadcast("optimism_deposits:new_deposits", "deposits", %{

deposits: deposits_count

})

Yeah, I meant that. But how about handling of the old event in notifier.ex? I don't see handler for the old event. Also, I don't see the definition of the old channel in user_socket.ex and user_socket_v2.ex.

vbaranov · 2024-11-20T12:37:59Z

apps/indexer/lib/indexer/fetcher/optimism/deposit.ex

+              timeout: :infinity
+            })
+
+          Publisher.broadcast(%{new_optimism_deposits: deposit_events}, :realtime)


Seems, there is no backward compatibility with the current frontend here. We should also broadcast the old event as well.

This line is responsible for the internal broadcasting of the new_optimism_deposits notifier event. It's caught by the handler in BlockScoutWeb.Notifiers.Optimism and then optimism:new_deposits and optimism_deposits:new_deposits topics are broadcasted to the socket

varasev · 2024-11-21T06:33:32Z

You mean the deposits handler from here? https://github.com/blockscout/blockscout/blob/50da37f33601b1b8192570497a594f6d0bebb17f/apps/block_scout_web/lib/block_scout_web/channels/optimism_deposit_channel.ex
It's here now:

blockscout/apps/block_scout_web/lib/block_scout_web/notifiers/optimism.ex

Lines 27 to 29 in c2c52dd

Endpoint.broadcast("optimism_deposits:new_deposits", "deposits", %{

deposits: deposits_count

})

Yeah, I meant that. But how about handling of the old event in notifier.ex? I don't see handler for the old event. Also, I don't see the definition of the old channel in user_socket.ex and user_socket_v2.ex.

Ah, forgot to remove : in the UserSocketV2. Fixed in de147bc. The event is handled here:

blockscout/apps/block_scout_web/lib/block_scout_web/notifiers/optimism.ex

Lines 30 to 32 in de147bc

    
           Endpoint.broadcast("optimism_deposits:new_deposits", "deposits", %{ 
        
             deposits: deposits_count 
        
           })

Regarding the user_socket.ex - do we really need that for Optimism? E.g., there is no Arbitrum handler there.

Don't worry, I locally tested the old and new socket topics with PieSocket Websocket Tester - everything works fine

apps/block_scout_web/lib/block_scout_web/channels/user_socket_v2.ex

* Add new envs for OP stack * Fix updating logs filter in OP Deposits fetcher * Add fallback envs for OP * Add socket notifier for OP batches * Update common-blockscout.env * Set infinity timeout for select db queries * Support transactions without `to` field * Add some docs * mix format * Restore OP fetcher after reorg and restart * Add specs and docs * Fix spelling * Refactoring and hardcode INDEXER_BEACON_BLOB_FETCHER_* envs * mix format * Update spelling * Small fix for Indexer.Fetcher.Optimism.Deposit * Rewrite Indexer.Fetcher.Optimism.Deposit * Update common-blockscout.env * Add todo comments for deprecated socket topic * Fix for the new websocket channel * Add todo comment --------- Co-authored-by: POA <[email protected]>

varasev mentioned this pull request Oct 30, 2024

Add new OP envs blockscout/docs#337

Merged

varasev added the ci:optimism label Oct 30, 2024

varasev mentioned this pull request Nov 1, 2024

Blockscout Optimism L3 (rollup to Optimism ) optimism_txn_batches error #11086

Closed

varasev added the ci:all label Nov 1, 2024

varasev requested a review from vbaranov November 1, 2024 12:19

varasev marked this pull request as ready for review November 1, 2024 12:19

varasev mentioned this pull request Nov 4, 2024

Optimism Bedrock blockscout 1.9.0 transaction batch #11101

Closed

vbaranov reviewed Nov 8, 2024

View reviewed changes

varasev force-pushed the va-op-improvements branch from 53c3042 to 79da9c9 Compare November 11, 2024 12:53

varasev added 16 commits November 13, 2024 21:31

Add new envs for OP stack

3401618

Fix updating logs filter in OP Deposits fetcher

acd8711

Add fallback envs for OP

f375199

Add socket notifier for OP batches

97ef404

Update common-blockscout.env

d80b432

Set infinity timeout for select db queries

89e53e1

Support transactions without to field

d633d78

Add some docs

ca1eafe

mix format

be3676d

Restore OP fetcher after reorg and restart

62c8451

Add specs and docs

59689bf

Fix spelling

538e895

Refactoring and hardcode INDEXER_BEACON_BLOB_FETCHER_* envs

253d0da

mix format

594a89a

Update spelling

c0e5372

Small fix for Indexer.Fetcher.Optimism.Deposit

7718245

varasev force-pushed the va-op-improvements branch from b6dbcb1 to 7718245 Compare November 13, 2024 17:32

varasev added 2 commits November 15, 2024 14:10

Rewrite Indexer.Fetcher.Optimism.Deposit

0e1334c

Update common-blockscout.env

8d8baf8

Add todo comments for deprecated socket topic

f5b14a4

vbaranov reviewed Nov 20, 2024

View reviewed changes

Fix for the new websocket channel

de147bc

vbaranov self-requested a review November 21, 2024 08:30

vbaranov reviewed Nov 21, 2024

View reviewed changes

apps/block_scout_web/lib/block_scout_web/channels/user_socket_v2.ex Show resolved Hide resolved

Add todo comment

d89f2c9

vbaranov self-requested a review November 21, 2024 11:51

vbaranov approved these changes Nov 21, 2024

View reviewed changes

vbaranov merged commit b745890 into master Nov 21, 2024
81 of 82 checks passed

vbaranov deleted the va-op-improvements branch November 21, 2024 11:53

varasev mentioned this pull request Nov 27, 2024

fix: Rework initialization of the RollupL1ReorgMonitor and fix read_system_config for fallback cases #11275

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: OP modules improvements #11073

chore: OP modules improvements #11073

varasev commented Oct 30, 2024 •

edited

Loading

vbaranov left a comment

varasev commented Nov 11, 2024 •

edited

Loading

varasev commented Nov 11, 2024

varasev commented Nov 15, 2024

vbaranov commented Nov 20, 2024

varasev commented Nov 20, 2024

vbaranov commented Nov 20, 2024

varasev commented Nov 20, 2024

vbaranov commented Nov 20, 2024

vbaranov Nov 20, 2024

varasev Nov 21, 2024

varasev commented Nov 21, 2024

chore: OP modules improvements #11073

chore: OP modules improvements #11073

Conversation

varasev commented Oct 30, 2024 • edited Loading

Motivation

New/updated sockets

New env variables

Fix for Indexer.Fetcher.Optimism.Deposit

Support transactions without to field

Finding correct block number to start from after reorg

Checklist for your Pull Request (PR)

vbaranov left a comment

Choose a reason for hiding this comment

varasev commented Nov 11, 2024 • edited Loading

varasev commented Nov 11, 2024

varasev commented Nov 15, 2024

vbaranov commented Nov 20, 2024

varasev commented Nov 20, 2024

vbaranov commented Nov 20, 2024

varasev commented Nov 20, 2024

vbaranov commented Nov 20, 2024

vbaranov Nov 20, 2024

Choose a reason for hiding this comment

varasev Nov 21, 2024

Choose a reason for hiding this comment

varasev commented Nov 21, 2024

varasev commented Oct 30, 2024 •

edited

Loading

Fix for `Indexer.Fetcher.Optimism.Deposit`

Support transactions without `to` field

varasev commented Nov 11, 2024 •

edited

Loading