CCIP-2206 Added support for retention in the contract_transmitter.go

[mateusz-sekara]

contract_transmitter enables filters in LogPoller that stream all the Transmit events from the chain to the database. Only to pick the latest one from the database.

The following db query is used for that (SelectLatestLogByEventSigWithConfs):

SELECT * FROM evm.logs
			WHERE evm_chain_id = :evm_chain_id
			AND event_sig = :event_sig
			AND address = :address
			AND block_number <= %s
			ORDER BY (block_number, log_index) DESC LIMIT 1

Unfortunately, the more transmits on the lane/ocr2 contract, the more logs we keep in the database only to pick the latest one for getting the current epoch and configDigest. This grows indefinitely because logs are never evicted from the database. Also, the query needs to gather all Transmit logs from a single contract, sort them, and pick the latest one.

Example plan on prod-testnet example

 Limit  (cost=17989.55..17989.55 rows=1 width=416) (actual time=597.180..597.183 rows=1 loops=1)
   Buffers: shared hit=37091 read=506
   I/O Timings: read=510.496
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.43..0.48 rows=1 width=16) (actual time=1.821..1.822 rows=1 loops=1)
           Buffers: shared hit=4 read=1
           I/O Timings: read=1.746
           ->  Index Only Scan using idx_evm_log_poller_blocks_order_by_block on log_poller_blocks  (cost=0.43..4380.14 rows=78039 width=16) (actual time=1.820..1.820 rows=1 loops=1)
                 Index Cond: (evm_chain_id = '43113'::numeric)
                 Heap Fetches: 1
                 Buffers: shared hit=4 read=1
                 I/O Timings: read=1.746
   ->  Sort  (cost=17989.06..18001.47 rows=4963 width=416) (actual time=597.179..597.179 rows=1 loops=1)
         Sort Key: (ROW(logs.block_number, logs.log_index)) DESC
         Sort Method: top-N heapsort  Memory: 26kB
         Buffers: shared hit=37091 read=506
         I/O Timings: read=510.496
         ->  Index Scan using idx_evm_logs_ordered_by_block_and_created_at on logs  (cost=0.56..17964.25 rows=4963 width=416) (actual time=4.768..573.746 rows=39707 loops=1)
               Index Cond: ((evm_chain_id = '43113'::numeric) AND (address = '\x3e0df884042c21e83276abb368b4388a17f78a82'::bytea) AND (event_sig = '\xb04e63db38c49950639fa09d29872f21f5d49d614f3a969d8adf3d4b52e41a62'::bytea) AND (block_number <= $0))
               Buffers: shared hit=37091 read=506
               I/O Timings: read=510.496
 Planning Time: 0.260 ms
 Execution Time: 597.230 ms
(23 rows)

The first step is to enable products to pass retention for that filter (each product can pass whatever the value is accurate for them). That way LogPoller will be able to keep this dataset bounded. Ideally, we should pass MaxLogsKept = 1, but it's not supported by LP yet.

Solution

• Add another constructor that allows passing retention. Default stays the same - no logs pruning.

• Make ordering much more efficient CCIP-2207 Boosting LogPoller performance by fixing how order by clauses are written #13026

[github-actions]

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

• #added For any new functionality added.
• #breaking_change For any functionality that requires manual action for the node to boot.
• #bugfix For bug fixes.
• #changed For any change to the existing functionality.
• #db_update For any feature that introduces updates to database schema.
• #deprecation_notice For any upcoming deprecation functionality.
• #internal For changesets that need to be excluded from the final changelog.
• #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
• #removed For any functionality/config that is removed.
• #updated For any functionality that is updated.
• #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

[github-actions]

I see you added a changeset file but it does not contain a tag. Please edit the text include at least one of the following tags:

• #added For any new functionality added.
• #breaking_change For any functionality that requires manual action for the node to boot.
• #bugfix For bug fixes.
• #changed For any change to the existing functionality.
• #db_update For any feature that introduces updates to database schema.
• #deprecation_notice For any upcoming deprecation functionality.
• #internal For changesets that need to be excluded from the final changelog.
• #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
• #removed For any functionality/config that is removed.
• #updated For any functionality that is updated.
• #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

[cl-sonarqube-production]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
66.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[reductionista]

Yeah, this was bumped way down in priority after the field was added but before the implementation was added. Shouldn't be difficult, but not sure when I'll have the time to get to it.