v1.7.1

Highlights

• Substreams engine is now able run Rust code that depends on solana_program in Solana land to decode and alloy/ether-rs in Ethereum land

How to use solana_program or alloy/ether-rs

Those libraries when used in a wasm32-unknown-unknown context creates in a bunch of wasmbindgen imports in the resulting Substreams Rust code, imports that led to runtime errors because Substreams engine didn't know about those special imports until today.

The Substreams engine is now able to "shims" those wasmbindgen imports enabling you to run code that depends libraries like solana_program and alloy/ether-rs which are known to pull those wasmbindgen imports. This is going to work as long as you do not actually call those special imports. Normal usage of those libraries don't accidentally call those methods normally. If they are called, the WASM module will fail at runtime and stall the Substreams module from going forward.

To enable this feature, you need to explicitly opt-in by appending a +wasm-bindgen-shims at the end of the binary's type in your Substreams manifest:

binaries:
  default:
    type: wasm/rust-v1
    file: <some_file>

to become

binaries:
  default:
    type: wasm/rust-v1+wasm-bindgen-shims
    file: <some_file>

Others

• substreams.yaml now supports localPath attribute under protobuf.descriptorSets, so you can pre-build a descriptor set using buf build --as-file-descriptor-set -o myfile.binpb and add it directly to your substreams package.

• Substreams clients now enable gzip compression over the network (already supported by servers).

• Substreams binary type can now be optionally composed of runtime extensions by appending a +<extension>,[<extesions...>] at the end of the binary type. Extensions are key[=value] that are runtime specifics.

  [!NOTE]
  If you were a library author and parsing generic Substreams manifest(s), you will now need to handle that possibility in the binary type. If you were reading the field without any processing, you don't have to change nothing.

• Fixed a failure in protogen where duplicate files would "appear multiple times" and fail.

• Fixed bug with block rate underflow in gui.

v1.7.0

Compatibility

Note As of graph-node release v0.35.0, substreams that use "index modules" and "set_sum" stores are not yet supported and cannot be used for Substreams-powered-Subgraphs

• Added store with update policy set_sum which allows the store to either sum a numerical value, or set it to a new value.

• Re-added Ethereum Sepolia support in substreams init.

• Fixed a bug with the new descriptorSets feature that wasn't ordered properly to correctly generate Protobuf bindings.

v1.6.2

• execout: preload only one file instead of two, log if undeleted caches found
• execout: add environment variable SUBSTREAMS_DISABLE_PRELOAD_EXEC_FILES to disable file preloading

v1.6.1

• Revert sanity check to support the special case of a substreams with only 'params' as input. This allows a chain-agnostic event to be sent, along with the clock.
• Fix error handling when resolved start-block == stop-block and stop-block is defined as non-zero

v1.6.0

Compatibility

Note As of graph-node release v0.35.0, substreams that use "index modules" are not yet supported and cannot be used for Substreams-powered-Subgraphs

Upgrading

Note Upgrading to v1.6.0 will require changing the tier1 and tier2 versions concurrently, as the internal protocol has changed.

Highlights

Index Modules and Block Filter

• Index Modules and Block Filter can now be used to speed up processing and reduce the amount of parsed data.
• When indexes are used along with the BlockFilter attribute on a mapper, blocks can be skipped completely: they will not be run in downstreams modules or sent in the output stream, except in live segment or in dev-mode, where an empty 'clock' is still sent.
• See https://github.com/streamingfast/substreams-foundational-modules for an example implementation
• Blocks that are skipped will still appear in the metering as "read bytes" (unless a full segment is skipped), but the index stores themselves are not "metered"

Scheduling / speed improvements

• The scheduler no longer duplicates work in the first segments of a request with multiple stages.
• Fix all issues with running a substreams where modules have different "initial blocks"
• Maximum Tier1 output speed improved for data that is already processed
• Tier1 'FileWalker' now polls more aggressively on local filesystem to prevent extra seconds of wait time.

Fixed

• Fix a bug in the gui that would crash when trying to restart the stream.
• fix total read bytes in case data already cache
• Fixed issues when processing modules with different initialBlocks

Added

• New environment variable SUBSTREAMS_WORKERS_RAMPUP_TIME can specify the initial delay before tier1 will reach the number of tier2 concurrent requests.
• Add 'clock' output to substreams run command, useful mostly for performance testing or pre-caching
• (alpha) Introduce the wasip1/tinygo-v1 binary type.

Changed / Removed

• Disabled otelcol:// tracing protocol, its mere presence affected performance.
• Previous value for SUBSTREAMS_WORKERS_RAMPUP_TIME was 4s, now set to 0, disabling the mechanism by default.

v1.5.6

Fixes

• Fix bug where substreams tier2 would sometimes write outputs with the wrong tag (leaked from another tier1 request)

Remove

• Removed MaxWasmFuel since it is not supported in Wazero

v1.5.5

Fixes

• bump wazero to fix issue with certain substreams causing the server process to freeze

Add

• add substreams_tier1_worker_retry_counter metric to count all worker errors returned by tier2
• add substreams_tier1_worker_rejected_overloaded_counter metric to count only worker errors with string "service currently overloaded"

v1.5.4

Fixes

• fix a possible panic() when an request is interrupted during the file loading phase of a squashing operation.
• fix a rare possibility of stalling if only some fullkv stores caches were deleted, but further segments were still present.
• fix stats counters for store operations time

v1.5.3

Performance, memory leak and bug fixes

Server

• fix memory leak on substreams execution (by bumping wazero dependency)
• prevent substreams-tier1 stopping if blocktype auto-detection times out
• allow specifying blocktype directly in Tier1 config to skip auto-detection
• fix missing error handling when writing output data to files. This could result in tier1 request just "hanging" waiting for the file never produced by tier2.
• fix handling of dstore error in tier1 'execout walker' causing stalling issues on S3 or on unexpected storage errors
• increase number of retries on storage when writing states or execouts (5 -> 10)
• prevent slow squashing when loading each segment from full KV store (can happen when a stage contains multiple stores)

Gui

• prevent 'gui' command from crashing on 'incomplete' spkgs without moduledocs (when using --skip-package-validation)

v1.5.2

• Fix a context leak causing tier1 responses to slow down progressively