Epic: State Archival Support (Protocol 23)

[sydneynotthecity]

Scope: This epic defines work as it pertains to Hubble Data Platform and internal analytics to support partial state archival (protocol 23)

References:

• Platform Scoping Doc
• Slack channel: #proj-state-archival
• Relevant CAP(s): CAP-57
  • CAP-62 - eviction of live state to a separate evicted bucketlistdb Note: ledger entries will not be deleted yet
  • CAP-63 - in-memory vs disk read resource types
• Rollout plan: Main State Archival Rollout Doc

Core is taking a tiered approach for releasing state archival. In Protocol 23, core will maintain two bucketlistdbs: a live state store, and a "hot" archived store, which will contain evicted ledger entries. No entries will be deleted from validator nodes, but the architecture will be set up so that removal to cold storage is easy. Protocol 23 will not implement a cold storage mechanism for ledger entries yet. For more details on the decision see this doc.

Evicted ledger entries will be auto-restored to the Network via invokeHostFunction operation. The restoreFootprint operation will be deprecated as it is redundant after autorestore support.

While Hubble does not interact with core, it does need to track ledger entry status to ensure all curated datasets reflect accurate metrics.

Changes will need to be made to all analytics components to ensure forwards compatibility with Protocol 23. This includes

• stellar-etl changes - updating libraries and XDR definitions to support State Archival primitives
• _Schema Revisions - adding new ledger change entry types and bucketlist db data structures that inform the user whether an entry is expired, evicted or live. Optionally we can include metadata surrounding BucketlistDB health
• Data Mart Compatibility - assessing and refactoring dbt models to support state archival.
  • Updates to network_stats_agg to include archived/expired entry counts
  • Updates to balance changes to correct aggregates if data is expired

Note that Hubble will not need to add support for the new core endpoints for P23. Hubble will extract/ingest data directly from the LedgerCloseMeta from Galexie/Core