Block Streaming Framework

[jcape]

This is a tracking issue to track the design and implementation of a block streaming framework:

• Initial draft of Tubes MCIP #1420
• Update rust to more recent nightly #1408
• Write LedgerDB src/sink #1566
• Block Stream Verification  #1618
• Streaming gRPC wrappers for Block Streaming #1573
• SCP Validation for Blockstreaming #1711
• Block streaming wrappers for S3 #1603
• Write a block stream multiplexer #1604
• Write MCIP for new S3 storage protobufs #1709
• Write MCIP for generic streaming traits #1710
• Include QuorumSet and Attestation Verification Report in ArchiveBlock #1682
• AVR-on-Chain in 1.3 #2104
• Validate on-chain AVRs #2197
• Write cargo bench benchmarks for Ledger Streaming Elements #1707
• End-to-end testing for block streaming #1708

Block Streaming 2022-06-28 (4.x):

• Finish E2E Demo/benchmarking
• Rewrite ledger-distribution (read from LMDB, serve gRPC, reliably push to S3)
  • Get requirements for L-D-ng from ops
• Write repeater app (good first issue)
• Enhanced Metadata Validation of streams
• Consensus service uses streaming for catchup instead of LedgerSync
• Streaming -> mobilecoind
• Streaming -> full-service
• Streaming -> Chronometer (Depending on SDLC work stream)
• Streaming -> Ingest
• Streaming -> Ledger