Additional data encode/decode methods

[magik6k]

Currently methods for encoding/decoding data to/from sectors are quite limited.

• Unseal/UnsealRange allow reading unsealed data from sealed sectors, but require TreeR generation, which is slow and expensive, and cannot be stored
  • Even if there was a way to generate TreeR/SectorKey (Method to generate sector key #1666), there is no method to decode data from the sealed sector.
  • If that method exists ideally it would look like UnsealRange, taking sealed/key streams, which would allow unsealing services to generate sector key before fetching data from SPs, and decode (parts of) data with the key in a streaming fashion after getting the key.
• Snapdeals encode/decode methods could also use streaming support.

[magik6k]

This is needed to enable filecoin-project/lotus#10316

[cryptonemo]

Noting this is for normal sectors, not snapdeals

[DrPeterVanNostrand]

ELI5: Lotus wants an API for encoding/decoding of "normal" (i.e. non CC) sectors.

• Accept a range of unsealed data and encoding key and return sealed data.
• Accept a range of sealed data and encoding key and return unsealed data.
• Lotus would like proofs API to accept file descriptors (e.g. to [un]sealed data and sealing keys).

[magik6k]

Apparently this is quite easy to do in Go, approximate code that may work:

import (
	"github.com/consensys/gnark-crypto/ecc/bls12-381/fr"
)

func encode(keyBytes []byte, valueBytes []byte) fr.Element {
	// Convert bytes to field elements
	key := new(fr.Element).SetBytes(keyBytes[:])
	value := new(fr.Element).SetBytes(valueBytes[:])

	// Add the field elements
	key.Add(key, value)

	return *key
}

We should probably test that + a snapdeals variant before adding rust-side APIs which may turn out to be redundant

[vmx]

SnapDeals decode is merged and available through the FFI via filecoin-project/filecoin-ffi#402. Normal replica encode/decode will be implemented directly in Go.