FRC: Retrieval Checking Requirements

[bajtos]

When we set out to build Spark, a protocol for testing whether payload of Filecoin deals can be retrieved back, we designed it based on how Boost worked at that time (mid-2023). Soon after FIL+ allocator compliance started to use Spark retrieval success score (Spark RSR) in mid-2024, we learned that Venus Droplet, an alternative miner software, is implemented slightly differently and requires tweaks to support Spark. Things evolved quite a bit since then. We need to overhaul most of the Spark protocol to support Direct Data Onboarding deals. We will need all miner software projects (Boost, Curio, Venus) to accommodate the new requirements imposed by the upcoming Spark v2 release.

This FRC has the following goals:

1. Document the retrieval process based on IPFS/IPLD.
2. Specify what Spark needs from miner software.
3. Collaborate with the community to tweak the requirements to work well for all parties involved.
4. Let this spec and the building blocks like IPNI Reverse Index empower other builders to design & implement their own retrieval-checking networks as alternatives to Spark.

Discussion

#1086

Progress

• Simple Summary
• Abstract
• Change Motivation
• Specification
• Design Rationale
• Backwards Compatibility
• Test Cases
• Security Considerations
• Incentive Considerations
• Product Considerations
• Implementation
• TODO

[bajtos]

Tagging @steven004 @LexLuthr @magik6k @masih @willscott @juliangruber @patrickwoodhead for visibility.

[LexLuthr]

This requirement cannot be fulfilled in Curio. We no longer have a concept of minerID <> Unique peerID binding. IPNI must be extended to support other keys types like worker key to sign ads.

[bajtos]

I am aware of that; see the note in the text below this paragraphs.

> [!NOTE]
> This is open to extensions in the future, we can support more than one form of linking
> index-provides to filecoin-miners. See e.g. [ipni/spec#33](https://github.com/ipni/specs/issues/33).

From my point of view, I prefer not to block progress on this FRC until the Curio team figures out how to extend IPNI to support other key types. Instead, I'd like this FRC to document the solution that works with Boost & Venus now and then enhance it with the new mechanism Curio needs once that new solution is agreed on.

[lanzafame]

I am probably mistaken here but Droplet (Venus' Boost) supports multiple minerIDs being associated with a single PeerID (see docs), does that mean if I am using Droplet, I need to limit myself to a 1:1 relationship to meet this requirement?

[bajtos]

Great call, @lanzafame! I am still learning more about how Venus Droplet work and what features they offer.

Based on the docs you linked to, I believe you can have multiple minerIDs associated with a single Droplet PeerID and still meet this requirement.

In Spark, we need the PeerID returned by Filecoin.StateMinerInfo to match the PeerID used in IPNI advertisements. Spark does not check whether that PeerID is unique or shared by multiple miners.

[bajtos]

Referencing for visibility - we are discussing a possible solution for this problem here:

filecoin-project/curio#377

[jsoares]

Left a few editorial comments. I do not know enough about the specific topic to be able to opine on a technical level. I also found the explanation somewhat unclear, but that could be a consequence of my lack of knowledge, so not holding that against the draft.

Others will be better suited to provide a full review.

[jsoares]

This reads more like motivation than an abstract. It'd be useful for the abstract to summarise the actual requirements/spec.

[jsoares]

Should this be in the FRC or is it our of scope? The link is fine, but trying to understand whether we see it as central.

[jsoares]

Suggested change

	Storage providers are requires to use the same libp2p peer ID for their block-chain identity as returned by `Filecoin.StateMinerInfo` and for the index provider identity used when communicating with IPNI instances like [cid.contact](https://cid.contact).
	Storage providers are required to use the same libp2p peer ID for their block-chain identity as returned by `Filecoin.StateMinerInfo` and for the index provider identity used when communicating with IPNI instances like [cid.contact](https://cid.contact).

[jsoares]

Not a huge fan of these arbitrary links on a document that's intended to be frozen for a long time.

[jsoares]

The 1 numbered list renders fine, but is not great for reading in raw md.

[rvagg]

Just like this FRC is dictating a convention, you could dictate a convention that at least one of the on-chain Multiaddrs stored in the miner actor of the SP is where you could retrieve from.

[jsoares]

Just to be clear, and given this is an FRC, what are we breaking exactly?

[jsoares]

Suggested change

	To make Filecoin a usable data storage offering, we need the content to be retrievable. It's difficult to improve what you don't measure; therefore, we need to measure the quality of retrieval service provided by each storage provider. To allow 3rd-party networks like [Spark](https://filspark.com) to sample active deals from the on-chain activity and check whether the SP is serving retrievals for the stored content, we need SPs to meet the following requirements:
	1. Link on-chain MinerId and IPNI provider identity ([spec](#link-on-chain-minerid-and-ipni-provider-identity)).
	2. Provide retrieval service using the [IPFS Trustless HTTP Gateway protocol](https://specs.ipfs.tech/http-gateways/trustless-gateway/).
	3. Advertise retrievals to IPNI.
	4. In IPNI advertisements, construct the `ContextID` field from `(PieceCID, PieceSize)` ([spec](#construct-ipni-contextid-from-piececid-piecesize))
	Meeting these requirements needs support in software implementations like Boost, Curio & Venus Droplet but potentially also updates in settings configured by the individual SPs.
	To make Filecoin a usable data storage offering, we need the content to be retrievable. It's difficult to improve what you don't measure; therefore, we need to measure the quality of retrieval service provided by each storage provider. This FRC outlines requirements that SPs and their software stacks should meet to allow 3rd-party networks to sample active deals from the on-chain activity and check whether the SP is serving retrievals for the stored content.

The goal here is not to go into technical detail. I left a non-binding suggestion; something along these lines would be preferable.

The content in 26-29 would potentially be a good fit for the abstract; see comment below.

[lanzafame]

Can a client opt out of their deal payload being indexing?

[rvagg]

It'll depend on the market software, Boost should support optionality here (see) and I imagine Curio will follow suit because there's already been demonstrated demand for "private" deals.

[willscott]

this is all in the context of data that's claimed to be 'retrievable'

We're not defining a standard here for signalling between client and SP / deal making for how a client communicates to an SP that data should be retrievable or not.

[bajtos]

Thank you for the feedback! I'll take a look and respond to your comments (early) next week.

[rvagg]

My quibble with this paragraph is that you're asserting a need for "3rd-party networks", yet the structure you outline below isn't the only way to achieve the goal you've stated (because it's a "need"). This is more like Spark's preferred path, which is fine, it's just not a generalisable need. Maybe it'd be best to not give the impression to the reader that this is the only way to do this.

I'd prefer a more direct approach here: To allow Spark to sample deals, we need the following to be true.

It's nice to have a standard that's specified via FRC but it's not like we have a queue of retrieval checkers waiting for such a standard in order to get going.

[rvagg]

Sequence that looks like a CARv1 will either start at offset 0 or be discoverable from a CARv2 header which is 51 bytes long from offset 0 or will alternatively be a PoDSI container which should also be discoverable from offset 0.

[rvagg]

CID in a CARv1 block header isn't a CID of the payload, it's typically (but not always, and may even be omitted) the root of the entire DAG within the CAR.

But the approach here could be generalised as something like the following: download a small chunk from the start of a piece, determine whether it's readable as either a CARv2, CARv1 or PoDSI container, then use that knowledge to read further sections of the containing CAR(s) to find CIDs to retrieve and build a catalog by progressive byte-range scanning of CAR section headers.

It's true that this is inefficient and would require a stored state to build up knowledge of a piece, but it is possible to do an IPLD block discovery by doing many, progressive, and small, piece retrievals if the SP is exposing the /piece/ endpoint.

You're still back at the problem of trusting the SP that the "piece" they are serving you corresponds to the PieceCID which you trust. As long as the SP is responsible for serving you the response or reporting CIDs to IPNI, you're at the mercy of the SP to tell you what a piece contains rather than the client who ought to have a canonical mapping of piece->blocks (or at least did have at the beginning). Or, to be properly trustless, downloading the entire piece and verifying the PieceCID matches the piece they gave you, but I guess you have to rule this option out and therefore be trusting the SP to play nice.

[willscott]

we should extend range reads over pieces to include the intermediate proof tree (which may already be in the podsci index, or can be generated on the side like the cid index) to allow validation that the range is indeed the expected sub-section of the overall PieceCID

[rvagg]

see my point above, this doesn't need to be a random offset