Create offchain::ipfs speculative grant

[aphelionz]

Grant Application

This application is (select one):

• Speculative (use this by default)
• an RFP response

This application is (select one):

• Public (fully)
• Public with private finances

Abstract

Based on community feedback we received after announcing the Rust IPFS initiative, we would like to enable the Substrate off-chain worker to make IPFS calls via a relay on the Substrate host. While this was at best cumbersome and at worst infeasible for Substrate users before, the availability of a Rust-native IPFS implementation not only makes this possible without external HTTP servers, but also optimizes network usage and performance significantly.

Once embedded, offchain::ipfs would enable common IPFS operations such as

• Storing data within IPFS via ipfs.add, ipfs.dag.put, and ipfs.block.put
• Retrieving data from IPFS via ipfs.get, ipfs.dag.get and ipfs.block.get
• Exchanging data with peers via ipfs.bitswap
• "Swarming" with the global IPFS network via libp2p

This would benefit the ecosystem by bringing together two mission-critical pieces of Web3 technology, allowing content and data to be marshaled to and from Polkadot parachains, and improving availability in both networks. We believe that this would be a valuable addition to Substrate and as such, we’d like to propose this to be available within the official Substrate repository. Of course, we are willing to take Parity’s guidance on that.

While there are other implementations of IPFS (Go and JavaScript) we would prefer to natively embed IPFS in Rust to keep the codebase consistent and leverage the performance and resource utilization benefits that Rust provides.

Checklist

• The grants document has been read and understood.
• The Google Form will be completed accurately. Note that the Google Form requires the pull request URL.
• Abstract (above) is succinct and complete.
• The application is being included into the correct directory: either 'targeted' or 'speculative'.
• The application includes a project description.
• The application includes all names of team members.
• The application includes a description of the team's experience.
• The application includes all necessary links (e.g. GitHub and LinkedIn)
• The "Development Roadmap" section in the application has a timeline of development ("milestones").
• The "Development Roadmap" section in the application has an estimate of funds required.
• The "Development Roadmap" section gives an indication of the team's long term plans.
• The "Development Roadmap" section includes documentation as a deliverable for at least one milestone.

[dvc94ch]

I'm not really sure how useful this will be in practice, can you provide an example of how this might be used?

A better approach might be to have the ipfs node bundled with a light client as demonstrated here: [0]. This allows sharing the db with ipfs. Transactions can take a Vec<u8> as an argument that represents an ipfs cid. This is useful for storing data that is not required for runtime execution, like for example the text of a grant proposal.

As a further application it might make sense to store merkle tries in ipfs blocks so that inclusion proofs can be provided. Assume you have an identity module that stores a map from account-id => cid. A different module requires to know some field of the identity. The transaction of that module takes as arguments the value of the field and a proof that can be checked by the runtime.

Another application useful might be a commit/reveal scheme, where the thing being commited/revealed is a cid. In this case also it's not very useful to have an ipfs api exposed to the runtime.

• [0] https://github.com/web3garden/sunshine-node/blob/master/client/src/light_client.rs

cc @4meta5

[4meta5]

@dvc94ch I'll open a separate grant that uses ipfs-embed instead of this approach. It is common for Web3 Foundation to fund 2 or more teams anyway for this sort of project so maybe both approaches will be supported.

With that said, I do think that it makes more sense to follow David's approach and we've been doing a lot of work on this already. EDIT: here's a link to a previous related discussion

[dvc94ch]

Based on community feedback we received after announcing the Rust IPFS initiative, we would like to enable the Substrate off-chain worker to make IPFS calls via a relay on the Substrate host.

I'm really curious what that feedback was exactly or how this is intended to be used.

[haadcode]

It seems to me that the proposed grant work exposes the full IPFS API to Substrate users, including IPLD.

The proposed sunshine-node approach as per the first comment uses IPFS merely as a key-value store (db) of cid->content.

From that perspective, it looks to me that the work in the proposed grant will enable a wider set of primitives to the user and I believe exposing the IPFS API will have larger impact on enabling Substrate users to work with IPFS/IPLD.

[dvc94ch]

Any data essential to the runtime logic needs to be stored in the runtime. Any data not required by the runtime logic only requires a hash of the data to be stored to ensure immutability.

[dvc94ch]

My understanding of offchain workers is that their primary usecase is for building oracles. That means that with your approach, you'll fetch a block, read it's contents, and then perform an onchain vote to agree on the value. The inclusion proofs I suggested can avoid this voting process.

I'm sure there are many things I haven't considered, I should have focused my initial comment on requesting clarification as opposed to proposing a different solution.

Can you explain one or two use cases of how this will be used?

[ljedrz]

Following Substrate's vision, it is not our intention to dictate how IPFS would be used. Same with voting - Substrate allows its users to tailor the consensus algorithm to their specific needs and we don't want to affect it. The goal is to enable DApp developers with the means to easily integrate IPFS into their custom pallets, which can offer a great host of functionalities, but are non-trivial to create. By integrating IPFS into the off-chain API we intend to make this process much simpler, which can benefit both the Substrate and IPFS ecosystems.

[4meta5]

By integrating IPFS into the off-chain API we intend to make this process much simpler

That makes complete sense. David is just trying to understand the application architecture that can benefit from this structure.

[subsocialdev]

This is a really interesting proposal. It would be useful to have an IPFS integration close to Substrate logic of our pallets. For example it would let us validate whether provided IPFS CIDs of a newly created blogs and posts on Subsocial point to exisiting contents in IPFS. And in case of invalid CIDs, offchain worker would hide such blogs or posts from the news feeds of the end users.