The Deep Dives format is designed to give all participants a taste of what it feels like to work on the big and meaty Open Problems by contributing directly to them in an intense design session - evaluating all the possible solutions, their advantages and shortcomings, and documenting the results.
This type of discussion is core to the genesis of IPFS. We’ve done multiple over the years which have yielded many valuable results such as IPLD, PubSub, GraphSync and other advancements to the original IPFS protocol.
These design sessions are not about explaining how things work, they were focused on finding the solution with the knowledge and resources we had available.
- Objective: Deep knowledge transfer that only happens in synchronous interactions.
- Activity: In a group of max 4 people, dive deep on a complex technical design topic.
- Outcome: The output of each of these sessions would be a RFC like document (can be very raw) that describes the proposed solution (even if with known shortcomings), or it could be a map of all the important concepts, obstacles, and related topics you covered in the session.
- Duration: 1:45h
| # | Topic | Description |
|---|---|---|
| 1 | Private DHT | |
| 2 | Scaling up the DHT | What can we do to optimise the performance of our DHT? Are there alternatives? |
| 3 | Reducing the information leakage on a libp2p dial | |
| 4 | Locking the IPFS repo for concurrent access | How do we manage concurrent access to an IPFS repo when we have multiple tabs on the same domain or web workers or service workers? How do we avoid being too restrictive (i.e. locking everything) whilst guarding against race conditions and concurrency problems? |
| 5 | Identity for DApps | What are the next challenges post IDM? |
| 6 | Live Streaming Capabilities through IPFS to X thousand | |
| 7 | Upgrading the HTTP-API to a RPC API | We desperately need a new remote API https://github.com/ipfs/http-api-spec/issues/116 |
| 8 | Offline First | How can IPFS currently be used to support communities with little to no network bandwidth? What improvements to the protocol would make IPFS offer better solutions to these challenges? |
| 9 | Package Managers: What Now, What Next? | Community evaluation and ideation session to recap the current (as of late June) state of the IPFS-enabled package manager landscape, discuss happiness and pain points, and do some ideation/co-creation around the direction of our next rounds of package manager efforts. |
| 10 | IPFS on Mobile | |
| 11 | Private content on IPFS | |
| 12 | Building the IPFS Community | Community, evangelism, engagement, on ramp. |
| 13 | IPFS for websites (static, dynamic, search, etc) | What are the best practices and open questions for using IPFS for static / dynamic websites? What about indexing and searching? |
| 14 | IPFS in blockchain | |
| 15 | Open research problems in IPFS | Content routing |
| 16 | Revocation/Rotating of IPNS Keys | The idea of using a public key to identify mutable data is cool, but what happens when that key becomes compromised? |
| 17 | Building Reusable Data Structures | IPLD and IPFS allow data to be portable and reused between applications. What are some good behaviors or work flows to promote data reuse instead of re-creation. |
| 18 | Service and Content Discovery under Partition | We tend to think of our systems as operating in either the “global internet” or a “local network”, but the reality can be more complicated than that. Do our systems meet that challenge and if not what would we like them to do? |
| 19 | Referencing HTTP/centralized web content from IPFS | Currently data is available to the IPFS network when a node advertises that they have data and provides it upon request. Would it be useful if we could advertise data provided by HTTP? What are the tradeoffs and mitigations we would need? |
| 20 | How can IPFS support sharding | How to best pin an arbitrary part of a DAG ipfs/kubo#5142 |
| 21 | A better HTTP API | Upgrading the IPFS HTTP API to be more RESTful and easier to use |
| 22 | Garbage Collection how to make more efficient large block stores | We currently stop-the-world and collect, what's involved with incremental, generational. Could we use a separate datastore, something else? |
| 23 | Connection prioritisation | How can we prioritise connections so that important connections are not dropped and important dials happen more quickly. ipfs swarm friend add <addr> - stay connected as long as possible, don't kill connections, reconnect on failure. Right now we add the cluster nodes to the gateway nodes (swarm connect) every minute. |
| 24 | Replication protocol | How can we more quickly sync a DAG or part of a DAG between two nodes? A compliment to bitswap, affectionately overloaded as GraphSync |
| 25 | Do not replicate lists - DNR | How to stop your node from replicating content due to take down notices or other reasons |
| 26 | Fast IPNS | How to increase the IPNS resolution speed |
| 27 | MultiWriter IPNS | Allowing multiple peers to update an IPNS record |
| 28 | WASM in JS IPFS | Should more WASM be used in JS IPFS, what for and where? |
| 29 | Resource utilization | What are the different places an IPFS node will operate and what are the appropriate resource utilization profiles for them? |
| 30 | IPFS Education and Documentation | What are the biggest improvements we can make to documentation for IPFS? |
| 31 | Mounting an IPFS filesystem | Is there a way to efficiently mount IPFS as a filesystem? FUSE support is here. What are current limitations, challenges? What needs to happen to enable mounting of MFS? WebDAV is often used in network storage drives, and mainstream operating systems come with native support for mounting WebDAV as drives. It could supersede Writable Gateway and create a surface for interop with apps that provide built-in client-side support for WebDAV. ipfs/in-web-browsers#146 |
| 32 | Faster content discovery | How can we make content discovery faster using ideas like peer hints or web seeds ipld/ipld#57 |
| 33 | Re-configuring IPFS on the fly | What would need to happen for an IPFS node to be able to be re-configured without a restart? |
| 34 | IPNS link rot | Right now, IPNS needs to be refreshed by original publisher every day, otherwise it stops working. There should be a way to pin IPNS address so that other peers keep refreshing the last valid entry when original node goes offline, keeping the last version alive even if the original publisher disappears forever. |
| 35 | Deterministic CIDs! Reproducible File Imports! Verifable HTTP Gateways! | How to deterministically get the same CID for the same data, even if ipfs.add defaults change? How can we validate data read via HTTP Gateway without running full IPFS node? ipfs/roadmap#19 (comment) ipfs/in-web-browsers#128 |
| 36 | Key Management | How it works today? Where we need to be? Eg: ipfs key is missing primitives for signing and verifying things using keys, we dont have interop for key import/export, keys are stored unencrypted on disk etc. I see this as "where we are"+"what devs need" → "where we need to be for 1.0" type of dive, producing some tangible AIs |
| 37 | IPFS in Web Workers | Where we are, limitations, known problems. ipfs/in-web-browsers#55 |
| 38 | UnixFS v2 | https://github.com/ipfs/unixfs-v2/blob/master/SPEC.md, ipfs/roadmap#19 |
| 39 | Using CRDTs to build real-time collaborative DApps | What are the hurdles to using CRDTs in dapps with IPFS? How can we make peer-base easier to understand and use? |
| 40 | libp2p resource/connection management | When you lose the server/client roles and networks become collaborative, the volume of peering increases, and so does connection count. This can kill residential routers. We have a rudimentary connection manager that keeps connection count within bounds, but we are conceptualising connection manager v2 with things like protocol quotas, burst modelling, and lazy scoring. Help us navigate the design choices we have to make! |
Use this template to create your one-pager description about your topic. Send a PR to https://github.com/ipfs/camp/tree/master/DEEP_DIVES with your topic description page e.g. FAST_IPNS.md
# <Deep Dive Topic NAME>
<Short summary of topic including Open Problem(s) to solve>
## Goal
<Define what we would get if the Open Problem(s) are solved for this Deep Dive>
## Requirements to consider
<Do we have requirements (i.e. work with 1MM nodes)?>
## Where to learn about it
<List of talks, links to github issues, code, notes and so on>