ipfs add --to-files=/path/in/mfs

[lidel]

TLDR: this proposes ipfs add --to-files= as atomic replacement for ipfs add --pin=false + ipfs files cp

Problem

There is a UX gap in our MFS API where users importing data to IPFS via ipfs add and ipfs files write get different CIDs due to different way chunks are assembled into a DAG (balanced vs trickle). Details in issues linked below.

While we hoped the problem will get solved by "/api/v1" it did not happen, and we are still confusing users just like we did in 2016:

• 2022:

  • This remains to be a confusing part of IPFS onboarding experience. Example:

• 2021:

  • https://discuss.ipfs.io/t/ipfs-desktop-uses-ipfs-files-write-or-ipfs-add/12412?u=lidel
  • https://discuss.ipfs.io/t/issue-ipfs-add-and-ipfs-files-write-the-difference-between-the-two/5147?u=lidel

• 2018: Upload via WebUI (ipfs.files.write --create) produces different hash than CLI (ipfs.add) ipfs-webui#676

• 2016: "ipfs add" and "ipfs files write" commands returns different hashes ipfs-inactive/support#45

Unpacking UX challenges

If a user wants to add data to MFS directly, they need to glue two (or more) commands together:

ipfs files cp /ipfs/$(ipfs add -Q test.mp4) /test/test.mp4

ipfs add by default creates a low-level pin which keeps data around even when the file is removed from MFS. This behavior is usually NOT what MFS users want, so they use ipfs files write and get different CID of trickle DAG, and can't benefit from common CID.

One could either do ipfs add with --pin=false (risky when GC is enabled) or remove low-level Pin after files cp is successful (still problematic, in case the CID was pinned for different reasons)

Either way, asking users to execute two or more commands is not nice. What users want and asked for is essentially a better API, an atomic ipfs files cp /ipfs/$(ipfs add -Q --pin=false test.mp4) /test/test.mp4 but with GC safety.

Proposed solution: ipfs files add

Add porcelain command that executes ipfs add --pin=false && ipfs files cp in atomic fashion that is safe from GC.

Main characteristics:

• ergonomics similar to ipfs files write but without ability to append data – instead, allow recursive import of entire dirs like ipfs add -r
• produce the same CID as ipfs add
• atomic operation safe from GC, end result should not add any new low-level pins

[kevincox]

I'd like to pile on saying that this is an important feature. The main reason that I want this is because the ipfs add approach creates a pin. This is problematic because pinning in IPFS is way too simple which can result in disaster. this means that if I want to add a file without pinning I do something like:

cid=$(ipfs add -Q foobar)
ipfs files cp /ipfs/$cid /test/foobar
ipfs pin rm /ipfs/$cid

However if that CID was already pinned I just accidentally removed it! If this was "temporary" I might then remove /test in MFS and now I accidentally removed a pin and lost the data!

The work around is of course to never use IPFS pins, but sometimes you do and don't want to have them accidentally getting deleted.

Adding this API would allow avoiding the pinning API entirely which is highly valuable.

[BigLep]

2022-03-11: @lidel is going to comment on the approach we think we should take here in light of verbal conversations when looking at #8637

[lidel]

#8637 revealed that we would have to duplicate a lot of code that we are not happy about with in the first place. This is not super urgent, I propose we do this in a way that will be easy to maintain.

Lets adjust the scope here, somehow.

Potential paths forward:

• (A) Wait until unixfs and MFS codebases are refactored into something easier to reuse.
• (B) Add opt-in ipfs add --to-mfs=/path/in/mfs flag which copies the root CI to provided path in MFS
  • this removes the need for duplicating any ipfs add code, and provides us with something we could easily add to docs/tutorials, removing the UX gap when added file is not in ipfs-webui's Files screen
• (C) wait until we implement local pins with labels, and then re-enable /pins screen in ipfs-webui to allow users to see added data

Realistically, ETA of (A) and (C) is unknown.

I think adding --to-mfs= suggested in (B) is pretty elegant way of solving the UX gap without being blocked by refactors etc.

Proposed solution: ipfs add --to-files=/path/in/mfs/..

It will be essentially taking the root CID produced by ipfs add and doing ipfs files cp /ipfs/{cid} /path/in/mfs/import-name as an additional step.

We should also do quick checks before doing ipfs add to avoid late errors, and error if

• parent directory /path/in/mfs/ does not exist
• import-name already exists

[BigLep]

@lidel : now that the scope is clarified, do you need to be the one to do this, or is this up for grabs?

[lidel]

Up for grabs!
@schomatis see "Proposed solution" in my previous comment – we would be adding optional --to-files=/path/in/mfs/ to ipfs add that does ipfs files cp

[schomatis]

The copy-option variant seems sane.

[schomatis]

@lidel Could you list here (in pseudo-code if easier) the sharness command tests you'd want for this?

[schomatis]

import-name already exists

Part of a bigger effort in #8264.