leaf nodes are different for the same files when using balanced and trickle DAG builders

[pgte]

Version information:

go-ipfs version: 0.4.4-
Repo version: 4
System version: amd64/darwin
Golang version: go1.7

Type:

Bug

Priority:

P1

Description:

When encoding the same file using the balanced and the trickle DAG builders separately, both trees end up not sharing any leaf hash.

@whyrusleeping did a preliminary analysis and it's probably due to the fact that the balanced builder uses UnixFS nodes of type file and the trickle builder apparently uses the type raw, hence the discrepancy.

[daviddias]

@whyrusleeping agrees in moving everything to use file, so that we get chunk dedup across both importers

[mib-kd743naq]

@whyrusleeping what is the historical reason to have both file and raw given they are (apparently) entirely interchangeable?

[pgte]

https://github.com/ipfs/js-ipfs-unixfs-engine/issues/119 depends on this

[Kubuxu]

@pgte @diasdavid what is the status of raw-leaves in js world? I wonder if it is worth to smooth out those edge cases if in future we switch to raw-leaves.

I don't know the whole scope and the plan itself but it might be worth considering.

---

Changing this would change hashes of repeated operations and could double storage in some instances if done improperly. I would advise some caution.

[daviddias]

@Kubuxu raw leaves are there, it is just a CID pointing to a binary blob anyway, unixfs-engine is not aware of it, it traverses with the expectations that files use the dag-pb IPLD Format.

Can the proposed change in the issue be part of the next release?

[kevina]

To state the obvious: Changing this will also be a backwards incompatible change. I.e. rehashing a node with a newer version could produce a different hash.

[whyrusleeping]

I think the actionable change here is to make the trickledag code use whatever the balanced dag code is using. Trickledag not being the default makes it a bit more okay to change.

[Jorropo]

This should be fixed by making --raw-leaves default, imo this shouldn't even be configurable.