Move to a streaming interface

[dignifiedquire]

IRC conversation between @diasdavid and @dignifiedquire

dignifiedquire> daviddias: I just realized that the streaming interface for the blockstore is a bit trickier than I hoped
we are passing instances of Block to it, to be written to disk
but that notion itself relies on having all the data in memory, in the associated buffer
<•daviddias> right, because you need the hash of it
so the question is can we make this a streaming interface all the way actually, and if so how
do you know if go has a streaming interface for the blockstoreß
daviddias: looks like go-ipfs is using in memory blocks only as well: https://github.com/ipfs/go-ipfs/blob/master/blocks/blocks.go#L25
<•daviddias> having streaming to disk
<•daviddias> is good because it is a way to capture the signal from the storage driver for backpressure
<•daviddias> but yeah, block-service has to be smart enough
<•daviddias> to capture those signals
<•daviddias> the streaming interface for reads is also important
<•daviddias> because we can pipe it to the network
<•daviddias> and even for writting, if we request a block
<•daviddias> we can pipe it to disk
<•daviddias> and then validate the hash with hashpipe
hmm
this will require a major refactor of a lot of parts :/
I think we should delay this
otherwise we will stretch this move to pull streams very long
daviddias: got everything passing with the current interface
<•daviddias> I see you point
<•daviddias> As always, please document in a issue
<•daviddias> So that we know when to come back to it :)

[dignifiedquire]

Just had a chat with @diasdavid and we plan to migrate the interfaces to streams in the following way

• repo.blockstore exposes read & write streams
• dagservice exposes read & write streams
• blockservice exposes read & write streams and put/get as helper methods

These streams will be pull-streams of Blocks. The individual blocks will stay as they are, i.e. raw buffer + hash.

[daviddias]

"These streams.." with exception of dag-service, which is a DAGNode stream

Also, make sure to PR changes of the interface to interface-ipfs-core object and block API

[daviddias]

done in the latest release