[feature request] support for dat / ipfs

[DanielRuf]

It might be interesting to also have some dat (and / or ipfs) support (share cache with others, download from (privately) shared URLs, leverage p2p in environments with bandwidth / connectivity issues, faster, a bit more decentralization, ...).

Is this already supported, does this make sense and how can we help here?

[benjamingr]

Hey @DanielRuf - I work in the p2p space and collaborated with ipfs folks when they asked for Node.js stuff before.

Deno is a server-side tech - if you want to leverage p2p you should already be able to use Deno once it exposes network APIs and communicate with other browsers and swarms.

Unless there is a concrete request or changes or features - then I'm not sure if there is anything left for Deno to do for this.

Note that p2p isn't a magic pill and requires a lot of work to get to reasonable real-world performance and throughput.

[DanielRuf]

once it exposes network APIs and communicate with other browsers and swarms

This is what we have to track and keep in mind and this issue is meant to keep track of current implementations and the support (if this is ok) and collect dis/advantages and solutions for the appropriate use cases.

a lot of work to get to reasonable real-world performance and throughput

I'm totally aware of this =)

[jedahan]

I think a good goal is support for custom protocol handlers so we can do import 'bloop' from 'dat://3726363838fab6363d...'

[benjamingr]

@jedahan well, it makes more sense to eventually support loaders rather than protocol handlers the way Node and client-side tooling does.

[evbogue]

@jedahan or ssb://%NPNNvcnTMZUFZSWl/2Z4XX+YSdqsqOhyPacp+lgpQUw=.sha256!

dat, ipfs, and ssb are all able to launch http servers, so you can load modules into deno with http out of the box from http servers launched to serve blobs from these p2p systems.

But having direct integration would be cool, but how best to do that? Maybe we could keep this issue open and continue to talk about it.

I'm not going to push too hard to implement this straight into deno right now, because as of right now deno is a baby program that doesn't quite work yet.

[benjamingr]

There is a straightforward hook point (the module loader) that both the ECMAScript language and the V8 engine itself expose and other platforms (Node) expose with a --loader flag.

I'm not going to push too hard to implement this straight into deno right now, because as of right now deno is a baby program that doesn't quite work yet.

Definitely, I think Deno is still figuring things out like basic I/O.

[bartlomieju]

@ry is this issue something we have plans for?

[ry]

oof. Not any time soon.

I can conceive of it happening some day, years in the future. But IPFS is not very good right now. And Deno is not very good right now. We cannot dedicate time to that at this point.

Closing tho I'm open to the idea. Will be open to discussing further in 2020.

[evbogue]

I thought import would work out of the box with ssb, with no need for a custom protocol handler. But it doesn't because Deno needs a proper filename.

Example:

import { serve } from "http://evbogue.com:8989/blobs/get/&CNnGcLf0eX1KPtq1t4GjU7jR24Q1bwuD5FRXT2x8U7U=.sha256";
const s = serve("0.0.0.0:8000");

async function main() {
  for await (const req of s) {
    req.respond({ body: new TextEncoder().encode("Hello World\n") });
  }
}

main();

fails with the error:

Downloading http://evbogue.com:8989/blobs/get/&CNnGcLf0eX1KPtq1t4GjU7jR24Q1bwuD5FRXT2x8U7U=.sha256...thread 'tokio-runtime-worker-2' panicked at 'called `Option::unwrap()` on a `None` value', libcore/option.rs:355:21
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Canceled', libcore/result.rs:1009:5

which I think is because the filename is .sha256 instead of .ts.

This doesn't have to be reopened until 2020, but I think this is the only reason why you can't use ssb's distributed blob store as a module manager right now.

[ry]

@evbogue Yes that' a bug. I've opened #1784 for it

[Velua]

Well it's 2020 now, who's keen!?

[abhishekangira]

Well it's 2020 now, who's keen!?

Aye

[SteveGremory]

aaand it's 2021.
still?

[hi-diego]

I think now its a great time to start adding ipfs support to deno, that idea of having "custom loaders" at protocol level sounds good.
So instead of :

import { XmlToJson } from '/https://raw.githubusercontent.com/hi-diego/json-to-xml/master/src/main.ts'

I will like to do something like:

import { XmlToJson } from '/ipfs/5d63845ce24efe53aa98d639aa37f18bd2da68ccbebdc83f05527987a1bab841.sha256.ts'

And avoid using an http to ipfs gateways:

import { XmlToJson } from 'https://ipfs.io/ipfs/5d63845ce24efe53aa98d639aa37f18bd2da68ccbebdc83f05527987a1bab841'

Maybe creating support for custom loaders is the way to go instead of providing support for each protocol natively.