is using ipfs for large scale content distribution a good idea?

[jillesvangurp]

One usecase that comes to mind for ipfs is hosting large amounts of content for e.g. games, virtual worlds, or flight simulators. This is currently something that requires expensive hosting, lots of bandwidth, and typically some sort of CDN to do cost effectively. Many smaller providers of content struggle with distribution for this reason and typically this stuff is done through bit torrent in the form of huge archives that take ages to expand after downloading.

To give one example, somebody created X-plane scenery for spain based on open data (i.e. distribution would be legal) that includes detailed 3D meshes, and high resolution photo scenery, etc. Awesome stuff; check here for details. The whole of spain comes in at about 0.5TB, distributed over gazillions of files of varying sizes. In other words, downloading all of this is rather time consuming. A typical use of this data would be taking off somewhere in Spain, and flying along some narrow path loading objects on a need to have basis. In other words, you'd be unlikely to access more than a tiny fraction of the data and would only need the high detail at low altitudes. So downloading 0.5 TB is probably a colossal waste of bandwidth for most users.

Question: would this be at all a usecase for ipfs to handle; or even remotely a good idea? Could it scale to these levels? Ideally, this stuff would load just in time, Google earth style. Or perhaps more similar to Flightgear's terrasync. That would require pretty high sustained throughput for the downloads and reasonable latency to start downloads of probably many files concurrently.

[flyingzumwalt]

Yes that is one of the target uses of IPFS. We absolutely intend for it to work at those scales. @whyrusleeping is eager to make it work at exabyte scale. In the meantime, here's some of the stuff we're doing to make IPFS work smoothly at large scales:

• One of our current sprints is focused on making IPFS work smoothly with all of the data.gov datasets (190k datasets, ~350TB) See Sprint: Data.gov (aka 300 TB Challenge) archives#87
• Later this quarter we will be building out an InterPlanetary Test Lab, which will allow us to do more thorough testing of the network at large scales
• We are also assembling a Giant Files Initiative to test IPFS performance with individual files that are larger than a Terabyte

[corysimmons]

@flyingzumwalt High level, how will IPFS be able to store exabytes? Is this in the paper or a blog post?

[flyingzumwalt]

Making IPFS performant at Exabyte scale is one of @whyrusleeping's favorite things to bring up in Roadmap discussions. He might be able to provide some technical reference points off the top of his head. We haven't had the opportunity to focus on that kind of scaling yet, but I'm optimistic that we will get to tackle it soon. At the very least, we have a list of known optimizations on this front that I hope to get on the roadmap this year, now that we have the basics of the InterPlanetary Test Lab ready to use.

[whyrusleeping]

@corysimmons when we say "store an exabyte", what we really mean is that the ipfs network will be able to have 1 exabyte of data in it, not that a single machine will have 1 exabyte of storage.

[flyingzumwalt]

Ah yes. I mis-remembered. The roadmap item @whyrusleeping proposed was making individual nodes work at Petabyte scale -- a worthy goal.

Regarding how the entire global network will scale, I'm not aware of any formal studies or blog posts about it.

[corysimmons]

Pretty sure I'll have to stare at the papers for a few days to figure out what's going on, but tl;dr: IPFS' "storage" is just p2p seeding/leeching right?

[flyingzumwalt]

@corysimmons getting other nodes to hold your content is an out-of-band concern for the IPFS protocol. There are a number of tools evolving to handle this. For example, you can use ipfs-cluster to form participating networks of nodes who share a pinset. Alternatively, Filecoin will allow you to pay the network to store stuff for you.

This thread on discourse gets into the topic: https://discuss.ipfs.io/t/ipfs-for-community-led-research/198

[flyingzumwalt]

This issue was moved to https://discuss.ipfs.io/t/is-using-ipfs-for-large-scale-content-distribution-a-good-idea/321