OpenStreetMap Viewer

[davidar]

Continued from ipfs-inactive/archives#11

Goal: create an OSM viewer with all of the assets (tiles) hosted on IPFS

[davidar]

https://ipfs.io/ipfs/QmNUFNQXWVzbhePhj6bscW6TPu5azraCpbejbNY9zyfFjo/

A higher-resolution map is ready to go, but blocked on ipfs/kubo#1924

CC: @jbenet @whyrusleeping

[davidar]

Thanks again to @osm2vectortiles @lukasmartinelli for making this possible :)

[lukasmartinelli]

https://ipfs.io/ipfs/QmNUFNQXWVzbhePhj6bscW6TPu5azraCpbejbNY9zyfFjo/

Really cool. Do you generate the vector tiles yourself or use osm2vectortiles downloads v1.0 or v2.0?
We could even use that as CDN in our showcases.

This is also only data for upper zoom levels? As I zoom in in Europe > zoom level 11 I miss data.

[davidar]

Really cool. Do you generate the vector tiles yourself or use osm2vectortiles downloads v1.0 or v2.0?

@lukasmartinelli I extracted them from the osm2vectortiles v1.0 download. What changes have been made in v2.0?

We could even use that as CDN in our showcases.

That would be cool :)

This is also only data for upper zoom levels? As I zoom in in Europe > zoom level 11 I miss data.

Currently it only goes up to level 9, but once ipfs/kubo#1924 is fixed I'll add the remaining zoom levels (10-14)

[Kubuxu]

Currently the viever doesn't work on local gateways as it tires to access global gateway, which is blocked by no-CORS. I reported CORS issue ipfs/infra#166 and created patch to enable those options by default ipfs/kubo#2778.

The best way of accessing data on IPFS using XHR is rooted relative link: /ipfs/Qm...AAA. Then if site is opened on global gateway, the global gateway will be used, and vice versa.

[lukasmartinelli]

@lukasmartinelli I extracted them from the osm2vectortiles v1.0 download. What changes have been made in v2.0?

Okay perfect. We are also just finished with v2.0, we will release a changelog.
http://osm2vectortiles.org/downloads/

In short: has much better quality, smaller filesize and biggest of all can now be updated in a weekly interval.

Really happy to have IPFS host this 😁 Thanks @davidar for your work.

[davidar]

@Kubuxu Fixed! https://ipfs.io/ipfs/QmNfwFPQqJARCQz87i1V3uAEF2t6WxAffTfA678Ad1JZiv/?/ipfs/QmUefFZttPf9xq4KTkk94rBbZEVrBsTreDi4JA8KYhQFX6/

[btrask]

This sounds very promising!

I would like to suggest a few features for preventing tracking and maximizing user privacy:

• Provide an easy way to pin map data for large geographic areas (like cities), so that queries can be made privately on the user's own machine
• Compress the map data as much as possible so that it's practical to store on e.g. mobile devices (not sure how large the OSM data is currently, if it's small this may not be an issue)
• Structure the data so that efficient differential updates are possible (again OSM may take care of this for you, or not)
• Have an optional warning before trackable remote requests are made (e.g. if the map isn't stored locally, don't just blindly fetch it, since that could leak information)

Thanks @davidar for letting me know about this project!

[davidar]

@lukasmartinelli Updated to osm2vectortiles v2.0 (z0-z8): https://ipfs.io/ipfs/QmTAEbxXKoBWDzNduLS9WmAoNkcuHgQbc3fm9oGBe3FfR4/?/ipfs/Qmf3KBYSv9H4xADgKqpvwWrXkiMh5BfZohFXCaaFh7u4nb/

• Provide an easy way to pin map data for large geographic areas (like cities), so that queries can be made privately on the user's own machine

@btrask Good idea. I'd have to think about the best way to do this.

• Compress the map data as much as possible so that it's practical to store on e.g. mobile devices (not sure how large the OSM data is currently, if it's small this may not be an issue)

It's about 50GB in total, but substantially less for lower levels of detail. Having said that, I don't think it's possible (or at least not recommended) to run an ipfs node on mobile devices yet.

• Structure the data so that efficient differential updates are possible (again OSM may take care of this for you, or not)

IPFS deduplicates tiles, so update efficiency would depend on how many tiles are changed.

• Have an optional warning before trackable remote requests are made (e.g. if the map isn't stored locally, don't just blindly fetch it, since that could leak information)

Yeah, the issue of tracking/privacy in the context of ipfs has been brought up before. In this case you could just use ipfs offline, but that would cause remote requests to silently fail rather than issue a visible warning, so probably isn't the best solution. @jbenet Thoughts?

[ghost]

It's about 50GB in total, but substantially less for lower levels of detail. Having said that, I don't think it's possible (or at least not recommended) to run an ipfs node on mobile devices yet.

It is! https://github.com/ligi/IPFSDroid -- it runs a go-ipfs node though, which needs lots more work regarding resource usage.

[davidar]

@lgierth Does it? The readme must be outdated

[ghost]

yes! -- ligi got it to work the other day, just in time for the Berlin meetup :)

[btrask]

@davidar sounds good!

Just one comment:

IPFS deduplicates tiles, so update efficiency would depend on how many tiles are changed.

Deduplication helps, but it's still up to the application to make sure that its data dedups effectively. That means making sure that small changes to the map result in localized changes within the database. (You probably already know this, but it doesn't hurt to document.)

[davidar]

@btrask localised changes to the map should only change a small number of tiles. @lukasmartinelli would probably be able to give a more concrete answer than me though :)

[lukasmartinelli]

@btrask localised changes to the map should only change a small number of tiles. @lukasmartinelli would probably be able to give a more concrete answer than me though :)

There are two things that go on.

A small number of tiles change very frequently. I can't say percent numbers but it feels like 20% of the tiles change every week and the others don't have any changes at all.

There are butterfly effects when changing boundaries.

Quote from @ImreSamu osm2vectortiles/osm2vectortiles#251

OSM butterfly effect : https://en.wikipedia.org/wiki/Butterfly_effect
The butterfly effect refers to a concept that small causes can have large effects.

for example modifying small part of France - Schweiz border : http://www.openstreetmap.org/changeset/39252252

Has an effect of other part of the French-administered territories ...

• Wikipedia: Overseas departments and territories of France
• Map of French Overseas Departments and Territories

[Kubuxu]

Thanks to de-duplication properties of IFPS it shouldn't really affect us, I might write a tool to calculate de-duplication ratio of two different hashes if there is need for that.

[davidar]

for example modifying small part of France - Schweiz border has an effect of other part of the French-administered territories

@lukasmartinelli why does that happen, out of interest?

[hackergrrl]

@davidar: We demo'd this a few times at the Decentralized Web conf and it wowed people each time! Awesome work. 🎉

[lukasmartinelli]

@lukasmartinelli why does that happen, out of interest?

Because all these lines are in the same superior relation "France".

[whyrusleeping]

@Kubuxu telling the deduplication factor between two hashes is pretty straightforward. The simplest way is to run ipfs refs -r on both, and use comm to take the differences of the set.

Alternatively, using ipfs object diff if the data is formatted in a nice enough way would provide a nice view

[daviddias]

Just had the time to play properly with this for the first time and my reaction was WOW! :D This looks and feels really good!

@lukasmartinelli what about http://osm2vectortiles.org/downloads/ are served by IPFS also, it would really only requires to add an hash to be there on the webpage since we are already hosting it, making the download of those 50GB way faster :)

[lukasmartinelli]

@lukasmartinelli what about http://osm2vectortiles.org/downloads/ are served by IPFS also, it would really only requires to add an hash to be there on the webpage since we are already hosting it, making the download of those 50GB way faster :)

Not entirely sure whether I get it. Could IPFS then serve the downloads - instead of S3?
But we still need to upload the downloads to S3? IPFS would mirror them.

So we would need to append a shortened sha-1 of the file to the filename and then IPFS could serve the downloads?

[daviddias]

We can have IPFS serving all the downloads or we can have the hybrid (the first option would reduce storage and bandwidth costs).

It would be a sha2-256 and you can get it by doing an ipfs add <path to the blob>. Once you get that hash, you will be able to get it through ipfs.io or a user can also 'reprovide' it by booting up a node and loading it through their IPFS node. I'm pulling the data set right now to add it to my local node and so that I can share with you the IPFS Hash. However, if you have the data set already, you can do it too :)

[daviddias]

Here you go :D

planet_z0-z8.mbtiles

» ipfs add planet_z0-z8.mbtiles
added QmeiadPfZUHxZwTSU8nNQGjQQfaan2Yj7GvBVdg1muevAu planet_z0-z8.mbtiles

planet_z0-z5.mbtiles

» ipfs add planet_z0-z5.mbtiles
added QmQ8hxi3wbdoYNG6UPHgZbWz72Ckgr2CtmG3SZQgSW9nW3 planet_z0-z5.mbtiles

planet.mbtiles

» ipfs add planet.mbtiles
added QmVHzEAwUNow7jhABCZEc9KECfCvfJq7ucNx4eXCrP26Da planet.mbtiles

If you run an local ipfs node, replace ipfs.io by localhost:8080 on the links :)

The Qm... are base58 encoded sha2-256 hashes

[jbenet]

cc @mikolalysenko

[davidar]

Graduating to its own repo: https://github.com/davidar/ipfs-maps