Open geospatial data hosting support by providing an IPFS Gateway

[hfu]

Background

• UN Smart Maps Group strives to keep web maps open by supporting cloud-native data production.
• However, hosting produced data for testing and prototyping was always a headache because we are an open practice community without a solid funding or other resources.
• We would like to provide multiple hosting options so that we can test new technologies in an easier way.

Existing hosting options

1. For a small dataset, we use GitHub Pages.
2. For a bigger dataset, we tested IPFS for since 2023-03 until 2024-03. Although our IPFS gateway was discontinued because a server resource donation program was expired, we can use IPFS through IPFS-supported browsers like Brave or anyone can host or use an IPFS gateway to access from other browsers.
3. We are also trying Source Cooperative.

Proposal

• The supporting body hosts an IPFS Gateway for use by UN Smart Maps community.
• The data publishing side hosts their data on their IPFS Node in principle.
• The data user on the web can consume the published data through the IPFS Gateway.

Considerations

1. If the IPFS Gateway has a big cache, it works as a CDN for cloud-native open geospatial data.

[hfu]

Service sequence

sequenceDiagram
  participant gateway as Service Host (IPFS Gateway)
  participant node as Data Host (IPFS Node)
  participant user as User (Web Browser)
  Note over gateway, node: node participte in the service
  gateway -->> node: node address
  node ->> gateway: ipfs swarm connect <node address>
  Note over node, user: host and advertise data
  node ->> node: ipfs add
  node -->> user: CID
  Note over gateway, user: data consumption through the service
  user ->> gateway: HTTP GET <CID> Request
  node -->> gateway: transfer
  gateway -->> gateway: cache
  gateway ->> user: HTTP GET Respose

Loading

[hfu]

Suggested first steps after you have the IPFS Node (a kubo instance)

0. [Node] check if your node can get a globally shared content.

Let's check if the following works on your node (VM).

$ ipfs cat QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
hello world

1. [Node] check node addresses

$ ipfs id | egrep 'ip(4|6)'
                "/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWQEy19hSUbDb7FAssBC7u2U7jTkhE2HMnfZerRMeRqrvQ",
...

This is the list of node addresses. Inside the node addresses, 12D3KooWQEy19hSUbDb7FAssBC7u2U7jTkhE2HMnfZerRMeRqrvQ is the unique identifier of your node. /ip4/127.0.0.1/tcp/4001/p2p/12D3KooWQEy19hSUbDb7FAssBC7u2U7jTkhE2HMnfZerRMeRqrvQ explains the way to connect your node.

Do you have any global accessible addresses in the list of node addresses? Then share it with @hfu so that @hfu can connect his node to your node.

2. [Node] check if your node cat get a content from @hfu

FIXME

3. [Gateway] make your gateway accessible from the Internet (via HTTP/2)

The gateway function is exposed only to local host by default. We normally create a reverse-proxy using nginx to make your gateway accessible from the Internet.

My nginx setting (/etc/nginx/sites-available/default) for my experimental server x.opgeo.org (no longer working) was like below:

server {
  listen 443 ssl http2;
  server_name x.optgeo.org;
  root /mnt/hdd/docs;
  ssl_certificate /etc/letsencrypt/live/x.optgeo.org/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/x.optgeo.org/privkey.pem;
  ssl_dhparam dhparam.pem;
  add_header Strict-Transport-Security "max-age=15768000; includeSubdomains";
  add_header Access-Control-Max-Age 1728000;
  add_header Access-Control-Allow-Origin "$http_origin" always;
  add_header Access-Control-Allow-Credentials 'true' always;
  add_header Access-Control-Allow-Methods 'GET, POST, PUT, DELETE, OPTIONS' always;
  add_header Access-Control-Allow-Headers 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,Range,User-Agent,X-Requested-With' always;
  if ($request_method = 'OPTIONS') {
    return 204;
  }

  ...

  location /ipfs {
    proxy_pass http://localhost:8080;
    proxy_hide_header Access-Control-Allow-Origin;
    proxy_set_header Host x.optgeo.org;
  }
  location /ipns {
    proxy_pass http://localhost:8080;
    proxy_hide_header Access-Control-Allow-Origin;
    proxy_set_header Host x.optgeo.org;
  }
}

[hfu]

🎮 Test contents

I have added the following contents for testing purposes.

⛰️ Smart Maps Global Elevation Tiles

CID: QmWsTYyEq9Trn31ZuU8cAJRnCm2WbzduiQLm4Uy7AfcNSs

🌍 OSMFJ OpenStreetMap PMTiles

CID: QmSMpoELQsJjpYYEuNhaoQPsQdq8dVTkbt63vt9ZSu9Dja

🎮 Demo site

I created an unlisted demo site at https://observablehq.com/d/ac71bc83fbe16bc9 for testing.

[hfu]

smb.optgeo.org is back!

Thanks to a kind offer by my friend, the name of smb.optgeo.org is back!
The demo site is now working with this name.

[hfu]

A script to create a backup IPFS Node

https://github.com/optgeo/booth/blob/main/Makefile