Containerize tileserver

[iandees]

This is a work in progress as I experiment with containerizing tileserver for easier development and more consistent production deploys.

[nvkelso]

Why are these Mapnik related installs necessary?

[nvkelso]

Looks like:

• https://github.com/tilezen/vector-datasource/wiki/Mapzen-Vector-Tile-Service

Has reference to Mapnik in error, compared to requirement file:

• https://github.com/tilezen/tileserver/blob/master/requirements.txt

So update the Wiki?

[zerebubuth]

Looks great!

I had to add this to the requirements.txt to get a docker build:

diff --git a/requirements.txt b/requirements.txt
index cf3e4e9..b104d93 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -25,3 +25,4 @@ zope.dottedname==4.1.0
 edtf==0.9.3
 mapbox-vector-tile==1.2.0
 git+https://github.com/mapzen/tilequeue@master#egg=tilequeue
+git+https://github.com/tilezen/raw_tiles@master#egg=raw_tiles

When I docker-compose up, tileserver complains about config.yaml being missing. I wondered whether you have that file, but perhaps it didn't get committed because it's listed in .gitignore? Perhaps we could store it under a different name (config.docker-compose.yaml or something) and have it picked up from there if it's not overridden by config.yaml existing? (I don't know if any of this is possible - this morning is the longest I've spent using Docker to date!)

[rmarianski]

The preview is a great idea!

[iandees]

Thanks for the suggestion on the config.yaml. I made that change and it seems to work for me. I'm not sure why it worked for me without the raw tiles requirement.

[bcamper]

Excited about this! I just tried pulling the latest and running docker-compose up, went smoothly for awhile but then errored out with:

Creating tileserver_redis_1 ... 
Creating tileserver_postgis_1 ... 
Creating tileserver_redis_1
Creating tileserver_redis_1 ... done
Creating tileserver_tileserver_1 ... 
Creating tileserver_tileserver_1 ... error

ERROR: for tileserver_tileserver_1  Cannot start service tileserver: driver failed programming external connectivity on endpoint tileserver_tileserver_1 (aa08ab23918b13b812506001bd164b885f1d21abd6f701847e738f829245a768): Error starting userland proxy: Bind for 0.0.0.0:8000 failed: port is already allocated

ERROR: for tileserver  Cannot start service tileserver: driver failed programming external connectivity on endpoint tileserver_tileserver_1 (aa08ab23918b13b812506001bd164b885f1d21abd6f701847e738f829245a768): Error starting userland proxy: Bind for 0.0.0.0:8000 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.

[bcamper]

Oh duh I do have port 8000 already bound... continuing...

[bcamper]

Follow up... postgis seems to have died on initial startup attempt:

postgis_1     | Loading PostGIS extensions into osm
postgis_1     | CREATE EXTENSION
postgis_1     | CREATE EXTENSION
postgis_1     | CREATE EXTENSION
postgis_1     | CREATE EXTENSION
postgis_1     | 
postgis_1     | LOG:  received fast shutdown request
postgis_1     | LOG:  aborting any active transactions
postgis_1     | LOG:  autovacuum launcher shutting down
postgis_1     | waiting for server to shut down....FATAL:  terminating connection due to administrator command
postgis_1     | LOG:  could not send data to client: Broken pipe
postgis_1     | LOG:  shutting down
postgis_1     | .LOG:  database system is shut down
postgis_1     |  done
postgis_1     | server stopped
postgis_1     | 
postgis_1     | PostgreSQL init process complete; ready for start up.
postgis_1     | 
postgis_1     | LOG:  database system was shut down at 2017-11-08 18:17:30 UTC
postgis_1     | LOG:  MultiXact member wraparound protections are now enabled
postgis_1     | LOG:  database system is ready to accept connections
postgis_1     | LOG:  autovacuum launcher started

I simply tried re-running docker-compose up and got:

postgis_1     | LOG:  database system was interrupted; last known up at 2017-11-08 18:22:55 UTC
postgis_1     | LOG:  database system was not properly shut down; automatic recovery in progress
postgis_1     | LOG:  invalid record length at 0/3040AB8
postgis_1     | LOG:  redo is not required
postgis_1     | LOG:  MultiXact member wraparound protections are now enabled
postgis_1     | LOG:  database system is ready to accept connections
postgis_1     | LOG:  autovacuum launcher started

But, any requests to preview fail with a missing hstore error:

tileserver_1  | Error handling request for /all/15/9647/12321.mvt: Traceback (most recent call last): |   File "/usr/src/app/tileserver/__init__.py", line 151, in __call__    response = self.handle_request(request) |   File "/usr/src/app/tileserver/__init__.py", line 239, in handle_request    source_rows = fetcher(nominal_zoom, unpadded_bounds) |   File "/usr/local/lib/python2.7/site-packages/tilequeue/query/postgres.py", line 183, in __call__    with self.sql_conn_pool.get_conns(n_conns) as sql_conns: |   File "/usr/local/lib/python2.7/site-packages/tilequeue/query/pool.py", line 53, in get_conns    conn = self._make_conn(conn_info_with_db) |   File "/usr/local/lib/python2.7/site-packages/tilequeue/query/pool.py", line 43, in _make_conn    register_hstore(conn) |   File "/usr/local/lib/python2.7/site-packages/psycopg2/extras.py", line 775, in register_hstore    "hstore type not found in the database. " | ProgrammingError: hstore type not found in the database. please install it from your 'contrib/hstore.sql' file

[iandees]

Yep, this is expected for now. There's a modification to the docker-compose.yml that I need to make to support setting up and inserting data into the PostGIS database you just created in one step.

For now, you can try my workaround:

1. After running docker-compose up once the database will be empty, but still should be running. Check to make sure it's still running with docker ps. If not, run docker-compose up again.

2. Check out vector-datasource and switch to the iandees/dockerize branch.

3. Run docker build -t mapzen/vector-datasource . to create an image locally to be used in the next step

4. Run the following to load in data from a New York metro extract:

   docker run \
   --link tileserver_postgis_1:postgres \
   --net tileserver_default \
   --env-file ../tileserver/.env \
   -it --rm --name vector-datasource-loader \
   mapzen/vector-datasource
   

5. You should get tiles at http://localhost:8000/preview.html

[bcamper]

I have tiles! 🎊 Thanks!

[iandees]

I'm going to merge this in and start seeking feedback on the Docker info.