From fad341d58f003ba5859e06e6b4cc23030d8c6caa Mon Sep 17 00:00:00 2001 From: David Schneider Date: Mon, 3 Dec 2018 14:41:29 -0800 Subject: [PATCH 1/8] Add top-level README. --- README.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..cb6a8b2 --- /dev/null +++ b/README.md @@ -0,0 +1,57 @@ +# cartodb-docker +### containerization resources for CartoDB (aka Carto) + +This repository provides Docker images for each of the services that comprise [Carto](https://carto.com), so that the application can be deployed in a containerized environment. The setup of each service is based on [the development documentation](https://cartodb.readthedocs.io) from Carto. We recommend you consult that document for up-to-date details on how to use the application. + +There are subdirectories corresponding to each of the Carto services: +- `editor`: the Carto web application (https://github.com/CartoDB/cartodb) +- `mapsapi`: map tile API server (https://github.com/CartoDB/Windshaft-cartodb) +- `sqlapi`: API server allowing arbitrary SQL queries against the database (https://github.com/CartoDB/CartoDB-SQL-API) +- `postgis`: PostGIS database server with Carto extensions (https://github.com/CartoDB/cartodb-postgresql) +- `redis`: data store used by other Carto components; an off-the-rack Redis image configured to persist data on disk + +In addition, we supply a simple nginx reverse-proxy server image (in the `router` subdirectory), configured to allow access to all Carto endpoints via a single base URL, and a sample `docker-compose.yml` as an example of how these services may be linked together to run the whole application. + +See the `README.md` file in each subdirectory for details on using the various Docker images. + +## Usage + +To run this application, you need to have a recent version of Docker installed. To spin up a bare-bones implementation locally, with default configuration, simply run this command from the project root: +``` +docker-compose -d up +``` + +In a more real-world scenario, you'll want to: +1. provide some custom configuration +2. deploy the application in an environment where the containers can be distributed across muliple logical hosts + +To configure Carto to your specific needs, at a minimum you'll need to supply your own `docker-compose.yml`, or whatever configuration file(s) are needed by your container orchestration engine. That would allow options like: +- configuring containers with custom environment variables +- configuring containers with a custom `command` and/or `entrypoint` +- changing the port on which the application is exposed to outside traffic +- attaching volumes to persist data beyond the life of the containers +- add your own containerized application(s) to consume the Carto services +- and more + +## Why use this incarnation of Carto? + +If you're reading this, perhaps you already have an idea why you want to use Carto. If not, have a look at the [Carto homepage](https://carto.com) for a description of the platform and what it's useful for. + +The Carto team provides [detailed instructions](https://cartodb.readthedocs.io/en/latest/install.html) for installing the system. These instructions assume you're installing all the components on a single logical host, though, and no resources are provided for containerization. For those who want a containerized Carto, several open-source solutions do exist. Why then use this one? + +As of this writing, most of the alternative solutions put the entire application in a single image. That certainly makes setup simpler, but it seems contrary to the container paradigm, where each process should ideally inhabit a separate container. Packing a PostGIS database server, a Redis instance, two Node.JS API servers, and a full Ruby on Rails web application into a single image has several disadvantages: +- the image is truly massive +- services can't be distributed or replicated across multiple hosts, limiting performance and scalability +- the system becomes a monolith in which all the parts are entangled + +This is of course not the appropriate place to detail the tradeoffs between a monolithic architecture versus a microservices approach, but the container paradigm is certainly geared towards the latter. This repo attempts to package Carto as an idiomatic containerized application, in which each service inhabits a separate container. + +We've also taken pains to make the images configurable and extensible. Most allow you to configure a container at runtime by modifying the `command` and/or `entrypoint`, and more advanced configuration changes can be accomplished simply by extending these images and adding one or more configuration files to your derived image. Again, see the `README.md` in each subdirectory for details. + +## Versioning and image tags + +Each image in this repo is versioned separately. In each subdirectory you'll find a file called `VERSION` that records two fields: +- `SOFTWARE_VERSION`: the version of the underlying piece of software we're containerizing +- `IMAGE_VERSION`: the version of the image itself, reflecting the contents of the Dockerfile and other resources in this repo used to create the image + +Each image tag uses the format: `-`. From 6f815c55231d1f9c3467a76d1caa1230772d722f Mon Sep 17 00:00:00 2001 From: David Schneider Date: Mon, 3 Dec 2018 15:33:02 -0800 Subject: [PATCH 2/8] Add README for Redis image. --- redis/README.md | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 redis/README.md diff --git a/redis/README.md b/redis/README.md new file mode 100644 index 0000000..4030bcf --- /dev/null +++ b/redis/README.md @@ -0,0 +1,6 @@ +# cartodb-redis +### Redis cache server for cartodb-docker + +This image lightly extends the [official Redis Docker image](https://hub.docker.com/_/redis/), configuring it to persist data on disk. Note, however, that you'll need to mount a volume on which to store this data (using the container path `/data`). The sample `docker-compose.yml` provided in this repo doesn't do so, because we don't know where you want to store the data. + +Further configuration can be passed to the Redis server by supplying a custom `command`. Existing configuration can be overriden by supplying a custom `entrypoint`. From 5152b676be87c3bc02bab5695341de0eb583ccd5 Mon Sep 17 00:00:00 2001 From: David Schneider Date: Mon, 3 Dec 2018 15:52:41 -0800 Subject: [PATCH 3/8] Add README for router image. --- router/README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 router/README.md diff --git a/router/README.md b/router/README.md new file mode 100644 index 0000000..66b350d --- /dev/null +++ b/router/README.md @@ -0,0 +1,14 @@ +# cartodb-router +### reverse proxy server for cartodb-docker + +This image extends the [official nginx Docker image](https://hub.docker.com/_/nginx/), configuring it to route Carto web traffic to the appropriate services. This simplifies requests, as all requests to Carto can use the same base URL and port. + +This server is not a part of Carto itself; it's just a router to make requests simpler. Feel free to modify it or replace it as your needs dictate. You could for instance add routing to your own custom web application that consumes Carto services. Or you could swap out this nginx server in favor of Varnish if you want an HTTP cache. + +## Routing + +| Service | URL | +| ------------ | -------------------- | +| Maps API | [baseURL]/api/v1/map | +| SQL API | [baseURL]/api/v2/sql | +| Carto Editor | [baseURL]/ | From 58f95194fc4abc922ac001c443138ff582575757 Mon Sep 17 00:00:00 2001 From: David Schneider Date: Tue, 4 Dec 2018 10:41:27 -0800 Subject: [PATCH 4/8] Add README for postgis image. --- postgis/README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 postgis/README.md diff --git a/postgis/README.md b/postgis/README.md new file mode 100644 index 0000000..9b6a33e --- /dev/null +++ b/postgis/README.md @@ -0,0 +1,12 @@ +# cartodb-postgis +### PostGIS database server for cartodb-docker + +This image contains a PostGIS database server with extensions specific to Carto, taken from the [cartodb-postgresql](https://github.com/CartoDB/cartodb-postgresql) GitHub repo. Its image tag references the corresponding version of that repo. + +## Data Persistence + +If you want to preserve the contents of the database beyond the life of the container itself (recommended), you'll need to mount a volume to the data directory (`/var/lib/postgresql/data` by default). The sample `docker-compose.yml` provided in this repo doesn't do so, because we don't know where you want to store the data. + +## Configuration + +There are several options for passing custom configuration to the server. The simplest option is to specify command-line flags to the `postgres` executable by passing them as the `command` to the container (`entrypoint` already invokes the executable itself via a startup script). We use the official [Postgres Docker image](https://hub.docker.com/_/postgres/) as a parent image. Consult the documentation for that image for details on how to further configure the server. From e7e4c977e7e9930915622fa10fe8038a1a419e5c Mon Sep 17 00:00:00 2001 From: David Schneider Date: Tue, 4 Dec 2018 11:17:19 -0800 Subject: [PATCH 5/8] Add READMEs for mapsapi and sqlapi. --- mapsapi/README.md | 17 +++++++++++++++++ sqlapi/README.md | 17 +++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 mapsapi/README.md create mode 100644 sqlapi/README.md diff --git a/mapsapi/README.md b/mapsapi/README.md new file mode 100644 index 0000000..6e9ae2d --- /dev/null +++ b/mapsapi/README.md @@ -0,0 +1,17 @@ +# cartodb-sqlapi +### Maps API for cartodb-docker + +This image contains the [Carto Maps API](https://github.com/CartoDB/Windshaft-cartodb), a map tile server. + +## Configuration + +The server can be configured via a `.js` file in a specific format. Carto provides several such files as examples: +- [development.js.example](https://github.com/CartoDB/Windshaft-cartodb/blob/master/config/environments/development.js.example) +- [production.js.example](https://github.com/CartoDB/Windshaft-cartodb/blob/master/config/environments/production.js.example) +- [staging.js.example](https://github.com/CartoDB/Windshaft-cartodb/blob/master/config/environments/staging.js.example) +- [test.js.example](https://github.com/CartoDB/Windshaft-cartodb/blob/master/config/environments/test.js.example) + +By default this image uses `development.js.example`, but you can choose a different file simply by specifying its name as the `command` to the container. Alternatively, you can supply your own configuration file (or files). To do so, extend this image by `COPY`ing a custom config file into `/Windshaft-cartodb/config/environments/` +and pass the name of your file(s) as the `command` to the container. If you supply multiple filenames, the configuration objects they contain will be merged. Note that with respect to merging, files earlier in the list take precedence over files later in the list. + +Note also that we supply some additional configuration, in the file `base.config.js`, to make this image work together with the other `cartodb-docker` images. These configuration settings have the highest precedence, overriding the same fields in any other configuration files supplied. If you need to override these settings, therefore, you'll need to override the container's `entrypoint`. diff --git a/sqlapi/README.md b/sqlapi/README.md new file mode 100644 index 0000000..07a613b --- /dev/null +++ b/sqlapi/README.md @@ -0,0 +1,17 @@ +# cartodb-sqlapi +### SQL API for cartodb-docker + +This image contains the [Carto SQL API](https://github.com/CartoDB/CartoDB-SQL-API), which allows making SQL queries against the PostGIS database via a REST API. + +## Configuration + +The server can be configured via a `.js` file in a specific format. Carto provides several such files as examples: +- [development.js.example](https://github.com/CartoDB/CartoDB-SQL-API/blob/master/config/environments/development.js.example) +- [production.js.example](https://github.com/CartoDB/CartoDB-SQL-API/blob/master/config/environments/production.js.example) +- [staging.js.example](https://github.com/CartoDB/CartoDB-SQL-API/blob/master/config/environments/staging.js.example) +- [test.js.example](https://github.com/CartoDB/CartoDB-SQL-API/blob/master/config/environments/test.js.example) + +By default this image uses `development.js.example`, but you can choose a different file simply by specifying its name as the `command` to the container. Alternatively, you can supply your own configuration file (or files). To do so, extend this image by `COPY`ing a custom config file into `/CartoDB-SQL-API/config/environments/` +and pass the name of your file(s) as the `command` to the container. If you supply multiple filenames, the configuration objects they contain will be merged. Note that with respect to merging, files earlier in the list take precedence over files later in the list. + +Note also that we supply some additional configuration, in the file `base.config.js`, to make this image work together with the other `cartodb-docker` images. These configuration settings have the highest precedence, overriding the same fields in any other configuration files supplied. If you need to override these settings, therefore, you'll need to override the container's `entrypoint`. From 353d4f7eacf8b0f855fbbe8a5798f2ba32da41a3 Mon Sep 17 00:00:00 2001 From: David Schneider Date: Tue, 4 Dec 2018 13:13:59 -0800 Subject: [PATCH 6/8] Add README for editor. --- editor/README.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 editor/README.md diff --git a/editor/README.md b/editor/README.md new file mode 100644 index 0000000..2168c24 --- /dev/null +++ b/editor/README.md @@ -0,0 +1,33 @@ +# cartodb-editor +### Carto web application + +This image contains the [Carto web application](https://github.com/CartoDB/cartodb), used to administer the Carto system. + +## Configuration + +Upon starting the application, a new user will be created so that you can log into the system. For security, we recommend you change the default login credentials by passing the following environment variables to the container: +- DEFAULT_USER (default: username) +- PASSWORD (default: password) +- EMAIL (default: username@example.com) + +To set the public URL on which the application will be visible to web traffic, you can pass hostname and port to the container via these environment variables: +- PUBLIC_HOST (default: localhost) +- PUBLIC_PORT (default: 80) + +The application is configured using two YAML files, `app_config.yml` and `database.yml`. Carto supplies sample versions of these files, `app_config.yml.sample` and `database.yml.sample` in the (container) directory `/cartodb/config/`. You can configure the system to your liking by supplying your own YAML files. Simply extend this image by `COPY`ing custom config files into `/cartodb/config/`, +and pass the names of your files via the `command` to the container like this: + +```dockerfile +CMD [ \ + "--app_config", "my_app_config.yml", \ + "--database", "my_database.yml" \ +] +``` + +If desired, you can supply multiple filenames for both `--app_config` and `--database`. The configuration objects they contain will be merged. Note that with respect to merging, files earlier in the list take precedence over files later in the list. + +Note also that we supply some additional configuration, in the files `base.app_config.yml` and `base.database.yml`, to make this image work together with the other `cartodb-docker` images. These configuration settings have the highest precedence, overriding the same fields in any other configuration files supplied. We don't recommend you override these files, because the container startup script, `docker-entrypoint.sh`, assumes they'll be present. If you do replace them, you'll likely need to modify the startup script too or else supply your own. + +# Usage + +The container generally takes a few minutes to initialize the system after starting up. Once it has finished initializing (watch the container log for the message "Starting the application..."), you should be able to access it from a web browser via the URL `PUBLIC_HOST:PUBLIC_PORT`, filling in the values you supplied for the corresponding environment variables (NB: if `PUBLIC_PORT` is 80, you can just use the URL `PUBLIC_HOST`). At the login prompt, enter the credentials you supplied to the container as `DEFAULT_USER` and `PASSWORD`. Now you can use Carto! From a6c226d244effd9250d0c7ac884e5b01d30f2cf2 Mon Sep 17 00:00:00 2001 From: David Schneider Date: Tue, 4 Dec 2018 14:22:20 -0800 Subject: [PATCH 7/8] Fix incorrect title in mapsapi/README.md (copy-paste error). --- mapsapi/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mapsapi/README.md b/mapsapi/README.md index 6e9ae2d..2cc7015 100644 --- a/mapsapi/README.md +++ b/mapsapi/README.md @@ -1,4 +1,4 @@ -# cartodb-sqlapi +# cartodb-mapsapi ### Maps API for cartodb-docker This image contains the [Carto Maps API](https://github.com/CartoDB/Windshaft-cartodb), a map tile server. From 5789443dfa835498ba3bac118742c7a934b9c0cf Mon Sep 17 00:00:00 2001 From: David Schneider Date: Tue, 4 Dec 2018 14:22:41 -0800 Subject: [PATCH 8/8] Add links to the images on DockerHub. --- README.md | 10 ++++++++++ editor/README.md | 2 ++ mapsapi/README.md | 2 ++ postgis/README.md | 2 ++ redis/README.md | 2 ++ router/README.md | 2 ++ sqlapi/README.md | 2 ++ 7 files changed, 22 insertions(+) diff --git a/README.md b/README.md index cb6a8b2..5b3c5e9 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,16 @@ In addition, we supply a simple nginx reverse-proxy server image (in the `router See the `README.md` file in each subdirectory for details on using the various Docker images. +## Images + +The images are available on DockerHub: +- [ihme/cartodb-editor](https://hub.docker.com/r/ihme/cartodb-editor/) +- [ihme/cartodb-mapsapi](https://hub.docker.com/r/ihme/cartodb-mapsapi/) +- [ihme/cartodb-sqlapi](https://hub.docker.com/r/ihme/cartodb-sqlapi/) +- [ihme/cartodb-postgis](https://hub.docker.com/r/ihme/cartodb-postgis/) +- [ihme/cartodb-redis](https://hub.docker.com/r/ihme/cartodb-redis/) +- [ihme/cartodb-router](https://hub.docker.com/r/ihme/cartodb-router/) + ## Usage To run this application, you need to have a recent version of Docker installed. To spin up a bare-bones implementation locally, with default configuration, simply run this command from the project root: diff --git a/editor/README.md b/editor/README.md index 2168c24..ae0d183 100644 --- a/editor/README.md +++ b/editor/README.md @@ -1,6 +1,8 @@ # cartodb-editor ### Carto web application +`ihme/cartodb-editor` image on DockerHub: https://hub.docker.com/r/ihme/cartodb-editor/ + This image contains the [Carto web application](https://github.com/CartoDB/cartodb), used to administer the Carto system. ## Configuration diff --git a/mapsapi/README.md b/mapsapi/README.md index 2cc7015..b6e2e25 100644 --- a/mapsapi/README.md +++ b/mapsapi/README.md @@ -1,6 +1,8 @@ # cartodb-mapsapi ### Maps API for cartodb-docker +`ihme/cartodb-mapsapi` image on DockerHub: https://hub.docker.com/r/ihme/cartodb-mapsapi/ + This image contains the [Carto Maps API](https://github.com/CartoDB/Windshaft-cartodb), a map tile server. ## Configuration diff --git a/postgis/README.md b/postgis/README.md index 9b6a33e..0c2c9f3 100644 --- a/postgis/README.md +++ b/postgis/README.md @@ -1,6 +1,8 @@ # cartodb-postgis ### PostGIS database server for cartodb-docker +`ihme/cartodb-postgis` image on DockerHub: https://hub.docker.com/r/ihme/cartodb-postgis/ + This image contains a PostGIS database server with extensions specific to Carto, taken from the [cartodb-postgresql](https://github.com/CartoDB/cartodb-postgresql) GitHub repo. Its image tag references the corresponding version of that repo. ## Data Persistence diff --git a/redis/README.md b/redis/README.md index 4030bcf..e980f0c 100644 --- a/redis/README.md +++ b/redis/README.md @@ -1,6 +1,8 @@ # cartodb-redis ### Redis cache server for cartodb-docker +`ihme/cartodb-redis` image on DockerHub: https://hub.docker.com/r/ihme/cartodb-redis/ + This image lightly extends the [official Redis Docker image](https://hub.docker.com/_/redis/), configuring it to persist data on disk. Note, however, that you'll need to mount a volume on which to store this data (using the container path `/data`). The sample `docker-compose.yml` provided in this repo doesn't do so, because we don't know where you want to store the data. Further configuration can be passed to the Redis server by supplying a custom `command`. Existing configuration can be overriden by supplying a custom `entrypoint`. diff --git a/router/README.md b/router/README.md index 66b350d..5870ee4 100644 --- a/router/README.md +++ b/router/README.md @@ -1,6 +1,8 @@ # cartodb-router ### reverse proxy server for cartodb-docker +`ihme/cartodb-router` image on DockerHub: https://hub.docker.com/r/ihme/cartodb-router/ + This image extends the [official nginx Docker image](https://hub.docker.com/_/nginx/), configuring it to route Carto web traffic to the appropriate services. This simplifies requests, as all requests to Carto can use the same base URL and port. This server is not a part of Carto itself; it's just a router to make requests simpler. Feel free to modify it or replace it as your needs dictate. You could for instance add routing to your own custom web application that consumes Carto services. Or you could swap out this nginx server in favor of Varnish if you want an HTTP cache. diff --git a/sqlapi/README.md b/sqlapi/README.md index 07a613b..5c135e0 100644 --- a/sqlapi/README.md +++ b/sqlapi/README.md @@ -1,6 +1,8 @@ # cartodb-sqlapi ### SQL API for cartodb-docker +`ihme/cartodb-sqlapi` image on DockerHub: https://hub.docker.com/r/ihme/cartodb-sqlapi/ + This image contains the [Carto SQL API](https://github.com/CartoDB/CartoDB-SQL-API), which allows making SQL queries against the PostGIS database via a REST API. ## Configuration