Secure MongoDB and Redis

.vols

@@ -0,0 +1 @@
+/home/oleger/kobo-docker-data/

README.md

@@ -28,6 +28,8 @@ database. They now each have their own, separate databases.
 **If you are upgrading an existing single-database installation, you must follow [these instructions](https://community.kobotoolbox.org/t/upgrading-to-separate-databases-for-kpi-and-kobocat/7202)** to migrate the KPI tables to a new database and adjust your configuration appropriately.
 This assumes your last upgrade was **more recent** than March 4, 2019. If not, you must [upgrade your databases](#important-notice-when-upgrading-from-commit-5c2ef02-march-4-2019-or-earlier) before proceeding.
 
+If you do not want to upgrade at this time, please use the [`shared-database-obsolete`](https://github.com/kobotoolbox/kobo-docker/tree/shared-database-obsolete) branch instead.
+
 
 ## Important notice when upgrading from commit [`5c2ef02` (March 4, 2019)](https://github.com/kobotoolbox/kobo-docker/commit/5c2ef0273339bee5c374830f72e52945947042a8) or earlier
 
@@ -42,9 +44,10 @@ Below is a diagram (made with [Lucidchart](https://www.lucidchart.com)) of the c
 
 ![Diagram of Docker Containers](./doc/container-diagram.svg)
 
-### Secure your installation!
-kobo-docker **opens ports on all interfaces** to let the `frontend` containers communicate with `backend` containers.
-A firewall is **HIGHLY recommended**. You **MUST** block PostgreSQL, Redis and MongoDB ports when the server is exposed publicly.
+### Secure your installation
+`kobo-docker` does **not** expose backend container ports.
+If you want to use `kobo-docker` with separated servers (one for frontend containers, one for backend containers),
+you will need to expose ports. A firewall is **HIGHLY recommended** to grant access `frontend` containers only to `PostgreSQL`, `redis` and `MongoDB` ports.
 
 
 ## Setup procedure
@@ -68,7 +71,7 @@ Already have an existing installation? Please see below.
     - `kobo-deployments/envfiles/kobocat.txt`
         ```diff
         - KOBOCAT_BROKER_URL=amqp://kobocat: kobocat@rabbit.[internal domain name]:5672/kobocat
-        + KOBOCAT_BROKER_URL =redis://redis-main.[internal domain name]:6389/2`
+        + KOBOCAT_BROKER_URL=redis://redis-main.[internal domain name]:6389/2`
         ```
 
 2. **Load balancing and redundancy**
@@ -88,34 +91,38 @@ Already have an existing installation? Please see below.
             - Redis
 
         Docker-compose for `frontend` can be started on its own server, same thing for `backend`. Users can start as many `frontend` servers they want. A load balancer can spread the traffic between `frontend` servers.
-        kobo-docker uses (private) domain names between `frontend` and `backend`.
+        `kobo-docker` uses (private) domain names between `frontend` and `backend`.
         It's fully customizable in configuration files. Once again, [kobo-install](https://github.com/kobotoolbox/kobo-install) does simplify the job by creating the configuration files for you.
 
     2. Redundancy
         `Backend` containers not redundant yet. Only `PostgreSQL` can be configured in `Master/Slave` mode where `Slave` is a real-time read-only replica.
 
-    This is a diagram that shows how kobo-docker can be used for a load-balanced/(almost) redundant solution.
+    This is a diagram that shows how `kobo-docker` can be used for a load-balanced/(almost) redundant solution.
 
     _NB: The diagram is based on AWS infrastructure, but it's not required to host your environment there._
 
     ![aws diagram](./doc/aws-diagram.svg)
 
 ## Usage
-It's recommended to create `*.override.yml` docker-compose files to customize your environment. It makes easier to update.
+It's recommended to create `*.override.yml` docker-compose files to customize your environment. It makes easier to update. 
+Samples are provided. Remove `.sample` extension and update them to match your environment. 
 
 - `docker-compose.frontend.override.yml`
 - `docker-compose.backend.master.override.yml`
 - `docker-compose.backend.slave.override.yml` (if a postgres replica is used)
 
-1. **Start/start containers**
+1. **Start/start containers** 
+
     ```
-    $kobo-docker> docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] up -d
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] up -d
-    $kobo-docker> docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] stop
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] stop
+    # Start
+    $kobo-docker> docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml up -d  
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml up -d
+
+    # Stop
+    $kobo-docker> docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml stop  
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml stop
     ```
 
-
 2. **Backups**
 
     Automatic, periodic backups of KoBoCAT media, MongoDB, PostgreSQL and Redis can be individually enabled by uncommenting (and optionally customizing) the `*_BACKUP_SCHEDULE` variables in your envfiles.
@@ -130,10 +137,10 @@ It's recommended to create `*.override.yml` docker-compose files to customize yo
     Backups **on disk** can also be manually triggered when kobo-docker is running by executing the the following commands:
 
     ```
-    $kobo-docker> docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] exec kobocat /srv/src/kobocat/docker/backup_media.bash
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] exec mongo /bin/bash /kobo-docker-scripts/backup-to-disk.bash
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] exec -e PGUSER=kobo postgres /bin/bash /kobo-docker-scripts/backup-to-disk.bash
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] exec redis_main /bin/bash /kobo-docker-scripts/backup-to-disk.bash
+    $kobo-docker> docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml exec kobocat /srv/src/kobocat/docker/backup_media.bash
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml exec mongo /bin/bash /kobo-docker-scripts/backup-to-disk.bash
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml exec -e PGUSER=kobo postgres /bin/bash /kobo-docker-scripts/backup-to-disk.bash
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml exec redis_main /bin/bash /kobo-docker-scripts/backup-to-disk.bash
     ```
 
 2. **Restore backups**
@@ -153,15 +160,15 @@ It's recommended to create `*.override.yml` docker-compose files to customize yo
     **Start**
 
     ```
-    docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] stop nginx
+    docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml stop nginx
     docker-compose -f docker-compose.maintenance.yml up -d
     ```
 
     **Stop**
 
     ```
     docker-compose -f docker-compose.maintenance.yml down
-    docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] up -d nginx
+    docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml up -d nginx
     ```
 
     There are 4 variables that can be customized in `docker-compose.maintenance.yml`
@@ -177,11 +184,11 @@ It's recommended to create `*.override.yml` docker-compose files to customize yo
     You can confirm that your containers are running with `docker ps`.
     To inspect the log output from:
 
-     - the frontend containers, execute `docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] logs -f`
-     - the master backend containers, execute `docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] logs -f`
-     - the slaved backend container, execute `docker-compose -f docker-compose.backend.slave.yml [-f docker-compose.backend.slave.override.yml] logs -f`
+     - the frontend containers, execute `docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml logs -f`
+     - the master backend containers, execute `docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml logs -f`
+     - the slaved backend container, execute `docker-compose -f docker-compose.backend.slave.yml -f docker-compose.backend.slave.override.yml logs -f`
 
-    For a specific container use e.g. `docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] logs -f redis_main`.
+    For a specific container use e.g. `docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml logs -f redis_main`.
 
     The documentation for Docker can be found at https://docs.docker.com.
 

backups

@@ -0,0 +1 @@
+/home/oleger/kobo-docker-data/.backups/

deployments/envfiles/databases.txt

@@ -6,6 +6,11 @@
 # Please see kobocat.txt to set container variables
 KOBO_MONGO_PORT=27017
 KOBO_MONGO_HOST=mongo.domain.name
+MONGO_INITDB_ROOT_USERNAME=root
+MONGO_INITDB_ROOT_PASSWORD=kobo
+MONGO_INITDB_DATABASE=formhub
+KOBO_MONGO_USERNAME=kobo
+KOBO_MONGO_PASSWORD=kobo
 
 # Default MongoDB backup schedule is weekly at 01:00 AM UTC on Sunday.
 #MONGO_BACKUP_SCHEDULE=0 1 * * 0
@@ -20,12 +25,14 @@ KOBO_MONGO_HOST=mongo.domain.name
 # `DATABASE_URL` environment variable.
 POSTGRES_PORT=5432
 POSTGRES_HOST=postgres.domain.name
-POSTGRES_DB=kobotoolbox
 POSTGRES_USER=kobo
 POSTGRES_PASSWORD=kobo
+KC_POSTGRES_DB=kobocat
+KPI_POSTGRES_DB=koboform
 
 # Postgres database used by kpi and kobocat Django apps
-DATABASE_URL=postgis://kobo:[email protected]:5432/kobotoolbox
+KC_DATABASE_URL=postgis://kobo:[email protected]:5432/kobotoolbox
+KPI_DATABASE_URL=postgis://kobo:[email protected]:5432/kobotoolbox
 
 # Replication. Password is mandatory
 KOBO_POSTGRES_REPLICATION_USER=kobo_replication
@@ -41,3 +48,6 @@ KOBO_POSTGRES_MASTER_ENDPOINT=primary.postgres.domain.name
 #--------------------------------------------------------------------------------
 
 #REDIS_BACKUP_SCHEDULE=0 3 * * 0
+
+REDIS_SESSION_URL=redis://:[email protected]:6390/2
+REDIS_PASSWORD=kobo

deployments/envfiles/kobocat.txt

@@ -5,7 +5,7 @@ USE_X_FORWARDED_HOST=False
 DJANGO_SETTINGS_MODULE=onadata.settings.kc_environ
 ENKETO_VERSION=Express
 
-KOBOCAT_BROKER_URL=redis://redis-main.domain.name:6379/2
+KOBOCAT_BROKER_URL=redis://:kobo@redis-main.domain.name:6379/2
 KOBOCAT_CELERY_LOG_FILE=/srv/logs/celery.log
 
 #ENKETO_OFFLINE_SURVEYS=True

deployments/envfiles/kpi.txt

@@ -5,7 +5,7 @@ USE_X_FORWARDED_HOST=False
 
 ENKETO_VERSION=Express
 KPI_PREFIX=/
-KPI_BROKER_URL=redis://redis-main.domain.name:6379/1
+KPI_BROKER_URL=redis://:kobo@redis-main.domain.name:6379/1
 
 KPI_MONGO_HOST=mongo.domain.name
 

docker-compose.backend.master.override.yml.sample

@@ -0,0 +1,46 @@
+# For public, HTTPS servers.
+version: '2.2'
+
+services:
+
+  postgres:
+    #environment:
+    #  - POSTGRES_BACKUP_FROM_SLAVE=True
+    # Uncomment `ports` section if you want to expose ports (e.g. use as separated servers)
+    #ports:
+    #  - 5432:5432
+    # Comment out `networks` section below if you want to expose ports
+    networks:
+      kobo-be-network:
+        aliases:
+          - postgres.kobo.private
+
+  mongo:
+    # Uncomment `ports` section if you want to expose ports (e.g. use as separated servers)
+    #ports:
+    #  - 27017:27017
+    # Comment out section below if you want to expose ports
+    networks:
+      kobo-be-network:
+        aliases:
+          - mongo.kobo.private
+
+  redis_main:
+    # Uncomment `ports` section if you want to expose ports (e.g. use as separated servers)
+    #ports:
+    #  - 6379:6379
+    # Comment out `networks` section below if you want to expose ports
+    networks:
+      kobo-be-network:
+        aliases:
+          - redis-main.kobo.private
+
+  redis_cache:
+    # Uncomment `ports` section if you want to expose ports (e.g. use as separated servers)
+    #ports:
+    #  - 6380:6380
+    # Comment out `networks` section below if you want to expose ports
+    networks:
+      kobo-be-network:
+        aliases:
+          - redis-cache.kobo.private

docker-compose.backend.master.yml

@@ -27,3 +27,7 @@ services:
       service: redis_cache
     environment:
       - KOBO_REDIS_SERVER_ROLE=cache
+
+networks:
+  kobo-be-network:
+    driver: bridge

docker-compose.backend.slave.yml

@@ -9,19 +9,6 @@ services:
     environment:
       - KOBO_POSTGRES_DB_SERVER_ROLE=slave
 
-#  mongo:
-#    extends:
-#      file: docker-compose.primary.backend.server.yml
-#      service: mongo
-#
-#  # Adapted from https://github.com/kobotoolbox/enketo-express/blob/docker/docker-compose.yml.
-#  redis_main:
-#    extends:
-#      file: docker-compose.primary.backend.server.yml
-#      service: redis_main
-#
-#  # Adapted from https://github.com/kobotoolbox/enketo-express/blob/docker/docker-compose.yml.
-#  redis_cache:
-#    extends:
-#      file: docker-compose.primary.backend.server.yml
-#      service: redis_cache
+networks:
+  kobo-be-network:
+    driver: bridge

docker-compose.backend.template.yml

@@ -16,6 +16,7 @@ services:
       - ./postgres:/kobo-docker-scripts
     command: "/bin/bash /kobo-docker-scripts/entrypoint.sh"
     restart: on-failure
+    stop_grace_period: 5m
     # Ports should be declared in `docker-compose.backend.master.override.yml` and docker-compose.backend.slave.override.yml`
     #ports:
     #  - 5432:5432
@@ -36,14 +37,13 @@ services:
       - ./log/mongo:/srv/logs
     restart: on-failure
     command: "/bin/bash /kobo-docker-scripts/entrypoint.sh"
+    stop_grace_period: 5m
     # Ports should be declared in `docker-compose.backend.master.override.yml` and docker-compose.backend.slave.override.yml`
     #ports:
     #  - 27017:27017
 
-  # Adapted from https://github.com/kobotoolbox/enketo-express/blob/docker/docker-compose.yml.
   redis_main:
     image: redis:3.2
-    # Map our "main" Redis config into the container.
     env_file:
       - ../kobo-deployments/envfile.txt
       - ../kobo-deployments/envfiles/databases.txt
@@ -56,23 +56,25 @@ services:
       - ./redis/entrypoint.sh:/tmp/redis/entrypoint.sh:ro
       - ./log/redis_main:/var/log/redis
     restart: on-failure
+    stop_grace_period: 2m30s
     # Ports should be declared in `docker-compose.backend.master.override.yml` and docker-compose.backend.slave.override.yml`
     #ports:
     #  - 6379:6379
     sysctls:
         - net.core.somaxconn=2048
     command: "/bin/bash /tmp/redis/entrypoint.sh"
 
-  # Adapted from https://github.com/kobotoolbox/enketo-express/blob/docker/docker-compose.yml.
   redis_cache:
     image: redis:3.2
-    # Map our "cache" Redis config into the container.
+    env_file:
+      - ../kobo-deployments/envfiles/databases.txt
     volumes:
       - ./.vols/redis_cache_data/:/data/
       - ./redis/redis-enketo-cache.conf.tmpl:/etc/redis/redis.conf.tmpl:ro
       - ./redis/entrypoint.sh:/tmp/redis/entrypoint.sh:ro
       - ./log/redis_cache:/var/log/redis
     restart: on-failure
+    stop_grace_period: 2m30s
     # Ports should be declared in `docker-compose.backend.master.override.yml` and docker-compose.backend.slave.override.yml`
     #ports:
     #  - 6380:6380

docker-compose.frontend.override.yml.sample

@@ -0,0 +1,67 @@
+# For public, HTTPS servers.
+version: '3'
+
+services:
+  kobocat:
+    environment:
+      # change `ENKETO_PROTOCOL` to http if HTTPS is not used
+      - ENKETO_PROTOCOL=https
+      # `NGINX_PUBLIC_PORT` is the port used to access KoBoToolbox (e.g. `https://kc.kobotoolbox.org:<NGINX_PUBLIC_PORT>`)
+      - NGINX_PUBLIC_PORT=80
+      # Uncomment the lines below to tweak uWSGI
+      #- KC_UWSGI_WORKERS_COUNT=2
+      #- KC_UWSGI_CHEAPER_WORKERS_COUNT=1
+      #- KC_UWSGI_MAX_REQUESTS=512
+      #- KC_UWSGI_CHEAPER_RSS_LIMIT_SOFT=134217728
+    networks:
+      kobo-be-network:
+        aliases:
+          - kobocat
+          - kobocat.docker.container
+
+  kpi:
+    environment:
+      # `NGINX_PUBLIC_PORT` is the port used to access KoBoToolbox (e.g. `https://kf.kobotoolbox.org:<NGINX_PUBLIC_PORT>`)
+      - NGINX_PUBLIC_PORT=80
+      # Uncomment the lines below to tweak uWSGI
+      #- KPI_UWSGI_WORKERS_COUNT=2
+      #- KPI_UWSGI_CHEAPER_WORKERS_COUNT=1
+      #- KPI_UWSGI_MAX_REQUESTS=512
+      #- KPI_UWSGI_CHEAPER_RSS_LIMIT_SOFT=134217728
+      # Comment out the line below if HTTPS is not used
+      - SECURE_PROXY_SSL_HEADER=HTTP_X_FORWARDED_PROTO, https
+    networks:
+      kobo-be-network:
+        aliases:
+          - kpi
+          - kpi.docker.container
+
+  nginx:
+    environment:
+      # `NGINX_PUBLIC_PORT` is the port used to access KoBoToolbox (e.g. `https://kf.kobotoolbox.org:<NGINX_PUBLIC_PORT>`)
+      - NGINX_PUBLIC_PORT=80
+    ports:
+      # <proxy_port>:80 . If no proxies, `proxy_port` should be the same as `NGINX_PUBLIC_PORT`
+      - 80:80
+    networks:
+      kobo-fe-network:
+        aliases:
+          - nginx
+          # These aliases must match the concatenation of `*_PUBLIC_SUBDOMAIN` and `INTERNAL_DOMAIN_NAME`
+          # found in `../kobo-deployments/envfile.txt`
+          - kf.docker.internal
+          - kc.docker.internal
+          - ee.docker.internal
+
+  enketo_express:
+    networks:
+      kobo-be-network:
+        aliases:
+          - enketo_express
+
+networks:
+  kobo-be-network:
+    external:
+      # name: <prefix>_kobo-be-network`, where `prefix` is usually the parent
+      # folder name
+      name: kobodocker_kobo-be-network