Merge branch 'kingfisher-process-deploy' of github.com:open-contracti…

…ng/deploy into kingfisher-process-deploy

docs/conf.py

@@ -61,7 +61,7 @@
     r'^https://(?:account|dcc)\.godaddy\.com',
     r'^https://(?:crm|redash)\.open-contracting\.org',
     r'^https://(?:docs\.google\.com/(?:document|spreadsheets)/d|drive\.google\.com/drive/folders)/',
-    r'^https://(?:ocp-library\.herokuapp|robot\.your-server|us-east-1\.console\.aws\.amazon)\.com',
+    r'^https://(?:ocp-library\.herokuapp|robot\.hetzner|us-east-1\.console\.aws\.amazon)\.com',
     r'^https://(?:postmaster|search)\.google\.com',
     r'^https://app\.(dmarcanalyzer|usefathom)\.com',
     r'^https://github\.com/open-contracting/[^/]+/issues/new',

docs/deploy/create_server.rst

@@ -281,26 +281,15 @@ Some IDs might fail (`#156 <https://github.com/open-contracting/deploy/issues/15
 #. If the server runs a database like PostgreSQL (``pg_dump``), MySQL (``mysqldump``) or Elasticsearch, copy the database
 #. If the server runs a web server like Apache or application server like uWSGI, optionally copy the log files
 
-Django applications
+Data support server
 ~~~~~~~~~~~~~~~~~~~
 
-#. Copy the ``media`` directory and the ``db.sqlite3`` file from the app's directory
-
-Kingfisher
-~~~~~~~~~~
-
-.. note::
-
-   Instructions are incomplete.
+See :doc:`data-support`.
 
-#. Adjust reserved disk space to 1% for large disks:
-
-   .. code-block:: bash
-
-      tune2fs -m 1 /dev/md2
+Django applications
+~~~~~~~~~~~~~~~~~~~
 
-#. Update the IP addresses and hostname in the ``pillar/kingfisher_replica.sls`` file
-#. Update the IP addresses in the ``pillar/tinyproxy.sls`` file, and deploy the ``docs`` service
+#. Copy the ``media`` directory and the ``db.sqlite3`` file from the app's directory
 
 OCDS documentation
 ~~~~~~~~~~~~~~~~~~

docs/deploy/data-support.rst

@@ -0,0 +1,141 @@
+Data support tasks
+==================
+
+Create a data support server
+----------------------------
+
+#. Adjust reserved disk space to 1% for large disks:
+
+   .. code-block:: bash
+
+      tune2fs -m 1 /dev/md2
+
+#. Update the IP addresses and hostname in the ``pillar/kingfisher_replica.sls`` file
+#. Update the IP addresses in the ``pillar/tinyproxy.sls`` file, and deploy the ``docs`` service
+
+Docker
+~~~~~~
+
+#. Check that ``docker.uid`` in the ``pillar/kingfisher_main.sls`` file matches the entry in the ``/etc/passwd`` file.
+
+Docker apps
+~~~~~~~~~~~
+
+#. Run migrations for :doc:`Docker apps<docker>` as the ``deployer`` user:
+
+   .. code-block:: bash
+
+      su - deployer
+
+      cd /data/deploy/kingfisher-process/
+      docker compose run --rm web python manage.py migrate
+
+      cd /data/deploy/pelican-frontend/
+      docker compose run --rm web python manage.py migrate
+
+#. :doc:`Pull new images and start new containers for each Docker app<docker>`.
+
+Pelican backend
+~~~~~~~~~~~~~~~
+
+.. warning::
+
+   A cron job updates the ``exchange_rates`` table every 12 hours. Complete this step before the next run.
+
+The initial migrations for Pelican backend are run by Salt.
+
+#. Connect to the old server, and dump the ``exchange_rates`` table:
+
+   .. code-block:: bash
+
+      sudo -i -u postgres psql -c '\copy exchange_rates (valid_on, rates, created, modified) to stdout' pelican_backend > exchange_rates.csv
+
+#. Copy the database dump to your local machine. For example:
+
+   .. code-block:: bash
+
+      rsync -avz [email protected]:~/exchange_rates.csv .
+
+#. Copy the database dump to the new server. For example:
+
+   .. code-block:: bash
+
+      rsync -avz exchange_rates.sql [email protected]:~/
+
+#. Populate the ``exchange_rates`` table:
+
+   .. code-block:: bash
+
+      psql -U pelican_backend -h localhost -c "\copy exchange_rates (valid_on, rates, created, modified) from 'exchange_rates.csv';" pelican_backend
+
+Kingfisher Collect
+~~~~~~~~~~~~~~~~~~
+
+#. Connect to the new server, switch to the ``incremental`` user, generate an SSH key pair, and get the public SSH key:
+
+      .. code-block:: bash
+
+         su - incremental
+         ssh-keygen -t rsa -b 4096 -C "incremental"
+         cat ~/.ssh/id_rsa.pub
+
+#. Add the public SSH key to the ``ssh.incremental`` list in the ``pillar/kingfisher_main.sls`` file:
+
+   .. code-block:: yaml
+
+      ssh:
+        incremental:
+          - ssh-rsa AAAB3N...
+
+#. Change ``cron.present`` to ``cron.absent`` in the ``salt/kingfisher/collect/incremental.sls`` file.
+#. :doc:`Deploy the old server and the new server<deploy>`.
+#. Connect to the old server:
+
+   #. Switch to the ``incremental`` user:
+
+      .. code-block:: bash
+
+         su - incremental
+
+   #. Stop any processes started by the cron jobs.
+   #. Dump the ``kingfisher_collect`` database:
+
+      .. code-block:: bash
+
+         pg_dump -U kingfisher_collect -h localhost -f kingfisher_collect.sql kingfisher_collect
+
+#. Connect to the new server:
+
+   #. Switch to the ``incremental`` user:
+
+      .. code-block:: bash
+
+         su - incremental
+
+   #. Copy the database dump from the old server. For example:
+
+      .. code-block:: bash
+
+         rsync -avz [email protected]:~/kingfisher_collect.sql .
+
+   #. Load the database dump:
+
+      .. code-block:: bash
+
+         psql -U kingfisher_collect -h localhost -f kingfisher_collect.sql kingfisher_collect
+
+   #. Copy the ``data`` directory from the old server. For example:
+
+      .. code-block:: bash
+
+         rsync -avz [email protected]:/home/incremental/data/ /home/incremental/data/
+
+   #. Copy the ``logs`` directory from the old server. For example:
+
+      .. code-block:: bash
+
+         rsync -avz [email protected]:/home/incremental/logs/ /home/incremental/logs/
+
+#. Remove the public SSH key from the ``ssh.incremental`` list in the ``pillar/kingfisher_main.sls`` file.
+#. Change ``cron.absent`` to ``cron.present`` in the ``salt/kingfisher/collect/incremental.sls`` file.
+#. :doc:`Deploy the new server<deploy>`.

docs/deploy/docker.rst

@@ -1,6 +1,10 @@
 Docker tasks
 ============
 
+.. note::
+
+   The commands assume Compose V2, which uses ``docker compose``. For Compose V1, use ``docker-compose`` (with hyphen).
+
 Change to the application's directory, replacing ``APP``:
 
 .. code-block:: bash
@@ -11,20 +15,20 @@ Change to the application's directory, replacing ``APP``:
 
 .. code-block:: bash
 
-   docker-compose pull
-   docker-compose up -d
+   docker compose pull
+   docker compose up -d
 
 Create a superuser:
 
 .. code-block:: bash
 
-   docker-compose run --rm web python manage.py createsuperuser
+   docker compose run --rm web python manage.py createsuperuser
 
 Migrate the database:
 
 .. code-block:: bash
 
-   docker-compose run --rm web python manage.py migrate
+   docker compose run --rm web python manage.py migrate
 
 .. note::
 
@@ -39,7 +43,7 @@ For example:
 
 .. code-block:: bash
 
-   psql -c 'SET ROLE pelican_backend' -c "\copy exchange_rates (valid_on, rates, created, modified) from '/opt/pelican-backend/exchange_rates.csv' delimiter ',' csv header;" pelican_backend
+   psql -U pelican_backend -h localhost -c "\copy exchange_rates (valid_on, rates, created, modified) from 'exchange_rates.csv';" pelican_backend
 
 Check that the ID sequence is correct:
 

docs/deploy/index.rst

@@ -18,6 +18,7 @@ Deployment Guides
    docker.rst
    docs.rst
    prometheus.rst
+   data-support.rst
    redash.rst
    redmine.rst
 

docs/deploy/redash.rst

@@ -70,7 +70,7 @@ Load the new server's database
 
    .. code-block:: bash
 
-      sudo -u postgres psql -f redash.sql redash
+      sudo -i -u postgres psql -f redash.sql redash
 
 #. Change to the non-root user and Redash directory:
 
@@ -129,8 +129,3 @@ Upgrade the Redash service
       docker-compose up -d
 
 #. Check that the new version is running by viewing the `System Status <https://redash.open-contracting.org/admin/status>`__ page and reading the *Version*. You may need to log in to Redash to access this page.
-
-Troubleshoot
-~~~~~~~~~~~~
-
-To troubleshoot ``docker-compose`` commands, add the ``--verbose`` option.

docs/develop/get_started.rst

@@ -45,7 +45,7 @@ You must first have access to three private repositories. Contact an owner of th
 
 .. tip::
 
-   To generate new SSH keys (if they do not already exist):
+   To generate an SSH key pair (if they do not already exist):
 
    .. code-block:: bash
 
@@ -88,6 +88,6 @@ This overwrites the files:
 
    On macOS, you might need to move ``Saltfile`` to ``~/.salt/Saltfile``.
 
-This script assumes your SSH keys are ``~/.ssh/id_rsa.pub`` and ``~/.ssh/id_rsa``.
+This script assumes your SSH key pair is ``~/.ssh/id_rsa.pub`` and ``~/.ssh/id_rsa``.
 
 You're now ready to :doc:`../deploy/deploy`.

docs/develop/update/docker.rst

@@ -64,6 +64,10 @@ This will:
 Add Docker Compose file
 -----------------------
 
+.. note::
+
+   The commands assume Compose V2, which uses ``docker compose``. For Compose V1, use ``docker-compose`` (with hyphen).
+
 Create an ``{app}.yaml`` file in the `salt/docker_apps/files <https://github.com/open-contracting/deploy/tree/main/salt/docker_apps/files>`__ directory. For example:
 
 .. code-block:: yaml
@@ -77,7 +81,7 @@ Validate the file, for example:
 
 .. code-block:: bash
 
-   docker-compose config -q salt/docker_apps/files/redash.yaml
+   docker compose config -q salt/docker_apps/files/redash.yaml
 
 .. seealso::
 
@@ -89,9 +93,9 @@ Validate the file, for example:
 
 .. admonition:: One-off commands
 
-   To run a one-off command, like a database migration, use `docker-compose run <https://docs.docker.com/engine/reference/commandline/compose_run/>`__ on the command line, instead of creating a one-time container. See :doc:`../../deploy/docker` for examples.
+   To run a one-off command, like a database migration, use `docker compose run <https://docs.docker.com/engine/reference/commandline/compose_run/>`__ on the command line, instead of creating a one-time container. See :doc:`../../deploy/docker` for examples.
 
-   If you need to run a scheduled task in a cron job, use ``docker-compose run`` and redirect the output with ``2> /dev/null``, since there's no `quiet option <https://github.com/docker/compose/issues/6026>`__.
+   If you need to run a scheduled task in a cron job, use ``docker compose run`` and redirect the output with ``2> /dev/null``, since there's no `quiet option <https://github.com/docker/compose/issues/6026>`__.
 
    Confirm the meaning of a cron expression using `Cronhub <https://crontab.cronhub.io>`__.