diff --git a/docs/installation/docker-available-commands.md b/docs/installation/docker-available-commands.md new file mode 100644 index 000000000..875125e7e --- /dev/null +++ b/docs/installation/docker-available-commands.md @@ -0,0 +1,51 @@ +# Available Commands + +There's a lot of useful commands available to you from within the `isle-dc` folder. + +## Rebuild docker-compose.yml + +`make -B docker-compose.yml` will rebuild your `docker-compose.yml` file based on configuration in your `.env` file. `make demo` and `make local` will automatically do this for you, but if you change configuration at a later point in time, you'll need to run this command to see your changes reflected. + +## Config Export + +`make config-export` will export your site's configuration to your site's config sync directory (usually `config/sync` inside your Drupal root folder). + +## Config Import + +`make config-import` will import site's configuration from your site's config sync directory (usually `config/sync` inside your Drupal root folder). + +## Database Dump + +`make drupal-database-dump DEST=/your/path/dump.sql` will dump your Drupal database and place the file at `DEST`. + +## Database Import + +`make drupal-database-import SRC=/your/path/dump.sql` will import your Drupal database from the file at `SRC`. + +## Export Public Files + +`make drupal-public-files-dump DEST=/your/path/public_files.tgz` will export your public filesystem and place it as a single zipped tarball at `DEST`. + +## Import Public Files + +`make drupal-public-files-import SRC=/your/path/public_files.tgz` will import your public filesystem from a single zipped tarball at `SRC`. + +## Fcrepo Export + +`make fcrepo-export DEST=/your/path/fcrepo-export.tgz` will export your Fedora repository and place it as a single zipped tarball at `DEST` + +## Fcrepo Import + +`make fcrepo-import SRC=/your/path/fcrepo-export.tgz` will import your Fedora repository from a single zipped tarball at `SRC` + +## Reindex Fedora Metadata + +`make reindex-fcrepo-metadata` will reindex RDF metadata from Drupal into Fedora. + +## Reindex Solr +`make reindex-solr` will rebuild rebuild Solr search index for your repository. + +## Reindex the Triplestore + +`make reindex-triplestore` will reindex RDF metadata from Drupal into Blazegraph. + diff --git a/docs/installation/docker-available-configuration.md b/docs/installation/docker-available-configuration.md new file mode 100644 index 000000000..0788e6d96 --- /dev/null +++ b/docs/installation/docker-available-configuration.md @@ -0,0 +1,228 @@ +# Available Configuration + +ISLE Docker Compose has a single configuration file, `.env`. Within it, here +is all the values you can set and their effects. This list is subjectively sorted +in order of importance and usefulness. + +### ENVIRONMENT + +Setting the `ENVIRONMENT` changes how your `docker-compose.yml` file gets constructed. When you +switch from a demo to development to production environment, this is the variable to change. + +Available values for this setting are + +- `demo` - For demo environments where you do not need access to the codebase +- `local` - For development environments where you need edit the codebase +- `custom` - For production environments where your codebase gets baked into a custom container + +By default, this is set to `demo`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### DOMAIN + +What domain your Islandora site will be available at. If you are deploying anywhere other than +your personal machine, you'll probably want to change this. + +By default, this is set to `islandora.traefik.me`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +!!! Warning "Changing this after installation has consequences" + If you are indexing RDF metadata in Fedora and/or a triplestore, please be aware of the implications of changing this once you have installed. By changing your domain, all of the subjects of your triples will be incorrect and you will have to reindex. There are `make` commands to streamline this proccess, but if you have lots of data it can potentially be costly in terms of time and effort. + +### REPOSITORY + +Repository to use for pulling isle-buildkit images. + +By default, this is set to `islandora`. Change to `local` to use images you have built locally with isle-buildkit, +or use your custom docker registry if you have set up one. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### TAG + +The version of the `isle-buildkit` images to use. Non `isle-buildkit` images have their versions specified explicitly +in their respective docker-compose files. + +By default, this is set to the latest stable release of `isle-buildkit`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### USE_SECRETS + +Whether or not you want to use secrets. For demo and development environments, secrets are +not required. They are essential if you are running a production environment, though. + +Secrets are contained in the `secrets` folder withiin your `isle-dc` installation. Each file represents +an individual secret, and its contents are the value you're trying to protect. + +Available values for this setting are +- `true` +- `false` + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### DRUPAL_INSTALL_PROFILE + +Which install profile to use when making an initial installation. Valid values for this setting are the machine +names of any Drupal profile. + +By default, this is set to `standard`. + +### INSTALL_EXISTING_CONFIG + +Set this to `true` if you want to install an existing Drupal site whose configuration was exported with +`drush config:export` + +Available values for this setting are +- `true` +- `false` + +By default, this is set to `false`. + +If you set this to `true`, be sure to set `DRUPAL_INSTALL_PROFILE` to `minimal`. + +### INCLUDE_WATCHTOWER_SERVICE + +Whether or not to include Watchtower as a service. When developing `isle-buildkit`, this is extremely useful and +will auto-deploy new containers as you make changes. You should _not_ use watchtower on prodution environments, though. + +Available values for this setting are +- `true` +- `false` + +By default, this is set to `true`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### INCLUDE_ETCD_SERVICE + +Whether or not to include `etcd` as a service. + +Available values for this setting are +- `true` +- `false` + +By default, this is set to `false`. If you don't know what `etcd` is, then leave this be. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### INCLUDE_CODE_SERVER_SERVICE + +Whether or not to include the `coder` IDE as a service. If you're developing on Islandora, this can +be pretty useful when developing. You should _not_ deploy this service on production environments. + +Available values for this setting are +- `true` +- `false` + +By default, this is set to `false`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### DRUPAL_DATABASE_SERVICE + +Which database engine to use for Drupal. + +Available values are +- mariadb +- postgres + +By default, this value is set to `mariadb`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +!!! Warning "Changing this after installation has consequences" + If you are changing from mariadb to potsrgres or vice versa, you _MUST_ migrate your data yourself. ISLE will not convert your database from one to the other, and it's generally not advised to change this once you've installed. + +### FCREPO_DATABASE_SERVICE + +Which database engine to use for Fedora. + +Available values are +- mariadb +- postgres + +By default, this value is set to `mariadb`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +!!! Warning "Changing this after installation has consequences" + If you are changing from mariadb to potsrgres or vice versa, you _MUST_ migrate your data yourself. ISLE will not convert your database from one to the other, and it's generally not advised to change this once you've installed. + +### COMPOSE_HTTP_TIMEOUT + +Sometimes when bringing up containers, you can encounter timeouts with an error like this: + +``` +ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information. +If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: XXX). +``` + +By default, this value is set to 480, but if you have slow network conditions and encounter this error, consider bumping it higher. + +### COMPOSE_PROJECT_NAME + +Used for naming services in traefik as well as defining network alias and urls. For example the `drupal` service will be found at +`${COMPOSE_PROJECT_NAME}_drupal_1`. This is useful to change if you are running mulitple ISLE instances on one machine. + +By default, this value is set to `isle-dc`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### COMPOSE_DOCKER_CLI_BUILD + +Allows building your custom image when setting `ENVIRONMENT=custom` + +By default, this is set to 1. + +You most likely will never have a need to change this. + +### DOCKER_BUILDKIT + +Instructs Docker to use buildkit when building your custom image. + +By default, this is set to 1. + +You most likely will never have a need to change this. Trust us, you want buildkit. + +### PROJECT_DRUPAL_DOCKERFILE + +The name of the Dockerfile to use after setting `ENVIRONMENT=custom`. + +By default, this is set to `Dockerfile`. + +You most likely will never have a need to change this. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + +### DISABLE_SYN + +Set this to `true` to disable JWT authentication for Fedora. This is neccessary +when performing a Fedora import using their import/export tools. + +By default, this is set to `false`. + +### FEDORA_6 + +Whether or not to use Fedora 6. If you set this to `false`, you will be given Fedora 5. +In general, unless you already have an existing site on Fedora 5, you'll want Fedora 6. + +By default, this is set to `true`. + +If you change this setting, you need to `make -B docker-compose.yml` to generate a new `docker-compose.yml` +file that contains the changes you've made. + diff --git a/docs/installation/docker-basic-usage.md b/docs/installation/docker-basic-usage.md new file mode 100644 index 000000000..e85f538ff --- /dev/null +++ b/docs/installation/docker-basic-usage.md @@ -0,0 +1,97 @@ +# Basic Usage + +After you've finished installing Islandora using ISLE, here's some useful information to keep close at hand +about running your site. + +## Important Files + +The `make` commands that you used to install Islandora will leave you with two very important files. + +| File | Purpose | +| :------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `.env` | A configuration file that is yours to customize. This file controls how the docker-compose.yml file gets generated to meet your use case.
It also allows you to set variables that make their way into the final `docker-compose.yml` file, such as your site's domain. | +| `docker-compose.yml` | A ready to run `docker-compose.yml` file based on your `.env` file. This file is considered disposable. When you change your `.env` file, you will generate a new one. | + +## Available Services + +Here's a list of all the available services. Note that there are some services over `http` and not `https`. +Those aren't meant to be exposed to the public, but internally people from your organization will want to +access them. In practice, you can restrict access to these services using firewall rules to just those who +you trust. + +| Service | Url | +| :---------- | :--------------------------------------------------------------------------------------------- | +| Drupal | [https://islandora.traefik.me](https://islandora.traefik.me) | +| Traefik | [http://islandora.traefik.me:8080](https://islandora.traefik.me:8080) | +| Fedora | [http://islandora.traefik.me:8081/fcrepo/rest](https://islandora.traefik.me:8081/fcrepo/rest) | +| Blazegraph | [http://islandora.traefik.me:8082/bigdata](https://islandora.traefik.me:8082/bigdata) | +| Activemq | [http://islandora.traefik.me:8161](https://islandora.traefik.me:8161) | +| Solr | [http://islandora.traefik.me:8983](https://islandora.traefik.me:8983) | +| Cantaloupe | [https://islandora.traefik.me/cantaloupe](https://islandora.traefik.me/cantaloupe) | +| Matomo | [https://islandora.traefik.me/matomo/](https://islandora.traefik.me/matomo/) | +| Code Server | [http://islandora.traefik.me:8443/](https://islandora.traefik.me:8443/) | + +## Basic Commands + +### Stopping Islandora + +If you want to stop Islandora, you can bring down all the containers with + +``` +docker-compose down +``` + +### Restarting Islandora + +If you want to start Islandora back up after stopping it, use + +``` +docker-compose up -d +``` + +### Deleting Islandora + +If you want to stop Islandora and delete all of its content, use + +``` +docker-compose down -v +``` + +### Regenerating docker-compose.yml + +If you make changes to configuration in the .env file, you may need to regnerate your `docker-compose.yml` file so that +those changes take effect. + +``` +make -B docker-compose.yml +``` + +Once you have a new `docker-compose.yml` file, you'll need to restart your containers that have had configuration change. +You can do this easily with + +``` +docker-compose up -d +``` + +Even if the site is up and running, that commnad will only retart the containers it needs to. + +### Listing services + +You can see a list of all the containers that you have running and their statuses by running + +``` +docker ps -a +``` + +### Tailing Logs + +You can tail logs using + +`docker-compose logs service_name` + +For example, to tail nginx logs for Drupal, use `docker-compose logs drupal`. + +If you don't know what you're looking for exactly, you can turn on the fire hose and look through all logs by dropping +the service name and simply using + +`docker-compose logs` diff --git a/docs/installation/docker-compose.md b/docs/installation/docker-compose.md deleted file mode 100644 index cae12ace5..000000000 --- a/docs/installation/docker-compose.md +++ /dev/null @@ -1,194 +0,0 @@ -# Install Islandora on Docker (ISLE) - -## Overview - -This page describes using ISLE-DC to launch Islandora as a suite of Docker Containers (ISLE). This is an alternative to the [Islandora Ansible Playbook](playbook.md), which creates the Islandora stack on a single virtual machine. At the end of this tutorial, you will have a local development environment, and be able to use Docker to launch, shut down, and edit your environment. - -### What is ISLE? - -ISLE, or ISLandora Enterprise, is a community initiative to ease the installation and maintenance of Islandora by using Docker. It was an initiative of the Islandora Collaboration Group, first using Islandora 7.x, and later with Islandora 8.x. The code for the Islandora Enterprise 8 Prototype (ISLE 8, sometimes just ISLE) is now under the purview of the Islandora Foundation. ISLE may also refer to the [ISLE system for Islandora 7](https://islandora-collaboration-group.github.io/ISLE/). - -### Why use Docker? - -[Docker](https://www.docker.com/) is a way to separate out the "state" of your site (i.e. all the content, files, and configurations that you've entered) from the underlying software that runs it (e.g. webserver, database engine, etc). This allows for easier upgrades, faster development, and more flexible deployment. - -### Where is ISLE 8? - -ISLE 8 is a suite of Docker containers that run the various components of Islandora required by Islandora Defaults: drupal, fedora, solr, alpaca, crayfish, matomo, etc. The individual containers are created (and automatically pushed to Docker Hub) by [ISLE BuildKit](https://github.com/Islandora-Devops/isle-buildkit). - -### How do I install ISLE? - -Install ISLE locally or remotely using the Docker Compose scripts (ISLE-DC) at ([Islandora-Devops/isle-dc](https://github.com/Islandora-Devops/isle-dc)). These are instructions to pull down and configure the necessary containers from Docker Hub. Unlike the [Islandora Playbook](playbook.md), this method does not require Ansible, or (for local installations) VirtualBox. Like the Playbook, it has been optimized for development environments but in theory could be used in production. - -These instructions describe using ISLE-DC locally to create a development Islandora sandbox. More detailed configuration options are described in the project's [README](https://github.com/Islandora-Devops/isle-dc). - - -## Requirements - -- A computer that can run Docker (version 19.x+) and has at least 8GB of RAM (ideally 16GB) -- An administrator account -- (Mac OS) Apple Developer Tools -- (Windows) The following setup has been tested: - - Windows 10 - - [Windows Subsystem for Linux v. 2 (WSL 2)](https://docs.microsoft.com/en-us/windows/wsl/install-win10) - - Ubuntu 20.04 running on WSL 2 - - GNU make, run `sudo apt update` and `sudo apt install make` to install - - Docker Desktop for Windows, using the WSL 2 based engine (Settings > General) and with the WSL integration active for Ubuntu (Settings > Resources > WSL integration) -- If using `make local`, see the [README](https://github.com/Islandora-Devops/isle-dc) for other requirements. - - -!!! Note "What are we missing?" - Are you, or your computer, new to spinning up development sandboxes? Do you have wisdom about installing make on Windows? We suspect these requirements may be incomplete and your experience would be appreciated at [Issue #1640](https://github.com/Islandora/documentation/issues/1640). - - -## Installing Docker - -To see if you have Docker installed, type `docker --version` in a terminal. - -If you need to install Docker, we recommend using the application [Docker Desktop](https://www.docker.com/products/docker-desktop). It provides a GUI for managing Docker container in Windows and MacOS, along with the Docker engine and suite of command-line tools. Linux users don't get a desktop client, but can download the Engine and command-line tools from that same link. - -There is also a legacy project called [Docker Toolbox](https://docs.docker.com/toolbox/overview/) which may be of interest if your machine cannot run Docker Desktop, or if you already have it installed. - -[Download Docker](https://www.docker.com/products/docker-desktop) - -!!! Warning "Memory, Processors, and Swap Requirements" - To run ISLE on Docker Desktop, you must increase the resources allocated to the software. See Docker docs on [setting resources on Windows](https://docs.docker.com/docker-for-windows/#resources) (see note on how to allocate/restrict memory when using WSL 2) or [setting resources on Mac](https://docs.docker.com/docker-for-mac/#resources). - - **CPUs (processors)**: The CPUs allowed to Docker Desktop are still shared with the host machine, so increasing this to the maximum value should allow both the Docker containers and your host machine to run simultaneously. - - **Memory (RAM)**: This memory is completely dedicated to Docker while Docker Desktop is running, so do not allocate more than you can spare and still run your host machine. Windows users may not require as much memory for Docker as Mac users. Current suggestions for memory allocated to Docker Desktop are below, but please edit this document if you have new information. - - - Sandbox (`make demo`): 4GB - - Development (`make local`): 8GB - - Production or production-like development: 16GB - - **Swap**: Swap space is space borrowed from your hard disk drive to serve as makeshift RAM as needed. If you cannot provide as much RAM as you would like, increase this as is reasonable given your free disk space. - -## Launching Islandora with Docker - -**Summary:** Using git, clone the isle-dc project. In that directory, enter the command `make demo` to build the docker infrastructure and a demo site. Then use the command `docker-compose up -d` to start the containers. - -!!! hint "ISLE-DC variants: `make demo` vs `make local`" - The `make demo` command alone will spin up a sandbox-like version of ISLE, but only with a front end. The code files will be inaccessible, and you will not be able to install additional modules or themes. The `make local` command creates a full-fleged development environment. It will copy the active Drupal codebase locally in a way that is also live to the ISLE site. This method takes longer (and may require multiple retries if your internet connection is spotty) but is required if you will be testing pull requests or writing code. The `make` command alone will build the infrastructure but not install anything (including Drupal!). - -```bash -git clone https://github.com/islandora-devops/isle-dc -cd isle-dc -make demo -``` - - -Results of `make demo`: -``` -isle-dc$ make demo - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed -100 3562 100 3562 0 0 12779 0 --:--:-- --:--:-- --:--:-- 12812 - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed -100 1708 100 1708 0 0 6809 0 --:--:-- --:--:-- --:--:-- 6832 -docker-compose pull -Pulling activemq ... done -Pulling alpaca ... done -Pulling blazegraph ... done -Pulling cantaloupe ... done -Pulling fcrepo ... done -Pulling fits ... done -Pulling crayfits ... done -Pulling gemini ... done -Pulling homarus ... done -Pulling houdini ... done -Pulling hypercube ... done -Pulling mariadb ... done -Pulling matomo ... done -Pulling milliner ... done -Pulling recast ... done -Pulling solr ... done -Pulling drupal ... done -Pulling traefik ... done -Pulling watchtower ... done -``` - -!!! Fail "Troubleshooting - Docker Versions" - If you get an error such as: `ERROR: Version in "./docker-compose.activemq.yml" is unsupported.`, then you need to upgrade Docker. Enter the command `make clean` before re-attempting to `make demo`. - -!!! Fail "Troubleshooting - re-attempting `make demo`" - If `make` fails for any reason, enter `make clean` before attempting to `make` again. If not, you may see an error such as: `ERROR: Top level object in './docker-compose.yml' needs to be an object not ''.` - -!!! Fail "Troubleshooting - docker containers exit without warning" - If you notice some Docker containers drop (exited(0)), and (in Docker Desktop) the isle-dc app icon is yellow instead of green, try increasing the resources allocated to Docker (see note above). - -!!! hint "Development version - access the codebase" - If you used `make local`, then you will have a new directory in the current (isle-dc) directory named `codebase`, containing the live Drupal root folder (containing your Drupal's composer files and the web/ subdirectory). - - -Once `make demo` has successfully completed, launch the ISLE containers using `docker-compose up`. The `-d` flag allows you to return to using the command line. Without it, your shell will be stuck in the `docker-compose` process as long as the containers are running. - - -```bash -docker-compose up -d -``` - -!!! Fail "Troubleshooting - connection timed out (Mac)." - If you are using Docker Desktop for Mac, and get timeout errors when spinning up the containers (during `docker-compose up -d` or during `make local`) such as this: - - ``` - ERROR: for isle-dc_mariadb_1 UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=480) - ``` - - you can try quitting Docker completely (make sure there is no whale icon in your top toolbar - may need to select "Quit" from the whale icon itself) and then restart Docker. - -## Visiting your Islandora site - -Direct a browser to [https://islandora.traefik.me/](https://islandora.traefik.me/). If your containers are still "spinning up", you will see a white screen with the words "Bad Gateway". This often lasts 2-5 minutes, and should be shorter for subsequent launches. If it takes more than a few minutes, check to make sure that none of your containers have failed to launch (see note above on "Troubleshooting - docker containers exit without warning"). When all containers are ready, you should see a basic Drupal login screen. - -!["Bad Gateway" white screen while still loading](../assets/docker_bad_gateway_still_loading.png) - -![Drupal login screen](../assets/docker_drupal_login_screen.png) - -To log in: - -* username: **admin** -* password: **password** - -!!! Note Docker logs and Docker Compose logs - `docker logs -tf isle-dc_drupal_1` shows Docker logs for the "drupal" container, which will continue to have new log entries as long as the drupal box is still starting up. When it gets to `confd using 'env' backend`, you're done. `docker-compose logs` is like a firehose, showing the log messages from all containers. - -## Spinning down your Islandora site - -To shut down the containers without destroying your site, use `docker-compose down`. To also destroy your "state" (i.e. your content, your database, your files), use `docker-compose down -v`. - -## Editing Code in Docker - -If you used `make local` then the drupal root folder is in a new directory in the isle-dc folder named `codebase`. This is live and editable in whatever development environment you would like. If you just did `make demo`, you will need to spin down your containers with `-v` to destroy your state before starting a new one with `make local`. - -Editing code for the back-end processes (alpaca, milliner, etc) is more complicated. Please ask on the #isle Slack channel and help us improve this documentation! - -### Testing a Pull Request - -Islandora modules in the `codebase/web/contrib/modules` folder are already set up with `git` and the `origin` remote is the canonical Islandora repository. You can follow the command-line instructions for testing pull requests available on Github. When finished, don't forget to `git checkout main` (or the default branch if not named main) so you can pull new code. - -## Updating ISLE - -Documentation to come - see [Issue #121](https://github.com/Islandora-Devops/isle-dc/issues/121) - -### Drupal Updates - -to come - -### Module Updates (non-Islandora) - -to come - -### Module Updates (Islandora) - -to come - -### Backend Providers (solr, mariadb, etc) - -to come - -### Updating the whole isle-dc repo - -to come - diff --git a/docs/installation/docker-custom.md b/docs/installation/docker-custom.md new file mode 100644 index 000000000..1633c7783 --- /dev/null +++ b/docs/installation/docker-custom.md @@ -0,0 +1,130 @@ +# Installing a Production/Staging Server + +If you are deploying Islandora on a server that is publicly accessible, there's extra precautions you should take for +performance and security reasons. If you follow these steps, you'll see how we can use our `codebase` folder to build +a custom Drupal container, and bake code into the container instead of bind-mounting it in. We'll also cover how to +store passwords as secrets and set up TLS. + +## Getting Started + +If you haven't already [made a local environment](../docker-local), you'll want to do that first. These instructions build off of having +a codebase folder. + +## Using your Domain + +At this point, we're assuming that you've purchased a domain to use for your repository. By default, `islandora.traefik.me` is used, which +is fine for `make demo` and `make local`. But for your production site, you'll need to set the domain you own in your .env file. + +``` +DOMAIN=example.org +``` + +## Secrets + +Sensitive information, such as passwords, should never be built into a container. It also shouldn't ever be bind-mounted in like we +do with our codebase folder. If you use secrets, it's like bind-mounting in a file, except that file is provided from the host machine +to the container using an encrypted channel. + +To use secrets, set the following in your .env file + +``` +USE_SECRETS=true +``` + +The secrets themselves are stored in the `secrets` folder of `isle-dc`. If you navigate to that directory, you'll see several small +files, where each file represents a different password. By default, they are all set to `password` and you can change them to set your +passwords for the system. + +## TLS + +All public facing sites need to use HTTPS, and it's definitely a stumbling block for the uninitiated. Fortunately, `isle-dc` is +set up to use HTTPS by default. Even when running `make demo`, your site runs over HTTPS at `https://islandora.traefik.me`. The +default certificates are stored in the `certs` folder of `isle-dc`, and you can simply overwrite them with certificates from your +certificate authority. As long as the certificates match the `DOMAIN` variable in your `.env` file, that is. + +| File | Purpose | +| :------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `privkey.pem` | A PEM encoded private key used to sign your certificate | +| `cert.pem` | A PEM encoded certificate that also contains the issuer's certificate as well. Most certificate authorities offer "Full Chain" or "With Issuer" certificates that contain everything you need. Occassionally, you may find yourself needing to manually concatenate your certificate with the issuer certificate by hand. In that case, the certificate for your site goes first, and the issuer's certificate is appended afterwards. | + +### Using Let's Encrypt instead of Certificate Files + +Full support for Let's Encrypt is not available yet with ISLE, but will be soon. It can be done by hand though. Just understand that it takes editing your `docker-compose.yml` file, and those changes will be destroyed if you regenerate your `docker-compose.yml` file for any reason. Be sure to back up your `docker-compose.yml` file once you have things in place. + +#### The acme.json file +Create an empty `acme` folder in `isle-dc` and bind mount it into Traefik. When Traefik start up, it will write `acme.json` to this folder. Your `volumes` section for +`traefik` in your `docker-compose.yml` file should look like + +``` + volumes: + - ./certs:/etc/ssl/traefik:rw + - ./tls.yml:/etc/traefik/tls.yml:rw + - ./acme:/acme:rw +``` + +#### Creating the certificate resolver + +Add the following to the `commands` section for `traefik` to tell it to use Let's Encrypt. + +``` + --certificatesresolvers.myresolver.acme.httpchallenge=true + --certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=http + --certificatesresolvers.myresolver.acme.email=your-mail@example.org + --certificatesresolvers.myresolver.acme.storage=/acme/acme.json + --certificatesResolvers.myresolver.acme.caServer=https://acme-v02.api.letsencrypt.org/directory +``` + +Be sure to replace `your-mail@example.org` with the email address you've associated with Let's Encrypt. + +#### Adding the certificate resolver to routes + +For the Drupal, Matomo, and Cantaloupe services, you'll need to add labels to instruct Traefik to use the `myresolver` certificate resolver you just created. + +For example, for Drupal + +``` +traefik.http.routers.isle-dc-drupal_https.tls.certresolver: myresolver +``` + +#### Troubleshooting + +If you are still getting security exceptions, check what certificate is being used through your browser. Setting `--log.level=DEBUG` in the `commands` section +for `traefik` will help out greatly when debugging. You can tail the logs with `docker-compose logs -tf traefik` + +If you aren't careful, you can hit Let's Encrypt's rate limit, and you'll be locked out for up to a week! If you want to use their staging server instead +while testing things out, use + +``` + --certificatesResolvers.myresolver.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory +``` + +You'll still get security exceptions when it's working, but you should be able to check the certificate from the browser and confirm you are +getting it from the staging server. + +## Building and Deploying Your Custom Container + +First, set your `ENVIRONMENT` variable to `custom` in your .env file in addition to the changes outlined above + +``` +ENVIRONMENT=custom +USE_SECREST=true +DOMAIN=your-domain.org +``` + +Then rebuild your dockerfile to have your changes take effect + +``` +make -B docker-compose.yml +``` + +After this, you can build your custom container with + +``` +make build +``` + +You then deploy the container with + +``` +docker-compose up -d +``` diff --git a/docs/installation/docker-demo.md b/docs/installation/docker-demo.md new file mode 100644 index 000000000..3cccb02f7 --- /dev/null +++ b/docs/installation/docker-demo.md @@ -0,0 +1,24 @@ +# Installing a Demo Server + +Using ISLE, you can spin up a repository that is exactly like future.islandora.ca, including the sample content. If you want to kick the tires and see what Islandora can do with the minimal amount of setup, this is for you. + +!!! Warning "Demonstration Purposes Only!" + Please be advised, the environment you are about to create is meant to be temporary. The drupal codebase is baked into a container and is ephemeral. If you [install new modules](../docker-maintain-drupal/), they will be gone if your drupal container goes down for any reason. + +## Getting Started + +To get started with a **demo** environment, run the following command from your `isle-dc` directory: + +```bash +make demo +``` + +## Kicking the Tires + +Your new Islandora instance will be available at [https://islandora.traefik.me](https://islandora.traefik.me). Don't let the +funny url fool you, it's a dummy domain that resolves to `127.0.0.1`. + +You can log into Drupal as `admin` using the default password, `password`. + +Enjoy your Islandora instance! Check out the [basic usage documentation](../docker-basic-usage) to see +all the endpoints that are available and how to do things like start and stop Islandora. diff --git a/docs/installation/docker-download.md b/docs/installation/docker-download.md new file mode 100644 index 000000000..e69de29bb diff --git a/docs/installation/docker-introduction.md b/docs/installation/docker-introduction.md new file mode 100644 index 000000000..66809a38e --- /dev/null +++ b/docs/installation/docker-introduction.md @@ -0,0 +1,15 @@ +# Introduction to ISLE + +## What is ISLE? + +ISLE, short for ISLandora Enterprise, is a community initiative to ease the installation and maintenance of Islandora by using Docker. It was originally started by the Islandora Collaboration Group for Islandora 7.x. When transitioning to Islandora 8, the project came under the purview of the Islandora Foundation. All documentation on this site assumes you're trying to install Islandora 8. [See here](https://islandora-collaboration-group.github.io/ISLE/) if you are looking for ISLE for Islandora 7. + +## Why use ISLE? + +ISLE's architecture using [Docker](https://www.docker.com/) separates out the "state" of your site (i.e. all the content, files, and configurations that you've entered) from the underlying software that runs it (e.g. webserver, database, microservices, etc). This allows for easier upgrades, faster development, and more flexible deployment. It is hands down the easiest way to install, run, and maintain and Islandora instance. + +## Where is ISLE? + +ISLE is a suite of Docker containers that run the various components of Islandora: drupal, fedora, solr, alpaca, crayfish, matomo, etc. The individual containers are created (and automatically pushed to [Docker Hub](https://hub.docker.com/u/islandora)) by [ISLE BuildKit](https://github.com/Islandora-Devops/isle-buildkit). + +In order to deploy the containers, however, you need to use a container orchestration tool. The ISLE project provides tools for running and maintaining the containers using docker-compose with [ISLE Docker Compose](https://github.com/Islandora-Devops/isle-dc). diff --git a/docs/installation/docker-local.md b/docs/installation/docker-local.md new file mode 100644 index 000000000..73ea426db --- /dev/null +++ b/docs/installation/docker-local.md @@ -0,0 +1,44 @@ +# Installing a Development Server + +When developing locally, your Drupal site resides in your `isle-dc/codebase` folder and is bind-mounted into your +Drupal container. This lets you update code using the IDE of your choice on your host machine, and the +changes are automatically reflected on the Drupal container. + +## Getting Started + +If you don't already have a Drupal site, you'll be given a basic setup using Drupal 9 and the +[Islandora install profile](https://github.com/islandora-devops/islandora_profile). + +If you do already have a Drupal site, use git to clone it into place as the `codebase` folder. + +``` +cd /path/to/isle-dc +git clone https://github.com/your_org/your_repo codebase +``` + +Now you'll need to tell `isle-dc` to look for it by setting the `ENVIRONMENT` variable in +your `.env` file. If you don't have one, copy over `sample.env` and name it `.env`. Then +set + +``` +ENVIRONMENT=local +``` + +If your site includes exported configuration from `drush config:export`, then you'll also +need to set + +``` +INSTALL_EXISTING_CONFIG=true +DRUPAL_INSTALL_PROFILE=minimal +``` + +Once you are ready, run + +```bash +make local +``` + +to install the Drupal site in your `codebase` folder and spin up all the other containers with it. + +Enjoy your Islandora instance! Check out the [basic usage documentation](../docker-basic-usage) to see +all the endpoints that are available and how to do things like start and stop Islandora. diff --git a/docs/installation/docker-maintain-drupal.md b/docs/installation/docker-maintain-drupal.md new file mode 100644 index 000000000..e9d2460e2 --- /dev/null +++ b/docs/installation/docker-maintain-drupal.md @@ -0,0 +1,61 @@ +# Maintaining Your Drupal Site + +Drupal has a dedicated security team, and regularly produces updates to address security issues +that are discovered. You should always keep your Drupal site up to date so that you are +protected against known vulnerabilities. Updating Drupal can be intimidating, but we have +step-by-step instructions to help you do it safely. + +## Running commands + +Once you have a `codebase` folder, how do you maintain it and keep it up to date? The quick answer is +"the same way you maintain any Drupal site with Composer and Drush"... with one small caveat. You most +likely do not have PHP or Composer on your machine, and even if you do, you want to make sure you're +using the exact same version that Islandora is using. So to ensure all the versions of things line +up, we use Docker to execute Drush and Composer from the Drupal container. The general template for +running a command looks like this: + +``` +docker-compose exec -T drupal with-contenv bash -lc 'YOUR COMMAND' +``` + +You can also just shell into the drupal container and run commands as well, +just be aware that if you cycle your container for any reason, you'll lose your +bash history. If you want to shell in to run commands, drop the `-lc 'YOUR COMMAND'` +bit. + +``` +docker-compose exec -T drupal with-contenv bash +``` + +## Updating your Drupal Site + +Use composer to update your site's modules and their dependencies. The working directory +of the drupal container is the Drupal root (a.k.a. `codebase`), so you don't need to `cd` +into any other directory before running the command. The following command will update +all modules and their dependencies that are not pinned to specific versions. + +``` +docker-compose exec -T drupal with-contenv bash -lc 'composer update -W' +``` + +After getting the newest code, you'll want to use Drush to update the Drupal database +and run any other update hooks that have been introduced. However, _YOU SHOULD BACK UP +YOUR DATABASE BEFORE GOING ANY FURTHER_. You never know when something will go wrong and +you don't want to be stuck with an unusable database and no plan B. + +``` +make drupal-database-dump DEST=/path/to/dump.sql +``` + +Now you can safely update the Drupal database with Drush via + +``` +docker-compose exec -T drupal with-contenv bash -lc 'drush updb' +``` + +If for any reason, something goes wrong, you can Restore the Drupal database at any time by running + +``` +make drupal-database-import SRC=/path/to/dump.sql +``` + diff --git a/docs/installation/docker-maintain-isle.md b/docs/installation/docker-maintain-isle.md new file mode 100644 index 000000000..f28556fbc --- /dev/null +++ b/docs/installation/docker-maintain-isle.md @@ -0,0 +1,49 @@ +# Maintaining Your ISLE Infrastructure + +You will regularly be updating your Drupal site as security patches and module updates are released. +Less often, you will also need to update the rest of your Islandora installation. ISLE makes this easy. +In fact, it was specifically designed to streamline this process. + +Since Islandora is not a single piece of software, but instead many pieces of software working together +in concert, maintaining all of it is a daunting task. There's nginx, tomcat, karaf, etc... Then there's +everything needed for the authentication layer and JWT keys. Plus there's all the microservices. You can +see that all this adds up to a significant maintenance burden. + +Now imagine if all that time and effort spent on security updates and getting the newest versions could +be boiled down to a handful of simple commands. That's exactly what ISLE does! + +## Updating ISLE + +Updating ISLE is easy. When a new release is made available, you update the `TAG` variable in your +`.env` file to the latest version. Suppose you are on ISLE 1.0.0, and ISLE 1.1.0 has been released. +Then we would set + +``` +TAG=1.1.0 +``` + +We'll then generate a new `docker-compose.yml` file that includes the new tag with + +``` +make -B docker-compose.yml +``` + +After that, we pull the new containers with + +``` +make pull +``` + +And finally we deploy the updated containers by running + +``` +docker-compose up -d +``` + +You can check that everything is running at the version you've specified with + +``` +docker ps -a +``` + +The version that's running can be confirmed by looking at the `IMAGE` column in the output. diff --git a/docs/installation/docker-prereq.md b/docs/installation/docker-prereq.md new file mode 100644 index 000000000..5ce9d8375 --- /dev/null +++ b/docs/installation/docker-prereq.md @@ -0,0 +1,47 @@ +# Prerequisites + +## Requirements for using ISLE with Docker Compose + +- Docker 19.x+ +- Docker Compose version 1.25.x+ +- GNU Make 4.0+ +- Git 2.0+ +- [ISLE Docker Compose](https://github.com/islandora-devops/isle-dc) +- At least 8GB of RAM (ideally 16GB) +- An administrator account your machine (a.k.a. the host machine) +- (Mac OS) Apple Developer Tools +- (Windows) The following setup has been tested: + - Windows 10 + - [Windows Subsystem for Linux v. 2 (WSL 2)](https://docs.microsoft.com/en-us/windows/wsl/install-win10) + - Ubuntu 20.04 running on WSL 2 + - GNU make, run `sudo apt update` and `sudo apt install make` to install + - Docker Desktop for Windows, using the WSL 2 based engine (Settings > General) and with the WSL integration active for Ubuntu (Settings > Resources > WSL integration) + +!!! Note "What are we missing?" + Are you, or your computer, new to spinning up development sandboxes? Do you have wisdom about installing make on Windows? We suspect these requirements may be incomplete and your experience would be appreciated at [Issue #1640](https://github.com/Islandora/documentation/issues/1640). + +## Installing Docker + +To see if you have Docker installed, type `docker --version` in a terminal. + +If you need to install Docker, we recommend using the application [Docker Desktop](https://www.docker.com/products/docker-desktop). It provides a GUI for managing Docker container in Windows and MacOS, along with the Docker engine and suite of command-line tools. Linux users don't get a desktop client, but can install the Engine and command-line tools the instructions [here](https://docs.docker.com/engine/install/). + +!!! Warning "Memory, Processors, and Swap Requirements" + To run ISLE on Docker Desktop, you must increase the resources allocated to the software. See Docker docs on [setting resources on Windows](https://docs.docker.com/docker-for-windows/#resources) (see note on how to allocate/restrict memory when using WSL 2) or [setting resources on Mac](https://docs.docker.com/docker-for-mac/#resources). + + **CPUs (processors)**: The CPUs allowed to Docker Desktop are still shared with the host machine, so increasing this to the maximum value should allow both the Docker containers and your host machine to run simultaneously. + + **Memory (RAM)**: This memory is completely dedicated to Docker while Docker Desktop is running, so do not allocate more than you can spare and still run your host machine. Windows users may not require as much memory for Docker as Mac users. Current suggestions for memory allocated to Docker Desktop are below, but please edit this document if you have new information. + + - Sandbox or development environment: 8GB + - Production or production-like development: 16GB + + **Swap**: Swap space is space borrowed from your hard disk drive to serve as makeshift RAM as needed. If you cannot provide as much RAM as you would like, increase this as is reasonable given your free disk space. + +## Installing ISLE Docker Compose + +Use Git to install the ISLE Docker Compose tool: + +`git clone https://github.com/islandora-devops/isle-dc` + +Tagged versions are available [here](https://github.com/Islandora-Devops/isle-dc/tags). diff --git a/docs/installation/docker-troubleshooting.md b/docs/installation/docker-troubleshooting.md new file mode 100644 index 000000000..e4a692326 --- /dev/null +++ b/docs/installation/docker-troubleshooting.md @@ -0,0 +1,25 @@ +# Troubleshooting + +### Docker Versions + +If you get an error such as: `ERROR: Version in "./docker-compose.activemq.yml" is unsupported.`, then you need to upgrade Docker. Enter the command `make clean` before re-attempting to `make demo`. + +### Re-attempting `make demo` + +If `make` fails for any reason, enter `make clean` before attempting to `make` again. If not, you may see an error such as: `ERROR: Top level object in './docker-compose.yml' needs to be an object not ''.` + +### Docker containers exit without warning + +If you notice some Docker containers drop (exited(0)), and (in Docker Desktop) the isle-dc app icon is yellow instead of green, try increasing the resources allocated to Docker (see note above). + +### Connection timed out (Mac). + +If you are using Docker Desktop for Mac, and get timeout errors when spinning up the containers (during `docker-compose up -d` or during `make local`) such as this: + +``` +ERROR: for isle-dc_mariadb_1 UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=480) +``` + +you can try quitting Docker completely (make sure there is no whale icon in your top toolbar - may need to select "Quit" from the whale icon itself) and then restart Docker. +### 504 Bad Gateway +If you get a white screen with a 504 Bad Gateway error, this means your containers haven't finished initializing themselves yet. If you've waiting an appropriate amount of time (2-5 minutes), then there is most likely an error in a container's startup script. Use `docker ps -a` to see which services have `Exited` status, and then tail their logs with `docker-compose logs service_name`. diff --git a/mkdocs.yml b/mkdocs.yml index 85e730bef..16b9a16f1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -67,7 +67,20 @@ nav: # particular recipe. The ISLE documentation page was written with a # 'non-technical' user in mind who wants to test. The Manual installation # guides mention that users will need additional knowledge about server administration. - - 'Docker Compose (ISLE-DC)': 'installation/docker-compose.md' + - 'Docker (ISLE)': + - 'Introduction to ISLE': 'installation/docker-introduction.md' + - 'Prerequisites': 'installation/docker-prereq.md' + - Install Islandora: + - 'Installing a Demo Server': 'installation/docker-demo.md' + - 'Installing a Development Server': 'installation/docker-local.md' + - 'Installing a Staging/Production Server': 'installation/docker-custom.md' + - Maintaining Islandora: + - 'Basic Usage': 'installation/docker-basic-usage.md' + - 'Maintaining Drupal': 'installation/docker-maintain-drupal.md' + - 'Maintaining ISLE': 'installation/docker-maintain-isle.md' + - 'Available Commands': 'installation/docker-available-commands.md' + - 'Available Configuration': 'installation/docker-available-configuration.md' + - 'Troubleshooting' : 'installation/docker-troubleshooting.md' - 'Ansible Playbook': 'installation/playbook.md' - Manual Installation: - 'Introduction': 'installation/manual/introduction.md'