From b913f82fed7abd1366cad6f155e7fbf2dcd34055 Mon Sep 17 00:00:00 2001 From: Josh d'Entremont Date: Tue, 21 May 2024 13:51:43 -0300 Subject: [PATCH 1/3] updated layout of docker installion pages --- .../{ => docker}/docker-introduction.md | 2 +- .../{ => docker}/docker-prereq.md | 11 +------ .../isle-dc}/docker-available-commands.md | 0 .../docker-available-configuration.md | 0 .../isle-dc}/docker-basic-usage.md | 0 .../{ => docker/isle-dc}/docker-custom.md | 0 .../{ => docker/isle-dc}/docker-local.md | 8 +++++ .../isle-dc}/docker-maintain-drupal.md | 0 .../isle-dc}/docker-maintain-isle.md | 0 .../isle-dc}/docker-troubleshooting.md | 0 .../site-template}/site-template.md | 0 mkdocs.yml | 30 ++++++++++--------- 12 files changed, 26 insertions(+), 25 deletions(-) rename docs/installation/{ => docker}/docker-introduction.md (94%) rename docs/installation/{ => docker}/docker-prereq.md (94%) rename docs/installation/{ => docker/isle-dc}/docker-available-commands.md (100%) rename docs/installation/{ => docker/isle-dc}/docker-available-configuration.md (100%) rename docs/installation/{ => docker/isle-dc}/docker-basic-usage.md (100%) rename docs/installation/{ => docker/isle-dc}/docker-custom.md (100%) rename docs/installation/{ => docker/isle-dc}/docker-local.md (89%) rename docs/installation/{ => docker/isle-dc}/docker-maintain-drupal.md (100%) rename docs/installation/{ => docker/isle-dc}/docker-maintain-isle.md (100%) rename docs/installation/{ => docker/isle-dc}/docker-troubleshooting.md (100%) rename docs/installation/{ => docker/site-template}/site-template.md (100%) diff --git a/docs/installation/docker-introduction.md b/docs/installation/docker/docker-introduction.md similarity index 94% rename from docs/installation/docker-introduction.md rename to docs/installation/docker/docker-introduction.md index 3724cef0b..7105948c8 100644 --- a/docs/installation/docker-introduction.md +++ b/docs/installation/docker/docker-introduction.md @@ -12,4 +12,4 @@ ISLE's architecture using [Docker](https://www.docker.com/) separates out the "s ISLE is a suite of Docker containers that run the various components of Islandora: drupal, fedora, solr, alpaca, crayfish, matomo, etc. The individual containers are created (and automatically pushed to [Docker Hub](https://hub.docker.com/u/islandora)) by [ISLE BuildKit](https://github.com/Islandora-Devops/isle-buildkit). -In order to deploy the containers, however, you need to use a container orchestration tool. The ISLE project provides tools for running and maintaining the containers using docker-compose with [ISLE Docker Compose](https://github.com/Islandora-Devops/isle-dc). +In order to deploy the containers, however, you need to use a container orchestration tool. The ISLE project provides tools for running and maintaining the containers using docker-compose with [ISLE Docker Compose](https://github.com/Islandora-Devops/isle-dc) or [ISLE Site Template](https://github.com/Islandora-Devops/isle-site-template). \ No newline at end of file diff --git a/docs/installation/docker-prereq.md b/docs/installation/docker/docker-prereq.md similarity index 94% rename from docs/installation/docker-prereq.md rename to docs/installation/docker/docker-prereq.md index 252a969ec..6e3f6d3aa 100644 --- a/docs/installation/docker-prereq.md +++ b/docs/installation/docker/docker-prereq.md @@ -38,7 +38,7 @@ Docker Compose is a tool to simplify the process of running multiple Docker cont ### GNU Make -Make allows us to define commands that simplify installing and maintaining our Islandora site. For a complete list of available commands see the Makefile included with ISLE. +Make allows us to define commands that simplify installing and maintaining our Islandora site. For a complete list of available commands see the Makefile included with ISLE-DC. ### Composer @@ -58,7 +58,6 @@ Drush is a command line tool for managing your Drupal site. It comes installed i - Docker Compose version 2.x+ - GNU Make 4.0+ - Git 2.0+ -- [ISLE Docker Compose](https://github.com/islandora-devops/isle-dc) - At least 8GB of RAM (ideally 16GB) - An administrator account your machine (a.k.a. the host machine) - (Mac OS) Apple Developer Tools @@ -89,11 +88,3 @@ If you need to install Docker, we recommend using the application [Docker Deskto - Production or production-like development: 16GB **Swap**: Swap space is space borrowed from your hard disk drive to serve as makeshift RAM as needed. If you cannot provide as much RAM as you would like, increase this as is reasonable given your free disk space. - -## Installing ISLE Docker Compose - -Use Git to install the ISLE Docker Compose tool: - -`git clone https://github.com/islandora-devops/isle-dc` - -Tagged versions are available [here](https://github.com/Islandora-Devops/isle-dc/tags). diff --git a/docs/installation/docker-available-commands.md b/docs/installation/docker/isle-dc/docker-available-commands.md similarity index 100% rename from docs/installation/docker-available-commands.md rename to docs/installation/docker/isle-dc/docker-available-commands.md diff --git a/docs/installation/docker-available-configuration.md b/docs/installation/docker/isle-dc/docker-available-configuration.md similarity index 100% rename from docs/installation/docker-available-configuration.md rename to docs/installation/docker/isle-dc/docker-available-configuration.md diff --git a/docs/installation/docker-basic-usage.md b/docs/installation/docker/isle-dc/docker-basic-usage.md similarity index 100% rename from docs/installation/docker-basic-usage.md rename to docs/installation/docker/isle-dc/docker-basic-usage.md diff --git a/docs/installation/docker-custom.md b/docs/installation/docker/isle-dc/docker-custom.md similarity index 100% rename from docs/installation/docker-custom.md rename to docs/installation/docker/isle-dc/docker-custom.md diff --git a/docs/installation/docker-local.md b/docs/installation/docker/isle-dc/docker-local.md similarity index 89% rename from docs/installation/docker-local.md rename to docs/installation/docker/isle-dc/docker-local.md index 746270c57..2e94e59e3 100644 --- a/docs/installation/docker-local.md +++ b/docs/installation/docker/isle-dc/docker-local.md @@ -4,6 +4,14 @@ When developing locally, your Drupal site resides in your `isle-dc/codebase` fol Drupal container. This lets you update code using the IDE of your choice on your host machine, and the changes are automatically reflected on the Drupal container. +## Installing ISLE Docker Compose + +Use Git to install the ISLE Docker Compose tool: + +`git clone https://github.com/islandora-devops/isle-dc` + +Tagged versions are available [here](https://github.com/Islandora-Devops/isle-dc/tags). + ## Getting Started If you don't already have a Drupal site, you'll be given a basic setup using Drupal 9 and the diff --git a/docs/installation/docker-maintain-drupal.md b/docs/installation/docker/isle-dc/docker-maintain-drupal.md similarity index 100% rename from docs/installation/docker-maintain-drupal.md rename to docs/installation/docker/isle-dc/docker-maintain-drupal.md diff --git a/docs/installation/docker-maintain-isle.md b/docs/installation/docker/isle-dc/docker-maintain-isle.md similarity index 100% rename from docs/installation/docker-maintain-isle.md rename to docs/installation/docker/isle-dc/docker-maintain-isle.md diff --git a/docs/installation/docker-troubleshooting.md b/docs/installation/docker/isle-dc/docker-troubleshooting.md similarity index 100% rename from docs/installation/docker-troubleshooting.md rename to docs/installation/docker/isle-dc/docker-troubleshooting.md diff --git a/docs/installation/site-template.md b/docs/installation/docker/site-template/site-template.md similarity index 100% rename from docs/installation/site-template.md rename to docs/installation/docker/site-template/site-template.md diff --git a/mkdocs.yml b/mkdocs.yml index 3d0badc5a..1e5b06535 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -111,20 +111,22 @@ nav: - '8.x-2.0 Release Notes': 'release-notes/8.x-2.0.md' - 'Older Release Notes': 'https://github.com/Islandora/documentation/tree/main/docs/release_notes' - 'Install a Demo': 'installation/install-a-demo.md' - - 'ISLE-DC': - - 'Introduction to ISLE': 'installation/docker-introduction.md' - - 'Prerequisites': 'installation/docker-prereq.md' - - Install Islandora: - - 'Installing a Development Server': 'installation/docker-local.md' - - 'Installing a Staging/Production Server': 'installation/docker-custom.md' - - Maintaining Islandora: - - 'Basic Usage': 'installation/docker-basic-usage.md' - - 'Maintaining Drupal': 'installation/docker-maintain-drupal.md' - - 'Maintaining ISLE': 'installation/docker-maintain-isle.md' - - 'Available Commands': 'installation/docker-available-commands.md' - - 'Available Configuration': 'installation/docker-available-configuration.md' - - 'Troubleshooting' : 'installation/docker-troubleshooting.md' - - 'ISLE Site Template': 'installation/site-template.md' + - 'Docker': + - 'Introduction to ISLE': 'installation/docker/docker-introduction.md' + - 'Prerequisites': 'installation/docker/docker-prereq.md' + - 'ISLE-DC': + - Installing: + - 'Installing a Development Server': 'installation/docker/isle-dc/docker-local.md' + - 'Installing a Staging/Production Server': 'installation/docker/isle-dc/docker-custom.md' + - Maintaining: + - 'Basic Usage': 'installation/docker/isle-dc/docker-basic-usage.md' + - 'Maintaining Drupal': 'installation/docker/isle-dc/docker-maintain-drupal.md' + - 'Maintaining ISLE': 'installation/docker/isle-dc/docker-maintain-isle.md' + - 'Available Commands': 'installation/docker/isle-dc/docker-available-commands.md' + - 'Available Configuration': 'installation/docker/isle-dc/docker-available-configuration.md' + - 'Troubleshooting' : 'installation/docker/isle-dc/docker-troubleshooting.md' + - 'ISLE Site Template': + - 'Introduction to Site Template': 'installation/docker/site-template/site-template.md' - 'Ansible Playbook': 'installation/playbook.md' - Manual Installation: - 'Introduction': 'installation/manual/introduction.md' From 3ddab804f2b57668c7845575ae1d82995766ef9d Mon Sep 17 00:00:00 2001 From: Rosie Le Faive Date: Wed, 29 May 2024 15:14:37 -0300 Subject: [PATCH 2/3] Redirects and update a link. --- docs/user-documentation/users.md | 2 +- mkdocs.yml | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/user-documentation/users.md b/docs/user-documentation/users.md index 1aaa87024..049166f5c 100644 --- a/docs/user-documentation/users.md +++ b/docs/user-documentation/users.md @@ -18,7 +18,7 @@ Additional user roles can be created and assigned customized permissions, as des ## Before you start - This How-To assumes a very basic familiarity with Drupal. -- This How-To is generally applicable for any Islandora site, but the examples given are taken from an Islandora demo using the (optional) **[Islandora Starter Site](https://github.com/Islandora/islandora-starter-site)** configuration. This configuration is deployed automatically if you build your Islandora site with the appropriate options, using the [Ansible Playbook](../installation/playbook.md), [ISLE with Docker-Compose](../installation/docker-introduction.md). +- This How-To is generally applicable for any Islandora site, but the examples given are taken from an Islandora demo using the (optional) **[Islandora Starter Site](https://github.com/Islandora/islandora-starter-site)** configuration. This configuration is deployed automatically if you build your Islandora site with the appropriate options, using the [Ansible Playbook](../installation/playbook.md), [ISLE with Docker-Compose](../installation/docker/docker-introduction.md). !!! islandora "Warning" If you are writing to Fedora, your username must not contain spaces. diff --git a/mkdocs.yml b/mkdocs.yml index 1e5b06535..27213aa33 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -45,6 +45,16 @@ plugins: 'contributing/docs_style_guide.md': 'contributing/docs-style-guide.md' 'contributing/readme_template.md': 'contributing/readme-template.md' 'installation/component_overview.md': 'installation/component-overview.md' + 'installation/docker-available-commands.md': 'installation/docker/isle-dc/docker-available-commands.md' + 'installation/docker-available-configuration.md': 'installation/docker/isle-dc/docker-available-configuration.md' + 'installation/docker-basic-usage.md': 'installation/docker/isle-dc/docker-basic-usage.md' + 'installation/docker-custom.md': 'installation/docker/isle-dc/docker-custom.md' + 'installation/docker-introduction.md': 'installation/docker/docker-introduction.md' + 'installation/docker-local.md': 'installation/docker/isle-dc/docker-local.md' + 'installation/docker-maintain-drupal.md': 'installation/docker/isle-dc/docker-maintain-drupal.md' + 'installation/docker-maintain-isle.md': 'installation/docker/isle-dc/docker-maintain-isle.md' + 'installation/docker-prereq.md': 'installation/docker/docker-prereq.md' + 'installation/docker-troubleshooting.md': 'installation/docker/isle-dc/docker-troubleshooting.md' 'installation/manual/configuring_drupal.md': 'installation/manual/configuring-drupal.md' 'installation/manual/installing_composer_drush_and_drupal.md': 'installation/manual/installing-composer-drush-and-drupal.md' 'installation/manual/installing_crayfish.md': 'installation/manual/installing-crayfish.md' @@ -54,6 +64,7 @@ plugins: 'installation/manual/installing_solr.md': 'installation/manual/installing-solr.md' 'installation/manual/installing_tomcat_and_cantaloupe.md': 'installation/manual/installing-tomcat-and-cantaloupe.md' 'installation/manual/preparing_a_webserver.md': 'installation/manual/preparing-a-webserver.md' + 'installation/site-template.md': 'installation/docker/site-template/site-template.md' 'technical-documentation/adding_format_jsonld.md': 'technical-documentation/adding-format-jsonld.md' 'technical-documentation/alpaca_tips.md': 'technical-documentation/alpaca-tips.md' 'technical-documentation/resizing_vm.md': 'technical-documentation/resizing-vm.md' @@ -66,6 +77,7 @@ plugins: 'user-documentation/linked_data.md': 'user-documentation/linked-data.md' 'user-documentation/metadata_harvesting.md': 'user-documentation/metadata-harvesting.md' 'user-documentation/recipes/alexa_search.md': 'user-documentation/recipes/alexa-search.md' + extra: font: From ed7e3060568e3e1fb67c5cfd18a302872b3373dd Mon Sep 17 00:00:00 2001 From: Josh d'Entremont Date: Mon, 3 Jun 2024 14:25:20 -0300 Subject: [PATCH 3/3] added back up and restore instructions --- .../docker/site-template/backup.md | 229 ++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 230 insertions(+) create mode 100644 docs/installation/docker/site-template/backup.md diff --git a/docs/installation/docker/site-template/backup.md b/docs/installation/docker/site-template/backup.md new file mode 100644 index 000000000..6cebcd804 --- /dev/null +++ b/docs/installation/docker/site-template/backup.md @@ -0,0 +1,229 @@ +# Backups & Transferring Data Between Sites + +The following instructions describe how to back up and restore an islandora site. This is also used to transfer data between a development and production site, or to sync a staging site with a production site. + +Note that containers are named with a suffix of -dev or -prod depending on your environment. For example, backing up your development site’s Drupal database requires running the command on the drupal-dev container, but backing up the production site’s database requires running the command on the drupal-prod container. The following instructions are mostly written for development sites, but the same commands will work on production containers. + +Before attempting the following commands, you should familiarize yourself with [running commands inside a container](/documentation/installation/docker/site-template/containers.md) and with [docker compose cp](https://docs.docker.com/reference/cli/docker/compose/cp/) + + +## Drupal Configuration + +The typical use case of this is to export your drupal configuration files out of your development site’s Drupal database and onto the host machine. This process lets you check your configuration into your git repository so you can import it into your production site. + + +### Export + +To export your development config run: + +``` +docker compose exec -T drupal-dev drush -l default config:export -y +``` + +Then commit and push your git repo so the new config files are included. + + +### Import + +To import your config to a production or staging site you first need to `git pull` the config files you pushed from your development site, then rebuild and restart your custom Drupal container. + +``` +git pull +docker compose --profile prod build +docker compose --profile prod down +docker compose --profile prod up -d +``` + +Once this is done, your config files will be included in the container. You can then import them by running + +``` +docker compose exec -T drupal-prod drush -l default config:import -y +``` + + +## Drupal Database + +The Drupal database contains all your Drupal settings and content. If you want to move configuration settings from development to production you should use the Drupal Configuration instructions above. + +These instructions can also be used to move your content from production to staging or development. + + +### Back Up + +Backing up the Drupal database involves doing a database dump in the drupal container: + +``` +docker compose exec -T drupal-dev with-contenv bash -lc 'mysqldump -u ${DRUPAL_DEFAULT_DB_USER} -p${DRUPAL_DEFAULT_DB_PASSWORD} -h mariadb ${DRUPAL_DEFAULT_DB_NAME} > /tmp/dump.sql' +``` + +and then copying that dump from the container to the host machine: + +``` +docker compose cp drupal-dev:/tmp/dump.sql [path/on/host/dump.sql] +``` + + +### Restore + +Restoring from a database dump requires copying the dump file into the drupal container: + +``` +docker compose cp [path-to/dump.sql] drupal-dev:/tmp/dump.sql +``` + +Then replacing the database with the dump: + +``` +docker compose exec -T drupal-dev with-contenv bash -lc 'chown root:root /tmp/dump.sql && mysql -u ${DRUPAL_DEFAULT_DB_USER} -p${DRUPAL_DEFAULT_DB_PASSWORD} -h mariadb ${DRUPAL_DEFAULT_DB_NAME} < /tmp/dump.sql' +``` + +And finally, rebuilding the Drupal cache: + +``` +docker compose exec drupal-dev drush cr +``` + + +## Drupal Public Files + +Drupal's public files contain any files used on static pages. This is also where your Islandora derivatives are stored, which includes FITS, thumbnails, etc. + +These instructions can also be used to move your content from production to staging or development. + + +### Back Up + +Drupal public files can be compressed to a tgz file in the Drupal container: + +``` +docker compose exec -T drupal-dev with-contenv bash -lc 'tar zcvf /tmp/public-files.tgz -C /var/www/drupal/web/sites/default/files .' +``` + +Then copied to the host machine: + +``` +docker compose cp drupal-dev:/tmp/public-files.tgz [path/on/host/public-files.tgz] +``` + + +### Restore + +Drupal files can be restored from a tgz file by copying them into the Drupal container: + +``` +docker compose cp public-files.tgz drupal-dev:/tmp/public-files.tgz +``` + +Then placed in the correct directory inside the Drupal volume: + +``` +docker compose exec -T drupal-dev with-contenv bash -lc 'tar zxvf /tmp/public-files.tgz -C /var/www/drupal/web/sites/default/files && chown -R nginx:nginx /var/www/drupal/web/sites/default/files && rm /tmp/public-files.tgz' +``` + +!!! note + This will overwrite existing files if they have the same filename, but does not remove existing files otherwise. If you want to make sure that the public files directory does not contain anything but the newly imported files, you will want to empty the directory before copying the new files in. + + ``` + docker compose exec -T drupal-dev with-contenv bash -lc 'rm -r /var/www/drupal/web/sites/default/files/*' + ``` + + +## Drupal Private Files + +These instructions can also be used to move your content from production to staging or development. + + + +### Back Up + +Drupal public files can be compressed to a tgz file in the Drupal container: + +``` +docker compose exec -T drupal-dev with-contenv bash -lc 'tar zcvf /tmp/private-files.tgz -C /var/www/drupal/private .' +``` + +Then copied to the host machine: + +``` +docker compose cp drupal-prod:/tmp/private-files.tgz [path/on/host/private-files.tgz] +``` + + +### Restore + +Drupal files can be restored from a tgz file by copying them into the Drupal container: + +``` +docker compose cp private-files.tgz drupal-dev:/tmp/private-files.tgz +``` + +Then placed in the correct directory inside the Drupal volume: + +``` +docker compose exec -T drupal-dev with-contenv bash -lc 'tar zxvf /tmp/private-files.tgz -C /var/www/drupal/private && chown -R nginx:nginx /var/www/drupal/private && rm /tmp/private-files.tgz' +``` + +!!! note + This will overwrite existing files if they have the same filename, but does not remove existing files otherwise. If you want to make sure that the private files directory does not contain anything but the newly imported files, you will want to empty the directory before copying the new files in. + + ``` + docker compose exec -T drupal-dev with-contenv bash -lc 'rm -r /var/www/drupal/private/*' + ``` + + +## Fedora + +Fedora 6 uses a file structure called [OCFL](https://ocfl.io/) to store files and metadata. The Fedora database is built based on this OCFL file structure, so we don't actually back up our fedora database. Instead, we back up the files and let Fedora rebuild the database based on them. + +!!! note + These instructions involve moving the entirety of your Fedora data. For small sites this is fine, but you may find this becomes cumbersome as your site grows. You may wish to bind your oclf-root directory as a volume to eliminate the need to run the docker compose cp commands. + +### Back Up + +To back up Fedora we create a .tgz file from the ocfl-root directory in the Fedora container: + +``` +docker compose exec -T fcrepo-dev with-contenv bash -lc 'tar zcvf /tmp/fcrepo-export.tgz -C /data/home/data/ocfl-root/ .' +``` + +Then we copy that file to the host machine: + +``` +docker compose cp fcrepo-dev:/tmp/fcrepo-export.tgz [path/on/host/fcrepo-export.tgz] +``` + + +### Restore + +To restore our fedora database we need to copy the backup into the Fedora container: + +``` +docker compose cp [path-to/fcrepo-export.tgz] fcrepo-dev:/tmp/fcrepo-export.tgz +``` + +Empty the existing ocfl-root directory: +``` +docker compose exec -T fcrepo-dev bash -lc 'rm -r /data/home/data/ocfl-root/*' +``` + +Put our backup files in the ocfl-root directory: + +``` +docker compose exec -T fcrepo-dev with-contenv bash -lc 'tar zxvf /tmp/fcrepo-export.tgz -C /data/home/data/ocfl-root/ && chown -R tomcat:tomcat /data/home/data/ocfl-root/ && rm /tmp/fcrepo-export.tgz' +``` + +Drop the existing Fedora database: + +``` +docker compose exec -T mariadb-dev bash -lc 'mysql -e "drop database fcrepo;"' +``` + +Restart Fedora to reinitialize the database from the ocfl-root directory: + +``` +docker compose --profile dev restart fcrepo-dev +``` + +!!! note + + It may take a while for the database to be restored. In the meantime, you may see error messages that say Fedora is not connected. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 7f822e551..2af5a1b51 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -139,6 +139,7 @@ nav: - 'Troubleshooting' : 'installation/docker/isle-dc/docker-troubleshooting.md' - 'ISLE Site Template': - 'Introduction to Site Template': 'installation/docker/site-template/site-template.md' + - 'Back Up and Restore': 'installation/docker/site-template/backup.md' - 'Ansible Playbook': 'installation/playbook.md' - Manual Installation: - 'Introduction': 'installation/manual/introduction.md'