diff --git a/docs/decisions/001-Purpose.md b/docs/decisions/001-Purpose.md deleted file mode 100644 index 77b29066..00000000 --- a/docs/decisions/001-Purpose.md +++ /dev/null @@ -1,32 +0,0 @@ -Purpose of this repo -==================== - - -The goal of this project is to make a flexible renderer of a manifest for openedx hosting. - - -Context: --------- - -- Explanation of groove and subsequent forum conversation: - * https://discuss.openedx.org/t/tech-talk-demo-deploying-multiple-open-edx-instances-onto-a-kubernetes-cluster-with-tutor/4641 - * https://forum.opencraft.com/t/tech-talk-demo-deploying-multiple-open-edx-instances-onto-a-kubernetes-cluster-with-tutor/887 -- Proposal of making tutor the supported installation https://discuss.openedx.org/t/lets-talk-about-the-native-installation/3269 -- oep-45: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0045-arch-ops-and-config.html -- oep-45-ADR-tutor: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0045-arch-ops-and-config.html -- Braindump on Configuration: Today and Future with Cory Lee from edx https://drive.google.com/file/d/1wWfTbhy2blF526Fi4IvJQyFLz_3rawIX/view -- Notes of the BTR meeting in Lisbon https://discuss.openedx.org/t/build-test-release-notes-lisbon-2022/7133 - - -Decision: ---------- - -1. Create a POC for a ver flexible builder of manifests that focuses on the usecases not covered already by tutor. -2. Use a very flexible architecture that is open for extension from the very conception. Hexagonal architecture was chosen for this. - - -Consequence: ------------- - -1. The builder should work as a tutor plugin. Even if it has other standalone cases. -2. Every piece of the builder should be able to be replaced by a different implementation of the same interface. diff --git a/drydock_backups/README.rst b/drydock_backups/README.rst deleted file mode 100644 index fbb58d18..00000000 --- a/drydock_backups/README.rst +++ /dev/null @@ -1,55 +0,0 @@ -Drydock-backups -================ - -This is a tutor plugin used to easily store backups of mysql and mongodb databases through k8s cronjobs. It backups the databases and stores them in a bucket. - -This plugin assumes that the destination bucket is already created and that the credentials to access it are already configured (Works with S3 and Minio buckets indicating his endpoint url through `BACKUP_CUSTOM_STORAGE_ENDPOINT`). - -Installation and activation ---------------------------- - -It is included with the Drydock installation, so there's no need to install it separately. - -You can enable it adding `drydock-backups` to the `plugins` section of the `config.yml` file. - -Configuration variables ------------------------ - -- **BACKUP_DOCKER_IMAGE**: The image used to run the cronjob. (default: `ednxops/shipyard-utils:v{{BACKUP_VERSION}}`) -- **BACKUP_CRON_SCHEDULE**: Cron schedule to run the backup. (default: `0 2 * * *`) -- **BACKUP_STORAGE_SERVICE**: Storage service to use. (default: `aws-s3`) (options: `aws-s3`, `azure-blob`) -- **BACKUP_AWS_ACCESS_KEY**: AWS access key to access the bucket or minIO user. -- **BACKUP_AWS_SECRET_KEY**: AWS secret key to access the bucket or minIO password. -- **BACKUP_BUCKET_NAME**: Name of the bucket where the backups will be stored. -- **BACKUP_BUCKET_PATH**: Path inside the bucket where the backups will be stored. -- **BACKUP_CUSTOM_STORAGE_ENDPOINT**: Custom endpoint to access the bucket. (default: `None`) -- **BACKUP_AZURE_CONTAINER_NAME**: Name of the container where the backups will be stored. -- **BACKUP_AZURE_ACCOUNT_NAME**: Name of the account to access the container. -- **BACKUP_AZURE_CONTAINER_SAS_TOKEN**: SAS token to access the container. -- **BACKUP_K8S_USE_EPHEMERAL_VOLUMES**: Use ephemeral volumes to set up the cronjob. (default: `False`) -- **BACKUP_K8S_EPHEMERAL_VOLUME_SIZE**: Size of the ephemeral volume. (default: `8Gi`) -- **BACKUP_MYSQL_USERNAME**: Username to access the mysql database. (default: `{{ MYSQL_ROOT_USERNAME }}`) -- **BACKUP_MYSQL_PASSWORD**: Password to access the mysql database. (default: `{{ MYSQL_ROOT_PASSWORD }}`) -- **BACKUP_MONGO_PASSWORD**: Password to access the mongodb database. (default: `{{ MONGODB_PASSWORD }}`) -- **BACKUP_MONGO_USERNAME**: Username to access the mongodb database. (default: `{{ MONGODB_USERNAME }}`) - -You can set ups these variables in the `config.yml` file. - -Docker image ------------- - -The docker image used to run the cronjob is `ednxops/shipyard-utils:v1.0.0` and it is available in `DockerHub `_. - -When `drydock-backups` is installed in your manifests, you can build the Docker image using the following commands: - -.. code-block:: bash - - tutor images build backups - tutor images push backups - -Utilities ---------- - -In the path `/utils` we have some utilities to help us managing the backups. - -- **s3_backups_expiration**: This terraform script is to add to S3 bucket lifecycle rules. Is used to delete the backups that are older than a certain number of days. diff --git a/drydock_backups/__about__.py b/drydock_backups/__about__.py deleted file mode 100644 index 3dc1f76b..00000000 --- a/drydock_backups/__about__.py +++ /dev/null @@ -1 +0,0 @@ -__version__ = "0.1.0" diff --git a/drydock_backups/__init__.py b/drydock_backups/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/drydock_backups/patches/.gitignore b/drydock_backups/patches/.gitignore deleted file mode 100644 index e69de29b..00000000 diff --git a/drydock_backups/patches/drydock-multipurpose-jobs b/drydock_backups/patches/drydock-multipurpose-jobs deleted file mode 100644 index 2fc4345c..00000000 --- a/drydock_backups/patches/drydock-multipurpose-jobs +++ /dev/null @@ -1,154 +0,0 @@ ---- -apiVersion: batch/v1 -kind: CronJob -metadata: - name: backup-mysql-cron - labels: - app.kubernetes.io/component: cronjob -spec: - suspend: false - schedule: {{ BACKUP_CRON_SCHEDULE }} - startingDeadlineSeconds: 900 - successfulJobsHistoryLimit: 3 - failedJobsHistoryLimit: 3 - jobTemplate: - spec: - template: - spec: - securityContext: - runAsUser: 1000 - runAsGroup: 1000 - fsGroup: 1000 - fsGroupChangePolicy: "OnRootMismatch" - restartPolicy: OnFailure - containers: - - name: backup - image: {{ BACKUP_DOCKER_IMAGE }} - env: - - name: MYSQL_HOST - value: '{{ MYSQL_HOST }}' - - name: MYSQL_PORT - value: '{{ MYSQL_PORT }}' - - name: MYSQL_ROOT_USERNAME - value: '{{ BACKUP_MYSQL_USERNAME }}' - - name: MYSQL_ROOT_PASSWORD - value: '{{ BACKUP_MYSQL_PASSWORD }}' - - name: BACKUP_STORAGE_SERVICE - value: '{{ BACKUP_STORAGE_SERVICE }}' - - name: BUCKET_PATH - value: '{{ BACKUP_BUCKET_PATH }}' - {%- if BACKUP_STORAGE_SERVICE == "aws-s3" %} - - name: AWS_ACCESS_KEY_ID - value: '{{ BACKUP_AWS_ACCESS_KEY }}' - - name: AWS_SECRET_ACCESS_KEY - value: '{{ BACKUP_AWS_SECRET_KEY }}' - - name: S3_BUCKET_NAME - value: '{{ BACKUP_BUCKET_NAME }}' - {%- if BACKUP_CUSTOM_STORAGE_ENDPOINT %} - - name: BACKUP_CUSTOM_STORAGE_ENDPOINT - value: '{{ BACKUP_CUSTOM_STORAGE_ENDPOINT }}' - {%- endif %} - {%- elif BACKUP_STORAGE_SERVICE == "azure-blob" %} - - name: AZURE_CONTAINER_NAME - value: '{{ BACKUP_AZURE_CONTAINER_NAME }}' - - name: AZURE_CONTAINER_SAS_TOKEN - value: '{{ BACKUP_AZURE_CONTAINER_SAS_TOKEN }}' - - name: AZURE_ACCOUNT_NAME - value: '{{ BACKUP_AZURE_ACCOUNT_NAME }}' - {%- endif %} - {%- if BACKUP_K8S_USE_EPHEMERAL_VOLUMES %} - volumeMounts: - - mountPath: /data/ - name: backup-volume - {%- endif %} - args: - - mysql - {%- if BACKUP_K8S_USE_EPHEMERAL_VOLUMES %} - volumes: - - name: backup-volume - ephemeral: - volumeClaimTemplate: - spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: {{ BACKUP_K8S_EPHEMERAL_VOLUME_SIZE }} - {%- endif %} ---- -apiVersion: batch/v1 -kind: CronJob -metadata: - name: backup-mongo-cron - labels: - app.kubernetes.io/component: cronjob -spec: - suspend: false - schedule: {{ BACKUP_CRON_SCHEDULE }} - startingDeadlineSeconds: 900 - successfulJobsHistoryLimit: 3 - failedJobsHistoryLimit: 3 - jobTemplate: - spec: - template: - spec: - securityContext: - runAsUser: 1000 - runAsGroup: 1000 - fsGroup: 1000 - fsGroupChangePolicy: "OnRootMismatch" - restartPolicy: OnFailure - containers: - - name: backup - image: {{ BACKUP_DOCKER_IMAGE }} - env: - - name: MONGODB_HOST - value: '{{ MONGODB_HOST }}' - - name: MONGODB_PORT - value: '{{ MONGODB_PORT }}' - - name: MONGODB_USERNAME - value: '{{ BACKUP_MONGO_USERNAME }}' - - name: MONGODB_PASSWORD - value: '{{ BACKUP_MONGO_PASSWORD }}' - - name: BACKUP_STORAGE_SERVICE - value: '{{ BACKUP_STORAGE_SERVICE }}' - - name: BUCKET_PATH - value: '{{ BACKUP_BUCKET_PATH }}' - {%- if BACKUP_STORAGE_SERVICE == "aws-s3" %} - - name: AWS_ACCESS_KEY_ID - value: '{{ BACKUP_AWS_ACCESS_KEY }}' - - name: AWS_SECRET_ACCESS_KEY - value: '{{ BACKUP_AWS_SECRET_KEY }}' - - name: S3_BUCKET_NAME - value: '{{ BACKUP_BUCKET_NAME }}' - {%- if BACKUP_CUSTOM_STORAGE_ENDPOINT %} - - name: BACKUP_CUSTOM_STORAGE_ENDPOINT - value: '{{ BACKUP_CUSTOM_STORAGE_ENDPOINT }}' - {%- endif %} - {%- elif BACKUP_STORAGE_SERVICE == "azure-blob" %} - - name: AZURE_CONTAINER_NAME - value: '{{ BACKUP_AZURE_CONTAINER_NAME }}' - - name: AZURE_CONTAINER_SAS_TOKEN - value: '{{ BACKUP_AZURE_CONTAINER_SAS_TOKEN }}' - - name: AZURE_ACCOUNT_NAME - value: '{{ BACKUP_AZURE_ACCOUNT_NAME }}' - {%- endif %} - {%- if BACKUP_K8S_USE_EPHEMERAL_VOLUMES %} - volumeMounts: - - mountPath: /data/ - name: backup-volume - {%- endif %} - args: - - mongo - {%- if BACKUP_K8S_USE_EPHEMERAL_VOLUMES %} - volumes: - - name: backup-volume - ephemeral: - volumeClaimTemplate: - spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: {{ BACKUP_K8S_EPHEMERAL_VOLUME_SIZE }} - {%- endif %} diff --git a/drydock_backups/plugin.py b/drydock_backups/plugin.py deleted file mode 100644 index 7930ca1f..00000000 --- a/drydock_backups/plugin.py +++ /dev/null @@ -1,230 +0,0 @@ -from __future__ import annotations - -import os -import os.path -from glob import glob - -import click -import pkg_resources -from tutor import hooks - -from drydock.__about__ import __version__ - -config = { - "defaults": { - "VERSION": __version__, - "DOCKER_IMAGE": "ednxops/shipyard-utils:v{{BACKUP_VERSION}}", - "CRON_SCHEDULE": '0 2 * * *', - "STORAGE_SERVICE": "aws-s3", - "AWS_ACCESS_KEY": "", - "AWS_SECRET_KEY": "", - "BUCKET_NAME": "", - "BUCKET_PATH": "backups", - "AZURE_CONTAINER_NAME": "", - "AZURE_CONTAINER_SAS_TOKEN": "", - "AZURE_ACCOUNT_NAME": "", - "CUSTOM_STORAGE_ENDPOINT": None, - "K8S_USE_EPHEMERAL_VOLUMES": False, - "K8S_EPHEMERAL_VOLUME_SIZE": "8Gi", - "MYSQL_USERNAME": '{{ MYSQL_ROOT_USERNAME }}', - "MYSQL_PASSWORD": '{{ MYSQL_ROOT_PASSWORD }}', - "MONGO_PASSWORD": '{{ MONGODB_PASSWORD }}', - "MONGO_USERNAME": '{{ MONGODB_USERNAME }}', - }, -} - -hooks.Filters.CONFIG_DEFAULTS.add_items( - [ - (f"BACKUP_{key}", value) - for key, value in config.get("defaults", {}).items() - ] -) - -hooks.Filters.CONFIG_UNIQUE.add_items( - [ - (f"BACKUP_{key}", value) - for key, value in config.get("unique", {}).items() - ] -) - -hooks.Filters.CONFIG_OVERRIDES.add_items( - list(config.get("overrides", {}).items()) -) - - -######################################## -# INITIALIZATION TASKS -######################################## - -# To add a custom initialization task, create a bash script template under: -# drydock_backups/templates/drydock-backups/jobs/init/ -# and then add it to the MY_INIT_TASKS list. Each task is in the format: -# ("", ("", "", "