Add a basic manifest repository implementation #1

MoisesGSalas · 2022-07-14T14:30:30Z

Description

Implement BaseManifest to render a drydock environment.

A manifest repository is a collection of configuration files that describe that describe the state of our applications. This collection is meant to be tracked by a version control tool and enable a GitOps framework.

The BaseManifest implementation serves as a basic approach to this for Kubernetes. It mixes Kustomize and a Tutor (v13.3.1) environment to render a new environment that uses the Tutor env as a base and adds additional extensions as new resources. A drydock environment generated using BaseManifest looks like this:

drydockenv
├── base
│   ├── apps
│   ├── k8s
│   ├── kustomization.yml
│   └── plugins
├── extensions
│   ├── flowers.yml
│   ├── hpa.yml
│   ├── ingress.yml
│   ├── kustomization.yml
│   └── overrides.yml
└── kustomization.yml

Where base is standard Tutor env (minus files not needed by k8s), and extension is an additional Kustomize overlay with extra resources or patches.

The drydock environment is generated by using the tutor renderer on the templates defined on default. The templates on default/base are taken almost verbatim from the latest tutor version for the Maple release (v13.3.1), the only difference being the option to skip jobs (The current implementation of Jobs on Tutor doesn't play nicely with GitOps).

# Kustomization.yml
 resources:
 - k8s/namespace.yml
 - k8s/deployments.yml
-- k8s/jobs.yml
 - k8s/services.yml
 - k8s/volumes.yml
+{%- if not DRYDOCK_DISABLE_JOBS | default(false) %}
+- k8s/jobs.yml
+{%- endif %}
+

The default/extension includes additional resources such as an ingress, an Horizontal Pod Autoscaler and a flowers deployment. It also includes an overrides.yml (inspired by this commit) to modify the values of the resources in default/base.

Additional info

BaseManifest is static in the sense that is tied to the templates found in default and those are locked to the ones found in tutor v13.3.1 for the most part. In the future it may be possible to have different versions of default.
Tutor is not exactly meant to be used as a library at the moment and the current implementation doesn't leverage the full tooling that it provides. At the moment we render a full tutor env (including our cherry-picked templates) and selectively move the ones we are interested.
We are not making checks for conflicting options (e.g. ENABLE_WEB_PROXY: true and DRYDOCK_INGRESS: true)

mariajgrimaldi · 2022-07-21T18:26:35Z

drydock/manifest_builder/infrastructure/tutor_based_manifests.py

+        with tempfile.TemporaryDirectory() as tmpdir:
+            src = path_join(tmpdir)
+            self.render(src, config)
+            shutil.rmtree(path=dst, ignore_errors=True)
+            self.relocate_env(src, dst)


This is my current folder structure:

├── config.yml ├── env │ ├── base │ │ ├── apps │ │ │ ├── caddy │ │ │ ├── openedx │ │ │ └── redis │ │ ├── k8s │ │ │ ├── deployments.yml │ │ │ ├── jobs.yml │ │ │ ├── namespace.yml │ │ │ ├── services.yml │ │ │ └── volumes.yml │ │ ├── kustomization.yml │ │ └── plugins │ │ ├── forum │ │ ├── notes │ │ └── settingmanager.yml │ ├── extensions │ │ ├── flowers.yml │ │ ├── hpa.yml │ │ ├── ingress.yml │ │ ├── kustomization.yml │ │ └── overrides.yml │ └── kustomization.yml ├── reference.yml └── requirements.txt

My tutor plugins list:

android (disabled) 13.0.0 discovery (disabled) 13.0.1 distro 2.1.0 drydock 0.1.0 ecommerce (disabled) 13.0.1 forum 13.0.0 license (disabled) 13.0.0 mfe (disabled) 13.0.6 minio 13.0.1 notes 13.0.1 richie (disabled) 13.0.2 settingmanager 0.1.0 webui (disabled) 13.0.2 xqueue (disabled) 13.0.0

Then I run:
tutor drydock save -r reference.yml
When the command finishes, I run:
tutor plugins list
And this happens:

⚠️ Failed to enable plugin 'settingmanager': plugin 'settingmanager' is not installed. android (disabled) 13.0.0 discovery (disabled) 13.0.1 distro 2.1.0 drydock 0.1.0 ecommerce (disabled) 13.0.1 forum 13.0.0 license (disabled) 13.0.0 mfe (disabled) 13.0.6 minio 13.0.1 notes 13.0.1 richie (disabled) 13.0.2 webui (disabled) 13.0.2 xqueue (disabled) 13.0.0

Is that okay?

Yes (actually is not ok, but is a current limitation). At the moment we don't render module nor yaml plugins on base/plugins. The current workaround is to define the TUTOR_PLUGINS_ROOTS outside env.

The folder structure would look something like this:

├── config.yml ├── env │ ├── base │ │ ├── apps │ │ │ ├── caddy │ │ │ ├── openedx │ │ │ └── redis │ │ ├── k8s │ │ │ ├── deployments.yml │ │ │ ├── jobs.yml │ │ │ ├── namespace.yml │ │ │ ├── services.yml │ │ │ └── volumes.yml │ │ ├── kustomization.yml │ │ └── plugins │ │ ├── forum │ │ └── notes │ ├── extensions │ │ ├── flowers.yml │ │ ├── hpa.yml │ │ ├── ingress.yml │ │ ├── kustomization.yml │ │ └── overrides.yml │ └── kustomization.yml ├── reference.yml └── requirements.txt └── plugins └── settingmanager.yml

and your TUTOR_PLUGINS_ROOT should be something like $TUTOR_ROOT/plugins

mariajgrimaldi · 2022-07-21T18:28:29Z

drydock/manifest_builder/infrastructure/tutor_config.py

+        self.defaults = {
+            "DRYDOCK_FLOWER": False,
+            "DRYDOCK_INGRESS": False,
+            "DRYDOCK_HPA": False,
+            "DRYDOCK_DISABLE_JOBS": False,
+            "DRYDOCK_ENABLE_OVERRIDES": False,
+            "DRYDOCK_LMS_MIN_REPLICAS": 1,
+            "DRYDOCK_CMS_MIN_REPLICAS": 3,
+            "DRYDOCK_LMS_MAX_REPLICAS": 1,
+            "DRYDOCK_CMS_MAX_REPLICAS": 3,
+            "DRYDOCK_LMS_LIMIT_CPU": 1,
+            "DRYDOCK_LMS_LIMIT_MEMORY": "2Gi",
+            "DRYDOCK_LMS_REQUEST_CPU": "600m",
+            "DRYDOCK_LMS_REQUEST_MEMORY": "1Gi",
+            "DRYDOCK_LMS_WORKER_LIMIT_CPU": 1,
+            "DRYDOCK_LMS_WORKER_LIMIT_MEMORY": "2Gi",
+            "DRYDOCK_LMS_WORKER_REQUEST_CPU": "600m",
+            "DRYDOCK_LMS_WORKER_REQUEST_MEMORY": "1Gi",
+            "DRYDOCK_CMS_LIMIT_CPU": 1,
+            "DRYDOCK_CMS_LIMIT_MEMORY": "2Gi",
+            "DRYDOCK_CMS_REQUEST_CPU": "600m",
+            "DRYDOCK_CMS_REQUEST_MEMORY": "1Gi",
+            "DRYDOCK_CMS_WORKER_LIMIT_CPU": 1,
+            "DRYDOCK_CMS_WORKER_LIMIT_MEMORY": "2Gi",
+            "DRYDOCK_CMS_WORKER_REQUEST_CPU": "600m",
+            "DRYDOCK_CMS_WORKER_REQUEST_MEMORY": "1Gi",
+        }


self.defaults changes? why this dictionary is not constant to the class?

I've opted to define the defaults on a defaults.yml on each template set and load them from there.

drydock/manifest_builder/infrastructure/tutor_based_manifests.py

mariajgrimaldi · 2022-07-21T18:35:08Z

How did you decide what to add as configurable in the base templates?

MoisesGSalas · 2022-07-21T19:42:30Z

How did you decide what to add as configurable in the base templates?

The ones in base I tried to keep almost the same as the ones in tutor. Minus the Jobs for reasons explained above.

The ones in extensions I've opted for making all the additional resources toggleable

mariajgrimaldi

This looks good! I tried it, and it seems to be working as expected. Just one more thing: you could add a short how-to describing how to generate manifests & how to apply them.

The `BaseManfests` builder will render a standard Tutor environment based on the templates used in version 13.3.1 of Tutor and use it as a base of Kustomization application with additional resources as overlays. The `TutorExtendedConfig` will return the Tutor configuration values of the current `TUTOR_ROOT` and will use the default values of the template set (defined in a file `defaults.yml`) as a fallback.

MoisesGSalas changed the title ~~[WIP] Add a basic manifest repository implementation~~ Add a basic manifest repository implementation Jul 20, 2022

MoisesGSalas requested a review from a team July 20, 2022 20:20

mariajgrimaldi reviewed Jul 21, 2022

View reviewed changes

drydock/manifest_builder/infrastructure/tutor_based_manifests.py Show resolved Hide resolved

mariajgrimaldi reviewed Jul 21, 2022

View reviewed changes

drydock/manifest_builder/infrastructure/tutor_based_manifests.py Outdated Show resolved Hide resolved

mariajgrimaldi requested a review from felipemontoya July 21, 2022 18:42

MoisesGSalas requested a review from a team July 22, 2022 17:29

mariajgrimaldi approved these changes Jul 22, 2022

View reviewed changes

jfavellar90 approved these changes Jul 22, 2022

View reviewed changes

MoisesGSalas force-pushed the mgs/base-manifests branch 2 times, most recently from a26f26c to dd494c1 Compare July 25, 2022 11:52

MoisesGSalas force-pushed the mgs/base-manifests branch from dd494c1 to e449d41 Compare July 25, 2022 11:52

MoisesGSalas merged commit 1f8208b into main Jul 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a basic manifest repository implementation #1

Add a basic manifest repository implementation #1

MoisesGSalas commented Jul 14, 2022 •

edited

Loading

mariajgrimaldi Jul 21, 2022

MoisesGSalas Jul 21, 2022

mariajgrimaldi Jul 21, 2022

MoisesGSalas Jul 22, 2022

mariajgrimaldi commented Jul 21, 2022

MoisesGSalas commented Jul 21, 2022

mariajgrimaldi left a comment

Add a basic manifest repository implementation #1

Add a basic manifest repository implementation #1

Conversation

MoisesGSalas commented Jul 14, 2022 • edited Loading

Description

Additional info

mariajgrimaldi Jul 21, 2022

Choose a reason for hiding this comment

MoisesGSalas Jul 21, 2022

Choose a reason for hiding this comment

mariajgrimaldi Jul 21, 2022

Choose a reason for hiding this comment

MoisesGSalas Jul 22, 2022

Choose a reason for hiding this comment

mariajgrimaldi commented Jul 21, 2022

MoisesGSalas commented Jul 21, 2022

mariajgrimaldi left a comment

Choose a reason for hiding this comment

MoisesGSalas commented Jul 14, 2022 •

edited

Loading