Introducing kustomize for HCO deployment (#463)

* Introducing kustomize for HCO deployment
Intended to replace deploy_marketplace.sh and deploy_imageregistry.sh scripts.
Deployment can be done by two ways:
1. Export relevant env vars, then run the "deploy_kustomize.sh" script using either "marketplace" or "image_registry" arguments.
2. Edit the relevant manifests, then run the overlay for the required HCO deployment combination.

Signed-off-by: orenc1 <[email protected]>

* removing templates and variable substitution, differentiating between make hco available and install hco, moving quay token and retry_loop functions to different files

Signed-off-by: orenc1 <[email protected]>

* Consolidating aux functions to main script, adding support for namespace change using patches, update README file.

Signed-off-by: orenc1 <[email protected]>

* Use 'namespace:' in kustomization.yaml to overwrite namespace in base resources.

Signed-off-by: orenc1 <[email protected]>

* updating readme file with manual instructions for kustomize deployment

Signed-off-by: orenc1 <[email protected]>

* Dropping the directory tree from the readme file.

Signed-off-by: orenc1 <[email protected]>

README.md

@@ -26,6 +26,11 @@ tab or deploy from the commandline:
 $ curl https://raw.githubusercontent.com/kubevirt/hyperconverged-cluster-operator/master/deploy/hco.yaml | kubectl create -f -
 ```
 
+## Installing HCO using kustomize (Openshift OLM Only)
+Refer to [kustomize deployment documentation](deploy/kustomize/README.md).
+
+**NOTE**: `deploy/deploy_marketplace.sh` and `deploy/deploy_imageregistry.sh` will be deprecated soon.
+
 ## Installing Unreleased Bundles Using Marketplace
 The hyperconverged cluster operator will publish the lastest bundles to [quay/kubevirt-hyperconvered/hco-operatohub](https://quay.io/application/kubevirt-hyperconverged/hco-operatorhub)
 before publishing to operatorhub.io.

deploy/deploy_imageregistry.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+echo "WARNING: THIS SCRIPT WILL BE DEPRECATED SOON. PLEASE USE kustomize/deploy_kustomize.sh INSTEAD."
+
 set -ex
 
 RED='\033[0;31m'

deploy/deploy_marketplace.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+echo "WARNING: THIS SCRIPT WILL BE DEPRECATED SOON. PLEASE USE kustomize/deploy_kustomize.sh INSTEAD."
+
 set -ex
 
 RED='\033[0;31m'

deploy/kustomize/README.md

@@ -0,0 +1,129 @@
+# Deploy HCO using kustomize
+The KubeVirt Hyperconverged Cluster Operator (HCO) is delivered and deployed on a running OCP/OKD cluster using the kustomize method. 
+
+# Kustomize Manifests
+In order to install HCO on your cluster, two necessary steps to be performed:
+1. **Delivery** - Make HCO recognized and available for the operator-lifecycle-manager (OLM).
+2. **Deployment** - Use OLM provided resources and APIs to deploy HCO on the cluster.
+
+The directory tree consists of kustomize-based manifests with default values, supporting various deployment configurations.
+
+## Delivery
+There are two distinct options to deliver HCO operator to OLM - Marketplace and Image Registry.
+
+### Marketplace
+This method is taking advantage of CatalogSourceConfig, pointing to an OperatorSource, which makes the operator available on OLM OperatorHub.
+To manually deliver HCO using marketplace, edit `spec.registryNamespace` of `marketplace/operator_source.yaml` to the desired value (default is "kubevirt-hyperconverged"), and run:
+```
+oc apply -k marketplace
+```
+Which will create the HCO catalog source with default configuration. After processing is complete, the package will be available in OperatorHub.
+
+#### Private Repo
+If the operator source is located in a private Quay.io registry, you should provide the OperatorSource resource with a secret, which can be extracted by:
+```
+curl -sH "Content-Type: application/json" -XPOST https://quay.io/cnr/api/v1/users/login -d '
+  {
+      "user": {
+          "username": "'"${QUAY_USERNAME}"'",
+          "password": "'"${QUAY_PASSWORD}"'"
+      }
+  }' | jq -r '.token'
+```
+The token should be inserted in `spec.authorizationToken.secretName` of `private_repo/operator_source.patch.yaml`, then run:
+```
+oc apply -k private_repo
+```
+
+### Image Registry
+This method is delivering the operator's bundle image via a grpc protocol from an image registry.
+To manually deliver HCO using image registry, edit `spec.image` of `image_registry/catalog_source.yaml` to the desired image bundle URL, and run:
+```
+oc apply -k image_registry
+```
+
+### Automation
+The shell script `deploy_kustomize.sh` can be used to automate delivery of HCO to OLM.
+
+#### Content-Only flag
+To make HCO available for deployment in the cluster, without actually deploy it, set "CONTENT_ONLY" to "true". That will stop script execution before entering the deployment phase.
+
+#### Marketplace
+Set environment variable "MARKETPLACE_MODE" to "true".
+
+##### Private Repo
+Set "PRIVATE_REPO" to "true" and provide credentials using "QUAY_USERNAME" and "QUAY_PASSWORD" environment variables.
+
+#### Image Registry
+Set environment variable "MARKETPLACE_MODE" to "false".
+
+## Deployment
+The deployment phase is consisting of 3 resources, located in `base` directory:
+* OperatorGroup
+* Subscription
+* HyperConverged Custom Resource
+
+In addition, a namespace must be deployed prior to the deployment of resources above. the namespace resource can be found in `namespace.yaml`.
+To deploy HCO with default settings, run:
+```
+cat <<EOF >kustomization.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+bases:
+  - base
+resources:
+  - namespace.yaml
+EOF
+
+oc apply -k .
+```
+
+### KVM Emulation
+If KVM emulation is required on your environment, use the following overlay to add the Subscription resource with relevant KVM config:
+```
+oc apply -k kvm_emulation
+```
+
+### Automation
+To automate the process of delivery **and** deployment, set the environment variable "CONTENT_ONLY" to "false", then run `./deploy_kustomize.sh`.
+To use the script in conjunction with KVM_EMULATION property, set "KVM_EMULATION" env var to "true" prior to running the script. 
+
+## Customizations
+Existing manifests in this repository are representing an HCO deployment with default settings.
+In order to make customizations to your deployment, you need to set up other environment variables and create kustomize overlays to override default settings.
+
+### Change Deployment Namespace
+The default namespace is `kubevirt-hyperconverged`.
+In order to change that to a custom value, you should edit `namespace.yaml` and update its `metadata.name` value, and run:
+```
+cat <<EOF >kustomization.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ${DESIRED_NAMESPACE}
+bases:
+  - base
+resources:
+  - namespace.yaml
+EOF
+
+oc apply -k .
+```
+
+### Modify HCO Channel and Version
+Create a Subscription patch to reflect the desired version and channel.
+```
+cat > subscription.patch.yaml << EOF
+apiVersion: operators.coreos.com/v1alpha1
+kind: Subscription
+metadata:
+  name: hco-operatorhub
+spec:
+  startingCSV: kubevirt-hyperconverged-operator.v${HCO_VERSION}
+  channel: "${HCO_CHANNEL}"
+```
+
+#### Deploy
+When customizations are ready, run `./deploy_kustomize.sh`.
+The script will prepare and submit kustimized manifests directories to the cluster. It will also check whenever deployment is complete (HCO CR reports Condition "Available" True), and finish successfully.

deploy/kustomize/base/hco_cr.yaml

@@ -0,0 +1,7 @@
+apiVersion: hco.kubevirt.io/v1alpha1
+kind: HyperConverged
+metadata:
+  name: kubevirt-hyperconverged
+  namespace: kubevirt-hyperconverged
+spec:
+  BareMetalPlatform: true

deploy/kustomize/base/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - operator_group.yaml
+  - subscription.yaml
+  - hco_cr.yaml

deploy/kustomize/base/operator_group.yaml

@@ -0,0 +1,8 @@
+apiVersion: operators.coreos.com/v1
+kind: OperatorGroup
+metadata:
+  name: kubevirt-hyperconverged-group
+  namespace: kubevirt-hyperconverged
+spec:
+  targetNamespaces:
+    - kubevirt-hyperconverged

deploy/kustomize/base/subscription.yaml

@@ -0,0 +1,12 @@
+apiVersion: operators.coreos.com/v1alpha1
+kind: Subscription
+metadata:
+  name: hco-operatorhub
+  namespace: kubevirt-hyperconverged
+spec:
+  source: hco-catalogsource-config
+  sourceNamespace: openshift-marketplace
+  name: kubevirt-hyperconverged
+  startingCSV: kubevirt-hyperconverged-operator.v1.1.0
+  channel: "1.1.0"
+

deploy/kustomize/deploy_kustomize.sh

@@ -0,0 +1,156 @@
+#!/bin/bash
+
+set -x
+
+# Setup Environment Variables
+HCO_VERSION="${HCO_VERSION:-1.1.0}"
+HCO_CHANNEL="${HCO_CHANNEL:-1.1.0}"
+MARKETPLACE_MODE="${MARKETPLACE_MODE:-true}"
+HCO_REGISTRY_IMAGE="${HCO_REGISTRY_IMAGE:-quay.io/kubevirt/hco-container-registry:latest}"
+PRIVATE_REPO="${PRIVATE_REPO:-false}"
+QUAY_USERNAME="${QUAY_USERNAME:-}"
+QUAY_PASSWORD="${QUAY_PASSWORD:-}"
+CONTENT_ONLY="${CONTENT_ONLY:-false}"
+KVM_EMULATION="${KVM_EMULATION:-false}"
+OC_TOOL="${OC_TOOL:-oc}"
+
+#####################
+
+main() {
+  SCRIPT_DIR="$(dirname "$0")"
+  TARGET_NAMESPACE=$(grep name: $SCRIPT_DIR/namespace.yaml | awk '{print $2}')
+
+  TMPDIR=$(mktemp -d)
+  cp -r $SCRIPT_DIR/* $TMPDIR
+
+  if [ "$PRIVATE_REPO" = 'true' ]; then
+    get_quay_token
+    oc create secret generic quay-registry-kubevirt-hyperconverged --from-literal=token="$QUAY_TOKEN" -n openshift-marketplace
+
+    cat <<EOF >$TMPDIR/kustomization.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+bases:
+  - private_repo
+EOF
+    oc apply -k $TMPDIR
+
+  else # not private repo
+    if [ "$MARKETPLACE_MODE" = 'true' ]; then
+      cat <<EOF >$TMPDIR/kustomization.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+bases:
+  - marketplace
+EOF
+      oc apply -k $TMPDIR
+    else
+      cat <<EOF >$TMPDIR/kustomization.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+bases:
+  - image_registry
+EOF
+      oc apply -k $TMPDIR
+    fi
+  fi
+
+  if [ "$CONTENT_ONLY" = 'true' ]; then
+    echo INFO: Content is ready for deployment in OLM.
+    exit 0
+  fi
+
+  # KVM_EMULATION setting is active only when a deployment is done.
+  if [ "$KVM_EMULATION" = 'true' ]; then
+    cat <<EOF >$TMPDIR/kustomization.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ${TARGET_NAMESPACE}
+bases:
+  - kvm_emulation
+resources:
+  - namespace.yaml
+EOF
+    exit
+    retry_loop $TMPDIR
+  else
+    # In case KVM_EMULATION is not set.
+    cat <<EOF >$TMPDIR/kustomization.yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ${TARGET_NAMESPACE}
+bases:
+  - base
+resources:
+  - namespace.yaml
+EOF
+    retry_loop $TMPDIR
+  fi
+}
+
+get_quay_token() {
+    token=$(curl -sH "Content-Type: application/json" -XPOST https://quay.io/cnr/api/v1/users/login -d '
+  {
+      "user": {
+          "username": "'"${QUAY_USERNAME}"'",
+          "password": "'"${QUAY_PASSWORD}"'"
+      }
+  }' | jq -r '.token')
+
+  if [ "$token" == "null" ]; then
+    echo [ERROR] Got invalid Token from Quay. Please check your credentials in QUAY_USERNAME and QUAY_PASSWORD.
+    exit 1
+  else
+    QUAY_TOKEN=\"$token\";
+  fi
+}
+
+# Deploy HCO and OLM Resources with retries
+retry_loop() {
+  success=0
+  iterations=0
+  sleep_time=10
+  max_iterations=72 # results in 12 minutes timeout
+  until [[ $success -eq 1 ]] || [[ $iterations -eq $max_iterations ]]
+  do
+    deployment_failed=0
+
+      if [[ ! -d $1 ]]; then
+        echo $1
+        echo "[ERROR] Manifests do not exist. Aborting..."
+        exit 1
+      fi
+
+      set +e
+      if ! ${OC_TOOL} apply -k $1
+      then
+        deployment_failed=1
+      fi
+      set -e
+
+    if [[ deployment_failed -eq 1 ]]; then
+      iterations=$((iterations + 1))
+      iterations_left=$((max_iterations - iterations))
+      echo "[WARN] At least one deployment failed, retrying in $sleep_time sec, $iterations_left retries left"
+      sleep $sleep_time
+      continue
+    fi
+    success=1
+  done
+
+  if [[ $success -eq 1 ]]; then
+    echo "[INFO] Deployment successful, waiting for HCO Operator to report Ready..."
+    ${OC_TOOL} wait -n ${TARGET_NAMESPACE} hyperconverged kubevirt-hyperconverged --for condition=Available --timeout=15m
+    ${OC_TOOL} wait "$(${OC_TOOL} get pods -n ${TARGET_NAMESPACE} -l name=hyperconverged-cluster-operator -o name)" -n "${TARGET_NAMESPACE}" --for condition=Ready --timeout=15m
+  else
+    echo "[ERROR] Deployment failed."
+    exit 1
+  fi
+}
+
+main

deploy/kustomize/image_registry/catalog_source.yaml

@@ -0,0 +1,11 @@
+apiVersion: operators.coreos.com/v1alpha1
+kind: CatalogSource
+metadata:
+  name: hco-catalogsource
+  namespace: openshift-marketplace
+  imagePullPolicy: Always
+spec:
+  sourceType: grpc
+  image: quay.io/kubevirt/hco-container-registry:latest
+  displayName: KubeVirt HyperConverged
+  publisher: KubeVirt Project

deploy/kustomize/image_registry/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - catalog_source.yaml

deploy/kustomize/kvm_emulation/kustomization.yaml

@@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+bases:
+  - ../base
+
+patchesStrategicMerge:
+  - subscription.patch.yaml

deploy/kustomize/kvm_emulation/subscription.patch.yaml

@@ -0,0 +1,12 @@
+apiVersion: operators.coreos.com/v1alpha1
+kind: Subscription
+metadata:
+  name: hco-operatorhub
+spec:
+  config:
+    selector:
+      matchLabels:
+        name: hyperconverged-cluster-operator
+    env:
+      - name: KVM_EMULATION
+        value: "true"