Adds CRC deployment automation and CI documentation #268

perdasilva · 2022-03-22T22:15:21Z

This PR:

Introduces new scripts to provision a crc cluster and deploy olm onto it
Introduces a new values-crc-e2e.yaml to adjust the images, arguments, and ImagePullPolicy of the deployments
Introduces additional *.crc.e2e.yaml yaml patch files to patch additional olm manifests
Adds new make targets for interacting with th deployment scripts
Adds documentation on how ci and downstream sync work
Adds documentation on how to use crc for e2e tests

openshift-ci · 2022-03-22T22:16:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: perdasilva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [perdasilva]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

perdasilva · 2022-03-23T17:43:33Z

/retest-required

perdasilva · 2022-03-23T18:38:38Z

/retest-required

docs/local-testing-with-crc.md

timflannagan · 2022-03-23T18:45:44Z

scripts/crc-deploy.sh

+
+# Set kubeconfig to CRC if necessary
+export KUBECONFIG=${KUBECONFIG:-${HOME}/.crc/machines/crc/kubeconfig}
+KUBE_ADMIN_PASSWORD=$(tr -d '\n' < "${HOME}/.crc/machines/crc/kubeadmin-password")


IIRC there's a dedicated crc console --credentials command that might be better to parse?

This is what I have locally as an alias:

$ alias crc_l crc_l='eval $(crc console --credentials | cut -d" -f2- | grep -v developer | tr -d ")'

Oh, found an even nicer way: crc console --credentials -o json | jq -r .clusterConfig.adminCredentials.password

timflannagan · 2022-03-23T18:47:38Z

scripts/crc-deploy.sh

+# Create port-forward to CRC registry
+kubectl port-forward -n ${OPENSHIFT_REGISTRY_NAMESPACE} svc/${IMAGE_REGISTRY_SVC} ${IMAGE_REGISTRY_PORT}:${IMAGE_REGISTRY_PORT} > /dev/null 2>&1 &
+PORT_FWD_PID=$!
+
+# Remember to close the port-forward
+trap 'kill "${PORT_FWD_PID}"' EXIT
+
+# give port-forward a second
+# I found that without this docker login would not work
+# as if it couldn't reach the docker server
+sleep 1
+
+oc whoami -t | docker login localhost:5000 --username user --password-stdin
+
+# Push olm image
+echo "Pushing olm image"
+docker tag olm:test localhost:5000/openshift/olm:test
+docker push localhost:5000/openshift/olm:test
+
+# Push registry image
+echo "Pushing registry image"
+docker tag opm:test localhost:5000/openshift/opm:test
+docker push localhost:5000/openshift/opm:test


Maybe this can be thrown in a function if we're trapping signals and require cleanup?

Also variable to track any of the opm/olm test images?

timflannagan · 2022-03-23T18:50:08Z

scripts/sync_pop_candidate.sh

+    make manifests
+    git add .
+    git status
+    git commit --amend --allow-empty --no-edit --trailer "Upstream-repository: ${remote}" --trailer "Upstream-commit: ${rc}"


Note: the --trailer command requires new git versions. I saw locally that I needed to build git from source as f34 (f35?) didn't have this in the dnf stack.

Do you happen to know from which version --trailer was introduced? I'm having trouble finding it. I could put in a check and a link.

IIRC it's git 2.33

perdasilva · 2022-03-24T08:33:01Z

/retest-required

timflannagan · 2022-03-25T23:53:22Z

/test all

perdasilva · 2022-03-30T08:31:44Z

/retest

perdasilva · 2022-03-30T10:40:07Z

/retest-required

perdasilva · 2022-03-31T09:28:13Z

/retest

dinhxuanvu

Just a comment on the order. The rest look good.

dinhxuanvu · 2022-03-31T17:46:09Z

scripts/crc-deploy.sh

+  # CRC_E2E_VALUES is already set in this script and exported
+  ./scripts/generate_crds_manifests.sh
+
+  # Apply patches


I think we should apply the OLM CRD manifests first and then apply the psm and profiler deployments.

Interestingly enough, it looks like do the exact opposite in the context of CVO run levels looking at our root manifests directory: https://github.com/openshift/operator-framework-olm/tree/master/manifests.

00 is higher priority right?

@dinhxuanvu that's right. I'm updating the script to pipe to sort. Then it should apply everything in priority order. Good catch! The comment here was stale. What I was doing before was deleting them in reverse priority order then applying them in priority order. But then I discovered replace.

timflannagan · 2022-03-31T18:52:01Z

Makefile

+	echo "Building olm image"
+	IMG="olm:test" $(MAKE) build/olm-container
+	echo "Building opm image"
+	rm -rf bin


Any idea on why this rm command was needed between build steps?

I'll add a comment. But basically, if we don't rm bin before building the opm container and the binaries are still in bin, it won't (re-)compile them and you'll end up with a binary that doesn't work inside the container =S

I'm cleaning this up by adding a clean target and calling it before the build/*-container targets

perdasilva · 2022-04-01T09:24:09Z

/retest-required

perdasilva · 2022-04-01T12:34:17Z

/retest-required

perdasilva · 2022-04-01T13:17:10Z

/test e2e-gcp-olm-flaky

Signed-off-by: perdasilva <[email protected]>

perdasilva · 2022-04-01T14:25:11Z

/test e2e-gcp-olm-flaky

timflannagan · 2022-04-01T16:31:58Z

scripts/crc-deploy.sh

+echo "Logging in as kubeadmin"
+oc login -u "${KUBE_ADMIN_USER}" -p "${KUBE_ADMIN_PASSWORD}" > /dev/null


I just tried pulling down these changes, running locally, and I got hit with the following error:

$ make crc ... echo "Deploying OLM" Deploying OLM ./scripts/crc-deploy.sh Deploying OLM to CRC Logging in as kubeadmin error: couldn't get https://127.0.0.1:42973/.well-known/oauth-authorization-server: unexpected response status 403 make: *** [Makefile:180: crc-deploy] Error 1

Going to try and manually run the crc-deploy Makefile target and see whether that authentication issue is reproducible.

Okay I figured out the issue: in the case where a user already has a defined $KUBECONFIG variable locally, and that kubeconfig contains many contexts', we're not changing the default context to the crc instance:

$ echo $KUBECONFIG /home/tflannag/.kube/config $ export KUBECONFIG=${KUBECONFIG:-${HOME}/.crc/machines/crc/kubeconfig} $ echo $KUBECONFIG /home/tflannag/.kube/config $ $ KUBE_ADMIN_USER=$(crc console --credentials -o json | jq -r .clusterConfig.adminCredentials.username) $ KUBE_ADMIN_PASSWORD=$(crc console --credentials -o json | jq -r .clusterConfig.adminCredentials.password) $ oc login -u "${KUBE_ADMIN_USER}" -p "${KUBE_ADMIN_PASSWORD}" > /dev/null Logging in as kubeadmin error: couldn't get https://127.0.0.1:42973/.well-known/oauth-authorization-server: unexpected response status 403 $ crc status CRC VM: Running OpenShift: Running (v4.10.3) Podman: Disk Usage: 19.83GB of 32.74GB (Inside the CRC VM) Cache Usage: 103.6GB Cache Directory: /home/tflannag/.crc/cache $ kubectx -c kind-kind $ # switch to the crc-admin context $ kubectx crc-admin $ oc login -u "${KUBE_ADMIN_USER}" -p "${KUBE_ADMIN_PASSWORD}" > /dev/null $ echo $? 0

timflannagan · 2022-04-01T16:40:59Z

scripts/crc-deploy.sh

+  ${YQ} write --inplace -s scripts/psm-operator-deployment.crc.e2e.patch.yaml manifests/0000_50_olm_06-psm-operator.deployment.yaml
+  ${YQ} write --inplace -s scripts/collect-profiles.crc.e2e.patch.yaml manifests/0000_50_olm_07-collect-profiles.cronjob.yaml


Ran into another error while running this locally and manually setting the current kubecontext to crc-admin:

$ make crc-deploy ... wrote ./tmp.hxG09zV0mc/chart/olm/templates/0000_50_olm_13-operatorgroup-default.yaml wrote ./tmp.hxG09zV0mc/chart/olm/templates/0000_50_olm_13-operatorgroup-default.yaml wrote ./tmp.hxG09zV0mc/chart/olm/templates/0000_90_olm_01-prometheus-rule.yaml wrote ./tmp.hxG09zV0mc/chart/olm/templates/0000_90_olm_00-service-monitor.yaml wrote ./tmp.hxG09zV0mc/chart/olm/templates/0000_90_olm_00-service-monitor.yaml go run: no packages loaded from ./vendor/github.com/mikefarah/yq/v3/ make: *** [Makefile:180: crc-deploy] Error 1

timflannagan · 2022-04-01T16:44:13Z

scripts/crc-deploy.sh

+}
+
+# YQ for applying yaml patches
+YQ="go run ./vendor/github.com/mikefarah/yq/v3/"


See my earlier comment about running into issues when getting to the part in this workflow where we use yq. I tried running the following command locally thinking we might have support for overriding YQ:

$ make crc-deploy YQ=/usr/local/bin/yq ... go run: no packages loaded from ./vendor/github.com/mikefarah/yq/v3/

Maybe this is something we can override just like the KUBECONFIG environment variable?

Andddd I tried manually overriding this in the crc-deploy.sh bash script and realized I have a v4 version locally:

$ make crc-deploy YQ=/usr/local/bin/yq ... Error: write inplace cannot be used with split file make: *** [Makefile:180: crc-deploy] Error 1 $ yq --version yq (https://github.com/mikefarah/yq/) version 4.23.1

It looks like we're stranding some changes made while generating the CRC patches:

$ git status modified: manifests/0000_50_olm_07-olm-operator.deployment.ibm-cloud-managed.yaml modified: manifests/0000_50_olm_07-olm-operator.deployment.yaml modified: manifests/0000_50_olm_08-catalog-operator.deployment.ibm-cloud-managed.yaml modified: manifests/0000_50_olm_08-catalog-operator.deployment.yaml modified: pkg/manifests/csv.yaml

Maybe we need to wrap this in it's own function so we can enact a cleanup function (e.g. git restore manifests) when we run into a non-zero exit code?

timflannagan · 2022-04-01T17:05:03Z

/lgtm

perdasilva · 2022-04-01T17:15:09Z

/override e2e-gcp
/override e2e-gcp-console-olm
/override e2e-upgrade

Overriding as these changes don't affect anything in the product or pipeline

openshift-ci · 2022-04-01T17:15:47Z

@perdasilva: /override requires a failed status context or a job name to operate on.
The following unknown contexts were given:

e2e-gcp
e2e-gcp-console-olm
e2e-upgrade

Only the following contexts were expected:

ci/prow/e2e-gcp
ci/prow/e2e-gcp-console-olm
ci/prow/e2e-gcp-olm
ci/prow/e2e-gcp-olm-flaky
ci/prow/e2e-upgrade
ci/prow/images
ci/prow/unit-api
ci/prow/unit-olm
ci/prow/unit-psm
ci/prow/unit-registry
ci/prow/verify
pull-ci-openshift-operator-framework-olm-master-e2e-gcp
pull-ci-openshift-operator-framework-olm-master-e2e-gcp-console-olm
pull-ci-openshift-operator-framework-olm-master-e2e-gcp-olm
pull-ci-openshift-operator-framework-olm-master-e2e-gcp-olm-flaky
pull-ci-openshift-operator-framework-olm-master-e2e-upgrade
pull-ci-openshift-operator-framework-olm-master-images
pull-ci-openshift-operator-framework-olm-master-unit-api
pull-ci-openshift-operator-framework-olm-master-unit-olm
pull-ci-openshift-operator-framework-olm-master-unit-psm
pull-ci-openshift-operator-framework-olm-master-unit-registry
pull-ci-openshift-operator-framework-olm-master-verify
tide

In response to this:

/override e2e-gcp
/override e2e-gcp-console-olm
/override e2e-upgrade

Overriding as these changes don't affect anything in the product or pipeline

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

timflannagan · 2022-04-01T17:26:20Z

/override ci/prow/e2e-gcp
/override ci/prow/e2e-upgrade

openshift-ci · 2022-04-01T17:26:53Z

@timflannagan: /override requires a failed status context or a job name to operate on.
The following unknown contexts were given:

ci/prow/e2e-gcp-console

Only the following contexts were expected:

ci/prow/e2e-gcp
ci/prow/e2e-gcp-console-olm
ci/prow/e2e-gcp-olm
ci/prow/e2e-gcp-olm-flaky
ci/prow/e2e-upgrade
ci/prow/images
ci/prow/unit-api
ci/prow/unit-olm
ci/prow/unit-psm
ci/prow/unit-registry
ci/prow/verify
pull-ci-openshift-operator-framework-olm-master-e2e-gcp
pull-ci-openshift-operator-framework-olm-master-e2e-gcp-console-olm
pull-ci-openshift-operator-framework-olm-master-e2e-gcp-olm
pull-ci-openshift-operator-framework-olm-master-e2e-gcp-olm-flaky
pull-ci-openshift-operator-framework-olm-master-e2e-upgrade
pull-ci-openshift-operator-framework-olm-master-images
pull-ci-openshift-operator-framework-olm-master-unit-api
pull-ci-openshift-operator-framework-olm-master-unit-olm
pull-ci-openshift-operator-framework-olm-master-unit-psm
pull-ci-openshift-operator-framework-olm-master-unit-registry
pull-ci-openshift-operator-framework-olm-master-verify
tide

In response to this:

/override ci/prow/e2e-gcp
/override ci/prow/e2e-upgrade
/override ci/prow/e2e-gcp-console

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

perdasilva · 2022-04-01T17:38:38Z

/override ci/prow/e2e-gcp
/override ci/prow/e2e-gcp-console-olm
/override ci/prow/e2e-upgrade

openshift-ci · 2022-04-01T17:41:41Z

@perdasilva: Overrode contexts on behalf of perdasilva: ci/prow/e2e-gcp, ci/prow/e2e-gcp-console-olm, ci/prow/e2e-upgrade

In response to this:

/override ci/prow/e2e-gcp
/override ci/prow/e2e-gcp-console-olm
/override ci/prow/e2e-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-04-01T17:41:54Z

@perdasilva: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Signed-off-by: Jordan Keister <[email protected]> Signed-off-by: Jordan Keister <[email protected]> Upstream-repository: api Upstream-commit: 028731a4f915f3a843634554ccc21d78b1d89859

openshift-ci bot requested review from exdx and kevinrizza March 22, 2022 22:16

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 22, 2022

perdasilva force-pushed the tools_and_docs branch from 1b073df to be0f5db Compare March 22, 2022 22:16

timflannagan reviewed Mar 23, 2022

View reviewed changes

docs/local-testing-with-crc.md Outdated Show resolved Hide resolved

timflannagan reviewed Mar 23, 2022

View reviewed changes

perdasilva force-pushed the tools_and_docs branch 2 times, most recently from 85a5852 to 36e984b Compare March 28, 2022 12:43

perdasilva force-pushed the tools_and_docs branch from 36e984b to 7fd8f2a Compare March 30, 2022 11:07

perdasilva force-pushed the tools_and_docs branch 2 times, most recently from 3ecbf71 to d2bbe5e Compare March 31, 2022 17:18

dinhxuanvu reviewed Mar 31, 2022

View reviewed changes

timflannagan reviewed Mar 31, 2022

View reviewed changes

perdasilva force-pushed the tools_and_docs branch 5 times, most recently from e90d0ad to a262379 Compare April 1, 2022 09:08

Add CRC startup and deployment automation

810287b

Signed-off-by: perdasilva <[email protected]>

perdasilva force-pushed the tools_and_docs branch from a262379 to 810287b Compare April 1, 2022 13:52

timflannagan reviewed Apr 1, 2022

View reviewed changes

openshift-ci bot assigned timflannagan Apr 1, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 1, 2022

openshift-merge-robot merged commit 98996c5 into openshift:master Apr 1, 2022

		echo "Logging in as kubeadmin"
		oc login -u "${KUBE_ADMIN_USER}" -p "${KUBE_ADMIN_PASSWORD}" > /dev/null

		${YQ} write --inplace -s scripts/psm-operator-deployment.crc.e2e.patch.yaml manifests/0000_50_olm_06-psm-operator.deployment.yaml
		${YQ} write --inplace -s scripts/collect-profiles.crc.e2e.patch.yaml manifests/0000_50_olm_07-collect-profiles.cronjob.yaml

Adds CRC deployment automation and CI documentation #268

Adds CRC deployment automation and CI documentation #268

Conversation

perdasilva commented Mar 22, 2022 • edited Loading

openshift-ci bot commented Mar 22, 2022

perdasilva commented Mar 23, 2022

perdasilva commented Mar 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

perdasilva commented Mar 24, 2022

timflannagan commented Mar 25, 2022

perdasilva commented Mar 30, 2022

perdasilva commented Mar 30, 2022

perdasilva commented Mar 31, 2022

dinhxuanvu left a comment

Choose a reason for hiding this comment

dinhxuanvu Mar 31, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

perdasilva commented Apr 1, 2022

perdasilva commented Apr 1, 2022

perdasilva commented Apr 1, 2022

perdasilva commented Apr 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timflannagan commented Apr 1, 2022

perdasilva commented Apr 1, 2022

openshift-ci bot commented Apr 1, 2022

timflannagan commented Apr 1, 2022 • edited Loading

openshift-ci bot commented Apr 1, 2022

perdasilva commented Apr 1, 2022

openshift-ci bot commented Apr 1, 2022

openshift-ci bot commented Apr 1, 2022

perdasilva commented Mar 22, 2022 •

edited

Loading

dinhxuanvu Mar 31, 2022 •

edited

Loading

timflannagan commented Apr 1, 2022 •

edited

Loading