feat(backend): Upgrade Argo server image version to v2.11.6 #4693

xinbinhuang · 2020-10-30T00:51:06Z

Description of your changes:

Part of #4553

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

PR titles examples:
- fix(frontend): fixes empty page. Fixes #1234
  Use fix to indicate that this PR fixes a bug.
- feat(backend): configurable service account. Fixes #1234, fixes #1235
  Use feat to indicate that this PR adds a new feature.
- chore: set up changelog generation tools
  Use chore to indicate that this PR makes some changes that users don't need to know.
- test: fix CI failure. Part of #1234
  Use part of to indicate that a PR is working on an issue, but shouldn't close the issue when merged.
Do you want this pull request (PR) cherry-picked into the current release branch?

If yes, use one of the following options:
- (Recommended.) Ask the PR approver to add the cherrypick-approved label to this PR. The release manager adds this PR to the release branch in a batch update.
- After this PR is merged, create a cherry-pick PR to add these changes to the release branch. (For more information about creating a cherry-pick PR, see the Kubeflow Pipelines release guide.)

kubeflow-bot · 2020-10-30T00:51:12Z

This change is

k8s-ci-robot · 2020-10-30T00:51:18Z

Hi @xinbinhuang. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

xinbinhuang · 2020-10-30T01:12:48Z

backend/Dockerfile

@@ -15,7 +15,7 @@ COPY backend/requirements.txt .
 RUN python3 -m pip install -r requirements.txt

 # Downloading Argo CLI so that the samples are validated
-#ADD  https://github.com/argoproj/argo/releases/download/v2.7.5/argo-linux-amd64 /usr/local/bin/argo
+#ADD  https://github.com/argoproj/argo/releases/download/v2.11.6/argo-linux-amd64 /usr/local/bin/argo


I replace all v2.7.5 -> v2.11.6 in the repo, but I am not sure if I need to update this one.

I think we should leave this one as well since it is the Kubeflow backend which will be running 2.7.7. Argo is in this case used to validate the workflows to check if they are valid according to argo.

It seems that we have several places that use Argo but they can be somehow independent of each other. Can you tell me how each of them is used so that I can get a clearer picture of how they work together?

xinbinhuang · 2020-10-30T01:13:02Z

test/install-argo.sh

@@ -25,7 +25,7 @@ ACCOUNT=$(gcloud info --format='value(config.account)')
 kubectl create clusterrolebinding PROW_BINDING --clusterrole=cluster-admin --user=$ACCOUNT --dry-run -o yaml | kubectl apply -f -
 kubectl create clusterrolebinding DEFAULT_BINDING --clusterrole=cluster-admin --serviceaccount=default:default --dry-run -o yaml | kubectl apply -f -

-ARGO_VERSION=v2.7.5
+ARGO_VERSION=v2.11.6


Not sure if I need to update this one

I think this will be needed for all the test infrastructure to run with argo 2.11.6

xinbinhuang · 2020-10-30T01:13:54Z

manifests/kustomize/base/argo/workflow-controller-deployment.yaml

@@ -27,7 +27,8 @@ spec:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
-        - gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance
+        - gcr.io/ml-pipeline/argoexec:v2.11.6-license-compliance
+        - --namespaced


This is the extra argument that I need to specifically add for my deployment.

Is there a corresponding documentation? Does omitting the argument make it cluster mode? We need to make sure it is customizable from an overlay to continue support multi-user mode.

The cluster vs namespace install seems to start since v2.5 - this can also be found in the v2.5 release branch.

In my case, it did become a cluster install if I simply replace the image version to argoproj/argoexec:v2.11.6 - I got permission issues when trying to launch new pipelines.

I am not familiar with the multi-user mode. Does it install the workflow-controller in cluster scope?

Error log from when trying to launch pipelines without --namespaced

E1107 01:14:06.943364 1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *unstructured.Unstructured: workflowtemplates.argoproj.io is forbidden: User "system:serviceaccount:kubeflow:argo" cannot list resource "workflowtemplates" in API group "argoproj.io" at the cluster scope

xinbinhuang · 2020-10-30T01:16:33Z

test/sample-test/Dockerfile

+#RUN ARGO_VERSION=v2.11.6 && curl -sSL -o /usr/local/bin/argo \
 RUN ARGO_VERSION=v2.4.3 && curl -sSL -o /usr/local/bin/argo \


Not sure about this one either.

This is used to kick of the ML pipeline sample e2e test, since the changes you do is only for the server side I think we should not update this to 2.11.6. But I think we should go with 2.7.7 since that is the version that is used by Kubeflow today.

There was a bug from argo that prevented using the argo cli from a container, that's why we were stuck on 2.4.3. I'm not sure about latest status, if the latest version works, we can upgrade too.
/cc @Ark-kun can you provide context about that issue?

xinbinhuang · 2020-10-30T01:17:07Z

Hey, @NikeNano @Bobgy . PTAL

My deployment is actually running v2.11.1 but I expect that there is no big difference. If there are any weird issues coming up, I can also upgrade my deployment to test it out.

/assign @Bobgy

xinbinhuang · 2020-10-30T01:24:58Z

BTW, I was following the instructions from here: https://github.com/kubeflow/pipelines/blob/master/third_party/README.md

I skipped step 6.

juliusvonkohout · 2020-10-31T15:16:22Z

I am also using 2.11.6 on my cluster. You should remove the executor image from the configmap. As far as I know it is deprecated and should only be in the workflow-controller deployment

NikeNano · 2020-10-31T20:44:55Z

I realised that your changes are only for the argo server and not for Kubeflow it self, thus this PR will not completely solve: #4553 as I understand it. I am also not sure we we should update the argo server on its own since this will result in that Kubeflow pipelines use 2.7 and the argo server uses 2.11 of argo.

Bobgy

You'd also need to update licenses, I created an issue to track updating documentation for these instructions: #4715.

Please keep your eyes on it.

Bobgy · 2020-11-04T06:22:21Z

manifests/kustomize/base/argo/workflow-controller-deployment.yaml

@@ -27,7 +27,8 @@ spec:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
-        - gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance
+        - gcr.io/ml-pipeline/argoexec:v2.11.6-license-compliance
+        - --namespaced


Is there a corresponding documentation? Does omitting the argument make it cluster mode? We need to make sure it is customizable from an overlay to continue support multi-user mode.

Bobgy · 2020-11-04T06:24:13Z

test/sample-test/Dockerfile

+#RUN ARGO_VERSION=v2.11.6 && curl -sSL -o /usr/local/bin/argo \
 RUN ARGO_VERSION=v2.4.3 && curl -sSL -o /usr/local/bin/argo \


There was a bug from argo that prevented using the argo cli from a container, that's why we were stuck on 2.4.3. I'm not sure about latest status, if the latest version works, we can upgrade too.
/cc @Ark-kun can you provide context about that issue?

k8s-ci-robot · 2020-11-07T00:44:27Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xinbinhuang
To complete the pull request process, please assign bobgy after the PR has been reviewed.
You can assign the PR to them by writing /assign @bobgy in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xinbinhuang · 2020-11-10T00:06:24Z

I realised that your changes are only for the argo server and not for Kubeflow it self, thus this PR will not completely solve: #4553 as I understand it. I am also not sure we we should update the argo server on its own since this will result in that Kubeflow pipelines use 2.7 and the argo server uses 2.11 of argo.

You are correct - this PR only upgrades argo server only. I can also bake in the changes for upgrading Kubeflow, but I wonder if we should separate it into a different PR? or should they go into the same one? @Bobgy @NikeNano

Bobgy · 2020-11-11T08:25:24Z

@xinbinhuang I think we can go with separate, previously we've upgraded both at different pace because they had different complexity.

Bobgy · 2020-11-11T08:26:30Z

/ok-to-test

xinbinhuang · 2020-11-12T05:50:00Z

/test kubeflow-pipeline-sample-test

xinbinhuang · 2020-11-12T06:18:05Z

/retest

k8s-ci-robot · 2020-11-12T06:33:11Z

@xinbinhuang: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
kubeflow-pipeline-upgrade-test	`44e41e6`	link	`/test kubeflow-pipeline-upgrade-test`
kubeflow-pipeline-e2e-test	`44e41e6`	link	`/test kubeflow-pipeline-e2e-test`
kubeflow-pipeline-sample-test	`44e41e6`	link	`/test kubeflow-pipeline-sample-test`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Bobgy · 2020-11-12T09:12:12Z

Hi @xinbinhuang, you'll need to update https://github.com/kubeflow/pipelines/tree/master/third_party/argo with instructions that should be documented in #4715. I'd suggest creating a separate PR for it. I'll publish images like gcr.io/ml-pipeline/workflow-controller:v2.11.6-license-compliance once your PR is merged.

xinbinhuang · 2021-01-16T00:02:49Z

@NikeNano @Bobgy @capri-xiyue I noticed some conversation on a more robust upgrading strategy to Argo3 #4999. I didn't quite have much time to finish this PR in the last year ending. I wonder if I should just close this PR so that whoever are upgrading it can start from scratch?

alexec · 2021-01-16T01:19:09Z

You might want to go to v2.12 which is LTS.

Bobgy · 2021-01-20T09:24:31Z

@alexec will we have the kubernetes client update backported to argo 2.12?

It was one of the big reasons we wanted to upgrade.

Bobgy · 2021-01-20T09:26:46Z

To @xinbinhuang, agree we can close this one if you do not have bandwidth to finish it.

/close

k8s-ci-robot · 2021-01-20T09:26:51Z

@Bobgy: Closed this PR.

In response to this:

To @xinbinhuang, agree we can close this one if you do not have bandwidth to finish it.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

google-cla bot added the cla: yes label Oct 30, 2020

k8s-ci-robot added size/XS needs-ok-to-test labels Oct 30, 2020

k8s-ci-robot requested review from Bobgy and rmgogogo October 30, 2020 00:51

xinbinhuang added 2 commits October 29, 2020 17:52

Upgrade Argo workflow controller manifest

538e167

fixup! Upgrade Argo workflow controller manifest

94b2947

xinbinhuang force-pushed the upgrade_argo_controller_version branch from 3b702c4 to 94b2947 Compare October 30, 2020 00:52

xinbinhuang changed the title ~~Upgrade Argo server version~~ feat(backend): Upgrade Argo server version Oct 30, 2020

Upgrade argo docker base image version

36684b1

k8s-ci-robot added size/S and removed size/XS labels Oct 30, 2020

xinbinhuang added 2 commits October 29, 2020 17:58

fixup! Upgrade argo docker base image version

c49b70c

fixup! fixup! Upgrade argo docker base image version

c4998bf

k8s-ci-robot added size/M and removed size/S labels Oct 30, 2020

xinbinhuang commented Oct 30, 2020

View reviewed changes

fixup! fixup! fixup! Upgrade argo docker base image version

80db306

xinbinhuang commented Oct 30, 2020

View reviewed changes

k8s-ci-robot assigned Bobgy Oct 30, 2020

xinbinhuang changed the title ~~feat(backend): Upgrade Argo server version~~ feat(backend): Upgrade Argo server image version Oct 30, 2020

Bobgy mentioned this pull request Nov 4, 2020

[Process] license management workflow #4715

Closed

Bobgy reviewed Nov 4, 2020

View reviewed changes

Bobgy changed the title ~~feat(backend): Upgrade Argo server image version~~ feat(backend): Upgrade Argo server image version to v2.11.6 Nov 4, 2020

Remove deprecated executorImage key from configmap

44e41e6

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Nov 11, 2020

This comment has been minimized.

Sign in to view

yanniszark mentioned this pull request Jan 12, 2021

Update Argo manifest to 2.5.0 kubeflow/manifests#1099

Closed

Ark-kun mentioned this pull request Jan 15, 2021

[WIP]feat(deployment):changed containerRuntimeExecutor to pns #4965

Closed

2 tasks

k8s-ci-robot closed this Jan 20, 2021

Bobgy mentioned this pull request Mar 9, 2021

feat: update argo image to v2.12.9 and automate update process. Fixes #5232 #5266

Merged

2 tasks

juliusvonkohout mentioned this pull request May 21, 2021

[feature] default to the emissary executor #5718

Closed

10 tasks

		#RUN ARGO_VERSION=v2.11.6 && curl -sSL -o /usr/local/bin/argo \
		RUN ARGO_VERSION=v2.4.3 && curl -sSL -o /usr/local/bin/argo \

feat(backend): Upgrade Argo server image version to v2.11.6 #4693

feat(backend): Upgrade Argo server image version to v2.11.6 #4693

Conversation

xinbinhuang commented Oct 30, 2020 • edited by Bobgy Loading

kubeflow-bot commented Oct 30, 2020

k8s-ci-robot commented Oct 30, 2020

xinbinhuang Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinbinhuang Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinbinhuang Nov 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinbinhuang commented Oct 30, 2020 • edited Loading

xinbinhuang commented Oct 30, 2020

juliusvonkohout commented Oct 31, 2020

NikeNano commented Oct 31, 2020

Bobgy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 7, 2020

xinbinhuang commented Nov 10, 2020

Bobgy commented Nov 11, 2020

Bobgy commented Nov 11, 2020

xinbinhuang commented Nov 12, 2020

xinbinhuang commented Nov 12, 2020

k8s-ci-robot commented Nov 12, 2020

Bobgy commented Nov 12, 2020

This comment has been minimized.

xinbinhuang commented Jan 16, 2021 • edited Loading

alexec commented Jan 16, 2021

Bobgy commented Jan 20, 2021

Bobgy commented Jan 20, 2021

k8s-ci-robot commented Jan 20, 2021

xinbinhuang commented Oct 30, 2020 •

edited by Bobgy

Loading

xinbinhuang Oct 30, 2020 •

edited

Loading

xinbinhuang Oct 30, 2020 •

edited

Loading

xinbinhuang Nov 7, 2020 •

edited

Loading

xinbinhuang commented Oct 30, 2020 •

edited

Loading

xinbinhuang commented Jan 16, 2021 •

edited

Loading