Add Jenkins scripts for running system tests on s390x nodes #6354

kun-lu20 · 2022-02-11T20:02:14Z

Type of change

Select the type of your PR

Enhancement / new feature

Description

Please describe your pull request

This PR is to add s390x Jenkins scripts so that system tests could be run on s390x Jenkins nodes.

Currently this PR includes two patches on s390x. However, they will be removed soon once next version of corresponding artifacts are released.

Patch for Kaniko image. We have raised PR to add s390x support. The Kaniko community has merged this PR and v1.8.0 will be released very soon. See Add s390x support to docker images GoogleContainerTools/kaniko#1749
Patch for rocksdbjni jar file. We have raised PRs in rocksdb repo and Apache Kafka repo to add s390x support. The communities have merged our PRs. rocskdbjni-6.27.3.jar has already been released and Kafka v3.2.0 will include it. See Add support for building on s390x platform facebook/rocksdb#8962 and KAFKA-13599: Upgrade RocksDB to 6.27.3 apache/kafka#11690

We've run Jenkins pipeline internally for testing. It went smoothly with STs in acceptance and other profiles. The logs are attached below. Also we're making good progress on the public Jenkins set up and will share the link to show live result once we're done.

run_acceptance_profile_on_s390x.log
run_keycloak_related_STs_on_s390x.log
run_azp_kafka_oauth_profile_on_s390x.log
run_azp_operators_profile_on_s390x.log
run_azp_connect_mirrormaker_profile_on_s390x.log
run_azp_rolling_update_bridge_profile_on_s390x.log
run_azp_security_profile_on_s390x.log
run_azp_dynconfig_listeners_tracing_watcher_profile_on_s390x.log
run_azp_remaining_profile_on_s390x.log

Checklist

Please go through this checklist and make sure all applicable tasks have been done

Write tests
Make sure all tests pass
Update documentation
Check RBAC rights for Kubernetes / OpenShift roles
Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
Reference relevant issue(s) and close them after merging
Update CHANGELOG.md
Supply screenshots for visual changes, such as Grafana dashboards

Signed-off-by: Kun-Lu <[email protected]>

strimzi-ci · 2022-02-11T20:07:02Z

Can one of the admins verify this patch?

kun-lu20 · 2022-02-11T20:10:27Z

I noticed that Operator v0.28.0 is on its way. Could you please take a look at this PR? Thanks very much! @scholzj @Frawless

Signed-off-by: Kun-Lu <[email protected]>

.jenkins/scripts/kafka-rocksdbjni-s390x.patch

Signed-off-by: Kun-Lu <[email protected]>

scholzj

Some more minor comments, but LGTM otherwise I guess. I have pretty much zero experience with Groovy or Jenkins. So I think the review from @Frawless ill be more important than mine.

scholzj · 2022-02-15T14:52:46Z

.jenkins/Jenkinsfile-s390x

+        OPERATOR_IMAGE_PULL_POLICY = "IfNotPresent"
+        COMPONENTS_IMAGE_PULL_POLICY = "IfNotPresent"
+        JAVA_VERSION = "11"
+        TEST_HELM2_VERSION = "v2.17.0"


Is this still needed? I don't think we use Helm 2 for anything anymore.

No, it is not needed. I've removed it.

scholzj · 2022-02-15T14:53:43Z

.jenkins/Jenkinsfile-s390x

+        PATH = "${env.JAVA_HOME}/bin:${env.PATH}"
+        DOCKER_BUILDX = "buildx"
+        DOCKER_BUILD_ARGS = "--platform linux/s390x --load"
+        BRIDGE_IMAGE = "quay.io/strimzi/kafka-bridge:latest"


What is this used for? I think the system tests should be normally using the pre-defined bridge version from the installation files and not latest.

I used version latest for testing before Bridge 0.21.4 which supports s390x was released. I've removed this line. Thanks @scholzj !

Right, makes sense.

Signed-off-by: Kun-Lu <[email protected]>

Frawless

I think you can remove a lot of duplicated code. For example jenkins-s390x.groovy contains a lot of code which is the same for x86.

You can also share parts of the pipeline itself and based on the parameter (node label) distinguish it.

Frawless · 2022-02-16T08:30:01Z

.jenkins/Jenkinsfile-s390x

+        failure {
+            println("Build failed")
+            script {
+                lib.sendMail(env.STRIMZI_MAILING_LIST, "failed", env.ghprbPullId, env.ghprbActualCommitAuthor, env.ghprbPullTitle, env.ghprbPullLink, env.BUILD_URL)


Don't forget to set STRIMZI_MAILING_LIST env in your Jenkins, otherwise it will fail on it I think

kun-lu20 · 2022-02-16T15:22:43Z

Thanks @Frawless ! I'll remove the duplicate code and get back to you soon.

Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 · 2022-02-18T17:08:21Z

Hi @Frawless , thanks for your advice!

I've refactored the scripts and removed the redundancy. Could you please take a look? Thank you so much!

kun-lu20 · 2022-02-18T22:51:27Z

Below is the log of running STs in acceptance profile in Jenkins pipeline on s390x after refactoring the scripts:
run_acceptance_profile_on_s390x_after_refactoring.log

kun-lu20 · 2022-02-23T15:17:40Z

Hi @Frawless , hope all is well.

Could you please have a look at the code change when you have some time? Thanks!

scholzj · 2022-02-23T15:19:08Z

@kun-lu20 Jakub is on PTO this and next week. I'm afraid his review is really needed for this (as he is pretty much the only maintainer who knows Jenkins), so we might need to wait.

kun-lu20 · 2022-02-23T15:26:04Z

@scholzj Got it. Thanks for letting me know!

Frawless · 2022-02-23T17:43:37Z

.jenkins/Jenkinsfile-pr

@@ -7,7 +7,7 @@ def DEFAULT_EXCLUDED_GROUPS = "loadbalancer,networkpolicies"
 pipeline {
    agent {
        node {
-            label "strimzi-pr"
+            label "strimzi-pr || strimzi-s390x"


I don't think this is a good idea. Which node will use it when jenkins will have both of the labels configured? I think we should have 2 different pipelines, one for s390x and one for amd64, and change specific parts like call different function to deploy minikube.

Thanks @Frawless for your time and detailed comments!

OK, I'll adjust the scripts to use 2 different pipelines.

Frawless · 2022-02-23T17:49:43Z

.jenkins/Jenkinsfile-pr

+                    if(env.TEST_ARCH.contains('s390x')){
+                        //Use hardcoding for ghprbCommentBody since we are not setting up GitHub PRB integration.
+                        env.ghprbCommentBody = "ok to test"
+                    }


so you don't want to run the tests from PR directly? If no, what value does this adds to us? Are you going to run some nightly builds? Is the jenkins instance available for the community to see the results?

Yes, we were going to run nightly builds. But we could also run the tests from PR directly, I'll make adjustments to the pipeline script. The Jenkins link will be available for the community to see the results.

Frawless · 2022-02-23T17:50:07Z

.jenkins/Jenkinsfile-pr

+                        if(env.TEST_ARCH.contains('s390x')){
+                            env.DOCKER_TAG="latest"
+                        }else{
+                            env.DOCKER_TAG="pr"
+                        }


Why you need to change tag here?

Yes, we do not need to change tag here since we're going to run the tests from PR directly as mentioned above.

Frawless · 2022-02-23T17:50:27Z

.jenkins/Jenkinsfile-pr

+                        if(env.TEST_ARCH.contains('s390x')){
+                            env.TEST_COUNT_RUNNING_IN_PARALLEL = "1"
+                        }else{
+                            env.TEST_COUNT_RUNNING_IN_PARALLEL = "3"
+                        }


Any specific reason for this?

We found that when setting this arg to 3, some test cases would become unstable and it will cost extra time to rerun them. Probably it's because that we've set MINIKUBE_MEMORY to 8192.

So we followed the arg value used in community's Azure Pipelines and the unstable issue disappeared.

Ye, that makes sense. @see-quick jfyi

Frawless · 2022-02-23T17:52:42Z

.jenkins/Jenkinsfile-pr

+                if(env.TEST_ARCH.contains('s390x')){
+                    sh(script: "minikube stop || true")
+                    sh(script: "minikube delete || true")
+                    sh(script: "docker stop s390x-registry || true")
+                    sh(script: "docker rm s390x-registry || true")
+                }


You are going to use some static machine that you need this? If so, you should probably also remove all images

Yes, we used a static VM to run the Jenkins pipeline. We could switch to run the pipeline in a docker container so that we do not need these commands.

1. Create a separate pipeline to run STs on s390x 2. Run STs in ub worker containers Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 · 2022-02-26T02:00:23Z

Hi @Frawless , I've updated the code as per your review comments. The main changes are:

Create a separate pipeline (Jenkinsfile-pr-s390x) to run STs on s390x.
Run STs in Jenkins worker containers instead of VMs.
Can run nightly builds or builds triggered by PRs.

Thank you so much for taking the time to do the previous review. We appreciate that. Please have a look at the code changes if you have some time. Thanks again!

Frawless · 2022-02-28T12:37:38Z

.azure/scripts/setup-kubernetes.sh

-    minikube addons enable registry
+    if [ "$ARCH" = "s390x" ]; then
+        git clone -b v1.9.11 --depth 1 https://github.com/kubernetes/kubernetes.git
+        cd kubernetes/cluster/addons/registry/images/


is this needed?

Yes, this is needed because currently the registry addon doesn't have s390x support yet, we need to build kube-registry-proxy image locally.

I mean the cd command, wouldn't be better just to use relative path?

I got it. I've changed the code to use relative path to build kube-registry-proxy image.

.jenkins/scripts/Dockerfile_base_s390x

Frawless · 2022-02-28T12:42:49Z

.jenkins/Jenkinsfile-pr-s390x

+                    sh(script: "sudo ln -s /opt/apache-maven-3.8.4/bin/mvn /usr/bin/mvn")
+                    sh(script: "sudo ln -s /opt/apache-maven-3.8.4/bin/mvn /usr/local/bin/mvn")


isn't there some better solution for this? Maybe on container preparation level?

Thanks! I've moved this to container preparation level (Dockerfile_base_s390x).

.jenkins/Jenkinsfile-pr-s390x

Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 · 2022-02-28T18:45:14Z

Hi @Frawless , I've made code changes as per your review comments. Please take a look when you have some time.

Thank you so much!

Frawless · 2022-02-28T19:09:01Z

.jenkins/Jenkinsfile-pr-s390x

+                        if ("${env.TEST_PROFILE}".contains("upgrade")) {
+                            println("[INFO] Update files for upgrade procedure")
+                            sh(script: """
+                                sed -i 's#:latest#:${env.DOCKER_TAG}#g' ${env.WORKSPACE}/systemtest/src/test/resources/upgrade/StrimziUpgradeST.json ${env.WORKSPACE}/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml
+                                sed -i 's#quay.io/strimzi/test-client:${env.DOCKER_TAG}#${env.DOCKER_REGISTRY}/strimzi/test-client:${env.DOCKER_TAG}#g' ${env.WORKSPACE}/systemtest/src/test/resources/upgrade/StrimziUpgradeST.json
+                                sed -i 's#quay.io/strimzi/#${env.DOCKER_REGISTRY}/strimzi/#g' ${env.WORKSPACE}/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml
+                                sed -i 's#/opt/${env.DOCKER_REGISTRY}#/opt#g' ${env.WORKSPACE}/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml
+                            """)
+                            sh(script: "cat ${env.WORKSPACE}/systemtest/src/test/resources/upgrade/StrimziUpgradeST.json")
+                            sh(script: "cat ${env.WORKSPACE}/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml")


I think you can put this into a method and call it in this and in x86 pipeline.

Frawless · 2022-02-28T19:14:51Z

@kun-lu20 I think the code is fine now. Now let's talk about access rights:

1 - Can you share jenkins URL?
2 - Who will be able to trigger the jobs?
3 - Who is the admin of the Jenkins? Is this person somehow connected to our community to listen to community needs?
4 - Which Jenkins plugin do you use for it?
5 - Which mailing list are you going to use for results reporting?

For amd64 we basically have everything on azure that is available to everyone. Comment trigger is available only to maintainers due to security reasons set by azure.

1. Use relative path when building kube-registry-proxy image 2. Add function prepareUpgradeSTs() in jenkins.groovy for reuse Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 · 2022-03-01T16:22:32Z

Hi @Frawless , thanks very much for your review comments. I've made the code change and here is the reply to your questions:

1 - The Jenkins url is https://148.100.84.136/job/kafka-op-build/job/add_jenkins_scripts_for_s390x/. We're still working on configurations and things might change in the future.

2 - We would like to start with running daily builds on acceptance profile. The pipeline will be triggered on a schedule. Since we are using a fixed number of VMs to host our Jenkins, we would like to limit triggering builds to maintainers as well.

3 - Our team will be the admin for Jenkins and we'll monitor the s390x builds and listen to community needs.

4 - We'll use PRB Jenkins plugin, I think the same plugin had been used on community's internal Jenkins.

5 - We're currently waiting for some ports to be opened on our vm's network so we don't have email reporting enabled yet. When it's working, can we use the same Strimzi mailing list as you do for internal Jenkins? Also do you think there are ways other than email that you can accept the results?

1. Replace hard-coded ids with build args in Dockerfile 2. Use $USER instead of env variable $ST_USER Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 · 2022-03-07T16:14:21Z

Hi @Frawless , could you please take a look? Do let me know if you have any further questions. Thank you so much!

BTW, the Jenkins url has been changed to https://148.100.84.211/job/strimzi-kafka-operator/job/Strimzi_Kafka_Operator_ST/. Please use this new link when accessing our Jenkins instance.

Frawless · 2022-03-07T20:22:49Z

@kun-lu20 Code changes are fine from my POV now. I have a few other points.

1 - The link should be available in some readme (I also expect it will be part of the checks on gh, but this is handled by the plugin itself)
2 - You need a user with contributor status in our organization, otherwise it won't be possible to scan the repo I think. Or have you tried it without credentials? For sure the results_info.sh won't work. Did you maybe think about using webhooks for triggering the jobs? It might be a better solution since your Jenkins is public
3 - mailing list - I think it might be possible, maybe better option to send it to PR author.

Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 · 2022-03-08T14:24:04Z

Hi @Frawless , according to your questions:

We've added a README-s390x.md in .jenkins folder.
Can we start with nightly builds on acceptance profile? Thus we do not need webhooks at the beginning. Once everything is stable, could you help us to create them later?
We do not have email connectivity yet and do not have an ETA. We're wondering if there are other ways to send you the results for now.

Could you please take a look? Thanks!

Frawless · 2022-03-09T09:40:55Z

1 - Info about which tests, how and when will be executed will be fine
2 - Yes we can.
3 - You can use Strimzi CNCF dev mailing list. However, you own this feature so you should react to any failures/changes needed for it at first.

Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 · 2022-03-09T15:59:24Z

Thanks @Frawless ! Reg your comments,

1 - Relevant info was added in README-s390x.md.
3 - Sure, we'll look into any failures/changes needed for it at first.

Frawless

Ok, PR looks good to me now. I assume you tried it and it's working without issues.

kun-lu20 · 2022-03-09T16:19:41Z

Yes, we've tried it and it's working fine. Thanks @Frawless !

kun-lu20 · 2022-03-09T18:07:18Z

Hi @scholzj , hope all is well.

Could you please merge this PR so that we can start to run our Jenkins nightly builds on main branch? Thank you very much!

scholzj · 2022-03-09T20:35:17Z

@kun-lu20 Thanks for the PR.

BTW: I opened #6494 which updates Kaniko to the new version. I believe it has your s390x support.

kun-lu20 · 2022-03-09T21:37:45Z

Thanks @scholzj !

Yes, Kaniko v1.8.0 has s390x support. Once #6494 is merged, I'll open another PR to drop the Kaniko patch from s390x Jenkins script.

Run s390x STs on Jenkins CI

831b5ed

Signed-off-by: Kun-Lu <[email protected]>

Minor changes

6ae06c5

Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 force-pushed the add_jenkins_scripts_for_s390x branch from ae48d74 to 6ae06c5 Compare February 11, 2022 21:19

scholzj added this to the 0.29.0 milestone Feb 12, 2022

scholzj reviewed Feb 13, 2022

View reviewed changes

.jenkins/scripts/kafka-rocksdbjni-s390x.patch Outdated Show resolved Hide resolved

scholzj requested a review from Frawless February 13, 2022 21:10

Remove rocksdb patch and update maven options

e85a11c

Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 force-pushed the add_jenkins_scripts_for_s390x branch from ec83560 to e85a11c Compare February 14, 2022 17:17

kun-lu20 requested a review from scholzj February 14, 2022 17:19

scholzj approved these changes Feb 15, 2022

View reviewed changes

Remove TEST_HELM2_VERSION and BRIDGE_IMAGE from env variables

ebe7a6b

Signed-off-by: Kun-Lu <[email protected]>

Frawless reviewed Feb 16, 2022

View reviewed changes

kun-lu20 added 3 commits February 17, 2022 17:55

Refactor to remove duplicate code

2f388ae

Signed-off-by: Kun-Lu <[email protected]>

Merge branch 'main' into add_jenkins_scripts_for_s390x

3c8f3ee

Signed-off-by: Kun-Lu <[email protected]>

Set the value of ARCH to amd64 if no arg is passed

252164b

Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 requested a review from Frawless February 18, 2022 17:08

Frawless reviewed Feb 23, 2022

View reviewed changes

Run STs in Jenkins worker container

d95a232

1. Create a separate pipeline to run STs on s390x 2. Run STs in ub worker containers Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 requested a review from Frawless February 26, 2022 02:00

Frawless reviewed Feb 28, 2022

View reviewed changes

Move ln commands to Dockerfile_base_s390x

01e7726

Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 requested a review from Frawless February 28, 2022 18:45

Frawless reviewed Feb 28, 2022

View reviewed changes

Code changes as per the review comments

f373112

1. Use relative path when building kube-registry-proxy image 2. Add function prepareUpgradeSTs() in jenkins.groovy for reuse Signed-off-by: Kun-Lu <[email protected]>

kun-lu20 requested a review from Frawless March 1, 2022 16:23

Minor changes

1c7da4d

1. Replace hard-coded ids with build args in Dockerfile 2. Use $USER instead of env variable $ST_USER Signed-off-by: Kun-Lu <[email protected]>

Add README-s390x.md in .jenkins dir

5fc1af6

Signed-off-by: Kun-Lu <[email protected]>

Update README_s390x.md

a8bb386

Signed-off-by: Kun-Lu <[email protected]>

Frawless approved these changes Mar 9, 2022

View reviewed changes

scholzj merged commit 7469c3c into strimzi:main Mar 9, 2022

Chandranana mentioned this pull request Apr 20, 2023

Add Jenkins Support for power architecture #8418

Merged

8 tasks

		sh(script: "sudo ln -s /opt/apache-maven-3.8.4/bin/mvn /usr/bin/mvn")
		sh(script: "sudo ln -s /opt/apache-maven-3.8.4/bin/mvn /usr/local/bin/mvn")

Add Jenkins scripts for running system tests on s390x nodes #6354

Add Jenkins scripts for running system tests on s390x nodes #6354

Conversation

kun-lu20 commented Feb 11, 2022 • edited Loading

Type of change

Description

Checklist

strimzi-ci commented Feb 11, 2022

kun-lu20 commented Feb 11, 2022

scholzj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Frawless left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kun-lu20 commented Feb 16, 2022

kun-lu20 commented Feb 18, 2022 • edited Loading

kun-lu20 commented Feb 18, 2022 • edited Loading

kun-lu20 commented Feb 23, 2022

scholzj commented Feb 23, 2022

kun-lu20 commented Feb 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kun-lu20 Feb 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kun-lu20 Feb 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kun-lu20 commented Feb 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kun-lu20 commented Feb 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Frawless commented Feb 28, 2022

kun-lu20 commented Mar 1, 2022

kun-lu20 commented Mar 7, 2022

Frawless commented Mar 7, 2022

kun-lu20 commented Mar 8, 2022

Frawless commented Mar 9, 2022

kun-lu20 commented Mar 9, 2022 • edited Loading

Frawless left a comment

Choose a reason for hiding this comment

kun-lu20 commented Mar 9, 2022

kun-lu20 commented Mar 9, 2022 • edited Loading

scholzj commented Mar 9, 2022

kun-lu20 commented Mar 9, 2022

kun-lu20 commented Feb 11, 2022 •

edited

Loading

Frawless left a comment •

edited

Loading

kun-lu20 commented Feb 18, 2022 •

edited

Loading

kun-lu20 commented Feb 18, 2022 •

edited

Loading

kun-lu20 Feb 23, 2022 •

edited

Loading

kun-lu20 Feb 23, 2022 •

edited

Loading

kun-lu20 commented Mar 9, 2022 •

edited

Loading

kun-lu20 commented Mar 9, 2022 •

edited

Loading