Add jenkins test pipeline for OpenJDK Project Builds #1192

jerboaa · 2019-06-25T12:33:31Z

This pipeline performs Github API calls so as to determine whether or not new releases are available for OpenJDK Project Builds released at:

JDK 11: https://github.com/AdoptOpenJDK/openjdk11-upstream-binaries/
JDK 8: https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries

If there are new releases, sanity.openjdk and sanity.system test jobs are being scheduled for each {version, architecture, platform} triplet.

Example run on Grinder_Sandbox (blue ocean view):
https://ci.adoptopenjdk.net/blue/organizations/jenkins/Grinder_Sandbox/detail/Grinder_Sandbox/495/pipeline

The idea would be to clone Grinder_Sandbox or Grinder Jenkins job configs and set up a separate job which runs daily or some such. The pipeline has functionality built in so as to not trigger new test jobs when the published_at date of the releases hasn't changed. Example for this scenario:

https://ci.adoptopenjdk.net/job/Grinder_Sandbox/494/console

Thoughts?

Closes #1183

This checks for updated releases first. Second, if there are any, trigger test build jobs for OpenJDK sanity and system tests for each platform/arch.

Example parallel job name: Test_openjdk8_hs_sanity.openjdk_x64_linux

- Set JDK_VERSION so as to not use the default (8) causing conflicts with AUTO_DETECT=true for JDK 11 test jobs - Schedule the API work on Linux x86 test nodes only as these are plenty.

smlambert · 2019-06-25T15:03:52Z

Also adding @sophia-guo as a reviewer. Thanks for pulling this together @jerboaa.

We can use the TestAutoGen job to generate the test jobs that this pipeline would kick off (and house those test jobs in a Test_upstream view/tab in Jenkins. (we may use a slightly different naming scheme than used here, in which case I will make some requests to update job names).

I will take a closer look later today, but am happy in principal with this initial step. We can refine or update later (if need be).

jerboaa · 2019-06-25T16:41:09Z

Also adding @sophia-guo as a reviewer. Thanks for pulling this together @jerboaa.

We can use the TestAutoGen job to generate the test jobs that this pipeline would kick off

Sure. I'd need guidance as to how to kick off jobs there via some parameters. If you could point me to an example, that'd be great.

(and house those test jobs in a Test_upstream view/tab in Jenkins. (we may use a slightly different naming scheme than used here, in which case I will make some requests to update job names).

OK. That would be a follow-up once the job needs creating right?

I will take a closer look later today, but am happy in principal with this initial step. We can refine or update later (if need be).

Thanks!

smlambert · 2019-06-25T16:47:16Z

Yes, my suggestions would be follow-up work (though we may decide not to need to do any updates).

I need to generate some other jobs today (relating to #1168), so I don't mind to try my hand to generate the jobs you need.

jerboaa · 2019-06-25T16:55:03Z

Yes, my suggestions would be follow-up work (though we may decide not to need to do any updates).

I see. Looking a bit closer TestAutoGen job seems to hard-code to buildenv/jenkins/testJobTemplate which doesn't seem flexible enough to generate those upstream test jobs. Perhaps you mean a new buildenv/jenkins/upstreamTestJobTemplate? I could perhaps update buildenv/jenkins/testJobTemplate to account for my use case too. Not sure.

I need to generate some other jobs today (relating to #1168), so I don't mind to try my hand to generate the jobs you need.

Oh, cool. I won't stop you ;-) Well then, a clone of the Grinder or Grinder_Sandbox config, with name say UpstreamAutotest or some such, would be great. I'll add the rest of the configury once a job has been set up. That job could be in Test_upstream view/tab in Jenkins as you suggest. Thanks!

sophia-guo · 2019-06-25T21:02:38Z

buildenv/jenkins/upstream_openjdk_tests

+                            parallelJobs["Test_openjdk${version}_hs_sanity.openjdk_${r.ARCH}_${r.OS}"] = {
+                                // Run a _sanity.openjdk test job with appropriate parameters
+                                build job: TEST_JOB_NAME, parameters: [
+                                    string(name: 'ADOPTOPENJDK_REPO', value: params.ADOPTOPENJDK_REPO ? params.ADOPTOPENJDK_REPO : "https://github.com/AdoptOpenJDK/openjdk-tests.git"),


I wondered if this is specific for upstream releases to run sanity.openjdk or sanity.system tests we should not need all those extra parameters to make it more clear. Those extra parameters are specific for grinder, i.e. for developers' personal build. Parameters like BUILD_LIST, TARGET, JDK_IMPL, SDK_RESOURCE, CUSTOMIZED_SDK_URL &CUSTOMIZED_SDK_SOURCE_URL should be enough.

Sure. I've kept them in as I wanted a way to test changes prior merge.

JDK_VERSION is needed too. As otherwise, it would inherit the default value from Grinder or Grinder_Sandbox which is 8 causing a conflict with AUTO_DETECT=true on JDK 11.

I see. The configuration of Grinder and Grinder_Sandbox is slightly different. No default JDK_VERSION value in Grinder and default value 8 in Grinder_sandbox. As AUTO_DETECT is default true JDK_VERSION should not set default value. I just removed default JDK_VERSION in Grinder_Sandbox.

Same as JDK_IMPL, I've removed the default value. So no need to set JDK_IMPL.

@jerboaa if you see the job configuration https://ci.adoptopenjdk.net/view/Test_upstream/job/Test_upstream_openjdk8_hs_sanity.openjdk_x86-64_linux/configure, all those parameter has been set default value as you needed! Thanks @smlambert

sophia-guo · 2019-06-25T21:11:13Z

My understanding is current idea is to trigger all test jobs required as a clone of Grinder_sandbox.

TEST_JOB_NAME = "${env.JOB_NAME}"
...
 build job: TEST_JOB_NAME, parameters: [ ... ]

Going forward would be reuse file Jenkinsfilebase to trigger jobs autogen-ed in Test_upstream view/tab in Jenkins, say some specific name as 'name related with os, arch, version, level'. And then

 parallelJobs[**"specific name"**] = {
                                // Run a systemtest test job with appropriate parameters
                                build job: **specific name**, parameters: [

sophia-guo · 2019-06-25T21:18:56Z

When it's merged the job will trigger by Grinder manually with extra parameters GH_USER GH_TOKEN FORCE_RETEST. Not sure if I got the whole idea?

jerboaa · 2019-06-26T08:09:42Z

When it's merged the job will trigger by Grinder manually with extra parameters GH_USER GH_TOKEN FORCE_RETEST. Not sure if I got the whole idea?

The initial work would be a new job, which would look very similar to Grinder in terms of configuration. Plus extra (optional) parameters like you said. These are:

    parameters {
        string(name:'VERSIONS', defaultValue: '8 11', description: "Space separated list of versions to check")
        string(name:'GH_USERNAME', description: "Github API username to authenticate as")
        string(name:'GH_TOKEN', description: "Github API OAuth personalized access token to be used for authentication")
        booleanParam(name:'INCLUDE_EA', defaultValue: true, description: "Whether or not to include EA builds")
        booleanParam(name:'FORCE_RETEST', defaultValue: false, description: "Whether or not to force a re-test irrespective previous runs")
    }

jerboaa · 2019-06-26T08:22:56Z

My understanding is current idea is to trigger all test jobs required as a clone of Grinder_sandbox.
TEST_JOB_NAME = "${env.JOB_NAME}"
...
 build job: TEST_JOB_NAME, parameters: [ ... ]
Going forward would be reuse file Jenkinsfilebase to trigger jobs autogen-ed in Test_upstream view/tab in Jenkins, say some specific name as 'name related with os, arch, version, level'. And then
 parallelJobs[**"specific name"**] = {

That's already done, AFAIK.

                                // Run a systemtest test job with appropriate parameters
                                build job: **specific name**, parameters: [

Wouldn't I need to know the job name ahead of time, though? I could probably do something like TestAutoGen job which is using buildenv/jenkins/testJobTemplate. Perhaps use a modified version of testJobTemplate which accommodates the customized JDK use-case. I'm confused, sorry.

Having said all that, perhaps we should get started somewhere. I propose this plan of action:

Get this merged with some review feedback of the jenkins file itself incorporated
Set up a new job in the Test_Upstream view, which will do the testing.
Then you can have a look at how it works and suggest a way forward to get it massaged into something we can all agree on. Right now, I'm a bit lost what exactly you want me to do next.

In step 2) a specific name will be used which makes it apparent that this is an upstream JDK test job. I'm guessing that's the biggest concern? Thoughts?

jerboaa · 2019-06-26T08:25:22Z

buildenv/jenkins/upstream_openjdk_tests

+                            def jdk_version = version
+                            def jenkins_file = r.getJenkinsFile()
+                            def jdk_jre_url = "$r.JDK_URL $r.JRE_URL"
+                            parallelJobs["Test_openjdk${version}_hs_sanity.openjdk_${r.ARCH}_${r.OS}"] = {


@sophia-guo What parallel branch name do you suggest instead of this? Test_upstream_openjdk... perhaps?

Ok, thanks!

smlambert · 2019-06-26T12:40:09Z

As I understand this there are fundamentally 2 separate pieces of functionality:

the piece that monitors for changes and decides if tests should be triggered (which uses the 3 parameters GH_USER, GH_TOKEN and FORCE_RETEST)
the test pipelines themselves (which can be handled by directly calling Jenkinsfilebase as is)

Ultimately for part 1, it would be great to be able to use existing Jenkins plugin functionality and do a post-commit trigger, however @jerboaa wants only to monitor changes to certain files in the GH repo he is monitoring. Part 1 could eventually also be accomplished by monitoring changes via openjdk-api v3 when that work completes.

Part 2 should just call the test pipelines directly with as @sophia-guo suggests, only the needed parameters being passed.

My mention of using the testJobTemplate was merely to make it easier to do the initial creation of the jobs required for this initial pass, but as there are only 10 required, it can also be easily done by cloning existing jobs. I have created one in the Test_upstream view, but not yet changed the defaults...

sophia-guo · 2019-06-26T12:51:11Z

buildenv/jenkins/upstream_openjdk_tests

+                            def jdk_jre_url = "$r.JDK_URL $r.JRE_URL"
+                            parallelJobs["Test_openjdk${version}_hs_sanity.openjdk_${r.ARCH}_${r.OS}"] = {
+                                // Run a _sanity.openjdk test job with appropriate parameters
+                                build job: TEST_JOB_NAME, parameters: [


This 'TEST_JOB_NAME' is exactly same as the branch name.

My understanding was that the key to the parallelJobs map is the "branch name". At least jenkins calls it that when it runs. "TEST_JOB_NAME" refers to the jenkins job name, no? Like "Grinder". Scheduling jobs that way assumes existing job config (with matching name referenced via TEST_JOB_NAME), does it not?

Branch name is not necessarily same as the test job name. It's nice to include the hint about the test jobs. For this case as long as the branch name including the information as '${version}, hotspot, sanity, arch, os ' it would be enough. So we know which test job will run.

Yes, TEST_JOB_NAME refers to the jenkins job name, which is the test jobs we have generated. for example: job https://ci.adoptopenjdk.net/view/Test_upstream/job/Test_upstream_openjdk8_hs_sanity.openjdk_x86-64_linux/

For now we only have one job generated, we can either generate others by duplicate copy or by autogen. If we don't want the job failed by non-generated jobs we could also use @Library('openjdk-jenkins-helper@master') by check if (JobHelper.jobIsRunnable("${TEST_JOB_NAME}")) { build job }

Hmm, I see you prefer to create a new job and all parallel test jobs are also clone of this new job?
Say the job number is 1, and then you have 6 test jobs. Those jobs will be job number 2,3,...,7.

We have tried this strategy before for parallel jobs and think it's not clear/straightforward for triage as all test jobs seem similar. You can only go to console and know jobs targets.

See this picture it's not clear to know which job is running against 11|8.

My original thought is to create a new job and this new job will trigger all jobs you want. Similarly to https://ci.adoptopenjdk.net/view/Test_external_weekly/job/openjdk_externaltest_extended_weekly/configure

But I'm ok with both.

@sophia-guo It makes sense what you are saying. I'll implement it so that the generated jobs have a good name as you suggest for discoverability. What I was trying to point out is that there seems to be a "branch name" and "job name" concept in jenkins (they're not necessarily the same). I understand the branch and job names should match for a revised version of this patch. It doesn't currently, which I'll fix. Thanks and sorry for the confusion.

Also use proper job name for building

smlambert · 2019-06-26T17:11:10Z

I've created 8/10 jobs needed for this, as I assumed same as @sophia-guo , that we would proceed with pre-created jobs (in that way you have job history and if we want to reference jobs from TRSS they are easily findable...), but its fine to proceed as @jerboaa has now too.

jerboaa · 2019-06-26T17:19:54Z

@smlambert The latest version now generates jobs in the pre-created set in the Test_Upstream view. There are missing 2 jobs, Test_upstream_openjdk11_hs_sanity.system_aarch64_linux and Test_upstream_openjdk11_hs_sanity.openjdk_aarch64_linux. If somebody could create them for me I can run a full test cycle with the latest update. Thanks. Copying from Test_upstream_openjdk11_hs_sanity.openjdk_x86-64_linux should work well.

Open to suggestions for the name of the new "Jenkinsfile" upstream_openjdk_tests currently. It still hooks onto various platform specific jenkinsfiles, since I'd otherwise need to duplicate the logic in there. Thoughts?

smlambert · 2019-06-26T17:34:49Z

All 10 jobs are there now. I am travelling, but will check in later today.

Let's leave the name of the file as is for now and get this merged. You can always update things later if you want. Approving now, but leave any other details to you and Sophia. Thanks!

sophia-guo

LGTM. Thanks @jerboaa !

jerboaa · 2019-06-26T18:20:04Z

Thanks all!

jerboaa added 10 commits June 17, 2019 15:06

Add jenkins file for upstream build testing

6ca9a98

This checks for updated releases first. Second, if there are any, trigger test build jobs for OpenJDK sanity and system tests for each platform/arch.

Move parallel execution outside of node()

d28a5b6

Fix types for comparing publish dates

50ac770

Print published date of release if no-retest needed

cdc1872

Do better API error checking

02d1827

Use test node for retrieving info

fab18bf

Add authentication to GH API calls

e160866

Test fixing workspace relative file

a136c02

Use better naming scheme for parallel tests

ba74ffe

Example parallel job name: Test_openjdk8_hs_sanity.openjdk_x64_linux

Improve test jobs' scheduling

d385691

- Set JDK_VERSION so as to not use the default (8) causing conflicts with AUTO_DETECT=true for JDK 11 test jobs - Schedule the API work on Linux x86 test nodes only as these are plenty.

karianna added the enhancement label Jun 25, 2019

karianna requested a review from smlambert June 25, 2019 12:34

smlambert requested a review from sophia-guo June 25, 2019 14:57

sophia-guo reviewed Jun 25, 2019

View reviewed changes

jerboaa commented Jun 26, 2019

View reviewed changes

sophia-guo reviewed Jun 26, 2019

View reviewed changes

jerboaa added 3 commits June 26, 2019 17:53

Use separate upstream test job names for test execution

df383cf

Use mapped architecture for test job names

417571b

Also use proper job name for building

Generate sanity.system and sanity.openjdk jobs in a loop

391cfc0

smlambert approved these changes Jun 26, 2019

View reviewed changes

sophia-guo approved these changes Jun 26, 2019

View reviewed changes

sophia-guo merged commit b360d81 into adoptium:master Jun 26, 2019

karianna added this to the June 2019 milestone Jun 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add jenkins test pipeline for OpenJDK Project Builds #1192

Add jenkins test pipeline for OpenJDK Project Builds #1192

jerboaa commented Jun 25, 2019

smlambert commented Jun 25, 2019

jerboaa commented Jun 25, 2019

smlambert commented Jun 25, 2019

jerboaa commented Jun 25, 2019

sophia-guo Jun 25, 2019

jerboaa Jun 26, 2019

sophia-guo Jun 26, 2019

sophia-guo Jun 26, 2019 •

edited

Loading

sophia-guo Jun 26, 2019

sophia-guo commented Jun 25, 2019

sophia-guo commented Jun 25, 2019

jerboaa commented Jun 26, 2019

jerboaa commented Jun 26, 2019

jerboaa Jun 26, 2019

jerboaa Jun 26, 2019

smlambert commented Jun 26, 2019

sophia-guo Jun 26, 2019

jerboaa Jun 26, 2019

sophia-guo Jun 26, 2019

sophia-guo Jun 26, 2019

sophia-guo Jun 26, 2019

jerboaa Jun 26, 2019

smlambert commented Jun 26, 2019

jerboaa commented Jun 26, 2019

smlambert commented Jun 26, 2019

sophia-guo left a comment

jerboaa commented Jun 26, 2019

Add jenkins test pipeline for OpenJDK Project Builds #1192

Add jenkins test pipeline for OpenJDK Project Builds #1192

Conversation

jerboaa commented Jun 25, 2019

smlambert commented Jun 25, 2019

jerboaa commented Jun 25, 2019

smlambert commented Jun 25, 2019

jerboaa commented Jun 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sophia-guo Jun 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sophia-guo commented Jun 25, 2019

sophia-guo commented Jun 25, 2019

jerboaa commented Jun 26, 2019

jerboaa commented Jun 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smlambert commented Jun 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smlambert commented Jun 26, 2019

jerboaa commented Jun 26, 2019

smlambert commented Jun 26, 2019

sophia-guo left a comment

Choose a reason for hiding this comment

jerboaa commented Jun 26, 2019

sophia-guo Jun 26, 2019 •

edited

Loading