diff --git a/README.md b/README.md index 7a563f708..f78528a5b 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,8 @@ Eclipse Adoptium makes use of these scripts to build binaries on the build farm ## Repository contents This repository contains several useful scripts in order to build OpenJDK -personally or at build farm scale. +personally or at build farm scale via jenkins. For the Temurin project at +Adoptium, this is done with the jenkins instance at https://ci.adotpium.net 1. The `docs` folder contains images and utility scripts to produce up to date documentation. @@ -14,6 +15,159 @@ documentation. (e.g. build | test | checksum | release). 3. The `tools` folder contains `pipelines/` analysis scripts that deliever success/failure trends and build scripts for code-tool dependancies for the build and test process (e.g. asmtools | jcov | jtharness | jtreg | sigtest). +## Overview of pipeline types + +The starting point on the jenkins instance from the perspective of the +overall build pipelines is the +[build-scripts folder](https://ci.adoptium.net/job/build-scripts/). This +contains the top level pipelines which are used to run the different types +of build. In the names in this document `XX` is the JDK version number e.g. +8, 17, 21 and so on. There is one of these for each JDK version which we +support. + +### openjdkXX-pipeline + +These were historically used for regular builds of each of our release +platforms using the current state of the master branch of the codebase - +which is it's default behaviour - but is now run each time there is a new +tag in the upstream openjdk codebase. These are triggered by the +`betaTrigger_XXea` jobs in the +[build-scripts/utils](https://ci.adoptium.net/job/build-scripts/job/utils/) +folder. + +The betaTrigger_XXea jobs use +[trigger_beta_build.groovy](https://github.com/adoptium/ci-jenkins-pipelines/blob/master/pipelines/build/common/trigger_beta_build.groovy) +to determine when to run a build. This contains a trap for the expected GA +release times to prevent triggering so that machine time is not used up +while we are performing release build and test cycles. + +Once complete, the openjdkXX-pipelines which will, by default, invoke the +separate +([refactor_openjdk_release_tool](https://ci.adoptium.net/job/build-scripts/job/release/job/refactor_openjdk_release_tool/)) +job which will publish them as an `ea-beta`-suffixed release in github under e.g. +[temurin-21-binaries](https://github.com/adoptium/temurin21-binaries/releases?q=ea-beta&expanded=true}). + +### release-openjdkXX-pipeline + +These are not publicly visible but are used to build the fully tested +production binaries on a quarterly basis. Similar to the openjdkXX-pipeline +jobs these are automatically triggered by the releaseTrigger_jdkXX jobs in +[build-scripts/utils](https://ci.adoptium.net/job/build-scripts/job/utils/) +every time a new `-ga` suffixed tag is detected. (TODO: Note on JDK8/arm32 +and potentially Win32) + +releaseTrigger_jdkXX runs once an hour between the 10th and 25th of release +months (Jan, Mar, Apr, Jul, Sep, Oct) and check for a new `-ga` tag. It runs +[triggerReleasePipeline.sh](https://github.com/adoptium/mirror-scripts/blob/master/triggerReleasePipeline.sh) +(from the [mirror-scripts](https://github.com/adoptium/mirror-scripts/) +repository). That script has a loop that checks 5 times with a ten minute +gap between them so that the overall trigger is checked every ten minutes +during the days when it is active based on checking for the "expected" tag +from [releasePlan.cfg](https://github.com/adoptium/mirror-scripts/blob/master/releasePlan.cfg) +using the readExpectedGATag function in +[common.sh](https://github.com/adoptium/mirror-scripts/blob/master/common.sh) + +### evaluation-openjdkXX-pipeline + +These are similar to the openjdkXX-pipeline jobs, and are triggered from the +same betaTrigger_XXea jobs. The evaluation pipelines are for platforms +which the Adoptium team are looking to potentially release at some point, +but they are not yet reliable or sufficiently tested. +which are not in the release. + +### weekly-openjdkXX-pipeline / weekly-evaluation-openjdkXX-pipeline + +These are no longer used. These were triggered over the weekend with an extended set +of tests, but since the regular openjdkXX-pipeline jobs are now running +approximately once a week (the usual cadence of new tags appearing in the +upstream codebases) we are running the full AQA suite in those pipelines. +These were triggered by timer and then invoked the openjdkXX-pipeline jobs +with the appropriate parameters. + +### trestle-openjdkXX-pipeline + +Trestle is the name of the experimental project to allow upstream openjdk +committers to run pipelines on our infrastructure in order to test code +changes in openjdk on the full set of platforms which Temurin supports. They +are triggered on demand from a subset of authorized users. + +### PR tester + +In addition to the main pipelines we have "PR tester" jobs that are run on +PRs to the pipelines repository in order to ensure they do not have any +unintended side effects before they are merged. These are triggered when +comments from authorized users are added into the PR comments. In that +folder in jenkins there are separate versions of all of the +openjdkXX-pipelines that can be used to run against PRs and will not +"pollute" the history of the main pipelines. + +## Subjobs of the top level pipelines + +Each of the top level pipelines described above invoke lower level jobs to +run the platform-specific builds. The jenkins folders containing these +scripts for each of the above top level pipelines are as follows: + +Top level pipeline | Platform-specific pipeline folder (TODO: Name these!) +---|--- +openjdkXX-pipeline | [jobs/jdkXX](https://ci.adoptium.net/job/build-scripts/job/jobs/) +evaluation-openjdkXX-pipeline | [jobs/evaluation/jdkXX](https://ci.adoptium.net/job/build-scripts/job/jobs/job/evaluation/) [†] +weekly-openjdkXX-pipeline | jobs/jdkXX (Shared with openjdkXX-pipeline) +release-openjdkXX-pipeline | [jobs/release/jobs/jdkXX](https://ci.adoptium.net/job/build-scripts/job/jobs/job/release/job/jobs/) (Restricted access) +PR testers | build-test/jobs/jdkXX + +[†] - The release jobs here are restricted access. The release folder here +should also not be confused with the build-scripts/release folder which +contains jobs used for the final publishing of the builds (early access or +GA) to github + +*Note: jdkXX is generally related to the name of the upstream codebase, which +will often have a `u` suffix. At the moment we have a separate set of jobs +for non-u and u versions when the upstream codebase changes. TODO: Add note +about the new branch process for jdk23+* + +Inside the jdkXX folders there are pipelines which perform a build of one +variant (e.g. Temurin) for on JDK version on one platform, for example +[jdk21u-linux-aarch64-temurin](https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-linux-aarch64-temurin/) +which are reponsible for running the build and initiating the tests and +other jobs against the completed build if successful. A "Smoke Test" job +such as +[jdk21u-linux-aarch64-temurin-SmokeTests](https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-linux-aarch64-temurin_SmokeTests/) +(TODO: Links to info about what that contains) which is initiated after the +build perfoms some basic tests against the build artefacts and acts as a +gate to kicking off the subsequent steps. Once complete, the +openjdkXX-pipelines which run the early access builds will generally invoke +the jobs to publish them as a release in github (e.g. +[temurin-21-binaries](https://github.com/adoptium/temurin21-binaries/releases?q=ea-beta&expanded=true}). + +## Job generation + +As you can see from the above sections, there are a lot of separate jobs in +jenkins which are used during the build process. These are not created +manually, but are autogenerated using the generator pipelines. TODO: More +information on this is at ... + +The top level +[build-pipeline-generator](https://ci.adoptium.net/job/build-scripts/job/utils/job/build-pipeline-generator/) +job uses +[build_pipeline_generator.groovy](https://github.com/adoptium/ci-jenkins-pipelines/blob/master/pipelines/build/regeneration/build_pipeline_generator.groovy) +to generate the pipelines. It will generate the top level +openjdkXX-pipeline jobs. Similarly there are pipeline_jobs_generator_jdkXX +jobs which use +[build_job_generator.groovy](pipelines/build/regeneration/build_job_generator.groovy) +to generate the subjobs for each platform/variant combination. Both of +these pipelines are triggered on a change (PR merge) to the +ci-jenkins-pipelines repository + +Similarly there is an evaluation-pipeline-generator and +evaluation-pipeline_jobs_generator_jdkXX for generating the evaluation jobs, +a trestle-pipeline-generator for those jobs, plus release-pipeline-generator +andand release_pipeline_jobs_generator_jdkXX for release jobs (the release +generators are not triggered automatically but are re-run manually at +certain points during each release cycle + +For more details on the processm see the +[regeneration documentation](https://github.com/adoptium/ci-jenkins-pipelines/blob/master/pipelines/build/regeneration/README.md) + ## Configuration Files The [pipelines/jobs/configurations](pipelines/jobs/configurations) directory contains two categories of configuration files that our jenkins pipelines use (Nicknamed [#Build Configs](#build) and [#Nightly Configs](#nightly) for short).