Jenkins pipeline with parallel cigroups #45285

brianseeders · 2019-09-10T16:08:19Z

Implement Jenkins Pipeline (i.e. Jenkinsfile) and run oss/x-pack ciGroups on in parallel on two machines, bringing the total number of jenkins agents per build down to four.

There are many changes that are temporary (such as anywhere you see $IS_PIPELINE_JOB), while we are in a transition period. Once all jobs are converted over to pipeline jobs, those can be cleaned up.

Also see #44925 for a discussion around visibility into logs/build status/etc, which is the biggest downside to this work.

elasticmachine · 2019-09-10T16:08:21Z

Pinging @elastic/kibana-operations

brianseeders · 2019-09-10T17:47:41Z

pipeline test

brianseeders · 2019-09-10T19:23:49Z

pipeline test

spalger · 2019-09-10T20:44:51Z

Failure handling looks good 👍

brianseeders · 2019-09-10T21:04:46Z

pipeline test

spalger

Ultimately I think we should be good to go here, but I do have some nitpicks that we can move to follow up issues if you prefer.

Jenkinsfile

packages/kbn-test/src/mocha/__tests__/junit_report_generation.js

packages/kbn-test/src/mocha/junit_report_generation.js

spalger · 2019-09-10T21:08:08Z

src/dev/ci_setup/setup.sh

-
-kbnBranch="$(jq -r .branch "$KIBANA_DIR/package.json")"
-export KIBANA_PKG_BRANCH="$kbnBranch"
+source src/dev/ci_setup/setup_env.sh true


What does true do here?

I split out setup_env, because I need to set up all of the environment stuff every time sh is called in the Jenkinsfile.

I was originally having issues with the parallel workers each trying to re-install node/yarn, so I introduced an argument to skip node/yarn installing, and just set up the env. But now that I look at the script again, I'm not sure why that was happening in the first place. Will take a second look

src/dev/ci_setup/setup.sh

src/dev/jest/integration_tests/junit_reporter.test.js

spalger · 2019-09-10T21:23:58Z

test/plugin_functional/test_suites/core_plugins/applications.js

@@ -32,6 +32,12 @@ export default function ({ getService, getPageObjects }) {
  const loadingScreenShown = () =>
    testSubjects.existOrFail('kbnLoadingMessage');

+  const getKibanaUrl = path => {
+    const host = process.env.TEST_KIBANA_HOST || 'localhost';


I know it's silly, but generally prefer to avoid creating properly formatted strings when there's a perfectly good format function available:

return url.format({ protocol: 'http:', hostname: process.env.TEST_KIBANA_HOST || 'localhost', port: process.env.TEST_KIBANA_PORT || '5620', path })

A lot of these formats have tricky little edge cases that pop up over time.

test/scripts/jenkins_build_kibana.sh

brianseeders · 2019-09-10T21:41:56Z

pipeline test

brianseeders · 2019-09-11T01:57:09Z

pipeline test

tylersmalley · 2019-09-11T02:05:02Z

Jenkinsfile

+  sendKibanaMail()
+}
+
+def sendInfraMail() {


Wonder if infra still wants these emails.

packages/kbn-test/src/mocha/junit_report_generation.js

tylersmalley · 2019-09-11T02:15:01Z

...ck/test/api_integration/apis/management/cross_cluster_replication/remote_clusters.helpers.js

@@ -16,13 +16,16 @@ export const registerHelpers = (supertest) => {
  const addCluster = (name = REMOTE_CLUSTER_NAME) => {
    remoteClustersCreated.push(name);

+    // TODO how could this get pulled out of the ES config instead?


Probably the only way would be to grab it from stdout during startup since it increments if the port isn't available.

brianseeders · 2019-09-11T03:17:11Z

pipeline test

brianseeders · 2019-09-11T16:09:14Z

pipeline test

elasticmachine · 2019-09-11T17:27:15Z

💚 Build Succeeded

pipeline test: continuous-integration/kibana-ci/pull-request # TODO fix after testing

* Pipeline * WIP some work for parallelization with ciGroups * Fix xpack kibana install dir, and add some debugging * Attempt to quick fix a few tests * Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)"" This reverts commit 078ac28. * Recombine test groups, and try runbld again * Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline * Fix typo in shell script * Remove some debug code * Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT * Fix test that uses hard-coded es transport port and add it back in to parallel groups * Disable checks reporter again for now * Set env var for TEST_ES_TRANSPORT_PORT in pipeline * Update Jenkinsfile for shorter testrunner labels * Fix another hard-coded transport port * Fix a new test with hard-coded URLs * Jenkinsfile cleanup and fix one of the groups * Fix double slash * Testing vault credentials on jenkins server * Add a non-existent credential * Revert "Add a non-existent credential" This reverts commit 0dc234c. * Try github-checks-reporter again * github-checks-reporter should only run for elastic/kibana, forks won't work * Clean up some debug code * Changing names around to try to make BlueOcean UI a little better * Add more stages * Make some changes to stage structure to mirror a nested example from CloudBees * Handle TODOs, and some cleanup in Jenkinsfile * Pass GIT_BRANCH when started without GHPRB, fix branch check * Fix mailer problem and add code that ensures all tests are in cigroups back in * Test adding worker/job name to junit report paths * Remove some duplication from ci_setup scripts * Fix unit test that uses junit path * Don't reinstall node every time setup_env is run * Fix yarn install logic * Fix another unit test that uses junit output dir * Download latest ES snapshot after kibana builds * Make sure junit reports are always processed * Add two failing tests for testing purposes * Add support to Jenkinsfile for kibana build e-mails * Remove some debug code for email sending * Change JOB env handling in junit paths and move it to a sub-directory * Revert "Add two failing tests for testing purposes" This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c. * Fix junit report path in test * Don't send kibana emails on build abort * Address PR feedback, formatting and use built-in url formatting library * Fix path formatting for functional test * Add email sending back in to Jenkinsfile * Fix another unit test with path problem (cherry picked from commit 27d23c4) # Conflicts: # src/dev/ci_setup/setup.sh # src/dev/failed_tests/cli.js

* Pipeline * WIP some work for parallelization with ciGroups * Fix xpack kibana install dir, and add some debugging * Attempt to quick fix a few tests * Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)"" This reverts commit 078ac28. * Recombine test groups, and try runbld again * Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline * Fix typo in shell script * Remove some debug code * Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT * Fix test that uses hard-coded es transport port and add it back in to parallel groups * Disable checks reporter again for now * Set env var for TEST_ES_TRANSPORT_PORT in pipeline * Update Jenkinsfile for shorter testrunner labels * Fix another hard-coded transport port * Fix a new test with hard-coded URLs * Jenkinsfile cleanup and fix one of the groups * Fix double slash * Testing vault credentials on jenkins server * Add a non-existent credential * Revert "Add a non-existent credential" This reverts commit 0dc234c. * Try github-checks-reporter again * github-checks-reporter should only run for elastic/kibana, forks won't work * Clean up some debug code * Changing names around to try to make BlueOcean UI a little better * Add more stages * Make some changes to stage structure to mirror a nested example from CloudBees * Handle TODOs, and some cleanup in Jenkinsfile * Pass GIT_BRANCH when started without GHPRB, fix branch check * Fix mailer problem and add code that ensures all tests are in cigroups back in * Test adding worker/job name to junit report paths * Remove some duplication from ci_setup scripts * Fix unit test that uses junit path * Don't reinstall node every time setup_env is run * Fix yarn install logic * Fix another unit test that uses junit output dir * Download latest ES snapshot after kibana builds * Make sure junit reports are always processed * Add two failing tests for testing purposes * Add support to Jenkinsfile for kibana build e-mails * Remove some debug code for email sending * Change JOB env handling in junit paths and move it to a sub-directory * Revert "Add two failing tests for testing purposes" This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c. * Fix junit report path in test * Don't send kibana emails on build abort * Address PR feedback, formatting and use built-in url formatting library * Fix path formatting for functional test * Add email sending back in to Jenkinsfile * Fix another unit test with path problem (cherry picked from commit 27d23c4) # Conflicts: # src/dev/ci_setup/setup.sh # src/dev/failed_tests/cli.js # test/plugin_functional/test_suites/core_plugins/applications.js # test/scripts/jenkins_xpack_ci_group.sh

* Pipeline * WIP some work for parallelization with ciGroups * Fix xpack kibana install dir, and add some debugging * Attempt to quick fix a few tests * Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)"" This reverts commit 078ac28. * Recombine test groups, and try runbld again * Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline * Fix typo in shell script * Remove some debug code * Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT * Fix test that uses hard-coded es transport port and add it back in to parallel groups * Disable checks reporter again for now * Set env var for TEST_ES_TRANSPORT_PORT in pipeline * Update Jenkinsfile for shorter testrunner labels * Fix another hard-coded transport port * Fix a new test with hard-coded URLs * Jenkinsfile cleanup and fix one of the groups * Fix double slash * Testing vault credentials on jenkins server * Add a non-existent credential * Revert "Add a non-existent credential" This reverts commit 0dc234c. * Try github-checks-reporter again * github-checks-reporter should only run for elastic/kibana, forks won't work * Clean up some debug code * Changing names around to try to make BlueOcean UI a little better * Add more stages * Make some changes to stage structure to mirror a nested example from CloudBees * Handle TODOs, and some cleanup in Jenkinsfile * Pass GIT_BRANCH when started without GHPRB, fix branch check * Fix mailer problem and add code that ensures all tests are in cigroups back in * Test adding worker/job name to junit report paths * Remove some duplication from ci_setup scripts * Fix unit test that uses junit path * Don't reinstall node every time setup_env is run * Fix yarn install logic * Fix another unit test that uses junit output dir * Download latest ES snapshot after kibana builds * Make sure junit reports are always processed * Add two failing tests for testing purposes * Add support to Jenkinsfile for kibana build e-mails * Remove some debug code for email sending * Change JOB env handling in junit paths and move it to a sub-directory * Revert "Add two failing tests for testing purposes" This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c. * Fix junit report path in test * Don't send kibana emails on build abort * Address PR feedback, formatting and use built-in url formatting library * Fix path formatting for functional test * Add email sending back in to Jenkinsfile * Fix another unit test with path problem (cherry picked from commit 27d23c4)

* Pipeline * WIP some work for parallelization with ciGroups * Fix xpack kibana install dir, and add some debugging * Attempt to quick fix a few tests * Revert "Revert "Revert "[ci] compress jobs for CI stability" (elastic#44584)"" This reverts commit 078ac28. * Recombine test groups, and try runbld again * Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline * Fix typo in shell script * Remove some debug code * Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT * Fix test that uses hard-coded es transport port and add it back in to parallel groups * Disable checks reporter again for now * Set env var for TEST_ES_TRANSPORT_PORT in pipeline * Update Jenkinsfile for shorter testrunner labels * Fix another hard-coded transport port * Fix a new test with hard-coded URLs * Jenkinsfile cleanup and fix one of the groups * Fix double slash * Testing vault credentials on jenkins server * Add a non-existent credential * Revert "Add a non-existent credential" This reverts commit 0dc234c. * Try github-checks-reporter again * github-checks-reporter should only run for elastic/kibana, forks won't work * Clean up some debug code * Changing names around to try to make BlueOcean UI a little better * Add more stages * Make some changes to stage structure to mirror a nested example from CloudBees * Handle TODOs, and some cleanup in Jenkinsfile * Pass GIT_BRANCH when started without GHPRB, fix branch check * Fix mailer problem and add code that ensures all tests are in cigroups back in * Test adding worker/job name to junit report paths * Remove some duplication from ci_setup scripts * Fix unit test that uses junit path * Don't reinstall node every time setup_env is run * Fix yarn install logic * Fix another unit test that uses junit output dir * Download latest ES snapshot after kibana builds * Make sure junit reports are always processed * Add two failing tests for testing purposes * Add support to Jenkinsfile for kibana build e-mails * Remove some debug code for email sending * Change JOB env handling in junit paths and move it to a sub-directory * Revert "Add two failing tests for testing purposes" This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c. * Fix junit report path in test * Don't send kibana emails on build abort * Address PR feedback, formatting and use built-in url formatting library * Fix path formatting for functional test * Add email sending back in to Jenkinsfile * Fix another unit test with path problem (cherry picked from commit 27d23c4)

* Jenkins pipeline with parallel cigroups (#45285) * Pipeline * WIP some work for parallelization with ciGroups * Fix xpack kibana install dir, and add some debugging * Attempt to quick fix a few tests * Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)"" This reverts commit 078ac28. * Recombine test groups, and try runbld again * Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline * Fix typo in shell script * Remove some debug code * Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT * Fix test that uses hard-coded es transport port and add it back in to parallel groups * Disable checks reporter again for now * Set env var for TEST_ES_TRANSPORT_PORT in pipeline * Update Jenkinsfile for shorter testrunner labels * Fix another hard-coded transport port * Fix a new test with hard-coded URLs * Jenkinsfile cleanup and fix one of the groups * Fix double slash * Testing vault credentials on jenkins server * Add a non-existent credential * Revert "Add a non-existent credential" This reverts commit 0dc234c. * Try github-checks-reporter again * github-checks-reporter should only run for elastic/kibana, forks won't work * Clean up some debug code * Changing names around to try to make BlueOcean UI a little better * Add more stages * Make some changes to stage structure to mirror a nested example from CloudBees * Handle TODOs, and some cleanup in Jenkinsfile * Pass GIT_BRANCH when started without GHPRB, fix branch check * Fix mailer problem and add code that ensures all tests are in cigroups back in * Test adding worker/job name to junit report paths * Remove some duplication from ci_setup scripts * Fix unit test that uses junit path * Don't reinstall node every time setup_env is run * Fix yarn install logic * Fix another unit test that uses junit output dir * Download latest ES snapshot after kibana builds * Make sure junit reports are always processed * Add two failing tests for testing purposes * Add support to Jenkinsfile for kibana build e-mails * Remove some debug code for email sending * Change JOB env handling in junit paths and move it to a sub-directory * Revert "Add two failing tests for testing purposes" This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c. * Fix junit report path in test * Don't send kibana emails on build abort * Address PR feedback, formatting and use built-in url formatting library * Fix path formatting for functional test * Add email sending back in to Jenkinsfile * Fix another unit test with path problem (cherry picked from commit 27d23c4) * remove reference to extract_bootstrap_cache.sh * remove extra jobs from Jenkinsfile * increment leadfoot server port for parallel workers * add worker index to port number, rather than concatenating with it * well, for some reason the PR job wasn't triggered for 351eaaf... * huh, PR job wasn't triggered for 1b99c69 either...

brianseeders added Team:Operations Team label for Operations Team Feature:CI Continuous integration labels Sep 10, 2019

brianseeders requested a review from a team September 10, 2019 16:08

brianseeders self-assigned this Sep 10, 2019

This comment has been minimized.

Sign in to view

spalger approved these changes Sep 10, 2019

View reviewed changes

This comment has been minimized.

Sign in to view

tylersmalley reviewed Sep 11, 2019

View reviewed changes

Jenkinsfile

sendKibanaMail()

}

def sendInfraMail() {

Copy link

Contributor

tylersmalley Sep 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if infra still wants these emails.

tylersmalley reviewed Sep 11, 2019

View reviewed changes

packages/kbn-test/src/mocha/junit_report_generation.js Show resolved Hide resolved

tylersmalley reviewed Sep 11, 2019

View reviewed changes

This comment has been minimized.

Sign in to view

brianseeders force-pushed the jenkins-pipeline-cigroups-parallel branch from 1ebf6c2 to ae6f73d Compare September 11, 2019 03:16

This comment has been minimized.

Sign in to view

brianseeders force-pushed the jenkins-pipeline-cigroups-parallel branch from ae6f73d to 4ffe6fd Compare September 11, 2019 13:50

brianseeders added 6 commits September 11, 2019 12:08

Fix junit report path in test

0a6acc8

Don't send kibana emails on build abort

2a6a838

Address PR feedback, formatting and use built-in url formatting library

e0944a7

Fix path formatting for functional test

3523893

Add email sending back in to Jenkinsfile

496ba75

Fix another unit test with path problem

3253cee

brianseeders force-pushed the jenkins-pipeline-cigroups-parallel branch from 3fef546 to 3253cee Compare September 11, 2019 16:08

spalger added release_note:skip Skip the PR/issue when compiling release notes v6.8.4 v7.3.2 v7.4.0 v7.5.0 v8.0.0 labels Sep 11, 2019

This comment has been minimized.

Sign in to view

spalger added the test-pipelines label Sep 11, 2019

brianseeders mentioned this pull request Sep 11, 2019

[CI] Re-examine node/yarn install logic for CI #45416

Open

spalger mentioned this pull request Sep 11, 2019

Final test of #45285 as a matrix job #45418

Closed

spalger merged commit 27d23c4 into elastic:master Sep 11, 2019

spalger mentioned this pull request Sep 13, 2019

[6.8] Jenkins pipeline with parallel cigroups (#45285) #45599

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jenkins pipeline with parallel cigroups #45285

Jenkins pipeline with parallel cigroups #45285

brianseeders commented Sep 10, 2019 •

edited by spalger

Loading

elasticmachine commented Sep 10, 2019

This comment has been minimized.

brianseeders commented Sep 10, 2019

This comment has been minimized.

brianseeders commented Sep 10, 2019

This comment has been minimized.

This comment has been minimized.

spalger commented Sep 10, 2019

brianseeders commented Sep 10, 2019

This comment has been minimized.

spalger left a comment

spalger Sep 10, 2019

brianseeders Sep 10, 2019

spalger Sep 10, 2019

brianseeders commented Sep 10, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

brianseeders commented Sep 11, 2019

tylersmalley Sep 11, 2019

tylersmalley Sep 11, 2019

This comment has been minimized.

brianseeders commented Sep 11, 2019

This comment has been minimized.

This comment has been minimized.

brianseeders commented Sep 11, 2019

This comment has been minimized.

elasticmachine commented Sep 11, 2019

Jenkins pipeline with parallel cigroups #45285

Jenkins pipeline with parallel cigroups #45285

Conversation

brianseeders commented Sep 10, 2019 • edited by spalger Loading

elasticmachine commented Sep 10, 2019

This comment has been minimized.

brianseeders commented Sep 10, 2019

This comment has been minimized.

brianseeders commented Sep 10, 2019

This comment has been minimized.

This comment has been minimized.

spalger commented Sep 10, 2019

brianseeders commented Sep 10, 2019

This comment has been minimized.

spalger left a comment

Choose a reason for hiding this comment

spalger Sep 10, 2019

Choose a reason for hiding this comment

brianseeders Sep 10, 2019

Choose a reason for hiding this comment

spalger Sep 10, 2019

Choose a reason for hiding this comment

brianseeders commented Sep 10, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

brianseeders commented Sep 11, 2019

tylersmalley Sep 11, 2019

Choose a reason for hiding this comment

tylersmalley Sep 11, 2019

Choose a reason for hiding this comment

This comment has been minimized.

brianseeders commented Sep 11, 2019

This comment has been minimized.

This comment has been minimized.

brianseeders commented Sep 11, 2019

This comment has been minimized.

elasticmachine commented Sep 11, 2019

💚 Build Succeeded

brianseeders commented Sep 10, 2019 •

edited by spalger

Loading