[CI] Add E2E for x-pack/elastic-agent (fleet) #24112

v1v · 2021-02-18T17:12:58Z

What does this PR do?

Support running the e2e stage within the main pipeline using the e2e pipeline job in addition to running the script if needed.
Enable the e2e for x-pack/elastic-agent
Support for running the e2e pipeline in order to benefit from the parallelisation

Why is it important?

As requested by the team.

Important

There are e2e test failures ...

Issues

Requires elastic/e2e-testing#793

Follow up

Refactor the calls to the e2e pipeline/jobs with same shared library step.

elasticmachine · 2021-02-18T18:09:20Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 168 min 39 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

v1v · 2021-02-22T09:28:40Z

/test

v1v · 2021-02-22T11:09:27Z

@mdelapenya , I might need some help here, the packaging finished as expected but I see a couple of issues:

The e2e didn't fail even there were errors.
Somehow the e2e is referring to the 7.10.3 branch

[2021-02-22T10:36:27.018Z] time="2021-02-22T10:36:26Z" level=error msg="Could not download the binary for the agent" arch=x86_64 artifact=elastic-agent error="stat /var/lib/jenkins/workspace/Beats_beats_PR-24112/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-7.10.3-SNAPSHOT-linux-x86_64.tar.gz: no such file or directory" extension=tar.gz os=linux version=7.10.3-SNAPSHOT
[2021-02-22T10:36:27.018Z] time="2021-02-22T10:36:26Z" level=fatal msg="Sorry, we could not download the installer" error="stat /var/lib/jenkins/workspace/Beats_beats_PR-24112/src/github.com/elastic/beats/x-pack/elastic-agent/build/distributions/elastic-agent-7.10.3-SNAPSHOT-linux-x86_64.tar.gz: no such file or directory" image=debian installer=tar
[2021-02-22T10:36:27.018Z] Makefile:71: recipe for target 'functional-test' failed
[2021-02-22T10:36:27.018Z] make: *** [functional-test] Error 1

While the target for this PR was the master branch, and artifacts were uploaded correctly:

The e2e logs can be found here and I don't see any references to the 7.10.3 branch :/, the below stacktrace is the header of the above build output:

[2021-02-22T10:14:49.439Z] + .ci/scripts/fleet-test.sh
[2021-02-22T10:14:49.439Z] + STACK_VERSION=8.0.0-SNAPSHOT
[2021-02-22T10:14:49.439Z] + SUITE=fleet
[2021-02-22T10:14:49.439Z] + .ci/scripts/functional-test.sh fleet '' 8.0.0-SNAPSHOT
[2021-02-22T10:14:49.439Z] + SUITE=fleet
[2021-02-22T10:14:49.439Z] + TAGS=
[2021-02-22T10:14:49.439Z] + STACK_VERSION=8.0.0-SNAPSHOT
[2021-02-22T10:14:49.439Z] + METRICBEAT_VERSION=8.0.0-SNAPSHOT
[2021-02-22T10:14:49.439Z] + .ci/scripts/install-test-dependencies.sh fleet
[2021-02-22T10:14:49.440Z] + SUITE=fleet
[2021-02-22T10:14:49.440Z] + '[' -f .ci/scripts/install-fleet-test-dependencies.sh ']'
[2021-02-22T10:14:49.440Z] + echo 'Not installing test dependencies for fleet'
[2021-02-22T10:14:49.440Z] Not installing test dependencies for fleet
[2021-02-22T10:14:49.440Z] + rm -rf outputs
[2021-02-22T10:14:49.440Z] + mkdir -p outputs
[2021-02-22T10:14:49.440Z] ++ pwd
[2021-02-22T10:14:49.440Z] + REPORT=/var/lib/jenkins/workspace/Beats_beats_PR-24112/src/github.com/elastic/e2e-testing/outputs/TEST-fleet
[2021-02-22T10:14:49.440Z] + SUITE=fleet
[2021-02-22T10:14:49.440Z] + TAGS=
[2021-02-22T10:14:49.440Z] + FORMAT=junit:/var/lib/jenkins/workspace/Beats_beats_PR-24112/src/github.com/elastic/e2e-testing/outputs/TEST-fleet.xml
[2021-02-22T10:14:49.440Z] + STACK_VERSION=8.0.0-SNAPSHOT
[2021-02-22T10:14:49.440Z] + METRICBEAT_VERSION=8.0.0-SNAPSHOT
[2021-02-22T10:14:49.440Z] + make --no-print-directory -C e2e functional-test

any hints?

mdelapenya · 2021-02-22T16:09:54Z

As discussed online, it seems the nightly tests (for the upgrade of the elastic-agent) are being executed, which is something we should not do for PRs. On the other hand, there are CI jobs that are already executing these tests on a nightly basis, so I think we are covered for branches.

Our solution is to always call the fleet entrypoint for this feature excluding the @nightly annotation/tag.

v1v · 2021-02-22T16:52:48Z

/test

v1v · 2021-02-22T16:56:41Z

I just mocked the functional-test make goal and it returned the errorlevel as expected

.ci/scripts/fleet-test.sh
+ STACK_VERSION=8.0.0-SNAPSHOT
+ SUITE=fleet
+ TAG='~@nightly'
+ .ci/scripts/functional-test.sh fleet '~@nightly' 8.0.0-SNAPSHOT
+ SUITE=fleet
+ TAGS='~@nightly'
+ STACK_VERSION=8.0.0-SNAPSHOT
+ METRICBEAT_VERSION=8.0.0-SNAPSHOT
+ .ci/scripts/install-test-dependencies.sh fleet
+ SUITE=fleet
+ '[' -f .ci/scripts/install-fleet-test-dependencies.sh ']'
+ echo 'Not installing test dependencies for fleet'
Not installing test dependencies for fleet
+ rm -rf outputs
+ mkdir -p outputs
++ pwd
+ REPORT=/Users/vmartinez/work/src/github.com/elastic/e2e-testing/outputs/TEST-fleet
+ SUITE=fleet
+ TAGS='~@nightly'
+ FORMAT=junit:/Users/vmartinez/work/src/github.com/elastic/e2e-testing/outputs/TEST-fleet.xml
+ STACK_VERSION=8.0.0-SNAPSHOT
+ METRICBEAT_VERSION=8.0.0-SNAPSHOT
+ make --no-print-directory -C e2e functional-test
go get -v github.com/cucumber/godog/cmd/[email protected]
go: found github.com/cucumber/godog/cmd/godog in github.com/cucumber/godog v0.11.0
exit 1
make: *** [functional-test] Error 1
➜  e2e-testing git:(master) ✗ echo $?
2

Maybe there is something with the pipeline 2.0

* upstream/master: [Elastic Agent] Fix docker entrypoint for elastic-agent. (elastic#24155) [PACKAGING] Push docker images with the architecture in the version (elastic#24121) [Agent] Add agent standalone manifests for system module & Pod's log collection (elastic#23938) indicator type url is in upper case (elastic#24152) [Filebeat] Document netflow internal_networks and set default (elastic#24110) [Filebeat] Adding fixes to the TI module (elastic#24133) [Enhancement] Add RotateOnStartup feature flag for file output (elastic#19347) [Ingest Manager] Fix: Successfully installed and enrolled agent running standalone (elastic#24128) Set Elastic licence type for APM server Beats update job (elastic#24122) Add logrotation section on Running Filebeat on k8s (elastic#24120) [CI] Run if manual UI (elastic#24116) [CI] enable x-pack/heartbeat in the CI (elastic#23873)

v1v · 2021-02-22T17:04:10Z

Indeed! b53ae70 should be the way

v1v · 2021-04-26T12:32:21Z

@Mergifyio update

mergify · 2021-04-26T12:32:36Z

Command update: failure

Base branch update has failed
merge conflict between base and head
err-code: 5A4B0

* upstream/master: (658 commits) Add complete k8s metadata through composable provider (elastic#27691) Revert "Fix issue where --insecure didn't propogate to Fleet Server ES connection (elastic#27969)" (elastic#27997) Remove deprecated kafka fields (elastic#27938) [Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input (elastic#27873) Improve httpjson template function join (elastic#27996) Remove kubernetes.container.image alias (elastic#27898) [Elastic Agent] Golden files for program tests (elastic#27862) [Elastic Agent] Disable modules.d in metricbeat (elastic#27860) libbeat/common/seccomp: provide default policy for linux arm64 (elastic#27955) Fix logger statement in aws-s3 input (elastic#27982) Fix wrong merge (elastic#27976) Fix issue where --insecure didn't propogate to Fleet Server ES connection (elastic#27969) Forward-port 7.14.2 changelog to master (elastic#27975) [Filebeat] Removing duplicate modules (aliases) Observability (elastic#27919) Fix path in vagrant windows script (elastic#27966) [Filebeat] Removing duplicate modules (aliases) and Cyberark (elastic#27915) No changelog for 8.0.0-alpha2 (elastic#27961) Add write access to 'url.value' from 'request.transforms'. (elastic#27937) Docker: remove deprecated fields (elastic#27933) Filebeat: Make all filesets disabled in default configuration (elastic#27762) ...

mergify · 2021-09-22T12:39:40Z

This pull request does not have a backport label. Could you fix it @v1v? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

v1v · 2021-09-22T13:00:31Z

/test

blakerouse · 2021-10-06T13:14:51Z

Whats the update on this? Can we get this merged soon?

v1v · 2021-10-06T13:19:50Z

Whats the update on this? Can we get this merged soon?

The e2e tests are stalled and therefore killed, see the logs

For clarity, this stage triggers -> https://github.com/elastic/e2e-testing/blob/master/.ci/scripts/fleet-test.sh.

@mdelapenya , any hints why the tests are slow and get killed?

v1v · 2021-10-11T12:34:18Z

Whats the update on this? Can we get this merged soon?

After discussing with Manu offline, it's required to support a different way to run those E2E tests since the entrypoint runs the script but it runs sequentially rather than in parallel, therefore it takes ages.

I'm gonna work to support to trigger a job, actually the e2e job for fleet. This should provide faster feedback.

* upstream/master: (73 commits) Remove GCP support from Functionbeat (elastic#28253) Move labels and annotations under kubernetes.namespace. (elastic#27917) Update go release version 1.17.1 (elastic#27543) Osquerybeat: Runner and Fetcher unit tests (elastic#28290) Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289) seccomp: allow clone3 syscall for x86 (elastic#28117) packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297) [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288) [cloud][docker] use the private docker namespace (elastic#28286) Update aws-lambda-go library version to 1.13.3 (elastic#28236) Deprecate common.Float (elastic#28280) Filebeat: Change compatibility test stage to test against previous minor instead of 7.11 (elastic#28274) x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling (elastic#28124) Explicitly pass http config to doppler consumer (elastic#28277) processors/actions/add_fields: Do not panic if event.Fields is nil map (elastic#28219) Resolved timestamp for defender atp (elastic#28272) [Winlogbeat] Tolerate faults when Windows Event Log session is interrupted (elastic#28191) [elastic-agent] proxy requests to subprocesses to their metrics endpoints (elastic#28165) Build cloud docker images for elastic-agent (elastic#28134) Upgrade k8s go-client library (elastic#28228) ...

v1v · 2021-10-11T19:41:42Z

/test

Jenkinsfile

x-pack/elastic-agent/Jenkinsfile.yml

v1v · 2021-10-13T09:19:25Z

/test

v1v · 2021-10-13T09:24:35Z

Status update

The e2e testing for the elastic-agent is now in place, and a particular GitHub check is created:

Though, I discussed offline with @adam-stokes and we thought to enable the e2e testing in incremental stages, then we can initially provide a subset of what the fleet-server/elastic-agent e2e scenarios are in place and then add incrementally more scenarios, this will avoid having any kind of flakiness if we move forward with this approach.

Short story, elastic/e2e-testing#1648 allows to configure those scenarios/suites in files, but need to agree with the rest of the team.

cc @blakerouse @mdelapenya

Jenkinsfile

(cherry picked from commit a2fde72)

(cherry picked from commit a2fde72) Co-authored-by: Victor Martinez <[email protected]>

* upstream/7.x: chore: use same branching schema for e2e (#28493) [Automation] Update elastic stack version to 7.16.0-8bf0b9b1 for testing (#28484) chore(ci): remove not used param when triggering e2e tests (#27823) (#27828) Add deprecation warning for filebeat disabled modules (#28393) Run Python tests in libbeat (#28438) (#28485) Register `storage_account` as second name for `storage` metricset (#28447) (#28470) [CI] Add E2E for x-pack/elastic-agent (fleet) (#24112) (#28465)

…28481 * upstream/7.15: chore(ci): remove not used param when triggering e2e tests (#27823) (#27830) Run Python tests in libbeat (#28438) (#28486) [CI] Add E2E for x-pack/elastic-agent (fleet) (#24112) (#28466)

[CI] Run e2e for fleet

eea3d6b

v1v self-assigned this Feb 18, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 18, 2021

v1v mentioned this pull request Feb 18, 2021

Fleet goal elastic/e2e-testing#784

Merged

2 tasks

andresrc added the Team:Automation Label for the Observability productivity team label Feb 18, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 18, 2021

v1v added 3 commits February 22, 2021 17:00

Fail the e2e instead get the status

b53ae70

PUblish packages earlier to be consumed if needed

049a1b9

Merge branch 'master' into feature/e2e-fleet

280fae6

v1v and others added 2 commits April 26, 2021 13:32

Merge branch 'master' into feature/e2e-fleet

bfbab2d

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 22, 2021

v1v added 2 commits October 11, 2021 16:48

ci: support e2e stage by running the given job

5733224

v1v commented Oct 12, 2021

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

Jenkinsfile Outdated Show resolved Hide resolved

x-pack/elastic-agent/Jenkinsfile.yml Outdated Show resolved Hide resolved

v1v mentioned this pull request Oct 12, 2021

ci: customise test matrix file elastic/e2e-testing#1648

Merged

v1v commented Oct 13, 2021

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

Use testMatrixFile

ba37774

v1v commented Oct 13, 2021

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

Update Jenkinsfile

fd2a2fe

v1v added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify labels Oct 13, 2021

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Oct 13, 2021

v1v changed the title ~~WIP [CI] Add E2E for x-pack/elastic-agent (fleet)~~ [CI] Add E2E for x-pack/elastic-agent (fleet) Oct 14, 2021

v1v requested review from blakerouse and a team October 14, 2021 07:32

v1v marked this pull request as ready for review October 14, 2021 07:32

mdelapenya approved these changes Oct 14, 2021

View reviewed changes

kuisathaverat approved these changes Oct 14, 2021

View reviewed changes

blakerouse approved these changes Oct 14, 2021

View reviewed changes

v1v merged commit a2fde72 into elastic:master Oct 15, 2021

mergify bot pushed a commit that referenced this pull request Oct 15, 2021

[CI] Add E2E for x-pack/elastic-agent (fleet) (#24112)

274467d

(cherry picked from commit a2fde72)

mergify bot mentioned this pull request Oct 15, 2021

[7.x](backport #24112) [CI] Add E2E for x-pack/elastic-agent (fleet) #28465

Merged

mergify bot pushed a commit that referenced this pull request Oct 15, 2021

[CI] Add E2E for x-pack/elastic-agent (fleet) (#24112)

d74940f

(cherry picked from commit a2fde72)

mergify bot mentioned this pull request Oct 15, 2021

[7.15](backport #24112) [CI] Add E2E for x-pack/elastic-agent (fleet) #28466

Merged

v1v deleted the feature/e2e-fleet branch October 15, 2021 11:31

v1v mentioned this pull request Oct 17, 2021

CI: disable e2e for elastic-agent on tags/branches #28481

Merged

v1v added a commit that referenced this pull request Oct 18, 2021

[CI] Add E2E for x-pack/elastic-agent (fleet) (#24112) (#28466)

bbc179f

(cherry picked from commit a2fde72) Co-authored-by: Victor Martinez <[email protected]>

v1v added a commit that referenced this pull request Oct 18, 2021

[CI] Add E2E for x-pack/elastic-agent (fleet) (#24112) (#28465)

0dcd64d

(cherry picked from commit a2fde72) Co-authored-by: Victor Martinez <[email protected]>

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

[CI] Add E2E for x-pack/elastic-agent (fleet) (elastic#24112)

9e4de85

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Add E2E for x-pack/elastic-agent (fleet) #24112

[CI] Add E2E for x-pack/elastic-agent (fleet) #24112

v1v commented Feb 18, 2021 •

edited

Loading

elasticmachine commented Feb 18, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

v1v commented Feb 22, 2021

v1v commented Feb 22, 2021 •

edited

Loading

mdelapenya commented Feb 22, 2021

v1v commented Feb 22, 2021

v1v commented Feb 22, 2021

v1v commented Feb 22, 2021

v1v commented Apr 26, 2021

mergify bot commented Apr 26, 2021

mergify bot commented Sep 22, 2021

v1v commented Sep 22, 2021

blakerouse commented Oct 6, 2021

v1v commented Oct 6, 2021

v1v commented Oct 11, 2021

v1v commented Oct 11, 2021

v1v commented Oct 13, 2021

v1v commented Oct 13, 2021

[CI] Add E2E for x-pack/elastic-agent (fleet) #24112

[CI] Add E2E for x-pack/elastic-agent (fleet) #24112

Conversation

v1v commented Feb 18, 2021 • edited Loading

What does this PR do?

Why is it important?

Important

Issues

Follow up

elasticmachine commented Feb 18, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

v1v commented Feb 22, 2021

v1v commented Feb 22, 2021 • edited Loading

mdelapenya commented Feb 22, 2021

v1v commented Feb 22, 2021

v1v commented Feb 22, 2021

v1v commented Feb 22, 2021

v1v commented Apr 26, 2021

mergify bot commented Apr 26, 2021

mergify bot commented Sep 22, 2021

v1v commented Sep 22, 2021

blakerouse commented Oct 6, 2021

v1v commented Oct 6, 2021

v1v commented Oct 11, 2021

v1v commented Oct 11, 2021

v1v commented Oct 13, 2021

v1v commented Oct 13, 2021

Status update

v1v commented Feb 18, 2021 •

edited

Loading

elasticmachine commented Feb 18, 2021 •

edited by jenkins-beats-ci bot

Loading

v1v commented Feb 22, 2021 •

edited

Loading