Migrate X-PACK Filebeat pipeline to main Beat pipeline #38941

alexsapran · 2024-04-15T13:29:56Z

Proposed commit message

Migrating the x-pack filebeat pipeline into the centralized Beats pipelines

Testing

Simple build "default" case
https://buildkite.com/elastic/beats/builds/5731#018ee6f4-2d18-4a97-882b-2aa0d90dfa65
Adding all the labels
https://buildkite.com/elastic/beats/builds/5764
Running cloud tests build
https://buildkite.com/elastic/beats-xpack-filebeat/builds/1655#018eeb54-b690-431f-8d4b-59cd33a9870f
Same build from Jenkins https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-38941/16/pipeline/2687/

Signed-off-by: Alexandros Sapranidis [email protected]

Signed-off-by: Alexandros Sapranidis <[email protected]>

elasticmachine · 2024-04-15T13:44:26Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 13 min 19 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Signed-off-by: Alexandros Sapranidis <[email protected]>

mergify · 2024-04-16T09:58:18Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b buildkite/x-pack-filebeat upstream/buildkite/x-pack-filebeat
git merge upstream/main
git push upstream buildkite/x-pack-filebeat

…lebeat Signed-off-by: Alexandros Sapranidis <[email protected]>

Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran · 2024-04-16T14:44:50Z

/test

Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran · 2024-04-16T15:24:08Z

/test x-pack/filebeat for aws cloud

Signed-off-by: Alexandros Sapranidis <[email protected]>

…lebeat

Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran · 2024-04-17T09:11:25Z

/test x-pack/filebeat for aws cloud

alexsapran · 2024-04-17T13:25:19Z

There are some failures, which I re-triggered, they are known flaky windows tests.

dliappis

Mostly LGTM left a few questions

dliappis · 2024-04-17T14:05:45Z

.buildkite/x-pack/pipeline.xpack.filebeat.yml

+              context: "x-pack/filebeat: Windows 11 Unit Tests"
+
+  - group: "x-pack/filebeat Extended Tests"
+    key: "x-pack-filebeat-extended-tests"


shouldn't this group be skipped for non pull requests? in Jenkins it is part of the extended group https://github.com/elastic/beats/blob/main/x-pack/filebeat/Jenkinsfile.yml

It is done, but the exception is in the steps not in the group.

This is how it was done with Filebeat.

dliappis · 2024-04-17T14:06:35Z

.buildkite/x-pack/pipeline.xpack.filebeat.yml

+    key: "x-pack-filebeat-extended-tests"
+    steps:
+      - label: ":mac: MacOS x86_64 Unit Tests"
+        if: build.env("BUILDKITE_PULL_REQUEST") == "false"  || build.env("GITHUB_PR_LABELS") =~ /.*(macOS|aws).*/


if we enable if: build.env("BUILDKITE_PULL_REQUEST") == "false" at the group level we can remove it from here

this is how we had it for Filebeat as well.

dliappis · 2024-04-17T14:06:40Z

.buildkite/x-pack/pipeline.xpack.filebeat.yml

+
+      - label: ":mac: MacOS arm64 Unit Tests"
+        skip: "https://github.com/elastic/beats/issues/33036"
+        if: build.env("BUILDKITE_PULL_REQUEST") == "false"  || build.env("GITHUB_PR_LABELS") =~ /.*(macOS|aws).*/


if we enable if: build.env("BUILDKITE_PULL_REQUEST") == "false" at the group level we can remove it from here

Signed-off-by: Alexandros Sapranidis <[email protected]>

dliappis

LGTM

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]> (cherry picked from commit fcd4be0) # Conflicts: # .buildkite/scripts/cloud_tests.sh # .buildkite/scripts/common.sh # .buildkite/x-pack/pipeline.xpack.filebeat.yml

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]> (cherry picked from commit fcd4be0)

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]>

… pipeline (#39036) * Migrate X-PACK Filebeat pipeline to main Beat pipeline (#38941) Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]> * Remove darwin arm64 Signed-off-by: Alexandros Sapranidis <[email protected]> * Remove hearbeat Signed-off-by: Alexandros Sapranidis <[email protected]> --------- Signed-off-by: Alexandros Sapranidis <[email protected]> Co-authored-by: Alexandros Sapranidis <[email protected]>

… pipeline (#39037) * Migrate X-PACK Filebeat pipeline to main Beat pipeline (#38941) Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]> * Fix missed merge conflict Signed-off-by: Alexandros Sapranidis <[email protected]> --------- Signed-off-by: Alexandros Sapranidis <[email protected]> Co-authored-by: Alexandros Sapranidis <[email protected]>

… pipeline (#39044) Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]> Co-authored-by: Alexandros Sapranidis <[email protected]>

alexsapran added 5 commits April 15, 2024 16:18

Remove bot trigger

27091d5

Signed-off-by: Alexandros Sapranidis <[email protected]>

Add X-pack Filebeat trigger step into main pipeline

8a8a120

Signed-off-by: Alexandros Sapranidis <[email protected]>

Remove x-pack filebeat from hooks

3be1c17

Signed-off-by: Alexandros Sapranidis <[email protected]>

Remove x-pack filebeat from setenv.sh

a8e0348

Signed-off-by: Alexandros Sapranidis <[email protected]>

Migrating x-pack filebeat pipeline

a3d9527

Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran added backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.13.0 Automated backport with mergify Team:Ingest-EngProd labels Apr 15, 2024

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 15, 2024

mergify bot assigned alexsapran Apr 15, 2024

alexsapran mentioned this pull request Apr 15, 2024

Remove X-PACK Filebeat bot trigger #38943

Merged

alexsapran changed the title ~~Remove bot trigger~~ Migrate X-PACK Filebeat pipeline to main Beat pipeline Apr 15, 2024

Finish up mandatory added ext and ext-win steps

83830ca

Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran added 7 commits April 16, 2024 15:05

Merge remote-tracking branch 'upstream/main' into buildkite/x-pack-fi…

068387f

…lebeat Signed-off-by: Alexandros Sapranidis <[email protected]>

Revert xpack-metricbeat

4ff6f12

Signed-off-by: Alexandros Sapranidis <[email protected]>

Add extended and packaging steps

b35a10a

Signed-off-by: Alexandros Sapranidis <[email protected]>

Remove dynamic pipeline

03be6d2

Signed-off-by: Alexandros Sapranidis <[email protected]>

Revert line

193a0e9

Signed-off-by: Alexandros Sapranidis <[email protected]>

Cleanup

6dc9df5

Signed-off-by: Alexandros Sapranidis <[email protected]>

Adjust extended triggering conditions

37d2c86

Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran added :Windows macOS Enable builds in the CI for darwin testing arm Enable builds in the CI for ARM testing aws Enable builds in the CI for aws cloud testing labels Apr 16, 2024

Adjust cloud_tests

ef77712

Signed-off-by: Alexandros Sapranidis <[email protected]>

Set the root dir when running teardown

2fdafc4

Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran removed the macOS Enable builds in the CI for darwin testing label Apr 17, 2024

alexsapran added 2 commits April 17, 2024 11:52

Merge remote-tracking branch 'upstream/main' into buildkite/x-pack-fi…

d342508

…lebeat

Fix bash error

235e6ef

Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran marked this pull request as ready for review April 17, 2024 13:23

alexsapran requested a review from a team as a code owner April 17, 2024 13:23

alexsapran requested a review from dliappis April 17, 2024 13:23

dliappis reviewed Apr 17, 2024

View reviewed changes

alexsapran requested a review from dliappis April 17, 2024 15:57

adjust step conditions

3565b08

Signed-off-by: Alexandros Sapranidis <[email protected]>

dliappis approved these changes Apr 18, 2024

View reviewed changes

dliappis added the backport-v8.14.0 Automated backport with mergify label Apr 18, 2024

alexsapran merged commit fcd4be0 into elastic:main Apr 18, 2024
130 of 139 checks passed

mergify bot pushed a commit that referenced this pull request Apr 18, 2024

Migrate X-PACK Filebeat pipeline to main Beat pipeline (#38941)

ae00898

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]> (cherry picked from commit fcd4be0)

This was referenced Apr 18, 2024

[7.17](backport #38941) Migrate X-PACK Filebeat pipeline to main Beat pipeline #39036

Merged

[8.13](backport #38941) Migrate X-PACK Filebeat pipeline to main Beat pipeline #39037

Merged

alexsapran added backport-v8.14.0 Automated backport with mergify and removed backport-v8.14.0 Automated backport with mergify labels Apr 18, 2024

mergify bot pushed a commit that referenced this pull request Apr 18, 2024

Migrate X-PACK Filebeat pipeline to main Beat pipeline (#38941)

9611b38

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]> (cherry picked from commit fcd4be0)

mergify bot mentioned this pull request Apr 18, 2024

[8.14](backport #38941) Migrate X-PACK Filebeat pipeline to main Beat pipeline #39044

Merged

alexsapran added a commit that referenced this pull request Apr 18, 2024

Migrate X-PACK Filebeat pipeline to main Beat pipeline (#38941)

b6bd442

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran added a commit that referenced this pull request Apr 18, 2024

Migrate X-PACK Filebeat pipeline to main Beat pipeline (#38941)

1b50a3b

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran added a commit that referenced this pull request Apr 18, 2024

Migrate X-PACK Filebeat pipeline to main Beat pipeline (#38941)

ac9f0b4

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]>

alexsapran added a commit that referenced this pull request Apr 18, 2024

Migrate X-PACK Filebeat pipeline to main Beat pipeline (#38941)

a3242f3

Migrate x-pack filebeat to single pipeline Signed-off-by: Alexandros Sapranidis <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate X-PACK Filebeat pipeline to main Beat pipeline #38941

Migrate X-PACK Filebeat pipeline to main Beat pipeline #38941

alexsapran commented Apr 15, 2024 •

edited

Loading

elasticmachine commented Apr 15, 2024 •

edited

Loading

Build stats

mergify bot commented Apr 16, 2024

alexsapran commented Apr 16, 2024

alexsapran commented Apr 16, 2024

alexsapran commented Apr 17, 2024

alexsapran commented Apr 17, 2024

dliappis left a comment

dliappis Apr 17, 2024

alexsapran Apr 17, 2024

dliappis Apr 17, 2024

alexsapran Apr 17, 2024

dliappis Apr 17, 2024

dliappis left a comment

Migrate X-PACK Filebeat pipeline to main Beat pipeline #38941

Migrate X-PACK Filebeat pipeline to main Beat pipeline #38941

Conversation

alexsapran commented Apr 15, 2024 • edited Loading

Proposed commit message

Testing

elasticmachine commented Apr 15, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

mergify bot commented Apr 16, 2024

alexsapran commented Apr 16, 2024

alexsapran commented Apr 16, 2024

alexsapran commented Apr 17, 2024

alexsapran commented Apr 17, 2024

dliappis left a comment

Choose a reason for hiding this comment

dliappis Apr 17, 2024

Choose a reason for hiding this comment

alexsapran Apr 17, 2024

Choose a reason for hiding this comment

dliappis Apr 17, 2024

Choose a reason for hiding this comment

alexsapran Apr 17, 2024

Choose a reason for hiding this comment

dliappis Apr 17, 2024

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment

alexsapran commented Apr 15, 2024 •

edited

Loading

elasticmachine commented Apr 15, 2024 •

edited

Loading