Update xpack/filebeat & xpack/metricbeat pipelines to match Jenkins steps

[oakrizan]

Proposed commit message

Updated x-pack/filebeat and x-pack/metricbeat pipelines to match same steps & commands as in Jenkins.

Jenkins vs. Buildkite comparison

• x-pack/filebeat: https://gist.github.com/oakrizan/0ef4a4df595547489538cbbc0ad1a4e5
• x-pack/metricbeat: https://gist.github.com/oakrizan/97eabb0635ad7727ae2b37e3f8469ff8

BK builds:

• x-pack/filebeat: https://buildkite.com/elastic/beats-xpack-filebeat/builds?branch=oakrizan%3Axpack-filebeat-fixes
• x-pack/metricbeat: https://buildkite.com/elastic/beats-xpack-metricbeat/builds?branch=oakrizan%3Axpack-filebeat-fixes

For both pipelines updated cloud test environment setup script and test commands to match Jenkins.

x-pack/filebeat: Cloud (MODULE) Tests failure is not related to current PR changes.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• .buildkite/x-pack/pipeline.xpack.filebeat.yml
• .buildkite/x-pack/pipeline.xpack.metricbeat.yml

How to test this PR locally

Related issues

• Relates: https://github.com/elastic/ingest-dev/issues/3254

Use cases

Screenshots

Logs

BK builds:
x-pack/filebeat: https://buildkite.com/elastic/beats-xpack-filebeat/builds?branch=oakrizan%3Axpack-filebeat-fixes
x-pack/metricbeat: https://buildkite.com/elastic/beats-xpack-metricbeat/builds?branch=oakrizan%3Axpack-filebeat-fixes

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @oakrizan? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-05-09T11:57:24.624+0000

• Duration: 12 min 13 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2
Skipped	0
Total	2

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b xpack-filebeat-fixes upstream/xpack-filebeat-fixes
git merge upstream/main
git push upstream xpack-filebeat-fixes

[oakrizan]

@dliappis PTAL 🙏

[dliappis]

LGTM

Left a small comment that can be addressed without another review cycle.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b xpack-filebeat-fixes upstream/xpack-filebeat-fixes
git merge upstream/main
git push upstream xpack-filebeat-fixes