Migrate needed .ci/* scripts into .buildkite/* ( elastic#3353 )

[ev1yehor]

Proposed commit message

Migrate needed .ci/* scripts into .buildkite/* and change path for them.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Relates https://github.com/elastic/ingest-dev/issues/3353

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ev1yehor? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[dliappis]

Looks good!

A couple of suggestions:

1. Let's wait that https://github.com/elastic/beats/pull/39648/files get's merged because it changes the version of localstack in the docker-compose file

2. Let's run the entire testing suite by adding all the optional GH labels aws, :Windows, macOS to ensure nothing breaks with this.

[dliappis]

@ev1yehor fyi, after adding labels (thanks for doing that!) we need to manually retrigger tests by leaving a comment like #39392 (comment) -- trigger phrase can be found in

beats/.buildkite/pull-requests.json

Lines 12 to 13 in 809e001

	"trigger_comment_regex": "^/(?:test)",
	"always_trigger_comment_regex": "^/(?:test)",

) . They won't be automatically retriggered just by changing labels. Another way to retrigger is to click "Update branch", this will also retrigger the test.

[ev1yehor]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b Migrate-files-issue-3353 upstream/Migrate-files-issue-3353
git merge upstream/main
git push upstream Migrate-files-issue-3353

[ev1yehor]

/test

[ev1yehor]

/test

[oakrizan]

There is no docker-compose.yml on 7.17. I believe that file shouldn't be backported there. WDYT @dliappis?

[dliappis]

There is no docker-compose.yml on 7.17. I believe that file shouldn't be backported there. WDYT @dliappis?

Indeed, thanks for catching. However, we still have references to the non-existing .ci/jobs/docker-compose file right now in the 7.17 branch in .buildkite/scripts/common.sh and .buildkite/scrpts/setup_cloud_env.sh so I'd say that specifically in the 7.17 backport of this we could do the needed clean ups and checks.

[dliappis]

LGTM

seems that docker-compose got trigger as expected: https://buildkite.com/elastic/beats-xpack-filebeat/builds/3013#018fba0c-7e0b-43ad-b640-57c0f0689171/157-158