workflows: staging build and test

[patrick-stephens]

Addresses issue 3753

New workflows added to automate the build and test of releases using the new staging environment.
No changes made to current process to ensure we can keep using it.

Build & test of packaging

Packages built to staging in S3 bucket: https://fluentbit-staging.s3.amazonaws.com
We then verify the packages using kitchen-dokken to spin up OS images as containers, install the relevant RPM/Deb and check the service is properly running then. We are testing that the packaging process is correct.

Containers build to Github Container Registry, gchr.io, using multi-arch manifests.
Container tests then verify each architecture runs locally as well as a simple Helm deployment on KIND.

All package and container build definitions brought into the repo from external sources - containers were in this repo and packages were not so that is now identical plus having them together makes it a lot easier to manage and use.

Security

Trivy and Dockle scanning added - ignores current failures so these should be reviewed and addressed as needed.
Hadolint and Shellcheck really should be used too but this can be a separate PR.

Cosigning of container images if a key is provided, and using the experimental keyless option too.
GPG signing of binary packages as well as normal.

Additional work

Initial promotion from staging to release provided using a new release environment for approval - this needs creating.
Initial multi-arch container image definition and workflow also added.

Follow up PRs to improve testing, build on self-hosted and cover the promotion to release process. Trying to prevent a big bag and reduce review overhead.

Infra updates

Create release and staging environments.
Create the following secrets:

• AWS_S3_BUCKET_STAGING
• AWS_S3_BUCKET_RELEASE
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• COSIGN_PRIVATE_KEY
• COSIGN_PASSWORD - optional if private key does not require
• COSIGN_PUBLIC_KEY
• FLUENTBITIO_HOST
• FLUENTBITIO_USERNAME
• FLUENTBITIO_SSHKEY
• GPG_PRIVATE_KEY

We can actually start breaking these secrets up into the two environments.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [NA] Example configuration file for the change
• [NA] Debug log output from testing the change

• [NA] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• [NA] Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[niedbalski]

• git rebase HEAD~250 --signoff
• update the workflows/README.md
• provide a list of the required secrets to be added into infra.
• remove act (?)

[patrick-stephens]

Closed as original repo had issues with squash+sign