workflows: staging build and test (#4432)

* Addresses #3753

New workflows added to automate the build and test of releases using the new staging environment.
No changes made to current process to ensure we can keep using it.

Build & test of packaging
Packages built to staging in S3 bucket: https://fluentbit-staging.s3.amazonaws.com
We then verify the packages using kitchen-dokken to spin up OS images as containers, install the relevant RPM/Deb and check the service is properly running then. We are testing that the packaging process is correct.

Containers build to Github Container Registry, gchr.io, using multi-arch manifests.
Container tests then verify each architecture runs locally as well as a simple Helm deployment on KIND.

All package and container build definitions brought into the repo from external sources - containers were in this repo and packages were not so that is now identical plus having them together makes it a lot easier to manage and use.

Security
Trivy and Dockle scanning added - ignores current failures so these should be reviewed and addressed as needed.
Hadolint and Shellcheck really should be used too but this can be a separate PR.

Cosigning of container images if a key is provided, and using the experimental keyless option too.
GPG signing of binary packages as well as normal.

Additional work
Initial promotion from staging to release provided using a new release environment for approval - this needs creating.
Initial multi-arch container image definition and workflow also added.

Follow up PRs to improve testing, build on self-hosted and cover the promotion to release process. Trying to prevent a big bag and reduce review overhead.

Infra updates
Create release and staging environments.
Create the following secrets:

AWS_S3_BUCKET_STAGING
AWS_S3_BUCKET_RELEASE
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
COSIGN_PRIVATE_KEY
COSIGN_PASSWORD - optional if private key does not require
COSIGN_PUBLIC_KEY
FLUENTBITIO_HOST
FLUENTBITIO_USERNAME
FLUENTBITIO_SSHKEY
GPG_PRIVATE_KEY
We can actually start breaking these secrets up into the two environments.

Signed-off-by: Patrick Stephens <[email protected]>

.github/containerscan/allowedlist.yaml

@@ -0,0 +1,30 @@
+general:
+  # Only unfixed should go here
+  # Order by priority then year then ascending value
+  vulnerabilities:
+    # Critical
+    - CVE-2021-33574
+    - CVE-2021-35942
+    # High
+    - CVE-2018-12886
+    - CVE-2019-3843
+    - CVE-2019-3844
+    - CVE-2019-12290
+    - CVE-2019-15847
+    - CVE-2020-1751
+    - CVE-2020-1752
+    - CVE-2021-3326
+    - CVE-2021-33560
+    - CVE-2021-43618
+  bestPracticeViolations:
+    # Fatal
+    - DKL-DI-0005 # Clear apt-get caches
+    # Warn
+    - CIS-DI-0001 # Create a user for the Container
+    - DKL-DI-0006 # Avoid latest tag
+    # Info
+    - CIS-DI-0005 # Enable content trust for Docker
+    - CIS-DI-0006 # Add HEALTHCHECK instruction to the container image
+    - CIS-DI-0008 # Confirm safety of setuid/setgid files
+    # Skip
+    - DKL-LI-0001 # Avoid empty password

.github/dependabot.yml

@@ -0,0 +1,7 @@
+version: 2
+updates:
+  # Maintain dependencies for GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "daily"

.github/workflows/README.md

@@ -0,0 +1,56 @@
+# Available workflows
+
+| Workflow file                                         | Description               | Run event                                         |
+| :---------------------------------------------------- | ------------------------  | ------------------------------------------------- |
+| [staging-build](./staging-build.yaml)            | Builds the distro packages and docker images from a tagged release into staging (S3 and GHCR) | on new release/tag |
+| [staging-test](./staging-build.yaml)            | Test the staging distro packages and docker images| manually or when `staging-build` completes successfully |
+| [staging-release](./staging-release.yaml)        | Publishes the docker images/manifest on hub.docker.io/fluent/ and the distro packages | manual approval |
+| [pr-closed-docker](./pr-closed-docker.yaml)      | Removes docker images for PR on hub.docker.io/fluentbitdev/| on pr closed|
+| [pr-compile-check](./pr-compile-check.yaml)      | Runs some compilation sanity checks on a PR |
+| [pr-stale](./pr-stale.yaml)                      | Closes stale PR(s) with no activity in 30 days | scheduled daily 01:30 AM UTC|
+| [integration-build-master](./integration-build-master.yaml)     | Builds a docker image to be used in integration testing (master branch) | on new commit/push on master|
+| [integration-build-pr](./integration-build-pr.yaml)     | Builds a docker image to be used in integration testing (pr branch) | on new commit/push on PR(s) |
+| [integration-run-master](./integration-run-master.yaml)     | Runs the integration testing suite on master | on new image built|
+| [integration-run-pr](./integration-run-pr.yaml)     | Runs the integration testing suite on a PR branch | pr opened / label created 'ok-to-test' |
+| [unit-tests](./unit-tests.yaml)     | Runs the unit tests suite on master push or new PR | PR opened, merge in master branch |
+
+## Available labels
+
+| Label name | Description |
+| :----------|-------------|
+| docs-required| default tag used to request documentation, has to be removed before merge |
+| ok-to-test | run all integration tests |
+| ok-to-merge | run mergebot and merge (rebase) current PR |
+| ci/integration-docker-ok | integration test is able to build docker image |
+| ci/integration-gcp-ok | integration test is able to run on GCP |
+| long-term | long running pull request, don't close |
+| exempt-stale | prevent stale checks running |
+
+## Required secrets
+
+* AWS_ACCESS_KEY_ID
+* AWS_SECRET_ACCESS_KEY
+* AWS_S3_BUCKET_STAGING
+* AWS_S3_BUCKET_RELEASE
+* GPG_PRIVATE_KEY
+
+These are only required for Cosign of the container images, will be skipped if not present:
+* COSIGN_PUBLIC_KEY
+* COSIGN_PRIVATE_KEY
+* COSIGN_PRIVATE_KEY_PASSWORD - if set otherwise not required
+
+## Environments
+
+Two environments are used:
+* `staging` for all staging builds
+* `release` for running the promotion of staging to release, this can have additional approvals added
+
+If an environment is not present then it will be created but this may not have the appropriate permissions then.
+## Pushing to Github Container Registry
+
+Github actions require specific permissions to push to packages, see: https://github.community/t/403-error-on-container-registry-push-from-github-action/173071/39
+For some reason this is not automatically done via permission inheritance or similar.
+
+1. Verify you can push with a simple test, e.g. `docker pull alpine && docker tag alpine:latest ghcr.io/<repo>/fluent-bit:latest && docker push ghcr.io/<repo>/fluent-bit:latest`
+2. Once this is working locally, you should then be able to set up action permissions for the repository. If you already have a package no need to push a test one.
+3. Go to `https://github.com/users/USER/packages/container/fluent-bit/settings` and ensure the repository has access to `Write`.