workflows: nightly build updates and regular release cadence

[patrick-stephens]

Once 1.9 release is out we want to move to a regular nightly build that produces a floating unstable tag.
We also want to support building from a branch (e.g. master or 1.9) without a tag and then creating a new tag (e.g. for 1.9.1) at the end of the process once we're happy.

We also need to distinguish the version we build from (i.e. the commit/branch) from the version we're creating (i.e. the tag/release for a new official version).
Currently the process is all assuming the current workflow of building tag X to make release X.

We want to move to commit rather than tag-based release process:

• Regularly build (nightly) latest commit on master
• Create a new/updated unstable Github release for this with all artefacts - derisked already.
• Update testing to use this unstable release URL for artefacts and containers.
• On successful testing move to staging as "ready to release" - plus this can then trigger performance ad resilience tests.
• The current release workflow will then take from staging (currently implemented) and create the tag (new).

The last step can also be automated to have a regular release cadence a well.
The goal is to always have something ready to release, i.e. the last green staging build.

There are some additional changes to get in here, e.g. update the containers to build from a commit rather than a tarball of the last tagged version.
This is already underway though for the multi-arch containers as well.

Currently the staging approach uses an S3 bucket and will overwrite this with the nightly build.
This update will simplify this so only "good" builds are in staging plus reduce S3 transfer costs.

Overview

graph TD
    direction LR
    Commit[Commit] -->|Cron - nightly| Unstable(Unstable)
    Unstable --> Testing{Testing}
    Testing -->|Passes| Staging(Staging)
    Testing -->|Fails| Commit
    Staging -->|Cron - weekly| Release(Release)
    Release -->|Update next version| Commit

Loading

As you can see, we should always have something to release from staging - if testing fails it does not go into staging. We can then just push whatever is in staging out to release.

Tasks

• unstable release per branch: workflows: unstable builds for supported branches #5117

• 1.8 builds

  • staging to fix: https://github.com/fluent/fluent-bit/runs/6005595200?check_suite_focus=true
  • test [ ] - need to adjust the smoke test targets
  • release [ ]

• pull from unstable release and test - update current staging test which uses bucket

• if successful, push to staging per branch - likely need to rebuild to remove the nightly information and then retest

• take staging and release (per branch)

• add SHA of Git commit to version output Add Git commit to version output #5006

• remove special characters in debug output Disable colourised output/control codes in log #5002

• add automated check to verify version and commit for release, prevent Containers for 1.8.13 reporting 1.8.12 #4970

• investigate how to promote the same binary rather than rebuilding for staging, issue is with the nightly setting

• support cmake versioning from the CI, so the build can generate the new version possibly to check in after release so people can update after if needs be - release pushes version, CMakeLists gets updated and checked in for main branch then

• Add the pre-built workflow Create Release

• Add in the workflow Bump Version

[patrick-stephens]

We can also integrate the changes for #4845 as part of this.

[patrick-stephens]

#5098 is part of this to remove any need to interact with the legacy server.

[alrz]

Would there be any nightly image? I wanted to try out some recent otlp fixes to see if it addressed some of my issues before 3.1 release.

[patrick-stephens]

Yes: ghcr.io/fluent/fluent-bit/unstable:master
See https://github.com/fluent/fluent-bit/actions/runs/9477546453/job/26112305225

All pushes to master also build a container: https://github.com/fluent/fluent-bit/tree/master/dockerfiles#ghcrio-topology

[edsiper]

@patrick-stephens is this ok to close ?

[patrick-stephens]

We have nightly builds but no regular release cadence, it's still on-demand I think.