fluent-bit repo data not available for RHEL/CentOS/AlmaLinux/Rocky 9

[vwbusguy]

Bug Report

Describe the bug
While the el8 package appears to work on el9 for now, the default repo config will reference version 9 in the repo URL which doesn't exist. Adding packages to support the current version of these distributions will make improve the experience of installing fluent-bit.

To Reproduce
Attempt to follow any install instructions from https://docs.fluentbit.io/manual/installation/linux/redhat-centos using any EL9 distribution.

• Steps to reproduce the problem:
  Either:

• Run the setup script and note that it fails due to missing packages for the current version
• Follow the manual steps in setting up a repo and note that it fails to install

Expected behavior
The setup script and instructions successfully result in fluent-bit being installed for the current version of these supported distros.

Your Environment

• Version used: fluent-bit-1.9.3-1.x86_64
• Configuration: Native RPM install
• Environment name and version (e.g. Kubernetes? What version?): Native RPM install
• Server type and version: Linux
• Operating System and version: CentOS Stream 9
• Filters and plugins: n/a

Additional context
For now we've implemented a hacky workaround in Ansible to attempt to identify an EL9 box and override the yum repo to point to the EL8 repos. The package installs fine for now, but it definitely added to the complexity of adopting fluent-bit to our environment.

Here's our current Ansible workaround in case others are similarly affected by this:

- name: Set fluentbit repo release version
  set_fact:
    fb_rel_ver: "{{ ansible_distribution_major_version }}"

- name: Set fluentbit repo release version (EL9)
  set_fact:
    fb_rel_ver: "8"
  when: ansible_distribution_major_version|int == 9

- name: Setup yum repo
  yum_repository:
    name: fluentbit
    description: Fluentbit Repo
    baseurl: "https://packages.fluentbit.io/centos/{{ fb_rel_ver }}/$basearch/"
    enabled: yes
    file: fluentbit
    gpgcheck: yes
    gpgkey: https://packages.fluentbit.io/fluentbit.key
    state: present

- name: Import rpm_key
  rpm_key:
    key: https://packages.fluentbit.io/fluentbit.key
    state: present

- name: Install fluentbit
  yum:
    name: fluent-bit
    state: latest

[patrick-stephens]

RHEL 9 targets are not officially supported yet (and it is only just GA): https://docs.fluentbit.io/manual/installation/supported-platforms.

There are other requests for this so it will be done "soon" although a PR to provide the support will be massively helpful.
With the new CentOS stream approach as well we may see more deviation between the various '9' targets so we do need to consider this as well - it's not as it was for CentOS 8 so we may have to target non-stream as the primary target and/or provide a split between upstream CentOS 9 stream support and the other more stable distributions. There is also no Rocky Linux 9 or Alma Linux 9 so currently the only "real" target is CentOS stream 9 (possibly RHEL 9 now it is GA) but I'm not happy targeting just that and assuming it'll work for the others.

This will be on the backlog and done as soon as possible but a PR will speed that up. cc @niedbalski

[patrick-stephens]

@vwbusguy could you submit a docs PR to cover the details for the installation page?
https://github.com/fluent/fluent-bit-docs/blob/master/installation/linux/redhat-centos.md

Feel free to include your workaround if you can but the main thing to cover is to modify 9 --> 8.
We can then update and remove this once we have 9 support official.

[patrick-stephens]

For reference, the main changes required are:

1. Decide on a primary distribution to support (I'm not convinced CentOS Stream is a good choice here, we may have to split into stream + stable, e.g. RockyLinux - a RHEL 9 build may be subject to licensing issues for CI and also for users).
2. Updates to provide the containerised build images for the targets we want (ARM and AMD - remember the changes in pagefile size on these so need to confirm): https://github.com/fluent/fluent-bit/blob/master/packaging/distros/centos/Dockerfile
3. Update the github actions build matrix to include these: https://github.com/fluent/fluent-bit/blob/master/.github/actions/generate-package-build-matrix/action.yaml
4. Update the packaging scripts to build repo metadata for any new targets: https://github.com/fluent/fluent-bit/blob/master/packaging/update-repos.sh
5. Add smoke tests for the new targets: https://github.com/fluent/fluent-bit/tree/master/packaging/testing/smoke/packages
6. (Anything else I've missed).

If we get #5098 sorted then we "only" need to concern ourselves with build changes and updates to the packaging scripts that create the repos. Otherwise we also have some release infrastructure set up to cope with too then the sync over to it.

[vwbusguy]

For what it's worth, we are running a mix of Alma, CentOS Stream, and RHEL and are using a common playbook for all of them from EL7-9 and everything seems to be happy so far (except for pointing EL9 stuff to 8). Targeting Stream for CI environments has been generally useful for us, since we find out sooner about changes that will impact Alma/RHEL, etc. sooner. The fact that EL8 packages work in CentOS Stream 9 is a good sign that it will probably work fine regardless of which flavor you pick :-).

That said, I'll try to get a PR for you for the docs page.

Thanks!

[patrick-stephens]

OK cool, good to know - although I'm still not convinced it won't deviate significantly as time marches on! 👍 KISS though would be to go with the combined target until it breaks for the others.

[patrick-stephens]

@vwbusguy not forgotten, this and Ubuntu 22 hopefully will be ready after the heatwave in Europe 👍

[patrick-stephens]

Ubuntu 22 is up now thankfully, I think this will likely have to wait for FB 2.0 unfortunately

[patrick-stephens]

@vwbusguy would you be able to test some preliminary packages across your various types of "RHEL 9 compatible" targets?

[vwbusguy]

Sure. I can do that on Monday (assuming I remember to check back here then).

[patrick-stephens]

@vwbusguy the staging bucket should have CentOS 9 packages in it now: https://fluentbit-staging.s3.amazonaws.com/centos-9.repo

[vwbusguy]

My apologies - it appears that I did not, in fact, remember to check back. If I get a moment today, I'll update update our playbook and try it against one of our EL9 hosts with that repo.

[vwbusguy]

I had a spare moment and updated the playbook to point to that repo for EL9 hosts and it works! Tested the playbook out against an AlmaLinux 9 node and it's up and running:

● fluent-bit.service - Fluent Bit
     Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; vendor preset: disabled)
     Active: active (running) since Thu 2022-10-20 16:04:43 PDT; 1min 54s ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 293607 (fluent-bit)
      Tasks: 5 (limit: 101976)
     Memory: 25.8M
        CPU: 240ms
     CGroup: /system.slice/fluent-bit.service
             └─293607 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

[patrick-stephens]

Great news @vwbusguy , obviously we tested as well but having an in the wild user confirm is super helpful!

[jtberry-godaddy]

Is there any update on status of when this will be ready for stable/prod repos?

[patrick-stephens]

It already is, should have CentOS 9 repos available for new releases. Need to close this.