fluentbit triggers OOM killer on Debian based systems

[jum]

Bug Report

Describe the bug

Running fluent-bit on Debian based systems appears to leak memory, the resident set size reaches gigabytes in a short time.

{
  "insertId": "ddgxlefks8baq",
  "jsonPayload": {
    "transport": "kernel",
    "syslog_identifier": "kernel\u0001",
    "machine_id": "298b0c1b4dc8464dbda33073d382c5ad",
    "boot_id": "6a4f4137a668408dab2bc03703e55618",
    "runtime_scope": "system",
    "message": "Out of memory: Killed process 1185587 (fluent-bit) total-vm:93200544kB, anon-rss:6249596kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:161220kB oom_score_adj:0\u0003",
    "source_monotonic_timestamp": "1466139876935",
    "hostname": "host.example.org\u0002",
    "priority": "3",
    "syslog_facility": "0"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "zone": "hosteurope",
      "project_id": "project-id",
      "instance_id": "instanceid"
    }
  },
  "timestamp": "2024-12-29T17:38:30.280174970Z",
  "severity": "ERROR",
  "logName": "projects/hosting-349701/logs/host.systemd",
  "receiveTimestamp": "2024-12-29T17:38:32.563449402Z"
}

To Reproduce

• Steps to reproduce the problem:

Run fluent-bit with the config.yaml below and watch the RSS for a while.

Expected behavior

The RSS of the fluent-bit process should not grow over time.

Your Environment

• Version used:

fluent-bit/bookworm,now 3.2.3 amd64 [installed]

• Configuration:

service:
  flush: 1
  daemon: Off
  log_level: info
  http_server: Off
  http_listen: 0.0.0.0
  http_port: 2020
  storage.metrics: on

pipeline:
  inputs:
    - name: systemd
      tag: host.systemd
      DB: /var/lib/fluent-bit/journal.db
      Lowercase: on
      Strip_Underscores: on
      processors:
        logs:
          - name: lua
            call: modify
            code: |
              function modify(tag, timestamp, record)
                new_record = record
                prio = record["priority"]
                if(prio == "7")
                then
                  new_record["severity"] = "DEBUG"
                elseif(prio == "6")
                then
                  new_record["severity"] = "INFO"
                elseif(prio == "5")
                then
                  new_record["severity"] = "NOTICE"
                elseif(prio == "4")
                then
                  new_record["severity"] = "WARNING"
                elseif(prio == "3")
                then
                  new_record["severity"] = "ERROR"
                elseif(prio == "2")
                then
                  new_record["severity"] = "CRITICAL"
                elseif(prio == "1")
                then
                  new_record["severity"] = "ALERT"
                elseif(prio == "0")
                then
                  new_record["severity"] = "EMERGENCY"
                end
                return 1, timestamp, new_record
              end
  outputs:
    - name: stackdriver
      match: '*'
      severity_key: severity
      google_service_credentials: /etc/fluent-bit/xxxx.json
      resource: gce_instance
      resource_labels: instance_id=XXXX,zone=hosteurope

• Environment name and version (e.g. Kubernetes? What version?):

Running via systemd.

• Server type and version:
• Operating System and version:

Debian Bookworm on AMD64 VM

• Filters and plugins:

journald input, lua filter and Stackdriver output, LUA filter as per the above config.

[bpetermannS11]

probably a duplicate of #9769, a memory leak in the systemd input plugin if its lowercase option is on. That's fixed with #9773

[jum]

Fixed with v3.2.4.