CPU Continuously growing with Fluent-bit version > 2.0.14 on Windows Server 2019 with Multiline Filter Plugin

[ganga1980]

Bug Report

Describe the bug
CPU Continuously growing with Fluent-bit version > 2.0.14 on Windows Server 2019 with Multiline Filter Plugin.
This is not issue with Fluent-bit version 2.0.14. Tried all the versions 2.1.* and 2.2.*, and all these versions has same issue.

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit-windows
  labels:
    app: fluent-bit-windows
spec:
  selector:
    matchLabels:
      app: fluent-bit-windows
  template:
    metadata:
      labels:
        app: fluent-bit-windows
    spec:
      affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: kubernetes.io/os
               operator: In
               values:
               - windows
             - key: kubernetes.azure.com/os-sku
               operator: In
               values:
               - Windows2019 # For Windows 2022, use Windows2022
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:windows-2019-2.2.2 # For Windows 2022, use 2022 instead of 2019
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: "750Mi"
            cpu: "500m"
          # requests:
          #   memory: "100Mi"
          #   cpu: "50m"
        volumeMounts:
        - mountPath: C:\ProgramData\docker\containers
          name: docker-windows-containers
          readOnly: true
        - mountPath: C:\var #Read + Write access on this for position file
          name: docker-windows-kuberenetes-container-logs
        - name: config-volume
          mountPath: /fluent-bit/etc/
      volumes:
      - name: docker-windows-kuberenetes-container-logs
        hostPath:
          path: C:\var
      - name: docker-windows-containers
        hostPath:
          path: C:\ProgramData\docker\containers
          type: DirectoryOrCreate
      - name: config-volume
        configMap:
          name: fluent-bit-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf

    [INPUT]
        Name                tail
        Path                C:\\var\\log\\containers\*.log
        Multiline.Parser    cri
        Tag                 windows.*
        Read_from_Head      true
        DB                  C:\\var\\log\\fblogs.db
        DB.Sync             Off
        Rotate_Wait         20
        Refresh_Interval    30
        Path_Key            filepath
        Skip_Long_Lines     On

    [FILTER]
        Name multiline
        Match *
        multiline.key_content log
        multiline.parser go,java,python

    [OUTPUT]
        Name          Null
        Match         *

• Steps to reproduce the problem:

1. Apply this yaml on to Kubernetes cluster with windows server 2019 nodes
   kubectl apply -f https://raw.githubusercontent.com/microsoft/Docker-Provider/gangams/fix-windows-agent-cpu-breaches/test/fluent-bit-windows.yaml
2. Deploy some test workloads for example
   kubectl apply -f https://raw.githubusercontent.com/microsoft/Docker-Provider/gangams/fix-windows-agent-cpu-breaches/test/scenario/multiline/dotnet-win.yaml
   kubectl apply -f https://raw.githubusercontent.com/microsoft/Docker-Provider/gangams/fix-windows-agent-cpu-breaches/test/scenario/multiline/golang-win.yaml
   kubectl apply -f https://raw.githubusercontent.com/microsoft/Docker-Provider/gangams/fix-windows-agent-cpu-breaches/test/scenario/multiline/java-win.yaml
   kubectl apply -f https://raw.githubusercontent.com/microsoft/Docker-Provider/gangams/fix-windows-agent-cpu-breaches/test/scenario/multiline/python-win.yaml

Expected behavior
CPU shouldnt grow increasingly grow and usage should be same as fluent-bit version 2.0.14.

Screenshots
CPU Usage with fluent-bit version 2.0.14 vs fluent-bit version 2.2.2

Your Environment

• Version used: Fluent-bit 2.0.14 and Fluent-bit 2.2.2
• Configuration: Kubernetes with Windows Server 2019
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes, version : v1.26.12
• Server type and version: Windows Server 2019
• Operating System and version:
• Filters and plugins: Tail, Multiline and Output

Additional context

[tieptoi]

Same as my issue #8925

[yangliuqing5470]

because multiline parser is slower with fluent-bit version >=2.1.0，and i compare the commit，it's "filter_multiline: updated to use the new log event abstraction layer" commit casue the issue