Splunk output plugin log in fluent-bit pods are not sufficient for debugging

[jonathanzhang02]

Bug Report

Describe the bug
Currently as described in #8046 (comment), the Splunk output plugin emits very few "info" level logs that would be sufficient for logging.
Even connection failures seem to be hidden.
If connection success could not be logged, at least there should be an attempt for the connection.
This is very inconvenient for debugging, especially when the destination Splunk server is not owned by the same developers that set up fluent-bit.

To Reproduce

• Rubular link if applicable:
  fluent-bit stops processing logs after reaching memory buffer limit #8046 (comment)
• Example log message if applicable:

Even the debug log info is very messy and cannot see real connection errors to splunk servers very clearly.

2024-12-17T12:00:01.08524453Z stderr F [2024/12/17 12:00:01] [debug] [upstream] KA connection #176 to splunk-hec.splunk.svc.cluster.local:8088 is connected
2024-12-17T12:00:01.08525883Z stderr F [2024/12/17 12:00:01] [debug] [output:splunk:splunk.1] Could not find hec_token in metadata
2024-12-17T12:00:01.085269631Z stderr F [2024/12/17 12:00:01] [debug] [http_client] not using http_proxy for header

2024-12-17T11:54:51.028062819Z stderr F [2024/12/17 11:54:51] [ info] [input:tail:splunkt1-alias] initializing
2024-12-17T11:54:51.02806782Z stderr F [2024/12/17 11:54:51] [ info] [input:tail:splunkt1-alias] storage_strategy='memory' (memory only)
2024-12-17T11:54:51.02808092Z stderr F [2024/12/17 11:54:51] [debug] [tail:splunkt1-alias] created event channels: read=58 write=59
2024-12-17T11:54:51.047261407Z stderr F [2024/12/17 11:54:51] [ info] [input:tail:splunkt1-alias] multiline core started
2024-12-17T11:54:51.047276408Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] flb_tail_fs_inotify_init() initializing inotify tail input
2024-12-17T11:54:51.047279808Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inotify watch fd=65
2024-12-17T11:54:51.047286608Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scanning path /var/log/containers/splunkt1*_splunkt1-tt1-agent-*.log
2024-12-17T11:54:51.047997433Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-tt1-agent-l6vgn_agents_splunkt1-tt1-agent-63209238028636ccadf41c7646f24eead55bbb8b2bc0d634fe6e72dffcf67642.log
2024-12-17T11:54:51.048011534Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3386019 with offset=19687 appended as /var/log/containers/splunkt1-tt1-agent-l6vgn_agents_splunkt1-tt1-agent-63209238028636ccadf41c7646f24eead55bbb8b2bc0d634fe6e72dffcf67642.log
2024-12-17T11:54:51.048014734Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-tt1-agent-l6vgn_agents_splunkt1-tt1-agent-63209238028636ccadf41c7646f24eead55bbb8b2bc0d634fe6e72dffcf67642.log, inode 3386019
2024-12-17T11:54:51.048017534Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] 1 new files found on path '/var/log/containers/splunkt1*_splunkt1-tt1-agent-*.log'
2024-12-17T11:54:51.048020434Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scanning path /var/log/containers/splunkt1*_splunkt1-cert-exporter-*.log
2024-12-17T11:54:51.048023134Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-cert-exporter-28905840-g5286_agents_splunkt1-cert-exporter-a554f4ee290b348c2c17cb77cb69abbf4524e0b0a623c7d523e4bf559d485baf.log
2024-12-17T11:54:51.048101637Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3386092 with offset=575 appended as /var/log/containers/splunkt1-cert-exporter-28905840-g5286_agents_splunkt1-cert-exporter-a554f4ee290b348c2c17cb77cb69abbf4524e0b0a623c7d523e4bf559d485baf.log
2024-12-17T11:54:51.048109937Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-cert-exporter-28905840-g5286_agents_splunkt1-cert-exporter-a554f4ee290b348c2c17cb77cb69abbf4524e0b0a623c7d523e4bf559d485baf.log, inode 3386092
2024-12-17T11:54:51.048112437Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-cert-exporter-28906560-7mgls_agents_splunkt1-cert-exporter-1a94de92787408ac609b1d166568c7436c9d8ec634f63628b65d3c6729b6c6de.log
2024-12-17T11:54:51.048114738Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3386040 with offset=574 appended as /var/log/containers/splunkt1-cert-exporter-28906560-7mgls_agents_splunkt1-cert-exporter-1a94de92787408ac609b1d166568c7436c9d8ec634f63628b65d3c6729b6c6de.log
2024-12-17T11:54:51.048117138Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-cert-exporter-28906560-7mgls_agents_splunkt1-cert-exporter-1a94de92787408ac609b1d166568c7436c9d8ec634f63628b65d3c6729b6c6de.log, inode 3386040
2024-12-17T11:54:51.048119338Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-cert-init-job-334pl-s8fpp_agents_splunkt1-cert-exporter-32b388c66ce2790534299bc93deccdc218bd3369a434ff298509caf806f86bb9.log
2024-12-17T11:54:51.048153639Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3386081 with offset=568 appended as /var/log/containers/splunkt1-cert-init-job-334pl-s8fpp_agents_splunkt1-cert-exporter-32b388c66ce2790534299bc93deccdc218bd3369a434ff298509caf806f86bb9.log
2024-12-17T11:54:51.04817364Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-cert-init-job-334pl-s8fpp_agents_splunkt1-cert-exporter-32b388c66ce2790534299bc93deccdc218bd3369a434ff298509caf806f86bb9.log, inode 3386081
2024-12-17T11:54:51.04817784Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-cert-init-job-7if7w-4n2qm_agents_splunkt1-cert-exporter-141451c26ee8e14432c3671e943ebec7fb5b4a2c4c79a77c9dc0e55d49c56fe4.log
2024-12-17T11:54:51.048208141Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3386065 with offset=574 appended as /var/log/containers/splunkt1-cert-init-job-7if7w-4n2qm_agents_splunkt1-cert-exporter-141451c26ee8e14432c3671e943ebec7fb5b4a2c4c79a77c9dc0e55d49c56fe4.log
2024-12-17T11:54:51.048215041Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-cert-init-job-7if7w-4n2qm_agents_splunkt1-cert-exporter-141451c26ee8e14432c3671e943ebec7fb5b4a2c4c79a77c9dc0e55d49c56fe4.log, inode 3386065
2024-12-17T11:54:51.048218041Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-cert-init-job-9p2nl-lwbrp_agents_splunkt1-cert-exporter-f7a80e06c4814cd81175bb6d44c5ceb4f1862d9199fbbf7ccaf6d57bfb2b73d9.log
2024-12-17T11:54:51.048224642Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3386052 with offset=574 appended as /var/log/containers/splunkt1-cert-init-job-9p2nl-lwbrp_agents_splunkt1-cert-exporter-f7a80e06c4814cd81175bb6d44c5ceb4f1862d9199fbbf7ccaf6d57bfb2b73d9.log
2024-12-17T11:54:51.048236442Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-cert-init-job-9p2nl-lwbrp_agents_splunkt1-cert-exporter-f7a80e06c4814cd81175bb6d44c5ceb4f1862d9199fbbf7ccaf6d57bfb2b73d9.log, inode 3386052
2024-12-17T11:54:51.048242142Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-cert-init-job-gpgcs-2r8ch_agents_splunkt1-cert-exporter-55bea4d8cb05c95ac70cef1aa4300d7058330ea0d668b1100c0cc29dabbf6f9a.log
2024-12-17T11:54:51.048282644Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3386079 with offset=574 appended as /var/log/containers/splunkt1-cert-init-job-gpgcs-2r8ch_agents_splunkt1-cert-exporter-55bea4d8cb05c95ac70cef1aa4300d7058330ea0d668b1100c0cc29dabbf6f9a.log
2024-12-17T11:54:51.048289544Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-cert-init-job-gpgcs-2r8ch_agents_splunkt1-cert-exporter-55bea4d8cb05c95ac70cef1aa4300d7058330ea0d668b1100c0cc29dabbf6f9a.log, inode 3386079
2024-12-17T11:54:51.048292844Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-cert-init-job-gvg15-w7sgv_agents_splunkt1-cert-exporter-3b31c2ebf12b64fce4d63b6ceb233ce1db0512992021485622144c688532f687.log
2024-12-17T11:54:51.048337546Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3386108 with offset=575 appended as /var/log/containers/splunkt1-cert-init-job-gvg15-w7sgv_agents_splunkt1-cert-exporter-3b31c2ebf12b64fce4d63b6ceb233ce1db0512992021485622144c688532f687.log
2024-12-17T11:54:51.048343846Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-cert-init-job-gvg15-w7sgv_agents_splunkt1-cert-exporter-3b31c2ebf12b64fce4d63b6ceb233ce1db0512992021485622144c688532f687.log, inode 3386108
2024-12-17T11:54:51.048347846Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/splunkt1-cert-init-job-x1d39-jz4j6_agents_splunkt1-cert-exporter-0bc5ee79d53d94e7dfbe329528330ac573dfdf7b03c724c31a24dd89106cb8db.log
2024-12-17T11:54:51.048384947Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] inode=3385953 with offset=575 appended as /var/log/containers/splunkt1-cert-init-job-x1d39-jz4j6_agents_splunkt1-cert-exporter-0bc5ee79d53d94e7dfbe329528330ac573dfdf7b03c724c31a24dd89106cb8db.log
2024-12-17T11:54:51.048400848Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] scan_glob add(): /var/log/containers/splunkt1-cert-init-job-x1d39-jz4j6_agents_splunkt1-cert-exporter-0bc5ee79d53d94e7dfbe329528330ac573dfdf7b03c724c31a24dd89106cb8db.log, inode 3385953
2024-12-17T11:54:51.048404848Z stderr F [2024/12/17 11:54:51] [debug] [input:tail:splunkt1-alias] 8 new files found on path '/var/log/containers/splunkt1*_splunkt1-cert-exporter-*.log'
2024-12-17T11:54:51.04845205Z stderr F [2024/12/17 11:54:51] [ info] [filter:multiline:multiline.0] created emitter: emitter_for_multiline.0
2024-12-17T11:54:51.048496551Z stderr F [2024/12/17 11:54:51] [ info] [input:emitter:emitter_for_multiline.0] initializing
2024-12-17T11:54:51.048502351Z stderr F [2024/12/17 11:54:51] [ info] [input:emitter:emitter_for_multiline.0] storage_strategy='memory' (memory only)
2024-12-17T11:54:51.048504552Z stderr F [2024/12/17 11:54:51] [debug] [emitter:emitter_for_multiline.0] created event channels: read=75 write=76

• Steps to reproduce the problem:

See configuration

Expected behavior
More log message that is info level and logs connection attempts to splunk server

Screenshots
N/A

Your Environment
see below

• Version used:
  3.2
• Configuration:

    [INPUT]
        Name              tail
        Alias             splunkt1-alias
        Tag               test-sp
        Path              /var/log/containers/splunkt1*_splunkt1-tt1-agent-*.log,/var/log/containers/splunkt1*_splunkt1-cert-exporter-*.log
        Multiline.parser  docker, cri
        Docker_Mode       true
        Docker_Mode_Flush 4
        Mem_Buf_Limit     10MB
        Skip_Long_Lines   true
        Refresh_Interval  5
        Buffer_Chunk_Size 320KB
        Buffer_Max_Size   768KB

    [OUTPUT]
        Name                  splunk
        Match                 test-sp
        Host                  $Splunk-endpoint
        Port                  $Splunk-port
        Splunk_Token          $Splunk-token
        Tls                   On
        Tls.verify            On
        Tls.debug             On
        Workers               1
        http_debug_bad_request on

• Environment name and version (e.g. Kubernetes? What version?):
  Kubernetes
• Server type and version:
• Operating System and version:
  Linux
• Filters and plugins:
  tail input plugin and splunk output plugin

[Ridhubharan]

Hi. I am using Fluent bit v3.2 and facing the same issue.

My config:

[INPUT]
    name cpu
    tag  cpu.local

[OUTPUT]
    name  splunk
    match  *
    host  127.0.0.1
    port  8088
    Splunk_Token  <token>
    tls  On
    tls.verify  Off
    Splunk_Send_Raw  On

Error from journalctl:

[output:splunk:splunk.1] Could not find hec_token in metadata