fluent-bit 3.0.3 upgrade has broken splunk output plugin when event_sourcetype_key is specified

[mannbiher]

Bug Report

Describe the bug
fluent-bit 3.0.3 upgrade has broken Splunk output plugin when event_sourcetype_key attribute is specified. It sends a wrong Authorization header to Splunk causing it to respond with an HTTP 401 error. It now sends Authorization: hec_token which should be Authorization: Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4 where db496524-e7e6-4ae9-b3f0-2287d8e65cd4 is a HEC token.

[2024/05/01 23:11:43] [ warn] [output:splunk:splunk.0] http_status=401:
{"text":"Invalid authorization","code":3}

To Reproduce

• Steps to reproduce the problem:
  On Linux

1. Create sample log.

cat <<EOF > sample.log
log line 1
log line 2
log line 3
EOF

2. Run nc in a different terminal as dummy Splunk server.

echo -e 'HTTP/1.1 200 OK\r\n' | nc -l -p 8081

3. Start fluent-bit 3.0.3 and wait for [http_client] broken connection to 127.0.0.1:8081 ? error. As we are using nc the broken connection is expected.

docker run -ti -v $PWD/sample.log:/sample.log \
--network="host" fluent/fluent-bit:3.0.3 /fluent-bit/bin/fluent-bit \
-i tail -p path=/sample.log -p read_from_head=true \
-o splunk -p port=8081 -psplunk_token=db496524-e7e6-4ae9-b3f0-2287d8e65cd4 -p 'event_sourcetype_key=sourcetype' \
-f 1

4. View the request sent to nc.

POST /services/collector/event HTTP/1.1
Host: 127.0.0.1:8081
Content-Length: 242
User-Agent: Fluent-Bit
Authorization: hec_token
Connection: keep-alive

{"time":1714604689.81654,"sourcetype":"sourcetype","event":{"log":"log line 1"}}{"time":1714604689.816549,"sourcetype":"sourcetype","event":{"log":"log line 2"}}{"time":1714604689.816551,"sourcetype":"sourcetype","event":{"log":"log line 3"}}

Expected behavior

Running the same steps with fluent-bit 3.0.2 docker image sends correct authorization header.

docker run -ti -v $PWD/sample.log:/sample.log \
--network="host" fluent/fluent-bit:3.0.2 /fluent-bit/bin/fluent-bit \
-i tail -p path=/sample.log -p read_from_head=true \
-o splunk -p port=8081 -psplunk_token=db496524-e7e6-4ae9-b3f0-2287d8e65cd4 -p 'event_sourcetype_key=sourcetype' \
-f 1

POST /services/collector/event HTTP/1.1
Host: 127.0.0.1:8081
Content-Length: 243
User-Agent: Fluent-Bit
Authorization: Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4
Connection: keep-alive

{"time":1714604726.505695,"sourcetype":"sourcetype","event":{"log":"log line 1"}}{"time":1714604726.505703,"sourcetype":"sourcetype","event":{"log":"log line 2"}}{"time":1714604726.505704,"sourcetype":"sourcetype","event":{"log":"log line 3"}}

Your Environment

• Version used: 3.0.3
• Configuration:

[INPUT]
    Name             tail
    Path             /sample.log
    Read_From_Head   On
[OUTPUT]
    Name                 splunk
    Match                *
    Port                 8081
    Splunk_Token         db496524-e7e6-4ae9-b3f0-2287d8e65cd4
    event_sourcetype_key sourcetype

• Environment name and version (e.g. Kubernetes? What version?): EKS 1.27
• Server type and version: NA
• Operating System and version: Amazon Linux 2023
• Filters and plugins: tail, splunk

[nirnanaaa]

Facing a very similar issue here. After some digging I found #8738 which might have caused this issue. Although I am not sure how to use the newly introduced event format correctly.

[mannbiher]

Hi @nirnanaaa I have created a PR to fix the issue. Would you be able to test the PR for your scenario. All inputs are welcome.

[neugeeug]

We have the same issue, Splunk integration stopped working since 3.0.3

[agup006]

@cosmo0920 fyi looks like an edge case on hec token