parser java exception does not work

[ZMMWMY]

i try to parser java exception on k8s platform, but it does not work.
that is my configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-forward.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Multiline         On
        Multiline_Flush   5
        Parser_Firstline  parse1
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        K8S-Logging.Parser  On
        K8s-Logging.Exclude Off
  output-forward.conf: |
    [OUTPUT]
        Name            forward
        Match           *
        Host            ip
        Port            23343
  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        Decode_Field_As   escaped_utf8    log
    [PARSER]
        Name        first_line
        Format      regex
        Time_Key    time
        Regex       (?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}.\d{1,3})
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep   On
        Decode_Field_As   escaped_utf8    log
    [PARSER]
        Name        parse1
        Format      regex
        Time_Key    time
        Regex       ^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}.\d{1,3}) (?<level>[^\s]+) \d{1,2} --- \[(?<thread>.*)\] (?<message>.*)
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep   On
        Decode_Field_As   escaped_utf8    log

the java log

2019-06-24 05:46:11.283  INFO 1 --- [           main] c.a.schedulerx.worker.SchedulerxWorker   : set applicationContext to SpringContext
2019-06-24 05:46:11.284  INFO 1 --- [           main] c.a.schedulerx.worker.SchedulerxWorker   : initializing bean...
2019-06-24 05:46:11.284  INFO 1 --- [           main] c.a.schedulerx.worker.SchedulerxWorker   : Schedulerx Worker starting...
2019-06-24 05:46:16.720 ERROR 1 --- [           main] c.a.schedulerx.worker.SchedulerxWorker   : get local host error

java.net.UnknownHostException: schedulerx-cn-hangzhou-inner.console.aliyun.com
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_212]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_212]
        at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_212]
        at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_212]

[apollo13]

@ZMMWMY Did you ever find a solution here? This is easy in fluentd https://docs.fluentd.org/parser/multiline but apparently not so much in fluent-bit

[apollo13]

I seem to have managed to get the wanted behaviour by just specifying Parser_Firstline, all following lines will get concatenated to the previous line then

[floriankoch]

@apollo13 can you share your config?

[apollo13]

@floriankoch I currently have the following (some things obfuscated):

[INPUT]
    Name tail
    Path ...
    # other common options
    Multiline On
    Parser_Firstline audit-firstline

[PARSER]
    Name        audit-firstline
    Format      regex
    Regex       ^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?<log>.+)$
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S
    Time_Keep   On

The regex ensures that every line starting with a timestamp becomes a new log entry. All other lines are concatenated to the "log" key.

[floriankoch]

@apollo13 thx, trying to solve the same Problem atm

[apollo13]

@floriankoch Let me know if you find any improvements to my approach

[floriankoch]

@apollo13 the regex is to specific, i have test exceptions (jboss and springboot) and they does not match
I think this is a good source for the regex https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions/blob/master/lib/fluent/plugin/exception_detector.rb

and here it can be tested https://regex101.com/

[apollo13]

Oh, I am not using it to parse exceptions, just multiline log messages

[floriankoch]

@apollo13 ah ok - then the regex is better, but all logs need to start with the same date format

[apollo13]

Yes, they do in my case

[yuzeshan]

Yes, they do in my case

Hi, have you ever encountered this problem： #2013
I configure multiline parser on k8s to collect java multiple line logs, and that can collect them successfully, but output following unexpected format

[avinashashukla]

I was able to resolve Java stack trace using multi line setting as follows:

Setup:
Java Docker App in Kubernetes -> fluent bit (1.4.1-Debug) -> ES -> Kibana

Fluent bit contianer logs (/var/log/containers):
{"log":"ERROR 2020-04-08 11:59:35,029 [main] org.springframework.boot.SpringApplication Application run failed\n","stream":"stdout","time":"2020-04-08T11:59:35.0327 56254Z"} {"log":"org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server\n","stream":"stdout","time":"2020-04-08T11:59:35.032776291Z"} {"log":"\u0009at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:215)\n","stream":"stdout","time":"2020-04-08T11:59:35.032783733Z"} {"log":"\u0009at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.startWebServer(ServletWebServerApplicationContext.java:297)\n","stream":"stdout"," time":"2020-04-08T11:59:35.032789816Z"} {"log":"\u0009at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:163)\n","stream":"stdout","t ime":"2020-04-08T11:59:35.032795714Z"} {"log":"\u0009at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)\n","stream":"stdout","time":"2020-04-08T11:59:35.03280 1469Z"} {"log":"INFO 2020-04-08 11:59:35,039 [main] org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor Shutting down ExecutorService 'applicationTaskExecutor '\n","stream":"stdout","time":"2020-04-08T11:59:35.040609512Z"}

fluent-bit.conf:
`
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File custom-parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020

[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
DB /var/log/flb_kube.db
DB.Sync Normal
Mem_Buf_Limit 100MB
Skip_Long_Lines On
Refresh_Interval 10
Multiline On
Multiline_Flush 4
Parser_Firstline first_line

[FILTER]
Name kubernetes
Match kube.*
Kube_Tag_Prefix kube.var.log.containers.
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Labels Off
Annotations Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
Merge_Log Off
Merge_Log_Key log_processed

[OUTPUT]
Name es
Match *
Host elastic
Port 9200
Logstash_Format On
Logstash_Prefix app
Replace_Dots On
Retry_Limit False

custom-parsers.conf
[PARSER]
Name first_line
Format regex
Regex ^(?.[INFO|WARN|ERROR]) (?.)$
Time_Key time
Time_Format %Y-%m-%d %H:%M:%S
Time_Keep On
`
Notes:

1. Regular expression is the key. The first line it matches becomes a new line otherwise the text get concatenated to the first key of your regex. In my case "log". Write a regular expression that would match start of both {"log":" and without. Example it should match: {"log":"INFO 2020-04-08 11:59:35,039 and INFO 2020-04-08 11:59:35,039
2. When you are viewing in Kibana, it may look like not all the lines are concatenated. So expand the Kibana entry and check the message and it should have a complete stack trace.

[Moulick]

@avinashashukla shouldn't the regex be Regex ^(?<something>.[INFO|WARN|ERROR]) (?<something-else>.)$ ? like we need to name the capture group