Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parser: support subsecond resolution with colon (%s:%L) #2210

Closed
tofele opened this issue May 29, 2020 · 5 comments · Fixed by #2722
Closed

parser: support subsecond resolution with colon (%s:%L) #2210

tofele opened this issue May 29, 2020 · 5 comments · Fixed by #2722
Assignees

Comments

@tofele
Copy link

tofele commented May 29, 2020

Bug Report

Describe the bug

I have a 3rd party application (Forgerock OpenAM) that writes timestamps as follows in some log files:

amAuthInternal:05/29/2020 11:58:21:127 AM UTC: Thread[localhost-startStop-1,5,main]: TransactionId[3da35379-317e-41fe-b9f2-0cae565a8480-0] AuthContext::getLoginStatus()

Note that it uses a colon to separate subseconds, hence of the format "%S:%L". As per #703, only "%S.%L" (dot) and "%S,%L" (comma) are currently supported by fluent-bit.

To Reproduce

Consider log entries as follows:

amAuthInternal:05/29/2020 11:58:21:127 AM UTC: Thread[localhost-startStop-1,5,main]: TransactionId[3da35379-317e-41fe-b9f2-0cae565a8480-0] AuthContext::getLoginStatus()
amAuthInternal:05/29/2020 11:58:21:128 AM UTC: Thread[localhost-startStop-1,5,main]: TransactionId[3da35379-317e-41fe-b9f2-0cae565a8480-0] AuthContext::getSubject()
amAuthInternal:05/29/2020 11:58:21:128 AM UTC: Thread[localhost-startStop-1,5,main]: TransactionId[3da35379-317e-41fe-b9f2-0cae565a8480-0] AuthContext::getAuthPrincipal(): [AuthPrincipal:  cn=dsameuser,ou=DSAME Users,ou=am-config]

Parser config:

    [PARSER]
        Name              am-debug
        Format            regex
        Regex             ^(?<module>[A-Za-z0-9]*):(?<timestamp>\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}:\d{3} [^\ ]* [^:]*): Thread\[(?<thread>[^\]]*)\]: TransactionId\[(?<transactionId>[^\]]*)\] (?<message>.*)$
        Time_Key          timestamp
        Time_Format       %m/%d/%Y %I:%M:%S:%L %p %Z
        Time_Keep         On

The result:

[2020/05/29 12:10:37] [ warn] [parser:am-debug] Invalid time format %m/%d/%Y %I:%M:%S:%L %p %Z for '05/29/2020 12:10:37:601 PM UTC'.

Expected behavior
It should be possible to have any separator between %S and %L. This should be defined via Time_Format.

Your Environment

  • Version used: 1.4.5
  • Filters and plugins: tail
@edsiper
Copy link
Member

edsiper commented Jun 2, 2020

I think we should go with the 'colon' support only and not make it configurable (just trying to avoid a config for a simple use case).

@fujimotos can you take care of this ?

@fujimotos fujimotos self-assigned this Jun 2, 2020
@fujimotos
Copy link
Member

can you take care of this ?

@edsiper All right, I assigned this issue to myself.

@thisisabhishek
Copy link

thisisabhishek commented Aug 31, 2020

@fujimotos @edsiper Don't you guys think that time in millis should also be supported by default.

We get the following logs with Log4j2 JSON layout:

{"timeMillis":1598609645725,"thread":"main","level":"WARN","loggerName":"com.abhishek.test.AppHelper","message":"Everything is working fine.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":1,"threadPriority":5}

Will having just %s or %L work for this?

@fujimotos
Copy link
Member

@tofele Just for your interest, I submitted #2722 that improves
the sub-second handling. It should cover your case as well.

@fujimotos
Copy link
Member

@thisisabhishek Can you submit a new issue for it?

My understanding is that you want Fluent Bit to be aware of "milliseconds
since epoch" (a common format in JS world).

I assume this support is possible and useful, but I think it is a different
topic than the orignal post.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants