Multiline parser: last item is not parsed

[vhadianto]

I'm using the multiline parser to parse Postgres CSV logs. It works well, except I'm struggling to get the last item in the log files to be picked up.

Following the docs here: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing

I'm configuring my multiline parser like this:

Rule      "start_state"   "/(\d{4}-\d{2}-\d{2}[\s|T]\d{2}.\d{2}.\d{2}.\d{3}\s[A-Z]{0,4}),(.*)/"    "cont"
Rule      "cont"          "^((?!\d{4}-\d{2}-\d{2}).)*$"                                            "cont"

It works well, but the last item in the log files will not get picked up until the entry. I assume because fluent-bit has not discovered the next "start" entry thus has not "closed" the current item.

There's no documentation on how to "end" the parsing. Is this possible? In our configuration, we print out the database name at the end of each CSV log entry. So in theory I could try to do something like this:

Rule      "start_state"   "/(\d{4}-\d{2}-\d{2}[\s|T]\d{2}.\d{2}.\d{2}.\d{3}\s[A-Z]{0,4}),(.*)/"    "cont"
Rule      "cont"          "^((?!\d{4}-\d{2}-\d{2}).)*$"                                            "end_state"
Rule     "end_state"   "\"dbname\""                                                                "start_state"

I tried the above, while it's not broken anything it also does not solve my issue.

Anyone can help?

[mickeypash]

Hey Victor!

Could you provide a sample log line or maybe even a Rubular link?
It's been a while since you posted so maybe if you solved it you can post your findings here 👍

[lyoshakarpenko]

Rule      "start_state"   "/(\d{4}-\d{2}-\d{2}[\s|T]\d{2}.\d{2}.\d{2}.\d{3}\s[A-Z]{0,4}),(.*)/"    "cont"
Rule      "cont"          "^((?!\d{4}-\d{2}-\d{2}).)*$"                                            "cont", "end_state"
Rule     "end_state"   "\"dbname\""                                                                "start_state"

It seems to me that the main question is how to make OR in the parser. Maybe something like that "cont", "end_state". A more complex example has not been specified anywhere.