Guidance needed; Fargate and Multiline logs

[f0o]

Hi,

I'm a bit at a loss how to configure firelens to allow multiline logs on Fargate.

right now it has:

          FirelensConfiguration:
            Type: fluentbit
            Options:
              enable-ecs-log-metadata: "true"

And the Work-Container:

          LogConfiguration:
            LogDriver: awsfirelens
            Options:
                Name: forward
                Host: fluentd.domain.tld
                Port: "24225"

So... Where to go from here? I'd like to avoid having to build my own image with custom config as would add maintenance costs.

[PettitWesley]

Try this: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/filter-multiline

Unfortunately on Fargate you still will need a custom image with a custom config file. I know it's not ideal. We are working on fixing that long term.

[f0o]

@PettitWesley It turns out that the issue wasnt Multiline log related but rather Maximum Log Length...
I was mistaken by the sample, it cut seemingly coincidental at a newline.

So I guess what it boils down to now (unfortunately unrelated to the issue) is what is the maximum line length of Firelens on Fargate?

Before we migrated to Fargate we used fluentbit on ECS/EC2 and didnt experience this issue so I dont believe the issue is related to fluentbit itself, or am I mistaken?

[PettitWesley]

@f0o FireLens uses the Fluentd Docker log driver: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

And it always splits logs at 16KB. So if you have logs emitted by the container that are longer than that, they will be truncated at 16KB and then the rest of the content will appear in a new log line.

Is this the case for your logs?

I think it might be possible to use the multiline feature to re-combine those logs but I have not tried that yet myself and so do not have a working example to reference.

[c-ameron]

Unfortunately on Fargate you still will need a custom image with a custom config file. I know it's not ideal. We are working on fixing that long term.

Jumping onto this - having a way to set the DEBUG level via an environment variable would also be pretty neat :D

[PettitWesley]

@c-ameron That is possible! It was actually the first feature I ever contributed to fluent bit core back in 2019, exactly for this use case. Set the env var FLB_LOG_LEVEL=debug

[c-ameron]

That's amazing! thanks so much!!! :D

[f0o]

@PettitWesley yeah I think we're hitting that 16KB Limit...

I think it might be possible to use the multiline feature to re-combine those logs but I have not tried that yet myself and so do not have a working example to reference.

I guess I'll be going onto an adventure now! Got no idea how to do this either, let's hope I can catch somebody on IRC haha

[PettitWesley]

@f0o Please use rubular to test your regex's: https://rubular.com/

Also, please do share any findings here for everyone's benefit.

[f0o]

@PettitWesley I've bit my teeth out trying to get concat with CRI logtag && Docker partials to run...

Have to check how the logs actually arrive on the fluentbit image, I got a suspicion that my attempts fail due to missing partial markers from the source.

Matching JSON (I guess?) with regex is going to be a nightmare haha - But I'll keep you posted

[f0o]

Moving thread due to duplicate of #100 & #25