Offline mode | Buffering strategy

[mickeypash]

Hey team,

As discussed in the recent community meeting. In my team we have a use case where we have devices/endpoints/unix hosts running on the customer's premises. They can be operational but without access to the internet for long periods of time (i.e. weeks or somethings months).

FluentBit supports on filesystem buffering however there are some short comings:

• If I'm tailing file-a.log and file-b.log and simply storing them in another file (FLB storage) - this feels redundant
• FLB uses exponential backoff but what happens when the device is offline for say 2 weeks?

I'd say there are generally 2 views:

• FluentBit came out of the embedded space so there might be or should be a way to natively support "offline mode" for embedded and IoT devices
• This should be a consideration of the embedded team working on the device

Talking points:

• @PettitWesley described it as "freezing" the FluentBit process till internet connectivity is established again.
• If I'm not mistaken FLB stores a DB with the offsets for each file it tracks, how could we use that to our advantage?

[PettitWesley]

I either didn't ask or forgot... does your use case involve collecting logs during the period without network? (Or actually even if that isn't your use case, should we be consider it when evaluating solutions?)

[mickeypash]

does your use case involve collecting logs during the period without network?

Yes our use-case involves exactly that.
(The application is running and logging while there is no network. If the app fails we need the logs to diagnose the problem)

should we be consider it when evaluating solutions

I would say so but I think we should see if there are other people in the community with this use-case.
(The solution right now is to zip journald logs send send them to S3. It's done on a systemd timer or via the user)

[JeffLuoo]

If I'm tailing file-a.log and file-b.log and simply storing them in another file (FLB storage) - this feels redundant

Fluent-bit buffers logs in either memory or disk. If your goal is to let fluent-bit collect logs when the device is offline, the data collected should somewhere be stored in the device itself. But I believe the *.flb chunk is already compressed to ~2MB which stores more data than the raw log file. From my best knowledge, the current solution is to configure unlimited retry in fluent-bit and filesystem buffering with a total limit cap. Once the buffered data exceeds the limit, fluent-bit will try to drop the oldest (but available to drop) chunks from the disk.

FLB uses exponential backoff but what happens when the device is offline for say 2 weeks?

iirc the cap is set to 2000s in the recent release (can't remember the commit...)

[mickeypash]

Thanks Jeff! This is all correct based on my research after asking this question. If I remember I will try to summarise my findings here.