Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tail fluent-bit can not collect new logs from django log files #9758

Open
prd-tri-tran opened this issue Dec 23, 2024 · 0 comments
Open

Tail fluent-bit can not collect new logs from django log files #9758

prd-tri-tran opened this issue Dec 23, 2024 · 0 comments

Comments

@prd-tri-tran
Copy link

prd-tri-tran commented Dec 23, 2024

Bug Report

Describe the bug

This is my config

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf
    Plugins_File plugins.conf
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
[INPUT]
    name              tail
    path              /fluent-bit/logs/siem-data.log
    tag               test_stream
    # Read_from_head    true
    Parser            json
    Refresh_Interval  5
    Buffer_Max_Size   5MB
    Buffer_Chunk_Size 400k
[OUTPUT]
    Name            kinesis_firehose
    Match           test_stream
    region          us-west-2
    delivery_stream my-localstack-firehose-stream

I using the Logger (Logging package for Python) in Django for producing logs into log file, using fluent-bit collect that log files and send it to firehose

Problem:

[2024/12/23 04:21:07] [debug] [input:tail:tail.0] scanning path /fluent-bit/logs/siem-data.log
[2024/12/23 04:21:07] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /fluent-bit/logs/siem-data.log, inode 13
[2024/12/23 04:21:07] [debug] [input:tail:tail.0] 0 new files found on path '/fluent-bit/logs/siem-data.log'
[2024/12/23 04:21:12] [debug] [input:tail:tail.0] scanning path /fluent-bit/logs/siem-data.log
[2024/12/23 04:21:12] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /fluent-bit/logs/siem-data.log, inode 13
[2024/12/23 04:21:12] [debug] [input:tail:tail.0] 0 new files found on path '/fluent-bit/logs/siem-data.log'
[2024/12/23 04:21:17] [debug] [input:tail:tail.0] scanning path /fluent-bit/logs/siem-data.log
[2024/12/23 04:21:17] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /fluent-bit/logs/siem-data.log, inode 13
[2024/12/23 04:21:17] [debug] [input:tail:tail.0] 0 new files found on path '/fluent-bit/logs/siem-data.log'
[2024/12/23 04:21:22] [debug] [input:tail:tail.0] scanning path /fluent-bit/logs/siem-data.log
[2024/12/23 04:21:22] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /fluent-bit/logs/siem-data.log, inode 13
[2024/12/23 04:21:22] [debug] [input:tail:tail.0] 0 new files found on path '/fluent-bit/logs/siem-data.log'
[2024/12/23 04:21:27] [debug] [input:tail:tail.0] scanning path /fluent-bit/logs/siem-data.log

Fluent bit can not detect the new log from log file

Some Additional information:
I try to manually paste log into the log files with editor (eg: VSCode) and it still worked
Do this problem relate to the offset make the Fluent bit can not detect the new logs

Could you help me to solve this problem and please help me explain how it is triggered
Thanks a lot guys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant