Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plugin: Detect config format and use a proper loader #9738

Merged
merged 1 commit into from
Dec 17, 2024

Conversation

cosmo0920
Copy link
Contributor

For classic mode and yaml mode of plugins_file parser should differ because classic mode of plugins_file has key-value pair which includes Path and the real path by contrast yaml mode of plugins_file just contains array under plugins section.
This should be caused by the difference between ini file and Yaml file syntax.
So, we need to detect the format of files when loading configurations for external plugins.

Closes #9729.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change

For yaml:

service:
  log_level: info
  plugins_file: plugins.yaml

pipeline:
  inputs:
    - name: random
    - name: gdummy

  outputs:
    - name: stdout
      match: '*'

plugins.yaml

plugins:
    - /path/to/in_gdummy.so
    - /path/to/out_gstdout.so

For classic mode:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Plugins_File plugins.yaml
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Hot_Reload On
    storage.metrics On

[INPUT]
    Name gdummy
    Tag  dummy.local.ins1
    Dummy {"compact": true, "schema": 1, "message": "ok"}
    # threaded on

[OUTPUT]
    Name  stdout
    Match *

plugins.conf

[PLUGINS]
  Path /path/to/in_gdummy.so
  Path /path/to/out_gstdout.so
  • Debug log output from testing the change

For yaml:

[2024/12/17 18:57:00] [ info] Configuration:
[2024/12/17 18:57:00] [ info]  flush time     | 1.000000 seconds
[2024/12/17 18:57:00] [ info]  grace          | 5 seconds
[2024/12/17 18:57:00] [ info]  daemon         | 0
[2024/12/17 18:57:00] [ info] ___________
[2024/12/17 18:57:00] [ info]  inputs:
[2024/12/17 18:57:00] [ info]      random
[2024/12/17 18:57:00] [ info]      gdummy
[2024/12/17 18:57:00] [ info] ___________
[2024/12/17 18:57:00] [ info]  filters:
[2024/12/17 18:57:00] [ info] ___________
[2024/12/17 18:57:00] [ info]  outputs:
[2024/12/17 18:57:00] [ info]      stdout.0
[2024/12/17 18:57:00] [ info] ___________
[2024/12/17 18:57:00] [ info]  collectors:
[2024/12/17 18:57:00] [ info] [fluent bit] version=3.2.3, commit=4242caa296, pid=799715
[2024/12/17 18:57:00] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/12/17 18:57:00] [debug] [input:gdummy:gdummy.1] [thread init] initialization OK
[2024/12/17 18:57:00] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/12/17 18:57:00] [ info] [output:stdout:stdout.0] worker #0 started
[2024/12/17 18:57:00] [ info] [simd    ] disabled
[2024/12/17 18:57:00] [ info] [cmetrics] version=0.9.9
[2024/12/17 18:57:00] [ info] [ctraces ] version=0.5.7
[2024/12/17 18:57:00] [ info] [input:random:random.0] initializing
[2024/12/17 18:57:00] [ info] [input:random:random.0] storage_strategy='memory' (memory only)
[2024/12/17 18:57:00] [debug] [random:random.0] created event channels: read=25 write=26
[2024/12/17 18:57:00] [debug] [input:random:random.0] interval_sec=1 interval_nsec=0
[2024/12/17 18:57:00] [ info] [input:gdummy:gdummy.1] initializing
[2024/12/17 18:57:00] [ info] [input:gdummy:gdummy.1] storage_strategy='memory' (memory only)
[2024/12/17 18:57:00] [ info] [input:gdummy:gdummy.1] thread instance initialized
[2024/12/17 18:57:00] [debug] [gdummy:gdummy.1] created event channels: read=40 write=41
[2024/12/17 18:57:00] [debug] [stdout:stdout.0] created event channels: read=44 write=45
[2024/12/17 18:57:00] [debug] [router] match rule random.0:stdout.0
[2024/12/17 18:57:00] [debug] [router] match rule gdummy.1:stdout.0
[2024/12/17 18:57:00] [ info] [sp] stream processor started
^C[2024/12/17 18:57:01] [engine] caught signal (SIGINT)
[2024/12/17 18:57:01] [debug] [task] created task=0x5f500d0 id=0 OK
[0] random.0: [[1734429421.170141674, {}], {"rand_value"=>17358370232073356975}]
[2024/12/17 18:57:01] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/12/17 18:57:01] [ warn] [engine] service will shutdown in max 5 seconds
[2024/12/17 18:57:01] [ info] [input] pausing random.0
[2024/12/17 18:57:01] [debug] [input:gdummy:gdummy.1] thread pause instance
[2024/12/17 18:57:01] [debug] [out flush] cb_destroy coro_id=0
[2024/12/17 18:57:01] [debug] [task] destroy task=0x5f500d0 (task_id=0)
2024/12/17 18:57:01 goroutine will be stopping: name="gdummy"
[2024/12/17 18:57:02] [ info] [engine] service has stopped (0 pending tasks)
[2024/12/17 18:57:02] [ info] [input] pausing random.0
[2024/12/17 18:57:02] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/12/17 18:57:02] [debug] [input:gdummy:gdummy.1] thread pause instance
[2024/12/17 18:57:02] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2024/12/17 18:57:02] [debug] [GO] running exit callback
[2024/12/17 18:57:02] [debug] [input:gdummy:gdummy.1] thread exit instance

For classic mode:

Fluent Bit v3.2.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/12/17 18:57:31] [ info] Configuration:
[2024/12/17 18:57:31] [ info]  flush time     | 5.000000 seconds
[2024/12/17 18:57:31] [ info]  grace          | 5 seconds
[2024/12/17 18:57:31] [ info]  daemon         | 0
[2024/12/17 18:57:31] [ info] ___________
[2024/12/17 18:57:31] [ info]  inputs:
[2024/12/17 18:57:31] [ info]      gdummy
[2024/12/17 18:57:31] [ info] ___________
[2024/12/17 18:57:31] [ info]  filters:
[2024/12/17 18:57:31] [ info] ___________
[2024/12/17 18:57:31] [ info]  outputs:
[2024/12/17 18:57:31] [ info]      stdout.0
[2024/12/17 18:57:31] [ info] ___________
[2024/12/17 18:57:31] [ info]  collectors:
[2024/12/17 18:57:32] [ info] [fluent bit] version=3.2.3, commit=4242caa296, pid=799807
[2024/12/17 18:57:32] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/12/17 18:57:32] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/12/17 18:57:32] [ info] [simd    ] disabled
[2024/12/17 18:57:32] [ info] [cmetrics] version=0.9.9
[2024/12/17 18:57:32] [ info] [ctraces ] version=0.5.7
[2024/12/17 18:57:32] [ info] [input:gdummy:gdummy.0] initializing
[2024/12/17 18:57:32] [ info] [input:gdummy:gdummy.0] storage_strategy='memory' (memory only)
[2024/12/17 18:57:32] [debug] [input:gdummy:gdummy.0] [thread init] initialization OK
[2024/12/17 18:57:32] [ info] [input:gdummy:gdummy.0] thread instance initialized
[2024/12/17 18:57:32] [debug] [gdummy:gdummy.0] created event channels: read=38 write=39
[2024/12/17 18:57:32] [debug] [stdout:stdout.0] created event channels: read=42 write=43
[2024/12/17 18:57:32] [ info] [output:stdout:stdout.0] worker #0 started
[2024/12/17 18:57:32] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/12/17 18:57:32] [ info] [sp] stream processor started
[2024/12/17 18:57:33] [debug] [gdummy] operation succeeded
[2024/12/17 18:57:34] [debug] [gdummy] operation succeeded
[2024/12/17 18:57:35] [debug] [gdummy] operation succeeded
[2024/12/17 18:57:36] [debug] [gdummy] operation succeeded
[2024/12/17 18:57:36] [debug] [task] created task=0x73c33403aeb0 id=0 OK
[2024/12/17 18:57:36] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] dummy.local.ins1: [[1734429453.019956279, {}], {"message"=>"dummy"}]
[1] dummy.local.ins1: [[1734429454.020030039, {}], {"message"=>"dummy"}]
[2] dummy.local.ins1: [[1734429455.020311757, {}], {"message"=>"dummy"}]
[2024/12/17 18:57:36] [debug] [out flush] cb_destroy coro_id=0
[2024/12/17 18:57:36] [debug] [task] destroy task=0x73c33403aeb0 (task_id=0)
^C[2024/12/17 18:57:36] [engine] caught signal (SIGINT)
[2024/12/17 18:57:36] [debug] [task] created task=0x73c334022150 id=0 OK
[2024/12/17 18:57:36] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/12/17 18:57:36] [ warn] [engine] service will shutdown in max 5 seconds
[0] dummy.local.ins1: [[1734429456.020360923, {}], {"message"=>"dummy"}]
[2024/12/17 18:57:36] [debug] [input:gdummy:gdummy.0] thread pause instance
[2024/12/17 18:57:36] [debug] [out flush] cb_destroy coro_id=1
[2024/12/17 18:57:36] [debug] [task] destroy task=0x73c334022150 (task_id=0)
2024/12/17 18:57:36 goroutine will be stopping: name="gdummy"
[2024/12/17 18:57:37] [ info] [engine] service has stopped (0 pending tasks)
[2024/12/17 18:57:37] [debug] [input:gdummy:gdummy.0] thread pause instance
[2024/12/17 18:57:37] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/12/17 18:57:37] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2024/12/17 18:57:37] [debug] [GO] running exit callback
[2024/12/17 18:57:37] [debug] [input:gdummy:gdummy.0] thread exit instance
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Copy link
Contributor

@patrick-stephens patrick-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some unit tests to prevent future regressions?

@cosmo0920
Copy link
Contributor Author

Can we add some unit tests to prevent future regressions?

Unfortunately, it's hard to implement unit testing for this feature...

@cosmo0920 cosmo0920 added this to the Fluent Bit v3.2.3 milestone Dec 17, 2024
@edsiper edsiper merged commit c461f3e into master Dec 17, 2024
52 checks passed
@edsiper edsiper deleted the cosmo0920-fix-loading-for-external-plugins branch December 17, 2024 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can't parse plugins.conf anymore
3 participants