plugins: exec: exit fluent-bit after oneshot and propagate exit code

[ringerc]

Enable use of fluent-bit as a log-processing command wrapper by enhancing the exec plugin with options to exit on child process exit and propagate exit code.

Adds new options to in_exec plugin:

Exit_After_Oneshot [bool]
Propagate_Exit_Code [bool]

Exit_After_Oneshot implies Oneshot and causes fluent-bit to terminate when the command run by the exec plugin terminates.

Propagate_Exit_Code causes fluent-bit to exit with the same exit code as the on-shot child process it invoked, following the shell convention for handling of signal exits.

$ ./bin/fluent-bit -q -o stdout -i exec -p propagate_exit_code=true -p exit_after_oneshot=true -p command='echo some command output; exit 2' 
[0] exec.0: [[1681771813.322498607, {}], {"exec"=>"some command output"}]
$ echo $?
2

Obviously practical uses will really be parsing and transforming the logs, shipping logs to external capture services, etc.

While exiting when the one-shot command exits would probably be a sensible default, that might surprise any existing users of the current one-shot option so it is not made the default. That way this change remains backwards compatible with existing oneshot uses.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Docs PR: fluent/fluent-bit-docs#1080

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[ringerc]

Config samples; these all expect a false.sh to exist with this content:

#!/bin/bash
echo "some output here"
exit 2

and chmod +x false.sh

fluent-bit-exec-conf.txt

$ ./bin/fluent-bit -c fluent-bit-exec-conf.txt            
Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/04/17 16:40:05] [ info] [fluent bit] version=2.1.0, commit=9efd3532a8, pid=2061089
[2023/04/17 16:40:05] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/04/17 16:40:05] [ info] [cmetrics] version=0.6.1
[2023/04/17 16:40:05] [ info] [ctraces ] version=0.3.0
[2023/04/17 16:40:05] [ info] [input:exec:exec.0] initializing
[2023/04/17 16:40:05] [ info] [input:exec:exec.0] storage_strategy='memory' (memory only)
[2023/04/17 16:40:05] [ info] [sp] stream processor started
[2023/04/17 16:40:05] [ info] [output:stdout:stdout.0] worker #0 started
[0] stdin: [[1681706405.178401054, {}], {"exec"=>"some output here"}]
^C[2023/04/17 16:38:36] [engine] caught signal (SIGINT)

Runs until signalled. Exiting behaviour.

fluent-bit-exec-and-exit-conf.txt

$ ./bin/fluent-bit -c fluent-bit-exec-and-exit-conf.txt 
Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/04/17 16:40:17] [ info] [fluent bit] version=2.1.0, commit=9efd3532a8, pid=2061175
[2023/04/17 16:40:17] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/04/17 16:40:17] [ info] [cmetrics] version=0.6.1
[2023/04/17 16:40:17] [ info] [ctraces ] version=0.3.0
[2023/04/17 16:40:17] [ info] [input:exec:exec.0] initializing
[2023/04/17 16:40:17] [ info] [input:exec:exec.0] storage_strategy='memory' (memory only)
[2023/04/17 16:40:17] [ info] [sp] stream processor started
[2023/04/17 16:40:17] [ info] [output:stdout:stdout.0] worker #0 started
[2023/04/17 16:40:17] [ warn] [engine] service will shutdown in max 5 seconds
[0] stdin: [[1681706417.285810237, {}], {"exec"=>"some output here"}]
[2023/04/17 16:40:18] [ info] [engine] service has stopped (0 pending tasks)
[2023/04/17 16:40:18] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/04/17 16:40:18] [ info] [output:stdout:stdout.0] thread worker #0 stopped
$ echo $?                                              
0

Exits once command exits. Even though command exited with nonzero, fluent-bit exits with zero.

fluent-bit-exec-and-exit-propagate-conf.txt

./bin/fluent-bit -c fluent-bit-exec-and-exit-propagate-conf.txt
Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/04/17 16:43:53] [ info] [fluent bit] version=2.1.0, commit=9efd3532a8, pid=2062027
[2023/04/17 16:43:53] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/04/17 16:43:53] [ info] [cmetrics] version=0.6.1
[2023/04/17 16:43:53] [ info] [ctraces ] version=0.3.0
[2023/04/17 16:43:53] [ info] [input:exec:exec.0] initializing
[2023/04/17 16:43:53] [ info] [input:exec:exec.0] storage_strategy='memory' (memory only)
[2023/04/17 16:43:53] [ info] [sp] stream processor started
[2023/04/17 16:43:53] [ info] [output:stdout:stdout.0] worker #0 started
sh: 1: false.sh: not found
[2023/04/17 16:43:53] [ warn] [engine] service will shutdown in max 5 seconds
[2023/04/17 16:43:54] [ info] [engine] service has stopped (0 pending tasks)
[2023/04/17 16:43:54] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/04/17 16:43:54] [ info] [output:stdout:stdout.0] thread worker #7085 
$ echo $?
127

Exits when command exits, propagates child process exit code.

[ringerc]

Updated PR to improve signal handling logic and related comments. Updated the docs PR to better discuss exit codes.

I can't see any existing tests for the exec plugin. I'd be happy to extend tests to cover the new features but I don't know how to write a fluent-bit plugin test from scratch.

[ringerc]

I went to run this with valgrind but I see a lot of valgrind errors on an unpatched fluent-bit using the valgrind.supp shipped with fluent-bit, so I'm not sure how useful it'll be. It is a very unclean run with a lot of noise within fluent-bit itself - sufficiently so that I couldn't write a sensible suppressions file in a reasonable time that wouldn't land up hiding anything within the plugin invocation as well.

Built on Ubuntu 22.10 with gcc 12.2.0 using default cmake options:

➜  fluent-bit git:(master) ✗ valgrind -s --gen-suppressions=all --leak-check=full --suppressions=./valgrind.supp --suppressions=/usr/lib/valgrind/debian.supp --suppressions=/usr/libexec/valgrind/default.supp --max-stackframe=28681792 ./bin/fluent-bit -q -o stdout -i exec -p oneshot=true -p command='echo some command output; exit 2' >& valgrind.out

➜  fluent-bit git:(master) ✗ wc -l valgrind.out 
11186 valgrind.out
➜  fluent-bit git:(master) ✗ tail -1 valgrind.out 
==2117629== ERROR SUMMARY: 14623 errors from 543 contexts (suppressed: 0 from 0)

I used the Perl script in https://wiki.wxwidgets.org/Parse_valgrind_suppressions.sh to generate a suppressions file; obviously hand-writing would be better, but it was just too huge. With a number of iterations I ended up with a suppressions file that matched for the simple case I ran, but since some of the suppressions are calls with variable paths into libraries, I found that the tiniest change in invocation led to new and unrelated valgrind errors.

Unless there's a current suppressions file somewhere, or some build/runtime options that encourage fluent-bit to run more valgrind-clean, I can't give you any useful valgrind output. All I can really do is attach the suppressions file that matches all the valgrind errors I see reported, which I will include here.

valgrind_supp.txt

Of these suppressions, the only ones that mention the in_exec plugin are calls in __pthread_disable_asynccancel or __pthread_enable_asynccancel, which I see widespread throughout runs on main too. They don't look relevant, and appear intermittently.

I'll try to attach a full valgrind run log anyway, but the noise is so great I don't expect it'll be much use.

valgrind_out.txt

[ringerc]

fluent-bit-exec-and-exit-propagate-debuglog.txt

Debug logs with the propagate flag set, run as

$ ./bin/fluent-bit -c fluent-bit-exec-and-exit-propagate-conf.txt -vvv >& fluent-bit-exec-and-exit-propagate-debuglog.txt

[ringerc]

What're the next steps here?

I'm currently working around this with a really ugly hack using the stdin plugin, fluent-bit running as a co-process with file descriptor redirection, etc.

[ringerc]

Is this project active? Is there anything else I need to do in order to get a PR checked?

[leonardo-albertovich]

Enabling the exec input plugin is something we planned on doing soon but I noticed the macros to check the process status might not be available yet something easy to overcome. Would you mind taking a look at that? I'd really appreciate it.

[leonardo-albertovich]

Btw, sorry about the lack of interaction, for some reason I didn't see this PR before.

[ringerc]

Thanks @leonardo-albertovich for taking a look.

but I noticed the macros to check the process status might not be available yet

I'm not sure what you mean by this. Do you mean the macros WIFEXITED, WEXITSTATUS, WIFSIGNALED and WTERMSIG ? They're provided by sys/wait.h and are specified by POSIX.1-2001 and have been around absolutely forever. See https://support.sas.com/documentation/onlinedoc/ccompiler/doc750/html/lr2/z2005123.htm .

Are you asking for a CMake feature test for these macros to be added? I could probably do that if required, though it's kind of implied by building on a posix-alike platform.

Is your concern re MS Windows support? I didn't find a lot of info about Windows in the fluent-bit docs, but cmake/windows-setup.cmake lists the FLB_IN_EXEC plugin as not supported on Windows anyway. It would probably require major changes to work on Windows, at least reliably. Windows process management is a whole different beast. Though the lack of argument-vector style execution in this plugin makes it a little easier - windows doesn't support anything like execv(...) (and no, spawnv(...) doesn't do what you probably hoped, its argument handling is dangerously stupid). You can use _popen if you compile fluentbit as a console application which you probably do - but IIRC there are some "interesting" warts around child process lifecycle mangement, blocking, signal handling, etc, and it lands up being generally better to use the native win32 APIs. You pretty much land up with two plugins, one for windows, one for not-windows, that happen to share some common code.

---

I took a look at the failing test runs, but there's no sign they're relevant to this change as opposed to being unstable tests:

• https://github.com/fluent/fluent-bit/actions/runs/4726708830/jobs/8474537865?pr=7207 failed for flb-rt-in_event_test and flb-rt-out_tcp which are not touched in any way in this PR.
• https://github.com/fluent/fluent-bit/actions/runs/4726708830/jobs/8474537996?pr=7207 failed for flb-rt-core-timeout, flb-rt-in_event_test and flb-rt-out_tcp and I don't see how those could be related to this PR's changes
• https://github.com/fluent/fluent-bit/actions/runs/4726708830/jobs/8474538119?pr=7207 failed for flb-rt-core_chunk_trace, flb-rt-in_random and flb-rt-out_tcp. Same deal.

All failures are on OSX. Looks like issues with OSX tests rather than this PR.

So I'm not quite sure what you're asking me to do.

[leonardo-albertovich]

What I was saying was that those macros are not present in windows and since the plugin already uses popen we were thinking of enabling it in windows even if that meant making it experimental and adding a big red label.

I understand the limitations and caveats that come with windows and a "universal" approach and at some point we might want to abstract the process management and console redirection in platform specific implementations shared by the same plugin but that's not what we had in mind for the moment.

So what I'm asking you to do is to make sure this feature doesn't make it harder for us to enable the plugin in windows and the two options are :

1. Read about those macros and how they are handled in windows by others (I found a thread in an old postgresql mailing list about it) and proceed accordingly (probably defining them locally for windows)
2. Wrap lines 183-213 in an preprocessor conditional and just set flb_exit_code to zero in windows which considering the fact that the plugin used to completely ignore the result wouldn't be a huge deviation (from a preexisting bad behavior I guess)

[dkmorgan]

I'm currently working around this with a really ugly hack using the stdin plugin, fluent-bit running as a co-process with file descriptor redirection, etc.

Hi @ringerc,
I would like to use your workaround until the PR gets merged.
Can you possibly share how you actually implemented your hack?

[ringerc]

@leonardo-albertovich OK, that makes sense, so it is about windows. I'll take a look.

[ringerc]

@dkmorgan something like

#!/usr/bin/env python3

import sys
import subprocess
import logging
import shlex

logger = logging.getLogger('main')

def main():
    "run child proc and redirect output via fluent-bit"

    child_cmd = sys.argv[1:]

    fluent_bit = subprocess.Popen(
            ["/opt/fluent-bit/bin/fluent-bit", "--config", "fluent-bit.conf", "--parser", "parsers.conf"],
            stdin=subprocess.PIPE)

    fluent_bit.stdin.write(b"Starting process " + shlex.join(child_cmd).encode(sys.getdefaultencoding()) + b"\n")
    fluent_bit.stdin.flush()

    try:
        child_proc = subprocess.Popen(child_cmd, stdout=fluent_bit.stdin, stderr=fluent_bit.stdin)
        exitcode = child_proc.wait()
        fluent_bit.stdin.write(f"Child process exited with {exitcode} \n".encode(sys.getdefaultencoding()))
        fluent_bit.stdin.flush()
    except Exception as ex:
        fluent_bit.stdin.write(b"Failed to invoke child process: " + str(ex).encode(sys.getdefaultencoding()) + b"\n")
        fluent_bit.stdin.flush()
        exitcode=1

    try:
        fluent_bit.stdin.close()
    except Exception:
        # Swallow so we can continue to wait for fluent-bit to exit. There's a timeout, so
        # we don't rely on stdin having closed successfully to avoid hanging forever.
        logger.warn("Exception while closing fluent-bit stdin")

    logger.info('waiting for fluentbit')
    try:
        fluent_bit.wait(timeout=60)
    except subprocess.TimeoutExpired:
        logger.info("Timed out waiting for fluentbit to exit")
        try:
            fluent_bit.terminate()
        except Exception:
            logger.warn("Exception while terminating fluent-bit")

    # bubble up the exit code from the wrapped command
    logger.info('fluentbit exited: %s', fluent_bit.returncode)
    exitcode=child_proc.returncode
    if exitcode < 0:
        # translate python subprocess negative signal exits to shell-convention
        # 128+N signal exits; see https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html
        exitcode = 128-child_proc.returncode
    sys.exit(exitcode)

if __name__ == '__main__':
    logging.basicConfig(level='INFO')
    main()

with fluent-bit.conf (obviously you'll use a different one, this is just the bare minimum):

[INPUT]
    # Override the stdin plugin's default json parsing by reading stdin line-by-line as raw strings instead of default (which is json)
    Name    stdin
    Tag     fluentbit_stdin
    Parser  stringify_message

[OUTPUT]
    Name   stdout
    Match  *

and parsers.conf

[PARSER]
# We initially read stdin as a single message with key "message".
# If this parser isn't configured, the stdin plugin will try to read the input as json and fail.
    name        stringify_message
    format      regex
    Key_Name    message
    regex       ^(?<message>.*)

I'm sure you can see why I bothered with the exec plugin changes.

Note that this approach does nothing special for signal handling. A SIGTERM is usually delivered to the whole process tree. AFAICS by default fluent-bit ignores the first SIGTERM it receives, so this tends to work ok. The script should possibly trap SIGTERM and ensure that it still waits for fluent-bit to exit, but I have not bothered fiddling with that.

[leonardo-albertovich]

Thanks @ringerc, we appreciate the effort. Please let me know once you make those changes and I'll merge this.

[ringerc]

Test steps for doing this with a canned MSVC VM on Azure:

• Deploy Azure VM https://azuremarketplace.microsoft.com/en/marketplace/apps/microsoftvisualstudio.visualstudio2022?tab=PlansAndPrice - I used "Visual Studio 2022 Community on Windows Server 2022 (x64)"
• Install CMake
• git clone my branch
• Ignore all dependencies since I only care about the core and in_exec plugin for this
• cmake . in my clone
• wait an eternity for sloooow windows build tooling to get around to checks
• cmake --build .

I found that Fluent Bit's cmake code does not raise a fatal error if openssl cannot be found, but then fails to build if openssl is absent, with errors like Cannot open include file: 'openssl/err.h': No such file or directory. That's a build config bug.

Grepping randomly around the repo to try to figure out how fluent-bit is built for Windows I stumbled on dockerfiles/Dockerfile.windows . I followed what it does to get an openssl build that's presumably compatible:

iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1')); `
    choco feature disable --name showDownloadProgress ; `
    choco install -y openssl;

Then re-ran the build

• cmake -DOPENSSL_ROOT_DIR='C:\Program Files\OpenSSL-Win64' .
• cmake --build .

which now fails with fluent-bit\lib\cmetrics\include\cmetrics/cmt_decode_prometheus.h(95,10): fatal error C1083: Cannot open include file: 'cmt_decode_prometheus_parser.h': No such file or directory

I'm guessing this might be due to missing flex and bison, and the fluent-bit build system failing to check for errors? Grabbed https://github.com/lexxmark/winflexbison/releases/download/v2.5.22/win_flex_bison-2.5.22.zip and unpacked like the Dockerfile does, then $env:PATH += "C:\WinFlexBison" and re-ran again, but same deal.

Now weirdly, I notice in plugins/CMakeLists.txt that in_node_exporter_metrics is only built if(${CMAKE_SYSTEM_NAME} MATCHES "Linux") ... so why is it being built here?

But

fluent-bit> cmake -L | findstr CMAKE_SYSTEM_NAME
-- CMAKE_SYSTEM_NAME:        Windows
-- #### CMAKE_SYSTEM_NAME is Windows

so I'm confused. This is a pain. I can't easily use the Dockerfile since I'm in a Windows VM without Hyper-V.

ok, installing flex and bison and re-running

• cmake -DOPENSSL_ROOT_DIR='C:\Program Files\OpenSSL-Win64' .
• cmake --build .

worked. Note that you have to unpack the whole winflexbision archive to \winflexbison or you'll get errors about "m4sugar".

This build needs some error checking of command exit values and required headers

At the very least the cmake run should abort if

• openssl/err.h cannot be found, since it is required to build
• flex cannot be found, since it is required to build
• bison cannot be found, since it is required to build

[ringerc]

New separate PR to fix the windows build complaints somewhat #7463

[ringerc]

@leonardo-albertovich Fixed tabs

[ringerc]

(Don't like that this PR doesn't add test cover, but I'm not writing a whole new test module for a plugin that has none already, especially given that fluent-bit uses its own bespoke test suite with limited docs and examples.)

[ringerc]

@leonardo-albertovich

The failing tests are

The following tests FAILED:
	  3 - flb-rt-in_event_test (SEGFAULT)
	  9 - flb-rt-in_statsd (Failed)
	 10 - flb-rt-in_syslog (Failed)

in https://github.com/fluent/fluent-bit/actions/runs/5063344577/jobs/9108776973?pr=7207 which are clearly unrelated to this change, and I've seen be unstable in multiple recent runs.

AFAICS this is ready for merge, how to I get it progressed to merge?

[leonardo-albertovich]

Don't worry about those, we are in the process of fixing them (it's a localized issue in macos). Your PR is fine.

[edsiper]

commits must be prefixed with the plugin name , e.g: in_exec: ... . I am fixing that now in the squash

[ringerc]

Thankyou @edsiper

You can probably enable the in_exec plugin for Windows now too, since this PR ported it over to build on Windows.

[ringerc]

@edsiper The docs PR for this change at fluent/fluent-bit-docs#1080 pending merge too

[patrick-stephens]

Docs will be merged once a release is done @ringerc - otherwise it confuses people to see the docs up for something not available in the release.

[patrick-stephens]

Thankyou @edsiper

You can probably enable the in_exec plugin for Windows now too, since this PR ported it over to build on Windows.

Can you submit a PR for this?

[ringerc]

@patrick-stephens #7664 as requested . I've torn down my Windows dev-env (it was an Azure VM) after this work closed out, so I can't easily do any manual testing of it, but the github CI stuff should run to validate the build at least.