out_parseable: Plugin for sending logs to Parseable #9622

AdheipSingh · 2024-11-20T10:23:12Z

Send kubernetes logs to https://www.parseable.com/

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

edsiper · 2024-11-21T18:40:38Z

can you please provide the link to the API spec ?

AdheipSingh · 2024-11-22T05:00:33Z

Here is the API doc for ingesting into parseable.

https://www.postman.com/parseable/parseable/request/au1fn5d/send-logs-to-a-log-stream-with-http-header

Here is the sample config i am using.

      inputs: |
      [INPUT]
          Name tail
          Path /var/log/containers/*.log
          multiline.parser docker, cri
          Tag kube.*
          Mem_Buf_Limit 5MB
          Skip_Long_Lines On

      [INPUT]
          name kubernetes_events
          tag k8s_events

      [INPUT]
          Name systemd
          Tag host.*
          Systemd_Filter _SYSTEMD_UNIT=kubelet.service
          Read_From_Tail On

    ## https://docs.fluentbit.io/manual/pipeline/filters
    filters: |
      [FILTER]
          Name                kubernetes
          Match               kube.*
          Merge_Log           On
          Keep_Log            Off
          K8S-Logging.Parser  On
          K8S-Logging.Exclude On

    ## https://docs.fluentbit.io/manual/pipeline/outputs
    outputs: |
      [OUTPUT]
          Name parseable
          Match kube.*
          Server_Host parseable.parseable.svc.cluster.local
          Username admin
          Password admin
          Server_Port 80
          Stream $NAMESPACE
          Exclude_Namespaces kube-system, default

      [OUTPUT]
          Name parseable
          Match k8s_events
          Server_Host parseable.parseable.svc.cluster.local
          Server_Port 80
          Username admin
          Password admin
          Stream k8s-events

patrick-stephens

Does this build for all targets and do we need any extra dependencies? We probably need some unit tests at least too.

I think at the moment you've not actually enabled it to build, only for the container but then we don't test or exercise that in unit tests or builds.

CMakeLists.txt

patrick-stephens · 2024-11-22T09:54:55Z

dockerfiles/Dockerfile

@@ -75,6 +75,7 @@ RUN cmake -DFLB_RELEASE=On \
    -DFLB_IN_SYSTEMD=On \
    -DFLB_OUT_KAFKA=On \
    -DFLB_OUT_PGSQL=On \
+    -DFLB_OUT_PARSEABLE=On \


Is this what should be used in the cmake config? It's already defaulting to on

feel free to suggest the best practice here. I tried to build without it but was not able to run the plugin.

plugins/out_parseable/parseable.c

AdheipSingh · 2024-11-22T13:14:59Z

Does this build for all targets and do we need any extra dependencies? We probably need some unit tests at least too.

I think at the moment you've not actually enabled it to build, only for the container but then we don't test or exercise that in unit tests or builds.

Please suggest some examples on unit tests and what can be done on building the ideal way. I'll implement this. ( i am not an expert in C codebase, so whatever is the best practice, ill will try to adhere to it ).

Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: AdheipSingh <[email protected]>

Signed-off-by: AdheipSingh <[email protected]>

AdheipSingh · 2024-12-05T10:57:16Z

Hi @patrick-stephens review comments have been addressed. Over to you :)
cc @nitisht

Signed-off-by: AdheipSingh <[email protected]>

nitisht · 2024-12-15T09:53:51Z

@edsiper @patrick-stephens would be great to have a next round of review please

edsiper

Before merging this, it needs some refactor. I added some comments , as well need to follow the coding style described in the contrib guide: https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#coding-style

edsiper · 2025-01-09T18:26:27Z

plugins/out_parseable/parseable.c

+    size_t b_sent;
+
+    msgpack_unpacked_init(&result);
+    while (msgpack_unpack_next(&result,


please use the flb_log_event_decode API, ref: https://github.com/fluent/fluent-bit/blob/master/plugins/out_stdout/stdout.c#L285

edsiper · 2025-01-09T18:26:48Z

plugins/out_parseable/parseable.c

+
+        /* Pack original key-value pairs */
+        msgpack_pack_map(&pk, p->via.map.size + 1);
+        for (int i = 0; i < p->via.map.size; i++) {


do not declare variables inside the for loop

edsiper · 2025-01-09T18:27:17Z

plugins/out_parseable/parseable.c

+            }
+
+
+            flb_sds_t namespace_name = flb_sds_create_size(256); // Dynamic string


declare the variable on top

edsiper · 2025-01-09T18:27:30Z

plugins/out_parseable/parseable.c

+            }
+
+
+            flb_sds_t namespace_name = flb_sds_create_size(256); // Dynamic string


validate if namespace_name is null

edsiper · 2025-01-09T18:28:28Z

plugins/out_parseable/parseable.c

+        msgpack_sbuffer_destroy(&sbuf);
+
+        /* Determine the value of the X-P-Stream header */
+        if (ctx->stream && strcmp(ctx->stream, "$NAMESPACE") == 0) {


it seems you are trying to accomplish what our record accessor API does, use the existing API

edsiper · 2025-01-09T18:28:55Z

plugins/out_parseable/parseable.c

+                                entry = mk_list_entry(head, struct flb_slist_entry, _head);
+                                flb_plg_info(ctx->ins, "Checking against exclude namespace: %s", entry->str);
+                                // flb_plg_info(ctx->ins, "namespace_name: %s %d", namespace_name,flb_sds_len(namespace_name));
+                                if (flb_sds_cmp(entry->str, namespace_name, flb_sds_len(namespace_name)) == 0) {


too many nested levels, refactor this in more clear functions

edsiper · 2025-01-09T18:31:04Z

plugins/out_parseable/parseable.h

+    flb_sds_t username;
+    flb_sds_t password;
+    flb_sds_t stream;
+    struct mk_list *exclude_namespaces;  // Use mk_list for namespace exclusion


use cfl_list

edsiper · 2025-01-09T18:31:37Z

lib/cprofiles/include/cprofiles/cprof_info.h

@@ -29,6 +29,9 @@
 #ifndef CPROF_HAVE_GMTIME_R
 #define CPROF_HAVE_GMTIME_R
 #endif
+#ifndef CPROF_HAVE_CLOCK_GET_TIME


do not touch this file

edsiper · 2025-01-09T18:32:34Z

note: there are 81 commits in this PR, it should be only one.

code review changes

AdheipSingh · 2025-01-12T20:07:53Z

@edsiper i have to close this PR due to many commits/ conflicts coming when trying to squash. I have moved all the changes you requested to #9826.
cc @nitisht

AdheipSingh requested review from niedbalski, patrick-stephens, celalettin1286, fujimotos, edsiper, leonardo-albertovich and koleini as code owners November 20, 2024 10:23

github-actions bot added the docs-required label Nov 20, 2024

AdheipSingh force-pushed the master branch from 68a5911 to f117b14 Compare November 20, 2024 10:42

patrick-stephens reviewed Nov 22, 2024

View reviewed changes

patrick-stephens added the ok-package-test Run PR packaging tests label Nov 22, 2024

AdheipSingh temporarily deployed to pr November 22, 2024 09:58 — with GitHub Actions Inactive

AdheipSingh temporarily deployed to pr November 22, 2024 10:18 — with GitHub Actions Inactive

AdheipSingh temporarily deployed to pr November 22, 2024 10:19 — with GitHub Actions Inactive

AdheipSingh force-pushed the master branch from 0f9e8bd to 13d21d7 Compare December 4, 2024 11:07

edsiper and others added 9 commits December 5, 2024 00:49

bitbake: bump to v3.2.1

54041d2

Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: AdheipSingh <[email protected]>

working http requests

abe21b5

Signed-off-by: AdheipSingh <[email protected]>

hacked code

7d24323

Signed-off-by: AdheipSingh <[email protected]>

add namespace_name filter

c5eba8e

Signed-off-by: AdheipSingh <[email protected]>

use stream dynamically

378393b

Signed-off-by: AdheipSingh <[email protected]>

update dockerfile

50a833e

Signed-off-by: AdheipSingh <[email protected]>

add debug

b82245d

Signed-off-by: AdheipSingh <[email protected]>

add headers debug

c36f54c

Signed-off-by: AdheipSingh <[email protected]>

add fix char size

0f9d637

Signed-off-by: AdheipSingh <[email protected]>

AdheipSingh added 4 commits December 5, 2024 03:32

hardcode port

2018767

Signed-off-by: AdheipSingh <[email protected]>

revert debug

98ed0a5

Signed-off-by: AdheipSingh <[email protected]>

debug ports

2c9ba03

Signed-off-by: AdheipSingh <[email protected]>

tweak configurations

a7c5961

Signed-off-by: AdheipSingh <[email protected]>

Merge branch 'master' into master

6d1f0a2

Signed-off-by: AdheipSingh <[email protected]>

edsiper requested changes Jan 9, 2025

View reviewed changes

AdheipSingh and others added 19 commits January 10, 2025 12:52

review changes

8911f8a

fixes to build

e1f8d91

comment out unsupported log event

ff621b1

update decoder

8fff3c4

update ra

ad25cf3

update record accessor and null checks

b8fef4f

update records

31b983e

update records

cd81d6b

update records

a1824d6

address comments

0a75176

update

2e500ed

update

b73c8cb

update

e6f0aec

update

23e4245

update

c872912

Merge pull request #1 from AdheipSingh/review

fb0875b

code review changes

update

83b6ef1

update

34fb8ca

Merge branch 'fluent:master' into master

3d1c021

AdheipSingh closed this Jan 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_parseable: Plugin for sending logs to Parseable #9622

out_parseable: Plugin for sending logs to Parseable #9622

AdheipSingh commented Nov 20, 2024 •

edited by patrick-stephens

Loading

edsiper commented Nov 21, 2024

AdheipSingh commented Nov 22, 2024 •

edited

Loading

patrick-stephens left a comment •

edited

Loading

patrick-stephens Nov 22, 2024

AdheipSingh Nov 22, 2024

AdheipSingh commented Nov 22, 2024

AdheipSingh commented Dec 5, 2024

nitisht commented Dec 15, 2024

edsiper left a comment

edsiper Jan 9, 2025

edsiper Jan 9, 2025

edsiper Jan 9, 2025

edsiper Jan 9, 2025

edsiper Jan 9, 2025

edsiper Jan 9, 2025

edsiper Jan 9, 2025

edsiper Jan 9, 2025

edsiper commented Jan 9, 2025

AdheipSingh commented Jan 12, 2025

		}


		flb_sds_t namespace_name = flb_sds_create_size(256); // Dynamic string

out_parseable: Plugin for sending logs to Parseable #9622

out_parseable: Plugin for sending logs to Parseable #9622

Conversation

AdheipSingh commented Nov 20, 2024 • edited by patrick-stephens Loading

edsiper commented Nov 21, 2024

AdheipSingh commented Nov 22, 2024 • edited Loading

patrick-stephens left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdheipSingh commented Nov 22, 2024

AdheipSingh commented Dec 5, 2024

nitisht commented Dec 15, 2024

edsiper left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edsiper commented Jan 9, 2025

AdheipSingh commented Jan 12, 2025

AdheipSingh commented Nov 20, 2024 •

edited by patrick-stephens

Loading

AdheipSingh commented Nov 22, 2024 •

edited

Loading

patrick-stephens left a comment •

edited

Loading