Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Loki output plugin #994

Closed
redbaron opened this issue Dec 21, 2018 · 14 comments
Closed

Add Loki output plugin #994

redbaron opened this issue Dec 21, 2018 · 14 comments
Labels
enhancement fixed

Comments

@redbaron
Copy link

redbaron commented Dec 21, 2018 •
edited
Loading

Describe the solution you'd like

Grafanalabs announced Loki log management system, it would be really nice to have output plugin in Fluentbit for it.

Describe alternatives you've considered

Loki comes with their own promtail log collector, nobody really needs yet another log collector.

Additional context

Loki API is dead simple and documented in https://github.com/grafana/loki/blob/master/docs/api.md
@redbaron redbaron mentioned this issue Dec 21, 2018
Closed
@cosmo0920
Copy link
Contributor

Hi, I've created grafana loki plugin which uses Golang interface: https://github.com/cosmo0920/fluent-bit-go-loki

@cosmo0920
Copy link
Contributor

I’ve sent a PR to create Grafana/Loki fluent-bit-go plugin and merged in grafana/loki#847

@dawidmalina
Copy link

Would be good if this plugin could be moved from grafana and promoted as official plugin under fluent-bit umbrella same as plugin for es or splunk.

simonswine added a commit to simonswine/fluent-bit that referenced this issue Jan 6, 2020
@simonswine
pack: add json_loki format
ddf8867 
This allows to use the existing HTTP output to push logs into Grafana
Loki. This uses the JSON format for pushing logs into Loki, as using
protobuf would require more dependencies. (cf. fluent#994)

Signed-off-by: Christian Simon <[email protected]>
@simonswine simonswine mentioned this issue Jan 6, 2020
Closed
@naseemkullah naseemkullah mentioned this issue Mar 27, 2020
Closed
@ghost
Copy link

ghost commented May 13, 2020 •
edited by ghost
Loading

Any update on this? I've currently created a hybrid fluent-bit (using the image provided by grafana and config from fluent bit itself). 🚀

With minimal tweaks I was able to get it working on Openshift. Both promtail and fluent-bit loki plugin (from Grafana) are proving to be very hard to get up and running. I can share my helm config if needed..

My C experience is very rusty so maybe not the best idea for me at this point 😅

@edsiper
Copy link
Member

edsiper commented May 13, 2020

Yes, there is a plan.

One feature that we are working on Fluent Bit core is the ability to send chunks of data "in order".

Fluent Bit was designed for fast performance and data delivery, meaning we send data in parallel. This could lead to having data chunks arriving at different intervals on the destination, for most of the services this is not a problem, but Loki requires that all data arrives ordered by timestamp which is something we cannot guarantee at the moment, consider this case:

  • Chunk A -> Failed to deliver (network issue or whatever)
  • Chunk A -> will be retried in N seconds
  • Chunk B -> Delivered
  • Chunk A -> Cannot deliver because timestamps are earlier than Chunk B timestamps

the solution is to implement a logic to deliver data in order without multiplexing and in order. Of course, this hurt performance (meaning: deliver data at a slow rate) but solves the problem. This part is "work in process"

@ghost
Copy link

ghost commented May 19, 2020

@edsiper thank you for elaborating this! Really appreciated 👌

@dojci
Copy link
Contributor

dojci commented May 27, 2020 •
edited
Loading

I've created buffering inside loki output plugin grafana/loki#2142 as an alternative to the future in order flb core buffering

@lanphan
Copy link

lanphan commented Jun 18, 2020

Hi @edsiper ,
Is there any update on this? We're evaluating fluent-bit as a very excellent log collector, but we need to send logs to Loki.
Looking forward to your response.
Thanks.

@edsiper
Copy link
Member

edsiper commented Jun 18, 2020

@lanphan it will be implemented but other previous tasks are required like custom buffering (work in process)

For now you can use the golang plugin available for Fluent Bit

@lanphan
Copy link

lanphan commented Jun 18, 2020

Thanks for your quick response, @edsiper .
As I'm new to Fluent-bit, I searched around and found that:

  1. https://docs.fluentbit.io/manual/pipeline/outputs : there is no output for Loki here
  2. grafana/loki@b27f92e : fluent-bit plugin in Loki repo

What you mentioned is #2 I found above?

@cosmo0920
Copy link
Contributor 

2\. [grafana/loki@b27f92e](https://github.com/grafana/loki/commit/b27f92e86bba86841d0947d416636cbe8d4eaf74) : fluent-bit plugin in Loki repo

Yeah, you can use official golang plugin for loki.

@lanphan
Copy link

lanphan commented Jun 18, 2020

Thank @cosmo0920 .
However, just a bit curious, as @edsiper explained above about the data order issue fluent-bit got, how can fluent-bit plugin of loki solved that problem?

@cosmo0920
Copy link
Contributor

fluent-bit golang plugin of Loki uses dque to implement FIFO queue which is described here:
https://github.com/grafana/loki/pull/2142/files#diff-447765ac3e6f9a153bbe056c0855b73bR84-R101

edsiper added a commit that referenced this issue Oct 14, 2020
@edsiper
out_loki: new Loki output plugin! (#994)
f12a969 
This patch adds a new plugin to deliver records to Loki (grafana).

It supports the following configuration options:

  host                    Loki hostname or IP address

  port                    Loki TCP port

  tenant_id               Tenant ID used by default to push logs to Loki. If
                          omitted or empty it assumes Loki is running in
                          single-tenant mode and no X-Scope-OrgID header is sent.
                          > type: string

  labels                  labels for API requests.
                          > default: job="fluent-bit", type: multiple comma delimited strings

  auto_kubernetes_labels  If set to true, it will add all Kubernetes labels to
                          Loki labels.
                          > default: false, type: boolean

  label_keys              Comma separated list of keys to use as stream labels.
                          > type: multiple comma delimited strings

One of the interesting features is that 'labels' and 'label_keys' support record
accessor mode, e.g:

    [OUTPUT]
        name        loki
        match       *
        labels      job=fluentbit, rec=$sub['field']
        label_keys  $somekey

Note that 'label_keys' must be prefixed with '$' since it's a record accessor pattern.

This plugin is still in development, community feedback is welcome.

Signed-off-by: Eduardo Silva <[email protected]>
edsiper added a commit that referenced this issue Oct 15, 2020
@edsiper
out_loki: new Loki output plugin! (#994)
01f5488 
This patch adds a new plugin to deliver records to Loki (grafana).

It supports the following configuration options:

  host                    Loki hostname or IP address

  port                    Loki TCP port

  tenant_id               Tenant ID used by default to push logs to Loki. If
                          omitted or empty it assumes Loki is running in
                          single-tenant mode and no X-Scope-OrgID header is sent.
                          > type: string

  labels                  labels for API requests.
                          > default: job="fluent-bit", type: multiple comma delimited strings

  auto_kubernetes_labels  If set to true, it will add all Kubernetes labels to
                          Loki labels.
                          > default: false, type: boolean

  label_keys              Comma separated list of keys to use as stream labels.
                          > type: multiple comma delimited strings

One of the interesting features is that 'labels' and 'label_keys' support record
accessor mode, e.g:

    [OUTPUT]
        name        loki
        match       *
        labels      job=fluentbit, rec=$sub['field']
        label_keys  $somekey

Note that 'label_keys' must be prefixed with '$' since it's a record accessor pattern.

This plugin is still in development, community feedback is welcome.

Signed-off-by: Eduardo Silva <[email protected]>
@edsiper
Copy link
Member

edsiper commented Nov 11, 2020

Fixed: https://docs.fluentbit.io/manual/pipeline/outputs/loki

@edsiper edsiper closed this as completed Nov 11, 2020
@cyriltovena cyriltovena mentioned this issue Nov 19, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement fixed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@redbaron @lanphan @edsiper @cosmo0920 @dawidmalina @dojci