Namespace support for fluentbit.io/exclude annotation

[ialidzhikov]

Is your feature request related to a problem? Please describe.

Currently fluentbit.io/exclude annotation works only with pods. We have a scenario in which we want to ignore all of the pods in given namespaces. Putting fluentbit.io/exclude annotation to each pod is overhead and it will lead to restart (the old pod will be evicted and a new pod with the given annotation will be launched).

Describe the solution you'd like

It would be great if we can annotate namespace with fluentbit.io/exclude: true and fluent-bit stops to track the pods in this namespace.

Describe alternatives you've considered

Additional context

fluent-plugin-kubernetes_metadata_filter for Fluentd also reads the namespace labels and annotations.

[16g]

Any updates?

[surajnarwade]

Hi all, any updates on this issue, I would like to see this feature in a fluent bit, it'll be very convenient to have this rather than doing multiple filters to achieve this.

[pierluigilenoci]

@edsiper any feedback about this?

[envybee]

Hi all, wondering if there's any update on this, or if anyone came across a workaround for it? Thanks

[rameca231190]

Hi, wondering if there's any update on this ?

[pierluigilenoci]

@edsiper @niedbalski is there any chance of receiving feedback regarding this issue?

[ypicard]

This would be amazing.

[pierluigilenoci]

@edsiper @niedbalski any news?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[evgkrsk]

anyone?

[pierluigilenoci]

@edsiper could you please take a look to this?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[pierluigilenoci]

@edsiper could you please take a look to this?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[pierluigilenoci]

@edsiper could you share your opinion about it?

[dmpe]

This is really needed, otherwise we end up having large amount of filters...

[srmars]

Any update on this feature

[dunkelbunt1]

would be nice if there would be a feature for it, dealing with all the filters isn't optimal especially once trying to template it all

[pierluigilenoci]

@edsiper @koleini @fujimotos @PettitWesley @clamoriniere @sxd @niedbalski who should we ask for feedback on this?

[pierluigilenoci]

Maybe @nokute78 @leonardo-albertovich @patrick-stephens @pwhelan @cosmo0920 @DavidKorczynski @tarruda ?

[gitkr999]

A much-needed functionality. The current addition of fluentbit.io/exclude is very complicated and difficult to implement with a large number of pods. With hundreds of pods it is very difficult. Please add exclude for namespace very much.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[pierluigilenoci]

@edsiper it seems incredible that such a requested feature has little feedback from the maintainers.

[leonardo-albertovich]

It would be good if we took a few steps back and switched to a more constructive approach don't you think?

I am not familiar with the topic but would be glad to help if anyone submitted a PR or at least a detailed enhancement proposal to steer this into the right direction.

[pierluigilenoci]

@leonardo-albertovich indeed, patiently waiting 1469 days to get feedback from one of the maintainers on the feasibility of the feature request is an undesirable attitude.

A thousand days ago, it would have been enough to write something like: "Guys, a nice idea! Unfortunately, we maintainers can't develop this feature. But anyone who feels able to do so can open a PR, and we'll be happy to review it".

[chrismedinapy]

Is anyone working on a PR for this soo much-needed feature?
I would like to help.

[pierluigilenoci]

@chrismedinapy I estimate that no one is working on it.
So if you want to do it, it is certainly welcome.

[HaveFun83]

any news here?

[Cajga]

So, what would be the best "workaround" today to solve this problem?

I found the following so far but none of them are good enough for complex envs:

• add the annotation to each pod (I guess, this could be "automated" by a mutation controller for given namespaces). This cannot handle cases when you have multiple outputs (like we have syslog and ES) and you want to be able to select which one to use
• exclude the namespaces from tail with Exclude_Path. The INPUT is not the right place to exclude things again as with 2 outputs, you cannot "select" where to send the logs (on top of this, It would be difficult to automate this)

Question for more advanced users then me: would it be possible to write a filter that adds an extra tag based on namespace annotations to the logs that then can be filtered on the output level with Match_Regex?

[patrick-stephens]

Namespace annotations/labels will be available in FB 3.0 from the K8S filter so @Cajga that may be possible.

[HaveFun83]

As a workaround we use the following grep filter

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log_Key log
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    # exclude specific namespaces
    [FILTER]
        Name    grep
        Match   kube.*
        Logical_Op or
        Exclude $kubernetes['namespace_name'] mynamespace
        Exclude $kubernetes['namespace_name'] default
        Exclude $kubernetes['namespace_name'] kube-system

    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under kubernetes

[Cajga]

@patrick-stephens excellent news. Exactly what we need to solve the use case in the ticket, as well as to solve our more involved use case (select to use or not use an output based on namespace). With namespace label/annotation metadata attached, we can use a rewrite-tag filter to add the extra information to the tag that can be then used in the routing at output.

@HaveFun83 , we have massive multi-tenant clusters with many namespaces. While we do generate the tenant config with helm template (so we could attach a label/annotation to the namespace automatically), modifying the fluent-bit config would be a manual process that we definitely wants to avoid (and do not want to start generating just for this).

[mrueg]

As a workaround we use the following grep filter

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log_Key log
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    # exclude specific namespaces
    [FILTER]
        Name    grep
        Match   kube.*
        Logical_Op or
        Exclude $kubernetes['namespace_name'] mynamespace
        Exclude $kubernetes['namespace_name'] default
        Exclude $kubernetes['namespace_name'] kube-system

    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under kubernetes

If I want to run an allow-list based on namespaces,

     [FILTER]
         Name    grep
         Match   kube.*
         Logical_Op or
         Regex $kubernetes['namespace_name'] mynamespace
         Regex $kubernetes['namespace_name'] default
         Regex $kubernetes['namespace_name'] kube-system

would this work?