Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics generator regex policy filter compiles on each attribute value match #3015

Closed
andriusluk opened this issue Oct 12, 2023 · 1 comment · Fixed by #3025
Closed

Metrics generator regex policy filter compiles on each attribute value match #3015

andriusluk opened this issue Oct 12, 2023 · 1 comment · Fixed by #3025
Labels
good first issue Good for newcomers help wanted Extra attention is needed keepalive Label to exempt Issues / PRs from stale workflow type/bug Something isn't working type/performance

Comments

@andriusluk
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Metrics generator has a filter policy with match type regex. Generator is iterating over attributes and performs regex compilation every time it tries to match value. This results in high memory consumption.
https://github.com/grafana/tempo/blob/main/pkg/spanfilter/spanfilter.go#L85

Describe the solution you'd like
Compile regex once and perform match with already compiled regex.

Describe alternatives you've considered
Used strict match type on the policy, which works well and memory consumption is stable. Although I have a use-case where I would like to use regex, but can't afford the memory consumption.

Additional context
Using strict type used to consume just over 1.2GB of memory. Changing to config presented below resulted in ~60GB of memory being used.

filter_policies:
  - include:
      match_type: regex
      attributes:
        - key: resource.telemetry.sdk.version
          value: 1\.1[6-9]\.[0-9]
        - key: service.namespace
          value: payments|access
@joe-elliott joe-elliott added type/bug Something isn't working keepalive Label to exempt Issues / PRs from stale workflow type/performance labels Oct 12, 2023
@joe-elliott
Copy link
Member

Nice find. This would be a nice improvement, but I don't have time to get to it atm. Going to tag it to look for some help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers help wanted Extra attention is needed keepalive Label to exempt Issues / PRs from stale workflow type/bug Something isn't working type/performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants