You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The logger factory io.confluent.common.logging.StructuredLoggerFactory keeps a ConcurrentHashmap of loggers used, never clearing them. If you look in PullQueryExecutor, the query id is passed as part of the name. This creates a large blowup in loggers as more queries come in.
To Reproduce
Steps to reproduce the behavior, include:
On master, generate a lot of pull queries, and set the java memory to something small. Users were able to hit this while generating high QPS for 30 minutes.
Expected behavior
No memory issues.
Screenshot from a user. This shows a persistent query (so doesn't fully display this case), but you can see that the query ids are part of the logger names and that they take a lot of memory:
The text was updated successfully, but these errors were encountered:
I did a memory profile on this and I've confirmed that the memory use keeps going up and up with lots of pull queries and the source is io.confluent.common.logging.StructuredLoggerFactory
The loggers are not written to be used this way since the slf4j logger factory itself appears to keep references to the loggers it gives out by name.
I think the solution and intended use is more something like and MDC (http://logback.qos.ch/manual/mdc.html). That way you could always add the query id and other structured data.
Describe the bug
Discussed here:
https://confluentcommunity.slack.com/archives/C6UJNMY67/p1590998942212800
The logger factory io.confluent.common.logging.StructuredLoggerFactory keeps a ConcurrentHashmap of loggers used, never clearing them. If you look in PullQueryExecutor, the query id is passed as part of the name. This creates a large blowup in loggers as more queries come in.
To Reproduce
Steps to reproduce the behavior, include:
On master, generate a lot of pull queries, and set the java memory to something small. Users were able to hit this while generating high QPS for 30 minutes.
Expected behavior
No memory issues.
Screenshot from a user. This shows a persistent query (so doesn't fully display this case), but you can see that the query ids are part of the logger names and that they take a lot of memory:
The text was updated successfully, but these errors were encountered: