Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A simple logger tree-based filtering #204

Merged
merged 10 commits into from
Jan 19, 2021
Merged

A simple logger tree-based filtering #204

merged 10 commits into from
Jan 19, 2021

Conversation

vigoo
Copy link
Contributor

@vigoo vigoo commented Jan 2, 2021

Introduces a simple hierarchic name-based filtering method that is similar to other logging systems. The filter is defined by a list of names tied to minimum log levels.

@vigoo
Copy link
Contributor Author

vigoo commented Jan 2, 2021

Very useful when using the SLF4j bridge (#202)

@pshemass pshemass self-requested a review January 2, 2021 16:29
def filterByTree[A](root: LogFilterNode): (LogContext, => A) => Boolean =
(ctx, _) => {
val loggerName = ctx.get(LogAnnotation.Name).flatMap(_.split('.'))
val logLevel = findMostSpecificLogLevel(loggerName, root)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you check what will be performance impact of this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet, I can add some benchmarks to have an understanding. And then it could be probably improved with a cache or something.

}

private def buildLogFilterTree(rootLevel: LogLevel, mappings: Seq[(String, LogLevel)]): LogFilterNode = {
def add(tree: LogFilterNode, names: List[String], level: LogLevel): LogFilterNode =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably should use Chunk instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but note that the LogAnnotation.Name has a type of List[String] too

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I forgot about. we should probably change that in the future

@vigoo vigoo requested review from mijicd and softinio as code owners January 3, 2021 14:29
@vigoo
Copy link
Contributor Author

vigoo commented Jan 3, 2021

I created some benchmarks and also wrote a simple TMap based caching version around the filter function. The results:

   * FilterBenchmarks.handWrittenFilterEval     thrpt    5  9177150.646 ± 125715.644  ops/s
   * FilterBenchmarks.filterTreeEval            thrpt    5  7298406.870 ±  87773.959  ops/s

   * FilterBenchmarks.noFilteringLogging        thrpt    5   267066.692 ±   2170.339  ops/s
   * FilterBenchmarks.handWrittenFilterLogging  thrpt    5   262466.006 ±   3641.051  ops/s
   * FilterBenchmarks.filterTreeLog             thrpt    5   252841.756 ±   2912.062  ops/s
   * FilterBenchmarks.cachedFilterTreeLog       thrpt    5   260752.769 ±   2625.707  ops/s

For me this looks like the tree-based filter is comparable to a hand-written one (for a few nodes in the tree, which is the normal use case to my experience - thinking of the number of differently configured loggers in a typical log4j config for example).
The caching makes it almost as fast as the hand-written version, but in the benchmarks measuring logging not just the filter function itself it seems like the filtering itself is not that significant part of the whole operation. For this reason I did not completely finish the cached filtering version, meaning it is not that convenient yet to initialize it (requires the TMap from outside). If you think it worth it I can make the cache construction part of the appender layer etc.

import scala.annotation.tailrec

object LogFiltering {
case class LogFilterNode(logLevel: LogLevel, children: Map[String, LogFilterNode])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

final

@vigoo
Copy link
Contributor Author

vigoo commented Jan 18, 2021

@jdegoes I believe this is ready to merge

@jdegoes jdegoes merged commit dcb3b48 into zio:master Jan 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants