Support Logback MDC (Mapped Diagnostic Context)

[rexorient]

See http://logback.qos.ch/manual/mdc.html

This is an easy way to add context information to every log event. Things like current user, transaction, source hostname/ip, etc. MDC includes a Servlet filter which injects information about the inbound connection, it would be awesome if we could map that to the ECS equivalent fields.

[felixbarny]

The MDC is already mapped to labels, unless you set <topLevelLabel>foo</topLevelLabel>.

So MDC.put("foo.bar", "baz") will be mapped to "labels.foo_bar": "baz" or "foo.bar": "baz", respectively.

[lanore78]

Hello, @felixbarny

I this still true, with version 0.4.0?

The MDC is already mapped to labels, unless you set foo.
So MDC.put("foo.bar", "baz") will be mapped to "labels.foo_bar": "baz" or "foo.bar": "baz", respectively.

Since, I got my MDC is located in the root level of my log as follow.
(I want to put my MDC value under 'labels' fields)

// code
MDC.put("my-id", "test id");
MDC.put("errorCode", "test error code");
logger.info("log method called");

// log
{"@timestamp":"2020-07-06T14:20:20.957Z", "log.level": "INFO", "message":"log method called", "service.name":"my-app","event.dataset":"my-app.log","process.thread.name":"http-nio-8080-exec-1","log.logger":"com.lanore.loggingexample.ExampleController","errorCode":"test error code","my-id":"test id","spanId":"d77fee53ada3452c","traceId":"d77fee53ada3452c"}

And, this is log4j2.xml file

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
    <Appenders>
        <Console name="LogToConsole" target="SYSTEM_OUT">
            <EcsLayout serviceName="my-app"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="LogToConsole"/>
        </Root>
    </Loggers>
</Configuration>

[felixbarny]

Since 0.3.0, MDC is mapped to the top level. If you want to nest it under labels, set MDC.put("labels.my-id", "test id");.

The topLevelLabel property has been removed accordingly.

[lanore78]

Hello @felixelastic,

Thank you for quick reply, and I found related issue HERE
However this could cause a problem when some library use 'url', which is one of defined key of ECS, and the format is not compatible with ECS(or Elasticsearch pipeline). In this case, when I collect log with Filebeat, the Filebeat keep producing pipeline error, related to 'url'

Here is example value of 'url' and 'jdbc' MDC.

"url" : "jdbc:sqlserver://xxx.xxx.xxx.xxx:61051;authenticationScheme=nativeAuthentication;xopenStates=false;sendTimeAsDatetime=true;trustServerCertificate=false;sendStringParametersAsUnicode=false;selectMethod=direct;responseBuffering=adaptive;packetSize=8000;multiSubnetFailover=false;loginTimeout=20;lockTimeout=-1;lastUpdateCount=true;encrypt=false;disableStatementPooling=true;databaseName=USER;applicationName=Microsoft JDBC Driver for SQL Server;applicationIntent=readwrite", "jdbc":"jdbc:sqlserver://192.168.201.234:61051;authenticationScheme=nativeAuthentication;xopenStates=false;sendTimeAsDatetime=true;trustServerCertificate=false;sendStringParametersAsUnicode=false;selectMethod=direct;responseBuffering=adaptive;packetSize=8000;multiSubnetFailover=false;loginTimeout=20;lockTimeout=-1;lastUpdateCount=true;encrypt=false;disableStatementPooling=true;databaseName=USER;applicationName=Microsoft JDBC Driver for SQL Server;applicationIntent=readwrite;, UserName=cji_global_serviceweb, Microsoft JDBC Driver 4.0 for SQL Server"

I commented this problem HERE

[felixbarny]

You could define either a Filebeat rename processor or an Elasticsearch ingest node rename processor to rename the field.

Tip: better create new issues with references to old issues rather than commenting on closed ones.

I hope this helps :)