Skip to content

Commit

Permalink
[Feature][Log] Integration log4j2
Browse files Browse the repository at this point in the history 
Maven dependencys:
- Add slf4j & log4j2 provider & bridges scope into root pom.xml dependencyManagement
  - include: slf4j-api、log4j2、log4j2-slf4j-impl(slf4j provider)、xxx-to-slf4j
  - exclude(provided): logback、log4j1、commons-logging、slf4j-other-provider、log4j2-to-slf4j
- Add slf4j & log4j2 & jcl-over-slf4j into root pom.xml dependencies
- Exclude all logging system packages in maven-sheade-plugin

SeaTunnel engine module:
- Enhanced logging http endpoint to dynamically change log4j2 level
- Config `hazelcast.logging.type: log4j2`

SeaTunnel Dist module:
- Use `maven-assembly-plugin` to copy slf4j & log4j2 & jcl-over-slf4j packages to `lib/logging` for use by starter shell

Engine Starter module[flink、spark]:
- Use `maven-dependency-plugin` to copy slf4j & log4j2 & jcl-over-slf4j packages to `target/logging-e2e` for use by E2E testcase
- Add `lib/logging` directory to java classpath in start shell for use by starter class
- Add jcl-over-slf4j package into shade jar
- Config `config/log4j2-console.properties` on engine-client
- Exclude log4j & commons-logging from shade jar

Engine Starter module[seatunnel]:
- Add `disruptor` jar and config `-Dlog4j2.contextSelector` to use asynchronous logger on engine-server
- Config `config/log4j2-file.properties` on engine-server
- Config `config/log4j2-console.properties` on engine-client
- Add slf4j & log4j2 & jcl-over-slf4j package into starter shade jar

Connector-v1 module[seatunnel-connector-flink-clickhouse]:
- Exclude commons-logging from shade jar

E2E:
- Remove `log4j.properties` files
- Add `log4j2-test.properties` output log to `STDOUT` or `STDERR`
- Copy starter[flink、spark] `target/logging-e2e` into engine container
- Improve execute job logs
  • Loading branch information
@hailin0
hailin0 committed Oct 10, 2022
1 parent c29370a commit 918c1fc
Show file tree
Hide file tree
Showing 61 changed files with 809 additions and 676 deletions.
41 changes: 41 additions & 0 deletions config/log4j2-console.properties
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

rootLogger.level = INFO
rootLogger.appenderRef.consoleInfo.ref = consoleInfo
rootLogger.appenderRef.consoleWarn.ref = consoleWarn

appender.consoleInfo.name = consoleInfo
appender.consoleInfo.type = CONSOLE
appender.consoleInfo.target = SYSTEM_OUT
appender.consoleInfo.layout.type = PatternLayout
appender.consoleInfo.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %c - %m%n
appender.consoleInfo.filter.acceptLtWarn.type = ThresholdFilter
appender.consoleInfo.filter.acceptLtWarn.level = WARN
appender.consoleInfo.filter.acceptLtWarn.onMatch = DENY
appender.consoleInfo.filter.acceptLtWarn.onMismatch = ACCEPT

appender.consoleWarn.name = consoleWarn
appender.consoleWarn.type = CONSOLE
appender.consoleWarn.target = SYSTEM_ERR
appender.consoleWarn.layout.type = PatternLayout
appender.consoleWarn.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %c - %m%n
appender.consoleWarn.filter.acceptGteWarn.type = ThresholdFilter
appender.consoleWarn.filter.acceptGteWarn.level = WARN
appender.consoleWarn.filter.acceptGteWarn.onMatch = ACCEPT
appender.consoleWarn.filter.acceptGteWarn.onMismatch = DENY
51 changes: 51 additions & 0 deletions config/log4j2-file.properties
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

property.file_path = logs
property.file_name = seatunnel
property.file_split_size = 100MB
property.file_count = 100
property.file_ttl = 7d

rootLogger.level = INFO
rootLogger.appenderRef.file.ref = fileAppender

appender.file.name = fileAppender
appender.file.type = RollingFile
appender.file.fileName = ${file_path}/${file_name}.log
appender.file.filePattern = ${file_path}/${file_name}.log.%d{yyyy-MM-dd}-%i
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %c - %m%n
appender.file.policies.type = Policies
appender.file.policies.time.type = TimeBasedTriggeringPolicy
appender.file.policies.time.modulate = true
appender.file.policies.size.type = SizeBasedTriggeringPolicy
appender.file.policies.size.size = ${file_split_size}
appender.file.strategy.type = DefaultRolloverStrategy
appender.file.strategy.fileIndex = nomax
appender.file.strategy.action.type = Delete
appender.file.strategy.action.basepath = ${file_path}
appender.file.strategy.action.maxDepth = 1
appender.file.strategy.action.condition.type = IfFileName
appender.file.strategy.action.condition.glob = ${file_name}.log*
appender.file.strategy.action.condition.nested_condition.type = IfAny
appender.file.strategy.action.condition.nested_condition.lastModify.type = IfLastModified
appender.file.strategy.action.condition.nested_condition.lastModify.age = ${file_ttl}
appender.file.strategy.action.condition.nested_condition.fileCount.type = IfAccumulatedFileCount
appender.file.strategy.action.condition.nested_condition.fileCount.exceeds = ${file_count}
183 changes: 154 additions & 29 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,11 +127,17 @@
<scala.binary.version>2.11</scala.binary.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>

<slf4j.version>1.7.25</slf4j.version>
<log4j2.version>2.17.1</log4j2.version>
<log4j2-disruptor.version>3.4.4</log4j2-disruptor.version>
<log4j.version>1.2.17</log4j.version>
<logback.version>1.2.3</logback.version>
<commons-logging.version>1.2</commons-logging.version>
<iceberg.version>0.13.1</iceberg.version>
<flink.1.13.6.version>1.13.6</flink.1.13.6.version>
<spark.2.4.0.version>2.4.0</spark.2.4.0.version>
<spark.binary.2.4.version>2.4</spark.binary.2.4.version>
<commons.logging.version>1.2</commons.logging.version>
<commons.beanutils.version>1.9.4</commons.beanutils.version>
<commons.cli.version>1.4</commons.cli.version>
<commons.configuration.version>1.7</commons.configuration.version>
Expand Down Expand Up @@ -179,17 +185,16 @@
<maven-helper-plugin.version>3.2.0</maven-helper-plugin.version>
<flatten-maven-plugin.version>1.3.0</flatten-maven-plugin.version>
<maven-license-maven-plugin>1.20</maven-license-maven-plugin>
<log4j-core.version>2.17.1</log4j-core.version>
<scalastyle-maven-plugin.version>1.0.0</scalastyle-maven-plugin.version>
<docker-maven-plugin.version>0.38.0</docker-maven-plugin.version>
<maven-dependency-plugin.version>3.1.1</maven-dependency-plugin.version>
<p3c-pmd.version>1.3.0</p3c-pmd.version>
<maven-scm-provider-jgit.version>1.9.5</maven-scm-provider-jgit.version>
<testcontainer.version>1.17.3</testcontainer.version>
<!-- Option args -->
<skipUT>false</skipUT>
<skipIT>true</skipIT>
<elasticsearch>7</elasticsearch>
<slf4j.version>1.7.25</slf4j.version>
<guava.version>19.0</guava.version>
<auto-service.version>1.0.1</auto-service.version>
<hadoop2.version>2.6.5</hadoop2.version>
Expand All @@ -202,6 +207,128 @@

<dependencyManagement>
<dependencies>
<!-- ***************** slf4j & provider & bridges start ***************** -->
<!-- Declare slf4j-api -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<!-- Declare slf4j-api provider: log4j2.x -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>${log4j2.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j2.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j2.version}</version>
</dependency>
<!-- Declare log4j2 asynchronous loggers provider: disruptor -->
<dependency>
<groupId>com.lmax</groupId>
<artifactId>disruptor</artifactId>
<version>${log4j2-disruptor.version}</version>
</dependency>
<!-- Include the logging bridges -->
<!-- commons-logging bridge to slf4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>${slf4j.version}</version>
</dependency>
<!-- jdk-logging bridge to slf4j -->
<!-- low performance, see: https://www.slf4j.org/legacy.html#jul-to-slf4j
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
<version>${slf4j.version}</version>
</dependency>
-->
<!-- log4j1.x bridge to slf4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
<version>${slf4j.version}</version>
</dependency>
<!-- Exclude the logging bridges via provided scope -->
<!-- slf4j binding to log4j1.x -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- log4j2.x binding to slf4j.
Use of the SLF4J adapter (log4j-to-slf4j-2.x.jar) together with the SLF4J bridge (log4j-slf4j-impl-2.x.jar) should never be attempted as it will cause events to endlessly be routed between SLF4J and Log4j 2
-->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-to-slf4j</artifactId>
<version>${log4j2.version}</version>
<scope>provided</scope>
</dependency>
<!-- slf4j binding to jdk-logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- slf4j binding to nop -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- slf4j binding to simple -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- slf4j binding to reload4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- Exclude other logging provider via provided scope -->
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>${commons-logging.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>${log4j.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>${logback.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<version>${logback.version}</version>
<scope>provided</scope>
</dependency>
<!-- ***************** slf4j & provider & bridges end ***************** -->

<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-config-shade</artifactId>
Expand Down Expand Up @@ -298,11 +425,6 @@
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j-core.version}</version>
</dependency>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
Expand All @@ -321,24 +443,6 @@
<version>${guava.version}</version>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
</dependency>

<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>${commons.logging.version}</version>
</dependency>

<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>checker-qual</artifactId>
Expand Down Expand Up @@ -392,16 +496,30 @@
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>

<!-- ***************** slf4j & provider & bridges start ***************** -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
<artifactId>jcl-over-slf4j</artifactId>
</dependency>
<!-- ***************** slf4j & provider & bridges end ***************** -->

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
Expand Down Expand Up @@ -532,6 +650,8 @@
<exclude>org.slf4j:*</exclude>
<exclude>ch.qos.logback:*</exclude>
<exclude>log4j:*</exclude>
<exclude>org.apache.logging.log4j:*</exclude>
<exclude>commons-logging:*</exclude>
</excludes>
</artifactSet>
<filters>
Expand Down Expand Up @@ -727,6 +847,11 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>${maven-dependency-plugin.version}</version>
</plugin>
</plugins>
</pluginManagement>

Expand Down
Loading

0 comments on commit 918c1fc

Please sign in to comment.