[Feature][Log] Integrate slf4j and log4j2 for unified management logs #3025

hailin0 · 2022-10-08T15:24:38Z

Purpose of this pull request

proposal #2725

Change logs:

Maven dependencys:

Add slf4j & log4j2 provider & bridges scope into root pom.xml dependencyManagement
- include: slf4j-api、log4j2、log4j2-slf4j-impl(slf4j provider)、xxx-to-slf4j
- exclude(provided): logback、log4j1、commons-logging、slf4j-other-provider、log4j2-to-slf4j
Add slf4j & log4j2 & jcl-over-slf4j into root pom.xml dependencies
Exclude all logging system packages in maven-sheade-plugin

SeaTunnel engine module:

Enhanced logging http endpoint to dynamically change log4j2 level
Config hazelcast.logging.type: log4j2

SeaTunnel Dist module:

Use maven-assembly-plugin to copy slf4j & log4j2 & jcl-over-slf4j packages to lib/logging for use by starter shell

Engine Starter module[flink、spark]:

Use maven-dependency-plugin to copy slf4j & log4j2 & jcl-over-slf4j packages to target/logging-e2e for use by E2E testcase
Add lib/logging directory to java classpath in start shell for use by starter class
Add jcl-over-slf4j package into shade jar
Config config/log4j2-console.properties on engine-client
Exclude log4j & commons-logging from shade jar
Fix shell script tail last line text

Engine Starter module[seatunnel]:

Add disruptor jar and config -Dlog4j2.contextSelector to use asynchronous logger on engine-server
Config config/log4j2-file.properties on engine-server
Config config/log4j2-console.properties on engine-client
Add slf4j & log4j2 & jcl-over-slf4j package into starter shade jar
Support create new log file for submit local mode job

Connector-v1 module[seatunnel-connector-flink-clickhouse]:

Exclude commons-logging from shade jar

E2E:

Remove log4j.properties files
Add log4j2-test.properties output log to STDOUT or STDERR
Copy starter[flink、spark] target/logging-e2e into engine container
Copy config dir into engine container
Improve execute job logs

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

hailin0 · 2022-10-08T16:09:17Z

Add notes to v1 documents：Need to add runtime jar to flink engine at runtime

hailin0 · 2022-10-08T16:17:08Z

seatunnel-e2e/seatunnel-e2e-common/src/test/resources/log4j2-test.properties

+appender.consoleWarn.filter.acceptGteWarn.type = ThresholdFilter
+appender.consoleWarn.filter.acceptGteWarn.level = WARN
+appender.consoleWarn.filter.acceptGteWarn.onMatch = ACCEPT
+appender.consoleWarn.filter.acceptGteWarn.onMismatch = DENY


log example: STDOUT、STDERR

hailin0 · 2022-10-09T04:54:17Z

...rver/src/main/java/org/apache/seatunnel/engine/server/log/Log4j2HttpGetCommandProcessor.java

+    public void handle(HttpGetCommand request) {
+        String uri = request.getURI();
+        if (uri.startsWith(HttpCommandProcessor.URI_LOG_LEVEL)) {
+            outputAllLoggerLevel(request);


hailin0 · 2022-10-09T04:55:32Z

...ver/src/main/java/org/apache/seatunnel/engine/server/log/Log4j2HttpPostCommandProcessor.java

+    public void handle(HttpPostCommand request) {
+        String uri = request.getURI();
+        if (uri.startsWith(HttpCommandProcessor.URI_LOG_LEVEL)) {
+            setLoggerLevel(request);


hailin0 · 2022-10-10T06:40:18Z

config/log4j2-file.properties

+property.file_split_size = 100MB
+property.file_count = 100
+property.file_ttl = 7d


log files：
max bytes = 100MB * 100
max times = 7d

hailin0 · 2022-10-10T06:44:01Z

seatunnel-core/pom.xml

+                            </goals>
+                            <configuration>
+                                <includeGroupIds>org.slf4j,org.apache.logging.log4j</includeGroupIds>
+                                <outputDirectory>${project.build.directory}/logging-e2e</outputDirectory>


hailin0 · 2022-10-10T06:46:49Z

seatunnel-core/pom.xml

-                <configuration>
-                    <shadedArtifactAttached>false</shadedArtifactAttached>
-                    <createDependencyReducedPom>true</createDependencyReducedPom>
-                    <!-- Make sure the transitive dependencies are written to the generated pom under <dependencies> -->
-                    <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
-                    <artifactSet>
-                        <excludes>
-                            <exclude>ch.qos.logback:*</exclude>
-                        </excludes>
-                    </artifactSet>
-                </configuration>


configuration from extends parent pom

hailin0 · 2022-10-10T06:47:45Z

seatunnel-core/seatunnel-starter/pom.xml

+                <configuration>
+                    <artifactSet>
+                        <excludes>
+                            <!-- not excluded slf4j-api & log4j2 & bridges -->
+                            <exclude>org.slf4j:slf4j-jdk14</exclude>
+                            <exclude>org.slf4j:slf4j-nop</exclude>
+                            <exclude>org.slf4j:slf4j-simple</exclude>
+                            <exclude>org.slf4j:slf4j-reload4j</exclude>
+                            <exclude>org.slf4j:slf4j-log4j12</exclude>
+                            <exclude>log4j:*</exclude>
+                            <exclude>commons-logging:*</exclude>
+                            <exclude>ch.qos.logback:*</exclude>
+                            <!--  -->
+                            <exclude>org.apache.logging.log4j:log4j-to-slf4j</exclude>
+                        </excludes>
+                    </artifactSet>
+                </configuration>


Override the parent's configuration

include： slf4j-api & log4j2

hailin0 · 2022-10-10T06:48:41Z

seatunnel-core/seatunnel-starter/src/main/bin/seatunnel-cluster.sh

+fi
+
+if [ -z $LOG4J2_CONTEXT_SELECTOR ]; then
+    LOG4J2_CONTEXT_SELECTOR="org.apache.logging.log4j.core.async.AsyncLoggerContextSelector"


Enable async logger

hailin0 · 2022-10-10T06:50:58Z

seatunnel-engine/seatunnel-engine-common/src/main/resources/hazelcast.yaml

+    rest-api:
+      enabled: true
+      endpoint-groups:
+        CLUSTER_WRITE:
+          enabled: true


Enable API: POST /hazelcast/rest/log-level

hailin0 · 2022-10-10T10:38:40Z

seatunnel-core/pom.xml

+                            </goals>
+                            <configuration>
+                                <includeGroupIds>org.slf4j,org.apache.logging.log4j</includeGroupIds>
+                                <outputDirectory>${project.build.directory}/logging-e2e</outputDirectory>


hailin0 · 2022-10-10T10:43:24Z

seatunnel-core/pom.xml

+                            <!-- not excluded: xxx-to-slf4j bridges, e.g. org.slf4j:jcl-over-slf4j  -->
+                            <exclude>org.slf4j:slf4j-api</exclude>
+                            <exclude>org.slf4j:slf4j-jdk14</exclude>
+                            <exclude>org.slf4j:slf4j-nop</exclude>
+                            <exclude>org.slf4j:slf4j-simple</exclude>
+                            <exclude>org.slf4j:slf4j-reload4j</exclude>
+                            <exclude>org.slf4j:slf4j-log4j12</exclude>
                            <exclude>ch.qos.logback:*</exclude>
+                            <exclude>log4j:*</exclude>
+                            <exclude>org.apache.logging.log4j:*</exclude>
+                            <exclude>commons-logging:*</exclude>
                        </excludes>


starter shade jar include jcl-over-slf4j

hailin0 · 2022-10-10T10:48:03Z

seatunnel-core/seatunnel-starter/pom.xml

+
+        <!-- Declare log4j2 asynchronous loggers provider: disruptor -->
+        <dependency>
+            <groupId>com.lmax</groupId>
+            <artifactId>disruptor</artifactId>
+        </dependency>
+
    </dependencies>

    <build>
        <finalName>${project.name}</finalName>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-shade-plugin</artifactId>
+                <configuration>
+                    <artifactSet>
+                        <excludes>
+                            <!-- not excluded: slf4j-api & log4j2 & xxx-to-slf4j bridges -->
+                            <exclude>org.slf4j:slf4j-jdk14</exclude>
+                            <exclude>org.slf4j:slf4j-nop</exclude>
+                            <exclude>org.slf4j:slf4j-simple</exclude>
+                            <exclude>org.slf4j:slf4j-reload4j</exclude>
+                            <exclude>org.slf4j:slf4j-log4j12</exclude>
+                            <exclude>log4j:*</exclude>
+                            <exclude>commons-logging:*</exclude>
+                            <exclude>ch.qos.logback:*</exclude>
+                            <exclude>org.apache.logging.log4j:log4j-to-slf4j</exclude>
+                        </excludes>


seatunnel starter shade jar include : slf4j-api、log4j-api、log4j-core、log4j-slf4j-impl、jcl-over-slf4j、disruptor

hailin0 · 2022-10-10T10:48:48Z

seatunnel-dist/src/main/assembly/assembly-bin-ci.xml

+            <includes>
+                <include>org.slf4j:slf4j-api:jar</include>
+                <include>org.slf4j:jcl-over-slf4j:jar</include>
+                <include>org.apache.logging.log4j:log4j-api:jar</include>
+                <include>org.apache.logging.log4j:log4j-core:jar</include>
+                <include>org.apache.logging.log4j:log4j-slf4j-impl:jar</include>
+            </includes>


hailin0 · 2022-10-10T10:51:35Z

...e/seatunnel-e2e-common/src/test/java/org/apache/seatunnel/e2e/common/util/ContainerUtil.java

+            MountableFile.forHostPath(loggingLibPath),
+            Paths.get(seatunnelHomeInContainer, "lib", "logging").toString());
+    }
+


mapping module dir to Container:

module dir:

Container dir:

hailin0 · 2022-10-10T11:10:46Z

Improve maven package：

starter[spark、flink]
change before:
starter shade jar include: slf4j、 log4j1.x

change after:
starter shade jar exclude: slf4j、 log4j1.x, include:jcl-over-slf4j

connector：example seatunnel-connector-flink-clickhouse
change before:
connector shade jar include: commons-logging, or other loggings

change after:
connector shade jar exclude all logging packages

EricJoy2048 · 2022-10-10T12:30:15Z

Please fix CI problem.

hailin0 · 2022-10-11T09:17:06Z

config/log4j2.properties

+property.file_split_size = 100MB
+property.file_count = 100
+property.file_ttl = 7d


log files：
max bytes = 100MB * 100
max times = 7d

hailin0 · 2022-10-11T09:17:23Z

config/log4j2.properties

+rootLogger.appenderRef.consoleWarn.ref = consoleWarn
+################################# console log #################################
+################################# file log    #################################
+#rootLogger.appenderRef.file.ref = fileAppender


Disable file log

hailin0 · 2022-10-11T09:21:54Z

seatunnel-core/seatunnel-core-flink-sql/src/main/bin/start-seatunnel-sql.sh

+  JAVA_OPTS="${JAVA_OPTS} -Dlog4j2.configurationFile=${CONF_DIR}/log4j2.properties"
+  JAVA_OPTS="${JAVA_OPTS} -Dseatunnel.logs.path=${APP_DIR}/logs"
+  JAVA_OPTS="${JAVA_OPTS} -Dseatunnel.logs.file_name=seatunnel-core-flink-sql"


Add log4j2 system properties

hailin0 · 2022-10-11T09:22:38Z

seatunnel-core/seatunnel-core-flink-sql/src/main/bin/start-seatunnel-sql.sh

+  JAVA_OPTS="${JAVA_OPTS} -Dseatunnel.logs.file_name=seatunnel-core-flink-sql"
+fi
+
+CLASS_PATH=${APP_DIR}/lib/logging/*:${APP_JAR}


add lib/logging/ into starter jar classpath

hailin0 · 2022-10-11T09:23:26Z

seatunnel-core/seatunnel-core-flink-sql/src/main/bin/start-seatunnel-sql.sh

+    echo "Execute SeaTunnel Flink SQL Job: $(echo "${CMD}" | tail -n 1)"
+    eval $(echo "${CMD}" | tail -n 1)


tail last line text

hailin0 · 2022-10-11T09:24:13Z

seatunnel-core/seatunnel-core-spark/src/main/bin/start-seatunnel-spark.sh

+    echo "Execute SeaTunnel Spark Job: $(echo "${CMD}" | tail -n 1)"
+    eval $(echo "${CMD}" | tail -n 1)


Fix tail last line text: ${CMD} -> "${CMD}"

hailin0 · 2022-10-12T03:07:30Z

@TyrantLucifer @EricJoy2048 @ashulin @Hisoka-X @CalvinKirs thx. PTAL

hailin0 · 2022-10-12T06:25:12Z

close #2474

EricJoy2048 · 2022-10-12T14:04:38Z

resolve conflicts please.

ashulin

@CalvinKirs CC, Does he need to resolve license conflicts?

CalvinKirs

These two can be removed and we do not need to declare the standard AL2

EricJoy2048

+1

Maven dependencys: - Add slf4j & log4j2 provider & bridges scope into root pom.xml dependencyManagement - include: slf4j-api、log4j2、log4j2-slf4j-impl(slf4j provider)、xxx-to-slf4j - exclude(provided): logback、log4j1、commons-logging、slf4j-other-provider、log4j2-to-slf4j - Add slf4j & log4j2 & jcl-over-slf4j into root pom.xml dependencies - Exclude all logging system packages in maven-sheade-plugin SeaTunnel engine module: - Enhanced logging http endpoint to dynamically change log4j2 level - Config `hazelcast.logging.type: log4j2` SeaTunnel Dist module: - Use `maven-assembly-plugin` to copy slf4j & log4j2 & jcl-over-slf4j packages to `lib/logging` for use by starter shell Engine Starter module[flink]: - Use `maven-dependency-plugin` to copy slf4j & log4j2 & jcl-over-slf4j packages to `target/logging-e2e` for use by E2E testcase - Add `lib/logging` directory to java classpath in start shell for use by starter class - Add jcl-over-slf4j package into shade jar - Config `config/log4j2-console.properties` on engine-client - Exclude log4j1 & log4j2 & logback & commons-logging from shade jar - Fix shell script tail last line text Engine Starter module[spark]: - Use `maven-dependency-plugin` to copy slf4j & log4j2 & jcl-over-slf4j packages to `target/logging-e2e` for use by E2E testcase - Add `lib/logging` directory to java classpath in start shell for use by starter class - Config `config/log4j2-console.properties` on engine-client - Exclude log4j1 & log4j2 & logback & commons-logging from shade jar - Fix shell script tail last line text Engine Starter module[seatunnel]: - Add `disruptor` jar and config `-Dlog4j2.contextSelector` to use asynchronous logger on engine-server - Config `config/log4j2-file.properties` on engine-server - Config `config/log4j2-console.properties` on engine-client - Add slf4j & log4j2 & jcl-over-slf4j & log4j-1.2-api package into starter shade jar Connector-v1 module[seatunnel-connector-flink-clickhouse]: - Exclude commons-logging from shade jar E2E: - Remove `log4j.properties` files - Add `log4j2-test.properties` output log to `STDOUT` or `STDERR` - Copy starter[flink、spark] `target/logging-e2e` into engine container - Copy config dir into engine container - Improve execute job logs

hailin0 · 2022-11-02T06:00:12Z

hailin0 commented Oct 8, 2022

View reviewed changes

hailin0 mentioned this pull request Oct 8, 2022

[ST-Engine][Logs] Log ecosystem Integration #2725

Closed

3 tasks

hailin0 commented Oct 9, 2022

View reviewed changes

hailin0 force-pushed the integration_log4j2 branch 6 times, most recently from 03df195 to 6572476 Compare October 10, 2022 06:35

hailin0 commented Oct 10, 2022

View reviewed changes

hailin0 force-pushed the integration_log4j2 branch 2 times, most recently from 372253f to 918c1fc Compare October 10, 2022 10:37

hailin0 commented Oct 10, 2022

View reviewed changes

hailin0 force-pushed the integration_log4j2 branch 4 times, most recently from 9617b2e to 4dec94e Compare October 11, 2022 09:14

hailin0 commented Oct 11, 2022

View reviewed changes

hailin0 changed the title ~~[Feature][Log] Integration log4j2~~ [Feature][Log] Integrate slf4j and log4j2 for unified management logs Oct 11, 2022

hailin0 force-pushed the integration_log4j2 branch 3 times, most recently from 2189cf0 to e253812 Compare October 11, 2022 16:08

ashulin reviewed Oct 12, 2022

View reviewed changes

hailin0 force-pushed the integration_log4j2 branch from e253812 to 60082ad Compare October 13, 2022 02:14

hailin0 force-pushed the integration_log4j2 branch 4 times, most recently from b1eee12 to ad9b5ea Compare October 20, 2022 11:55

hailin0 force-pushed the integration_log4j2 branch 5 times, most recently from 60afeb2 to 15b890d Compare October 29, 2022 14:38

CalvinKirs reviewed Oct 31, 2022

View reviewed changes

hailin0 force-pushed the integration_log4j2 branch 3 times, most recently from d538c63 to a57c9b2 Compare October 31, 2022 11:10

EricJoy2048 previously approved these changes Nov 1, 2022

View reviewed changes

hailin0 added 2 commits November 1, 2022 17:26

merge dev change

35b8222

hailin0 dismissed EricJoy2048’s stale review via 35b8222 November 1, 2022 09:41

hailin0 force-pushed the integration_log4j2 branch from a57c9b2 to 35b8222 Compare November 1, 2022 09:41

Merge branch 'dev' into integration_log4j2

fc76847

CalvinKirs approved these changes Nov 2, 2022

View reviewed changes

Hisoka-X approved these changes Nov 2, 2022

View reviewed changes

hailin0 requested review from ashulin, EricJoy2048 and Hisoka-X and removed request for ashulin, EricJoy2048 and Hisoka-X November 2, 2022 08:54

Hisoka-X merged commit 8241ec1 into apache:dev Nov 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Log] Integrate slf4j and log4j2 for unified management logs #3025

[Feature][Log] Integrate slf4j and log4j2 for unified management logs #3025

hailin0 commented Oct 8, 2022 •

edited

Loading

hailin0 commented Oct 8, 2022

hailin0 Oct 8, 2022

hailin0 Oct 9, 2022

hailin0 Oct 9, 2022

hailin0 Oct 9, 2022

hailin0 Oct 10, 2022 •

edited

Loading

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 Oct 10, 2022

hailin0 commented Oct 10, 2022

EricJoy2048 commented Oct 10, 2022

hailin0 Oct 11, 2022

hailin0 Oct 11, 2022 •

edited

Loading

hailin0 Oct 11, 2022

hailin0 Oct 11, 2022

hailin0 Oct 11, 2022

hailin0 Oct 11, 2022

hailin0 commented Oct 12, 2022 •

edited

Loading

hailin0 commented Oct 12, 2022

EricJoy2048 commented Oct 12, 2022

ashulin left a comment

CalvinKirs left a comment

EricJoy2048 left a comment

hailin0 commented Nov 2, 2022

		echo "Execute SeaTunnel Flink SQL Job: $(echo "${CMD}" \| tail -n 1)"
		eval $(echo "${CMD}" \| tail -n 1)

		echo "Execute SeaTunnel Spark Job: $(echo "${CMD}" \| tail -n 1)"
		eval $(echo "${CMD}" \| tail -n 1)

[Feature][Log] Integrate slf4j and log4j2 for unified management logs #3025

[Feature][Log] Integrate slf4j and log4j2 for unified management logs #3025

Conversation

hailin0 commented Oct 8, 2022 • edited Loading

Purpose of this pull request

Check list

hailin0 commented Oct 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 Oct 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Oct 10, 2022

EricJoy2048 commented Oct 10, 2022

Choose a reason for hiding this comment

hailin0 Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Oct 12, 2022 • edited Loading

hailin0 commented Oct 12, 2022

EricJoy2048 commented Oct 12, 2022

ashulin left a comment

Choose a reason for hiding this comment

CalvinKirs left a comment

Choose a reason for hiding this comment

EricJoy2048 left a comment

Choose a reason for hiding this comment

hailin0 commented Nov 2, 2022

hailin0 commented Oct 8, 2022 •

edited

Loading

hailin0 Oct 10, 2022 •

edited

Loading

hailin0 Oct 11, 2022 •

edited

Loading

hailin0 commented Oct 12, 2022 •

edited

Loading