[Feature][Flink] Support multiple tables read and write #7713

PeppaPage · 2024-09-22T06:58:23Z

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

hailin0 · 2024-09-22T14:35:11Z

Please check ci error

hailin0 · 2024-09-22T15:33:13Z

close #7008

PeppaPage · 2024-09-22T15:44:10Z

Please check ci error

Thank you for your review, I think this error has nothing to do with my commit, because this error test point(testStreamReadPaimon) is completely based on the seatunnel engine, and it did not pass this test point in the previous submission

PeppaPage · 2024-09-22T16:08:46Z

Please check ci error

Thank you for your review, I think this error has nothing to do with my commit, because this error test point(testStreamReadPaimon) is completely based on the seatunnel engine, and it did not pass this test point in the previous submission

Hisoka-X · 2024-09-23T02:00:50Z

Do not close/reopen. It would lost ci status.

Hisoka-X · 2024-09-23T02:01:30Z

CI https://github.com/PeppaPage/seatunnel/actions/runs/10977855173/job/30483008717

PeppaPage · 2024-09-23T03:44:54Z

Do not close/reopen. It would lost ci status.

I'm sorry for that,I want to close my comment so i click "close with comment" button,but close my pull request instead,should I make a new pull request ?

Hisoka-X · 2024-09-23T03:46:24Z

Do not close/reopen. It would lost ci status.

I'm sorry for that,I want to close my comment so i click "close with comment" button,but close my pull request instead,should I make a new pull request ?

No, just push an emtpy commit can re-trigger it.

PeppaPage · 2024-09-23T04:04:35Z

Do not close/reopen. It would lost ci status.

I'm sorry for that,I want to close my comment so i click "close with comment" button,but close my pull request instead,should I make a new pull request ?

No, just push an emtpy commit can re-trigger it.

Thank you very much，I have just pushed.

PeppaPage · 2024-09-24T01:19:02Z

Please check ci error

No errors now

TyrantLucifer · 2024-09-24T06:28:36Z

@Hisoka-X @hailin0 PTAL

hailin0 · 2024-09-24T08:20:28Z

Do you need to clean up the commit list that does not belong to you?

hailin0 · 2024-09-24T11:43:29Z

...nk-starter-common/src/main/java/org/apache/seatunnel/core/starter/flink/utils/TableUtil.java

@@ -42,4 +44,25 @@ public static DataStream<Row> tableToDataStream(
    public static boolean tableExists(TableEnvironment tableEnvironment, String name) {
        return Arrays.asList(tableEnvironment.listTables()).contains(name);
    }
+
+    // catalogName.databaseName.[schemeName].tableName -> databaseName.[schemeName].tableName
+    public static String extractTableIdName(TableIdentifier tableIdentifier) {


Use TableIdentifier#toTablePath().toString() ?

thank you for suggestion, I've modified it.

Hisoka-X

Overall LGTM, I left some minior update.

Hisoka-X · 2024-09-24T12:12:00Z

...er/src/main/java/org/apache/seatunnel/core/starter/flink/execution/SinkExecuteProcessor.java

-            Optional<SaveModeHandler> saveModeHandler =
-                    ((SupportSaveMode) sink).getSaveModeHandler();
+    public void handleSaveMode(SeaTunnelSink seaTunnelSink) {
+        if (SupportSaveMode.class.isAssignableFrom(seaTunnelSink.getClass())) {


Suggested change

if (SupportSaveMode.class.isAssignableFrom(seaTunnelSink.getClass())) {

if (seaTunnelSink instanceof SupportSaveMode) {

Hisoka-X · 2024-09-24T12:13:45Z

...on/src/main/java/org/apache/seatunnel/core/starter/flink/execution/SinkExecuteProcessor.java

-            Optional<SaveModeHandler> saveModeHandler =
-                    ((SupportSaveMode) sink).getSaveModeHandler();
+    public void handleSaveMode(SeaTunnelSink seaTunnelSink) {
+        if (SupportSaveMode.class.isAssignableFrom(seaTunnelSink.getClass())) {


PeppaPage · 2024-09-24T12:59:06Z

Overall LGTM, I left some minior update.

thank you for suggestion, I've modified it.

hailin0

LGTM

Hisoka-X · 2024-09-25T03:13:42Z

...e-common/src/test/java/org/apache/seatunnel/e2e/common/container/flink/Flink18Container.java

+ */
+@NoArgsConstructor
+@AutoService(TestContainer.class)
+public class Flink18Container extends AbstractTestFlinkContainer {


Please update doc for flink 1.17 and 1.18
https://github.com/apache/seatunnel/blob/7b7f06437a62c1683f002db78458ec7dd90de108/docs/en/start-v2/locally/quick-start-flink.md#step-4-run-seatunnel-application

cc @TyrantLucifer any other doc/shell we should update?

thank you for suggestion, I've modified it, also https://github.com/apache/seatunnel/blob/7b7f06437a62c1683f002db78458ec7dd90de108/docs/zh/start-v2/locally/quick-start-flink.md#step-4-run-seatunnel-application.

Carl-Zhou-CN · 2024-09-25T03:30:43Z

...-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCIT.java

-            type = {EngineType.SPARK, EngineType.FLINK},
-            disabledReason = "Currently SPARK and FLINK do not support multi table")
+            type = {EngineType.SPARK},
+            disabledReason = "Currently SPARK do not support multi table")


spark already supports multiple tables, but not cdc

thank you for suggestion, I've modified it.

Carl-Zhou-CN · 2024-09-25T03:31:23Z

...-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCIT.java

-            type = {EngineType.SPARK, EngineType.FLINK},
-            disabledReason = "Currently SPARK and FLINK do not support multi table")
+            type = {EngineType.SPARK},
+            disabledReason = "Currently SPARK do not support multi table")


Same as above

Carl-Zhou-CN · 2024-09-25T03:31:34Z

...e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/postgres/OpengaussCDCIT.java

-            type = {EngineType.SPARK, EngineType.FLINK},
-            disabledReason = "Currently SPARK and FLINK do not support multi table")
+            type = {EngineType.SPARK},
+            disabledReason = "Currently SPARK do not support multi table")


Same as above

Carl-Zhou-CN · 2024-09-25T03:32:12Z

...acle-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/oracle/OracleCDCIT.java

-            type = {EngineType.SPARK, EngineType.FLINK},
-            disabledReason = "Currently SPARK and FLINK do not support multi table")
+            type = {EngineType.SPARK},
+            disabledReason = "Currently SPARK do not support multi table")


Same as above

Carl-Zhou-CN · 2024-09-25T03:32:26Z

...-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/postgres/PostgresCDCIT.java

-            type = {EngineType.SPARK, EngineType.FLINK},
-            disabledReason = "Currently SPARK and FLINK do not support multi table")
+            type = {EngineType.SPARK},
+            disabledReason = "Currently SPARK do not support multi table")


Same as above

Carl-Zhou-CN · 2024-09-25T03:39:01Z

seatunnel/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/execution/PluginUtil.java

Line 106 in 7b7f064

// if (catalogTables.size() != 1) {

Please remove the comment here @PeppaPage

Carl-Zhou-CN

LGTM

Hisoka-X

Thanks @PeppaPage ! Welcome joining SeaTunnel!

Hisoka-X · 2024-11-19T08:11:51Z

...tor-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/CheckpointEnableIT.java

@@ -225,7 +226,7 @@ public void testZetaStreamingCheckpointNoInterval(TestContainer container)

    @TestTemplate
    @DisabledOnContainer(
-            value = {},
+            value = {TestContainerId.FLINK_1_17, TestContainerId.FLINK_1_18},


hi @PeppaPage , why disabled this? cc @TyrantLucifer

Hi, flink 1.17 and 1.18 change the default ExecutionGraph in batch mode , it makes the dynamic-graph true, so it will skip the checkpoint because dynamic graph doesn't support.

in 1.18:

in 1.15:

this problem can be fixed by setting jobmanager.scheduler: Ng

but i'm not sure whether it will conflict with other properties.

github-actions bot added CI&CD core SeaTunnel core module flink e2e labels Sep 22, 2024

hailin0 linked an issue Sep 22, 2024 that may be closed by this pull request

[Feature][Flink] Support multiple tables read and write #7008

Closed

3 tasks

PeppaPage closed this Sep 22, 2024

PeppaPage reopened this Sep 22, 2024

github-actions bot added document Zeta connectors-v2 api labels Sep 23, 2024

PeppaPage force-pushed the flink-multitable-pr2 branch from 8bc7722 to 6890a61 Compare September 23, 2024 13:53

TyrantLucifer approved these changes Sep 24, 2024

View reviewed changes

github-actions bot added approved reviewed labels Sep 24, 2024

TyrantLucifer force-pushed the flink-multitable-pr2 branch from 6890a61 to e203655 Compare September 24, 2024 09:32

github-actions bot removed document Zeta labels Sep 24, 2024

github-actions bot removed connectors-v2 api labels Sep 24, 2024

TyrantLucifer previously approved these changes Sep 24, 2024

View reviewed changes

PeppaPage dismissed TyrantLucifer’s stale review via f487289 September 24, 2024 10:21

PeppaPage force-pushed the flink-multitable-pr2 branch from e203655 to f487289 Compare September 24, 2024 10:21

hailin0 reviewed Sep 24, 2024

View reviewed changes

Hisoka-X reviewed Sep 24, 2024

View reviewed changes

PeppaPage force-pushed the flink-multitable-pr2 branch from f487289 to 78af4fb Compare September 24, 2024 12:41

Hisoka-X added the First-time contributor First-time contributor label Sep 24, 2024

hailin0 previously approved these changes Sep 25, 2024

View reviewed changes

Hisoka-X reviewed Sep 25, 2024

View reviewed changes

Carl-Zhou-CN reviewed Sep 25, 2024

View reviewed changes

[Feature][Flink] Support multiple tables read and write

afbc563

PeppaPage dismissed hailin0’s stale review via afbc563 September 25, 2024 04:02

PeppaPage force-pushed the flink-multitable-pr2 branch from 78af4fb to afbc563 Compare September 25, 2024 04:02

github-actions bot added the document label Sep 25, 2024

Carl-Zhou-CN approved these changes Sep 25, 2024

View reviewed changes

Hisoka-X approved these changes Sep 25, 2024

View reviewed changes

TyrantLucifer merged commit 65312ff into apache:dev Sep 25, 2024
10 checks passed

Hisoka-X reviewed Nov 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Flink] Support multiple tables read and write #7713

[Feature][Flink] Support multiple tables read and write #7713

PeppaPage commented Sep 22, 2024

hailin0 commented Sep 22, 2024

hailin0 commented Sep 22, 2024

PeppaPage commented Sep 22, 2024

PeppaPage commented Sep 22, 2024

Hisoka-X commented Sep 23, 2024 •

edited

Loading

Hisoka-X commented Sep 23, 2024

PeppaPage commented Sep 23, 2024

Hisoka-X commented Sep 23, 2024

PeppaPage commented Sep 23, 2024

PeppaPage commented Sep 24, 2024

TyrantLucifer commented Sep 24, 2024

hailin0 commented Sep 24, 2024

hailin0 Sep 24, 2024

PeppaPage Sep 24, 2024

Hisoka-X left a comment

Hisoka-X Sep 24, 2024

Hisoka-X Sep 24, 2024

PeppaPage commented Sep 24, 2024

hailin0 left a comment

Hisoka-X Sep 25, 2024

PeppaPage Sep 25, 2024

Carl-Zhou-CN Sep 25, 2024

PeppaPage Sep 25, 2024

Carl-Zhou-CN Sep 25, 2024

Carl-Zhou-CN Sep 25, 2024

Carl-Zhou-CN Sep 25, 2024

Carl-Zhou-CN Sep 25, 2024

Carl-Zhou-CN commented Sep 25, 2024

Carl-Zhou-CN left a comment

Hisoka-X left a comment

Hisoka-X Nov 19, 2024

PeppaPage Nov 20, 2024

	if (SupportSaveMode.class.isAssignableFrom(seaTunnelSink.getClass())) {
	if (seaTunnelSink instanceof SupportSaveMode) {

[Feature][Flink] Support multiple tables read and write #7713

[Feature][Flink] Support multiple tables read and write #7713

Conversation

PeppaPage commented Sep 22, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

hailin0 commented Sep 22, 2024

hailin0 commented Sep 22, 2024

PeppaPage commented Sep 22, 2024

PeppaPage commented Sep 22, 2024

Hisoka-X commented Sep 23, 2024 • edited Loading

Hisoka-X commented Sep 23, 2024

PeppaPage commented Sep 23, 2024

Hisoka-X commented Sep 23, 2024

PeppaPage commented Sep 23, 2024

PeppaPage commented Sep 24, 2024

TyrantLucifer commented Sep 24, 2024

hailin0 commented Sep 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PeppaPage commented Sep 24, 2024

hailin0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Carl-Zhou-CN commented Sep 25, 2024

Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X commented Sep 23, 2024 •

edited

Loading