When `source_table_name` is specified, the current plug-in is processing the data set corresponding to this parameter. | +:::warn + +The old configuration name `source_table_name` is deprecated, please migrate to the new name `plugin_input` as soon as possible. + +::: + +| Name | Type | Required | Default | Description | +|--------------|--------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| plugin_input | String | No | - | When `plugin_input` is not specified, the current plug-in processes the data set `dataset` output by the previous plugin in the configuration file
When `plugin_input` is specified, the current plug-in is processing the data set corresponding to this parameter. | # Important note -When the job configuration `source_table_name` you must set the `result_table_name` parameter +When the job configuration `plugin_input` you must set the `plugin_output` parameter ## Task Example @@ -24,34 +30,34 @@ When the job configuration `source_table_name` you must set the `result_table_na source { FakeSourceStream { parallelism = 2 - result_table_name = "fake" + plugin_output = "fake" field_name = "name,age" } } transform { Filter { - source_table_name = "fake" + plugin_input = "fake" fields = [name] - result_table_name = "fake_name" + plugin_output = "fake_name" } Filter { - source_table_name = "fake" + plugin_input = "fake" fields = [age] - result_table_name = "fake_age" + plugin_output = "fake_age" } } sink { Console { - source_table_name = "fake_name" + plugin_input = "fake_name" } Console { - source_table_name = "fake_age" + plugin_input = "fake_age" } } ``` -> If the job only have one source and one(or zero) transform and one sink, You do not need to specify `source_table_name` and `result_table_name` for connector. -> If the number of any operator in source, transform and sink is greater than 1, you must specify the `source_table_name` and `result_table_name` for each connector in the job. +> If the job only have one source and one(or zero) transform and one sink, You do not need to specify `plugin_input` and `plugin_output` for connector. +> If the number of any operator in source, transform and sink is greater than 1, you must specify the `plugin_input` and `plugin_output` for each connector in the job. diff --git a/docs/en/connector-v2/sink/AmazonSqs.md b/docs/en/connector-v2/sink/AmazonSqs.md index 8efabfa395b..4a43349b388 100644 --- a/docs/en/connector-v2/sink/AmazonSqs.md +++ b/docs/en/connector-v2/sink/AmazonSqs.md @@ -70,7 +70,7 @@ source { } } } - result_table_name = "fake" + plugin_output = "fake" } } diff --git a/docs/en/connector-v2/sink/Assert.md b/docs/en/connector-v2/sink/Assert.md index bc5b4c1bf32..026adddfae3 100644 --- a/docs/en/connector-v2/sink/Assert.md +++ b/docs/en/connector-v2/sink/Assert.md @@ -267,13 +267,13 @@ source { ] } ] - result_table_name = "fake" + plugin_output = "fake" } } sink{ Assert { - source_table_name = "fake" + plugin_input = "fake" rules = { row_rules = [ diff --git a/docs/en/connector-v2/sink/Console.md b/docs/en/connector-v2/sink/Console.md index a1c3b570baf..3493915d029 100644 --- a/docs/en/connector-v2/sink/Console.md +++ b/docs/en/connector-v2/sink/Console.md @@ -44,7 +44,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" schema = { fields { name = "string" @@ -56,7 +56,7 @@ source { sink { Console { - source_table_name = "fake" + plugin_input = "fake" } } ``` @@ -73,7 +73,7 @@ env { source { FakeSource { - result_table_name = "fake1" + plugin_output = "fake1" schema = { fields { id = "int" @@ -84,7 +84,7 @@ source { } } FakeSource { - result_table_name = "fake2" + plugin_output = "fake2" schema = { fields { name = "string" @@ -96,10 +96,10 @@ source { sink { Console { - source_table_name = "fake1" + plugin_input = "fake1" } Console { - source_table_name = "fake2" + plugin_input = "fake2" } } ``` diff --git a/docs/en/connector-v2/sink/DB2.md b/docs/en/connector-v2/sink/DB2.md index 92df20bd63d..7902c31f08c 100644 --- a/docs/en/connector-v2/sink/DB2.md +++ b/docs/en/connector-v2/sink/DB2.md @@ -101,7 +101,7 @@ source { # This is a example source plugin **only for test and demonstrate the feature source plugin** FakeSource { parallelism = 1 - result_table_name = "fake" + plugin_output = "fake" row.num = 16 schema = { fields { diff --git a/docs/en/connector-v2/sink/HdfsFile.md b/docs/en/connector-v2/sink/HdfsFile.md index 9c2aec0c54b..3060e8ac8b2 100644 --- a/docs/en/connector-v2/sink/HdfsFile.md +++ b/docs/en/connector-v2/sink/HdfsFile.md @@ -93,7 +93,7 @@ source { # This is a example source plugin **only for test and demonstrate the feature source plugin** FakeSource { parallelism = 1 - result_table_name = "fake" + plugin_output = "fake" row.num = 16 schema = { fields { diff --git a/docs/en/connector-v2/sink/Hive.md b/docs/en/connector-v2/sink/Hive.md index 147fd766a9f..20f3d22cb86 100644 --- a/docs/en/connector-v2/sink/Hive.md +++ b/docs/en/connector-v2/sink/Hive.md @@ -8,7 +8,7 @@ Write data to Hive. :::tip -In order to use this connector, You must ensure your spark/flink cluster already integrated hive. The tested hive version is 2.3.9. +In order to use this connector, You must ensure your spark/flink cluster already integrated hive. The tested hive version is 2.3.9 and 3.1.3 . If you use SeaTunnel Engine, You need put seatunnel-hadoop3-3.1.4-uber.jar and hive-exec-3.1.3.jar and libfb303-0.9.3.jar in $SEATUNNEL_HOME/lib/ dir. ::: @@ -182,6 +182,78 @@ sink { } ``` +### example2: Kerberos + +```bash +sink { + Hive { + table_name = "default.test_hive_sink_on_hdfs_with_kerberos" + metastore_uri = "thrift://metastore:9083" + hive_site_path = "/tmp/hive-site.xml" + kerberos_principal = "hive/metastore.seatunnel@EXAMPLE.COM" + kerberos_keytab_path = "/tmp/hive.keytab" + krb5_path = "/tmp/krb5.conf" + } +} +``` + +Description: + +- `hive_site_path`: The path to the `hive-site.xml` file. +- `kerberos_principal`: The principal for Kerberos authentication. +- `kerberos_keytab_path`: The keytab file path for Kerberos authentication. +- `krb5_path`: The path to the `krb5.conf` file used for Kerberos authentication. + +Run the case: + +```bash +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FakeSource { + schema = { + fields { + pk_id = bigint + name = string + score = int + } + primaryKey { + name = "pk_id" + columnNames = [pk_id] + } + } + rows = [ + { + kind = INSERT + fields = [1, "A", 100] + }, + { + kind = INSERT + fields = [2, "B", 100] + }, + { + kind = INSERT + fields = [3, "C", 100] + } + ] + } +} + +sink { + Hive { + table_name = "default.test_hive_sink_on_hdfs_with_kerberos" + metastore_uri = "thrift://metastore:9083" + hive_site_path = "/tmp/hive-site.xml" + kerberos_principal = "hive/metastore.seatunnel@EXAMPLE.COM" + kerberos_keytab_path = "/tmp/hive.keytab" + krb5_path = "/tmp/krb5.conf" + } +} +``` + ## Hive on s3 ### Step 1 @@ -395,26 +467,3 @@ sink { } } ``` - -## Changelog - -### 2.2.0-beta 2022-09-26 - -- Add Hive Sink Connector - -### 2.3.0-beta 2022-10-20 - -- [Improve] Hive Sink supports automatic partition repair ([3133](https://github.com/apache/seatunnel/pull/3133)) - -### 2.3.0 2022-12-30 - -- [BugFix] Fixed the following bugs that failed to write data to files ([3258](https://github.com/apache/seatunnel/pull/3258)) - - When field from upstream is null it will throw NullPointerException - - Sink columns mapping failed - - When restore writer from states getting transaction directly failed - -### Next version - -- [Improve] Support kerberos authentication ([3840](https://github.com/apache/seatunnel/pull/3840)) -- [Improve] Added partition_dir_expression validation logic ([3886](https://github.com/apache/seatunnel/pull/3886)) - diff --git a/docs/en/connector-v2/sink/Hudi.md b/docs/en/connector-v2/sink/Hudi.md index 6c424fde15e..ea4c066d2f8 100644 --- a/docs/en/connector-v2/sink/Hudi.md +++ b/docs/en/connector-v2/sink/Hudi.md @@ -8,7 +8,7 @@ Used to write data to Hudi. ## Key features -- [x] [exactly-once](../../concept/connector-v2-features.md) +- [ ] [exactly-once](../../concept/connector-v2-features.md) - [x] [cdc](../../concept/connector-v2-features.md) - [x] [support multiple table write](../../concept/connector-v2-features.md) @@ -21,7 +21,6 @@ Base configuration: | table_dfs_path | string | yes | - | | conf_files_path | string | no | - | | table_list | Array | no | - | -| auto_commit | boolean | no | true | | schema_save_mode | enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST| | common-options | Config | no | - | @@ -44,6 +43,7 @@ Table list configuration: | index_type | enum | no | BLOOM | | index_class_name | string | no | - | | record_byte_size | Int | no | 1024 | +| cdc_enabled | boolean| no | false | Note: When this configuration corresponds to a single table, you can flatten the configuration items in table_list to the outer layer. @@ -115,9 +115,9 @@ Note: When this configuration corresponds to a single table, you can flatten the `max_commits_to_keep` The max commits to keep of hudi table. -### auto_commit [boolean] +### cdc_enabled [boolean] -`auto_commit` Automatic transaction commit is enabled by default. +`cdc_enabled` Whether to persist the CDC change log. When enable, persist the change data if necessary, and the table can be queried as a CDC query mode. ### schema_save_mode [Enum] diff --git a/docs/en/connector-v2/sink/Iceberg.md b/docs/en/connector-v2/sink/Iceberg.md index 721c5ea7c08..54c46b849bf 100644 --- a/docs/en/connector-v2/sink/Iceberg.md +++ b/docs/en/connector-v2/sink/Iceberg.md @@ -59,7 +59,7 @@ libfb303-xxx.jar ## Sink Options -| Name | Type | Required | Default | Description | +| Name | Type | Required | Default | Description | |----------------------------------------|---------|----------|------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | catalog_name | string | yes | default | User-specified catalog name. default is `default` | | namespace | string | yes | default | The iceberg database name in the backend catalog. default is `default` | @@ -76,6 +76,7 @@ libfb303-xxx.jar | iceberg.table.upsert-mode-enabled | boolean | no | false | Set to `true` to enable upsert mode, default is `false` | | schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | the schema save mode, please refer to `schema_save_mode` below | | data_save_mode | Enum | no | APPEND_DATA | the data save mode, please refer to `data_save_mode` below | +| custom_sql | string | no | - | Custom `delete` data sql for data save mode. e.g: `delete from ... where ...` | | iceberg.table.commit-branch | string | no | - | Default branch for commits | ## Task Example @@ -91,7 +92,7 @@ env { source { MySQL-CDC { - result_table_name = "customers_mysql_cdc_iceberg" + plugin_output = "customers_mysql_cdc_iceberg" server-id = 5652 username = "st_user" password = "seatunnel" diff --git a/docs/en/connector-v2/sink/Jdbc.md b/docs/en/connector-v2/sink/Jdbc.md index 1ddbdd507d9..9b86a27721d 100644 --- a/docs/en/connector-v2/sink/Jdbc.md +++ b/docs/en/connector-v2/sink/Jdbc.md @@ -226,7 +226,7 @@ In the case of is_exactly_once = "true", Xa transactions are used. This requires there are some reference value for params above. -| datasource | driver | url | xa_data_source_class_name | maven | +| datasource | driver | url | xa_data_source_class_name | maven | |-------------------|----------------------------------------------|--------------------------------------------------------------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| | MySQL | com.mysql.cj.jdbc.Driver | jdbc:mysql://localhost:3306/test | com.mysql.cj.jdbc.MysqlXADataSource | https://mvnrepository.com/artifact/mysql/mysql-connector-java | | PostgreSQL | org.postgresql.Driver | jdbc:postgresql://localhost:5432/postgres | org.postgresql.xa.PGXADataSource | https://mvnrepository.com/artifact/org.postgresql/postgresql | @@ -235,7 +235,7 @@ there are some reference value for params above. | SQL Server | com.microsoft.sqlserver.jdbc.SQLServerDriver | jdbc:sqlserver://localhost:1433 | com.microsoft.sqlserver.jdbc.SQLServerXADataSource | https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc | | Oracle | oracle.jdbc.OracleDriver | jdbc:oracle:thin:@localhost:1521/xepdb1 | oracle.jdbc.xa.OracleXADataSource | https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8 | | sqlite | org.sqlite.JDBC | jdbc:sqlite:test.db | / | https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc | -| GBase8a | com.gbase.jdbc.Driver | jdbc:gbase://e2e_gbase8aDb:5258/test | / | https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar | +| GBase8a | com.gbase.jdbc.Driver | jdbc:gbase://e2e_gbase8aDb:5258/test | / | https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar | | StarRocks | com.mysql.cj.jdbc.Driver | jdbc:mysql://localhost:3306/test | / | https://mvnrepository.com/artifact/mysql/mysql-connector-java | | db2 | com.ibm.db2.jcc.DB2Driver | jdbc:db2://localhost:50000/testdb | com.ibm.db2.jcc.DB2XADataSource | https://mvnrepository.com/artifact/com.ibm.db2.jcc/db2jcc/db2jcc4 | | saphana | com.sap.db.jdbc.Driver | jdbc:sap://localhost:39015 | / | https://mvnrepository.com/artifact/com.sap.cloud.db.jdbc/ngdbc | @@ -245,9 +245,10 @@ there are some reference value for params above. | Snowflake | net.snowflake.client.jdbc.SnowflakeDriver | jdbc:snowflake://
+> Flink
+> SeaTunnel Zeta
+ +## Key Features + +- [ ] [exactly-once](../../concept/connector-v2-features.md) +- [ ] [cdc](../../concept/connector-v2-features.md) +- [x] [support multiple table write](../../concept/connector-v2-features.md) + +## Description + +Used to launch web hooks using data. + +> For example, if the data from upstream is [`label: {"__name__": "test1"}, value: 1.2.3,time:2024-08-15T17:00:00`], the body content is the following: `{"label":{"__name__": "test1"}, "value":"1.23","time":"2024-08-15T17:00:00"}` + +**Tips: Prometheus sink only support `post json` webhook and the data from source will be treated as body content in web hook.And does not support passing past data** + +## Supported DataSource Info + +In order to use the Http connector, the following dependencies are required. +They can be downloaded via install-plugin.sh or from the Maven central repository. + +| Datasource | Supported Versions | Dependency | +|------------|--------------------|------------------------------------------------------------------------------------------------------------------| +| Http | universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2/connector-prometheus) | + +## Sink Options + +| Name | Type | Required | Default | Description | +|-----------------------------|--------|----------|---------|-------------------------------------------------------------------------------------------------------------| +| url | String | Yes | - | Http request url | +| headers | Map | No | - | Http headers | +| retry | Int | No | - | The max retry times if request http return to `IOException` | +| retry_backoff_multiplier_ms | Int | No | 100 | The retry-backoff times(millis) multiplier if request http failed | +| retry_backoff_max_ms | Int | No | 10000 | The maximum retry-backoff times(millis) if request http failed | +| connect_timeout_ms | Int | No | 12000 | Connection timeout setting, default 12s. | +| socket_timeout_ms | Int | No | 60000 | Socket timeout setting, default 60s. | +| key_timestamp | Int | NO | - | prometheus timestamp key . | +| key_label | String | yes | - | prometheus label key | +| key_value | Double | yes | - | prometheus value | +| batch_size | Int | false | 1024 | prometheus batch size write | +| flush_interval | Long | false | 300000L | prometheus flush commit interval | +| common-options | | No | - | Sink plugin common parameters, please refer to [Sink Common Options](../sink-common-options.md) for details | + +## Example + +simple: + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FakeSource { + schema = { + fields { + c_map = "map
+> Flink
+> Seatunnel Zeta
+ +## Key Features + +- [ ] [exactly-once](../../concept/connector-v2-features.md) +- [ ] [cdc](../../concept/connector-v2-features.md) + +## Description + +Sink connector for Aliyun Sls. + +## Supported DataSource Info + +In order to use the Sls connector, the following dependencies are required. +They can be downloaded via install-plugin.sh or from the Maven central repository. + +| Datasource | Supported Versions | Maven | +|------------|--------------------|-----------------------------------------------------------------------------------| +| Sls | Universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-sls) | + +## Source Options + +| Name | Type | Required | Default | Description | +|-------------------------------------|---------|----------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------| +| project | String | Yes | - | [Aliyun Sls Project](https://help.aliyun.com/zh/sls/user-guide/manage-a-project?spm=a2c4g.11186623.0.0.6f9755ebyfaYSl) | +| logstore | String | Yes | - | [Aliyun Sls Logstore](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore?spm=a2c4g.11186623.0.0.13137c08nfuiBC) | +| endpoint | String | Yes | - | [Aliyun Access Endpoint](https://help.aliyun.com/zh/sls/developer-reference/api-sls-2020-12-30-endpoint?spm=a2c4g.11186623.0.0.548945a8UyJULa) | +| access_key_id | String | Yes | - | [Aliyun AccessKey ID](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479) | +| access_key_secret | String | Yes | - | [Aliyun AccessKey Secret](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479) | +| source | String | No | SeaTunnel-Source | Data Source marking in sls | +| topic | String | No | SeaTunnel-Topic | Data topic marking in sls | + +## Task Example + +### Simple + +> This example write data to the sls's logstore1.And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in Install SeaTunnel to install and deploy SeaTunnel. And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in [Install SeaTunnel](../../start-v2/locally/deployment.md) to install and deploy SeaTunnel. And then follow the instructions in [Quick Start With SeaTunnel Engine](../../start-v2/locally/quick-start-seatunnel-engine.md) to run this job. + +[Create RAM user and authorization](https://help.aliyun.com/zh/sls/create-a-ram-user-and-authorize-the-ram-user-to-access-log-service?spm=a2c4g.11186623.0.i4),Please ensure thr ram user have sufficient rights to perform, reference [RAM Custom Authorization Example](https://help.aliyun.com/zh/sls/use-custom-policies-to-grant-permissions-to-a-ram-user?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#reference-s3z-m1l-z2b) + +```hocon +# Defining the runtime environment +env { + parallelism = 2 + job.mode = "STREAMING" + checkpoint.interval = 30000 +} +source { + FakeSource { + row.num = 10 + map.size = 10 + array.size = 10 + bytes.length = 10 + string.length = 10 + schema = { + fields = { + id = "int" + name = "string" + description = "string" + weight = "string" + } + } + } +} + +sink { + Sls { + endpoint = "cn-hangzhou-intranet.log.aliyuncs.com" + project = "project1" + logstore = "logstore1" + access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx" + access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" + } +} +``` + diff --git a/docs/en/connector-v2/sink/Snowflake.md b/docs/en/connector-v2/sink/Snowflake.md index f40cb2b675d..dd84bcc2cef 100644 --- a/docs/en/connector-v2/sink/Snowflake.md +++ b/docs/en/connector-v2/sink/Snowflake.md @@ -89,7 +89,7 @@ source { # This is a example source plugin **only for test and demonstrate the feature source plugin** FakeSource { parallelism = 1 - result_table_name = "fake" + plugin_output = "fake" row.num = 16 schema = { fields { diff --git a/docs/en/connector-v2/sink/Socket.md b/docs/en/connector-v2/sink/Socket.md index 581a1a5caab..cdd5b7b034d 100644 --- a/docs/en/connector-v2/sink/Socket.md +++ b/docs/en/connector-v2/sink/Socket.md @@ -39,7 +39,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" schema = { fields { name = "string" diff --git a/docs/en/connector-v2/sink/SqlServer.md b/docs/en/connector-v2/sink/SqlServer.md index a9f6bdd8955..3a03d3a2df8 100644 --- a/docs/en/connector-v2/sink/SqlServer.md +++ b/docs/en/connector-v2/sink/SqlServer.md @@ -147,7 +147,7 @@ sink { ``` Jdbc { - source_table_name = "customers" + plugin_input = "customers" driver = com.microsoft.sqlserver.jdbc.SQLServerDriver url = "jdbc:sqlserver://localhost:1433;databaseName=column_type_test" user = SA diff --git a/docs/en/connector-v2/sink/Typesense.md b/docs/en/connector-v2/sink/Typesense.md index 8700d68dc77..f3c78af1617 100644 --- a/docs/en/connector-v2/sink/Typesense.md +++ b/docs/en/connector-v2/sink/Typesense.md @@ -77,7 +77,7 @@ Simple example: ```bash sink { Typesense { - source_table_name = "typesense_test_table" + plugin_input = "typesense_test_table" hosts = ["localhost:8108"] collection = "typesense_to_typesense_sink_with_query" max_retry_count = 3 diff --git a/docs/en/connector-v2/sink/Vertica.md b/docs/en/connector-v2/sink/Vertica.md index ef303b59453..04aa77f0e6f 100644 --- a/docs/en/connector-v2/sink/Vertica.md +++ b/docs/en/connector-v2/sink/Vertica.md @@ -109,7 +109,7 @@ source { # This is a example source plugin **only for test and demonstrate the feature source plugin** FakeSource { parallelism = 1 - result_table_name = "fake" + plugin_output = "fake" row.num = 16 schema = { fields { diff --git a/docs/en/connector-v2/source-common-options.md b/docs/en/connector-v2/source-common-options.md index a66eb34a44c..1c40f287796 100644 --- a/docs/en/connector-v2/source-common-options.md +++ b/docs/en/connector-v2/source-common-options.md @@ -6,14 +6,20 @@ sidebar_position: 3 > Common parameters of source connectors -| Name | Type | Required | Default | Description | -|-------------------|--------|----------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| result_table_name | String | No | - | When `result_table_name` is not specified, the data processed by this plugin will not be registered as a data set `(dataStream/dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)`
When `result_table_name` is specified, the data processed by this plugin will be registered as a data set `(dataStream/dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)` . The data set `(dataStream/dataset)` registered here can be directly accessed by other plugins by specifying `source_table_name` . | -| parallelism | Int | No | - | When `parallelism` is not specified, the `parallelism` in env is used by default.
When parallelism is specified, it will override the parallelism in env. | +:::warn + +The old configuration name `result_table_name` is deprecated, please migrate to the new name `plugin_output` as soon as possible. + +::: + +| Name | Type | Required | Default | Description | +|---------------|--------|----------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| plugin_output | String | No | - | When `plugin_output` is not specified, the data processed by this plugin will not be registered as a data set `(dataStream/dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)`
When `plugin_output` is specified, the data processed by this plugin will be registered as a data set `(dataStream/dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)` . The data set `(dataStream/dataset)` registered here can be directly accessed by other plugins by specifying `plugin_input` . | +| parallelism | Int | No | - | When `parallelism` is not specified, the `parallelism` in env is used by default.
When parallelism is specified, it will override the parallelism in env. | # Important note -When the job configuration `result_table_name` you must set the `source_table_name` parameter +When the job configuration `plugin_output` you must set the `plugin_input` parameter ## Task Example @@ -24,7 +30,7 @@ When the job configuration `result_table_name` you must set the `source_table_na ```bash source { FakeSourceStream { - result_table_name = "fake_table" + plugin_output = "fake_table" } } ``` @@ -40,7 +46,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -64,9 +70,9 @@ source { transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" - # the query table name must same as field 'source_table_name' + plugin_input = "fake" + plugin_output = "fake1" + # the query table name must same as field 'plugin_input' query = "select id, regexp_replace(name, '.+', 'b') as name, age+1 as age, pi() as pi, c_timestamp, c_date, c_map, c_array, c_decimal, c_row from fake" } # The SQL transform support base function and criteria operation @@ -75,10 +81,10 @@ transform { sink { Console { - source_table_name = "fake1" + plugin_input = "fake1" } Console { - source_table_name = "fake" + plugin_input = "fake" } } ``` diff --git a/docs/en/connector-v2/source/Cassandra.md b/docs/en/connector-v2/source/Cassandra.md index d4d4e97088a..32966de1e36 100644 --- a/docs/en/connector-v2/source/Cassandra.md +++ b/docs/en/connector-v2/source/Cassandra.md @@ -67,7 +67,7 @@ source { datacenter = "datacenter1" keyspace = "test" cql = "select * from source_table" - result_table_name = "source_table" + plugin_output = "source_table" } } ``` diff --git a/docs/en/connector-v2/source/Clickhouse.md b/docs/en/connector-v2/source/Clickhouse.md index 47907bd3025..e3048894ff7 100644 --- a/docs/en/connector-v2/source/Clickhouse.md +++ b/docs/en/connector-v2/source/Clickhouse.md @@ -80,7 +80,7 @@ source { username = "xxxxx" password = "xxxxx" server_time_zone = "UTC" - result_table_name = "test" + plugin_output = "test" clickhouse.config = { "socket_timeout": "300000" } diff --git a/docs/en/connector-v2/source/CosFile.md b/docs/en/connector-v2/source/CosFile.md index 702439c3062..1cbda880139 100644 --- a/docs/en/connector-v2/source/CosFile.md +++ b/docs/en/connector-v2/source/CosFile.md @@ -45,7 +45,7 @@ To use this connector you need put hadoop-cos-{hadoop.version}-{version}.jar and ## Options -| name | type | required | default value | +| name | type | required | default value | |---------------------------|---------|----------|---------------------| | path | string | yes | - | | file_format_type | string | yes | - | @@ -64,7 +64,7 @@ To use this connector you need put hadoop-cos-{hadoop.version}-{version}.jar and | sheet_name | string | no | - | | xml_row_tag | string | no | - | | xml_use_attr_format | boolean | no | - | -| file_filter_pattern | string | no | - | +| file_filter_pattern | string | no | | | compress_codec | string | no | none | | archive_compress_codec | string | no | none | | encoding | string | no | UTF-8 | @@ -275,6 +275,55 @@ Specifies Whether to process data using the tag attribute format. Filter pattern, which used for filtering files. +The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. +There are some examples. + +File Structure Example: +``` +/data/seatunnel/20241001/report.txt +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +/data/seatunnel/20241012/logo.png +``` +Matching Rules Example: + +**Example 1**: *Match all .txt files*，Regular Expression: +``` +/data/seatunnel/20241001/.*\.txt +``` +The result of this example matching is: +``` +/data/seatunnel/20241001/report.txt +``` +**Example 2**: *Match all file starting with abc*，Regular Expression: +``` +/data/seatunnel/20241002/abc.* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +``` +**Example 3**: *Match all file starting with abc，And the fourth character is either h or g*, the Regular Expression: +``` +/data/seatunnel/20241007/abc[h,g].* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +``` +**Example 4**: *Match third level folders starting with 202410 and files ending with .csv*, the Regular Expression: +``` +/data/seatunnel/202410\d*/.*\.csv +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +``` + ### compress_codec [string] The compress codec of files and the details that supported as the following shown: @@ -294,6 +343,7 @@ The compress codec of archive files and the details that supported as the follow | ZIP | txt,json,excel,xml | .zip | | TAR | txt,json,excel,xml | .tar | | TAR_GZ | txt,json,excel,xml | .tar.gz | +| GZ | txt,json,xml | .gz | | NONE | all | .* | ### encoding [string] @@ -372,6 +422,33 @@ sink { ``` +### Filter File + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + CosFile { + bucket = "cosn://seatunnel-test-1259587829" + secret_id = "xxxxxxxxxxxxxxxxxxx" + secret_key = "xxxxxxxxxxxxxxxxxxx" + region = "ap-chengdu" + path = "/seatunnel/read/binary/" + file_format_type = "binary" + // file example abcD2024.csv + file_filter_pattern = "abc[DX]*.*" + } +} + +sink { + Console { + } +} +``` + ## Changelog ### next version diff --git a/docs/en/connector-v2/source/Doris.md b/docs/en/connector-v2/source/Doris.md index c67444b58c8..373b84f8fdd 100644 --- a/docs/en/connector-v2/source/Doris.md +++ b/docs/en/connector-v2/source/Doris.md @@ -13,15 +13,14 @@ - [x] [batch](../../concept/connector-v2-features.md) - [ ] [stream](../../concept/connector-v2-features.md) - [ ] [exactly-once](../../concept/connector-v2-features.md) -- [x] [schema projection](../../concept/connector-v2-features.md) +- [x] [column projection](../../concept/connector-v2-features.md) - [x] [parallelism](../../concept/connector-v2-features.md) - [x] [support user-defined split](../../concept/connector-v2-features.md) +- [x] [support multiple table read](../../concept/connector-v2-features.md) ## Description -Used to read data from Doris. -Doris Source will send a SQL to FE, FE will parse it into an execution plan, send it to BE, and BE will -directly return the data +Used to read data from Apache Doris. ## Supported DataSource Info @@ -29,11 +28,6 @@ directly return the data |------------|--------------------------------------|--------|-----|-------| | Doris | Only Doris2.0 or later is supported. | - | - | - | -## Database Dependency - -> Please download the support list corresponding to 'Maven' and copy it to the '$SEATNUNNEL_HOME/plugins/jdbc/lib/' -> working directory
- ## Data Type Mapping | Doris Data type | SeaTunnel Data type | @@ -54,29 +48,40 @@ directly return the data ## Source Options +Base configuration: + | Name | Type | Required | Default | Description | |----------------------------------|--------|----------|------------|-----------------------------------------------------------------------------------------------------| | fenodes | string | yes | - | FE address, the format is `"fe_host:fe_http_port"` | | username | string | yes | - | User username | | password | string | yes | - | User password | +| doris.request.retries | int | no | 3 | Number of retries to send requests to Doris FE. | +| doris.request.read.timeout.ms | int | no | 30000 | | +| doris.request.connect.timeout.ms | int | no | 30000 | | +| query-port | string | no | 9030 | Doris QueryPort | +| doris.request.query.timeout.s | int | no | 3600 | Timeout period of Doris scan data, expressed in seconds. | +| table_list | string | 否 | - | table list | + +Table list configuration: + +| Name | Type | Required | Default | Description | +|----------------------------------|--------|----------|------------|-----------------------------------------------------------------------------------------------------| | database | string | yes | - | The name of Doris database | | table | string | yes | - | The name of Doris table | | doris.read.field | string | no | - | Use the 'doris.read.field' parameter to select the doris table columns to read | -| query-port | string | no | 9030 | Doris QueryPort | | doris.filter.query | string | no | - | Data filtering in doris. the format is "field = value",example : doris.filter.query = "F_ID > 2" | | doris.batch.size | int | no | 1024 | The maximum value that can be obtained by reading Doris BE once. | -| doris.request.query.timeout.s | int | no | 3600 | Timeout period of Doris scan data, expressed in seconds. | | doris.exec.mem.limit | long | no | 2147483648 | Maximum memory that can be used by a single be scan request. The default memory is 2G (2147483648). | -| doris.request.retries | int | no | 3 | Number of retries to send requests to Doris FE. | -| doris.request.read.timeout.ms | int | no | 30000 | | -| doris.request.connect.timeout.ms | int | no | 30000 | | + +Note: When this configuration corresponds to a single table, you can flatten the configuration items in table_list to the outer layer. ### Tips > It is not recommended to modify advanced parameters at will -## Task Example +## Example +### single table > This is an example of reading a Doris table and writing to Console. ``` @@ -159,4 +164,49 @@ sink { Console {} } ``` +### Multiple table +``` +env{ + parallelism = 1 + job.mode = "BATCH" +} +source{ + Doris { + fenodes = "xxxx:8030" + username = root + password = "" + table_list = [ + { + database = "st_source_0" + table = "doris_table_0" + doris.read.field = "F_ID,F_INT,F_BIGINT,F_TINYINT" + doris.filter.query = "F_ID >= 50" + }, + { + database = "st_source_1" + table = "doris_table_1" + } + ] + } +} + +transform {} + +sink{ + Doris { + fenodes = "xxxx:8030" + schema_save_mode = "RECREATE_SCHEMA" + username = root + password = "" + database = "st_sink" + table = "${table_name}" + sink.enable-2pc = "true" + sink.label-prefix = "test_json" + doris.config = { + format="json" + read_json_by_line="true" + } + } +} +``` diff --git a/docs/en/connector-v2/source/FakeSource.md b/docs/en/connector-v2/source/FakeSource.md index 6f6b259736b..48333e244c0 100644 --- a/docs/en/connector-v2/source/FakeSource.md +++ b/docs/en/connector-v2/source/FakeSource.md @@ -142,7 +142,7 @@ source { c_timestamp = timestamp } } - result_table_name = "fake" + plugin_output = "fake" } } ``` diff --git a/docs/en/connector-v2/source/FtpFile.md b/docs/en/connector-v2/source/FtpFile.md index 656f7a00422..f65255bfd77 100644 --- a/docs/en/connector-v2/source/FtpFile.md +++ b/docs/en/connector-v2/source/FtpFile.md @@ -38,7 +38,7 @@ If you use SeaTunnel Engine, It automatically integrated the hadoop jar when you ## Options -| name | type | required | default value | +| name | type | required | default value | |---------------------------|---------|----------|---------------------| | host | string | yes | - | | port | int | yes | - | @@ -62,6 +62,7 @@ If you use SeaTunnel Engine, It automatically integrated the hadoop jar when you | compress_codec | string | no | none | | archive_compress_codec | string | no | none | | encoding | string | no | UTF-8 | +| null_format | string | no | - | | common-options | | no | - | ### host [string] @@ -84,6 +85,59 @@ The target ftp password is required The source file path. +### file_filter_pattern [string] + +Filter pattern, which used for filtering files. + +The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. +There are some examples. + +File Structure Example: +``` +/data/seatunnel/20241001/report.txt +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +/data/seatunnel/20241012/logo.png +``` +Matching Rules Example: + +**Example 1**: *Match all .txt files*，Regular Expression: +``` +/data/seatunnel/20241001/.*\.txt +``` +The result of this example matching is: +``` +/data/seatunnel/20241001/report.txt +``` +**Example 2**: *Match all file starting with abc*，Regular Expression: +``` +/data/seatunnel/20241002/abc.* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +``` +**Example 3**: *Match all file starting with abc，And the fourth character is either h or g*, the Regular Expression: +``` +/data/seatunnel/20241007/abc[h,g].* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +``` +**Example 4**: *Match third level folders starting with 202410 and files ending with .csv*, the Regular Expression: +``` +/data/seatunnel/202410\d*/.*\.csv +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +``` + ### file_format_type [string] File type, supported as the following file types: @@ -275,6 +329,7 @@ The compress codec of archive files and the details that supported as the follow | ZIP | txt,json,excel,xml | .zip | | TAR | txt,json,excel,xml | .tar | | TAR_GZ | txt,json,excel,xml | .tar.gz | +| GZ | txt,json,xml | .gz | | NONE | all | .* | ### encoding [string] @@ -282,6 +337,13 @@ The compress codec of archive files and the details that supported as the follow Only used when file_format_type is json,text,csv,xml. The encoding of the file to read. This param will be parsed by `Charset.forName(encoding)`. +### null_format [string] + +Only used when file_format_type is text. +null_format to define which strings can be represented as null. + +e.g: `\N` + ### common options Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details. @@ -306,6 +368,67 @@ Source plugin common parameters, please refer to [Source Common Options](../sour ``` +### Multiple Table + +```hocon + +FtpFile { + tables_configs = [ + { + schema { + table = "student" + } + path = "/tmp/seatunnel/sink/text" + host = "192.168.31.48" + port = 21 + user = tyrantlucifer + password = tianchao + file_format_type = "parquet" + }, + { + schema { + table = "teacher" + } + path = "/tmp/seatunnel/sink/text" + host = "192.168.31.48" + port = 21 + user = tyrantlucifer + password = tianchao + file_format_type = "parquet" + } + ] +} + +``` + +```hocon + +FtpFile { + tables_configs = [ + { + schema { + fields { + name = string + age = int + } + } + path = "/apps/hive/demo/student" + file_format_type = "json" + }, + { + schema { + fields { + name = string + age = int + } + } + path = "/apps/hive/demo/teacher" + file_format_type = "json" + } +} + +``` + ### Transfer Binary File ```hocon @@ -339,6 +462,33 @@ sink { ``` +### Filter File + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FtpFile { + host = "192.168.31.48" + port = 21 + user = tyrantlucifer + password = tianchao + path = "/seatunnel/read/binary/" + file_format_type = "binary" + // file example abcD2024.csv + file_filter_pattern = "abc[DX]*.*" + } +} + +sink { + Console { + } +} +``` + ## Changelog ### 2.2.0-beta 2022-09-26 diff --git a/docs/en/connector-v2/source/HdfsFile.md b/docs/en/connector-v2/source/HdfsFile.md index 7413c0428b8..caaf9972a06 100644 --- a/docs/en/connector-v2/source/HdfsFile.md +++ b/docs/en/connector-v2/source/HdfsFile.md @@ -41,7 +41,7 @@ Read data from hdfs file system. ## Source Options -| Name | Type | Required | Default | Description | +| Name | Type | Required | Default | Description | |---------------------------|---------|----------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | path | string | yes | - | The source file path. | | file_format_type | string | yes | - | We supported as the following file types:`text` `csv` `parquet` `orc` `json` `excel` `xml` `binary`.Please note that, The final file name will end with the file_format's suffix, the suffix of the text file is `txt`. | @@ -62,15 +62,70 @@ Read data from hdfs file system. | sheet_name | string | no | - | Reader the sheet of the workbook,Only used when file_format is excel. | | xml_row_tag | string | no | - | Specifies the tag name of the data rows within the XML file, only used when file_format is xml. | | xml_use_attr_format | boolean | no | - | Specifies whether to process data using the tag attribute format, only used when file_format is xml. | +| file_filter_pattern | string | no | | Filter pattern, which used for filtering files. | | compress_codec | string | no | none | The compress codec of files | | archive_compress_codec | string | no | none | | encoding | string | no | UTF-8 | | +| null_format | string | no | - | Only used when file_format_type is text. null_format to define which strings can be represented as null. e.g: `\N` | | common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details. | ### delimiter/field_delimiter [string] **delimiter** parameter will deprecate after version 2.3.5, please use **field_delimiter** instead. +### file_filter_pattern [string] + +Filter pattern, which used for filtering files. + +The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. +There are some examples. + +File Structure Example: +``` +/data/seatunnel/20241001/report.txt +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +/data/seatunnel/20241012/logo.png +``` +Matching Rules Example: + +**Example 1**: *Match all .txt files*，Regular Expression: +``` +/data/seatunnel/20241001/.*\.txt +``` +The result of this example matching is: +``` +/data/seatunnel/20241001/report.txt +``` +**Example 2**: *Match all file starting with abc*，Regular Expression: +``` +/data/seatunnel/20241002/abc.* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +``` +**Example 3**: *Match all file starting with abc，And the fourth character is either h or g*, the Regular Expression: +``` +/data/seatunnel/20241007/abc[h,g].* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +``` +**Example 4**: *Match third level folders starting with 202410 and files ending with .csv*, the Regular Expression: +``` +/data/seatunnel/202410\d*/.*\.csv +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +``` + ### compress_codec [string] The compress codec of files and the details that supported as the following shown: @@ -90,6 +145,7 @@ The compress codec of archive files and the details that supported as the follow | ZIP | txt,json,excel,xml | .zip | | TAR | txt,json,excel,xml | .tar | | TAR_GZ | txt,json,excel,xml | .tar.gz | +| GZ | txt,json,xml | .gz | | NONE | all | .* | ### encoding [string] @@ -146,3 +202,26 @@ sink { } ``` +### Filter File + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + HdfsFile { + path = "/apps/hive/demo/student" + file_format_type = "json" + fs.defaultFS = "hdfs://namenode001" + // file example abcD2024.csv + file_filter_pattern = "abc[DX]*.*" + } +} + +sink { + Console { + } +} +``` diff --git a/docs/en/connector-v2/source/Hive.md b/docs/en/connector-v2/source/Hive.md index 5669906c3b9..d87739f1034 100644 --- a/docs/en/connector-v2/source/Hive.md +++ b/docs/en/connector-v2/source/Hive.md @@ -8,7 +8,7 @@ Read data from Hive. :::tip -In order to use this connector, You must ensure your spark/flink cluster already integrated hive. The tested hive version is 2.3.9. +In order to use this connector, You must ensure your spark/flink cluster already integrated hive. The tested hive version is 2.3.9 and 3.1.3 . If you use SeaTunnel Engine, You need put seatunnel-hadoop3-3.1.4-uber.jar and hive-exec-3.1.3.jar and libfb303-0.9.3.jar in $SEATUNNEL_HOME/lib/ dir. ::: @@ -120,6 +120,24 @@ Source plugin common parameters, please refer to [Source Common Options](../sour ``` ### Example 2: Multiple tables +> Note: Hive is a structured data source and should be use 'table_list', and 'tables_configs' will be removed in the future. + +```bash + + Hive { + table_list = [ + { + table_name = "default.seatunnel_orc_1" + metastore_uri = "thrift://namenode001:9083" + }, + { + table_name = "default.seatunnel_orc_2" + metastore_uri = "thrift://namenode001:9083" + } + ] + } + +``` ```bash @@ -138,6 +156,95 @@ Source plugin common parameters, please refer to [Source Common Options](../sour ``` +### Example3 : Kerberos + +```bash +source { + Hive { + table_name = "default.test_hive_sink_on_hdfs_with_kerberos" + metastore_uri = "thrift://metastore:9083" + hive.hadoop.conf-path = "/tmp/hadoop" + result_table_name = hive_source + hive_site_path = "/tmp/hive-site.xml" + kerberos_principal = "hive/metastore.seatunnel@EXAMPLE.COM" + kerberos_keytab_path = "/tmp/hive.keytab" + krb5_path = "/tmp/krb5.conf" + } +} +``` + +Description: + +- `hive_site_path`: The path to the `hive-site.xml` file. +- `kerberos_principal`: The principal for Kerberos authentication. +- `kerberos_keytab_path`: The keytab file path for Kerberos authentication. +- `krb5_path`: The path to the `krb5.conf` file used for Kerberos authentication. + +Run the case: + +```bash +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + Hive { + table_name = "default.test_hive_sink_on_hdfs_with_kerberos" + metastore_uri = "thrift://metastore:9083" + hive.hadoop.conf-path = "/tmp/hadoop" + result_table_name = hive_source + hive_site_path = "/tmp/hive-site.xml" + kerberos_principal = "hive/metastore.seatunnel@EXAMPLE.COM" + kerberos_keytab_path = "/tmp/hive.keytab" + krb5_path = "/tmp/krb5.conf" + } +} + +sink { + Assert { + source_table_name = hive_source + rules { + row_rules = [ + { + rule_type = MAX_ROW + rule_value = 3 + } + ], + field_rules = [ + { + field_name = pk_id + field_type = bigint + field_value = [ + { + rule_type = NOT_NULL + } + ] + }, + { + field_name = name + field_type = string + field_value = [ + { + rule_type = NOT_NULL + } + ] + }, + { + field_name = score + field_type = int + field_value = [ + { + rule_type = NOT_NULL + } + ] + } + ] + } + } +} +``` + ## Hive on s3 ### Step 1 @@ -265,15 +372,3 @@ sink { } } ``` - -## Changelog - -### 2.2.0-beta 2022-09-26 - -- Add Hive Source Connector - -### Next version - -- [Improve] Support kerberos authentication ([3840](https://github.com/apache/seatunnel/pull/3840)) -- Support user-defined partitions ([3842](https://github.com/apache/seatunnel/pull/3842)) - diff --git a/docs/en/connector-v2/source/HiveJdbc.md b/docs/en/connector-v2/source/HiveJdbc.md index 19619d924c1..23227aa306f 100644 --- a/docs/en/connector-v2/source/HiveJdbc.md +++ b/docs/en/connector-v2/source/HiveJdbc.md @@ -72,7 +72,7 @@ Read external data source data through JDBC. | partition_num | Int | No | job parallelism | The number of partition count, only support positive integer. default value is job parallelism | | fetch_size | Int | No | 0 | For queries that return a large number of objects,you can configure
the row fetch size used in the query toimprove performance by
reducing the number database hits required to satisfy the selection criteria.
Zero means use jdbc default value. | | common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details | -| useKerberos | Boolean | No | no | Whether to enable Kerberos, default is false | +| use_kerberos | Boolean | No | no | Whether to enable Kerberos, default is false | | kerberos_principal | String | No | - | When use kerberos, we should set kerberos principal such as 'test_user@xxx'. | | kerberos_keytab_path | String | No | - | When use kerberos, we should set kerberos principal file path such as '/home/test/test_user.keytab' . | | krb5_path | String | No | /etc/krb5.conf | When use kerberos, we should set krb5 path file path such as '/seatunnel/krb5.conf' or use the default path '/etc/krb5.conf '. | diff --git a/docs/en/connector-v2/source/Http.md b/docs/en/connector-v2/source/Http.md index 9c60b4c9aa4..511ba04132d 100644 --- a/docs/en/connector-v2/source/Http.md +++ b/docs/en/connector-v2/source/Http.md @@ -78,7 +78,7 @@ env { source { Http { - result_table_name = "http" + plugin_output = "http" url = "http://mockserver:1080/example/http" method = "GET" format = "json" diff --git a/docs/en/connector-v2/source/Iceberg.md b/docs/en/connector-v2/source/Iceberg.md index 4203c85bb87..8bb21eb7b63 100644 --- a/docs/en/connector-v2/source/Iceberg.md +++ b/docs/en/connector-v2/source/Iceberg.md @@ -127,7 +127,7 @@ source { } namespace = "database1" table = "source" - result_table_name = "iceberg" + plugin_output = "iceberg" } } @@ -136,7 +136,7 @@ transform { sink { Console { - source_table_name = "iceberg" + plugin_input = "iceberg" } } ``` @@ -160,7 +160,7 @@ source { } namespace = "your_iceberg_database" table = "your_iceberg_table" - result_table_name = "iceberg_test" + plugin_output = "iceberg_test" } } ``` diff --git a/docs/en/connector-v2/source/Jdbc.md b/docs/en/connector-v2/source/Jdbc.md index 27b3d875580..2b5897cbaea 100644 --- a/docs/en/connector-v2/source/Jdbc.md +++ b/docs/en/connector-v2/source/Jdbc.md @@ -113,7 +113,7 @@ The JDBC Source connector supports parallel reading of data from tables. SeaTunn there are some reference value for params above. -| datasource | driver | url | maven | +| datasource | driver | url | maven | |-------------------|-----------------------------------------------------|------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| | mysql | com.mysql.cj.jdbc.Driver | jdbc:mysql://localhost:3306/test | https://mvnrepository.com/artifact/mysql/mysql-connector-java | | postgresql | org.postgresql.Driver | jdbc:postgresql://localhost:5432/postgres | https://mvnrepository.com/artifact/org.postgresql/postgresql | @@ -122,7 +122,7 @@ there are some reference value for params above. | sqlserver | com.microsoft.sqlserver.jdbc.SQLServerDriver | jdbc:sqlserver://localhost:1433 | https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc | | oracle | oracle.jdbc.OracleDriver | jdbc:oracle:thin:@localhost:1521/xepdb1 | https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8 | | sqlite | org.sqlite.JDBC | jdbc:sqlite:test.db | https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc | -| gbase8a | com.gbase.jdbc.Driver | jdbc:gbase://e2e_gbase8aDb:5258/test | https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar | +| gbase8a | com.gbase.jdbc.Driver | jdbc:gbase://e2e_gbase8aDb:5258/test | https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar | | starrocks | com.mysql.cj.jdbc.Driver | jdbc:mysql://localhost:3306/test | https://mvnrepository.com/artifact/mysql/mysql-connector-java | | db2 | com.ibm.db2.jcc.DB2Driver | jdbc:db2://localhost:50000/testdb | https://mvnrepository.com/artifact/com.ibm.db2.jcc/db2jcc/db2jcc4 | | tablestore | com.alicloud.openservices.tablestore.jdbc.OTSDriver | "jdbc:ots:http s://myinstance.cn-hangzhou.ots.aliyuncs.com/myinstance" | https://mvnrepository.com/artifact/com.aliyun.openservices/tablestore-jdbc | @@ -133,10 +133,11 @@ there are some reference value for params above. | Redshift | com.amazon.redshift.jdbc42.Driver | jdbc:redshift://localhost:5439/testdb?defaultRowFetchSize=1000 | https://mvnrepository.com/artifact/com.amazon.redshift/redshift-jdbc42 | | Vertica | com.vertica.jdbc.Driver | jdbc:vertica://localhost:5433 | https://repo1.maven.org/maven2/com/vertica/jdbc/vertica-jdbc/12.0.3-0/vertica-jdbc-12.0.3-0.jar | | Kingbase | com.kingbase8.Driver | jdbc:kingbase8://localhost:54321/db_test | https://repo1.maven.org/maven2/cn/com/kingbase/kingbase8/8.6.0/kingbase8-8.6.0.jar | -| OceanBase | com.oceanbase.jdbc.Driver | jdbc:oceanbase://localhost:2881 | https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/2.4.11/oceanbase-client-2.4.11.jar | +| OceanBase | com.oceanbase.jdbc.Driver | jdbc:oceanbase://localhost:2881 | https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/2.4.12/oceanbase-client-2.4.12.jar | | Hive | org.apache.hive.jdbc.HiveDriver | jdbc:hive2://localhost:10000 | https://repo1.maven.org/maven2/org/apache/hive/hive-jdbc/3.1.3/hive-jdbc-3.1.3-standalone.jar | | xugu | com.xugu.cloudjdbc.Driver | jdbc:xugu://localhost:5138 | https://repo1.maven.org/maven2/com/xugudb/xugu-jdbc/12.2.0/xugu-jdbc-12.2.0.jar | | InterSystems IRIS | com.intersystems.jdbc.IRISDriver | jdbc:IRIS://localhost:1972/%SYS | https://raw.githubusercontent.com/intersystems-community/iris-driver-distribution/main/JDBC/JDK18/intersystems-jdbc-3.8.4.jar | +| opengauss | org.opengauss.Driver | jdbc:opengauss://localhost:5432/postgres | https://repo1.maven.org/maven2/org/opengauss/opengauss-jdbc/5.1.0-og/opengauss-jdbc-5.1.0-og.jar | ## Example diff --git a/docs/en/connector-v2/source/kafka.md b/docs/en/connector-v2/source/Kafka.md similarity index 94% rename from docs/en/connector-v2/source/kafka.md rename to docs/en/connector-v2/source/Kafka.md index 90c183c2c13..dfc23a7572f 100644 --- a/docs/en/connector-v2/source/kafka.md +++ b/docs/en/connector-v2/source/Kafka.md @@ -59,6 +59,7 @@ They can be downloaded via install-plugin.sh or from the Maven central repositor ### Simple > This example reads the data of kafka's topic_1, topic_2, topic_3 and prints it to the client.And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in Install SeaTunnel to install and deploy SeaTunnel. And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in [Install SeaTunnel](../../start-v2/locally/deployment.md) to install and deploy SeaTunnel. And then follow the instructions in [Quick Start With SeaTunnel Engine](../../start-v2/locally/quick-start-seatunnel-engine.md) to run this job. +> In batch mode, during the enumerator sharding process, it will fetch the latest offset for each partition and use it as the stopping point. ```hocon # Defining the runtime environment @@ -188,6 +189,65 @@ source { > This is written to the same pg table according to different formats and topics of parsing kafka Perform upsert operations based on the id +> Note: Kafka is an unstructured data source and should be use 'tables_configs', and 'table_list' will be removed in the future. + +```hocon + +env { + execution.parallelism = 1 + job.mode = "BATCH" +} + +source { + Kafka { + bootstrap.servers = "kafka_e2e:9092" + tables_configs = [ + { + topic = "^test-ogg-sou.*" + pattern = "true" + consumer.group = "ogg_multi_group" + start_mode = earliest + schema = { + fields { + id = "int" + name = "string" + description = "string" + weight = "string" + } + }, + format = ogg_json + }, + { + topic = "test-cdc_mds" + start_mode = earliest + schema = { + fields { + id = "int" + name = "string" + description = "string" + weight = "string" + } + }, + format = canal_json + } + ] + } +} + +sink { + Jdbc { + driver = org.postgresql.Driver + url = "jdbc:postgresql://postgresql:5432/test?loggerLevel=OFF" + user = test + password = test + generate_sink_sql = true + database = test + table = public.sink + primary_keys = ["id"] + } +} +``` + ```hocon env { @@ -289,7 +349,7 @@ source { """ bootstrap.servers = "kafkaCluster:9092" start_mode = "earliest" - result_table_name = "kafka_table" + plugin_output = "kafka_table" } } ``` diff --git a/docs/en/connector-v2/source/Klaviyo.md b/docs/en/connector-v2/source/Klaviyo.md index 10b4ed42e9e..848fe38ef8f 100644 --- a/docs/en/connector-v2/source/Klaviyo.md +++ b/docs/en/connector-v2/source/Klaviyo.md @@ -45,7 +45,7 @@ http request url API private key for login, you can get more detail at this link: -https://developers.klaviyo.com/en/docs/retrieve_api_credentials +https://developers.klaviyo.com/en/docs/authenticate_#private-key-authentication ### revision [String] diff --git a/docs/en/connector-v2/source/Kudu.md b/docs/en/connector-v2/source/Kudu.md index ccd63e090b3..a6fee76f12c 100644 --- a/docs/en/connector-v2/source/Kudu.md +++ b/docs/en/connector-v2/source/Kudu.md @@ -78,7 +78,7 @@ source { kudu { kudu_masters = "kudu-master:7051" table_name = "kudu_source_table" - result_table_name = "kudu" + plugin_output = "kudu" enable_kerberos = true kerberos_principal = "xx@xx.COM" kerberos_keytab = "xx.keytab" @@ -90,11 +90,11 @@ transform { sink { console { - source_table_name = "kudu" + plugin_input = "kudu" } kudu { - source_table_name = "kudu" + plugin_input = "kudu" kudu_masters = "kudu-master:7051" table_name = "kudu_sink_table" enable_kerberos = true @@ -125,7 +125,7 @@ source { table_name = "kudu_source_table_2" } ] - result_table_name = "kudu" + plugin_output = "kudu" } } diff --git a/docs/en/connector-v2/source/LocalFile.md b/docs/en/connector-v2/source/LocalFile.md index 6d11b992e3a..477a4d41399 100644 --- a/docs/en/connector-v2/source/LocalFile.md +++ b/docs/en/connector-v2/source/LocalFile.md @@ -43,7 +43,7 @@ If you use SeaTunnel Engine, It automatically integrated the hadoop jar when you ## Options -| name | type | required | default value | +| name | type | required | default value | |---------------------------|---------|----------|--------------------------------------| | path | string | yes | - | | file_format_type | string | yes | - | @@ -58,10 +58,11 @@ If you use SeaTunnel Engine, It automatically integrated the hadoop jar when you | sheet_name | string | no | - | | xml_row_tag | string | no | - | | xml_use_attr_format | boolean | no | - | -| file_filter_pattern | string | no | - | +| file_filter_pattern | string | no | | | compress_codec | string | no | none | | archive_compress_codec | string | no | none | | encoding | string | no | UTF-8 | +| null_format | string | no | - | | common-options | | no | - | | tables_configs | list | no | used to define a multiple table task | @@ -254,6 +255,55 @@ Specifies Whether to process data using the tag attribute format. Filter pattern, which used for filtering files. +The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. +There are some examples. + +File Structure Example: +``` +/data/seatunnel/20241001/report.txt +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +/data/seatunnel/20241012/logo.png +``` +Matching Rules Example: + +**Example 1**: *Match all .txt files*，Regular Expression: +``` +/data/seatunnel/20241001/.*\.txt +``` +The result of this example matching is: +``` +/data/seatunnel/20241001/report.txt +``` +**Example 2**: *Match all file starting with abc*，Regular Expression: +``` +/data/seatunnel/20241002/abc.* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +``` +**Example 3**: *Match all file starting with abc，And the fourth character is either h or g*, the Regular Expression: +``` +/data/seatunnel/20241007/abc[h,g].* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +``` +**Example 4**: *Match third level folders starting with 202410 and files ending with .csv*, the Regular Expression: +``` +/data/seatunnel/202410\d*/.*\.csv +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +``` + ### compress_codec [string] The compress codec of files and the details that supported as the following shown: @@ -273,6 +323,7 @@ The compress codec of archive files and the details that supported as the follow | ZIP | txt,json,excel,xml | .zip | | TAR | txt,json,excel,xml | .tar | | TAR_GZ | txt,json,excel,xml | .tar.gz | +| GZ | txt,json,xml | .gz | | NONE | all | .* | ### encoding [string] @@ -280,6 +331,13 @@ The compress codec of archive files and the details that supported as the follow Only used when file_format_type is json,text,csv,xml. The encoding of the file to read. This param will be parsed by `Charset.forName(encoding)`. +### null_format [string] + +Only used when file_format_type is text. +null_format to define which strings can be represented as null. + +e.g: `\N` + ### common options Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details @@ -406,6 +464,30 @@ sink { ``` +### Filter File + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + LocalFile { + path = "/data/seatunnel/" + file_format_type = "csv" + skip_header_row_number = 1 + // file example abcD2024.csv + file_filter_pattern = "abc[DX]*.*" + } +} + +sink { + Console { + } +} +``` + ## Changelog ### 2.2.0-beta 2022-09-26 @@ -417,4 +499,6 @@ sink { - [BugFix] Fix the bug of incorrect path in windows environment ([2980](https://github.com/apache/seatunnel/pull/2980)) - [Improve] Support extract partition from SeaTunnelRow fields ([3085](https://github.com/apache/seatunnel/pull/3085)) - [Improve] Support parse field from file path ([2985](https://github.com/apache/seatunnel/pull/2985)) +### 2.3.9-beta 2024-11-12 +- [Improve] Support parse field from file path ([8019](https://github.com/apache/seatunnel/issues/8019)) diff --git a/docs/en/connector-v2/source/Mivlus.md b/docs/en/connector-v2/source/Milvus.md similarity index 85% rename from docs/en/connector-v2/source/Mivlus.md rename to docs/en/connector-v2/source/Milvus.md index a56df4c5fe7..e9560489762 100644 --- a/docs/en/connector-v2/source/Mivlus.md +++ b/docs/en/connector-v2/source/Milvus.md @@ -4,7 +4,11 @@ ## Description -Read data from Milvus or Zilliz Cloud +This Milvus source connector reads data from Milvus or Zilliz Cloud, it has the following features: +- support read and write data by partition +- support read dynamic schema data into Metadata Column +- json data will be converted to json string and sink as json as well +- retry automatically to bypass ratelimit and grpc limit ## Key Features @@ -53,3 +57,5 @@ source { } ``` +## Changelog + diff --git a/docs/en/connector-v2/source/MongoDB-CDC.md b/docs/en/connector-v2/source/MongoDB-CDC.md index 301d7075738..d7e6c7e440f 100644 --- a/docs/en/connector-v2/source/MongoDB-CDC.md +++ b/docs/en/connector-v2/source/MongoDB-CDC.md @@ -105,13 +105,14 @@ For specific types in MongoDB, we use Extended JSON format to map them to Seatun ## Source Options -| Name | Type | Required | Default | Description | +| Name | Type | Required | Default | Description | |------------------------------------|--------|----------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | hosts | String | Yes | - | The comma-separated list of hostname and port pairs of the MongoDB servers. eg. `localhost:27017,localhost:27018` | | username | String | No | - | Name of the database user to be used when connecting to MongoDB. | | password | String | No | - | Password to be used when connecting to MongoDB. | | database | List | Yes | - | Name of the database to watch for changes. If not set then all databases will be captured. The database also supports regular expressions to monitor multiple databases matching the regular expression. eg. `db1,db2`. | | collection | List | Yes | - | Name of the collection in the database to watch for changes. If not set then all collections will be captured. The collection also supports regular expressions to monitor multiple collections matching fully-qualified collection identifiers. eg. `db1.coll1,db2.coll2`. | +| schema | | yes | - | The structure of the data, including field names and field types. | | connection.options | String | No | - | The ampersand-separated connection options of MongoDB. eg. `replicaSet=test&connectTimeoutMS=300000`. | | batch.size | Long | No | 1024 | The cursor batch size. | | poll.max.batch.size | Enum | No | 1024 | Maximum number of change stream documents to include in a single batch when polling for new data. | @@ -185,6 +186,14 @@ source { collection = ["inventory.products"] username = stuser password = stpw + schema = { + fields { + "_id" : string, + "name" : string, + "description" : string, + "weight" : string + } + } } } @@ -204,76 +213,6 @@ sink { } ``` -## Multi-table Synchronization - -The following example demonstrates how to create a data synchronization job that read the cdc data of multiple library tables mongodb and prints it on the local client: - -```hocon -env { - # You can set engine configuration here - parallelism = 1 - job.mode = "STREAMING" - checkpoint.interval = 5000 -} - -source { - MongoDB-CDC { - hosts = "mongo0:27017" - database = ["inventory","crm"] - collection = ["inventory.products","crm.test"] - username = stuser - password = stpw - } -} - -# Console printing of the read Mongodb data -sink { - Console { - parallelism = 1 - } -} -``` - -### Tips: - -> 1.The cdc synchronization of multiple library tables cannot specify the schema, and can only output json data downstream. -> This is because MongoDB does not provide metadata information for querying, so if you want to support multiple tables, all tables can only be read as one structure. - -## Regular Expression Matching for Multiple Tables - -The following example demonstrates how to create a data synchronization job that through regular expression read the data of multiple library tables mongodb and prints it on the local client: - -| Matching example | Expressions | | Describe | -|------------------|-------------|---|----------------------------------------------------------------------------------------| -| Prefix matching | ^(test).* | | Match the database name or table name with the prefix test, such as test1, test2, etc. | -| Suffix matching | .*[p$] | | Match the database name or table name with the suffix p, such as cdcp, edcp, etc. | - -```hocon -env { - # You can set engine configuration here - parallelism = 1 - job.mode = "STREAMING" - checkpoint.interval = 5000 -} - -source { - MongoDB-CDC { - hosts = "mongo0:27017" - # So this example is used (^(test).*|^(tpc).*|txc|.*[p$]|t{2}).(t[5-8]|tt),matching txc.tt、test2.test5. - database = ["(^(test).*|^(tpc).*|txc|.*[p$]|t{2})"] - collection = ["(t[5-8]|tt)"] - username = stuser - password = stpw - } -} - -# Console printing of the read Mongodb data -sink { - Console { - parallelism = 1 - } -} -``` ## Format of real-time streaming data @@ -309,4 +248,3 @@ sink { } } ``` - diff --git a/docs/en/connector-v2/source/MySQL-CDC.md b/docs/en/connector-v2/source/MySQL-CDC.md index fc2ea4d8ff0..cc58ec44596 100644 --- a/docs/en/connector-v2/source/MySQL-CDC.md +++ b/docs/en/connector-v2/source/MySQL-CDC.md @@ -169,14 +169,14 @@ When an initial consistent snapshot is made for large databases, your establishe ## Source Options -| Name | Type | Required | Default | Description | +| Name | Type | Required | Default | Description | |------------------------------------------------|----------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | base-url | String | Yes | - | The URL of the JDBC connection. Refer to a case: `jdbc:mysql://localhost:3306:3306/test`. | | username | String | Yes | - | Name of the database to use when connecting to the database server. | | password | String | Yes | - | Password to use when connecting to the database server. | | database-names | List | No | - | Database name of the database to monitor. | | table-names | List | Yes | - | Table name of the database to monitor. The table name needs to include the database name, for example: `database_name.table_name` | -| table-names-config | List | No | - | Table config list. for example: [{"table": "db1.schema1.table1","primaryKeys":["key1"]}] | +| table-names-config | List | No | - | Table config list. for example: [{"table": "db1.schema1.table1","primaryKeys": ["key1"],"snapshotSplitColumn": "key2"}] | | startup.mode | Enum | No | INITIAL | Optional startup mode for MySQL CDC consumer, valid enumerations are `initial`, `earliest`, `latest` and `specific`.
`initial`: Synchronize historical data at startup, and then synchronize incremental data.
`earliest`: Startup from the earliest offset possible.
`latest`: Startup from the latest offset.
`specific`: Startup from user-supplied specific offsets. | | startup.specific-offset.file | String | No | - | Start from the specified binlog file name. **Note, This option is required when the `startup.mode` option used `specific`.** | | startup.specific-offset.pos | Long | No | - | Start from the specified binlog file position. **Note, This option is required when the `startup.mode` option used `specific`.** | diff --git a/docs/en/connector-v2/source/Opengauss-CDC.md b/docs/en/connector-v2/source/Opengauss-CDC.md index 81691ea1ff4..26825202963 100644 --- a/docs/en/connector-v2/source/Opengauss-CDC.md +++ b/docs/en/connector-v2/source/Opengauss-CDC.md @@ -64,31 +64,31 @@ select 'ALTER TABLE ' || schemaname || '.' || tablename || ' REPLICA IDENTITY FU ## Source Options -| Name | Type | Required | Default | Description | -|------------------------------------------------|----------|----------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| base-url | String | Yes | - | The URL of the JDBC connection. Refer to a case: `jdbc:postgresql://localhost:5432/postgres_cdc?loggerLevel=OFF`. | -| username | String | Yes | - | Username of the database to use when connecting to the database server. | -| password | String | Yes | - | Password to use when connecting to the database server. | -| database-names | List | No | - | Database name of the database to monitor. | -| table-names | List | Yes | - | Table name of the database to monitor. The table name needs to include the database name, for example: `database_name.table_name` | -| table-names-config | List | No | - | Table config list. for example: [{"table": "db1.schema1.table1","primaryKeys":["key1"]}] | -| startup.mode | Enum | No | INITIAL | Optional startup mode for Opengauss CDC consumer, valid enumerations are `initial`, `earliest`, `latest` and `specific`.
`initial`: Synchronize historical data at startup, and then synchronize incremental data.
`earliest`: Startup from the earliest offset possible.
`latest`: Startup from the latest offset.
`specific`: Startup from user-supplied specific offsets. | -| snapshot.split.size | Integer | No | 8096 | The split size (number of rows) of table snapshot, captured tables are split into multiple splits when read the snapshot of table. | -| snapshot.fetch.size | Integer | No | 1024 | The maximum fetch size for per poll when read table snapshot. | -| slot.name | String | No | - | The name of the Opengauss logical decoding slot that was created for streaming changes from a particular plug-in for a particular database/schema. The server uses this slot to stream events to the connector that you are configuring. Default is seatunnel. | -| decoding.plugin.name | String | No | pgoutput | The name of the Postgres logical decoding plug-in installed on the server,Supported values are decoderbufs, wal2json, wal2json_rds, wal2json_streaming,wal2json_rds_streaming and pgoutput. | -| server-time-zone | String | No | UTC | The session time zone in database server. If not set, then ZoneId.systemDefault() is used to determine the server time zone. | -| connect.timeout.ms | Duration | No | 30000 | The maximum time that the connector should wait after trying to connect to the database server before timing out. | -| connect.max-retries | Integer | No | 3 | The max retry times that the connector should retry to build database server connection. | -| connection.pool.size | Integer | No | 20 | The jdbc connection pool size. | +| Name | Type | Required | Default | Description | +|------------------------------------------------|----------|----------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| base-url | String | Yes | - | The URL of the JDBC connection. Refer to a case: `jdbc:postgresql://localhost:5432/postgres_cdc?loggerLevel=OFF`. | +| username | String | Yes | - | Username of the database to use when connecting to the database server. | +| password | String | Yes | - | Password to use when connecting to the database server. | +| database-names | List | No | - | Database name of the database to monitor. | +| table-names | List | Yes | - | Table name of the database to monitor. The table name needs to include the database name, for example: `database_name.table_name` | +| table-names-config | List | No | - | Table config list. for example: [{"table": "db1.schema1.table1","primaryKeys":["key1"]}] | +| startup.mode | Enum | No | INITIAL | Optional startup mode for Opengauss CDC consumer, valid enumerations are `initial`, `earliest`, `latest`.
`initial`: Synchronize historical data at startup, and then synchronize incremental data.
`earliest`: Startup from the earliest offset possible.
`latest`: Startup from the latest offset. | +| snapshot.split.size | Integer | No | 8096 | The split size (number of rows) of table snapshot, captured tables are split into multiple splits when read the snapshot of table. | +| snapshot.fetch.size | Integer | No | 1024 | The maximum fetch size for per poll when read table snapshot. | +| slot.name | String | No | - | The name of the Opengauss logical decoding slot that was created for streaming changes from a particular plug-in for a particular database/schema. The server uses this slot to stream events to the connector that you are configuring. Default is seatunnel. | +| decoding.plugin.name | String | No | pgoutput | The name of the Postgres logical decoding plug-in installed on the server,Supported values are decoderbufs, wal2json, wal2json_rds, wal2json_streaming,wal2json_rds_streaming and pgoutput. | +| server-time-zone | String | No | UTC | The session time zone in database server. If not set, then ZoneId.systemDefault() is used to determine the server time zone. | +| connect.timeout.ms | Duration | No | 30000 | The maximum time that the connector should wait after trying to connect to the database server before timing out. | +| connect.max-retries | Integer | No | 3 | The max retry times that the connector should retry to build database server connection. | +| connection.pool.size | Integer | No | 20 | The jdbc connection pool size. | | chunk-key.even-distribution.factor.upper-bound | Double | No | 100 | The upper bound of the chunk key distribution factor. This factor is used to determine whether the table data is evenly distributed. If the distribution factor is calculated to be less than or equal to this upper bound (i.e., (MAX(id) - MIN(id) + 1) / row count), the table chunks would be optimized for even distribution. Otherwise, if the distribution factor is greater, the table will be considered as unevenly distributed and the sampling-based sharding strategy will be used if the estimated shard count exceeds the value specified by `sample-sharding.threshold`. The default value is 100.0. | -| chunk-key.even-distribution.factor.lower-bound | Double | No | 0.05 | The lower bound of the chunk key distribution factor. This factor is used to determine whether the table data is evenly distributed. If the distribution factor is calculated to be greater than or equal to this lower bound (i.e., (MAX(id) - MIN(id) + 1) / row count), the table chunks would be optimized for even distribution. Otherwise, if the distribution factor is less, the table will be considered as unevenly distributed and the sampling-based sharding strategy will be used if the estimated shard count exceeds the value specified by `sample-sharding.threshold`. The default value is 0.05. | -| sample-sharding.threshold | Integer | No | 1000 | This configuration specifies the threshold of estimated shard count to trigger the sample sharding strategy. When the distribution factor is outside the bounds specified by `chunk-key.even-distribution.factor.upper-bound` and `chunk-key.even-distribution.factor.lower-bound`, and the estimated shard count (calculated as approximate row count / chunk size) exceeds this threshold, the sample sharding strategy will be used. This can help to handle large datasets more efficiently. The default value is 1000 shards. | -| inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. | -| exactly_once | Boolean | No | false | Enable exactly once semantic. | -| format | Enum | No | DEFAULT | Optional output format for Opengauss CDC, valid enumerations are `DEFAULT`, `COMPATIBLE_DEBEZIUM_JSON`. | -| debezium | Config | No | - | Pass-through [Debezium's properties](https://github.com/debezium/debezium/blob/v1.9.8.Final/documentation/modules/ROOT/pages/connectors/postgresql.adoc#connector-configuration-properties) to Debezium Embedded Engine which is used to capture data changes from Opengauss server. | -| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details | +| chunk-key.even-distribution.factor.lower-bound | Double | No | 0.05 | The lower bound of the chunk key distribution factor. This factor is used to determine whether the table data is evenly distributed. If the distribution factor is calculated to be greater than or equal to this lower bound (i.e., (MAX(id) - MIN(id) + 1) / row count), the table chunks would be optimized for even distribution. Otherwise, if the distribution factor is less, the table will be considered as unevenly distributed and the sampling-based sharding strategy will be used if the estimated shard count exceeds the value specified by `sample-sharding.threshold`. The default value is 0.05. | +| sample-sharding.threshold | Integer | No | 1000 | This configuration specifies the threshold of estimated shard count to trigger the sample sharding strategy. When the distribution factor is outside the bounds specified by `chunk-key.even-distribution.factor.upper-bound` and `chunk-key.even-distribution.factor.lower-bound`, and the estimated shard count (calculated as approximate row count / chunk size) exceeds this threshold, the sample sharding strategy will be used. This can help to handle large datasets more efficiently. The default value is 1000 shards. | +| inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. | +| exactly_once | Boolean | No | false | Enable exactly once semantic. | +| format | Enum | No | DEFAULT | Optional output format for Opengauss CDC, valid enumerations are `DEFAULT`, `COMPATIBLE_DEBEZIUM_JSON`. | +| debezium | Config | No | - | Pass-through [Debezium's properties](https://github.com/debezium/debezium/blob/v1.9.8.Final/documentation/modules/ROOT/pages/connectors/postgresql.adoc#connector-configuration-properties) to Debezium Embedded Engine which is used to capture data changes from Opengauss server. | +| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details | ## Task Example @@ -109,7 +109,7 @@ env { source { Opengauss-CDC { - result_table_name = "customers_opengauss_cdc" + plugin_output = "customers_opengauss_cdc" username = "gaussdb" password = "openGauss@123" database-names = ["opengauss_cdc"] @@ -126,7 +126,7 @@ transform { sink { jdbc { - source_table_name = "customers_opengauss_cdc" + plugin_input = "customers_opengauss_cdc" url = "jdbc:postgresql://opengauss_cdc_e2e:5432/opengauss_cdc" driver = "org.postgresql.Driver" user = "dailai" @@ -149,7 +149,7 @@ sink { ``` source { Opengauss-CDC { - result_table_name = "customers_opengauss_cdc" + plugin_output = "customers_opengauss_cdc" username = "gaussdb" password = "openGauss@123" database-names = ["opengauss_cdc"] diff --git a/docs/en/connector-v2/source/Oracle-CDC.md b/docs/en/connector-v2/source/Oracle-CDC.md index feef58a0d2e..dad52faa0b1 100644 --- a/docs/en/connector-v2/source/Oracle-CDC.md +++ b/docs/en/connector-v2/source/Oracle-CDC.md @@ -220,7 +220,7 @@ exit; ## Source Options -| Name | Type | Required | Default | Description | +| Name | Type | Required | Default | Description | |------------------------------------------------|----------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | base-url | String | Yes | - | The URL of the JDBC connection. Refer to a case: `idbc:oracle:thin:datasource01:1523:xe`. | | username | String | Yes | - | Name of the database to use when connecting to the database server. | @@ -228,7 +228,7 @@ exit; | database-names | List | No | - | Database name of the database to monitor. | | schema-names | List | No | - | Schema name of the database to monitor. | | table-names | List | Yes | - | Table name of the database to monitor. The table name needs to include the database name, for example: `database_name.table_name` | -| table-names-config | List | No | - | Table config list. for example: [{"table": "db1.schema1.table1","primaryKeys":["key1"]}] | +| table-names-config | List | No | - | Table config list. for example: [{"table": "db1.schema1.table1","primaryKeys": ["key1"],"snapshotSplitColumn": "key2"}] | | startup.mode | Enum | No | INITIAL | Optional startup mode for Oracle CDC consumer, valid enumerations are `initial`, `earliest`, `latest` and `specific`.
`initial`: Synchronize historical data at startup, and then synchronize incremental data.
`earliest`: Startup from the earliest offset possible.
`latest`: Startup from the latest offset.
`specific`: Startup from user-supplied specific offsets. | | startup.specific-offset.file | String | No | - | Start from the specified binlog file name. **Note, This option is required when the `startup.mode` option used `specific`.** | | startup.specific-offset.pos | Long | No | - | Start from the specified binlog file position. **Note, This option is required when the `startup.mode` option used `specific`.** | @@ -262,7 +262,7 @@ exit; source { # This is a example source plugin **only for test and demonstrate the feature source plugin** Oracle-CDC { - result_table_name = "customers" + plugin_output = "customers" username = "system" password = "oracle" database-names = ["XE"] @@ -280,7 +280,7 @@ source { > source { > # This is a example source plugin **only for test and demonstrate the feature source plugin** > Oracle-CDC { -> result_table_name = "customers" +> plugin_output = "customers" > use_select_count = true > username = "system" > password = "oracle" @@ -299,7 +299,7 @@ source { > source { > # This is a example source plugin **only for test and demonstrate the feature source plugin** > Oracle-CDC { -> result_table_name = "customers" +> plugin_output = "customers" > skip_analyze = true > username = "system" > password = "oracle" @@ -318,7 +318,7 @@ source { source { Oracle-CDC { - result_table_name = "customers" + plugin_output = "customers" base-url = "jdbc:oracle:thin:system/oracle@oracle-host:1521:xe" source.reader.close.timeout = 120000 username = "system" diff --git a/docs/en/connector-v2/source/OssFile.md b/docs/en/connector-v2/source/OssFile.md index d5326cb86a4..42163a9d13e 100644 --- a/docs/en/connector-v2/source/OssFile.md +++ b/docs/en/connector-v2/source/OssFile.md @@ -190,7 +190,7 @@ If you assign file type to `parquet` `orc`, schema option not required, connecto ## Options -| name | type | required | default value | Description | +| name | type | required | default value | Description | |---------------------------|---------|----------|---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | path | string | yes | - | The Oss path that needs to be read can have sub paths, but the sub paths need to meet certain format requirements. Specific requirements can be referred to "parse_partition_from_path" option | | file_format_type | string | yes | - | File type, supported as the following file types: `text` `csv` `parquet` `orc` `json` `excel` `xml` `binary` | @@ -211,7 +211,8 @@ If you assign file type to `parquet` `orc`, schema option not required, connecto | xml_use_attr_format | boolean | no | - | Specifies whether to process data using the tag attribute format, only used when file_format is xml. | | compress_codec | string | no | none | Which compress codec the files used. | | encoding | string | no | UTF-8 | -| file_filter_pattern | string | no | | `*.txt` means you only need read the files end with `.txt` | +| null_format | string | no | - | Only used when file_format_type is text. null_format to define which strings can be represented as null. e.g: `\N` | +| file_filter_pattern | string | no | | Filter pattern, which used for filtering files. | | common-options | config | no | - | Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details. | ### compress_codec [string] @@ -233,6 +234,55 @@ The encoding of the file to read. This param will be parsed by `Charset.forName( Filter pattern, which used for filtering files. +The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. +There are some examples. + +File Structure Example: +``` +/data/seatunnel/20241001/report.txt +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +/data/seatunnel/20241012/logo.png +``` +Matching Rules Example: + +**Example 1**: *Match all .txt files*，Regular Expression: +``` +/data/seatunnel/20241001/.*\.txt +``` +The result of this example matching is: +``` +/data/seatunnel/20241001/report.txt +``` +**Example 2**: *Match all file starting with abc*，Regular Expression: +``` +/data/seatunnel/20241002/abc.* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +``` +**Example 3**: *Match all file starting with abc，And the fourth character is either h or g*, the Regular Expression: +``` +/data/seatunnel/20241007/abc[h,g].* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +``` +**Example 4**: *Match third level folders starting with 202410 and files ending with .csv*, the Regular Expression: +``` +/data/seatunnel/202410\d*/.*\.csv +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +``` + ### schema [config] Only need to be configured when the file_format_type are text, json, excel, xml or csv ( Or other format we can't read the schema from metadata). @@ -344,7 +394,7 @@ source { file_format_type = "orc" } ] - result_table_name = "fake" + plugin_output = "fake" } } @@ -461,7 +511,7 @@ source { } } ] - result_table_name = "fake" + plugin_output = "fake" } } @@ -474,6 +524,33 @@ sink { } ``` +### Filter File + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + OssFile { + path = "/seatunnel/orc" + bucket = "oss://tyrantlucifer-image-bed" + access_key = "xxxxxxxxxxxxxxxxx" + access_secret = "xxxxxxxxxxxxxxxxxxxxxx" + endpoint = "oss-cn-beijing.aliyuncs.com" + file_format_type = "orc" + // file example abcD2024.csv + file_filter_pattern = "abc[DX]*.*" + } +} + +sink { + Console { + } +} +``` + ## Changelog ### 2.2.0-beta 2022-09-26 diff --git a/docs/en/connector-v2/source/OssJindoFile.md b/docs/en/connector-v2/source/OssJindoFile.md index d5bd6d14fa3..9b83a0b0501 100644 --- a/docs/en/connector-v2/source/OssJindoFile.md +++ b/docs/en/connector-v2/source/OssJindoFile.md @@ -49,7 +49,7 @@ It only supports hadoop version **2.9.X+**. ## Options -| name | type | required | default value | +| name | type | required | default value | |---------------------------|---------|----------|---------------------| | path | string | yes | - | | file_format_type | string | yes | - | @@ -68,10 +68,11 @@ It only supports hadoop version **2.9.X+**. | sheet_name | string | no | - | | xml_row_tag | string | no | - | | xml_use_attr_format | boolean | no | - | -| file_filter_pattern | string | no | - | +| file_filter_pattern | string | no | | | compress_codec | string | no | none | | archive_compress_codec | string | no | none | | encoding | string | no | UTF-8 | +| null_format | string | no | - | | common-options | | no | - | ### path [string] @@ -267,6 +268,55 @@ Reader the sheet of the workbook. Filter pattern, which used for filtering files. +The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. +There are some examples. + +File Structure Example: +``` +/data/seatunnel/20241001/report.txt +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +/data/seatunnel/20241012/logo.png +``` +Matching Rules Example: + +**Example 1**: *Match all .txt files*，Regular Expression: +``` +/data/seatunnel/20241001/.*\.txt +``` +The result of this example matching is: +``` +/data/seatunnel/20241001/report.txt +``` +**Example 2**: *Match all file starting with abc*，Regular Expression: +``` +/data/seatunnel/20241002/abc.* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +``` +**Example 3**: *Match all file starting with abc，And the fourth character is either h or g*, the Regular Expression: +``` +/data/seatunnel/20241007/abc[h,g].* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +``` +**Example 4**: *Match third level folders starting with 202410 and files ending with .csv*, the Regular Expression: +``` +/data/seatunnel/202410\d*/.*\.csv +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +``` + ### compress_codec [string] The compress codec of files and the details that supported as the following shown: @@ -286,6 +336,7 @@ The compress codec of archive files and the details that supported as the follow | ZIP | txt,json,excel,xml | .zip | | TAR | txt,json,excel,xml | .tar | | TAR_GZ | txt,json,excel,xml | .tar.gz | +| GZ | txt,json,xml | .gz | | NONE | all | .* | ### encoding [string] @@ -293,6 +344,13 @@ The compress codec of archive files and the details that supported as the follow Only used when file_format_type is json,text,csv,xml. The encoding of the file to read. This param will be parsed by `Charset.forName(encoding)`. +### null_format [string] + +Only used when file_format_type is text. +null_format to define which strings can be represented as null. + +e.g: `\N` + ### common options Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details. @@ -364,6 +422,33 @@ sink { ``` +### Filter File + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + OssJindoFile { + bucket = "oss://tyrantlucifer-image-bed" + access_key = "xxxxxxxxxxxxxxxxx" + access_secret = "xxxxxxxxxxxxxxxxxxxxxx" + endpoint = "oss-cn-beijing.aliyuncs.com" + path = "/seatunnel/read/binary/" + file_format_type = "binary" + // file example abcD2024.csv + file_filter_pattern = "abc[DX]*.*" + } +} + +sink { + Console { + } +} +``` + ## Changelog ### next version diff --git a/docs/en/connector-v2/source/Paimon.md b/docs/en/connector-v2/source/Paimon.md index e586a4fd9d8..cbe3b592f8b 100644 --- a/docs/en/connector-v2/source/Paimon.md +++ b/docs/en/connector-v2/source/Paimon.md @@ -82,6 +82,11 @@ Properties in hadoop conf The specified loading path for the 'core-site.xml', 'hdfs-site.xml', 'hive-site.xml' files +## Filesystems +The Paimon connector supports writing data to multiple file systems. Currently, the supported file systems are hdfs and s3. +If you use the s3 filesystem. You can configure the `fs.s3a.access-key`、`fs.s3a.secret-key`、`fs.s3a.endpoint`、`fs.s3a.path.style.access`、`fs.s3a.aws.credentials.provider` properties in the `paimon.hadoop.conf` option. +Besides, the warehouse should start with `s3a://`. + ## Examples ### Simple example @@ -109,6 +114,33 @@ source { } ``` +### S3 example +```hocon +env { + execution.parallelism = 1 + job.mode = "BATCH" +} + +source { + Paimon { + warehouse = "s3a://test/" + database = "seatunnel_namespace11" + table = "st_test" + paimon.hadoop.conf = { + fs.s3a.access-key=G52pnxg67819khOZ9ezX + fs.s3a.secret-key=SHJuAQqHsLrgZWikvMa3lJf5T0NfM5LMFliJh9HF + fs.s3a.endpoint="http://minio4:9000" + fs.s3a.path.style.access=true + fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider + } + } +} + +sink { + Console{} +} +``` + ### Hadoop conf example ```hocon diff --git a/docs/en/connector-v2/source/PostgreSQL-CDC.md b/docs/en/connector-v2/source/PostgreSQL-CDC.md index be87d03edd4..21afa42f701 100644 --- a/docs/en/connector-v2/source/PostgreSQL-CDC.md +++ b/docs/en/connector-v2/source/PostgreSQL-CDC.md @@ -86,15 +86,15 @@ ALTER TABLE your_table_name REPLICA IDENTITY FULL; ## Source Options -| Name | Type | Required | Default | Description | +| Name | Type | Required | Default | Description | |------------------------------------------------|----------|----------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | base-url | String | Yes | - | The URL of the JDBC connection. Refer to a case: `jdbc:postgresql://localhost:5432/postgres_cdc?loggerLevel=OFF`. | | username | String | Yes | - | Name of the database to use when connecting to the database server. | | password | String | Yes | - | Password to use when connecting to the database server. | | database-names | List | No | - | Database name of the database to monitor. | | table-names | List | Yes | - | Table name of the database to monitor. The table name needs to include the database name, for example: `database_name.table_name` | -| table-names-config | List | No | - | Table config list. for example: [{"table": "db1.schema1.table1","primaryKeys":["key1"]}] | -| startup.mode | Enum | No | INITIAL | Optional startup mode for PostgreSQL CDC consumer, valid enumerations are `initial`, `earliest`, `latest` and `specific`.
`initial`: Synchronize historical data at startup, and then synchronize incremental data.
`earliest`: Startup from the earliest offset possible.
`latest`: Startup from the latest offset.
`specific`: Startup from user-supplied specific offsets. | +| table-names-config | List | No | - | Table config list. for example: [{"table": "db1.schema1.table1","primaryKeys": ["key1"],"snapshotSplitColumn": "key2"}] | +| startup.mode | Enum | No | INITIAL | Optional startup mode for PostgreSQL CDC consumer, valid enumerations are `initial`, `earliest` and `latest`.
`initial`: Synchronize historical data at startup, and then synchronize incremental data.
`earliest`: Startup from the earliest offset possible.
`latest`: Startup from the latest offset. | | snapshot.split.size | Integer | No | 8096 | The split size (number of rows) of table snapshot, captured tables are split into multiple splits when read the snapshot of table. | | snapshot.fetch.size | Integer | No | 1024 | The maximum fetch size for per poll when read table snapshot. | | slot.name | String | No | - | The name of the PostgreSQL logical decoding slot that was created for streaming changes from a particular plug-in for a particular database/schema. The server uses this slot to stream events to the connector that you are configuring. Default is seatunnel. | @@ -132,7 +132,7 @@ env { source { Postgres-CDC { - result_table_name = "customers_Postgre_cdc" + plugin_output = "customers_Postgre_cdc" username = "postgres" password = "postgres" database-names = ["postgres_cdc"] @@ -148,7 +148,7 @@ transform { sink { jdbc { - source_table_name = "customers_Postgre_cdc" + plugin_input = "customers_Postgre_cdc" url = "jdbc:postgresql://postgres_cdc_e2e:5432/postgres_cdc?loggerLevel=OFF" driver = "org.postgresql.Driver" user = "postgres" @@ -169,7 +169,7 @@ sink { ``` source { Postgres-CDC { - result_table_name = "customers_mysql_cdc" + plugin_output = "customers_mysql_cdc" username = "postgres" password = "postgres" database-names = ["postgres_cdc"] diff --git a/docs/en/connector-v2/source/PostgreSQL.md b/docs/en/connector-v2/source/PostgreSQL.md index 101902d3618..d383b113c2e 100644 --- a/docs/en/connector-v2/source/PostgreSQL.md +++ b/docs/en/connector-v2/source/PostgreSQL.md @@ -261,7 +261,7 @@ source{ partition_column= "id" # The name of the table returned - result_table_name = "jdbc" + plugin_output = "jdbc" partition_lower_bound = 1 partition_upper_bound = 50 partition_num = 5 diff --git a/docs/en/connector-v2/source/Prometheus.md b/docs/en/connector-v2/source/Prometheus.md new file mode 100644 index 00000000000..ba8979f023e --- /dev/null +++ b/docs/en/connector-v2/source/Prometheus.md @@ -0,0 +1,152 @@ +# Prometheus + +> Prometheus source connector + +## Description + +Used to read data from Prometheus. + +## Key features + +- [x] [batch](../../concept/connector-v2-features.md) +- [ ] [stream](../../concept/connector-v2-features.md) +- [ ] [parallelism](../../concept/connector-v2-features.md) + +## Options + +| name | type | required | default value | +|-----------------------------|---------|----------|-----------------| +| url | String | Yes | - | +| query | String | Yes | - | +| query_type | String | Yes | Instant | +| content_field | String | Yes | $.data.result.* | +| schema.fields | Config | Yes | - | +| format | String | No | json | +| params | Map | Yes | - | +| poll_interval_millis | int | No | - | +| retry | int | No | - | +| retry_backoff_multiplier_ms | int | No | 100 | +| retry_backoff_max_ms | int | No | 10000 | +| enable_multi_lines | boolean | No | false | +| common-options | config | No | - | + +### url [String] + +http request url + +### query [String] + +Prometheus expression query string + +### query_type [String] + +Instant/Range + +1. Instant : The following endpoint evaluates an instant query at a single point in time +2. Range : The following endpoint evaluates an expression query over a range of time + +https://prometheus.io/docs/prometheus/latest/querying/api/ + +### params [Map] + +http request params + +### poll_interval_millis [int] + +request http api interval(millis) in stream mode + +### retry [int] + +The max retry times if request http return to `IOException` + +### retry_backoff_multiplier_ms [int] + +The retry-backoff times(millis) multiplier if request http failed + +### retry_backoff_max_ms [int] + +The maximum retry-backoff times(millis) if request http failed + +### format [String] + +the format of upstream data, default `json`. + +### schema [Config] + +Fill in a fixed value + +```hocon + schema = { + fields { + metric = "map
- txt: `lzo` `None`
- json: `lzo` `None`
- csv: `lzo` `None`
- orc: `lzo` `snappy` `lz4` `zlib` `None`
- parquet: `lzo` `snappy` `lz4` `gzip` `brotli` `zstd` `None`
Tips: excel type does Not support any compression format | | archive_compress_codec | string | no | none | | encoding | string | no | UTF-8 | +| null_format | string | no | - | Only used when file_format_type is text. null_format to define which strings can be represented as null. e.g: `\N` | | common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details. | +### file_filter_pattern [string] + +Filter pattern, which used for filtering files. + +The pattern follows standard regular expressions. For details, please refer to https://en.wikipedia.org/wiki/Regular_expression. +There are some examples. + +File Structure Example: +``` +/data/seatunnel/20241001/report.txt +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +/data/seatunnel/20241012/logo.png +``` +Matching Rules Example: + +**Example 1**: *Match all .txt files*，Regular Expression: +``` +/data/seatunnel/20241001/.*\.txt +``` +The result of this example matching is: +``` +/data/seatunnel/20241001/report.txt +``` +**Example 2**: *Match all file starting with abc*，Regular Expression: +``` +/data/seatunnel/20241002/abc.* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +``` +**Example 3**: *Match all file starting with abc，And the fourth character is either h or g*, the Regular Expression: +``` +/data/seatunnel/20241007/abc[h,g].* +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +``` +**Example 4**: *Match third level folders starting with 202410 and files ending with .csv*, the Regular Expression: +``` +/data/seatunnel/202410\d*/.*\.csv +``` +The result of this example matching is: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +``` + ### file_format_type [string] File type, supported as the following file types: @@ -182,11 +236,12 @@ The compress codec of files and the details that supported as the following show The compress codec of archive files and the details that supported as the following shown: | archive_compress_codec | file_format | archive_compress_suffix | -|------------------------|--------------------|-------------------------| -| ZIP | txt,json,excel,xml | .zip | -| TAR | txt,json,excel,xml | .tar | -| TAR_GZ | txt,json,excel,xml | .tar.gz | -| NONE | all | .* | +|--------------------|--------------------|---------------------| +| ZIP | txt,json,excel,xml | .zip | +| TAR | txt,json,excel,xml | .tar | +| TAR_GZ | txt,json,excel,xml | .tar.gz | +| GZ | txt,json,xml | .gz | +| NONE | all | .* | ### encoding [string] @@ -219,7 +274,7 @@ source { password = pass path = "tmp/seatunnel/read/json" file_format_type = "json" - result_table_name = "sftp" + plugin_output = "sftp" schema = { fields { c_map = "map
**Note, This option is required when** the **"startup.mode" option used `'timestamp'`.** | @@ -141,7 +141,7 @@ env { source { # This is a example source plugin **only for test and demonstrate the feature source plugin** SqlServer-CDC { - result_table_name = "customers" + plugin_output = "customers" username = "sa" password = "Y.sa123456" startup.mode="initial" @@ -156,7 +156,7 @@ transform { sink { console { - source_table_name = "customers" + plugin_input = "customers" } ``` @@ -177,7 +177,7 @@ source { SqlServer-CDC { # Set up accurate one read exactly_once=true - result_table_name = "customers" + plugin_output = "customers" username = "sa" password = "Y.sa123456" startup.mode="latest" @@ -192,7 +192,7 @@ transform { sink { console { - source_table_name = "customers" + plugin_input = "customers" } ``` diff --git a/docs/en/connector-v2/source/StarRocks.md b/docs/en/connector-v2/source/StarRocks.md index d46105cc9af..1c1a109480a 100644 --- a/docs/en/connector-v2/source/StarRocks.md +++ b/docs/en/connector-v2/source/StarRocks.md @@ -19,25 +19,25 @@ delivers the query plan as a parameter to BE nodes, and then obtains data result ## Options -| name | type | required | default value | -|-------------------------|--------|----------|-------------------| -| node_urls | list | yes | - | -| username | string | yes | - | -| password | string | yes | - | -| database | string | yes | - | -| table | string | yes | - | -| scan_filter | string | no | - | -| schema | config | yes | - | -| request_tablet_size | int | no | Integer.MAX_VALUE | -| scan_connect_timeout_ms | int | no | 30000 | -| scan_query_timeout_sec | int | no | 3600 | -| scan_keep_alive_min | int | no | 10 | -| scan_batch_rows | int | no | 1024 | -| scan_mem_limit | long | no | 2147483648 | -| max_retries | int | no | 3 | -| scan.params.* | string | no | - | - -### node_urls [list] +| name | type | required | default value | +|--------------------------|--------|----------|-------------------| +| nodeUrls | list | yes | - | +| username | string | yes | - | +| password | string | yes | - | +| database | string | yes | - | +| table | string | yes | - | +| scan_filter | string | no | - | +| schema | config | yes | - | +| request_tablet_size | int | no | Integer.MAX_VALUE | +| scan_connect_timeout_ms | int | no | 30000 | +| scan_query_timeout_sec | int | no | 3600 | +| scan_keep_alive_min | int | no | 10 | +| scan_batch_rows | int | no | 1024 | +| scan_mem_limit | long | no | 2147483648 | +| max_retries | int | no | 3 | +| scan.params.* | string | no | - | + +### nodeUrls [list] `StarRocks` cluster address, the format is `["fe_ip:fe_http_port", ...]` diff --git a/docs/en/connector-v2/source/TDengine.md b/docs/en/connector-v2/source/TDengine.md index a24744d5c17..26480c12354 100644 --- a/docs/en/connector-v2/source/TDengine.md +++ b/docs/en/connector-v2/source/TDengine.md @@ -78,7 +78,7 @@ source { stable : "meters" lower_bound : "2018-10-03 14:38:05.000" upper_bound : "2018-10-03 14:38:16.800" - result_table_name = "tdengine_result" + plugin_output = "tdengine_result" } } ``` diff --git a/docs/en/connector-v2/source/TiDB-CDC.md b/docs/en/connector-v2/source/TiDB-CDC.md index ffa307f8bc1..1cce8ec3ac2 100644 --- a/docs/en/connector-v2/source/TiDB-CDC.md +++ b/docs/en/connector-v2/source/TiDB-CDC.md @@ -91,7 +91,7 @@ env { source { # This is a example source plugin **only for test and demonstrate the feature source plugin** TiDB-CDC { - result_table_name = "products_tidb_cdc" + plugin_output = "products_tidb_cdc" base-url = "jdbc:mysql://tidb0:4000/inventory" driver = "com.mysql.cj.jdbc.Driver" tikv.grpc.timeout_in_ms = 20000 @@ -108,7 +108,7 @@ transform { sink { jdbc { - source_table_name = "products_tidb_cdc" + plugin_input = "products_tidb_cdc" url = "jdbc:mysql://tidb0:4000/inventory" driver = "com.mysql.cj.jdbc.Driver" user = "root" diff --git a/docs/en/contribution/how-to-create-your-connector.md b/docs/en/contribution/how-to-create-your-connector.md new file mode 100644 index 00000000000..b99bc85d999 --- /dev/null +++ b/docs/en/contribution/how-to-create-your-connector.md @@ -0,0 +1,3 @@ +# Develop Your Own Connector + +If you want to develop your own connector for the new SeaTunnel connector API (Connector V2), please check [here](https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/README.md). \ No newline at end of file diff --git a/docs/en/contribution/setup.md b/docs/en/contribution/setup.md index b2579e1ee1e..8fd632a24b0 100644 --- a/docs/en/contribution/setup.md +++ b/docs/en/contribution/setup.md @@ -80,7 +80,7 @@ After all the above things are done, you just finish the environment setup and c of box. All examples are in module `seatunnel-examples`, you could pick one you are interested in, [Running Or Debugging It In IDEA](https://www.jetbrains.com/help/idea/run-debug-configuration.html) as you wish. -Here we use `seatunnel-examples/seatunnel-engine-examples/src/main/java/org/apache/seatunnel/example/engine/SeaTunnelEngineExample.java` +Here we use `seatunnel-examples/seatunnel-engine-examples/src/main/java/org/apache/seatunnel/example/engine/SeaTunnelEngineLocalExample.java` as an example, when you run it successfully you can see the output as below: ```log diff --git a/docs/en/faq.md b/docs/en/faq.md index 02c125ad4fd..6a4e838eaed 100644 --- a/docs/en/faq.md +++ b/docs/en/faq.md @@ -1,332 +1,116 @@ -# FAQs +# FAQ -## Why should I install a computing engine like Spark or Flink? +## What data sources and destinations does SeaTunnel support? +SeaTunnel supports various data sources and destinations. You can find a detailed list on the following list: +- Supported data sources (Source): [Source List](https://seatunnel.apache.org/docs/connector-v2/source) +- Supported data destinations (Sink): [Sink List](https://seatunnel.apache.org/docs/connector-v2/sink) -SeaTunnel now uses computing engines such as Spark and Flink to complete resource scheduling and node communication, so we can focus on the ease of use of data synchronization and the development of high-performance components. But this is only temporary. +## Does SeaTunnel support batch and streaming processing? +SeaTunnel supports both batch and streaming processing modes. You can select the appropriate mode based on your specific business scenarios and needs. Batch processing is suitable for scheduled data integration tasks, while streaming processing is ideal for real-time integration and Change Data Capture (CDC). -## I have a question, and I cannot solve it by myself +## Is it necessary to install engines like Spark or Flink when using SeaTunnel? +Spark and Flink are not mandatory. SeaTunnel supports Zeta, Spark, and Flink as integration engines, allowing you to choose one based on your needs. The community highly recommends Zeta, a new generation high-performance integration engine specifically designed for integration scenarios. Zeta is affectionately called "Ultraman Zeta" by community users! The community offers extensive support for Zeta, making it the most feature-rich option. -I have encountered a problem when using SeaTunnel and I cannot solve it by myself. What should I do? First, search in [Issue List](https://github.com/apache/seatunnel/issues) or [Mailing List](https://lists.apache.org/list.html?dev@seatunnel.apache.org) to see if someone has already asked the same question and got an answer. If you cannot find an answer to your question, you can contact community members for help in [These Ways](https://github.com/apache/seatunnel#contact-us). +## What data transformation functions does SeaTunnel provide? +SeaTunnel supports multiple data transformation functions, including field mapping, data filtering, data format conversion, and more. You can implement data transformations through the `transform` module in the configuration file. For more details, refer to the SeaTunnel [Transform Documentation](https://seatunnel.apache.org/docs/transform-v2). -## How do I declare a variable? +## Can SeaTunnel support custom data cleansing rules? +Yes, SeaTunnel supports custom data cleansing rules. You can configure custom rules in the `transform` module, such as cleaning up dirty data, removing invalid records, or converting fields. -Do you want to know how to declare a variable in SeaTunnel's configuration, and then dynamically replace the value of the variable at runtime? +## Does SeaTunnel support real-time incremental integration? +SeaTunnel supports incremental data integration. For example, the CDC connector allows real-time capture of data changes, which is ideal for scenarios requiring real-time data integration. -Since `v1.2.4`, SeaTunnel supports variable substitution in the configuration. This feature is often used for timing or non-timing offline processing to replace variables such as time and date. The usage is as follows: +## What CDC data sources are currently supported by SeaTunnel? +SeaTunnel currently supports MongoDB CDC, MySQL CDC, OpenGauss CDC, Oracle CDC, PostgreSQL CDC, SQL Server CDC, TiDB CDC, and more. For more details, refer to the [Source List](https://seatunnel.apache.org/docs/connector-v2/source). -Configure the variable name in the configuration. Here is an example of sql transform (actually, anywhere in the configuration file the value in `'key = value'` can use the variable substitution): +## How do I enable permissions required for SeaTunnel CDC integration? +Please refer to the official SeaTunnel documentation for the necessary steps to enable permissions for each connector’s CDC functionality. -``` -... -transform { - sql { - query = "select * from user_view where city ='"${city}"' and dt = '"${date}"'" - } -} -... -``` +## Does SeaTunnel support CDC from MySQL replicas? How are logs pulled? +Yes, SeaTunnel supports CDC from MySQL replicas by subscribing to binlog logs, which are then parsed on the SeaTunnel server. -Taking Spark Local mode as an example, the startup command is as follows: +## Does SeaTunnel support CDC integration for tables without primary keys? +SeaTunnel does not support CDC integration for tables without primary keys. The reason is that if two identical records exist in the upstream and one is deleted or modified, the downstream cannot determine which record to delete or modify, leading to potential issues. Primary keys are essential to ensure data uniqueness. -```bash -./bin/start-seatunnel-spark.sh \ --c ./config/your_app.conf \ --e client \ --m local[2] \ --i city=shanghai \ --i date=20190319 -``` +## Does SeaTunnel support automatic table creation? +Before starting an integration task, you can select different handling schemes for existing table structures on the target side, controlled via the `schema_save_mode` parameter. Available options include: +- **`RECREATE_SCHEMA`**: Creates the table if it does not exist; if the table exists, it is deleted and recreated. +- **`CREATE_SCHEMA_WHEN_NOT_EXIST`**: Creates the table if it does not exist; skips creation if the table already exists. +- **`ERROR_WHEN_SCHEMA_NOT_EXIST`**: Throws an error if the table does not exist. +- **`IGNORE`**: Ignores table handling. + Many connectors currently support automatic table creation. Refer to the specific connector documentation, such as [Jdbc sink](https://seatunnel.apache.org/docs/2.3.8/connector-v2/sink/Jdbc#schema_save_mode-enum), for more information. -You can use the parameter `-i` or `--variable` followed by `key=value` to specify the value of the variable, where the key needs to be same as the variable name in the configuration. +## Does SeaTunnel support handling existing data before starting a data integration task? +Yes, you can specify different processing schemes for existing data on the target side before starting an integration task, controlled via the `data_save_mode` parameter. Available options include: +- **`DROP_DATA`**: Retains the database structure but deletes the data. +- **`APPEND_DATA`**: Retains both the database structure and data. +- **`CUSTOM_PROCESSING`**: User-defined processing. +- **`ERROR_WHEN_DATA_EXISTS`**: Throws an error if data already exists. + Many connectors support handling existing data; please refer to the respective connector documentation, such as [Jdbc sink](https://seatunnel.apache.org/docs/connector-v2/sink/Jdbc#data_save_mode-enum). -## How do I write a configuration item in multi-line text in the configuration file? +## Does SeaTunnel support exactly-once consistency? +SeaTunnel supports exactly-once consistency for some data sources, such as MySQL and PostgreSQL, ensuring data consistency during integration. Note that exactly-once consistency depends on the capabilities of the underlying database. -When a configured text is very long and you want to wrap it, you can use three double quotes to indicate its start and end: +## Can SeaTunnel execute scheduled tasks? +You can use Linux cron jobs to achieve periodic data integration, or leverage scheduling tools like Apache DolphinScheduler or Apache Airflow to manage complex scheduled tasks. -``` -var = """ - whatever you want -""" -``` +## I encountered an issue with SeaTunnel that I cannot resolve. What should I do? +If you encounter issues with SeaTunnel, here are a few ways to get help: +1. Search the [Issue List](https://github.com/apache/seatunnel/issues) or [Mailing List](https://lists.apache.org/list.html?dev@seatunnel.apache.org) to see if someone else has faced a similar issue. +2. If you cannot find an answer, reach out to the community through [these methods](https://github.com/apache/seatunnel#contact-us). -## How do I implement variable substitution for multi-line text? +## How do I declare variables? +Would you like to declare a variable in SeaTunnel's configuration and dynamically replace it at runtime? This feature is commonly used in both scheduled and ad-hoc offline processing to replace time, date, or other variables. Here's an example: -It is a little troublesome to do variable substitution in multi-line text, because the variable cannot be included in three double quotation marks: - -``` -var = """ -your string 1 -"""${you_var}""" your string 2""" -``` - -Refer to: [lightbend/config#456](https://github.com/lightbend/config/issues/456). - -## Is SeaTunnel supported in Azkaban, Oozie, DolphinScheduler? - -Of course! See the screenshot below: - - - - - -## Does SeaTunnel have a case for configuring multiple sources, such as configuring elasticsearch and hdfs in source at the same time? - -``` -env { - ... -} - -source { - hdfs { ... } - elasticsearch { ... } - jdbc {...} -} +Define the variable in the configuration. For example, in an SQL transformation (the value in any "key = value" pair in the configuration file can be replaced with variables): +```plaintext +... transform { - ... -} - -sink { - elasticsearch { ... } -} -``` - -## Are there any HBase plugins? - -There is a HBase input plugin. You can download it from here: https://github.com/garyelephant/waterdrop-input-hbase . - -## How can I use SeaTunnel to write data to Hive? - -``` -env { - spark.sql.catalogImplementation = "hive" - spark.hadoop.hive.exec.dynamic.partition = "true" - spark.hadoop.hive.exec.dynamic.partition.mode = "nonstrict" -} - -source { - sql = "insert into ..." -} - -sink { - // The data has been written to hive through the sql source. This is just a placeholder, it does not actually work. - stdout { - limit = 1 - } -} -``` - -In addition, SeaTunnel has implemented a `Hive` output plugin after version `1.5.7` in `1.x` branch; in `2.x` branch. The Hive plugin for the Spark engine has been supported from version `2.0.5`: https://github.com/apache/seatunnel/issues/910. - -## How does SeaTunnel write multiple instances of ClickHouse to achieve load balancing? - -1. Write distributed tables directly (not recommended) - -2. Add a proxy or domain name (DNS) in front of multiple instances of ClickHouse: - - ``` - { - output { - clickhouse { - host = "ck-proxy.xx.xx:8123" - # Local table - table = "table_name" - } - } - } - ``` -3. Configure multiple instances in the configuration: - - ``` - { - output { - clickhouse { - host = "ck1:8123,ck2:8123,ck3:8123" - # Local table - table = "table_name" - } - } - } - ``` -4. Use cluster mode: - - ``` - { - output { - clickhouse { - # Configure only one host - host = "ck1:8123" - cluster = "clickhouse_cluster_name" - # Local table - table = "table_name" - } - } - } - ``` - -## How can I solve OOM when SeaTunnel consumes Kafka? - -In most cases, OOM is caused by not having a rate limit for consumption. The solution is as follows: - -For the current limit of Spark consumption of Kafka: - -1. Suppose the number of partitions of Kafka `Topic 1` you consume with KafkaStream = N. - -2. Assuming that the production speed of the message producer (Producer) of `Topic 1` is K messages/second, the speed of write messages to the partition must be uniform. - -3. Suppose that, after testing, it is found that the processing capacity of Spark Executor per core per second is M. - -The following conclusions can be drawn: - -1. If you want to make Spark's consumption of `Topic 1` keep up with its production speed, then you need `spark.executor.cores` * `spark.executor.instances` >= K / M - -2. When a data delay occurs, if you want the consumption speed not to be too fast, resulting in spark executor OOM, then you need to configure `spark.streaming.kafka.maxRatePerPartition` <= (`spark.executor.cores` * `spark.executor.instances`) * M / N - -3. In general, both M and N are determined, and the conclusion can be drawn from 2: The size of `spark.streaming.kafka.maxRatePerPartition` is positively correlated with the size of `spark.executor.cores` * `spark.executor.instances`, and it can be increased while increasing the resource `maxRatePerPartition` to speed up consumption. - - - -## How can I solve the Error `Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE`? - -The reason is that the version of httpclient.jar that comes with the CDH version of Spark is lower, and The httpclient version that ClickHouse JDBC is based on is 4.5.2, and the package versions conflict. The solution is to replace the jar package that comes with CDH with the httpclient-4.5.2 version. - -## The default JDK of my Spark cluster is JDK7. After I install JDK8, how can I specify that SeaTunnel starts with JDK8? - -In SeaTunnel's config file, specify the following configuration: - -```shell -spark { - ... - spark.executorEnv.JAVA_HOME="/your/java_8_home/directory" - spark.yarn.appMasterEnv.JAVA_HOME="/your/java_8_home/directory" - ... + Sql { + query = "select * from user_view where city ='${city}' and dt = '${date}'" + } } +... ``` -## What should I do if OOM always appears when running SeaTunnel in Spark local[*] mode? - -If you run in local mode, you need to modify the `start-seatunnel.sh` startup script. After `spark-submit`, add a parameter `--driver-memory 4g` . Under normal circumstances, local mode is not used in the production environment. Therefore, this parameter generally does not need to be set during On YARN. See: [Application Properties](https://spark.apache.org/docs/latest/configuration.html#application-properties) for details. - -## Where can I place self-written plugins or third-party jdbc.jars to be loaded by SeaTunnel? - -Place the Jar package under the specified structure of the plugins directory: +To start SeaTunnel in Zeta Local mode with variables: ```bash -cd SeaTunnel -mkdir -p plugins/my_plugins/lib -cp third-part.jar plugins/my_plugins/lib +$SEATUNNEL_HOME/bin/seatunnel.sh \ +-c $SEATUNNEL_HOME/config/your_app.conf \ +-m local[2] \ +-i city=Singapore \ +-i date=20231110 ``` -`my_plugins` can be any string. - -## How do I configure logging-related parameters in SeaTunnel-V1(Spark)? - -There are three ways to configure logging-related parameters (such as Log Level): - -- [Not recommended] Change the default `$SPARK_HOME/conf/log4j.properties`. - - This will affect all programs submitted via `$SPARK_HOME/bin/spark-submit`. -- [Not recommended] Modify logging related parameters directly in the Spark code of SeaTunnel. - - This is equivalent to hardcoding, and each change needs to be recompiled. -- [Recommended] Use the following methods to change the logging configuration in the SeaTunnel configuration file (The change only takes effect if SeaTunnel >= 1.5.5 ): - - ``` - env { - spark.driver.extraJavaOptions = "-Dlog4j.configuration=file:
+
+
+#### Request Parameters
+
+#### Parameters (Add in the `params` field of the request body)
+
+> | Parameter Name | Required | Type | Description |
+> |----------------------|------------|---------|---------------------------------|
+> | jobId | optional | string | job id |
+
+When `jobId` is empty, it returns log information for all nodes; otherwise, it returns the log list of the specified `jobId` across all nodes.
+
+#### Response
+
+Returns a list of logs and content from the requested nodes.
+
+#### Get All Log Files List
+
+If you'd like to view the log list first, you can use a `GET` request to retrieve the log list:
+`http://localhost:5801/hazelcast/rest/maps/logs?format=json`
+
+```json
+[
+ {
+ "node": "localhost:5801",
+ "logLink": "http://localhost:5801/hazelcast/rest/maps/logs/job-899485770241277953.log",
+ "logName": "job-899485770241277953.log"
+ },
+ {
+ "node": "localhost:5801",
+ "logLink": "http://localhost:5801/hazelcast/rest/maps/logs/job-899470314109468673.log",
+ "logName": "job-899470314109468673.log"
+ }
+]
+```
+
+The supported formats are `json` and `html`, with `html` as the default.
+
+#### Examples
+
+Retrieve logs for all nodes with the `jobId` of `733584788375666689`: `http://localhost:5801/hazelcast/rest/maps/logs/733584788375666689`
+Retrieve the log list for all nodes: `http://localhost:5801/hazelcast/rest/maps/logs`
+Retrieve the log list for all nodes in JSON format: `http://localhost:5801/hazelcast/rest/maps/logs?format=json`
+Retrieve log file content: `http://localhost:5801/hazelcast/rest/maps/logs/job-898380162133917698.log`
+
+
+
+### Get Log Content from a Single Node
+
+GET /hazelcast/rest/maps/logs/:jobId (Returns a list of logs.)
+
+#### Request Parameters
+
+#### Parameters (Add in the `params` field of the request body)
+
+> | Parameter Name | Required | Type | Description |
+> |----------------------|------------|---------|---------------------------------|
+> | jobId | optional | string | job id |
+
+When `jobId` is empty, it returns log information for all nodes; otherwise, it returns the log list of the specified `jobId` across all nodes.
+
+#### Response
+
+Returns a list of logs and content from the requested nodes.
+
+#### Get All Log Files List
+
+If you'd like to view the log list first, you can use a `GET` request to retrieve the log list:
+`http://localhost:5801/hazelcast/rest/maps/logs?format=json`
+
+```json
+[
+ {
+ "node": "localhost:5801",
+ "logLink": "http://localhost:5801/hazelcast/rest/maps/logs/job-899485770241277953.log",
+ "logName": "job-899485770241277953.log"
+ },
+ {
+ "node": "localhost:5801",
+ "logLink": "http://localhost:5801/hazelcast/rest/maps/logs/job-899470314109468673.log",
+ "logName": "job-899470314109468673.log"
+ }
+]
+```
+
+The supported formats are `json` and `html`, with `html` as the default.
+
+#### Examples
+
+Retrieve logs for all nodes with the `jobId` of `733584788375666689`: `http://localhost:5801/hazelcast/rest/maps/logs/733584788375666689`
+Retrieve the log list for all nodes: `http://localhost:5801/hazelcast/rest/maps/logs`
+Retrieve the log list for all nodes in JSON format: `http://localhost:5801/hazelcast/rest/maps/logs?format=json`
+Retrieve log file content: `http://localhost:5801/hazelcast/rest/maps/logs/job-898380162133917698.log`
+
+
+
+
+#### Response
+
+Returns a list of logs from the requested node.
+
+#### Examples
+
+To get a list of logs from the current node: `http://localhost:5801/hazelcast/rest/maps/log`
+To get the content of a log file: `http://localhost:5801/hazelcast/rest/maps/log/job-898380162133917698.log`
+
+
\ No newline at end of file
diff --git a/docs/en/seatunnel-engine/rest-api-v2.md b/docs/en/seatunnel-engine/rest-api-v2.md
new file mode 100644
index 00000000000..8a5e3a8d7d3
--- /dev/null
+++ b/docs/en/seatunnel-engine/rest-api-v2.md
@@ -0,0 +1,847 @@
+---
+sidebar_position: 12
+---
+
+# RESTful API V2
+
+SeaTunnel has a monitoring API that can be used to query status and statistics of running jobs, as well as recent
+completed jobs. The monitoring API is a RESTful API that accepts HTTP requests and responds with JSON data.
+
+## Overview
+
+The v2 version of the api uses jetty support. It is the same as the interface specification of v1 version
+, you can specify the port and context-path by modifying the configuration items in `seatunnel.yaml`,
+you can configure `enable-dynamic-port` to enable dynamic ports (the default port is accumulated starting from `port`), and the default is closed,
+If enable-dynamic-port is true, We will use the unused port in the range within the range of `port` and `port` + `port-range`, default range is 100
+
+```yaml
+
+seatunnel:
+ engine:
+ http:
+ enable-http: true
+ port: 8080
+ enable-dynamic-port: false
+ port-range: 100
+```
+
+Context-path can also be configured as follows:
+
+```yaml
+
+seatunnel:
+ engine:
+ http:
+ enable-http: true
+ port: 8080
+ context-path: /seatunnel
+```
+
+## API reference
+
+### Returns an overview over the Zeta engine cluster.
+
+GET /hazelcast/rest/maps/log (Returns a list of logs.)
+
+#### Response
+
+Returns a list of logs from the requested node.
+
+#### Examples
+
+To get a list of logs from the current node: `http://localhost:5801/hazelcast/rest/maps/log`
+To get the content of a log file: `http://localhost:5801/hazelcast/rest/maps/log/job-898380162133917698.log`
+
+
+
+
+#### Parameters
+
+> | name | type | data type | description |
+> |----------|----------|-----------|------------------------------------------------------------------------------------------------------|
+> | tag_name | optional | string | the tags filter, you can add tag filter to get those matched worker count, and slot on those workers |
+
+#### Responses
+
+```json
+{
+ "projectVersion":"2.3.5-SNAPSHOT",
+ "gitCommitAbbrev":"DeadD0d0",
+ "totalSlot":"0",
+ "unassignedSlot":"0",
+ "works":"1",
+ "runningJobs":"0",
+ "finishedJobs":"0",
+ "failedJobs":"0",
+ "cancelledJobs":"0"
+}
+```
+
+**Notes:**
+- If you use `dynamic-slot`, the `totalSlot` and `unassignedSlot` always be `0`. when you set it to fix slot number, it will return the correct total and unassigned slot number
+- If the url has tag filter, the `works`, `totalSlot` and `unassignedSlot` will return the result on the matched worker. but the job related metric will always return the cluster level information.
+
+
+
+------------------------------------------------------------------------------------------
+
+### Returns An Overview And State Of All Jobs
+
+GET /overview?tag1=value1&tag2=value2 (Returns an overview over the Zeta engine cluster.)
+
+#### Parameters
+
+> | name | type | data type | description |
+> |----------|----------|-----------|------------------------------------------------------------------------------------------------------|
+> | tag_name | optional | string | the tags filter, you can add tag filter to get those matched worker count, and slot on those workers |
+
+#### Responses
+
+```json
+{
+ "projectVersion":"2.3.5-SNAPSHOT",
+ "gitCommitAbbrev":"DeadD0d0",
+ "totalSlot":"0",
+ "unassignedSlot":"0",
+ "works":"1",
+ "runningJobs":"0",
+ "finishedJobs":"0",
+ "failedJobs":"0",
+ "cancelledJobs":"0"
+}
+```
+
+**Notes:**
+- If you use `dynamic-slot`, the `totalSlot` and `unassignedSlot` always be `0`. when you set it to fix slot number, it will return the correct total and unassigned slot number
+- If the url has tag filter, the `works`, `totalSlot` and `unassignedSlot` will return the result on the matched worker. but the job related metric will always return the cluster level information.
+
+
+
+
+#### Parameters
+
+#### Responses
+
+```json
+[
+ {
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "envOptions": {
+ },
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false,
+ "metrics": {
+ "sourceReceivedCount": "",
+ "sinkWriteCount": ""
+ }
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### Return Details Of A Job
+
+GET /running-jobs (Returns an overview over all jobs and their current state.)
+
+#### Parameters
+
+#### Responses
+
+```json
+[
+ {
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "envOptions": {
+ },
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false,
+ "metrics": {
+ "sourceReceivedCount": "",
+ "sinkWriteCount": ""
+ }
+ }
+]
+```
+
+
+
+
+#### Parameters
+
+> | name | type | data type | description |
+> |-------|----------|-----------|-------------|
+> | jobId | required | long | job id |
+
+#### Responses
+
+```json
+{
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": {
+ "sourceReceivedCount": "",
+ "sinkWriteCount": ""
+ },
+ "finishedTime": "",
+ "errorMsg": null,
+ "envOptions": {
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false
+}
+```
+
+`jobId`, `jobName`, `jobStatus`, `createTime`, `jobDag`, `metrics` always be returned.
+`envOptions`, `pluginJarsUrls`, `isStartWithSavePoint` will return when job is running.
+`finishedTime`, `errorMsg` will return when job is finished.
+
+When we can't get the job info, the response will be:
+
+```json
+{
+ "jobId" : ""
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### Return Details Of A Job
+
+This API has been deprecated, please use /job-info/:jobId instead
+
+GET /job-info/:jobId (Return details of a job. )
+
+#### Parameters
+
+> | name | type | data type | description |
+> |-------|----------|-----------|-------------|
+> | jobId | required | long | job id |
+
+#### Responses
+
+```json
+{
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": {
+ "sourceReceivedCount": "",
+ "sinkWriteCount": ""
+ },
+ "finishedTime": "",
+ "errorMsg": null,
+ "envOptions": {
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false
+}
+```
+
+`jobId`, `jobName`, `jobStatus`, `createTime`, `jobDag`, `metrics` always be returned.
+`envOptions`, `pluginJarsUrls`, `isStartWithSavePoint` will return when job is running.
+`finishedTime`, `errorMsg` will return when job is finished.
+
+When we can't get the job info, the response will be:
+
+```json
+{
+ "jobId" : ""
+}
+```
+
+
+
+
+#### Parameters
+
+> | name | type | data type | description |
+> |-------|----------|-----------|-------------|
+> | jobId | required | long | job id |
+
+#### Responses
+
+```json
+{
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": {
+ "SourceReceivedCount": "",
+ "SourceReceivedQPS": "",
+ "SourceReceivedBytes": "",
+ "SourceReceivedBytesPerSeconds": "",
+ "SinkWriteCount": "",
+ "SinkWriteQPS": "",
+ "SinkWriteBytes": "",
+ "SinkWriteBytesPerSeconds": "",
+ "TableSourceReceivedCount": {},
+ "TableSourceReceivedBytes": {},
+ "TableSourceReceivedBytesPerSeconds": {},
+ "TableSourceReceivedQPS": {},
+ "TableSinkWriteCount": {},
+ "TableSinkWriteQPS": {},
+ "TableSinkWriteBytes": {},
+ "TableSinkWriteBytesPerSeconds": {}
+ },
+ "finishedTime": "",
+ "errorMsg": null,
+ "envOptions": {
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false
+}
+```
+
+`jobId`, `jobName`, `jobStatus`, `createTime`, `jobDag`, `metrics` always be returned.
+`envOptions`, `pluginJarsUrls`, `isStartWithSavePoint` will return when job is running.
+`finishedTime`, `errorMsg` will return when job is finished.
+
+When we can't get the job info, the response will be:
+
+```json
+{
+ "jobId" : ""
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### Return All Finished Jobs Info
+
+GET /running-job/:jobId (Return details of a job. )
+
+#### Parameters
+
+> | name | type | data type | description |
+> |-------|----------|-----------|-------------|
+> | jobId | required | long | job id |
+
+#### Responses
+
+```json
+{
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": {
+ "SourceReceivedCount": "",
+ "SourceReceivedQPS": "",
+ "SourceReceivedBytes": "",
+ "SourceReceivedBytesPerSeconds": "",
+ "SinkWriteCount": "",
+ "SinkWriteQPS": "",
+ "SinkWriteBytes": "",
+ "SinkWriteBytesPerSeconds": "",
+ "TableSourceReceivedCount": {},
+ "TableSourceReceivedBytes": {},
+ "TableSourceReceivedBytesPerSeconds": {},
+ "TableSourceReceivedQPS": {},
+ "TableSinkWriteCount": {},
+ "TableSinkWriteQPS": {},
+ "TableSinkWriteBytes": {},
+ "TableSinkWriteBytesPerSeconds": {}
+ },
+ "finishedTime": "",
+ "errorMsg": null,
+ "envOptions": {
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false
+}
+```
+
+`jobId`, `jobName`, `jobStatus`, `createTime`, `jobDag`, `metrics` always be returned.
+`envOptions`, `pluginJarsUrls`, `isStartWithSavePoint` will return when job is running.
+`finishedTime`, `errorMsg` will return when job is finished.
+
+When we can't get the job info, the response will be:
+
+```json
+{
+ "jobId" : ""
+}
+```
+
+
+
+
+#### Parameters
+
+> | name | type | data type | description |
+> |-------|----------|-----------|------------------------------------------------------------------|
+> | state | optional | string | finished job status. `FINISHED`,`CANCELED`,`FAILED`,`UNKNOWABLE` |
+
+#### Responses
+
+```json
+[
+ {
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "errorMsg": null,
+ "createTime": "",
+ "finishTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": ""
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### Returns System Monitoring Information
+
+GET /finished-jobs/:state (Return all finished Jobs Info.)
+
+#### Parameters
+
+> | name | type | data type | description |
+> |-------|----------|-----------|------------------------------------------------------------------|
+> | state | optional | string | finished job status. `FINISHED`,`CANCELED`,`FAILED`,`UNKNOWABLE` |
+
+#### Responses
+
+```json
+[
+ {
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "errorMsg": null,
+ "createTime": "",
+ "finishTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": ""
+ }
+]
+```
+
+
+
+
+#### Parameters
+
+#### Responses
+
+```json
+[
+ {
+ "processors":"8",
+ "physical.memory.total":"16.0G",
+ "physical.memory.free":"16.3M",
+ "swap.space.total":"0",
+ "swap.space.free":"0",
+ "heap.memory.used":"135.7M",
+ "heap.memory.free":"440.8M",
+ "heap.memory.total":"576.5M",
+ "heap.memory.max":"3.6G",
+ "heap.memory.used/total":"23.54%",
+ "heap.memory.used/max":"3.73%",
+ "minor.gc.count":"6",
+ "minor.gc.time":"110ms",
+ "major.gc.count":"2",
+ "major.gc.time":"73ms",
+ "load.process":"24.78%",
+ "load.system":"60.00%",
+ "load.systemAverage":"2.07",
+ "thread.count":"117",
+ "thread.peakCount":"118",
+ "cluster.timeDiff":"0",
+ "event.q.size":"0",
+ "executor.q.async.size":"0",
+ "executor.q.client.size":"0",
+ "executor.q.client.query.size":"0",
+ "executor.q.client.blocking.size":"0",
+ "executor.q.query.size":"0",
+ "executor.q.scheduled.size":"0",
+ "executor.q.io.size":"0",
+ "executor.q.system.size":"0",
+ "executor.q.operations.size":"0",
+ "executor.q.priorityOperation.size":"0",
+ "operations.completed.count":"10",
+ "executor.q.mapLoad.size":"0",
+ "executor.q.mapLoadAllKeys.size":"0",
+ "executor.q.cluster.size":"0",
+ "executor.q.response.size":"0",
+ "operations.running.count":"0",
+ "operations.pending.invocations.percentage":"0.00%",
+ "operations.pending.invocations.count":"0",
+ "proxy.count":"8",
+ "clientEndpoint.count":"0",
+ "connection.active.count":"2",
+ "client.connection.count":"0",
+ "connection.count":"0"
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### Submit A Job
+
+GET /system-monitoring-information (Returns system monitoring information.)
+
+#### Parameters
+
+#### Responses
+
+```json
+[
+ {
+ "processors":"8",
+ "physical.memory.total":"16.0G",
+ "physical.memory.free":"16.3M",
+ "swap.space.total":"0",
+ "swap.space.free":"0",
+ "heap.memory.used":"135.7M",
+ "heap.memory.free":"440.8M",
+ "heap.memory.total":"576.5M",
+ "heap.memory.max":"3.6G",
+ "heap.memory.used/total":"23.54%",
+ "heap.memory.used/max":"3.73%",
+ "minor.gc.count":"6",
+ "minor.gc.time":"110ms",
+ "major.gc.count":"2",
+ "major.gc.time":"73ms",
+ "load.process":"24.78%",
+ "load.system":"60.00%",
+ "load.systemAverage":"2.07",
+ "thread.count":"117",
+ "thread.peakCount":"118",
+ "cluster.timeDiff":"0",
+ "event.q.size":"0",
+ "executor.q.async.size":"0",
+ "executor.q.client.size":"0",
+ "executor.q.client.query.size":"0",
+ "executor.q.client.blocking.size":"0",
+ "executor.q.query.size":"0",
+ "executor.q.scheduled.size":"0",
+ "executor.q.io.size":"0",
+ "executor.q.system.size":"0",
+ "executor.q.operations.size":"0",
+ "executor.q.priorityOperation.size":"0",
+ "operations.completed.count":"10",
+ "executor.q.mapLoad.size":"0",
+ "executor.q.mapLoadAllKeys.size":"0",
+ "executor.q.cluster.size":"0",
+ "executor.q.response.size":"0",
+ "operations.running.count":"0",
+ "operations.pending.invocations.percentage":"0.00%",
+ "operations.pending.invocations.count":"0",
+ "proxy.count":"8",
+ "clientEndpoint.count":"0",
+ "connection.active.count":"2",
+ "client.connection.count":"0",
+ "connection.count":"0"
+ }
+]
+```
+
+
+
+
+#### Parameters
+
+> | name | type | data type | description |
+> |----------------------|----------|-----------|-----------------------------------|
+> | jobId | optional | string | job id |
+> | jobName | optional | string | job name |
+> | isStartWithSavePoint | optional | string | if job is started with save point |
+> | format | optional | string | config format, support json and hocon, default json |
+
+#### Body
+
+You can choose json or hocon to pass request body.
+The json format example:
+``` json
+{
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 100,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+}
+```
+The hocon format example:
+``` hocon
+env {
+ job.mode = "batch"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ row.num = 100
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ card = "int"
+ }
+ }
+ }
+}
+
+transform {
+}
+
+sink {
+ Console {
+ source_table_name = "fake"
+ }
+}
+
+```
+
+
+#### Responses
+
+```json
+{
+ "jobId": 733584788375666689,
+ "jobName": "rest_api_test"
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### Batch Submit Jobs
+
+POST /submit-job (Returns jobId and jobName if job submitted successfully.)
+
+#### Parameters
+
+> | name | type | data type | description |
+> |----------------------|----------|-----------|-----------------------------------|
+> | jobId | optional | string | job id |
+> | jobName | optional | string | job name |
+> | isStartWithSavePoint | optional | string | if job is started with save point |
+> | format | optional | string | config format, support json and hocon, default json |
+
+#### Body
+
+You can choose json or hocon to pass request body.
+The json format example:
+``` json
+{
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 100,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+}
+```
+The hocon format example:
+``` hocon
+env {
+ job.mode = "batch"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ row.num = 100
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ card = "int"
+ }
+ }
+ }
+}
+
+transform {
+}
+
+sink {
+ Console {
+ source_table_name = "fake"
+ }
+}
+
+```
+
+
+#### Responses
+
+```json
+{
+ "jobId": 733584788375666689,
+ "jobName": "rest_api_test"
+}
+```
+
+
+
+
+#### Parameters (add in the `params` field in the request body)
+
+> | Parameter Name | Required | Type | Description |
+> |----------------------|--------------|---------|---------------------------------------|
+> | jobId | optional | string | job id |
+> | jobName | optional | string | job name |
+> | isStartWithSavePoint | optional | string | if the job is started with save point |
+
+#### Request Body
+
+```json
+[
+ {
+ "params":{
+ "jobId":"123456",
+ "jobName":"SeaTunnel-01"
+ },
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 1000,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+ },
+ {
+ "params":{
+ "jobId":"1234567",
+ "jobName":"SeaTunnel-02"
+ },
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 1000,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+ }
+]
+```
+
+#### Response
+
+```json
+[
+ {
+ "jobId": "123456",
+ "jobName": "SeaTunnel-01"
+ },{
+ "jobId": "1234567",
+ "jobName": "SeaTunnel-02"
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### Stop A Job
+
+POST /submit-jobs (Returns jobId and jobName if the job is successfully submitted.)
+
+#### Parameters (add in the `params` field in the request body)
+
+> | Parameter Name | Required | Type | Description |
+> |----------------------|--------------|---------|---------------------------------------|
+> | jobId | optional | string | job id |
+> | jobName | optional | string | job name |
+> | isStartWithSavePoint | optional | string | if the job is started with save point |
+
+#### Request Body
+
+```json
+[
+ {
+ "params":{
+ "jobId":"123456",
+ "jobName":"SeaTunnel-01"
+ },
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 1000,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+ },
+ {
+ "params":{
+ "jobId":"1234567",
+ "jobName":"SeaTunnel-02"
+ },
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 1000,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+ }
+]
+```
+
+#### Response
+
+```json
+[
+ {
+ "jobId": "123456",
+ "jobName": "SeaTunnel-01"
+ },{
+ "jobId": "1234567",
+ "jobName": "SeaTunnel-02"
+ }
+]
+```
+
+
+
+
+#### Body
+
+```json
+{
+ "jobId": 733584788375666689,
+ "isStopWithSavePoint": false # if job is stopped with save point
+}
+```
+
+#### Responses
+
+```json
+{
+"jobId": 733584788375666689
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+### Batch Stop Jobs
+
+POST /stop-job (Returns jobId if job stoped successfully.)
+
+#### Body
+
+```json
+{
+ "jobId": 733584788375666689,
+ "isStopWithSavePoint": false # if job is stopped with save point
+}
+```
+
+#### Responses
+
+```json
+{
+"jobId": 733584788375666689
+}
+```
+
+
+
+
+#### Request Body
+
+```json
+[
+ {
+ "jobId": 881432421482889220,
+ "isStopWithSavePoint": false
+ },
+ {
+ "jobId": 881432456517910529,
+ "isStopWithSavePoint": false
+ }
+]
+```
+
+#### Response
+
+```json
+[
+ {
+ "jobId": 881432421482889220
+ },
+ {
+ "jobId": 881432456517910529
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+### Encrypt Config
+
+POST /stop-jobs (Returns jobId if the job is successfully stopped.)
+
+#### Request Body
+
+```json
+[
+ {
+ "jobId": 881432421482889220,
+ "isStopWithSavePoint": false
+ },
+ {
+ "jobId": 881432456517910529,
+ "isStopWithSavePoint": false
+ }
+]
+```
+
+#### Response
+
+```json
+[
+ {
+ "jobId": 881432421482889220
+ },
+ {
+ "jobId": 881432456517910529
+ }
+]
+```
+
+
+
+For more information about customize encryption, please refer to the documentation [config-encryption-decryption](../connector-v2/Config-Encryption-Decryption.md).
+
+#### Body
+
+```json
+{
+ "env": {
+ "parallelism": 1,
+ "shade.identifier":"base64"
+ },
+ "source": [
+ {
+ "plugin_name": "MySQL-CDC",
+ "schema" : {
+ "fields": {
+ "name": "string",
+ "age": "int"
+ }
+ },
+ "plugin_output": "fake",
+ "parallelism": 1,
+ "hostname": "127.0.0.1",
+ "username": "seatunnel",
+ "password": "seatunnel_password",
+ "table-name": "inventory_vwyw0n"
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Clickhouse",
+ "host": "localhost:8123",
+ "database": "default",
+ "table": "fake_all",
+ "username": "seatunnel",
+ "password": "seatunnel_password"
+ }
+ ]
+}
+```
+
+#### Responses
+
+```json
+{
+ "env": {
+ "parallelism": 1,
+ "shade.identifier": "base64"
+ },
+ "source": [
+ {
+ "plugin_name": "MySQL-CDC",
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int"
+ }
+ },
+ "plugin_output": "fake",
+ "parallelism": 1,
+ "hostname": "127.0.0.1",
+ "username": "c2VhdHVubmVs",
+ "password": "c2VhdHVubmVsX3Bhc3N3b3Jk",
+ "table-name": "inventory_vwyw0n"
+ }
+ ],
+ "transform": [],
+ "sink": [
+ {
+ "plugin_name": "Clickhouse",
+ "host": "localhost:8123",
+ "database": "default",
+ "table": "fake_all",
+ "username": "c2VhdHVubmVs",
+ "password": "c2VhdHVubmVsX3Bhc3N3b3Jk"
+ }
+ ]
+}
+```
+
+
+
+
+------------------------------------------------------------------------------------------
+
+### Update the tags of running node
+
+POST /encrypt-config (Returns the encrypted config if config is encrypted successfully.)
+For more information about customize encryption, please refer to the documentation [config-encryption-decryption](../connector-v2/Config-Encryption-Decryption.md).
+
+#### Body
+
+```json
+{
+ "env": {
+ "parallelism": 1,
+ "shade.identifier":"base64"
+ },
+ "source": [
+ {
+ "plugin_name": "MySQL-CDC",
+ "schema" : {
+ "fields": {
+ "name": "string",
+ "age": "int"
+ }
+ },
+ "plugin_output": "fake",
+ "parallelism": 1,
+ "hostname": "127.0.0.1",
+ "username": "seatunnel",
+ "password": "seatunnel_password",
+ "table-name": "inventory_vwyw0n"
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Clickhouse",
+ "host": "localhost:8123",
+ "database": "default",
+ "table": "fake_all",
+ "username": "seatunnel",
+ "password": "seatunnel_password"
+ }
+ ]
+}
+```
+
+#### Responses
+
+```json
+{
+ "env": {
+ "parallelism": 1,
+ "shade.identifier": "base64"
+ },
+ "source": [
+ {
+ "plugin_name": "MySQL-CDC",
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int"
+ }
+ },
+ "plugin_output": "fake",
+ "parallelism": 1,
+ "hostname": "127.0.0.1",
+ "username": "c2VhdHVubmVs",
+ "password": "c2VhdHVubmVsX3Bhc3N3b3Jk",
+ "table-name": "inventory_vwyw0n"
+ }
+ ],
+ "transform": [],
+ "sink": [
+ {
+ "plugin_name": "Clickhouse",
+ "host": "localhost:8123",
+ "database": "default",
+ "table": "fake_all",
+ "username": "c2VhdHVubmVs",
+ "password": "c2VhdHVubmVsX3Bhc3N3b3Jk"
+ }
+ ]
+}
+```
+
+POST/update-tagsBecause the update can only target a specific node, the current node's `ip:port` needs to be used for the update(If the update is successful, return a success message)
+
+
+#### update node tags
+##### Body
+If the request parameter is a `Map` object, it indicates that the tags of the current node need to be updated
+```json
+{
+ "tag1": "dev_1",
+ "tag2": "dev_2"
+}
+```
+##### Responses
+
+```json
+{
+ "status": "success",
+ "message": "update node tags done."
+}
+```
+#### remove node tags
+##### Body
+If the parameter is an empty `Map` object, it means that the tags of the current node need to be cleared
+```json
+{}
+```
+##### Responses
+
+```json
+{
+ "status": "success",
+ "message": "update node tags done."
+}
+```
+
+#### Request parameter exception
+- If the parameter body is empty
+
+##### Responses
+
+```json
+{
+ "status": "fail",
+ "message": "Request body is empty."
+}
+```
+- If the parameter is not a `Map` object
+##### Responses
+
+```json
+{
+ "status": "fail",
+ "message": "Invalid JSON format in request body."
+}
+```
+
+
+
+#### Request Parameters
+
+#### Parameters (to be added in the `params` field of the request body)
+
+> | Parameter Name | Required | Type | Description |
+> |-----------------------|--------------|---------|------------------------------------|
+> | jobId | optional | string | job id |
+
+If `jobId` is empty, the request will return logs from all nodes. Otherwise, it will return the list of logs for the specified `jobId` from all nodes.
+
+#### Response
+
+Returns a list of logs from the requested nodes along with their content.
+
+#### Return List of All Log Files
+
+If you want to view the log list first, you can retrieve it via a `GET` request: `http://localhost:8080/logs?format=json`
+
+```json
+[
+ {
+ "node": "localhost:8080",
+ "logLink": "http://localhost:8080/logs/job-899485770241277953.log",
+ "logName": "job-899485770241277953.log"
+ },
+ {
+ "node": "localhost:8080",
+ "logLink": "http://localhost:8080/logs/job-899470314109468673.log",
+ "logName": "job-899470314109468673.log"
+ }
+]
+```
+
+Supported formats are `json` and `html`, with `html` as the default.
+
+#### Examples
+
+Retrieve logs for `jobId` `733584788375666689` across all nodes: `http://localhost:8080/logs/733584788375666689`
+Retrieve the list of logs from all nodes: `http://localhost:8080/logs`
+Retrieve the list of logs in JSON format: `http://localhost:8080/logs?format=json`
+Retrieve the content of a specific log file: `http://localhost:8080/logs/job-898380162133917698.log`
+
+
+
+### Get Log Content from a Single Node
+
+GET /logs/:jobId (Returns a list of logs.)
+
+#### Request Parameters
+
+#### Parameters (to be added in the `params` field of the request body)
+
+> | Parameter Name | Required | Type | Description |
+> |-----------------------|--------------|---------|------------------------------------|
+> | jobId | optional | string | job id |
+
+If `jobId` is empty, the request will return logs from all nodes. Otherwise, it will return the list of logs for the specified `jobId` from all nodes.
+
+#### Response
+
+Returns a list of logs from the requested nodes along with their content.
+
+#### Return List of All Log Files
+
+If you want to view the log list first, you can retrieve it via a `GET` request: `http://localhost:8080/logs?format=json`
+
+```json
+[
+ {
+ "node": "localhost:8080",
+ "logLink": "http://localhost:8080/logs/job-899485770241277953.log",
+ "logName": "job-899485770241277953.log"
+ },
+ {
+ "node": "localhost:8080",
+ "logLink": "http://localhost:8080/logs/job-899470314109468673.log",
+ "logName": "job-899470314109468673.log"
+ }
+]
+```
+
+Supported formats are `json` and `html`, with `html` as the default.
+
+#### Examples
+
+Retrieve logs for `jobId` `733584788375666689` across all nodes: `http://localhost:8080/logs/733584788375666689`
+Retrieve the list of logs from all nodes: `http://localhost:8080/logs`
+Retrieve the list of logs in JSON format: `http://localhost:8080/logs?format=json`
+Retrieve the content of a specific log file: `http://localhost:8080/logs/job-898380162133917698.log`
+
+
+
+
+#### Response
+
+Returns a list of logs from the requested node.
+
+#### Examples
+
+To get a list of logs from the current node: `http://localhost:5801/log`
+To get the content of a log file: `http://localhost:5801/log/job-898380162133917698.log`
+
+
diff --git a/docs/en/seatunnel-engine/separated-cluster-deployment.md b/docs/en/seatunnel-engine/separated-cluster-deployment.md
index dcfe58d2f55..91215eb459a 100644
--- a/docs/en/seatunnel-engine/separated-cluster-deployment.md
+++ b/docs/en/seatunnel-engine/separated-cluster-deployment.md
@@ -71,7 +71,7 @@ SeaTunnel Engine implements cluster management based on [Hazelcast IMDG](https:/
The `backup count` is a parameter that defines the number of synchronous backups. For example, if it is set to 1, the backup of the partition will be placed on one other member. If it is set to 2, it will be placed on two other members.
-We recommend that the value of `backup-count` be `min(1, max(5, N/2))`. `N` is the number of cluster nodes.
+We recommend that the value of `backup-count` be `max(1, min(5, N/2))`. `N` is the number of cluster nodes.
```yaml
seatunnel:
@@ -173,7 +173,7 @@ seatunnel:
This configuration mainly solves the problem of resource leakage caused by continuously creating and attempting to destroy class loaders.
If you encounter an exception related to metaspace space overflow, you can try to enable this configuration.
In order to reduce the frequency of creating class loaders, after enabling this configuration, SeaTunnel will not try to release the corresponding class loader when the job is completed, so that it can be used by subsequent jobs, that is to say, when not too many types of Source/Sink connector are used in the running job, it is more effective.
-The default value is false.
+The default value is true.
Example
```yaml
@@ -280,6 +280,23 @@ netty-common-4.1.89.Final.jar
seatunnel-hadoop3-3.1.4-uber.jar
```
+### 4.7 Job Scheduling Strategy
+
+When resources are insufficient, the job scheduling strategy can be configured in the following two modes:
+
+1. `WAIT`: Wait for resources to be available.
+
+2. `REJECT`: Reject the job, default value.
+
+Example
+
+```yaml
+seatunnel:
+ engine:
+ job-schedule-strategy: WAIT
+```
+When `dynamic-slot: true` is used, the `job-schedule-strategy: WAIT` configuration will become invalid and will be forcibly changed to `job-schedule-strategy: REJECT`, because this parameter is meaningless in dynamic slots.
+
## 5. Configuring SeaTunnel Engine Network Services
All network-related configurations of the SeaTunnel Engine are in the `hazelcast-master.yaml` and `hazelcast-worker.yaml` files.
@@ -431,4 +448,4 @@ Now that the cluster has been deployed, you can complete the job submission and
### 8.2 Submit Jobs With The REST API
-The SeaTunnel Engine provides a REST API for submitting and managing jobs. For more information, please refer to [REST API](rest-api.md)
\ No newline at end of file
+The SeaTunnel Engine provides a REST API for submitting and managing jobs. For more information, please refer to [REST API V2](rest-api-v2.md)
diff --git a/docs/en/seatunnel-engine/telemetry.md b/docs/en/seatunnel-engine/telemetry.md
index 1febb3f08e6..530385392a3 100644
--- a/docs/en/seatunnel-engine/telemetry.md
+++ b/docs/en/seatunnel-engine/telemetry.md
@@ -1,5 +1,5 @@
---
-sidebar_position: 13
+sidebar_position: 14
---
# Telemetry
@@ -48,8 +48,8 @@ Note: All metrics both have the same labelName `cluster`, that's value is the co
| hazelcast_executor_queueSize | Gauge | **type**, the type of executor, including: "async" "client" "clientBlocking" "clientQuery" "io" "offloadable" "scheduled" "system" | The hazelcast executor queueSize of seatunnel cluster node |
| hazelcast_partition_partitionCount | Gauge | - | The partitionCount of seatunnel cluster node |
| hazelcast_partition_activePartition | Gauge | - | The activePartition of seatunnel cluster node |
-| hazelcast_partition_isClusterSafe | Gauge | - | Weather is cluster safe of partition |
-| hazelcast_partition_isLocalMemberSafe | Gauge | - | Weather is local member safe of partition |
+| hazelcast_partition_isClusterSafe | Gauge | - | Whether is cluster safe of partition |
+| hazelcast_partition_isLocalMemberSafe | Gauge | - | Whether is local member safe of partition |
### Thread Pool Status
diff --git a/docs/en/seatunnel-engine/user-command.md b/docs/en/seatunnel-engine/user-command.md
index 2504198b2b1..f8957a98276 100644
--- a/docs/en/seatunnel-engine/user-command.md
+++ b/docs/en/seatunnel-engine/user-command.md
@@ -1,5 +1,5 @@
---
-sidebar_position: 12
+sidebar_position: 13
---
# Command Line Tool
@@ -122,3 +122,13 @@ This command will cancel the specified job. After canceling the job, the job wil
Supports batch cancellation of jobs, and can cancel multiple jobs at one time.
All breakpoint information of the canceled job will be deleted and cannot be resumed by seatunnel.sh -r <jobId>.
+
+## Configure The JVM Options
+
+We can configure the JVM options for the SeaTunnel Engine client in the following ways:
+
+1. Add the JVM options to `$SEATUNNEL_HOME/config/jvm_client_options`.
+
+ Modify the JVM parameters in the `$SEATUNNEL_HOME/config/jvm_client_options` file. Please note that the JVM parameters in this file will be applied to all jobs submitted using `seatunnel.sh`, including Local Mode and Cluster Mode.
+
+2. Add JVM options when submitting jobs. For example, `sh bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template -DJvmOption="-Xms2G -Xmx2G"`
diff --git a/docs/en/seatunnel-engine/web-ui.md b/docs/en/seatunnel-engine/web-ui.md
new file mode 100644
index 00000000000..84c73d2fae8
--- /dev/null
+++ b/docs/en/seatunnel-engine/web-ui.md
@@ -0,0 +1,48 @@
+# Web UI
+
+## Access
+
+Before accessing the web ui we need to enable the http rest api. first you need to configure it in the `seatunnel.yaml` configuration file
+
+```
+seatunnel:
+ engine:
+ http:
+ enable-http: true
+ port: 8080
+
+```
+
+Then visit `http://ip:8080/#/overview`
+
+## Overview
+
+The Web UI of Apache SeaTunnel offers a user-friendly interface for monitoring and managing SeaTunnel jobs. Through the Web UI, users can view real-time information on currently running jobs, finished jobs, and the status of worker and master nodes within the cluster. The main functional modules include Jobs, Workers, and Master, each providing detailed status information and operational options to help users efficiently manage and optimize their data processing workflows.
+
+
+## Jobs
+
+### Running Jobs
+
+The "Running Jobs" section lists all SeaTunnel jobs that are currently in execution. Users can view basic information for each job, including Job ID, submission time, status, execution time, and more. By clicking on a specific job, users can access detailed information such as task distribution, resource utilization, and log outputs, allowing for real-time monitoring of job progress and timely handling of potential issues.
+
+
+
+### Finished Jobs
+
+The "Finished Jobs" section displays all SeaTunnel jobs that have either successfully completed or failed. This section provides execution results, completion times, durations, and failure reasons (if any) for each job. Users can review past job records through this module to analyze job performance, troubleshoot issues, or rerun specific jobs as needed.
+
+
+## Workers
+
+### Workers Information
+
+The "Workers" section displays detailed information about all worker nodes in the cluster, including each worker's address, running status, CPU and memory usage, number of tasks being executed, and more. Through this module, users can monitor the health of each worker node, promptly identify and address resource bottlenecks or node failures, ensuring the stable operation of the SeaTunnel cluster.
+
+
+## Master
+
+### Master Information
+
+The "Master" section provides the status and configuration information of the master node in the SeaTunnel cluster. Users can view the master's address, running status, job scheduling responsibilities, and overall resource allocation within the cluster. This module helps users gain a comprehensive understanding of the cluster's core management components, facilitating cluster configuration optimization and troubleshooting.
+
diff --git a/docs/en/start-v2/docker/docker.md b/docs/en/start-v2/docker/docker.md
index 3dfe3ec8c85..2c2c7824f4f 100644
--- a/docs/en/start-v2/docker/docker.md
+++ b/docs/en/start-v2/docker/docker.md
@@ -40,7 +40,7 @@ You can download the source code from the [download page](https://seatunnel.apac
```shell
cd seatunnel
# Use already sett maven profile
-sh ./mvnw -B clean install -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dlicense.skipAddThirdParty=true -D"docker.build.skip"=false -D"docker.verify.skip"=false -D"docker.push.skip"=true -D"docker.tag"=2.3.8 -Dmaven.deploy.skip --no-snapshot-updates -Pdocker,seatunnel
+sh ./mvnw -B clean install -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dlicense.skipAddThirdParty=true -D"docker.build.skip"=false -D"docker.verify.skip"=false -D"docker.push.skip"=true -D"docker.tag"=2.3.9 -Dmaven.deploy.skip -D"skip.spotless"=true --no-snapshot-updates -Pdocker,seatunnel
# Check the docker image
docker images | grep apache/seatunnel
@@ -53,10 +53,10 @@ sh ./mvnw clean package -DskipTests -Dskip.spotless=true
# Build docker image
cd seatunnel-dist
-docker build -f src/main/docker/Dockerfile --build-arg VERSION=2.3.8 -t apache/seatunnel:2.3.8 .
+docker build -f src/main/docker/Dockerfile --build-arg VERSION=2.3.9 -t apache/seatunnel:2.3.9 .
# If you build from dev branch, you should add SNAPSHOT suffix to the version
-docker build -f src/main/docker/Dockerfile --build-arg VERSION=2.3.8-SNAPSHOT -t apache/seatunnel:2.3.8-SNAPSHOT .
+docker build -f src/main/docker/Dockerfile --build-arg VERSION=2.3.9-SNAPSHOT -t apache/seatunnel:2.3.9-SNAPSHOT .
# Check the docker image
docker images | grep apache/seatunnel
@@ -167,24 +167,26 @@ docker run -d --name seatunnel_master \
- get created container ip
```shell
-docker inspect master-1
+docker inspect seatunnel_master
```
run this command to get the pod ip.
- start worker node
```shell
+# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST`
docker run -d --name seatunnel_worker_1 \
--network seatunnel-network \
--rm \
- -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ # set master container ip to here
+ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r worker
## start worker2
-docker run -d --name seatunnel_worker_2 \
+# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST`
+docker run -d --name seatunnel_worker_2 \
--network seatunnel-network \
--rm \
- -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ # set master container ip to here
+ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r worker
@@ -194,20 +196,22 @@ docker run -d --name seatunnel_worker_2 \
run this command to start master node.
```shell
+# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST`
docker run -d --name seatunnel_master \
--network seatunnel-network \
--rm \
- -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ # set exist master container ip to here
+ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r master
```
run this command to start worker node.
```shell
+# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST`
docker run -d --name seatunnel_worker_1 \
--network seatunnel-network \
--rm \
- -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ # set master container ip to here
+ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r worker
```
@@ -371,21 +375,23 @@ and run `docker-compose up -d` command, the new worker node will start, and the
#### use docker as a client
- submit job :
```shell
+# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST`
docker run --name seatunnel_client \
--network seatunnel-network \
+ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
--rm \
apache/seatunnel \
- -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ # set it as master node container ip
- ./bin/seatunnel.sh -c config/v2.batch.config.template # this is an default config, if you need submit your self config, you can mount config file.
+ ./bin/seatunnel.sh -c config/v2.batch.config.template
```
- list job
```shell
+# you need update yourself master container ip to `ST_DOCKER_MEMBER_LIST`
docker run --name seatunnel_client \
--network seatunnel-network \
+ -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
--rm \
apache/seatunnel \
- -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \ # set it as master node container ip
./bin/seatunnel.sh -l
```
@@ -395,5 +401,5 @@ more command please refer [user-command](../../seatunnel-engine/user-command.md)
#### use rest api
-please refer [Submit A Job](../../seatunnel-engine/rest-api.md#submit-a-job)
+please refer [Submit A Job](../../seatunnel-engine/rest-api-v2.md#submit-a-job)
diff --git a/docs/en/start-v2/kubernetes/kubernetes.mdx b/docs/en/start-v2/kubernetes/kubernetes.mdx
index eb231850514..f3cc9e6b0d5 100644
--- a/docs/en/start-v2/kubernetes/kubernetes.mdx
+++ b/docs/en/start-v2/kubernetes/kubernetes.mdx
@@ -44,7 +44,7 @@ To run the image with SeaTunnel, first create a `Dockerfile`:
```Dockerfile
FROM flink:1.13
-ENV SEATUNNEL_VERSION="2.3.8"
+ENV SEATUNNEL_VERSION="2.3.9"
ENV SEATUNNEL_HOME="/opt/seatunnel"
RUN wget https://dlcdn.apache.org/seatunnel/${SEATUNNEL_VERSION}/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
@@ -56,13 +56,13 @@ RUN cd ${SEATUNNEL_HOME} && sh bin/install-plugin.sh ${SEATUNNEL_VERSION}
Then run the following commands to build the image:
```bash
-docker build -t seatunnel:2.3.8-flink-1.13 -f Dockerfile .
+docker build -t seatunnel:2.3.9-flink-1.13 -f Dockerfile .
```
-Image `seatunnel:2.3.8-flink-1.13` needs to be present in the host (minikube) so that the deployment can take place.
+Image `seatunnel:2.3.9-flink-1.13` needs to be present in the host (minikube) so that the deployment can take place.
Load image to minikube via:
```bash
-minikube image load seatunnel:2.3.8-flink-1.13
+minikube image load seatunnel:2.3.9-flink-1.13
```
@@ -72,7 +72,7 @@ minikube image load seatunnel:2.3.8-flink-1.13
```Dockerfile
FROM openjdk:8
-ENV SEATUNNEL_VERSION="2.3.8"
+ENV SEATUNNEL_VERSION="2.3.9"
ENV SEATUNNEL_HOME="/opt/seatunnel"
RUN wget https://dlcdn.apache.org/seatunnel/${SEATUNNEL_VERSION}/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
@@ -84,13 +84,13 @@ RUN cd ${SEATUNNEL_HOME} && sh bin/install-plugin.sh ${SEATUNNEL_VERSION}
Then run the following commands to build the image:
```bash
-docker build -t seatunnel:2.3.8 -f Dockerfile .
+docker build -t seatunnel:2.3.9 -f Dockerfile .
```
-Image `seatunnel:2.3.8` need to be present in the host (minikube) so that the deployment can take place.
+Image `seatunnel:2.3.9` need to be present in the host (minikube) so that the deployment can take place.
Load image to minikube via:
```bash
-minikube image load seatunnel:2.3.8
+minikube image load seatunnel:2.3.9
```
@@ -100,7 +100,7 @@ minikube image load seatunnel:2.3.8
```Dockerfile
FROM openjdk:8
-ENV SEATUNNEL_VERSION="2.3.8"
+ENV SEATUNNEL_VERSION="2.3.9"
ENV SEATUNNEL_HOME="/opt/seatunnel"
RUN wget https://dlcdn.apache.org/seatunnel/${SEATUNNEL_VERSION}/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
@@ -112,13 +112,13 @@ RUN cd ${SEATUNNEL_HOME} && sh bin/install-plugin.sh ${SEATUNNEL_VERSION}
Then run the following commands to build the image:
```bash
-docker build -t seatunnel:2.3.8 -f Dockerfile .
+docker build -t seatunnel:2.3.9 -f Dockerfile .
```
-Image `seatunnel:2.3.8` needs to be present in the host (minikube) so that the deployment can take place.
+Image `seatunnel:2.3.9` needs to be present in the host (minikube) so that the deployment can take place.
Load image to minikube via:
```bash
-minikube image load seatunnel:2.3.8
+minikube image load seatunnel:2.3.9
```
@@ -191,7 +191,7 @@ none
]}>
GET /log (Returns a list of logs.)
+
+#### Response
+
+Returns a list of logs from the requested node.
+
+#### Examples
+
+To get a list of logs from the current node: `http://localhost:5801/log`
+To get the content of a log file: `http://localhost:5801/log/job-898380162133917698.log`
+
+When `source_table_name` is specified, the current plugin is processing the data set corresponding to this parameter. | -| source_table_name | String | No | - | When `result_table_name` is not specified, the data processed by this plugin will not be registered as a data set that can be directly accessed by other plugins, or called a temporary table `(table)`;
When `result_table_name` is specified, the data processed by this plugin will be registered as a data set `(dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)` . The dataset registered here can be directly accessed by other plugins by specifying `source_table_name` . | +:::warn + +The old configuration name `source_table_name`/`result_table_name` is deprecated, please migrate to the new name `plugin_input`/`plugin_output` as soon as possible. + +::: + +| Name | Type | Required | Default | Description | +|---------------|--------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| plugin_output | String | No | - | When `plugin_input` is not specified, the current plugin processes the data set `(dataset)` output by the previous plugin in the configuration file;
When `plugin_input` is specified, the current plugin is processing the data set corresponding to this parameter. | +| plugin_input | String | No | - | When `plugin_output` is not specified, the data processed by this plugin will not be registered as a data set that can be directly accessed by other plugins, or called a temporary table `(table)`;
When `plugin_output` is specified, the data processed by this plugin will be registered as a data set `(dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)` . The dataset registered here can be directly accessed by other plugins by specifying `plugin_input` . | ## Task Example @@ -24,7 +30,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -48,9 +54,9 @@ source { transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" - # the query table name must same as field 'source_table_name' + plugin_input = "fake" + plugin_output = "fake1" + # the query table name must same as field 'plugin_input' query = "select id, regexp_replace(name, '.+', 'b') as name, age+1 as age, pi() as pi, c_timestamp, c_date, c_map, c_array, c_decimal, c_row from fake" } # The SQL transform support base function and criteria operation @@ -59,10 +65,10 @@ transform { sink { Console { - source_table_name = "fake1" + plugin_input = "fake1" } Console { - source_table_name = "fake" + plugin_input = "fake" } } ``` diff --git a/docs/en/transform-v2/copy.md b/docs/en/transform-v2/copy.md index 7a0e73f44be..eede3f7d077 100644 --- a/docs/en/transform-v2/copy.md +++ b/docs/en/transform-v2/copy.md @@ -36,8 +36,8 @@ We want copy fields `name`、`age` to a new fields `name1`、`name2`、`age1`, w ``` transform { Copy { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" fields { name1 = name name2 = name diff --git a/docs/en/transform-v2/dynamic-compile.md b/docs/en/transform-v2/dynamic-compile.md index fb5500880ac..d5f21f2708d 100644 --- a/docs/en/transform-v2/dynamic-compile.md +++ b/docs/en/transform-v2/dynamic-compile.md @@ -82,13 +82,13 @@ Use this DynamicCompile to add a new column `compile_language`, and update the ` ```hacon transform { DynamicCompile { - source_table_name = "fake" - result_table_name = "groovy_out" + plugin_input = "fake" + plugin_output = "groovy_out" compile_language="GROOVY" compile_pattern="SOURCE_CODE" source_code=""" import org.apache.seatunnel.api.table.catalog.Column - import org.apache.seatunnel.transform.common.SeaTunnelRowAccessor + import org.apache.seatunnel.api.table.type.SeaTunnelRowAccessor import org.apache.seatunnel.api.table.catalog.CatalogTable import org.apache.seatunnel.api.table.catalog.PhysicalColumn; import org.apache.seatunnel.api.table.type.*; @@ -140,13 +140,13 @@ transform { ```hacon transform { DynamicCompile { - source_table_name = "fake" - result_table_name = "java_out" + plugin_input = "fake" + plugin_output = "java_out" compile_language="JAVA" compile_pattern="SOURCE_CODE" source_code=""" import org.apache.seatunnel.api.table.catalog.Column; - import org.apache.seatunnel.transform.common.SeaTunnelRowAccessor; + import org.apache.seatunnel.api.table.type.SeaTunnelRowAccessor; import org.apache.seatunnel.api.table.catalog.*; import org.apache.seatunnel.api.table.type.*; import java.util.ArrayList; @@ -195,8 +195,8 @@ transform { ```hacon transform { DynamicCompile { - source_table_name = "fake" - result_table_name = "groovy_out" + plugin_input = "fake" + plugin_output = "groovy_out" compile_language="GROOVY" compile_pattern="ABSOLUTE_PATH" absolute_path="""/tmp/GroovyFile""" diff --git a/docs/en/transform-v2/embedding.md b/docs/en/transform-v2/embedding.md index 046f72789ac..350a23fc555 100644 --- a/docs/en/transform-v2/embedding.md +++ b/docs/en/transform-v2/embedding.md @@ -166,13 +166,13 @@ source { "Herman Melville (1819–1891) was an American novelist, short story writer, and poet of the American Renaissance period. Born in New York City, Melville gained initial fame with novels such as Typee and Omoo, but it was Moby-Dick, published in 1851, that would later be recognized as his masterpiece. Melville’s work is known for its complexity, symbolism, and exploration of themes such as man’s place in the universe, the nature of evil, and the quest for meaning. Despite facing financial difficulties and critical neglect during his lifetime, Melville’s reputation soared posthumously, and he is now considered one of the great American authors." ], kind = INSERT} ] - result_table_name = "fake" + plugin_output = "fake" } } transform { Embedding { - source_table_name = "fake" + plugin_input = "fake" embedding_model_provider = QIANFAN model = bge_large_en api_key = xxxxxxxxxx @@ -182,13 +182,13 @@ transform { book_intro_vector = book_intro author_biography_vector = author_biography } - result_table_name = "embedding_output" + plugin_output = "embedding_output" } } sink { Assert { - source_table_name = "embedding_output" + plugin_input = "embedding_output" rules = { field_rules = [ @@ -293,13 +293,13 @@ source { "Herman Melville (1819–1891) was an American novelist, short story writer, and poet of the American Renaissance period. Born in New York City, Melville gained initial fame with novels such as Typee and Omoo, but it was Moby-Dick, published in 1851, that would later be recognized as his masterpiece. Melville’s work is known for its complexity, symbolism, and exploration of themes such as man’s place in the universe, the nature of evil, and the quest for meaning. Despite facing financial difficulties and critical neglect during his lifetime, Melville’s reputation soared posthumously, and he is now considered one of the great American authors." ], kind = INSERT} ] - result_table_name = "fake" + plugin_output = "fake" } } transform { Embedding { - source_table_name = "fake" + plugin_input = "fake" model_provider = CUSTOM model = text-embedding-3-small api_key = xxxxxxxx @@ -320,13 +320,13 @@ transform { inputx = ["${input}"] } } - result_table_name = "embedding_output_1" + plugin_output = "embedding_output_1" } } sink { Assert { - source_table_name = "embedding_output_1" + plugin_input = "embedding_output_1" rules = { field_rules = [ diff --git a/docs/en/transform-v2/field-mapper.md b/docs/en/transform-v2/field-mapper.md index e0bd32e1492..fa54ced741e 100644 --- a/docs/en/transform-v2/field-mapper.md +++ b/docs/en/transform-v2/field-mapper.md @@ -36,8 +36,8 @@ We want to delete `age` field and update the filed order to `id`, `card`, `name` ``` transform { FieldMapper { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" field_mapper = { id = id card = card diff --git a/docs/en/transform-v2/filter-rowkind.md b/docs/en/transform-v2/filter-rowkind.md index e6ef5ba98cd..68aab44b973 100644 --- a/docs/en/transform-v2/filter-rowkind.md +++ b/docs/en/transform-v2/filter-rowkind.md @@ -39,7 +39,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -53,15 +53,15 @@ source { transform { FilterRowKind { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" exclude_kinds = ["INSERT"] } } sink { Console { - source_table_name = "fake1" + plugin_input = "fake1" } } ``` diff --git a/docs/en/transform-v2/filter.md b/docs/en/transform-v2/filter.md index f9f28b8398a..748934e621a 100644 --- a/docs/en/transform-v2/filter.md +++ b/docs/en/transform-v2/filter.md @@ -43,8 +43,8 @@ we want to keep the field named `name`, `card`, we can add a `Filter` Transform ``` transform { Filter { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" include_fields = [name, card] } } @@ -55,8 +55,8 @@ Or we can delete the field named `age` by adding a `Filter` Transform with `excl ``` transform { Filter { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" exclude_fields = [age] } } diff --git a/docs/en/transform-v2/jsonpath.md b/docs/en/transform-v2/jsonpath.md index 1948f5ca694..f787487069e 100644 --- a/docs/en/transform-v2/jsonpath.md +++ b/docs/en/transform-v2/jsonpath.md @@ -93,8 +93,8 @@ Assuming we want to use JsonPath to extract properties. ```json transform { JsonPath { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" columns = [ { "src_field" = "data" @@ -175,8 +175,8 @@ The JsonPath transform converts the values of seatunnel into an array, ```hocon transform { JsonPath { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" row_error_handle_way = FAIL columns = [ diff --git a/docs/en/transform-v2/llm.md b/docs/en/transform-v2/llm.md index 8ee5a36a9ab..c1c9798abe3 100644 --- a/docs/en/transform-v2/llm.md +++ b/docs/en/transform-v2/llm.md @@ -11,7 +11,7 @@ more. ## Options | name | type | required | default value | -|------------------------| ------ | -------- |---------------| +|------------------------|--------|----------|---------------| | model_provider | enum | yes | | | output_data_type | enum | no | String | | output_column_name | string | no | llm_output | @@ -28,7 +28,9 @@ more. ### model_provider The model provider to use. The available options are: -OPENAI, DOUBAO, KIMIAI, CUSTOM +OPENAI, DOUBAO, KIMIAI, MICROSOFT, CUSTOM + +> tips: If you use Microsoft, please make sure api_path cannot be empty ### output_data_type @@ -254,6 +256,7 @@ sink { } } ``` + ### Customize the LLM model ```hocon @@ -277,13 +280,13 @@ source { {fields = [4, "Eric"], kind = INSERT} {fields = [5, "Guangdong Liu"], kind = INSERT} ] - result_table_name = "fake" + plugin_output = "fake" } } transform { LLM { - source_table_name = "fake" + plugin_input = "fake" model_provider = CUSTOM model = gpt-4o-mini api_key = sk-xxx @@ -308,13 +311,13 @@ transform { }] } } - result_table_name = "llm_output" + plugin_output = "llm_output" } } sink { Assert { - source_table_name = "llm_output" + plugin_input = "llm_output" rules = { field_rules = [ diff --git a/docs/en/transform-v2/metadata.md b/docs/en/transform-v2/metadata.md new file mode 100644 index 00000000000..abae10e4483 --- /dev/null +++ b/docs/en/transform-v2/metadata.md @@ -0,0 +1,85 @@ +# Metadata + +> Metadata transform plugin + +## Description +Metadata transform plugin for adding metadata fields to data + +## Available Metadata + +| Key | DataType | Description | +|:---------:|:--------:|:---------------------------------------------------------------------------------------------------| +| Database | string | Name of the table that contain the row. | +| Table | string | Name of the table that contain the row. | +| RowKind | string | The type of operation | +| EventTime | Long | The time at which the connector processed the event. | +| Delay | Long | The difference between data extraction time and database change time | +| Partition | string | Contains the partition field of the corresponding number table of the row, multiple using `,` join | + +### note + `Delay` `Partition` only worked on cdc series connectors for now , except TiDB-CDC + +## Options + +| name | type | required | default value | Description | +|:---------------:|------|----------|---------------|---------------------------------------------------------------------------| +| metadata_fields | map | yes | | A mapping metadata input fields and their corresponding output fields. | + +### metadata_fields [map] + +A mapping between metadata fields and their respective output fields. + +```hocon +metadata_fields { + Database = c_database + Table = c_table + RowKind = c_rowKind + EventTime = c_ts_ms + Delay = c_delay +} +``` + +## Examples + +```yaml + +env { + parallelism = 1 + job.mode = "STREAMING" + checkpoint.interval = 5000 + read_limit.bytes_per_second = 7000000 + read_limit.rows_per_second = 400 +} + +source { + MySQL-CDC { + plugin_output = "customers_mysql_cdc" + server-id = 5652 + username = "root" + password = "zdyk_Dev@2024" + table-names = ["source.user"] + base-url = "jdbc:mysql://172.16.17.123:3306/source" + } +} + +transform { + Metadata { + metadata_fields { + Database = database + Table = table + RowKind = rowKind + EventTime = ts_ms + Delay = delay + } + plugin_output = "trans_result" + } +} + +sink { + Console { + plugin_input = "custom_name" + } +} + +``` + diff --git a/docs/en/transform-v2/replace.md b/docs/en/transform-v2/replace.md index 1cc99c0ace7..ebb15a9c8ba 100644 --- a/docs/en/transform-v2/replace.md +++ b/docs/en/transform-v2/replace.md @@ -56,8 +56,8 @@ We want to replace the char ` ` to `_` at the `name` field. Then we can add a `R ``` transform { Replace { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" replace_field = "name" pattern = " " replacement = "_" @@ -84,7 +84,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -97,8 +97,8 @@ source { transform { Replace { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" replace_field = "name" pattern = ".+" replacement = "b" @@ -108,7 +108,7 @@ transform { sink { Console { - source_table_name = "fake1" + plugin_input = "fake1" } } ``` diff --git a/docs/en/transform-v2/rowkind-extractor.md b/docs/en/transform-v2/rowkind-extractor.md new file mode 100644 index 00000000000..a2ee384c347 --- /dev/null +++ b/docs/en/transform-v2/rowkind-extractor.md @@ -0,0 +1,113 @@ +# RowKindExtractor + +> RowKindExtractor transform plugin + +## Description + +transform cdc row to append only row that contains the cdc RowKind.
+Example:
+CDC row: -D 1, test1, test2
+transformed Row: +I 1,test1,test2,DELETE + +## Options + +| name | type | required | default value | +|-------------------|--------|----------|---------------| +| custom_field_name | string | yes | row_kind | +| transform_type | enum | yes | SHORT | + +### custom_field_name [string] + +Custom field name of the RowKind field + +### transform_type [enum] + +the RowKind field value formatting , the option can be `SHORT` or `FULL` + +`SHORT` : +I, -U , +U, -D +`FULL` : INSERT, UPDATE_BEFORE, UPDATE_AFTER , DELETE + +## Examples + + +```yaml + +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FakeSource { + schema = { + fields { + pk_id = bigint + name = string + score = int + } + primaryKey { + name = "pk_id" + columnNames = [pk_id] + } + } + rows = [ + { + kind = INSERT + fields = [1, "A", 100] + }, + { + kind = INSERT + fields = [2, "B", 100] + }, + { + kind = INSERT + fields = [3, "C", 100] + }, + { + kind = INSERT + fields = [4, "D", 100] + }, + { + kind = UPDATE_BEFORE + fields = [1, "A", 100] + }, + { + kind = UPDATE_AFTER + fields = [1, "F", 100] + } + { + kind = UPDATE_BEFORE + fields = [2, "B", 100] + }, + { + kind = UPDATE_AFTER + fields = [2, "G", 100] + }, + { + kind = DELETE + fields = [3, "C", 100] + }, + { + kind = DELETE + fields = [4, "D", 100] + } + ] + } +} + +transform { + RowKindExtractor { + custom_field_name = "custom_name" + transform_type = FULL + plugin_output = "trans_result" + } +} + +sink { + Console { + plugin_input = "custom_name" + } +} + +``` + diff --git a/docs/en/transform-v2/split.md b/docs/en/transform-v2/split.md index ecfe94c854b..0df9afbdef2 100644 --- a/docs/en/transform-v2/split.md +++ b/docs/en/transform-v2/split.md @@ -46,8 +46,8 @@ We want split `name` field to `first_name` and `second name`, we can add `Split` ``` transform { Split { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" separator = " " split_field = "name" output_fields = [first_name, second_name] diff --git a/docs/en/transform-v2/sql-functions.md b/docs/en/transform-v2/sql-functions.md index 3438a24de9c..31a33989375 100644 --- a/docs/en/transform-v2/sql-functions.md +++ b/docs/en/transform-v2/sql-functions.md @@ -302,6 +302,14 @@ Example: REPLACE(NAME, ' ') +### SPLIT + +Split a string into an array. + +Example: + +select SPLIT(test,';') as arrays + ### SOUNDEX ```SOUNDEX(string)``` @@ -973,3 +981,37 @@ It is used to determine whether the condition is valid and return different valu Example: case when c_string in ('c_string') then 1 else 0 end + +### UUID + +```UUID()``` + +Generate a uuid through java function. + +Example: + +select UUID() as seatunnel_uuid + +### ARRAY + +Generate an array. + +Example: + +select ARRAY('test1','test2','test3') as arrays + + +### LATERAL VIEW +#### EXPLODE + +explode array column to rows. +OUTER EXPLODE will return NULL, while array is NULL or empty +EXPLODE(SPLIT(FIELD_NAME,separator))Used to split string type. The first parameter of SPLIT function is the field name, the second parameter is the separator +EXPLODE(ARRAY(value1,value2)) Used to custom array type. +``` +SELECT * FROM fake + LATERAL VIEW EXPLODE ( SPLIT ( NAME, ',' ) ) AS NAME + LATERAL VIEW EXPLODE ( SPLIT ( pk_id, ';' ) ) AS pk_id + LATERAL VIEW OUTER EXPLODE ( age ) AS age + LATERAL VIEW OUTER EXPLODE ( ARRAY(1,1) ) AS num +``` diff --git a/docs/en/transform-v2/sql-udf.md b/docs/en/transform-v2/sql-udf.md index df5d3b93fe5..a857fe4c51f 100644 --- a/docs/en/transform-v2/sql-udf.md +++ b/docs/en/transform-v2/sql-udf.md @@ -110,8 +110,8 @@ We use UDF of SQL query to transform the source data like this: ``` transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" query = "select id, example(name) as name, age from fake" } } diff --git a/docs/en/transform-v2/sql.md b/docs/en/transform-v2/sql.md index a3bdb9bbfc1..a8f12568d53 100644 --- a/docs/en/transform-v2/sql.md +++ b/docs/en/transform-v2/sql.md @@ -12,11 +12,11 @@ SQL transform use memory SQL engine, we can via SQL functions and ability of SQL | name | type | required | default value | |-------------------|--------|----------|---------------| -| source_table_name | string | yes | - | -| result_table_name | string | yes | - | +| plugin_input | string | yes | - | +| plugin_output | string | yes | - | | query | string | yes | - | -### source_table_name [string] +### plugin_input [string] The source table name, the query SQL table name must match this field. @@ -43,8 +43,8 @@ We use SQL query to transform the source data like this: ``` transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" query = "select id, concat(name, '_') as name, age+1 as age from fake where id>0" } } @@ -66,7 +66,7 @@ if your upstream data schema is like this: ```hacon source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 string.template = ["innerQuery"] schema = { @@ -123,7 +123,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -137,15 +137,15 @@ source { transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" query = "select id, concat(name, '_') as name, age+1 as age from fake where id>0" } } sink { Console { - source_table_name = "fake1" + plugin_input = "fake1" } } ``` diff --git a/docs/en/transform-v2/transform-multi-table.md b/docs/en/transform-v2/transform-multi-table.md new file mode 100644 index 00000000000..e642ec9cd2d --- /dev/null +++ b/docs/en/transform-v2/transform-multi-table.md @@ -0,0 +1,128 @@ +--- +sidebar_position: 2 +--- + +# Multi-Table Transform in SeaTunnel + +SeaTunnel’s transform feature supports multi-table transformations, which is especially useful when the upstream plugin outputs multiple tables. This allows you to complete all necessary transformation operations within a single transform configuration. Currently, many connectors in SeaTunnel support multi-table outputs, such as `JDBCSource` and `MySQL-CDC`. All transforms can be configured for multi-table transform as described below. + +:::tip + +Multi-table Transform has no limitations on Transform capabilities; any Transform configuration can be used in a multi-table Transform. The purpose of multi-table Transform is to handle multiple tables in the data stream individually and merge the Transform configurations of multiple tables into one Transform for easier management. + +::: + +## Properties + +| Name | Type | Required | Default | Description | +|----------------------------|--------|----------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| table_match_regex | String | No | .* | A regular expression to match the tables that require transformation. By default, it matches all tables. Note that this table name refers to the actual upstream table name, not `result_table_name`. | +| table_transform | List | No | - | You can use a list in `table_transform` to specify rules for individual tables. If a transformation rule is configured for a specific table in `table_transform`, the outer rules will not apply to that table. The rules in `table_transform` take precedence. | +| table_transform.table_path | String | No | - | When configuring a transformation rule for a table in `table_transform`, you need to specify the table path using the `table_path` field. The table path should include `databaseName[.schemaName].tableName`. | + +## Matching Logic + +Suppose we read five tables from upstream: `test.abc`, `test.abcd`, `test.xyz`, `test.xyzxyz`, and `test.www`. They share the same structure, each having three fields: `id`, `name`, and `age`. + +| id | name | age | + +Now, let's say we want to copy the data from these five tables using the Copy transform with the following specific requirements: +- For tables `test.abc` and `test.abcd`, we need to copy the `name` field to a new field `name1`. +- For `test.xyz`, we want to copy the `name` field to `name2`. +- For `test.xyzxyz`, we want to copy the `name` field to `name3`. +- For `test.www`, no changes are needed. + +We can configure this as follows: + +```hocon +transform { + Copy { + source_table_name = "fake" // Optional dataset name to read from + result_table_name = "fake1" // Optional dataset name for output + + table_match_regex = "test.a.*" // 1. Matches tables needing transformation, here matching `test.abc` and `test.abcd` + src_field = "name" // Source field + dest_field = "name1" // Destination field + + table_transform = [{ + table_path = "test.xyz" // 2. Specifies the table name for transformation + src_field = "name" // Source field + dest_field = "name2" // Destination field + }, { + table_path = "test.xyzxyz" + src_field = "name" + dest_field = "name3" + }] + } +} +``` + +### Explanation + +1. With the regular expression and corresponding Copy transform options, we match tables `test.abc` and `test.abcd` and copy the `name` field to `name1`. +2. Using the `table_transform` configuration, we specify that for table `test.xyz`, the `name` field should be copied to `name2`. + +This allows us to handle transformations for multiple tables within a single transform configuration. + +For each table, the priority of configuration is: `table_transform` > `table_match_regex`. If no rules match a table, no transformation will be applied. + +Below are the transform configurations for each table: + +- **test.abc** and **test.abcd** + +```hocon +transform { + Copy { + src_field = "name" + dest_field = "name1" + } +} +``` + +Output structure: + +| id | name | age | name1 | + +- **test.xyz** + +```hocon +transform { + Copy { + src_field = "name" + dest_field = "name2" + } +} +``` + +Output structure: + +| id | name | age | name2 | + +- **test.xyzxyz** + +```hocon +transform { + Copy { + src_field = "name" + dest_field = "name3" + } +} +``` + +Output structure: + +| id | name | age | name3 | + +- **test.www** + +```hocon +transform { + // No transformation needed +} +``` + +Output structure: + +| id | name | age | + +In this example, we used the Copy transform, but all transforms in SeaTunnel support multi-table transformations, and you can configure them similarly within the corresponding transform block. \ No newline at end of file diff --git a/docs/images/icons/Apache Iceberg.svg b/docs/images/icons/Apache Iceberg.svg new file mode 100644 index 00000000000..d04e866a0f6 --- /dev/null +++ b/docs/images/icons/Apache Iceberg.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Doris.svg b/docs/images/icons/Doris.svg new file mode 100644 index 00000000000..2729c9a6985 --- /dev/null +++ b/docs/images/icons/Doris.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/FtpFile.svg b/docs/images/icons/FtpFile.svg new file mode 100644 index 00000000000..4cf14476e97 --- /dev/null +++ b/docs/images/icons/FtpFile.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Greenplum.svg b/docs/images/icons/Greenplum.svg new file mode 100644 index 00000000000..ead7dc6bfeb --- /dev/null +++ b/docs/images/icons/Greenplum.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git "a/docs/images/icons/Hdfs\346\226\207\344\273\266.svg" "b/docs/images/icons/Hdfs\346\226\207\344\273\266.svg" new file mode 100644 index 00000000000..7bc4a938f74 --- /dev/null +++ "b/docs/images/icons/Hdfs\346\226\207\344\273\266.svg" @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Hive.svg b/docs/images/icons/Hive.svg new file mode 100644 index 00000000000..70859e23b97 --- /dev/null +++ b/docs/images/icons/Hive.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/HiveJdbc.svg b/docs/images/icons/HiveJdbc.svg new file mode 100644 index 00000000000..70859e23b97 --- /dev/null +++ b/docs/images/icons/HiveJdbc.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Http.svg b/docs/images/icons/Http.svg new file mode 100644 index 00000000000..e9fcaf50aca --- /dev/null +++ b/docs/images/icons/Http.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/InfluxDB.svg b/docs/images/icons/InfluxDB.svg new file mode 100644 index 00000000000..a0bd1c639b6 --- /dev/null +++ b/docs/images/icons/InfluxDB.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/IoTDB.svg b/docs/images/icons/IoTDB.svg new file mode 100644 index 00000000000..1aad0988b75 --- /dev/null +++ b/docs/images/icons/IoTDB.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/JDBC.svg b/docs/images/icons/JDBC.svg new file mode 100644 index 00000000000..00365006920 --- /dev/null +++ b/docs/images/icons/JDBC.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Jira.svg b/docs/images/icons/Jira.svg new file mode 100644 index 00000000000..e49c6d768f9 --- /dev/null +++ b/docs/images/icons/Jira.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Kafka.png b/docs/images/icons/Kafka.png deleted file mode 100644 index a4b5359b866..00000000000 Binary files a/docs/images/icons/Kafka.png and /dev/null differ diff --git a/docs/images/icons/Kafka.svg b/docs/images/icons/Kafka.svg new file mode 100644 index 00000000000..094d598c4c2 --- /dev/null +++ b/docs/images/icons/Kafka.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Kingbase.svg b/docs/images/icons/Kingbase.svg new file mode 100644 index 00000000000..65a72ff2122 --- /dev/null +++ b/docs/images/icons/Kingbase.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Klaviyo.svg b/docs/images/icons/Klaviyo.svg new file mode 100644 index 00000000000..77f75c139fa --- /dev/null +++ b/docs/images/icons/Klaviyo.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/LocalFile.svg b/docs/images/icons/LocalFile.svg new file mode 100644 index 00000000000..414c3dde3b9 --- /dev/null +++ b/docs/images/icons/LocalFile.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Maxcompute.svg b/docs/images/icons/Maxcompute.svg new file mode 100644 index 00000000000..dca95d03c36 --- /dev/null +++ b/docs/images/icons/Maxcompute.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Milvus.svg b/docs/images/icons/Milvus.svg new file mode 100644 index 00000000000..a057c16e418 --- /dev/null +++ b/docs/images/icons/Milvus.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/MySQL CDC.svg b/docs/images/icons/MySQL CDC.svg new file mode 100644 index 00000000000..92cca4e38d0 --- /dev/null +++ b/docs/images/icons/MySQL CDC.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Notion.svg b/docs/images/icons/Notion.svg new file mode 100644 index 00000000000..3c6e3b0f72f --- /dev/null +++ b/docs/images/icons/Notion.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/ObsFile.png b/docs/images/icons/ObsFile.png new file mode 100644 index 00000000000..be943c607ac Binary files /dev/null and b/docs/images/icons/ObsFile.png differ diff --git a/docs/images/icons/OceanBase.svg b/docs/images/icons/OceanBase.svg new file mode 100644 index 00000000000..e4589987ea6 --- /dev/null +++ b/docs/images/icons/OceanBase.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/OneSignal.svg b/docs/images/icons/OneSignal.svg new file mode 100644 index 00000000000..8f0c26700da --- /dev/null +++ b/docs/images/icons/OneSignal.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/OpenMldb.png b/docs/images/icons/OpenMldb.png new file mode 100644 index 00000000000..b66e8dedef4 Binary files /dev/null and b/docs/images/icons/OpenMldb.png differ diff --git a/docs/images/icons/Oracle CDC.svg b/docs/images/icons/Oracle CDC.svg new file mode 100644 index 00000000000..9f739d77862 --- /dev/null +++ b/docs/images/icons/Oracle CDC.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Oracle.svg b/docs/images/icons/Oracle.svg new file mode 100644 index 00000000000..c4865624c3e --- /dev/null +++ b/docs/images/icons/Oracle.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Paimon.svg b/docs/images/icons/Paimon.svg new file mode 100644 index 00000000000..9dac157fdb6 --- /dev/null +++ b/docs/images/icons/Paimon.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Persistiq.svg b/docs/images/icons/Persistiq.svg new file mode 100644 index 00000000000..2ab14f08a78 --- /dev/null +++ b/docs/images/icons/Persistiq.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Phoenix.svg b/docs/images/icons/Phoenix.svg new file mode 100644 index 00000000000..6fa6e48a403 --- /dev/null +++ b/docs/images/icons/Phoenix.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/PostgreSQL CDC.svg b/docs/images/icons/PostgreSQL CDC.svg new file mode 100644 index 00000000000..38547f16078 --- /dev/null +++ b/docs/images/icons/PostgreSQL CDC.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/PostgreSQL.svg b/docs/images/icons/PostgreSQL.svg new file mode 100644 index 00000000000..38547f16078 --- /dev/null +++ b/docs/images/icons/PostgreSQL.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Pulsar.svg b/docs/images/icons/Pulsar.svg new file mode 100644 index 00000000000..cabedf1e022 --- /dev/null +++ b/docs/images/icons/Pulsar.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Qdrant.svg b/docs/images/icons/Qdrant.svg new file mode 100644 index 00000000000..b431d111a6a --- /dev/null +++ b/docs/images/icons/Qdrant.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Rabbitmq.svg b/docs/images/icons/Rabbitmq.svg new file mode 100644 index 00000000000..a4ecbc6cfbf --- /dev/null +++ b/docs/images/icons/Rabbitmq.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Redis.svg b/docs/images/icons/Redis.svg new file mode 100644 index 00000000000..4cbd41cada9 --- /dev/null +++ b/docs/images/icons/Redis.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/RocketMQ.svg b/docs/images/icons/RocketMQ.svg new file mode 100644 index 00000000000..3fd2c1adba9 --- /dev/null +++ b/docs/images/icons/RocketMQ.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/S3File.svg b/docs/images/icons/S3File.svg new file mode 100644 index 00000000000..ddd50aeff00 --- /dev/null +++ b/docs/images/icons/S3File.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/SQL Server.svg b/docs/images/icons/SQL Server.svg new file mode 100644 index 00000000000..db4b76ca740 --- /dev/null +++ b/docs/images/icons/SQL Server.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Sftp.svg b/docs/images/icons/Sftp.svg new file mode 100644 index 00000000000..2a8015eb504 --- /dev/null +++ b/docs/images/icons/Sftp.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Snowflake.svg b/docs/images/icons/Snowflake.svg new file mode 100644 index 00000000000..fb4c2868fba --- /dev/null +++ b/docs/images/icons/Snowflake.svg @@ -0,0 +1,3 @@ + diff --git a/docs/images/icons/StarRocks.svg b/docs/images/icons/StarRocks.svg new file mode 100644 index 00000000000..10a52bbf355 --- /dev/null +++ b/docs/images/icons/StarRocks.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/TDengine.svg b/docs/images/icons/TDengine.svg new file mode 100644 index 00000000000..588347b3727 --- /dev/null +++ b/docs/images/icons/TDengine.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Tablestore.svg b/docs/images/icons/Tablestore.svg new file mode 100644 index 00000000000..24526c988b9 --- /dev/null +++ b/docs/images/icons/Tablestore.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/images/icons/Typesense.png b/docs/images/icons/Typesense.png new file mode 100644 index 00000000000..f25cc7e9e71 Binary files /dev/null and b/docs/images/icons/Typesense.png differ diff --git a/docs/images/icons/Web3j.png b/docs/images/icons/Web3j.png new file mode 100644 index 00000000000..ec031cb3280 Binary files /dev/null and b/docs/images/icons/Web3j.png differ diff --git a/docs/images/ui/detail.png b/docs/images/ui/detail.png new file mode 100644 index 00000000000..a376b6e4880 Binary files /dev/null and b/docs/images/ui/detail.png differ diff --git a/docs/images/ui/finished.png b/docs/images/ui/finished.png new file mode 100644 index 00000000000..fa800bd6029 Binary files /dev/null and b/docs/images/ui/finished.png differ diff --git a/docs/images/ui/master.png b/docs/images/ui/master.png new file mode 100644 index 00000000000..5e42d2854ee Binary files /dev/null and b/docs/images/ui/master.png differ diff --git a/docs/images/ui/overview.png b/docs/images/ui/overview.png new file mode 100644 index 00000000000..67123532499 Binary files /dev/null and b/docs/images/ui/overview.png differ diff --git a/docs/images/ui/running.png b/docs/images/ui/running.png new file mode 100644 index 00000000000..889edb303b1 Binary files /dev/null and b/docs/images/ui/running.png differ diff --git a/docs/images/ui/workers.png b/docs/images/ui/workers.png new file mode 100644 index 00000000000..a2bf39ec218 Binary files /dev/null and b/docs/images/ui/workers.png differ diff --git a/docs/sidebars.js b/docs/sidebars.js index e3bb42f9e3f..3257181b11a 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -202,10 +202,12 @@ const sidebars = { "seatunnel-engine/engine-jar-storage-mode", "seatunnel-engine/tcp", "seatunnel-engine/resource-isolation", - "seatunnel-engine/rest-api", + "seatunnel-engine/rest-api-v1", + "seatunnel-engine/rest-api-v2", "seatunnel-engine/user-command", "seatunnel-engine/logging", - "seatunnel-engine/telemetry" + "seatunnel-engine/telemetry", + "seatunnel-engine/web-ui" ] }, { @@ -224,6 +226,7 @@ const sidebars = { 'contribution/new-license', 'contribution/coding-guide', 'contribution/contribute-transform-v2-guide', + 'contribution/how-to-create-your-connector' ], }, "faq" diff --git a/docs/zh/about.md b/docs/zh/about.md index c938cc7b62e..244b27af1ae 100644 --- a/docs/zh/about.md +++ b/docs/zh/about.md @@ -62,7 +62,7 @@ SeaTunnel 拥有大量用户。 您可以在[用户](https://seatunnel.apache.or
-SeaTunnel 丰富了CNCF 云原生景观。 +SeaTunnel 丰富了CNCF 云原生景观。 ## 了解更多 diff --git a/docs/zh/concept/config.md b/docs/zh/concept/config.md index 2206441e801..98bf85735fe 100644 --- a/docs/zh/concept/config.md +++ b/docs/zh/concept/config.md @@ -15,6 +15,12 @@ 配置文件类似下面这个例子： +:::warn + +旧的配置名称 `result_table_name`/`source_table_name` 已经过时，请尽快迁移到新名称 `plugin_output`/`plugin_input`。 + +::: + ### hocon ```hocon @@ -24,7 +30,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -38,8 +44,8 @@ source { transform { Filter { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" fields = [name, card] } } @@ -52,7 +58,7 @@ sink { fields = ["name", "card"] username = "default" password = "" - source_table_name = "fake1" + plugin_input = "fake1" } } ``` @@ -72,7 +78,7 @@ sink { source用于定义SeaTunnel在哪儿检索数据，并将检索的数据用于下一步。 可以同时定义多个source。目前支持的source请看[Source of SeaTunnel](../../en/connector-v2/source)。每种source都有自己特定的参数用来 -定义如何检索数据，SeaTunnel也抽象了每种source所使用的参数，例如 `result_table_name` 参数，用于指定当前source生成的数据的名称， +定义如何检索数据，SeaTunnel也抽象了每种source所使用的参数，例如 `plugin_output` 参数，用于指定当前source生成的数据的名称， 方便后续其他模块使用。 ### transform @@ -87,7 +93,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -107,7 +113,7 @@ sink { fields = ["name", "age", "card"] username = "default" password = "" - source_table_name = "fake1" + plugin_input = "fake1" } } ``` @@ -123,10 +129,10 @@ sink模块，你可以快速高效地完成这个操作。Sink和source非常相 ### 其它 -你会疑惑当定义了多个source和多个sink时，每个sink读取哪些数据，每个transform读取哪些数据？我们使用`result_table_name` 和 -`source_table_name` 两个配置。每个source模块都会配置一个`result_table_name`来指示数据源生成的数据源名称，其它transform和sink -模块可以使用`source_table_name` 引用相应的数据源名称，表示要读取数据进行处理。然后transform，作为一个中间的处理模块，可以同时使用 -`result_table_name` 和 `source_table_name` 配置。但你会发现在上面的配置例子中，不是每个模块都配置了这些参数，因为在SeaTunnel中， +你会疑惑当定义了多个source和多个sink时，每个sink读取哪些数据，每个transform读取哪些数据？我们使用`plugin_output` 和 +`plugin_input` 两个配置。每个source模块都会配置一个`plugin_output`来指示数据源生成的数据源名称，其它transform和sink +模块可以使用`plugin_input` 引用相应的数据源名称，表示要读取数据进行处理。然后transform，作为一个中间的处理模块，可以同时使用 +`plugin_output` 和 `plugin_input` 配置。但你会发现在上面的配置例子中，不是每个模块都配置了这些参数，因为在SeaTunnel中， 有一个默认的约定，如果这两个参数没有配置，则使用上一个节点的最后一个模块生成的数据。当只有一个source时这是非常方便的。 ## 多行文本支持 @@ -155,7 +161,7 @@ sql = """ select * from "table" """ "source": [ { "plugin_name": "FakeSource", - "result_table_name": "fake", + "plugin_output": "fake", "row.num": 100, "schema": { "fields": { @@ -169,8 +175,8 @@ sql = """ select * from "table" """ "transform": [ { "plugin_name": "Filter", - "source_table_name": "fake", - "result_table_name": "fake1", + "plugin_input": "fake", + "plugin_output": "fake1", "fields": ["name", "card"] } ], @@ -183,7 +189,7 @@ sql = """ select * from "table" """ "fields": ["name", "card"], "username": "default", "password": "", - "source_table_name": "fake1" + "plugin_input": "fake1" } ] } @@ -218,7 +224,7 @@ env { source { FakeSource { - result_table_name = "${resName:fake_test}_table" + plugin_output = "${resName:fake_test}_table" row.num = "${rowNum:50}" string.template = ${strTemplate} int.template = [20, 21] @@ -233,8 +239,8 @@ source { transform { sql { - source_table_name = "${resName:fake_test}_table" - result_table_name = "sql" + plugin_input = "${resName:fake_test}_table" + plugin_output = "sql" query = "select * from ${resName:fake_test}_table where name = '${nameVal}' " } @@ -242,7 +248,7 @@ transform { sink { Console { - source_table_name = "sql" + plugin_input = "sql" username = ${username} password = ${password} } @@ -277,7 +283,7 @@ env { source { FakeSource { - result_table_name = "fake_test_table" + plugin_output = "fake_test_table" row.num = 50 string.template = ['abc','d~f','hi'] int.template = [20, 21] @@ -292,8 +298,8 @@ source { transform { sql { - source_table_name = "fake_test_table" - result_table_name = "sql" + plugin_input = "fake_test_table" + plugin_output = "sql" query = "select * from fake_test_table where name = 'abc' " } @@ -301,7 +307,7 @@ transform { sink { Console { - source_table_name = "sql" + plugin_input = "sql" username = "seatunnel=2.3.1" password = "$a^b%c.d~e0*9(" } diff --git a/docs/zh/concept/schema-evolution.md b/docs/zh/concept/schema-evolution.md index 16f0f5dbc81..57d562946ff 100644 --- a/docs/zh/concept/schema-evolution.md +++ b/docs/zh/concept/schema-evolution.md @@ -6,15 +6,19 @@ ### 源 [Mysql-CDC](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/MySQL-CDC.md) +[Oracle-CDC](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/Oracle-CDC.md) ### 目标 [Jdbc-Mysql](https://github.com/apache/seatunnel/blob/dev/docs/zh/connector-v2/sink/Jdbc.md) +[Jdbc-Oracle](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Jdbc.md) -注意: 目前模式演进不支持transform. +注意: 目前模式演进不支持transform。不同类型数据库(Oracle-CDC -> Jdbc-Mysql)的模式演进目前不支持ddl中列的默认值。 +当你使用Oracle-CDC时，你不能使用用户名`SYS`或`SYSTEM`来修改表结构，否则ddl事件将被过滤，这可能导致模式演进不起作用； +另外，如果你的表名以`ORA_TEMP_`开头，也会有相同的问题。 ## 启用Schema evolution功能 -在CDC源连接器中模式演进默认是关闭的。你需要在CDC连接器中配置`debezium.include.schema.changes = true`来启用它。 +在CDC源连接器中模式演进默认是关闭的。你需要在CDC连接器中配置`debezium.include.schema.changes = true`来启用它。当你使用Oracle-CDC并且启用schema-evolution时，你必须将`debezium`属性中的`log.mining.strategy`指定为`redo_log_catalog`。 ## 示例 @@ -57,3 +61,92 @@ sink { } } ``` + +### Oracle-cdc -> Jdbc-Oracle +``` +env { + # You can set engine configuration here + parallelism = 1 + job.mode = "STREAMING" + checkpoint.interval = 5000 +} + +source { + # This is a example source plugin **only for test and demonstrate the feature source plugin** + Oracle-CDC { + plugin_output = "customers" + username = "dbzuser" + password = "dbz" + database-names = ["ORCLCDB"] + schema-names = ["DEBEZIUM"] + table-names = ["ORCLCDB.DEBEZIUM.FULL_TYPES"] + base-url = "jdbc:oracle:thin:@oracle-host:1521/ORCLCDB" + source.reader.close.timeout = 120000 + connection.pool.size = 1 + debezium { + include.schema.changes = true + log.mining.strategy = redo_log_catalog + } + } +} + +sink { + Jdbc { + plugin_input = "customers" + driver = "oracle.jdbc.driver.OracleDriver" + url = "jdbc:oracle:thin:@oracle-host:1521/ORCLCDB" + user = "dbzuser" + password = "dbz" + generate_sink_sql = true + database = "ORCLCDB" + table = "DEBEZIUM.FULL_TYPES_SINK" + batch_size = 1 + primary_keys = ["ID"] + connection.pool.size = 1 + } +} +``` + +### Oracle-cdc -> Jdbc-Mysql +``` +env { + # You can set engine configuration here + parallelism = 1 + job.mode = "STREAMING" + checkpoint.interval = 5000 +} + +source { + # This is a example source plugin **only for test and demonstrate the feature source plugin** + Oracle-CDC { + plugin_output = "customers" + username = "dbzuser" + password = "dbz" + database-names = ["ORCLCDB"] + schema-names = ["DEBEZIUM"] + table-names = ["ORCLCDB.DEBEZIUM.FULL_TYPES"] + base-url = "jdbc:oracle:thin:@oracle-host:1521/ORCLCDB" + source.reader.close.timeout = 120000 + connection.pool.size = 1 + debezium { + include.schema.changes = true + log.mining.strategy = redo_log_catalog + } + } +} + +sink { + jdbc { + plugin_input = "customers" + url = "jdbc:mysql://oracle-host:3306/oracle_sink" + driver = "com.mysql.cj.jdbc.Driver" + user = "st_user_sink" + password = "mysqlpw" + generate_sink_sql = true + # You need to configure both database and table + database = oracle_sink + table = oracle_cdc_2_mysql_sink_table + primary_keys = ["ID"] + } +} +``` diff --git a/docs/zh/concept/schema-feature.md b/docs/zh/concept/schema-feature.md index d719a7953e5..b504d264f83 100644 --- a/docs/zh/concept/schema-feature.md +++ b/docs/zh/concept/schema-feature.md @@ -172,6 +172,46 @@ constraintKeys = [ | INDEX_KEY | 键 | | UNIQUE_KEY | 唯一键 | +## 多表Schema + +``` +tables_configs = [ + { + schema { + table = "database.schema.table1" + schema_first = false + comment = "comment" + columns = [ + ... + ] + primaryKey { + ... + } + constraintKeys { + ... + } + } + }, + { + schema = { + table = "database.schema.table2" + schema_first = false + comment = "comment" + columns = [ + ... + ] + primaryKey { + ... + } + constraintKeys { + ... + } + } + } +] + +``` + ## 如何使用schema ### 推荐 @@ -180,7 +220,7 @@ constraintKeys = [ source { FakeSource { parallelism = 2 - result_table_name = "fake" + plugin_output = "fake" row.num = 16 schema { table = "FakeDatabase.FakeTable" @@ -234,7 +274,7 @@ source { source { FakeSource { parallelism = 2 - result_table_name = "fake" + plugin_output = "fake" row.num = 16 schema = { fields { diff --git a/docs/zh/connector-v2/Config-Encryption-Decryption.md b/docs/zh/connector-v2/Config-Encryption-Decryption.md index e7b13aea86a..c2b7ced0ab6 100644 --- a/docs/zh/connector-v2/Config-Encryption-Decryption.md +++ b/docs/zh/connector-v2/Config-Encryption-Decryption.md @@ -43,7 +43,7 @@ Base64编码支持加密以下参数： source { MySQL-CDC { - result_table_name = "fake" + plugin_output = "fake" parallelism = 1 server-id = 5656 port = 56725 @@ -97,7 +97,7 @@ Base64编码支持加密以下参数： "port" : 56725, "database-name" : "inventory_vwyw0n", "parallelism" : 1, - "result_table_name" : "fake", + "plugin_output" : "fake", "table-name" : "products", "plugin_name" : "MySQL-CDC", "server-id" : 5656, diff --git a/docs/zh/connector-v2/formats/avro.md b/docs/zh/connector-v2/formats/avro.md index 7176f4e507f..826fc27b448 100644 --- a/docs/zh/connector-v2/formats/avro.md +++ b/docs/zh/connector-v2/formats/avro.md @@ -51,7 +51,7 @@ source { } } } - result_table_name = "fake" + plugin_output = "fake" } } @@ -76,7 +76,7 @@ source { Kafka { bootstrap.servers = "kafkaCluster:9092" topic = "test_avro_topic" - result_table_name = "kafka_table" + plugin_output = "kafka_table" start_mode = "earliest" format = avro format_error_handle_way = skip @@ -104,7 +104,7 @@ source { sink { Console { - source_table_name = "kafka_table" + plugin_input = "kafka_table" } } ``` diff --git a/docs/zh/connector-v2/formats/canal-json.md b/docs/zh/connector-v2/formats/canal-json.md index 92c4338eb56..fc3344d963c 100644 --- a/docs/zh/connector-v2/formats/canal-json.md +++ b/docs/zh/connector-v2/formats/canal-json.md @@ -86,7 +86,7 @@ source { Kafka { bootstrap.servers = "kafkaCluster:9092" topic = "products_binlog" - result_table_name = "kafka_name" + plugin_output = "kafka_name" start_mode = earliest schema = { fields { diff --git a/docs/zh/connector-v2/formats/cdc-compatible-debezium-json.md b/docs/zh/connector-v2/formats/cdc-compatible-debezium-json.md index e34a5b39a22..8febab18fb0 100644 --- a/docs/zh/connector-v2/formats/cdc-compatible-debezium-json.md +++ b/docs/zh/connector-v2/formats/cdc-compatible-debezium-json.md @@ -17,7 +17,7 @@ env { source { MySQL-CDC { - result_table_name = "table1" + plugin_output = "table1" base-url="jdbc:mysql://localhost:3306/test" "startup.mode"=INITIAL @@ -43,9 +43,10 @@ source { sink { Kafka { - source_table_name = "table1" + plugin_input = "table1" bootstrap.servers = "localhost:9092" + topic = "${topic}" # compatible_debezium_json options format = compatible_debezium_json diff --git a/docs/zh/connector-v2/formats/debezium-json.md b/docs/zh/connector-v2/formats/debezium-json.md index 3e70a5d31ed..88b32540395 100644 --- a/docs/zh/connector-v2/formats/debezium-json.md +++ b/docs/zh/connector-v2/formats/debezium-json.md @@ -85,7 +85,7 @@ source { Kafka { bootstrap.servers = "kafkaCluster:9092" topic = "products_binlog" - result_table_name = "kafka_name" + plugin_output = "kafka_name" start_mode = earliest schema = { fields { diff --git a/docs/zh/connector-v2/formats/kafka-compatible-kafkaconnect-json.md b/docs/zh/connector-v2/formats/kafka-compatible-kafkaconnect-json.md index d0ceb58ac6c..027d90ded07 100644 --- a/docs/zh/connector-v2/formats/kafka-compatible-kafkaconnect-json.md +++ b/docs/zh/connector-v2/formats/kafka-compatible-kafkaconnect-json.md @@ -16,7 +16,7 @@ source { Kafka { bootstrap.servers = "localhost:9092" topic = "jdbc_source_record" - result_table_name = "kafka_table" + plugin_output = "kafka_table" start_mode = earliest schema = { fields { diff --git a/docs/zh/connector-v2/formats/ogg-json.md b/docs/zh/connector-v2/formats/ogg-json.md index 7b64f5b5e41..80c88e6ac13 100644 --- a/docs/zh/connector-v2/formats/ogg-json.md +++ b/docs/zh/connector-v2/formats/ogg-json.md @@ -66,7 +66,7 @@ source { Kafka { bootstrap.servers = "127.0.0.1:9092" topic = "ogg" - result_table_name = "kafka_name" + plugin_output = "kafka_name" start_mode = earliest schema = { fields { diff --git a/docs/zh/connector-v2/formats/protobuf.md b/docs/zh/connector-v2/formats/protobuf.md new file mode 100644 index 00000000000..5fac7f93211 --- /dev/null +++ b/docs/zh/connector-v2/formats/protobuf.md @@ -0,0 +1,164 @@ +# Protobuf 格式 + +Protobuf（Protocol Buffers）是一种由Google开发的语言中立、平台无关的数据序列化格式。它提供了一种高效的方式来编码结构化数据，同时支持多种编程语言和平台。 + +目前支持在 Kafka 中使用 protobuf 格式。 + +## Kafka 使用示例 + +- 模拟随机生成数据源,并以 protobuf 的格式 写入 kafka 的实例 + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FakeSource { + parallelism = 1 + plugin_output = "fake" + row.num = 16 + schema = { + fields { + c_int32 = int + c_int64 = long + c_float = float + c_double = double + c_bool = boolean + c_string = string + c_bytes = bytes + + Address { + city = string + state = string + street = string + } + attributes = "map
+> Flink
+> SeaTunnel Zeta
+ +## 主要特性 + +- [ ] [exactly-once](../../concept/connector-v2-features.md) +- [ ] [cdc](../../concept/connector-v2-features.md) +- [x] [support multiple table write](../../concept/connector-v2-features.md) + +## 描述 + +接收Source端传入的数据，利用数据触发 web hooks。 + +> 例如，来自上游的数据为 [`label: {"__name__": "test1"}, value: 1.2.3,time:2024-08-15T17:00:00`], 则body内容如下: `{"label":{"__name__": "test1"}, "value":"1.23","time":"2024-08-15T17:00:00"}` + +**Tips: Prometheus 数据接收器 仅支持 `post json` 类型的 web hook，source 数据将被视为 webhook 中的 body 内容。并且不支持传递过去太久的数据** + +## 支持的数据源信息 + +想使用 Prometheus 连接器，需要安装以下必要的依赖。可以通过运行 install-plugin.sh 脚本或者从 Maven 中央仓库下载这些依赖 + +| 数据源 | 支持版本 | 依赖 | +|------|-----------|------------------------------------------------------------------------------------------------------------------| +| Http | universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2/connector-prometheus) | + +## 接收器选项 + +| Name | Type | Required | Default | Description | +|-----------------------------|--------|----------|---------|-------------------------------------------------------------------| +| url | String | Yes | - | Http 请求链接 | +| headers | Map | No | - | Http 标头 | +| retry | Int | No | - | 如果请求http返回`IOException`的最大重试次数 | +| retry_backoff_multiplier_ms | Int | No | 100 | http请求失败，重试回退次数（毫秒）乘数 | +| retry_backoff_max_ms | Int | No | 10000 | http请求失败，最大重试回退时间(毫秒) | +| connect_timeout_ms | Int | No | 12000 | 连接超时设置，默认12s | +| socket_timeout_ms | Int | No | 60000 | 套接字超时设置，默认为60s | +| key_timestamp | Int | NO | - | prometheus时间戳的key. | +| key_label | String | yes | - | prometheus标签的key | +| key_value | Double | yes | - | prometheus值的key | +| batch_size | Int | false | 1024 | prometheus批量写入大小 | +| flush_interval | Long | false | 300000L | prometheus定时写入 | +| common-options | | No | - | Sink插件常用参数，请参考 [Sink常用选项 ](../sink-common-options.md) 了解详情 | + +## 示例 + +简单示例: + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FakeSource { + schema = { + fields { + c_map = "map
+> Flink
+> Seatunnel Zeta
+ +## 主要特性 + +- [ ] [exactly-once](../../concept/connector-v2-features.md) +- [ ] [cdc](../../concept/connector-v2-features.md) + +## 描述 + +Sink connector for Aliyun Sls. + +从写入数据到阿里云Sls日志服务 + +为了使用Sls连接器，需要以下依赖关系。 +它们可以通过install-plugin.sh或Maven中央存储库下载。 + +| Datasource | Supported Versions | Maven | +|------------|--------------------|-----------------------------------------------------------------------------------| +| Sls | Universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-sls) | + +## 支持的数据源信息 + +| Name | Type | Required | Default | Description | +|-------------------------------------|----------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------| +| project | String | Yes | - | [阿里云 Sls 项目](https://help.aliyun.com/zh/sls/user-guide/manage-a-project?spm=a2c4g.11186623.0.0.6f9755ebyfaYSl) | +| logstore | String | Yes | - | [阿里云 Sls 日志库](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore?spm=a2c4g.11186623.0.0.13137c08nfuiBC) | +| endpoint | String | Yes | - | [阿里云访问服务点](https://help.aliyun.com/zh/sls/developer-reference/api-sls-2020-12-30-endpoint?spm=a2c4g.11186623.0.0.548945a8UyJULa) | +| access_key_id | String | Yes | - | [阿里云访问用户ID](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479) | +| access_key_secret | String | Yes | - | [阿里云访问用户密码](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479) | +| source | String | No | SeaTunnel-Source | 在sls中数据来源标记 | +| topic | String | No | SeaTunnel-Topic | 在sls中数据主题标记 | + +## 任务示例 + +### 简单示例 + +> 此示例写入sls的logstore1的数据。如果您尚未安装和部署SeaTunnel，则需要按照安装SeaTunnel中的说明安装和部署SeaTunnel。然后按照[快速启动SeaTunnel引擎](../../Start-v2/locale/Quick-Start SeaTunnel Engine.md)中的说明运行此作业。 + +[创建RAM用户及授权](https://help.aliyun.com/zh/sls/create-a-ram-user-and-authorize-the-ram-user-to-access-log-service?spm=a2c4g.11186623.0.i4), 请确认RAM用户有足够的权限来读取及管理数据，参考：[RAM自定义授权示例](https://help.aliyun.com/zh/sls/use-custom-policies-to-grant-permissions-to-a-ram-user?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#reference-s3z-m1l-z2b) + +```hocon +# Defining the runtime environment +env { + parallelism = 2 + job.mode = "STREAMING" + checkpoint.interval = 30000 +} +source { + FakeSource { + row.num = 10 + map.size = 10 + array.size = 10 + bytes.length = 10 + string.length = 10 + schema = { + fields = { + id = "int" + name = "string" + description = "string" + weight = "string" + } + } + } +} + +sink { + Sls { + endpoint = "cn-hangzhou-intranet.log.aliyuncs.com" + project = "project1" + logstore = "logstore1" + access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx" + access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" + } +} +``` + diff --git a/docs/zh/connector-v2/sink/Typesense.md b/docs/zh/connector-v2/sink/Typesense.md index 99017f32cb5..f6c06e5f2b5 100644 --- a/docs/zh/connector-v2/sink/Typesense.md +++ b/docs/zh/connector-v2/sink/Typesense.md @@ -79,7 +79,7 @@ Sink插件常用参数，请参考 [Sink常用选项](../sink-common-options.md) ```bash sink { Typesense { - source_table_name = "typesense_test_table" + plugin_input = "typesense_test_table" hosts = ["localhost:8108"] collection = "typesense_to_typesense_sink_with_query" max_retry_count = 3 diff --git a/docs/zh/connector-v2/source-common-options.md b/docs/zh/connector-v2/source-common-options.md index e8f8798d628..9a95c163390 100644 --- a/docs/zh/connector-v2/source-common-options.md +++ b/docs/zh/connector-v2/source-common-options.md @@ -6,14 +6,20 @@ sidebar_position: 3 > Source connector 的常用参数 -| 名称 | 类型 | 必填 | 默认值 | 描述 | -|-------------------|--------|----|-----|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| result_table_name | String | 否 | - | 当未指定 `result_table_name` 时，此插件处理的数据将不会被注册为可由其他插件直接访问的数据集 `(dataStream/dataset)`，或称为临时表 `(table)`。
当指定了 `result_table_name` 时，此插件处理的数据将被注册为可由其他插件直接访问的数据集 `(dataStream/dataset)`，或称为临时表 `(table)`。此处注册的数据集 `(dataStream/dataset)` 可通过指定 `source_table_name` 直接被其他插件访问。 | -| parallelism | Int | 否 | - | 当未指定 `parallelism` 时，默认使用环境中的 `parallelism`。
当指定了 `parallelism` 时，将覆盖环境中的 `parallelism` 设置。 | +:::warn + +旧的配置名称 `result_table_name` 已经过时，请尽快迁移到新名称 `plugin_output`。 + +::: + +| 名称 | 类型 | 必填 | 默认值 | 描述 | +|---------------|--------|----|-----|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| plugin_output | String | 否 | - | 当未指定 `plugin_output` 时，此插件处理的数据将不会被注册为可由其他插件直接访问的数据集 `(dataStream/dataset)`，或称为临时表 `(table)`。
当指定了 `plugin_output` 时，此插件处理的数据将被注册为可由其他插件直接访问的数据集 `(dataStream/dataset)`，或称为临时表 `(table)`。此处注册的数据集 `(dataStream/dataset)` 可通过指定 `plugin_input` 直接被其他插件访问。 | +| parallelism | Int | 否 | - | 当未指定 `parallelism` 时，默认使用环境中的 `parallelism`。
当指定了 `parallelism` 时，将覆盖环境中的 `parallelism` 设置。 | # 重要提示 -在作业配置中使用 `result_table_name` 时，必须设置 `source_table_name` 参数。 +在作业配置中使用 `plugin_output` 时，必须设置 `plugin_input` 参数。 ## 任务示例 @@ -24,7 +30,7 @@ sidebar_position: 3 ```bash source { FakeSourceStream { - result_table_name = "fake_table" + plugin_output = "fake_table" } } ``` @@ -40,7 +46,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -64,9 +70,9 @@ source { transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" - # 查询表名必须与字段 'source_table_name' 相同 + plugin_input = "fake" + plugin_output = "fake1" + # 查询表名必须与字段 'plugin_input' 相同 query = "select id, regexp_replace(name, '.+', 'b') as name, age+1 as age, pi() as pi, c_timestamp, c_date, c_map, c_array, c_decimal, c_row from fake" } # SQL 转换支持基本函数和条件操作 @@ -75,10 +81,10 @@ transform { sink { Console { - source_table_name = "fake1" + plugin_input = "fake1" } Console { - source_table_name = "fake" + plugin_input = "fake" } } ``` diff --git a/docs/zh/connector-v2/source/Doris.md b/docs/zh/connector-v2/source/Doris.md new file mode 100644 index 00000000000..ba3549473a5 --- /dev/null +++ b/docs/zh/connector-v2/source/Doris.md @@ -0,0 +1,212 @@ +# Doris + +> Doris 源连接器 + +## 支持的引擎 + +> Spark
+> Flink
+> SeaTunnel Zeta
+ +## 主要功能 + +- [x] [批处理](../../concept/connector-v2-features.md) +- [ ] [流处理](../../concept/connector-v2-features.md) +- [ ] [精确一次](../../concept/connector-v2-features.md) +- [x] [列投影](../../concept/connector-v2-features.md) +- [x] [并行度](../../concept/connector-v2-features.md) +- [x] [支持用户自定义分片](../../concept/connector-v2-features.md) +- [x] [支持多表读](../../concept/connector-v2-features.md) + +## 描述 + +用于 Apache Doris 的源连接器。 + +## 支持的数据源信息 + +| 数据源 | 支持版本 | 驱动 | Url | Maven | +|------------|--------------------------------------|--------|-----|-------| +| Doris | 仅支持Doris2.0及以上版本. | - | - | - | + +## 数据类型映射 + +| Doris 数据类型 | SeaTunnel 数据类型 | +|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------| +| INT | INT | +| TINYINT | TINYINT | +| SMALLINT | SMALLINT | +| BIGINT | BIGINT | +| LARGEINT | STRING | +| BOOLEAN | BOOLEAN | +| DECIMAL | DECIMAL((Get the designated column's specified column size)+1,
(Gets the designated column's number of digits to right of the decimal point.))) | +| FLOAT | FLOAT | +| DOUBLE | DOUBLE | +| CHAR
VARCHAR
STRING
TEXT | STRING | +| DATE | DATE | +| DATETIME
DATETIME(p) | TIMESTAMP | +| ARRAY | ARRAY | + +## 源选项 + +基础配置: + +| 名称 | 类型 | 是否必须 | 默认值 | 描述 | +|----------------------------------|--------|----------|------------|-----------------------------------------------------------------------------------------------------| +| fenodes | string | yes | - | FE 地址, 格式：`"fe_host:fe_http_port"` | +| username | string | yes | - | 用户名 | +| password | string | yes | - | 密码 | +| doris.request.retries | int | no | 3 | 请求Doris FE的重试次数 | +| doris.request.read.timeout.ms | int | no | 30000 | | +| doris.request.connect.timeout.ms | int | no | 30000 | | +| query-port | string | no | 9030 | Doris查询端口 | +| doris.request.query.timeout.s | int | no | 3600 | Doris扫描数据的超时时间，单位秒 | +| table_list | string | 否 | - | 表清单 | + +表清单配置: + +| 名称 | 类型 | 是否必须 | 默认值 | 描述 | +|----------------------------------|--------|----------|------------|-----------------------------------------------------------------------------------------------------| +| database | string | yes | - | 数据库 | +| table | string | yes | - | 表名 | +| doris.read.field | string | no | - | 选择要读取的Doris表字段 | +| doris.filter.query | string | no | - | 数据过滤. 格式："字段 = 值", 例如：doris.filter.query = "F_ID > 2" | +| doris.batch.size | int | no | 1024 | 每次能够从BE中读取到的最大行数 | +| doris.exec.mem.limit | long | no | 2147483648 | 单个be扫描请求可以使用的最大内存。默认内存为2G（2147483648） | + +注意: 当此配置对应于单个表时，您可以将table_list中的配置项展平到外层。 + +### 提示 + +> 不建议随意修改高级参数 + +## 例子 + +### 单表 +> 这是一个从doris读取数据后，输出到控制台的例子： + +``` +env { + parallelism = 2 + job.mode = "BATCH" +} +source{ + Doris { + fenodes = "doris_e2e:8030" + username = root + password = "" + database = "e2e_source" + table = "doris_e2e_table" + } +} + +transform { + # If you would like to get more information about how to configure seatunnel and see full list of transform plugins, + # please go to https://seatunnel.apache.org/docs/transform/sql +} + +sink { + Console {} +} +``` + +使用`doris.read.field`参数来选择需要读取的Doris表字段： + +``` +env { + parallelism = 2 + job.mode = "BATCH" +} +source{ + Doris { + fenodes = "doris_e2e:8030" + username = root + password = "" + database = "e2e_source" + table = "doris_e2e_table" + doris.read.field = "F_ID,F_INT,F_BIGINT,F_TINYINT,F_SMALLINT" + } +} + +transform { + # If you would like to get more information about how to configure seatunnel and see full list of transform plugins, + # please go to https://seatunnel.apache.org/docs/transform/sql +} + +sink { + Console {} +} +``` + +使用`doris.filter.query`来过滤数据，参数值将作为过滤条件直接传递到doris： + +``` +env { + parallelism = 2 + job.mode = "BATCH" +} +source{ + Doris { + fenodes = "doris_e2e:8030" + username = root + password = "" + database = "e2e_source" + table = "doris_e2e_table" + doris.filter.query = "F_ID > 2" + } +} + +transform { + # If you would like to get more information about how to configure seatunnel and see full list of transform plugins, + # please go to https://seatunnel.apache.org/docs/transform/sql +} + +sink { + Console {} +} +``` +### 多表 +``` +env{ + parallelism = 1 + job.mode = "BATCH" +} + +source{ + Doris { + fenodes = "xxxx:8030" + username = root + password = "" + table_list = [ + { + database = "st_source_0" + table = "doris_table_0" + doris.read.field = "F_ID,F_INT,F_BIGINT,F_TINYINT" + doris.filter.query = "F_ID >= 50" + }, + { + database = "st_source_1" + table = "doris_table_1" + } + ] + } +} + +transform {} + +sink{ + Doris { + fenodes = "xxxx:8030" + schema_save_mode = "RECREATE_SCHEMA" + username = root + password = "" + database = "st_sink" + table = "${table_name}" + sink.enable-2pc = "true" + sink.label-prefix = "test_json" + doris.config = { + format="json" + read_json_by_line="true" + } + } +} +``` diff --git a/docs/zh/connector-v2/source/HdfsFile.md b/docs/zh/connector-v2/source/HdfsFile.md index 0f983a80bcf..4de3014f5c0 100644 --- a/docs/zh/connector-v2/source/HdfsFile.md +++ b/docs/zh/connector-v2/source/HdfsFile.md @@ -39,7 +39,7 @@ ## 源选项 -| 名称 | 类型 | 是否必须 | 默认值 | 描述 | +| 名称 | 类型 | 是否必须 | 默认值 | 描述 | |---------------------------|---------|------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | path | string | 是 | - | 源文件路径。 | | file_format_type | string | 是 | - | 我们支持以下文件类型：`text` `json` `csv` `orc` `parquet` `excel`。请注意，最终文件名将以文件格式的后缀结束，文本文件的后缀是 `txt`。 | @@ -55,6 +55,8 @@ | kerberos_principal | string | 否 | - | kerberos 的 principal。 | | kerberos_keytab_path | string | 否 | - | kerberos 的 keytab 路径。 | | skip_header_row_number | long | 否 | 0 | 跳过前几行，但仅适用于 txt 和 csv。例如，设置如下：`skip_header_row_number = 2`。然后 Seatunnel 将跳过源文件中的前两行。 | +| file_filter_pattern | string | 否 | - | 过滤模式，用于过滤文件。 | +| null_format | string | 否 | - | 定义哪些字符串可以表示为 null，但仅适用于 txt 和 csv. 例如: `\N` | | schema | config | 否 | - | 上游数据的模式字段。 | | sheet_name | string | 否 | - | 读取工作簿的表格，仅在文件格式为 excel 时使用。 | | compress_codec | string | 否 | none | 文件的压缩编解码器。 | @@ -64,6 +66,60 @@ **delimiter** 参数在版本 2.3.5 后将被弃用，请改用 **field_delimiter**。 +### file_filter_pattern [string] + +过滤模式，用于过滤文件。 + +这个过滤规则遵循正则表达式. 关于详情，请参考 https://en.wikipedia.org/wiki/Regular_expression 学习 + +这里是一些例子. + +文件清单: +``` +/data/seatunnel/20241001/report.txt +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +/data/seatunnel/20241012/logo.png +``` +匹配规则: + +**例子 1**: *匹配所有txt为后缀名的文件*，匹配正则为: +``` +/data/seatunnel/20241001/.*\.txt +``` +匹配的结果是: +``` +/data/seatunnel/20241001/report.txt +``` +**例子 2**: *匹配所有文件名以abc开头的文件*，匹配正则为: +``` +/data/seatunnel/20241002/abc.* +``` +匹配的结果是: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +``` +**例子 3**: *匹配所有文件名以abc开头，并且文件第四个字母是 h 或者 g 的文件*, 匹配正则为: +``` +/data/seatunnel/20241007/abc[h,g].* +``` +匹配的结果是: +``` +/data/seatunnel/20241007/abch202410.csv +``` +**例子 4**: *匹配所有文件夹第三级以 202410 开头并且文件后缀名是.csv的文件*, 匹配正则为: +``` +/data/seatunnel/202410\d*/.*\.csv +``` +匹配的结果是: +``` +/data/seatunnel/20241007/abch202410.csv +/data/seatunnel/20241002/abcg202410.csv +/data/seatunnel/20241005/old_data.csv +``` + ### compress_codec [string] 文件的压缩编解码器及支持的详细信息如下所示： @@ -125,3 +181,25 @@ sink { } ``` +### Filter File + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + HdfsFile { + path = "/apps/hive/demo/student" + file_format_type = "json" + fs.defaultFS = "hdfs://namenode001" + file_filter_pattern = "abc[DX]*.*" + } +} + +sink { + Console { + } +} +``` \ No newline at end of file diff --git a/docs/zh/connector-v2/source/Hive.md b/docs/zh/connector-v2/source/Hive.md new file mode 100644 index 00000000000..00a322cc408 --- /dev/null +++ b/docs/zh/connector-v2/source/Hive.md @@ -0,0 +1,368 @@ +# Hive + +> Hive 源连接器 + +## 描述 + +从 Hive 读取数据。 + +:::提示 + +为了使用此连接器，您必须确保您的 Spark/Flink 集群已经集成了 Hive。测试过的 Hive 版本是 2.3.9 和 3.1.3。 + +如果您使用 SeaTunnel 引擎，您需要将 `seatunnel-hadoop3-3.1.4-uber.jar`、`hive-exec-3.1.3.jar` 和 `libfb303-0.9.3.jar` 放在 `$SEATUNNEL_HOME/lib/` 目录中。 +::: + +## 关键特性 + +- [x] [批处理](../../concept/connector-v2-features.md) +- [ ] [流处理](../../concept/connector-v2-features.md) +- [x] [精确一次](../../concept/connector-v2-features.md) + +在 `pollNext` 调用中读取分片中的所有数据。读取的分片将保存在快照中。 + +- [x] [schema 投影](../../concept/connector-v2-features.md) +- [x] [并行度](../../concept/connector-v2-features.md) +- [ ] [支持用户定义的分片](../../concept/connector-v2-features.md) +- [x] 文件格式 + - [x] 文本 + - [x] CSV + - [x] Parquet + - [x] ORC + - [x] JSON + +## 选项 + +| 名称 | 类型 | 必需 | 默认值 | +|-----------------------|--------|------|---------| +| table_name | string | 是 | - | +| metastore_uri | string | 是 | - | +| krb5_path | string | 否 | /etc/krb5.conf | +| kerberos_principal | string | 否 | - | +| kerberos_keytab_path | string | 否 | - | +| hdfs_site_path | string | 否 | - | +| hive_site_path | string | 否 | - | +| hive.hadoop.conf | Map | 否 | - | +| hive.hadoop.conf-path | string | 否 | - | +| read_partitions | list | 否 | - | +| read_columns | list | 否 | - | +| compress_codec | string | 否 | none | +| common-options | | 否 | - | + +### table_name [string] + +目标 Hive 表名，例如：`db1.table1` + +### metastore_uri [string] + +Hive 元存储 URI + +### hdfs_site_path [string] + +`hdfs-site.xml` 的路径，用于加载 Namenode 的高可用配置 + +### hive.hadoop.conf [map] + +Hadoop 配置中的属性（`core-site.xml`、`hdfs-site.xml`、`hive-site.xml`） + +### hive.hadoop.conf-path [string] + +指定加载 `core-site.xml`、`hdfs-site.xml`、`hive-site.xml` 文件的路径 + +### read_partitions [list] + +用户希望从 Hive 表中读取的目标分区，如果用户未设置此参数，将读取 Hive 表中的所有数据。 + +**提示：分区列表中的每个分区应具有相同的目录层级。例如，一个 Hive 表有两个分区：`par1` 和 `par2`，如果用户设置如下：** +**`read_partitions = [par1=xxx, par1=yyy/par2=zzz]`，这是不合法的** + +### krb5_path [string] + +`krb5.conf` 的路径，用于 Kerberos 认证 + +### kerberos_principal [string] + +Kerberos 认证的主体 + +### kerberos_keytab_path [string] + +Kerberos 认证的 keytab 文件路径 + +### read_columns [list] + +数据源的读取列列表，用户可以使用它来实现字段投影。 + +### compress_codec [string] + +文件的压缩编解码器，支持的详细信息如下所示： + +- txt: `lzo` `none` +- json: `lzo` `none` +- csv: `lzo` `none` +- orc/parquet: + 自动识别压缩类型，无需额外设置。 + +### 通用选项 + +源插件的通用参数，请参阅 [Source Common Options](../source-common-options.md) 了解详细信息。 + +## 示例 + +### 示例 1：单表 + +```bash + Hive { + table_name = "default.seatunnel_orc" + metastore_uri = "thrift://namenode001:9083" + } +``` + +### 示例 2：多表 +> 注意：Hive 是结构化数据源，应使用 `table_list`，`tables_configs` 将在未来移除。 + +```bash + Hive { + table_list = [ + { + table_name = "default.seatunnel_orc_1" + metastore_uri = "thrift://namenode001:9083" + }, + { + table_name = "default.seatunnel_orc_2" + metastore_uri = "thrift://namenode001:9083" + } + ] + } +``` + +```bash + Hive { + tables_configs = [ + { + table_name = "default.seatunnel_orc_1" + metastore_uri = "thrift://namenode001:9083" + }, + { + table_name = "default.seatunnel_orc_2" + metastore_uri = "thrift://namenode001:9083" + } + ] + } +``` + +### 示例 3：Kerberos + +```bash +source { + Hive { + table_name = "default.test_hive_sink_on_hdfs_with_kerberos" + metastore_uri = "thrift://metastore:9083" + hive.hadoop.conf-path = "/tmp/hadoop" + result_table_name = hive_source + hive_site_path = "/tmp/hive-site.xml" + kerberos_principal = "hive/metastore.seatunnel@EXAMPLE.COM" + kerberos_keytab_path = "/tmp/hive.keytab" + krb5_path = "/tmp/krb5.conf" + } +} +``` + +描述： + +- `hive_site_path`：`hive-site.xml` 文件的路径。 +- `kerberos_principal`：Kerberos 认证的主体。 +- `kerberos_keytab_path`：Kerberos 认证的 keytab 文件路径。 +- `krb5_path`：用于 Kerberos 认证的 `krb5.conf` 文件路径。 + +运行案例： + +```bash +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + Hive { + table_name = "default.test_hive_sink_on_hdfs_with_kerberos" + metastore_uri = "thrift://metastore:9083" + hive.hadoop.conf-path = "/tmp/hadoop" + result_table_name = hive_source + hive_site_path = "/tmp/hive-site.xml" + kerberos_principal = "hive/metastore.seatunnel@EXAMPLE.COM" + kerberos_keytab_path = "/tmp/hive.keytab" + krb5_path = "/tmp/krb5.conf" + } +} + +sink { + Assert { + source_table_name = hive_source + rules { + row_rules = [ + { + rule_type = MAX_ROW + rule_value = 3 + } + ], + field_rules = [ + { + field_name = pk_id + field_type = bigint + field_value = [ + { + rule_type = NOT_NULL + } + ] + }, + { + field_name = name + field_type = string + field_value = [ + { + rule_type = NOT_NULL + } + ] + }, + { + field_name = score + field_type = int + field_value = [ + { + rule_type = NOT_NULL + } + ] + } + ] + } + } +} +``` + +## Hive on s3 + +### 步骤 1 + +为 EMR 的 Hive 创建 lib 目录。 + +```shell +mkdir -p ${SEATUNNEL_HOME}/plugins/Hive/lib +``` + +### 步骤 2 + +从 Maven 中心获取 jar 文件到 lib。 + +```shell +cd ${SEATUNNEL_HOME}/plugins/Hive/lib +wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.6.5/hadoop-aws-2.6.5.jar +wget https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.9/hive-exec-2.3.9.jar +``` + +### 步骤 3 + +从您的 EMR 环境中复制 jar 文件到 lib 目录。 + +```shell +cp /usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.60.0.jar ${SEATUNNEL_HOME}/plugins/Hive/lib +cp /usr/share/aws/emr/hadoop-state-pusher/lib/hadoop-common-3.3.6-amzn-1.jar ${SEATUNNEL_HOME}/plugins/Hive/lib +cp /usr/share/aws/emr/hadoop-state-pusher/lib/javax.inject-1.jar ${SEATUNNEL_HOME}/plugins/Hive/lib +cp /usr/share/aws/emr/hadoop-state-pusher/lib/aopalliance-1.0.jar ${SEATUNNEL_HOME}/plugins/Hive/lib +``` + +### 步骤 4 + +运行案例。 + +```shell +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + Hive { + table_name = "test_hive.test_hive_sink_on_s3" + metastore_uri = "thrift://ip-192-168-0-202.cn-north-1.compute.internal:9083" + hive.hadoop.conf-path = "/home/ec2-user/hadoop-conf" + hive.hadoop.conf = { + bucket="s3://ws-package" + fs.s3a.aws.credentials.provider="com.amazonaws.auth.InstanceProfileCredentialsProvider" + } + read_columns = ["pk_id", "name", "score"] + } +} + +sink { + Hive { + table_name = "test_hive.test_hive_sink_on_s3_sink" + metastore_uri = "thrift://ip-192-168-0-202.cn-north-1.compute.internal:9083" + hive.hadoop.conf-path = "/home/ec2-user/hadoop-conf" + hive.hadoop.conf = { + bucket="s3://ws-package" + fs.s3a.aws.credentials.provider="com.amazonaws.auth.InstanceProfileCredentialsProvider" + } + } +} +``` + +## Hive on oss + +### 步骤 1 + +为 EMR 的 Hive 创建 lib 目录。 + +```shell +mkdir -p ${SEATUNNEL_HOME}/plugins/Hive/lib +``` + +### 步骤 2 + +从 Maven 中心获取 jar 文件到 lib。 + +```shell +cd ${SEATUNNEL_HOME}/plugins/Hive/lib +wget https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.9/hive-exec-2.3.9.jar +``` + +### 步骤 3 + +从您的 EMR 环境中复制 jar 文件到 lib 目录并删除冲突的 jar。 + +```shell +cp -r /opt/apps/JINDOSDK/jindosdk-current/lib/jindo-*.jar ${SEATUNNEL_HOME}/plugins/Hive/lib +rm -f ${SEATUNNEL_HOME}/lib/hadoop-aliyun-*.jar +``` + +### 步骤 4 + +运行案例。 + +```shell +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + Hive { + table_name = "test_hive.test_hive_sink_on_oss" + metastore_uri = "thrift://master-1-1.c-1009b01725b501f2.cn-wulanchabu.emr.aliyuncs.com:9083" + hive.hadoop.conf-path = "/tmp/hadoop" + hive.hadoop.conf = { + bucket="oss://emr-osshdfs.cn-wulanchabu.oss-dls.aliyuncs.com" + } + } +} + +sink { + Hive { + table_name = "test_hive.test_hive_sink_on_oss_sink" + metastore_uri = "thrift://master-1-1.c-1009b01725b501f2.cn-wulanchabu.emr.aliyuncs.com:9083" + hive.hadoop.conf-path = "/tmp/hadoop" + hive.hadoop.conf = { + bucket="oss://emr-osshdfs.cn-wulanchabu.oss-dls.aliyuncs.com" + } + } +} +``` diff --git a/docs/zh/connector-v2/source/Kafka.md b/docs/zh/connector-v2/source/Kafka.md index 44e27215564..04820cc7c13 100644 --- a/docs/zh/connector-v2/source/Kafka.md +++ b/docs/zh/connector-v2/source/Kafka.md @@ -181,6 +181,65 @@ source { > 根据不同的 Kafka 主题和格式解析数据，并基于 ID 执行 upsert 操作。 +> 注意: Kafka是一个非结构化数据源，应该使用`tables_configs`，将来会删除`table_list` + +```hocon + +env { + execution.parallelism = 1 + job.mode = "BATCH" +} + +source { + Kafka { + bootstrap.servers = "kafka_e2e:9092" + tables_configs = [ + { + topic = "^test-ogg-sou.*" + pattern = "true" + consumer.group = "ogg_multi_group" + start_mode = earliest + schema = { + fields { + id = "int" + name = "string" + description = "string" + weight = "string" + } + }, + format = ogg_json + }, + { + topic = "test-cdc_mds" + start_mode = earliest + schema = { + fields { + id = "int" + name = "string" + description = "string" + weight = "string" + } + }, + format = canal_json + } + ] + } +} + +sink { + Jdbc { + driver = org.postgresql.Driver + url = "jdbc:postgresql://postgresql:5432/test?loggerLevel=OFF" + user = test + password = test + generate_sink_sql = true + database = test + table = public.sink + primary_keys = ["id"] + } +} +``` + ```hocon env { execution.parallelism = 1 @@ -283,7 +342,7 @@ source { """ bootstrap.servers = "kafkaCluster:9092" start_mode = "earliest" - result_table_name = "kafka_table" + plugin_output = "kafka_table" } } ``` diff --git a/docs/zh/connector-v2/source/Opengauss-CDC.md b/docs/zh/connector-v2/source/Opengauss-CDC.md index 83da40b363e..b175f611ecb 100644 --- a/docs/zh/connector-v2/source/Opengauss-CDC.md +++ b/docs/zh/connector-v2/source/Opengauss-CDC.md @@ -63,31 +63,31 @@ select 'ALTER TABLE ' || schemaname || '.' || tablename || ' REPLICA IDENTITY FU ## 源端可选项 -| Name | Type | Required | Default | Description | -|------------------------------------------------|------|----------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| base-url | 字符串 | 是 | - | JDBC连接的URL. 参考: `jdbc:postgresql://localhost:5432/postgres_cdc?loggerLevel=OFF`. | -| username | 字符串 | 是 | - | 连接数据库的用户名 | -| password | 字符串 | 是 | - | 连接数据库的密码 | -| database-names | 列表 | 否 | - | 监控的数据库名称 | -| table-names | 列表 | 是 | - | 监控的数据表名称. 表名需要包含数据库名称, 例如: `database_name.table_name` | -| table-names-config | 列表 | 否 | - | 表配置的列表集合. 例如: [{"table": "db1.schema1.table1","primaryKeys":["key1"]}] | -| startup.mode | 枚举 | 否 | INITIAL | Opengauss CDC消费者的可选启动模式, 有效的枚举是`initial`, `earliest`, `latest` and `specific`.
`initial`: 启动时同步历史数据，然后同步增量数据
`earliest`: 从可能的最早偏移量启动
`latest`: 从最近的偏移量启动
`specific`: 从用户指定的偏移量启动 | -| snapshot.split.size | 整型 | 否 | 8096 | 表快照的分割大小（行数），在读取表的快照时，捕获的表被分割成多个split | -| snapshot.fetch.size | 整型 | 否 | 1024 | 读取表快照时，每次轮询的最大读取大小 | -| slot.name | 字符串 | 否 | - | Opengauss逻辑解码插槽的名称，该插槽是为特定数据库/模式的特定插件的流式更改而创建的。服务器使用此插槽将事件流传输到正在配置的连接器。默认值为seatunnel | -| decoding.plugin.name | 字符串 | 否 | pgoutput | 安装在服务器上的Postgres逻辑解码插件的名称，支持的值是decoderbufs、wal2json、wal2json_rds、wal2json_streaming、wal2json_rds_streaming和pgoutput | -| server-time-zone | 字符串 | 否 | UTC | 数据库服务器中的会话时区。如果没有设置，则使用ZoneId.systemDefault()来确定服务器的时区 | -| connect.timeout.ms | 时间间隔 | 否 | 30000 | 在尝试连接数据库服务器之后，连接器在超时之前应该等待的最大时间 | -| connect.max-retries | 整型 | 否 | 3 | 连接器在建立数据库服务器连接时应该重试的最大次数 | -| connection.pool.size | 整型 | 否 | 20 | jdbc连接池的大小 | -| chunk-key.even-distribution.factor.upper-bound | 双浮点型 | 否 | 100 | chunk的key分布因子的上界。该因子用于确定表数据是否均匀分布。如果分布因子被计算为小于或等于这个上界(即(MAX(id) - MIN(id) + 1) /行数)，表的所有chunk将被优化以达到均匀分布。否则，如果分布因子更大，则认为表分布不均匀，如果估计的分片数量超过`sample-sharding.threshold`指定的值，则将使用基于采样的分片策略。默认值为100.0。 | -| chunk-key.even-distribution.factor.lower-bound | 双浮点型 | 否 | 0.05 | chunk的key分布因子的下界。该因子用于确定表数据是否均匀分布。如果分布因子的计算结果大于或等于这个下界(即(MAX(id) - MIN(id) + 1) /行数)，那么表的所有块将被优化以达到均匀分布。否则，如果分布因子较小，则认为表分布不均匀，如果估计的分片数量超过`sample-sharding.threshold`指定的值，则使用基于采样的分片策略。缺省值为0.05。 | -| sample-sharding.threshold | 整型 | 否 | 1000 | 此配置指定了用于触发采样分片策略的估计分片数的阈值。当分布因子超出了由`chunk-key.even-distribution.factor.upper-bound `和`chunk-key.even-distribution.factor.lower-bound`，并且估计的分片计数(以近似的行数/块大小计算)超过此阈值，则将使用样本分片策略。这有助于更有效地处理大型数据集。默认值为1000个分片。 | -| inverse-sampling.rate | 整型 | 否 | 1000 | 采样分片策略中使用的采样率的倒数。例如，如果该值设置为1000，则意味着在采样过程中应用了1/1000的采样率。该选项提供了控制采样粒度的灵活性，从而影响最终的分片数量。当处理非常大的数据集时，它特别有用，其中首选较低的采样率。缺省值为1000。 | -| exactly_once | 布尔 | 否 | false | 启用exactly once语义 | -| format | 枚举 | 否 | DEFAULT | Opengauss CDC可选的输出格式, 有效的枚举是`DEFAULT`, `COMPATIBLE_DEBEZIUM_JSON`. | -| debezium | 配置 | 否 | - | 将 [Debezium的属性](https://github.com/debezium/debezium/blob/v1.9.8.Final/documentation/modules/ROOT/pages/connectors/postgresql.adoc#connector-configuration-properties) 传递到Debezium嵌入式引擎，该引擎用于捕获来自Opengauss服务的数据更改 | -| common-options | | 否 | - | 源码插件通用参数, 请参考[Source Common Options](../source-common-options.md)获取详情 | +| Name | Type | Required | Default | Description | +|------------------------------------------------|------|----------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| base-url | 字符串 | 是 | - | JDBC连接的URL. 参考: `jdbc:postgresql://localhost:5432/postgres_cdc?loggerLevel=OFF`. | +| username | 字符串 | 是 | - | 连接数据库的用户名 | +| password | 字符串 | 是 | - | 连接数据库的密码 | +| database-names | 列表 | 否 | - | 监控的数据库名称 | +| table-names | 列表 | 是 | - | 监控的数据表名称. 表名需要包含数据库名称, 例如: `database_name.table_name` | +| table-names-config | 列表 | 否 | - | 表配置的列表集合. 例如: [{"table": "db1.schema1.table1","primaryKeys":["key1"]}] | +| startup.mode | 枚举 | 否 | INITIAL | Opengauss CDC消费者的可选启动模式, 有效的枚举是`initial`, `earliest`, `latest`.
`initial`: 启动时同步历史数据，然后同步增量数据
`earliest`: 从可能的最早偏移量启动
`latest`: 从最近的偏移量启动 | +| snapshot.split.size | 整型 | 否 | 8096 | 表快照的分割大小（行数），在读取表的快照时，捕获的表被分割成多个split | +| snapshot.fetch.size | 整型 | 否 | 1024 | 读取表快照时，每次轮询的最大读取大小 | +| slot.name | 字符串 | 否 | - | Opengauss逻辑解码插槽的名称，该插槽是为特定数据库/模式的特定插件的流式更改而创建的。服务器使用此插槽将事件流传输到正在配置的连接器。默认值为seatunnel | +| decoding.plugin.name | 字符串 | 否 | pgoutput | 安装在服务器上的Postgres逻辑解码插件的名称，支持的值是decoderbufs、wal2json、wal2json_rds、wal2json_streaming、wal2json_rds_streaming和pgoutput | +| server-time-zone | 字符串 | 否 | UTC | 数据库服务器中的会话时区。如果没有设置，则使用ZoneId.systemDefault()来确定服务器的时区 | +| connect.timeout.ms | 时间间隔 | 否 | 30000 | 在尝试连接数据库服务器之后，连接器在超时之前应该等待的最大时间 | +| connect.max-retries | 整型 | 否 | 3 | 连接器在建立数据库服务器连接时应该重试的最大次数 | +| connection.pool.size | 整型 | 否 | 20 | jdbc连接池的大小 | +| chunk-key.even-distribution.factor.upper-bound | 双浮点型 | 否 | 100 | chunk的key分布因子的上界。该因子用于确定表数据是否均匀分布。如果分布因子被计算为小于或等于这个上界(即(MAX(id) - MIN(id) + 1) /行数)，表的所有chunk将被优化以达到均匀分布。否则，如果分布因子更大，则认为表分布不均匀，如果估计的分片数量超过`sample-sharding.threshold`指定的值，则将使用基于采样的分片策略。默认值为100.0。 | +| chunk-key.even-distribution.factor.lower-bound | 双浮点型 | 否 | 0.05 | chunk的key分布因子的下界。该因子用于确定表数据是否均匀分布。如果分布因子的计算结果大于或等于这个下界(即(MAX(id) - MIN(id) + 1) /行数)，那么表的所有块将被优化以达到均匀分布。否则，如果分布因子较小，则认为表分布不均匀，如果估计的分片数量超过`sample-sharding.threshold`指定的值，则使用基于采样的分片策略。缺省值为0.05。 | +| sample-sharding.threshold | 整型 | 否 | 1000 | 此配置指定了用于触发采样分片策略的估计分片数的阈值。当分布因子超出了由`chunk-key.even-distribution.factor.upper-bound `和`chunk-key.even-distribution.factor.lower-bound`，并且估计的分片计数(以近似的行数/块大小计算)超过此阈值，则将使用样本分片策略。这有助于更有效地处理大型数据集。默认值为1000个分片。 | +| inverse-sampling.rate | 整型 | 否 | 1000 | 采样分片策略中使用的采样率的倒数。例如，如果该值设置为1000，则意味着在采样过程中应用了1/1000的采样率。该选项提供了控制采样粒度的灵活性，从而影响最终的分片数量。当处理非常大的数据集时，它特别有用，其中首选较低的采样率。缺省值为1000。 | +| exactly_once | 布尔 | 否 | false | 启用exactly once语义 | +| format | 枚举 | 否 | DEFAULT | Opengauss CDC可选的输出格式, 有效的枚举是`DEFAULT`, `COMPATIBLE_DEBEZIUM_JSON`. | +| debezium | 配置 | 否 | - | 将 [Debezium的属性](https://github.com/debezium/debezium/blob/v1.9.8.Final/documentation/modules/ROOT/pages/connectors/postgresql.adoc#connector-configuration-properties) 传递到Debezium嵌入式引擎，该引擎用于捕获来自Opengauss服务的数据更改 | +| common-options | | 否 | - | 源码插件通用参数, 请参考[Source Common Options](../source-common-options.md)获取详情 | ## 任务示例 @@ -108,7 +108,7 @@ env { source { Opengauss-CDC { - result_table_name = "customers_opengauss_cdc" + plugin_output = "customers_opengauss_cdc" username = "gaussdb" password = "openGauss@123" database-names = ["opengauss_cdc"] @@ -125,7 +125,7 @@ transform { sink { jdbc { - source_table_name = "customers_opengauss_cdc" + plugin_input = "customers_opengauss_cdc" url = "jdbc:postgresql://opengauss_cdc_e2e:5432/opengauss_cdc" driver = "org.postgresql.Driver" user = "dailai" @@ -148,7 +148,7 @@ sink { ``` source { Opengauss-CDC { - result_table_name = "customers_opengauss_cdc" + plugin_output = "customers_opengauss_cdc" username = "gaussdb" password = "openGauss@123" database-names = ["opengauss_cdc"] diff --git a/docs/zh/connector-v2/source/Prometheus.md b/docs/zh/connector-v2/source/Prometheus.md new file mode 100644 index 00000000000..1dca6b463cb --- /dev/null +++ b/docs/zh/connector-v2/source/Prometheus.md @@ -0,0 +1,152 @@ +# Prometheus + +> Prometheus 数据源连接器 + +## 描述 + +用于读取prometheus数据。 + +## 主要特性 + +- [x] [批处理](../../concept/connector-v2-features.md) +- [ ] [流处理](../../concept/connector-v2-features.md) +- [ ] [并行](../../concept/connector-v2-features.md) + +## 源选项 + +| 名称 | 类型 | 是否必填 | 默认值 | +|-----------------------------|---------|------|-----------------| +| url | String | Yes | - | +| query | String | Yes | - | +| query_type | String | Yes | Instant | +| content_field | String | Yes | $.data.result.* | +| schema.fields | Config | Yes | - | +| format | String | No | json | +| params | Map | Yes | - | +| poll_interval_millis | int | No | - | +| retry | int | No | - | +| retry_backoff_multiplier_ms | int | No | 100 | +| retry_backoff_max_ms | int | No | 10000 | +| enable_multi_lines | boolean | No | false | +| common-options | config | No | | + +### url [String] + +http 请求路径。 + +### query [String] + +Prometheus 表达式查询字符串 + +### query_type [String] + +Instant/Range + +1. Instant : 简单指标的即时查询。 +2. Range : 一段时间内指标数据。 + +https://prometheus.io/docs/prometheus/latest/querying/api/ + +### params [Map] + +http 请求参数 + +### poll_interval_millis [int] + +流模式下请求HTTP API间隔(毫秒) + +### retry [int] + +The max retry times if request http return to `IOException` + +### retry_backoff_multiplier_ms [int] + +请求http返回到' IOException '的最大重试次数 + +### retry_backoff_max_ms [int] + +http请求失败，最大重试回退时间(毫秒) + +### format [String] + +上游数据的格式，默认为json。 + +### schema [Config] + +按照如下填写一个固定值 + +```hocon + schema = { + fields { + metric = "map
+
+
+#### 请求参数
+
+#### 参数(在请求体中params字段中添加)
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |----------------------|----------|--------|-----------------------------------|
+> | jobId | optional | string | job id |
+
+当`jobId`为空时，返回所有节点的日志信息，否则返回指定`jobId`在所有节点的的日志列表。
+
+#### 响应
+
+返回请求节点的日志列表、内容
+
+#### 返回所有日志文件列表
+
+如果你想先查看日志列表，可以通过`GET`请求获取日志列表，`http://localhost:5801/hazelcast/rest/maps/logs?format=json`
+
+```json
+[
+ {
+ "node": "localhost:5801",
+ "logLink": "http://localhost:5801/hazelcast/rest/maps/logs/job-899485770241277953.log",
+ "logName": "job-899485770241277953.log"
+ },
+ {
+ "node": "localhost:5801",
+ "logLink": "http://localhost:5801/hazelcast/rest/maps/logs/job-899470314109468673.log",
+ "logName": "job-899470314109468673.log"
+ }
+]
+```
+
+当前支持的格式有`json`和`html`，默认为`html`。
+
+#### 例子
+
+获取所有节点jobId为`733584788375666689`的日志信息：`http://localhost:5801/hazelcast/rest/maps/logs/733584788375666689`
+获取所有节点日志列表：`http://localhost:5801/hazelcast/rest/maps/logs`
+获取所有节点日志列表以JSON格式返回：`http://localhost:5801/hazelcast/rest/maps/logs?format=json`
+获取日志文件内容：`http://localhost:5801/hazelcast/rest/maps/logs/job-898380162133917698.log``
+
+
+
+
+
+### 获取单节点日志内容
+
+GET /hazelcast/rest/maps/logs/:jobId (返回日志列表。)
+
+#### 请求参数
+
+#### 参数(在请求体中params字段中添加)
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |----------------------|----------|--------|-----------------------------------|
+> | jobId | optional | string | job id |
+
+当`jobId`为空时，返回所有节点的日志信息，否则返回指定`jobId`在所有节点的的日志列表。
+
+#### 响应
+
+返回请求节点的日志列表、内容
+
+#### 返回所有日志文件列表
+
+如果你想先查看日志列表，可以通过`GET`请求获取日志列表，`http://localhost:5801/hazelcast/rest/maps/logs?format=json`
+
+```json
+[
+ {
+ "node": "localhost:5801",
+ "logLink": "http://localhost:5801/hazelcast/rest/maps/logs/job-899485770241277953.log",
+ "logName": "job-899485770241277953.log"
+ },
+ {
+ "node": "localhost:5801",
+ "logLink": "http://localhost:5801/hazelcast/rest/maps/logs/job-899470314109468673.log",
+ "logName": "job-899470314109468673.log"
+ }
+]
+```
+
+当前支持的格式有`json`和`html`，默认为`html`。
+
+#### 例子
+
+获取所有节点jobId为`733584788375666689`的日志信息：`http://localhost:5801/hazelcast/rest/maps/logs/733584788375666689`
+获取所有节点日志列表：`http://localhost:5801/hazelcast/rest/maps/logs`
+获取所有节点日志列表以JSON格式返回：`http://localhost:5801/hazelcast/rest/maps/logs?format=json`
+获取日志文件内容：`http://localhost:5801/hazelcast/rest/maps/logs/job-898380162133917698.log``
+
+
+
+
+
+#### 响应
+
+返回请求节点的日志列表
+
+#### 例子
+
+获取当前节点的日志列表：`http://localhost:5801/hazelcast/rest/maps/log`
+获取日志文件内容：`http://localhost:5801/hazelcast/rest/maps/log/job-898380162133917698.log`
+
+
diff --git a/docs/zh/seatunnel-engine/rest-api-v2.md b/docs/zh/seatunnel-engine/rest-api-v2.md
new file mode 100644
index 00000000000..0e3b3e2657d
--- /dev/null
+++ b/docs/zh/seatunnel-engine/rest-api-v2.md
@@ -0,0 +1,852 @@
+---
+sidebar_position: 12
+---
+
+# RESTful API V2
+
+SeaTunnel有一个用于监控的API，可用于查询运行作业的状态和统计信息，以及最近完成的作业。监控API是RESTful风格的，它接受HTTP请求并使用JSON数据格式进行响应。
+
+## 概述
+
+v2版本的api使用jetty支持，与v1版本的接口规范相同 ,可以通过修改`seatunnel.yaml`中的配置项来指定端口和context-path，
+同时可以配置 `enable-dynamic-port` 开启动态端口(默认从 `port` 开始累加)，默认为关闭，
+如果`enable-dynamic-port`为`true`，我们将使用`port`和`port`+`port-range`范围内未使用的端口，默认范围是100。
+
+```yaml
+
+seatunnel:
+ engine:
+ http:
+ enable-http: true
+ port: 8080
+ enable-dynamic-port: false
+ port-range: 100
+```
+
+同时也可以配置context-path,配置如下：
+
+```yaml
+
+seatunnel:
+ engine:
+ http:
+ enable-http: true
+ port: 8080
+ context-path: /seatunnel
+```
+
+## API参考
+
+### 返回Zeta集群的概览
+
+GET /hazelcast/rest/maps/log (返回日志列表。)
+
+#### 响应
+
+返回请求节点的日志列表
+
+#### 例子
+
+获取当前节点的日志列表：`http://localhost:5801/hazelcast/rest/maps/log`
+获取日志文件内容：`http://localhost:5801/hazelcast/rest/maps/log/job-898380162133917698.log`
+
+
+
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |--------|------|------|--------------------------|
+> | tag键值对 | 否 | 字符串 | 一组标签值, 通过该标签值过滤满足条件的节点信息 |
+
+#### 响应
+
+```json
+{
+ "projectVersion":"2.3.5-SNAPSHOT",
+ "gitCommitAbbrev":"DeadD0d0",
+ "totalSlot":"0",
+ "unassignedSlot":"0",
+ "works":"1",
+ "runningJobs":"0",
+ "finishedJobs":"0",
+ "failedJobs":"0",
+ "cancelledJobs":"0"
+}
+```
+
+**注意:**
+- 当你使用`dynamic-slot`时, 返回结果中的`totalSlot`和`unassignedSlot`将始终为0. 设置为固定的slot值后, 将正确返回集群中总共的slot数量以及未分配的slot数量.
+- 当添加标签过滤后, `works`, `totalSlot`, `unassignedSlot`将返回满足条件的节点的相关指标. 注意`runningJobs`等job相关指标为集群级别结果, 无法根据标签进行过滤.
+
+
+
+------------------------------------------------------------------------------------------
+
+### 返回所有作业及其当前状态的概览
+
+GET /overview?tag1=value1&tag2=value2 (Returns an overview over the Zeta engine cluster.)
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |--------|------|------|--------------------------|
+> | tag键值对 | 否 | 字符串 | 一组标签值, 通过该标签值过滤满足条件的节点信息 |
+
+#### 响应
+
+```json
+{
+ "projectVersion":"2.3.5-SNAPSHOT",
+ "gitCommitAbbrev":"DeadD0d0",
+ "totalSlot":"0",
+ "unassignedSlot":"0",
+ "works":"1",
+ "runningJobs":"0",
+ "finishedJobs":"0",
+ "failedJobs":"0",
+ "cancelledJobs":"0"
+}
+```
+
+**注意:**
+- 当你使用`dynamic-slot`时, 返回结果中的`totalSlot`和`unassignedSlot`将始终为0. 设置为固定的slot值后, 将正确返回集群中总共的slot数量以及未分配的slot数量.
+- 当添加标签过滤后, `works`, `totalSlot`, `unassignedSlot`将返回满足条件的节点的相关指标. 注意`runningJobs`等job相关指标为集群级别结果, 无法根据标签进行过滤.
+
+
+
+
+#### 参数
+
+#### 响应
+
+```json
+[
+ {
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false,
+ "metrics": {
+ "sourceReceivedCount": "",
+ "sinkWriteCount": ""
+ }
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 返回作业的详细信息
+
+GET /running-jobs (返回所有作业及其当前状态的概览。)
+
+#### 参数
+
+#### 响应
+
+```json
+[
+ {
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false,
+ "metrics": {
+ "sourceReceivedCount": "",
+ "sinkWriteCount": ""
+ }
+ }
+]
+```
+
+
+
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |-------|------|------|--------|
+> | jobId | 是 | long | job id |
+
+#### 响应
+
+```json
+{
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": {
+ "SourceReceivedCount": "",
+ "SourceReceivedQPS": "",
+ "SourceReceivedBytes": "",
+ "SourceReceivedBytesPerSeconds": "",
+ "SinkWriteCount": "",
+ "SinkWriteQPS": "",
+ "SinkWriteBytes": "",
+ "SinkWriteBytesPerSeconds": "",
+ "TableSourceReceivedCount": {},
+ "TableSourceReceivedBytes": {},
+ "TableSourceReceivedBytesPerSeconds": {},
+ "TableSourceReceivedQPS": {},
+ "TableSinkWriteCount": {},
+ "TableSinkWriteQPS": {},
+ "TableSinkWriteBytes": {},
+ "TableSinkWriteBytesPerSeconds": {}
+ },
+ "finishedTime": "",
+ "errorMsg": null,
+ "envOptions": {
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false
+}
+```
+
+`jobId`, `jobName`, `jobStatus`, `createTime`, `jobDag`, `metrics` 字段总会返回.
+`envOptions`, `pluginJarsUrls`, `isStartWithSavePoint` 字段在Job在RUNNING状态时会返回
+`finishedTime`, `errorMsg` 字段在Job结束时会返回，结束状态为不为RUNNING，可能为FINISHED，可能为CANCEL
+
+当我们查询不到这个Job时，返回结果为：
+
+```json
+{
+ "jobId" : ""
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 返回作业的详细信息
+
+此API已经弃用，请使用/job-info/:jobId替代。
+
+GET /job-info/:jobId (返回作业的详细信息。)
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |-------|------|------|--------|
+> | jobId | 是 | long | job id |
+
+#### 响应
+
+```json
+{
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": {
+ "SourceReceivedCount": "",
+ "SourceReceivedQPS": "",
+ "SourceReceivedBytes": "",
+ "SourceReceivedBytesPerSeconds": "",
+ "SinkWriteCount": "",
+ "SinkWriteQPS": "",
+ "SinkWriteBytes": "",
+ "SinkWriteBytesPerSeconds": "",
+ "TableSourceReceivedCount": {},
+ "TableSourceReceivedBytes": {},
+ "TableSourceReceivedBytesPerSeconds": {},
+ "TableSourceReceivedQPS": {},
+ "TableSinkWriteCount": {},
+ "TableSinkWriteQPS": {},
+ "TableSinkWriteBytes": {},
+ "TableSinkWriteBytesPerSeconds": {}
+ },
+ "finishedTime": "",
+ "errorMsg": null,
+ "envOptions": {
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false
+}
+```
+
+`jobId`, `jobName`, `jobStatus`, `createTime`, `jobDag`, `metrics` 字段总会返回.
+`envOptions`, `pluginJarsUrls`, `isStartWithSavePoint` 字段在Job在RUNNING状态时会返回
+`finishedTime`, `errorMsg` 字段在Job结束时会返回，结束状态为不为RUNNING，可能为FINISHED，可能为CANCEL
+
+当我们查询不到这个Job时，返回结果为：
+
+```json
+{
+ "jobId" : ""
+}
+```
+
+
+
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |-------|------|------|--------|
+> | jobId | 是 | long | job id |
+
+#### 响应
+
+```json
+{
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": {
+ "sourceReceivedCount": "",
+ "sinkWriteCount": ""
+ },
+ "finishedTime": "",
+ "errorMsg": null,
+ "envOptions": {
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false
+}
+```
+
+`jobId`, `jobName`, `jobStatus`, `createTime`, `jobDag`, `metrics` 字段总会返回.
+`envOptions`, `pluginJarsUrls`, `isStartWithSavePoint` 字段在Job在RUNNING状态时会返回
+`finishedTime`, `errorMsg` 字段在Job结束时会返回，结束状态为不为RUNNING，可能为FINISHED，可能为CANCEL
+
+当我们查询不到这个Job时，返回结果为：
+
+```json
+{
+ "jobId" : ""
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 返回所有已完成的作业信息
+
+GET /running-job/:jobId (返回作业的详细信息。)
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |-------|------|------|--------|
+> | jobId | 是 | long | job id |
+
+#### 响应
+
+```json
+{
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "createTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": {
+ "sourceReceivedCount": "",
+ "sinkWriteCount": ""
+ },
+ "finishedTime": "",
+ "errorMsg": null,
+ "envOptions": {
+ },
+ "pluginJarsUrls": [
+ ],
+ "isStartWithSavePoint": false
+}
+```
+
+`jobId`, `jobName`, `jobStatus`, `createTime`, `jobDag`, `metrics` 字段总会返回.
+`envOptions`, `pluginJarsUrls`, `isStartWithSavePoint` 字段在Job在RUNNING状态时会返回
+`finishedTime`, `errorMsg` 字段在Job结束时会返回，结束状态为不为RUNNING，可能为FINISHED，可能为CANCEL
+
+当我们查询不到这个Job时，返回结果为：
+
+```json
+{
+ "jobId" : ""
+}
+```
+
+
+
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |-------|----------|--------|------------------------------------------------------------------|
+> | state | optional | string | finished job status. `FINISHED`,`CANCELED`,`FAILED`,`UNKNOWABLE` |
+
+#### 响应
+
+```json
+[
+ {
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "errorMsg": null,
+ "createTime": "",
+ "finishTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": ""
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 返回系统监控信息
+
+GET /finished-jobs/:state (返回所有已完成的作业信息。)
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |-------|----------|--------|------------------------------------------------------------------|
+> | state | optional | string | finished job status. `FINISHED`,`CANCELED`,`FAILED`,`UNKNOWABLE` |
+
+#### 响应
+
+```json
+[
+ {
+ "jobId": "",
+ "jobName": "",
+ "jobStatus": "",
+ "errorMsg": null,
+ "createTime": "",
+ "finishTime": "",
+ "jobDag": {
+ "jobId": "",
+ "envOptions": [],
+ "vertexInfoMap": [
+ {
+ "vertexId": 1,
+ "type": "",
+ "vertexName": "",
+ "tablePaths": [
+ ""
+ ]
+ }
+ ],
+ "pipelineEdges": {}
+ },
+ "metrics": ""
+ }
+]
+```
+
+
+
+
+#### 参数
+
+#### 响应
+
+```json
+[
+ {
+ "processors":"8",
+ "physical.memory.total":"16.0G",
+ "physical.memory.free":"16.3M",
+ "swap.space.total":"0",
+ "swap.space.free":"0",
+ "heap.memory.used":"135.7M",
+ "heap.memory.free":"440.8M",
+ "heap.memory.total":"576.5M",
+ "heap.memory.max":"3.6G",
+ "heap.memory.used/total":"23.54%",
+ "heap.memory.used/max":"3.73%",
+ "minor.gc.count":"6",
+ "minor.gc.time":"110ms",
+ "major.gc.count":"2",
+ "major.gc.time":"73ms",
+ "load.process":"24.78%",
+ "load.system":"60.00%",
+ "load.systemAverage":"2.07",
+ "thread.count":"117",
+ "thread.peakCount":"118",
+ "cluster.timeDiff":"0",
+ "event.q.size":"0",
+ "executor.q.async.size":"0",
+ "executor.q.client.size":"0",
+ "executor.q.client.query.size":"0",
+ "executor.q.client.blocking.size":"0",
+ "executor.q.query.size":"0",
+ "executor.q.scheduled.size":"0",
+ "executor.q.io.size":"0",
+ "executor.q.system.size":"0",
+ "executor.q.operations.size":"0",
+ "executor.q.priorityOperation.size":"0",
+ "operations.completed.count":"10",
+ "executor.q.mapLoad.size":"0",
+ "executor.q.mapLoadAllKeys.size":"0",
+ "executor.q.cluster.size":"0",
+ "executor.q.response.size":"0",
+ "operations.running.count":"0",
+ "operations.pending.invocations.percentage":"0.00%",
+ "operations.pending.invocations.count":"0",
+ "proxy.count":"8",
+ "clientEndpoint.count":"0",
+ "connection.active.count":"2",
+ "client.connection.count":"0",
+ "connection.count":"0"
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 提交作业
+
+GET /system-monitoring-information (返回系统监控信息。)
+
+#### 参数
+
+#### 响应
+
+```json
+[
+ {
+ "processors":"8",
+ "physical.memory.total":"16.0G",
+ "physical.memory.free":"16.3M",
+ "swap.space.total":"0",
+ "swap.space.free":"0",
+ "heap.memory.used":"135.7M",
+ "heap.memory.free":"440.8M",
+ "heap.memory.total":"576.5M",
+ "heap.memory.max":"3.6G",
+ "heap.memory.used/total":"23.54%",
+ "heap.memory.used/max":"3.73%",
+ "minor.gc.count":"6",
+ "minor.gc.time":"110ms",
+ "major.gc.count":"2",
+ "major.gc.time":"73ms",
+ "load.process":"24.78%",
+ "load.system":"60.00%",
+ "load.systemAverage":"2.07",
+ "thread.count":"117",
+ "thread.peakCount":"118",
+ "cluster.timeDiff":"0",
+ "event.q.size":"0",
+ "executor.q.async.size":"0",
+ "executor.q.client.size":"0",
+ "executor.q.client.query.size":"0",
+ "executor.q.client.blocking.size":"0",
+ "executor.q.query.size":"0",
+ "executor.q.scheduled.size":"0",
+ "executor.q.io.size":"0",
+ "executor.q.system.size":"0",
+ "executor.q.operations.size":"0",
+ "executor.q.priorityOperation.size":"0",
+ "operations.completed.count":"10",
+ "executor.q.mapLoad.size":"0",
+ "executor.q.mapLoadAllKeys.size":"0",
+ "executor.q.cluster.size":"0",
+ "executor.q.response.size":"0",
+ "operations.running.count":"0",
+ "operations.pending.invocations.percentage":"0.00%",
+ "operations.pending.invocations.count":"0",
+ "proxy.count":"8",
+ "clientEndpoint.count":"0",
+ "connection.active.count":"2",
+ "client.connection.count":"0",
+ "connection.count":"0"
+ }
+]
+```
+
+
+
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |----------------------|----------|-----------------------------------|-----------------------------------|
+> | jobId | optional | string | job id |
+> | jobName | optional | string | job name |
+> | isStartWithSavePoint | optional | string | if job is started with save point |
+> | format | optional | string | 配置风格,支持json和hocon,默认 json |
+
+#### 请求体
+
+你可以选择用json或者hocon的方式来传递请求体。
+Json请求示例：
+```json
+{
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 100,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+}
+```
+
+Hocon请求示例：
+```hocon
+env {
+ job.mode = "batch"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ row.num = 100
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ card = "int"
+ }
+ }
+ }
+}
+
+transform {
+}
+
+sink {
+ Console {
+ source_table_name = "fake"
+ }
+}
+
+```
+#### 响应
+
+```json
+{
+ "jobId": 733584788375666689,
+ "jobName": "rest_api_test"
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+
+### 批量提交作业
+
+POST /submit-job (如果作业提交成功，返回jobId和jobName。)
+
+#### 参数
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |----------------------|----------|-----------------------------------|-----------------------------------|
+> | jobId | optional | string | job id |
+> | jobName | optional | string | job name |
+> | isStartWithSavePoint | optional | string | if job is started with save point |
+> | format | optional | string | 配置风格,支持json和hocon,默认 json |
+
+#### 请求体
+
+你可以选择用json或者hocon的方式来传递请求体。
+Json请求示例：
+```json
+{
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 100,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+}
+```
+
+Hocon请求示例：
+```hocon
+env {
+ job.mode = "batch"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ row.num = 100
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ card = "int"
+ }
+ }
+ }
+}
+
+transform {
+}
+
+sink {
+ Console {
+ source_table_name = "fake"
+ }
+}
+
+```
+#### 响应
+
+```json
+{
+ "jobId": 733584788375666689,
+ "jobName": "rest_api_test"
+}
+```
+
+
+
+
+#### 参数(在请求体中params字段中添加)
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |----------------------|----------|--------|-----------------------------------|
+> | jobId | optional | string | job id |
+> | jobName | optional | string | job name |
+> | isStartWithSavePoint | optional | string | if job is started with save point |
+
+
+
+#### 请求体
+
+```json
+[
+ {
+ "params":{
+ "jobId":"123456",
+ "jobName":"SeaTunnel-01"
+ },
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 1000,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+ },
+ {
+ "params":{
+ "jobId":"1234567",
+ "jobName":"SeaTunnel-02"
+ },
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 1000,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+ }
+]
+```
+
+#### 响应
+
+```json
+[
+ {
+ "jobId": "123456",
+ "jobName": "SeaTunnel-01"
+ },{
+ "jobId": "1234567",
+ "jobName": "SeaTunnel-02"
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 停止作业
+
+POST /submit-jobs (如果作业提交成功，返回jobId和jobName。)
+
+#### 参数(在请求体中params字段中添加)
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |----------------------|----------|--------|-----------------------------------|
+> | jobId | optional | string | job id |
+> | jobName | optional | string | job name |
+> | isStartWithSavePoint | optional | string | if job is started with save point |
+
+
+
+#### 请求体
+
+```json
+[
+ {
+ "params":{
+ "jobId":"123456",
+ "jobName":"SeaTunnel-01"
+ },
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 1000,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+ },
+ {
+ "params":{
+ "jobId":"1234567",
+ "jobName":"SeaTunnel-02"
+ },
+ "env": {
+ "job.mode": "batch"
+ },
+ "source": [
+ {
+ "plugin_name": "FakeSource",
+ "plugin_output": "fake",
+ "row.num": 1000,
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int",
+ "card": "int"
+ }
+ }
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Console",
+ "plugin_input": ["fake"]
+ }
+ ]
+ }
+]
+```
+
+#### 响应
+
+```json
+[
+ {
+ "jobId": "123456",
+ "jobName": "SeaTunnel-01"
+ },{
+ "jobId": "1234567",
+ "jobName": "SeaTunnel-02"
+ }
+]
+```
+
+
+
+
+#### 请求体
+
+```json
+{
+ "jobId": 733584788375666689,
+ "isStopWithSavePoint": false # if job is stopped with save point
+}
+```
+
+#### 响应
+
+```json
+{
+"jobId": 733584788375666689
+}
+```
+
+
+
+
+------------------------------------------------------------------------------------------
+
+### 批量停止作业
+
+POST /stop-job (如果作业成功停止，返回jobId。)
+
+#### 请求体
+
+```json
+{
+ "jobId": 733584788375666689,
+ "isStopWithSavePoint": false # if job is stopped with save point
+}
+```
+
+#### 响应
+
+```json
+{
+"jobId": 733584788375666689
+}
+```
+
+
+
+
+#### 请求体
+
+```json
+[
+ {
+ "jobId": 881432421482889220,
+ "isStopWithSavePoint": false
+ },
+ {
+ "jobId": 881432456517910529,
+ "isStopWithSavePoint": false
+ }
+]
+```
+
+#### 响应
+
+```json
+[
+ {
+ "jobId": 881432421482889220
+ },
+ {
+ "jobId": 881432456517910529
+ }
+]
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 加密配置
+
+POST /stop-jobs (如果作业成功停止，返回jobId。)
+
+#### 请求体
+
+```json
+[
+ {
+ "jobId": 881432421482889220,
+ "isStopWithSavePoint": false
+ },
+ {
+ "jobId": 881432456517910529,
+ "isStopWithSavePoint": false
+ }
+]
+```
+
+#### 响应
+
+```json
+[
+ {
+ "jobId": 881432421482889220
+ },
+ {
+ "jobId": 881432456517910529
+ }
+]
+```
+
+
+
+有关自定义加密的更多信息，请参阅文档[配置-加密-解密](../connector-v2/Config-Encryption-Decryption.md).
+
+#### 请求体
+
+```json
+{
+ "env": {
+ "parallelism": 1,
+ "shade.identifier":"base64"
+ },
+ "source": [
+ {
+ "plugin_name": "MySQL-CDC",
+ "schema" : {
+ "fields": {
+ "name": "string",
+ "age": "int"
+ }
+ },
+ "plugin_output": "fake",
+ "parallelism": 1,
+ "hostname": "127.0.0.1",
+ "username": "seatunnel",
+ "password": "seatunnel_password",
+ "table-name": "inventory_vwyw0n"
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Clickhouse",
+ "host": "localhost:8123",
+ "database": "default",
+ "table": "fake_all",
+ "username": "seatunnel",
+ "password": "seatunnel_password"
+ }
+ ]
+}
+```
+
+#### 响应
+
+```json
+{
+ "env": {
+ "parallelism": 1,
+ "shade.identifier": "base64"
+ },
+ "source": [
+ {
+ "plugin_name": "MySQL-CDC",
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int"
+ }
+ },
+ "plugin_output": "fake",
+ "parallelism": 1,
+ "hostname": "127.0.0.1",
+ "username": "c2VhdHVubmVs",
+ "password": "c2VhdHVubmVsX3Bhc3N3b3Jk",
+ "table-name": "inventory_vwyw0n"
+ }
+ ],
+ "transform": [],
+ "sink": [
+ {
+ "plugin_name": "Clickhouse",
+ "host": "localhost:8123",
+ "database": "default",
+ "table": "fake_all",
+ "username": "c2VhdHVubmVs",
+ "password": "c2VhdHVubmVsX3Bhc3N3b3Jk"
+ }
+ ]
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 更新运行节点的tags
+
+POST /encrypt-config (如果配置加密成功，则返回加密后的配置。)
+有关自定义加密的更多信息，请参阅文档[配置-加密-解密](../connector-v2/Config-Encryption-Decryption.md).
+
+#### 请求体
+
+```json
+{
+ "env": {
+ "parallelism": 1,
+ "shade.identifier":"base64"
+ },
+ "source": [
+ {
+ "plugin_name": "MySQL-CDC",
+ "schema" : {
+ "fields": {
+ "name": "string",
+ "age": "int"
+ }
+ },
+ "plugin_output": "fake",
+ "parallelism": 1,
+ "hostname": "127.0.0.1",
+ "username": "seatunnel",
+ "password": "seatunnel_password",
+ "table-name": "inventory_vwyw0n"
+ }
+ ],
+ "transform": [
+ ],
+ "sink": [
+ {
+ "plugin_name": "Clickhouse",
+ "host": "localhost:8123",
+ "database": "default",
+ "table": "fake_all",
+ "username": "seatunnel",
+ "password": "seatunnel_password"
+ }
+ ]
+}
+```
+
+#### 响应
+
+```json
+{
+ "env": {
+ "parallelism": 1,
+ "shade.identifier": "base64"
+ },
+ "source": [
+ {
+ "plugin_name": "MySQL-CDC",
+ "schema": {
+ "fields": {
+ "name": "string",
+ "age": "int"
+ }
+ },
+ "plugin_output": "fake",
+ "parallelism": 1,
+ "hostname": "127.0.0.1",
+ "username": "c2VhdHVubmVs",
+ "password": "c2VhdHVubmVsX3Bhc3N3b3Jk",
+ "table-name": "inventory_vwyw0n"
+ }
+ ],
+ "transform": [],
+ "sink": [
+ {
+ "plugin_name": "Clickhouse",
+ "host": "localhost:8123",
+ "database": "default",
+ "table": "fake_all",
+ "username": "c2VhdHVubmVs",
+ "password": "c2VhdHVubmVsX3Bhc3N3b3Jk"
+ }
+ ]
+}
+```
+
+
+
+
+
+#### 更新节点tags
+##### 请求体
+如果请求参数是`Map`对象，表示要更新当前节点的tags
+```json
+{
+ "tag1": "dev_1",
+ "tag2": "dev_2"
+}
+```
+##### 响应
+
+```json
+{
+ "status": "success",
+ "message": "update node tags done."
+}
+```
+#### 移除节点tags
+##### 请求体
+如果参数为空`Map`对象，表示要清除当前节点的tags
+```json
+{}
+```
+##### 响应
+响应体将为：
+```json
+{
+ "status": "success",
+ "message": "update node tags done."
+}
+```
+
+#### 请求参数异常
+- 如果请求参数为空
+
+##### 响应
+
+```json
+{
+ "status": "fail",
+ "message": "Request body is empty."
+}
+```
+- 如果参数不是`Map`对象
+##### 响应
+
+```json
+{
+ "status": "fail",
+ "message": "Invalid JSON format in request body."
+}
+```
+
+
+
+------------------------------------------------------------------------------------------
+
+### 获取所有节点日志内容
+
+POST/update-tags因为更新只能针对于某个节点，因此需要用当前节点ip:port用于更新(如果更新成功，则返回"success"信息)
+
+
+#### 更新节点tags
+##### 请求体
+如果请求参数是`Map`对象，表示要更新当前节点的tags
+```json
+{
+ "tag1": "dev_1",
+ "tag2": "dev_2"
+}
+```
+##### 响应
+
+```json
+{
+ "status": "success",
+ "message": "update node tags done."
+}
+```
+#### 移除节点tags
+##### 请求体
+如果参数为空`Map`对象，表示要清除当前节点的tags
+```json
+{}
+```
+##### 响应
+响应体将为：
+```json
+{
+ "status": "success",
+ "message": "update node tags done."
+}
+```
+
+#### 请求参数异常
+- 如果请求参数为空
+
+##### 响应
+
+```json
+{
+ "status": "fail",
+ "message": "Request body is empty."
+}
+```
+- 如果参数不是`Map`对象
+##### 响应
+
+```json
+{
+ "status": "fail",
+ "message": "Invalid JSON format in request body."
+}
+```
+
+
+
+#### 请求参数
+
+#### 参数(在请求体中params字段中添加)
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |----------------------|----------|--------|-----------------------------------|
+> | jobId | optional | string | job id |
+
+当`jobId`为空时，返回所有节点的日志信息，否则返回指定`jobId`在所有节点的的日志列表。
+
+#### 响应
+
+返回请求节点的日志列表、内容
+
+#### 返回所有日志文件列表
+
+如果你想先查看日志列表，可以通过`GET`请求获取日志列表，`http://localhost:8080/logs?format=json`
+
+```json
+[
+ {
+ "node": "localhost:8080",
+ "logLink": "http://localhost:8080/logs/job-899485770241277953.log",
+ "logName": "job-899485770241277953.log"
+ },
+ {
+ "node": "localhost:8080",
+ "logLink": "http://localhost:8080/logs/job-899470314109468673.log",
+ "logName": "job-899470314109468673.log"
+ }
+]
+```
+
+当前支持的格式有`json`和`html`，默认为`html`。
+
+
+#### 例子
+
+获取所有节点jobId为`733584788375666689`的日志信息：`http://localhost:8080/logs/733584788375666689`
+获取所有节点日志列表：`http://localhost:8080/logs`
+获取所有节点日志列表以JSON格式返回：`http://localhost:8080/logs?format=json`
+获取日志文件内容：`http://localhost:8080/logs/job-898380162133917698.log`
+
+
+
+
+
+### 获取单节点日志内容
+
+GET /logs/:jobId (返回日志列表。)
+
+#### 请求参数
+
+#### 参数(在请求体中params字段中添加)
+
+> | 参数名称 | 是否必传 | 参数类型 | 参数描述 |
+> |----------------------|----------|--------|-----------------------------------|
+> | jobId | optional | string | job id |
+
+当`jobId`为空时，返回所有节点的日志信息，否则返回指定`jobId`在所有节点的的日志列表。
+
+#### 响应
+
+返回请求节点的日志列表、内容
+
+#### 返回所有日志文件列表
+
+如果你想先查看日志列表，可以通过`GET`请求获取日志列表，`http://localhost:8080/logs?format=json`
+
+```json
+[
+ {
+ "node": "localhost:8080",
+ "logLink": "http://localhost:8080/logs/job-899485770241277953.log",
+ "logName": "job-899485770241277953.log"
+ },
+ {
+ "node": "localhost:8080",
+ "logLink": "http://localhost:8080/logs/job-899470314109468673.log",
+ "logName": "job-899470314109468673.log"
+ }
+]
+```
+
+当前支持的格式有`json`和`html`，默认为`html`。
+
+
+#### 例子
+
+获取所有节点jobId为`733584788375666689`的日志信息：`http://localhost:8080/logs/733584788375666689`
+获取所有节点日志列表：`http://localhost:8080/logs`
+获取所有节点日志列表以JSON格式返回：`http://localhost:8080/logs?format=json`
+获取日志文件内容：`http://localhost:8080/logs/job-898380162133917698.log`
+
+
+
+
+
+#### 响应
+
+返回请求节点的日志列表
+
+#### 例子
+
+获取当前节点的日志列表：`http://localhost:5801/log`
+获取日志文件内容：`http://localhost:5801/log/job-898380162133917698.log``
+
+
diff --git a/docs/zh/seatunnel-engine/separated-cluster-deployment.md b/docs/zh/seatunnel-engine/separated-cluster-deployment.md
index 9c6a230537e..bdc369ff8c0 100644
--- a/docs/zh/seatunnel-engine/separated-cluster-deployment.md
+++ b/docs/zh/seatunnel-engine/separated-cluster-deployment.md
@@ -75,7 +75,7 @@ SeaTunnel Engine 基于 [Hazelcast IMDG](https://docs.hazelcast.com/imdg/4.1/)
`backup count` 是定义同步备份数量的参数。例如，如果设置为 1，则分区的备份将放置在一个其他成员上。如果设置为 2，则将放置在两个其他成员上。
-我们建议 `backup-count` 的值为 `min(1, max(5, N/2))`。 `N` 是集群节点的数量。
+我们建议 `backup-count` 的值为 `max(1, min(5, N/2))`。 `N` 是集群节点的数量。
```yaml
seatunnel:
@@ -284,6 +284,23 @@ netty-common-4.1.89.Final.jar
seatunnel-hadoop3-3.1.4-uber.jar
```
+### 4.7 作业调度策略
+
+当资源不足时，作业调度策略可以配置为以下两种模式：
+
+1. `WAIT`：等待资源可用。
+2. `REJECT`：拒绝作业，默认值。
+
+示例
+
+```yaml
+seatunnel:
+ engine:
+ job-schedule-strategy: WAIT
+```
+
+当`dynamic-slot: ture`时，`job-schedule-strategy: WAIT` 配置会失效，将被强制修改为`job-schedule-strategy: REJECT`，因为动态Slot时该参数没有意义，可以直接提交。
+
## 5. 配置 SeaTunnel Engine 网络服务
所有 SeaTunnel Engine 网络相关的配置都在 `hazelcast-master.yaml`和`hazelcast-worker.yaml` 文件中.
@@ -448,4 +465,4 @@ hazelcast-client:
### 8.2 使用 REST API 提交作业
-SeaTunnel Engine 提供了 REST API 用于提交作业。有关详细信息，请参阅 [REST API](rest-api.md)
+SeaTunnel Engine 提供了 REST API 用于提交作业。有关详细信息，请参阅 [REST API V2](rest-api-v2.md)
diff --git a/docs/zh/seatunnel-engine/telemetry.md b/docs/zh/seatunnel-engine/telemetry.md
index 31972a44bca..b65a3d0225b 100644
--- a/docs/zh/seatunnel-engine/telemetry.md
+++ b/docs/zh/seatunnel-engine/telemetry.md
@@ -1,5 +1,5 @@
---
-sidebar_position: 13
+sidebar_position: 14
---
# Telemetry
diff --git a/docs/zh/seatunnel-engine/telemetry/grafana-dashboard.json b/docs/zh/seatunnel-engine/telemetry/grafana-dashboard.json
index e69de29bb2d..7a87e47ff38 100644
--- a/docs/zh/seatunnel-engine/telemetry/grafana-dashboard.json
+++ b/docs/zh/seatunnel-engine/telemetry/grafana-dashboard.json
@@ -0,0 +1 @@
+{"annotations":{"list":[{"builtIn":1,"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","target":{"limit":100,"matchAny":false,"tags":[],"type":"dashboard"},"type":"dashboard"}]},"editable":true,"fiscalYearStartMonth":0,"graphTooltip":0,"id":8,"links":[],"liveNow":false,"panels":[{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"super-light-blue","value":null},{"color":"red","value":100000}]}},"overrides":[]},"gridPos":{"h":4,"w":12,"x":0,"y":0},"id":17,"options":{"colorMode":"background","graphMode":"none","justifyMode":"center","orientation":"auto","percentChangeColorMode":"standard","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"showPercentChange":false,"textMode":"auto","wideLayout":true},"pluginVersion":"11.2.0","targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"node_count{instance=~\"$instance\"}","interval":"","legendFormat":"","range":true,"refId":"A"}],"title":"Total Node Count","type":"stat"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":4,"w":12,"x":12,"y":0},"id":18,"options":{"colorMode":"background","graphMode":"none","justifyMode":"auto","orientation":"auto","percentChangeColorMode":"standard","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"showPercentChange":false,"text":{"titleSize":1},"textMode":"auto","wideLayout":true},"pluginVersion":"11.2.0","targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"sum(node_state{instance=~\"$instance\"})","interval":"","legendFormat":"__auto","range":true,"refId":"A"}],"title":"UP Node Count","type":"stat"},{"collapsed":false,"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"gridPos":{"h":1,"w":24,"x":0,"y":4},"id":22,"panels":[],"targets":[{"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"refId":"A"}],"title":"Hazelcast Partition","type":"row"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":7,"w":12,"x":0,"y":5},"id":32,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_partition_partitionCount{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}","range":true,"refId":"A"}],"title":"partitionCount","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"linear","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":7,"w":12,"x":12,"y":5},"id":33,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_partition_activePartition{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}","range":true,"refId":"A"}],"title":"activePartition","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":7,"w":12,"x":0,"y":12},"id":34,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"pluginVersion":"8.3.3","targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_partition_isClusterSafe{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}","range":true,"refId":"A"}],"title":"isClusterSafe","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":7,"w":12,"x":12,"y":12},"id":35,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_partition_isLocalMemberSafe{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}","range":true,"refId":"A"}],"title":"isLocalMemberSafe","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"gridPos":{"h":1,"w":24,"x":0,"y":19},"id":20,"targets":[{"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"refId":"A"}],"title":"Hazelcast Executor","type":"row"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":6,"w":24,"x":0,"y":20},"id":24,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_executor_executedCount{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"}],"title":"executedCount","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":26},"id":26,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_executor_isTerminated{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"}],"title":"isTerminated","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":26},"id":25,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_executor_isShutdown{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"}],"title":"isShutdown","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":34},"id":28,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_executor_poolSize{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"}],"title":"poolSize","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":34},"id":27,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_executor_maxPoolSize{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"}],"title":"maxPoolSize","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"linear","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":42},"id":30,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_executor_queueRemainingCapacity{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"}],"title":"queueRemainingCapacity","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"linear","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":42},"id":29,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"hazelcast_executor_queueSize{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"}],"title":"queueSize","type":"timeseries"},{"collapsed":false,"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"gridPos":{"h":1,"w":24,"x":0,"y":50},"id":7,"panels":[],"targets":[{"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"refId":"A"}],"title":"System","type":"row"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":18,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineStyle":{"fill":"solid"},"lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":51},"id":9,"interval":"300","options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"rate(process_cpu_seconds_total{instance=~\"$instance\"}[$__interval])*100","interval":"","legendFormat":"{{instance}}","range":true,"refId":"A"}],"title":"Cpu Usage","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":22,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":51},"id":10,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"100 * (jvm_memory_bytes_used{instance=~\"$instance\",area=\"heap\"} / jvm_memory_bytes_max{instance=~\"$instance\",area=\"heap\"})","interval":"","legendFormat":"{{instance}}","range":true,"refId":"A"}],"title":"Heap Memory Usage","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":59},"id":12,"interval":"300","options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"increase(jvm_gc_collection_seconds_count[$__interval])","interval":"","legendFormat":"{{instance}}-{{gc}}","range":true,"refId":"A"}],"title":"GC Count","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":59},"id":13,"interval":"300","options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"increase(jvm_gc_collection_seconds_sum{instance=~\"$instance\"}[$__interval])*1000","interval":"","legendFormat":"{{instance}}-{{gc}}","range":true,"refId":"A"}],"title":"GC Cost Time","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"linear","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":8,"w":24,"x":0,"y":67},"id":14,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"jvm_threads_current{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-current","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"jvm_threads_daemon{instance=~\"$instance\"}","hide":false,"interval":"","legendFormat":"{{instance}}-daemon","range":true,"refId":"B"},{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"jvm_threads_peak{instance=~\"$instance\"}","hide":false,"interval":"","legendFormat":"{{instance}}-peak","range":true,"refId":"C"},{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"jvm_threads_deadlocked{instance=~\"$instance\"}","hide":false,"interval":"","legendFormat":"{{instance}}-deadlocked","range":true,"refId":"D"}],"title":"Jvm Threads","type":"timeseries"},{"collapsed":false,"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"gridPos":{"h":1,"w":24,"x":0,"y":75},"id":5,"panels":[],"targets":[{"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"refId":"A"}],"title":"Job","type":"row"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"continuous-YlBl"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]}},"overrides":[]},"gridPos":{"h":6,"w":24,"x":0,"y":76},"id":2,"options":{"displayMode":"basic","maxVizHeight":300,"minVizHeight":16,"minVizWidth":8,"namePlacement":"auto","orientation":"vertical","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"showUnfilled":true,"sizing":"auto","valueMode":"color"},"pluginVersion":"11.2.0","targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"sum(job_count) by (type) ","hide":false,"interval":"","legendFormat":"__auto","range":true,"refId":"A"}],"title":"Job Count","type":"bargauge"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":6,"w":12,"x":0,"y":82},"id":3,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"job_thread_pool_activeCount{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"job_thread_pool_corePoolSize{instance=~\"$instance\"}","hide":false,"interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"B"},{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"job_thread_pool_maximumPoolSize{instance=~\"$instance\"}","hide":true,"interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"C"},{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"job_thread_pool_poolSize{instance=~\"$instance\"}","hide":false,"interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"D"}],"title":"Job Thread Pool","type":"timeseries"},{"datasource":{"default":true,"type":"prometheus","uid":"edwo9tknxxgqof"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"barWidthFactor":0.6,"drawStyle":"line","fillOpacity":12,"gradientMode":"opacity","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"smooth","lineWidth":1,"pointSize":1,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":6,"w":12,"x":12,"y":82},"id":15,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"job_thread_pool_completedTask_total{instance=~\"$instance\"}","interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"jUi2yaj4k"},"editorMode":"code","exemplar":true,"expr":"job_thread_pool_task_total{instance=~\"$instance\"}","hide":false,"interval":"","legendFormat":"{{instance}}-{{type}}","range":true,"refId":"B"}],"title":"Job Thread Pool Total","type":"timeseries"}],"refresh":"30s","schemaVersion":39,"tags":[],"templating":{"list":[{"current":{"selected":true,"text":["All"],"value":["$__all"]},"datasource":{"type":"prometheus","uid":"edwo9tknxxgqof"},"definition":"label_values(cluster_info,instance)","description":"instance","hide":0,"includeAll":true,"label":"","multi":true,"name":"instance","options":[],"query":{"qryType":5,"query":"label_values(cluster_info,instance)","refId":"PrometheusVariableQueryEditor-VariableQuery"},"refresh":1,"regex":"","skipUrlSync":false,"sort":0,"type":"query"}]},"time":{"from":"now-15m","to":"now"},"timepicker":{},"timezone":"","title":"Seatunnel","uid":"bdx1j097hmku8d","version":11,"weekStart":""}
\ No newline at end of file
diff --git a/docs/zh/seatunnel-engine/user-command.md b/docs/zh/seatunnel-engine/user-command.md
index 9311ad912f6..1ceea35c85d 100644
--- a/docs/zh/seatunnel-engine/user-command.md
+++ b/docs/zh/seatunnel-engine/user-command.md
@@ -1,5 +1,5 @@
---
-sidebar_position: 12
+sidebar_position: 13
---
# 命令行工具
@@ -138,3 +138,12 @@ bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template
被cancel的作业的所有断点信息都将被删除，无法通过seatunnel.sh -r <jobId>恢复。
+## 配置JVM参数
+
+我们可以通过以下方式为 SeaTunnel Engine 客户端配置 JVM 参数：
+
+1. 添加JVM参数到`$SEATUNNEL_HOME/config/jvm_client_options`文件中。
+
+ 在 `$SEATUNNEL_HOME/config/jvm_client_options` 文件中修改 JVM 参数。请注意，该文件中的 JVM 参数将应用于使用 `seatunnel.sh` 提交的所有作业，包括 Local 模式和 Cluster 模式。
+
+2. 在提交作业时添加 JVM 参数。例如，`sh bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template -DJvmOption="-Xms2G -Xmx2G"`
diff --git a/docs/zh/seatunnel-engine/web-ui.md b/docs/zh/seatunnel-engine/web-ui.md
new file mode 100644
index 00000000000..35b551225ec
--- /dev/null
+++ b/docs/zh/seatunnel-engine/web-ui.md
@@ -0,0 +1,47 @@
+# Web UI
+
+## 访问
+
+在访问 web ui 之前我们需要开启 http rest api。首先需要在`seatunnel.yaml`配置文件中配置
+
+```
+seatunnel:
+ engine:
+ http:
+ enable-http: true
+ port: 8080
+```
+
+然后访问 `http://ip:8080/#/overview`
+
+## 概述
+
+Apache SeaTunnel 的 Web UI 提供了一个友好的用户界面，用于监控和管理 SeaTunnel 作业。通过 Web UI，用户可以实时查看当前运行的作业、已完成的作业，以及集群中工作节点和管理节点的状态。主要功能模块包括 Jobs、Workers 和 Master，每个模块都提供了详细的状态信息和操作选项，帮助用户高效地管理和优化其数据处理流程。
+
+
+## 作业
+
+### 运行中的作业
+
+“运行中的作业”模块列出了当前正在执行的所有 SeaTunnel 作业。用户可以查看每个作业的基本信息，包括作业 ID、提交时间、状态、执行时间等。点击具体作业可以查看更多详细信息，如任务分布、资源使用情况和日志输出，便于用户实时监控作业进度并及时处理潜在问题。
+
+
+
+### 已完成的作业
+
+“已完成的作业”模块展示了所有已成功完成或失败的 SeaTunnel 作业。此部分提供了每个作业的执行结果、完成时间、耗时以及失败原因（如果有）。用户可以通过此模块回顾过去的作业记录，分析作业性能，进行故障排查或重复执行某些特定作业。
+
+
+## 工作节点
+
+### 工作节点信息
+
+“工作节点”模块展示了集群中所有工作节点的详细信息，包括每个工作节点的地址、运行状态、CPU 和内存使用情况、正在执行的任务数量等。通过该模块，用户可以监控各个工作节点的健康状况，及时发现和处理资源瓶颈或节点故障，确保 SeaTunnel 集群的稳定运行。
+
+
+## 管理节点
+
+### 管理节点信息
+
+“管理节点”模块提供了 SeaTunnel 集群中主节点的状态和配置信息。用户可以查看 Master 节点的地址、运行状态、负责的作业调度情况以及整体集群的资源分配情况。该模块帮助用户全面了解集群的核心管理部分，便于进行集群配置优化和故障排查。
+
diff --git a/docs/zh/start-v2/docker/docker.md b/docs/zh/start-v2/docker/docker.md
index 309417a5993..1c4bc5d4b10 100644
--- a/docs/zh/start-v2/docker/docker.md
+++ b/docs/zh/start-v2/docker/docker.md
@@ -40,7 +40,7 @@ docker run --rm -it -v /tmp/job/:/config apache/seatunnel:GET /log (返回日志列表。)
+
+#### 响应
+
+返回请求节点的日志列表
+
+#### 例子
+
+获取当前节点的日志列表：`http://localhost:5801/log`
+获取日志文件内容：`http://localhost:5801/log/job-898380162133917698.log``
+
++Example:
+CDC row: -D 1, test1, test2
+transformed Row: +I 1,test1,test2,DELETE + +## Options + +| name | type | required | default value | +|-------------------|--------|----------|---------------| +| custom_field_name | string | yes | row_kind | +| transform_type | enum | yes | SHORT | + +### custom_field_name [string] + +RowKind列的自定义名 + +### transform_type [enum] + +格式化RowKind值 , 配置为 `SHORT` 或 `FULL` + +`SHORT` : +I, -U , +U, -D +`FULL` : INSERT, UPDATE_BEFORE, UPDATE_AFTER , DELETE + +## Examples + +```yaml + +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FakeSource { + schema = { + fields { + pk_id = bigint + name = string + score = int + } + primaryKey { + name = "pk_id" + columnNames = [pk_id] + } + } + rows = [ + { + kind = INSERT + fields = [1, "A", 100] + }, + { + kind = INSERT + fields = [2, "B", 100] + }, + { + kind = INSERT + fields = [3, "C", 100] + }, + { + kind = INSERT + fields = [4, "D", 100] + }, + { + kind = UPDATE_BEFORE + fields = [1, "A", 100] + }, + { + kind = UPDATE_AFTER + fields = [1, "F", 100] + } + { + kind = UPDATE_BEFORE + fields = [2, "B", 100] + }, + { + kind = UPDATE_AFTER + fields = [2, "G", 100] + }, + { + kind = DELETE + fields = [3, "C", 100] + }, + { + kind = DELETE + fields = [4, "D", 100] + } + ] + } +} + +transform { + RowKindExtractor { + custom_field_name = "custom_name" + transform_type = FULL + plugin_output = "trans_result" + } +} + +sink { + Console { + plugin_input = "custom_name" + } +} + +``` + diff --git a/docs/zh/transform-v2/split.md b/docs/zh/transform-v2/split.md index ef8c3f58540..7fba623520a 100644 --- a/docs/zh/transform-v2/split.md +++ b/docs/zh/transform-v2/split.md @@ -46,8 +46,8 @@ ``` transform { Split { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" separator = " " split_field = "name" output_fields = [first_name, second_name] diff --git a/docs/zh/transform-v2/sql-functions.md b/docs/zh/transform-v2/sql-functions.md index 57c440a39b3..7e3f8454e1d 100644 --- a/docs/zh/transform-v2/sql-functions.md +++ b/docs/zh/transform-v2/sql-functions.md @@ -302,6 +302,15 @@ REPEAT(NAME || ' ', 10) REPLACE(NAME, ' ') + +### SPLIT + +将字符串切分成数组。 + +示例: + +select SPLIT(test,';') as arrays + ### SOUNDEX ```SOUNDEX(string)``` @@ -964,3 +973,37 @@ from 示例: case when c_string in ('c_string') then 1 else 0 end + +### UUID + +```UUID()``` + +通过java函数生成uuid + +示例: + +select UUID() as seatunnel_uuid + + +### ARRAY + +生成一个数组。 + +示例: + +select ARRAY('test1','test2','test3') as arrays + +### LATERAL VIEW +#### EXPLODE + +将 array 列展开成多行。 +OUTER EXPLODE 当 array 为NULL或者为空时，返回NULL +EXPLODE(SPLIT(FIELD_NAME,separator))用来切分字符串类型，SPLIT 第一个参数是字段名，第二个参数是分隔符 +EXPLODE(ARRAY(value1,value2)) 用于自定义数组切分，在原有基础上生成一个新的字段。 +``` +SELECT * FROM fake + LATERAL VIEW EXPLODE ( SPLIT ( NAME, ',' ) ) AS NAME + LATERAL VIEW EXPLODE ( SPLIT ( pk_id, ';' ) ) AS pk_id + LATERAL VIEW OUTER EXPLODE ( age ) AS age + LATERAL VIEW OUTER EXPLODE ( ARRAY(1,1) ) AS num +``` diff --git a/docs/zh/transform-v2/sql-udf.md b/docs/zh/transform-v2/sql-udf.md index 4c1a3777408..df03b5db5c2 100644 --- a/docs/zh/transform-v2/sql-udf.md +++ b/docs/zh/transform-v2/sql-udf.md @@ -109,8 +109,8 @@ public class ExampleUDF implements ZetaUDF { ``` transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" query = "select id, example(name) as name, age from fake" } } diff --git a/docs/zh/transform-v2/sql.md b/docs/zh/transform-v2/sql.md index 1b56f1fef3f..87d9380e7a0 100644 --- a/docs/zh/transform-v2/sql.md +++ b/docs/zh/transform-v2/sql.md @@ -12,11 +12,11 @@ SQL 转换使用内存中的 SQL 引擎，我们可以通过 SQL 函数和 SQL | 名称 | 类型 | 是否必须 | 默认值 | |-------------------|--------|------|-----| -| source_table_name | string | yes | - | -| result_table_name | string | yes | - | +| plugin_input | string | yes | - | +| plugin_output | string | yes | - | | query | string | yes | - | -### source_table_name [string] +### plugin_input [string] 源表名称，查询 SQL 表名称必须与此字段匹配。 @@ -43,8 +43,8 @@ SQL 转换使用内存中的 SQL 引擎，我们可以通过 SQL 函数和 SQL ``` transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" query = "select id, concat(name, '_') as name, age+1 as age from fake where id>0" } } @@ -66,7 +66,7 @@ transform { ```hacon source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 string.template = ["innerQuery"] schema = { @@ -123,7 +123,7 @@ env { source { FakeSource { - result_table_name = "fake" + plugin_output = "fake" row.num = 100 schema = { fields { @@ -137,15 +137,15 @@ source { transform { Sql { - source_table_name = "fake" - result_table_name = "fake1" + plugin_input = "fake" + plugin_output = "fake1" query = "select id, concat(name, '_') as name, age+1 as age from fake where id>0" } } sink { Console { - source_table_name = "fake1" + plugin_input = "fake1" } } ``` diff --git a/docs/zh/transform-v2/transform-multi-table.md b/docs/zh/transform-v2/transform-multi-table.md new file mode 100644 index 00000000000..2881f319e7a --- /dev/null +++ b/docs/zh/transform-v2/transform-multi-table.md @@ -0,0 +1,124 @@ +--- +sidebar_position: 2 +--- + +# Transform的多表转换 + +SeaTunnel transform支持多表转换，在上游插件输出多个表的时候特别有用，能够在一个transform中完成所有的转换操作。目前SeaTunnel很多Connectors支持多表输出，比如`JDBCSource`、`MySQL-CDC` +等。所有的Transform都可以通过如下配置实现多表转换。 + +:::tip + +多表Transform没有对Transform能力的限制，任何Transform的配置都可以在多表Transform中使用。多表Transform的作用针对数据流中的多个表进行单独的处理，并将多个表的Transform配置合并到一个Transform中，方便用户管理。 + +::: + +## 属性 + +| Name | Type | Required | Default | Description | +|----------------------------|--------|----------|---------|--------------------------------------------------------------------------------------------------| +| table_match_regex | String | No | .* | 表名的正则表达式，通过正则表达式来匹配需要进行转换的表，默认匹配所有的表。注意这个表名是上游的真正表名，不是result_table_name。 | +| table_transform | List | No | - | 可以通过table_transform列表来指定部分表的规则，当在table_transform中配置某个表的转换规则后，外层针对当前表的规则不会生效，以table_transform中的为准 | +| table_transform.table_path | String | No | - | 当在table_transform中配置某个表的转换规则后，需要使用table_path字段指定表名，表名需要包含`databaseName[.schemaName].tableName`。 | + +## 匹配逻辑 + +假设我们从上游读取了5张表，分别为`test.abc`，`test.abcd`，`test.xyz`，`test.xyzxyz`，`test.www`。他们的表结构一致，都有`id`、`name`、`age`三个字段。 + +| id | name | age | + +现在我们想通过Copy transform将这5张表的数据进行复制，具体需求是，`test.abc`，`test.abcd`表需要将`name`复制为`name1`， +`test.xyz`表需要复制为`name2`，`test.xyzxyz`表需要复制为`name3`，`test.www`数据结构不变。那么我们可以通过如下配置来实现： + +```hocon +transform { + Copy { + source_table_name = "fake" // 可选的读取数据集名 + result_table_name = "fake1" // 可选的输出数据集名 + + table_match_regex = "test.a.*" // 1. 通过正则表达式匹配需要进行转换的表，test.a.*表示匹配test.abc和test.abcd + src_field = "name" // 源字段 + dest_field = "name1" // 目标字段 + table_transform = [{ + table_path = "test.xyz" // 2. 指定表名进行转换 + src_field = "name" // 源字段 + dest_field = "name2" // 目标字段 + }, { + table_path = "test.xyzxyz" + src_field = "name" + dest_field = "name3" + }] + } +} +``` + +### 解释 + +1. 通过第一层的正则表达式，和对应的Copy transform options配置，我们可以匹配到`test.abc`和`test.abcd`表，将`name`字段复制为`name1`。 +2. 通过`table_transform`配置，我们可以指定`test.xyz`表，将`name`字段复制为`name2`。 + +这样我们就可以通过一个transform完成对多个表的转换操作。 + +对于每个表来说，配置的优先级是：`table_transform` > `table_match_regex`。如果所有的规则都没有匹配到，那么该表将不会进行任何转换操作。 + +针对每个表来说，他们的Transform配置是： + +- **test.abc**和**test.abcd** + +```hocon +transform { + Copy { + src_field = "name" + dest_field = "name1" + } +} +``` + +输出表结构： + +| id | name | age | name1 | + +- **test.xyz** + +```hocon +transform { + Copy { + src_field = "name" + dest_field = "name2" + } +} +``` + +输出表结构： + +| id | name | age | name2 | + +- **test.xyzxyz** + +```hocon +transform { + Copy { + src_field = "name" + dest_field = "name3" + } +} +``` + +输出表结构： + +| id | name | age | name3 | + +- **test.www** + +```hocon +transform { + // 无需转换 +} +``` + +输出表结构： + +| id | name | age | + +我们使用了Copy Transform作为了示例，实际上所有的Transform都支持多表转换，只需要在对应的Transform中配置即可。 + diff --git a/plugin-mapping.properties b/plugin-mapping.properties index 3673ade48f6..c494686161e 100644 --- a/plugin-mapping.properties +++ b/plugin-mapping.properties @@ -132,12 +132,16 @@ seatunnel.sink.ObsFile = connector-file-obs seatunnel.source.Milvus = connector-milvus seatunnel.sink.Milvus = connector-milvus seatunnel.sink.ActiveMQ = connector-activemq +seatunnel.source.Prometheus = connector-prometheus +seatunnel.sink.Prometheus = connector-prometheus seatunnel.source.Qdrant = connector-qdrant seatunnel.sink.Qdrant = connector-qdrant seatunnel.source.Sls = connector-sls +seatunnel.sink.Sls = connector-sls seatunnel.source.Typesense = connector-typesense seatunnel.sink.Typesense = connector-typesense seatunnel.source.Opengauss-CDC = connector-cdc-opengauss + seatunnel.transform.Sql = seatunnel-transforms-v2 seatunnel.transform.FieldMapper = seatunnel-transforms-v2 seatunnel.transform.Filter = seatunnel-transforms-v2 @@ -149,4 +153,5 @@ seatunnel.transform.Copy = seatunnel-transforms-v2 seatunnel.transform.DynamicCompile = seatunnel-transforms-v2 seatunnel.transform.LLM = seatunnel-transforms-v2 seatunnel.transform.Embedding = seatunnel-transforms-v2 - +seatunnel.transform.RowKindExtractor = seatunnel-transforms-v2 +seatunnel.transform.Metadata = seatunnel-transforms-v2 diff --git a/pom.xml b/pom.xml index ab8453af400..e050018b2b8 100644 --- a/pom.xml +++ b/pom.xml @@ -56,7 +56,7 @@
If the splits or enumerator state is null, the corresponding field in the returned state
+ * will be null.
+ *
+ * @param checkpoint
+ * @return
+ * @param