Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
Hisoka-X committed Dec 13, 2024
1 parent a8e9904 commit 8988052
Show file tree
Hide file tree
Showing 2 changed files with 59 additions and 9 deletions.
65 changes: 58 additions & 7 deletions docs/en/connector-v2/sink/Maxcompute.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,64 @@ Used to read data from Maxcompute.

`overwrite` Whether to overwrite the table or partition, default: false.

### save_mode_create_template

We use templates to automatically create MaxCompute tables,
which will create corresponding table creation statements based on the type of upstream data and schema type,
and the default template can be modified according to the situation. Only work on multi-table mode at now.

Default template:

```sql
CREATE TABLE IF NOT EXISTS `${table}` (
${rowtype_fields}
);
```

If a custom field is filled in the template, such as adding an `id` field

```sql
CREATE TABLE IF NOT EXISTS `${table}`
(
id,
${rowtype_fields}
);
```

The connector will automatically obtain the corresponding type from the upstream to complete the filling,
and remove the id field from `rowtype_fields`. This method can be used to customize the modification of field types and attributes.

You can use the following placeholders

- database: Used to get the database in the upstream schema
- table_name: Used to get the table name in the upstream schema
- rowtype_fields: Used to get all the fields in the upstream schema, we will automatically map to the field
description of MaxCompute
- rowtype_primary_key: Used to get the primary key in the upstream schema (maybe a list)
- rowtype_unique_key: Used to get the unique key in the upstream schema (maybe a list)

### schema_save_mode[Enum]

Before the synchronous task is turned on, different treatment schemes are selected for the existing surface structure of the target side.
Option introduction:
`RECREATE_SCHEMA` :Will create when the table does not exist, delete and rebuild when the table is saved
`CREATE_SCHEMA_WHEN_NOT_EXIST` :Will Created when the table does not exist, skipped when the table is saved
`ERROR_WHEN_SCHEMA_NOT_EXIST` :Error will be reported when the table does not exist
`IGNORE` :Ignore the treatment of the table

### data_save_mode[Enum]

Before the synchronous task is turned on, different processing schemes are selected for data existing data on the target side.
Option introduction:
`DROP_DATA`: Preserve database structure and delete data
`APPEND_DATA`:Preserve database structure, preserve data
`CUSTOM_PROCESSING`:User defined processing
`ERROR_WHEN_DATA_EXISTS`:When there is data, an error is reported

### custom_sql[String]

When data_save_mode selects CUSTOM_PROCESSING, you should fill in the CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be executed. SQL will be executed before synchronization tasks.

### common options

Sink plugin common parameters, please refer to [Sink Common Options](../sink-common-options.md) for details.
Expand All @@ -70,10 +128,3 @@ sink {
}
}
```

## Changelog

### next version

- [Feature] Add Maxcompute Sink Connector([3640](https://github.com/apache/seatunnel/pull/3640))

Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
import org.apache.seatunnel.common.constants.PluginType;
import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSimpleSink;
import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter;
import org.apache.seatunnel.connectors.seatunnel.maxcompute.catalog.MaxComputeCatalog;
import org.apache.seatunnel.connectors.seatunnel.maxcompute.config.MaxcomputeConfig;
import org.apache.seatunnel.connectors.seatunnel.maxcompute.exception.MaxcomputeConnectorException;
Expand Down Expand Up @@ -61,7 +60,7 @@ public String getPluginName() {
}

@Override
public AbstractSinkWriter<SeaTunnelRow, Void> createWriter(SinkWriter.Context context) {
public MaxcomputeWriter createWriter(SinkWriter.Context context) {
return new MaxcomputeWriter(this.readonlyConfig, this.catalogTable.getSeaTunnelRowType());
}

Expand Down

0 comments on commit 8988052

Please sign in to comment.