Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC][Connector-V2][Kudu] update kudu doc #5231

Closed
wants to merge 2 commits into from
Closed

[DOC][Connector-V2][Kudu] update kudu doc #5231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0f0b67d
[DOC][Connector-V2][Kudu] update kudu doc
Aug 6, 2023
7823961
[DOC][Connector-V2][Kudu] update kudu doc
Aug 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 65 additions & 35 deletions docs/en/connector-v2/sink/Kudu.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,50 +2,80 @@

> Kudu sink connector

## Description
## Support Those Engines

Write data to Kudu.

The tested kudu version is 1.11.1.
> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>

## Key features

- [x] [batch](../../concept/connector-v2-features.md)
- [x] [stream](../../concept/connector-v2-features.md)
- [ ] [exactly-once](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|----------------|--------|----------|---------------|
| kudu_master | string | yes | - |
| kudu_table | string | yes | - |
| save_mode | string | yes | - |
| common-options | | no | - |

### kudu_master [string]

`kudu_master` The address of kudu master,such as '192.168.88.110:7051'.

### kudu_table [string]

`kudu_table` The name of kudu table..

### save_mode [string]

Storage mode, we need support `overwrite` and `append`. `append` is now supported.

### common options

Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details.

## Example

```bash

kudu {
- [x] [column projection](../../concept/connector-v2-features.md)
- [x] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

## Data Type Mapping

| SeaTunnel Data type | kudu Data type |
|---------------------|--------------------------|
| BOOLEAN | BOOL |
| INT | INT8<br/>INT16<br/>INT32 |
| BIGINT | INT64 |
| DECIMAL(20,0) | DECIMAL32 |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| STRING | STRING |
| TIMESTAMP | UNIXTIME_MICROS |
| ARRAY | BINARY |

## Sink Options

| Name | Type | Required | Default | Description |
|----------------|--------|----------|---------|----------------------------------------------------------------------------------------------------------|
| kudu_master | String | Yes | - | The address of kudu master,such as '192.168.88.110:7051'. |
| kudu_table | String | Yes | - | The name of kudu table. |
| save_mode | String | No | - | Storage mode, support `overwrite` and `append`. |
| common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details. |

## Task Example

### Simple:

> The following example refers to a Kudu table named "studentlyh2" with four fields: id, name, age, and sex, which will be imported into another Kudu table named "studentlyhresultflink" with the same table structure

```hocon

# Defining the runtime environment
env {
# You can set flink configuration here
execution.parallelism = 2
job.mode = "BATCH"
}

source {
Kudu {
result_table_name = "studentlyh2"
kudu_master = "192.168.88.110:7051"
kudu_table = "studentlyh2"
columnsList = "id,name,age,sex"
}
}

transform {
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/transform-v2/sql
}

sink {
kudu {
kudu_master = "192.168.88.110:7051"
kudu_table = "studentlyhresultflink"
save_mode="append"
}
}

```

Expand Down
71 changes: 47 additions & 24 deletions docs/en/connector-v2/source/Kudu.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,57 +2,80 @@

> Kudu source connector

## Description

Used to read data from Kudu.
## Support Those Engines

The tested kudu version is 1.11.1.
> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>

## Key features

- [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
- [x] [stream](../../concept/connector-v2-features.md)
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [ ] [column projection](../../concept/connector-v2-features.md)
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [x] [column projection](../../concept/connector-v2-features.md)
- [x] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|----------------|--------|----------|---------------|
| kudu_master | string | yes | - |
| kudu_table | string | yes | - |
| columnsList | string | yes | - |
| common-options | | no | - |
## Description

### kudu_master [string]
Used to read data from Kudu.

`kudu_master` The address of kudu master,such as '192.168.88.110:7051'.
The tested kudu version is 1.11.1.

### kudu_table [string]
## Data Type Mapping

`kudu_table` The name of kudu table..
| kudu Data type | SeaTunnel Data type |
|--------------------------|---------------------|
| BOOL | BOOLEAN |
| INT8<br/>INT16<br/>INT32 | INT |
| INT64 | BIGINT |
| DECIMAL32 | DECIMAL(20,0) |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| STRING | STRING |
| UNIXTIME_MICROS | TIMESTAMP |
| BINARY | ARRAY |

### columnsList [string]
## Source Options

`columnsList` Specifies the column names of the table.
| Name | Type | Required | Default | Description |
|----------------|--------|----------|---------|----------------------------------------------------------------------------------------------------------|
| kudu_master | String | Yes | - | The address of kudu master,such as '192.168.88.110:7051'. |
| kudu_table | String | Yes | - | The name of kudu table. |
| columnsList | String | No | - | Specifies the column names of the table. |
| common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details. |

### common options
## Task Example

Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details.
### Simple:

## Examples
> The following example is for a Kudu table named "studentlyh2" which includes four fields: id, name, age, and sex. The goal is to print the data from this table on the console

```hocon
# Defining the runtime environment
env {
# You can set flink configuration here
execution.parallelism = 2
job.mode = "BATCH"
}

source {
Kudu {
result_table_name = "studentlyh2"
kudu_master = "192.168.88.110:7051"
kudu_table = "studentlyh2"
columnsList = "id,name,age,sex"
}
}

transform {
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/transform-v2/sql
}

sink {
Console {}
}
```

Expand Down