Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Doc] Add doc about connector-v2 schema feature #3296

Merged
merged 7 commits into from
Nov 8, 2022
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions docs/en/concept/schema-feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Intro to schema feature

## Why we need schema

Some nosql databases are not strongly limited schema, so the schema cannot be obtained through the api. At this time, a schema needs to be defined to convert to SeaTunnelRowType and obtain data.
liugddx marked this conversation as resolved.
Show resolved Hide resolved

## What type supported at now

| Data type | Description |
| :-------- | :----------------------------------------------------------- |
| string | string |
| boolean | boolean |
| tinyint | -128 to 127 regular. 0 to 255 unsigned*. Specify the maximum number of digits in parentheses. |
| smallint | -32768 to 32767 General. 0 to 65535 unsigned*. Specify the maximum number of digits in parentheses. |
| int | All numbers from -2,147,483,648 to 2,147,483,647 are allowed. |
| bigint | All numbers between -9,223,372,036,854,775,808 and 9,223,372,036,854,775,807 are allowed. |
| float | Float-precision numeric data from -1.79E+308 to 1.79E+308. |
| double | Double precision floating point. Handle most decimals. |
| decimal | DOUBLE type stored as a string, allowing a fixed decimal point. |
| null | null |
| bytes | bytes. |
| date | Only the date is stored. From January 1, 0001 to December 31, 9999. |
| time | Only store time. Accuracy is 100 nanoseconds. |
| timestamp | Stores a unique number that is updated whenever a row is created or modified. timestamp is based on the internal clock and does not correspond to real time. There can only be one timestamp variable per table. |
| row | Row type,can be nested. |

## How to use schema

### step1: Configure schema in config

```
source {
FakeSource {
parallelism = 2
result_table_name = "fake"
row.num = 16
schema = {
fields {
id = bigint
c_map = "map<string, smallint>"
c_array = "array<tinyint>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_decimal = "decimal(2, 1)"
c_bytes = bytes
c_date = date
c_timestamp = timestamp
}
}
}
}
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please Add When we should use it or not?, because I can't know I can use schema on which source , I can't use schema on which source. (just only need metioned the connector config have schema key can use this feature)


step2:Add dependency
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following is an introduction for developers. It is not necessary to put it here. This article is mainly to provide an introduction for users.


```
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>connector-common</artifactId>
<version>${project.version}</version>
</dependency>
```

### step2:Initialize SeaTunnelSchema

```
SeaTunnelSchema.buildWithConfig(pluginConfig.getConfig(CommonConfig.SCHEMA));
```

### step3:Get seaTunnelRowType and construct SeaTunnelRow

```
SeaTunnelRowType seaTunnelRowType = schema.getSeaTunnelRowType();
```

Please see `FakeDataGenerator`.
1 change: 1 addition & 0 deletions docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ const sidebars = {
items: [
'concept/config',
'concept/connector-v2-features',
'concept/schema-feature',
],
},
'Connector-v2-release-state',
Expand Down