Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector-V2][My Hours]Add My Hours Source Connector #3228

Merged
merged 42 commits into from
Nov 8, 2022
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
016acdd
[Feature][Connector-V2]My Hours Source Connector
TaoZex Oct 22, 2022
fbbadb3
update
TaoZex Oct 22, 2022
74cfb9f
update
TaoZex Oct 22, 2022
34599d9
update
TaoZex Oct 23, 2022
19dee80
update
TaoZex Oct 23, 2022
8e095b0
Merge branch 'apache:dev' into myhours
TaoZex Oct 24, 2022
8c75f98
Remove useless annotation
TaoZex Oct 24, 2022
74b14d1
Merge branch 'myhours' of https://github.com/TaoZex/incubator-seatunn…
TaoZex Oct 24, 2022
f199448
Merge branch 'apache:dev' into myhours
TaoZex Oct 25, 2022
84f8638
[Feature][Connector-V2][My Hours]Add My Hours Source Connector With URL
TaoZex Oct 25, 2022
8714a46
[Feature][Connector-V2][My Hours]Add My Hours Source Connector With URL
TaoZex Oct 25, 2022
70ba19c
Merge branch 'myhours' of https://github.com/TaoZex/incubator-seatunn…
TaoZex Oct 29, 2022
bb9418e
[Feature][Connector-V2][My Hours]Change release log
TaoZex Oct 29, 2022
abb58d1
Merge branch 'apache:dev' into myhours
TaoZex Oct 29, 2022
d8a4985
[Feature][Connector-V2][My Hours]Add fields for example
TaoZex Oct 30, 2022
037b5de
Merge branch 'myhours' of https://github.com/TaoZex/incubator-seatunn…
TaoZex Oct 30, 2022
f02d8cd
Merge branch 'apache:dev' into myhours
TaoZex Oct 30, 2022
49c751d
Merge branch 'apache:dev' into myhours
TaoZex Oct 30, 2022
ac86dc0
Merge branch 'apache:dev' into myhours
TaoZex Oct 31, 2022
fc07087
Merge branch 'apache:dev' into myhours
TaoZex Oct 31, 2022
7317eb4
Merge branch 'apache:dev' into myhours
TaoZex Oct 31, 2022
2216b8c
[Feature][Connector-V2][My Hours]Fix doc format
TaoZex Nov 1, 2022
d769a77
Merge branch 'apache:dev' into myhours
TaoZex Nov 2, 2022
f58958e
[Feature][Connector-V2][My Hours]Delete stream feature
TaoZex Nov 2, 2022
4b53060
Merge branch 'apache:dev' into myhours
TaoZex Nov 3, 2022
ca825f2
Merge branch 'apache:dev' into myhours
TaoZex Nov 3, 2022
1fda841
Merge branch 'apache:dev' into myhours
TaoZex Nov 3, 2022
1a68199
Merge branch 'dev' into myhours
TaoZex Nov 4, 2022
d586d45
[Feature][Connector-V2][My Hours]Add pugin_config
TaoZex Nov 4, 2022
c3c9921
Merge branch 'apache:dev' into myhours
TaoZex Nov 4, 2022
c9cd88e
[Feature][Connector-V2][My Hours]Delete myhours in plugin_config
TaoZex Nov 4, 2022
550915d
Merge branch 'apache:dev' into myhours
TaoZex Nov 4, 2022
e1f1d6d
Merge branch 'apache:dev' into myhours
TaoZex Nov 4, 2022
3dc48cb
Merge branch 'apache:dev' into myhours
TaoZex Nov 4, 2022
8d97c40
[Feature][Connector-V2][My Hours]Fix format
TaoZex Nov 4, 2022
fc394ea
Merge branch 'dev' into myhours
TaoZex Nov 7, 2022
555cd87
[Feature][Connector-V2][My Hours]Update format
TaoZex Nov 7, 2022
1029029
Merge branch 'apache:dev' into myhours
TaoZex Nov 8, 2022
986c37e
[Feature][Connector-V2][My Hours]delete redundant code
TaoZex Nov 8, 2022
33bdf10
Merge branch 'dev' into myhours
EricJoy2048 Nov 8, 2022
73463b0
Merge branch 'dev' into myhours
EricJoy2048 Nov 8, 2022
ab85b1e
Update docs/en/connector-v2/source/MyHours.md
EricJoy2048 Nov 8, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/en/Connector-v2-release-state.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ SeaTunnel uses a grading system for connectors to help you understand what to ex
| [LocalFile](connector-v2/source/LocalFile.md) | Source | Beta | 2.2.0-beta |
| [MongoDB](connector-v2/source/MongoDB.md) | Source | Beta | 2.2.0-beta |
| [MongoDB](connector-v2/sink/MongoDB.md) | Sink | Beta | 2.2.0-beta |
| [MyHours](connector-v2/source/MyHours.md) | Source | Alpha | 2.2.0-beta |
| [Neo4j](connector-v2/sink/Neo4j.md) | Sink | Alpha | 2.2.0-beta |
| [OssFile](connector-v2/sink/OssFile.md) | Sink | Alpha | 2.2.0-beta |
| [OssFile](connector-v2/source/OssFile.md) | Source | Beta | 2.2.0-beta |
Expand Down
177 changes: 177 additions & 0 deletions docs/en/connector-v2/source/MyHours.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
# My Hours

> My Hours source connector

## Description

Used to read data from My Hours.

## Key features

- [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [x] [schema projection](../../concept/connector-v2-features.md)
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
| --------------------------- | ------ | -------- | ------------- |
| url | String | Yes | - |
| email | String | Yes | - |
| password | String | Yes | - |
| method | String | No | get |
| schema.fields | Config | No | - |
| format | String | No | json |
| params | Map | No | - |
| body | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
| retry_backoff_max_ms | int | No | 10000 |
| common-options | | No | - |
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved

### url [String]

http request url

### email [String]

email for login

### password [String]

password for login

### method [String]

http request method, only supports GET, POST method

### params [Map]

http params

### body [String]

http body

### poll_interval_ms [int]

request http api interval(millis) in stream mode

### retry [int]

The max retry times if request http return to `IOException`

### retry_backoff_multiplier_ms [int]

The retry-backoff times(millis) multiplier if request http failed

### retry_backoff_max_ms [int]

The maximum retry-backoff times(millis) if request http failed

### format [String]

the format of upstream data, now only support `json` `text`, default `json`.

when you assign format is `json`, you should also assign schema option, for example:

upstream data is the following:

```json

{"code": 200, "data": "get success", "success": true}

```

you should assign schema as the following:

```hocon

schema {
fields {
code = int
data = string
success = boolean
}
}

```

connector will generate data as the following:

| code | data | success |
|------|-------------|---------|
| 200 | get success | true |

when you assign format is `text`, connector will do nothing for upstream data, for example:

upstream data is the following:

```json

{"code": 200, "data": "get success", "success": true}

```

connector will generate data as the following:

| content |
|---------|
| {"code": 200, "data": "get success", "success": true} |

### schema [Config]

#### fields [Config]

the schema fields of upstream data

### common options

Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details

## Example

```hocon
MyHours{
url = "https://api2.myhours.com/api/Projects/getAll"
email = "[email protected]"
password = "seatunnel"
schema {
fields {
name = string
archived = boolean
dateArchived = string
dateCreated = string
clientName = string
budgetAlertPercent = string
budgetType = int
totalTimeLogged = double
budgetValue = double
totalAmount = double
totalExpense = double
laborCost = double
totalCost = double
billableTimeLogged = double
totalBillableAmount = double
billable = boolean
roundType = int
roundInterval = int
budgetSpentPercentage = double
budgetTarget = int
budgetPeriodType = string
budgetSpent = string
id = string
}
}
}
```

## Changelog

### next version

- Add My Hours Source Connector
1 change: 1 addition & 0 deletions plugin-mapping.properties
Original file line number Diff line number Diff line change
Expand Up @@ -140,4 +140,5 @@ seatunnel.sink.S3File = connector-file-s3
seatunnel.source.Amazondynamodb = connector-amazondynamodb
seatunnel.sink.Amazondynamodb = connector-amazondynamodb
seatunnel.sink.StarRocks = connector-starrocks
seatunnel.source.MyHours = connector-http-myhours
seatunnel.sink.InfluxDB = connector-influxdb
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,11 @@ public <V> void onRetry(Attempt<V> attempt) {
.build();
}

public HttpResponse execute(String url, String method, Map<String, String> headers, Map<String, String> params) throws Exception {
public HttpResponse execute(String url, String method, Map<String, String> headers, Map<String, String> params, String body) throws Exception {
// convert method option to uppercase
method = method.toUpperCase(Locale.ROOT);
if (HttpPost.METHOD_NAME.equals(method)) {
return doPost(url, headers, params);
return doPost(url, headers, params, body);
}
if (HttpGet.METHOD_NAME.equals(method)) {
return doGet(url, headers, params);
Expand Down Expand Up @@ -243,6 +243,31 @@ public HttpResponse doPost(String url, Map<String, String> headers, String body)
return getResponse(httpPost);
}

/**
* Send a post request with request headers , request parameters and request body
*
* @param url request address
* @param headers request header map
* @param params request parameter map
* @param body request body
* @return http response result
* @throws Exception information
*/
public HttpResponse doPost(String url, Map<String, String> headers, Map<String, String> params, String body) throws Exception {
// create a new http get
HttpPost httpPost = new HttpPost(url);
// set default request config
httpPost.setConfig(REQUEST_CONFIG);
// set request header
addHeaders(httpPost, headers);
// set request params
addParameters(httpPost, params);
// add body in request
addBody(httpPost, body);
// return http response
return getResponse(httpPost);
}

/**
* Send a put request without request parameters
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@
@Data
@SuppressWarnings("MagicNumber")
public class HttpParameter implements Serializable {
private String url;
private String method;
private Map<String, String> headers;
private Map<String, String> params;
private String body;
private int pollIntervalMillis;
private int retry;
private int retryBackoffMultiplierMillis = 100;
private int retryBackoffMaxMillis = 10000;
protected String url;
protected String method;
protected Map<String, String> headers;
protected Map<String, String> params;
protected String body;
protected int pollIntervalMillis;
protected int retry;
protected int retryBackoffMultiplierMillis = 100;
protected int retryBackoffMaxMillis = 10000;

public void buildWithConfig(Config pluginConfig) {
// set url
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ public void prepare(Config pluginConfig) throws PrepareFailException {
throw new PrepareFailException(getPluginName(), PluginType.SOURCE, result.getMsg());
}
this.httpParameter.buildWithConfig(pluginConfig);
buildSchemaWithConfig(pluginConfig);
}

protected void buildSchemaWithConfig(Config pluginConfig) {
if (pluginConfig.hasPath(HttpConfig.SCHEMA)) {
Config schema = pluginConfig.getConfig(HttpConfig.SCHEMA);
this.rowType = SeaTunnelSchema.buildWithConfig(schema).getSeaTunnelRowType();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ public void close() throws IOException {
@Override
public void pollNext(Collector<SeaTunnelRow> output) throws Exception {
try {
HttpResponse response = httpClient.execute(this.httpParameter.getUrl(), this.httpParameter.getMethod(), this.httpParameter.getHeaders(), this.httpParameter.getParams());
HttpResponse response = httpClient.execute(this.httpParameter.getUrl(), this.httpParameter.getMethod(), this.httpParameter.getHeaders(), this.httpParameter.getParams(), this.httpParameter.getBody());
if (HttpResponse.STATUS_OK == response.getCode()) {
String content = response.getContent();
if (!Strings.isNullOrEmpty(content)) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--

Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>connector-http</artifactId>
<groupId>org.apache.seatunnel</groupId>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>connector-http-myhours</artifactId>

<dependencies>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>connector-http-base</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>

</project>
Loading