Skip to content

Commit

Permalink
[Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # m…
Browse files Browse the repository at this point in the history
…odify doc iceberg source
  • Loading branch information
4chicat committed Apr 8, 2024
1 parent 861a07c commit d321ae0
Showing 1 changed file with 25 additions and 1 deletion.
26 changes: 25 additions & 1 deletion docs/en/connector-v2/source/Iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Source connector for Apache Iceberg. It can support batch and stream mode.

## Database Dependency

> In order to be compatible with different versions of Hadoop and Hive, the scope of hive-exec in the project pom file are provided, so if you use the Flink engine, first you may need to add the following Jar packages to <FLINK_HOME>/lib directory, if you are using the Spark engine and integrated with Hadoop, then you do not need to add the following Jar packages.
> In order to be compatible with different versions of Hadoop and Hive, the scope of hive-exec in the project pom file are provided, so if you use the Flink engine, first you may need to add the following Jar packages to <FLINK_HOME>/lib directory, if you are using the Spark engine and integrated with Hadoop, then you do not need to add the following Jar packages. If you are using the hadoop s3 catalog, you need to add the hadoop-aws,aws-java-sdk jars for your Flink and Spark engine versions. (Additional locations: <FLINK_HOME>/lib, <SPARK_HOME>/jars)
```
hive-exec-xxx.jar
Expand Down Expand Up @@ -141,6 +141,30 @@ sink {
}
```

### Hadoop S3 Catalog:

```hocon
source {
iceberg {
catalog_name = "seatunnel"
iceberg.catalog.config={
"type"="hadoop"
"warehouse"="s3a://your_bucket/spark/warehouse/"
}
hadoop.config={
"fs.s3a.aws.credentials.provider" = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
"fs.s3a.endpoint" = "s3.cn-north-1.amazonaws.com.cn"
"fs.s3a.access.key" = "xxxxxxxxxxxxxxxxx"
"fs.s3a.secret.key" = "xxxxxxxxxxxxxxxxx"
"fs.defaultFS" = "s3a://your_bucket"
}
namespace = "your_iceberg_database"
table = "your_iceberg_table"
result_table_name = "iceberg_test"
}
}
```

### Hive Catalog:

```hocon
Expand Down

0 comments on commit d321ae0

Please sign in to comment.