[Improve][Connector-V2][Iceberg] Add hadoop s3 catalog e2e testcase #5745

4chicat · 2023-10-27T13:29:01Z

Purpose of this pull request

We use S3 as iceberg warehouse storage.
If catalog_type is hadoop, only hdfs is supported, so it is being customized and used.
I want to contribute my customized code, so I push a pull request.

Does this PR introduce any user-facing change?

we can use the iceberg table saved in s3.

config example

source {
  iceberg {
    catalog_name = "seatunnel"
    iceberg.catalog.config={
      "type"="hadoop"
      "warehouse"="s3a://your_bucket/spark/warehouse/"
    }
    hadoop.config={
      "fs.s3a.path.style.access" = "true"
      "fs.s3a.connection.ssl.enabled" = "false"
      "fs.s3a.signing.algorithm" = "S3SignerType"
      "fs.s3a.encryption.algorithm" = "AES256"
      "fs.s3a.connection.timeout" = "3000"
      "fs.s3a.aws.credentials.provider" = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
      "fs.s3a.endpoint" = "http://minio:9000"
      "fs.s3a.access.key" = "xxx"
      "fs.s3a.secret.key" = "xxx"
      "fs.defaultFS" = "s3a://xxx"
    }
    namespace = "your_iceberg_database"
    table = "your_iceberg_table"
    result_table_name = "iceberg_test"
  }
}

How was this patch tested?

add e2e test case : connector-iceberg-s3-e2e

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

Hisoka-X · 2023-10-30T03:56:48Z

cc @ic4y

EricJoy2048

LGTM, cc @ic4y

Hisoka-X

Shall we add a test case for this feature?

4chicat · 2023-10-31T03:58:54Z

Shall we add a test case for this feature?

I don't think it's going to be easy because it requires an S3 bucket.

Hisoka-X · 2023-10-31T04:05:21Z

I think we can create a minio container in e2e test case for s3a file read.

4chicat · 2023-11-04T12:25:01Z

@EricJoy2048
Thank you so much for the review.  
I made an additional commit today by adding the e2e test case.
However, your previous review was automatically dismissed by the system.
I apologize for that.

4chicat · 2023-11-11T03:29:36Z

I think we can create a minio container in e2e test case for s3a file read.

I created minio test container and added e2e test case.
It passes on spark3, but fails on flink/seatunnel/spark2.

I think I need to add hadoop-aws/aws-java-sdk jar file to the test container docker image and need your help.

spark2 : hadoop-aws-2.7.5, aws-java-sdk-1.7.4

I'm also looking to see if there are other workarounds.

Hisoka-X · 2023-11-11T03:37:52Z

Thanks @4chicat , I will take a look in the few days.

liugddx · 2024-02-05T08:21:14Z

...-e2e/seatunnel-connector-v2-e2e/connector-iceberg-s3-e2e/src/test/resources/log4j.properties

Why add this file

connector-iceberg-s3-e2e e2e case was written based on connector-iceberg-hadoop3-e2e.
This file was created by copying the file below.
https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-hadoop3-e2e/src/test/resources/log4j.properties

Should I delete it?

connector-iceberg-s3-e2e e2e case was written based on connector-iceberg-hadoop3-e2e. This file was created by copying the file below. https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-hadoop3-e2e/src/test/resources/log4j.properties

Should I delete it?

It is already included in the engine and can be removed

connector-iceberg-s3-e2e e2e case was written based on connector-iceberg-hadoop3-e2e. This file was created by copying the file below. https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-hadoop3-e2e/src/test/resources/log4j.properties
Should I delete it?

It is already included in the engine and can be removed

Removed log4j.properties file.

hailin0

LGTM

cc @Hisoka-X

Hisoka-X · 2024-02-19T02:31:39Z

Look like this PR bring new leak thread in seatunnel server. Can you try closing these leaking threads? If not, please add it to the known issues list.

seatunnel/seatunnel-e2e/seatunnel-e2e-common/src/test/java/org/apache/seatunnel/e2e/common/container/seatunnel/SeaTunnelContainer.java

Line 291 in 92f847c

private boolean isIssueWeAlreadyKnow(String threadName) {

Hisoka-X · 2024-02-19T08:14:33Z

...e/seatunnel-e2e-common/src/test/java/org/apache/seatunnel/e2e/common/util/ContainerUtil.java

+    public static List<TestContainer> discoverTestSpark3Containers() {
+        return discoverTestContainers().stream()
+                .filter(testContainer -> testContainer instanceof Spark3Container)
+                .collect(Collectors.toList());
+    }


please remove

It has been removed.

Hisoka-X · 2024-02-19T08:15:18Z

...on/src/test/java/org/apache/seatunnel/e2e/common/container/seatunnel/SeaTunnelContainer.java

+                                || s.contains("java-sdk-http-connection-reaper")
+                                || s.contains("Timer for 's3a-file-system' metrics system")
+                                || s.startsWith("MutableQuantiles-")


please move it into https://github.com/apache/seatunnel/pull/5745/files#diff-828bb95593582c2297e60ef9a6ced4733f251eef77389235d8c3b506dd0beb47L291

Moved to isIssueWeAlreadyKnow method.

EricJoy2048 · 2024-04-07T12:49:44Z

This PR is very valuable and I think we need to look at it again and review it.

Carl-Zhou-CN · 2024-04-08T01:22:43Z

...berg-s3-e2e/src/test/java/org/apache/seatunnel/e2e/connector/iceberg/s3/IcebergSourceIT.java

+@DisabledOnContainer(
+        value = {TestContainerId.SPARK_2_4},
+        type = {EngineType.FLINK},
+        disabledReason = "Needs hadoop-aws,aws-java-sdk jar for flink and spark2.4.")


Could you please help me annotate this section in the usage document and provide an example for AWS S3？

Thanks for your comment. Added to document.

Carl-Zhou-CN · 2024-04-10T01:22:09Z

...on/src/test/java/org/apache/seatunnel/e2e/common/container/seatunnel/SeaTunnelContainer.java

+                // Iceberg S3 Hadoop catalog
+                || threadName.contains("java-sdk-http-connection-reaper")
+                || threadName.contains("Timer for 's3a-file-system' metrics system")
+                || threadName.startsWith("MutableQuantiles-");


@Hisoka-X cc

Hisoka-X · 2024-04-10T02:50:50Z

seatunnel-shade/seatunnel-hadoop3-3.1.4-uber/pom.xml

+        <dependency>
+            <groupId>org.apache.hadoop</groupId>
+            <artifactId>hadoop-aws</artifactId>
+            <version>${hadoop3.version}</version>
+        </dependency>


cc @EricJoy2048

Why add hadoop-aws to hadoop shade jar?

Changed to not add hadoop-aws.

EricJoy2048 · 2024-04-30T03:25:54Z

I think we can create a minio container in e2e test case for s3a file read.

I created minio test container and added e2e test case. It passes on spark3, but fails on flink/seatunnel/spark2.

I think I need to add hadoop-aws/aws-java-sdk jar file to the test container docker image and need your help.

spark2 : hadoop-aws-2.7.5, aws-java-sdk-1.7.4

I'm also looking to see if there are other workarounds.

Like the driver, you can add hadoop-aws and aws-java-sdk to the test container docker like seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-oss-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/oss/OssFileIT.java

@TestContainerExtension
    private final ContainerExtendedFactory extendedFactory =
            container -> {
                Container.ExecResult extraCommands =
                        container.execInContainer(
                                "bash",
                                "-c",
                                "mkdir -p /tmp/seatunnel/plugins/oss/lib && cd /tmp/seatunnel/plugins/oss/lib && curl -O "
                                        + OSS_SDK_DOWNLOAD);
                Assertions.assertEquals(0, extraCommands.getExitCode());

                extraCommands =
                        container.execInContainer(
                                "bash",
                                "-c",
                                "cd /tmp/seatunnel/plugins/oss/lib && curl -O " + JDOM_DOWNLOAD);
                Assertions.assertEquals(0, extraCommands.getExitCode());

                extraCommands =
                        container.execInContainer(
                                "bash",
                                "-c",
                                "cd /tmp/seatunnel/plugins/oss/lib && curl -O "
                                        + HADOOP_ALIYUN_DOWNLOAD);
                Assertions.assertEquals(0, extraCommands.getExitCode());

                extraCommands =
                        container.execInContainer(
                                "bash",
                                "-c",
                                "cd /tmp/seatunnel/lib && curl -O " + OSS_SDK_DOWNLOAD);
                Assertions.assertEquals(0, extraCommands.getExitCode());

                extraCommands =
                        container.execInContainer(
                                "bash", "-c", "cd /tmp/seatunnel/lib && curl -O " + JDOM_DOWNLOAD);
                Assertions.assertEquals(0, extraCommands.getExitCode());

                extraCommands =
                        container.execInContainer(
                                "bash",
                                "-c",
                                "cd /tmp/seatunnel/lib && curl -O " + HADOOP_ALIYUN_DOWNLOAD);
                Assertions.assertEquals(0, extraCommands.getExitCode());
            };

[Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # bugfix [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # add e2e test case [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # add @DisabledOnContainer [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # remove log4j.properties [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # add removeSystemThread [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # move and remove code [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # resolve conflict [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # mvn spotless:apply [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # modify IcebergSourceIT [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # modify doc iceberg source [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # copy jars to test container docker

…isabled seatunnel engine

hailin0

LGTM

…pache#5745)

4chicat force-pushed the support-s3-warehouse branch 2 times, most recently from 931af66 to b1c6e40 Compare October 27, 2023 14:13

4chicat changed the title ~~[Improve][Connector-V2][Iceberg] Support for S3File in hadoop catalog~~ [Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog Oct 27, 2023

4chicat force-pushed the support-s3-warehouse branch from b1c6e40 to 7ceedd0 Compare October 27, 2023 14:18

Hisoka-X added improve connectors-v2 labels Oct 30, 2023

Hisoka-X added the First-time contributor First-time contributor label Oct 30, 2023

EricJoy2048 reviewed Oct 30, 2023

View reviewed changes

EricJoy2048 previously approved these changes Oct 30, 2023

View reviewed changes

github-actions bot added approved reviewed labels Oct 30, 2023

Hisoka-X reviewed Oct 31, 2023

View reviewed changes

Hisoka-X added the need add test case label Oct 31, 2023

Hisoka-X removed the need add test case label Oct 31, 2023

Hisoka-X added the need add test case label Nov 1, 2023

4chicat dismissed EricJoy2048’s stale review via d82a841 November 4, 2023 12:14

github-actions bot removed approved reviewed labels Nov 4, 2023

4chicat force-pushed the support-s3-warehouse branch 4 times, most recently from 21f2ce6 to 2dfb53a Compare November 10, 2023 15:16

4chicat force-pushed the support-s3-warehouse branch 2 times, most recently from 98537cc to 1e195be Compare December 8, 2023 00:28

4chicat force-pushed the support-s3-warehouse branch from 1e195be to 4cdebc3 Compare February 5, 2024 08:14

liugddx reviewed Feb 5, 2024

View reviewed changes

4chicat requested a review from liugddx February 5, 2024 08:37

hailin0 reviewed Feb 18, 2024

View reviewed changes

4chicat force-pushed the support-s3-warehouse branch from d6536b1 to 2fb59ec Compare February 19, 2024 07:55

Hisoka-X reviewed Feb 19, 2024

View reviewed changes

4chicat force-pushed the support-s3-warehouse branch 2 times, most recently from 2078dff to cedd449 Compare February 20, 2024 13:45

4chicat force-pushed the support-s3-warehouse branch from cedd449 to 97222fc Compare March 8, 2024 01:32

4chicat requested review from Hisoka-X and hailin0 March 8, 2024 04:10

4chicat force-pushed the support-s3-warehouse branch from 97222fc to 861a07c Compare April 8, 2024 00:45

Carl-Zhou-CN reviewed Apr 8, 2024

View reviewed changes

4chicat requested a review from Carl-Zhou-CN April 8, 2024 02:10

Carl-Zhou-CN reviewed Apr 10, 2024

View reviewed changes

Hisoka-X reviewed Apr 10, 2024

View reviewed changes

4chicat changed the title ~~[Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog~~ [Improve][Connector-V2][Iceberg] Add hadoop s3 catalog e2e testcase May 1, 2024

4chicat force-pushed the support-s3-warehouse branch from d321ae0 to 1a89e8e Compare May 1, 2024 08:39

[Improve][Connector-V2][Iceberg] Support for S3 in hadoop catalog # d…

fe14b3a

…isabled seatunnel engine

4chicat force-pushed the support-s3-warehouse branch from 6624f28 to fe14b3a Compare May 8, 2024 10:35

EricJoy2048 approved these changes Jun 14, 2024

View reviewed changes

github-actions bot added approved reviewed labels Jun 14, 2024

hailin0 approved these changes Jun 15, 2024

View reviewed changes

hailin0 merged commit 00a7014 into apache:dev Jun 15, 2024
6 checks passed

chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024

[Improve][Connector-V2][Iceberg] Add hadoop s3 catalog e2e testcase (a…

29bb996

…pache#5745)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improve][Connector-V2][Iceberg] Add hadoop s3 catalog e2e testcase #5745

[Improve][Connector-V2][Iceberg] Add hadoop s3 catalog e2e testcase #5745

4chicat commented Oct 27, 2023 •

edited

Loading

Hisoka-X commented Oct 30, 2023

EricJoy2048 left a comment

Hisoka-X left a comment

4chicat commented Oct 31, 2023

Hisoka-X commented Oct 31, 2023

4chicat commented Nov 4, 2023

4chicat commented Nov 11, 2023

Hisoka-X commented Nov 11, 2023

liugddx Feb 5, 2024

4chicat Feb 5, 2024

liugddx Feb 5, 2024

4chicat Feb 5, 2024

hailin0 left a comment •

edited

Loading

Hisoka-X commented Feb 19, 2024

Hisoka-X Feb 19, 2024

4chicat Feb 19, 2024

Hisoka-X Feb 19, 2024

4chicat Feb 19, 2024 •

edited

Loading

EricJoy2048 commented Apr 7, 2024

Carl-Zhou-CN Apr 8, 2024

4chicat Apr 8, 2024

Carl-Zhou-CN Apr 10, 2024

Hisoka-X Jun 15, 2024

Hisoka-X Apr 10, 2024

EricJoy2048 Apr 30, 2024

4chicat May 1, 2024

EricJoy2048 commented Apr 30, 2024

hailin0 left a comment

[Improve][Connector-V2][Iceberg] Add hadoop s3 catalog e2e testcase #5745

[Improve][Connector-V2][Iceberg] Add hadoop s3 catalog e2e testcase #5745

Conversation

4chicat commented Oct 27, 2023 • edited Loading

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X commented Oct 30, 2023

EricJoy2048 left a comment

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

4chicat commented Oct 31, 2023

Hisoka-X commented Oct 31, 2023

4chicat commented Nov 4, 2023

4chicat commented Nov 11, 2023

Hisoka-X commented Nov 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 left a comment • edited Loading

Choose a reason for hiding this comment

Hisoka-X commented Feb 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

4chicat Feb 19, 2024 • edited Loading

Choose a reason for hiding this comment

EricJoy2048 commented Apr 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricJoy2048 commented Apr 30, 2024

hailin0 left a comment

Choose a reason for hiding this comment

4chicat commented Oct 27, 2023 •

edited

Loading

hailin0 left a comment •

edited

Loading

4chicat Feb 19, 2024 •

edited

Loading