[Paimon]support projection for paimon source #6343

TaoZex · 2024-02-07T06:32:46Z

Purpose of this pull request

support projection for paimon source

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

...el-connector-v2-e2e/connector-paimon-e2e/src/test/resources/paimon_projection_to_assert.conf

# Conflicts: # seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonConfig.java

TaoZex · 2024-04-11T06:55:43Z

PTAL @Hisoka-X @EricJoy2048

Hisoka-X · 2024-04-11T07:02:12Z

cc @dailai

dailai · 2024-04-11T07:11:13Z

Please support projection by field name.

TaoZex · 2024-04-11T12:24:16Z

Please support projection by field name.

Fixed. Thanks.

dailai · 2024-04-12T00:18:29Z

LGTM

Hisoka-X · 2024-04-13T09:01:55Z

docs/en/connector-v2/source/Paimon.md

@@ -40,6 +41,10 @@ The table you want to access

 The file path of `hdfs-site.xml`

+### projection [string]


Suggested change

### projection [string]

### read_columns [string]

How about like other connectors, named read_columns https://seatunnel.apache.org/docs/connector-v2/source/LocalFile#read_columns-list

@Hisoka-X @hailin0 I have fixed it as suggested, please review it again.

Hisoka-X · 2024-06-04T12:39:13Z

Hi @TaoZex , could you solve the conflict?

TaoZex · 2024-06-04T12:43:32Z

Hi @TaoZex , could you solve the conflict?

Of course.

Hisoka-X · 2024-06-05T12:50:21Z

docs/en/connector-v2/source/Paimon.md

@@ -23,6 +23,7 @@ Read data from Apache Paimon.
 | database       | String | Yes      | -             |
 | table          | String | Yes      | -             |
 | hdfs_site_path | String | No       | -             |
+| projection     | String | No       | -             |


please change the document too

Fixed, thanks.

Carl-Zhou-CN · 2024-06-06T03:39:39Z

For #6887
There also seems to be an intention to cover the projection, at which point we will have two projection parameters,Do we need to keep both? @Hisoka-X @TaoZex @dailai

TaoZex · 2024-06-06T03:51:40Z

For #6887 There also seems to be an intention to cover the projection, at which point we will have two projection parameters,Do we need to keep both? @Hisoka-X @TaoZex @dailai

This pr implements the projection for paimon. After the pr was merged, #6887 can be further through SQL for the projection.

Carl-Zhou-CN · 2024-06-06T03:57:03Z

For #6887 There also seems to be an intention to cover the projection, at which point we will have two projection parameters,Do we need to keep both? @Hisoka-X @TaoZex @dailai

This pr implements the projection for paimon. After the pr was merged, #6887 can be further through SQL for the projection.

@TaoZex I think 'read_columns' can be overlaid in 'query sql'. Can we transform this pr based on sql？

TaoZex · 2024-06-07T01:29:05Z

For #6887 There also seems to be an intention to cover the projection, at which point we will have two projection parameters,Do we need to keep both? @Hisoka-X @TaoZex @dailai

This pr implements the projection for paimon. After the pr was merged, #6887 can be further through SQL for the projection.

@TaoZex I think 'read_columns' can be overlaid in 'query sql'. Can we transform this pr based on sql？

I had a discussion with @dailai on wechat. Currently his pr only supports select * sql queries. I will wait for his pr to merge, and modify this pr code to select id, name sql query (Assuming multiple fields, but only two fields are queried by projection)
cc @hailin0 @Hisoka-X

hailin0 · 2024-06-12T15:19:56Z

this pr #6887 is merged
@TaoZex

TaoZex · 2024-06-13T10:35:27Z

this pr #6887 is merged @TaoZex

I have changed the code. PTAL. @Hisoka-X @hailin0

hailin0 · 2024-06-13T15:54:01Z

cc @dailai

Hisoka-X

LGTM. wating @dailai

dailai

LGTM @hailin0

...he/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java

...ava/org/apache/seatunnel/connectors/seatunnel/paimon/source/PaimonSourceSplitEnumerator.java

TaoZex · 2024-06-14T11:40:09Z

LGTM @hailin0

Thanks for review. Done.

hailin0

LGTM

TaoZex added 2 commits February 7, 2024 14:16

[Paimon]support projection for paimon source

a6ffe4d

fix code style

9beb226

hailin0 reviewed Feb 18, 2024

View reviewed changes

...el-connector-v2-e2e/connector-paimon-e2e/src/test/resources/paimon_projection_to_assert.conf Outdated Show resolved Hide resolved

hailin0 added connectors-v2 paimon labels Feb 18, 2024

TaoZex added 2 commits April 11, 2024 03:57

Merge remote-tracking branch 'origin/dev' into hudi_sink

89c43a8

# Conflicts: # seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonConfig.java

fix

33d4c94

fix for field name

e6b26c5

fix doc

87f9bc4

dailai approved these changes Apr 12, 2024

View reviewed changes

github-actions bot added the reviewed label Apr 12, 2024

Hisoka-X reviewed Apr 13, 2024

View reviewed changes

dailai mentioned this pull request May 23, 2024

[Improve][Connector-V2] Improve the paimon source #6887

Merged

4 tasks

TaoZex added 3 commits June 5, 2024 14:15

Merge branch 'dev' into hudi_sink

5ee8bf8

fix ci

b86177d

fix

28f433f

Hisoka-X reviewed Jun 5, 2024

View reviewed changes

fix

60e4bbc

Hisoka-X previously approved these changes Jun 6, 2024

View reviewed changes

github-actions bot added the approved label Jun 6, 2024

Merge branch 'dev' into hudi_sink

ef2f3ec

TaoZex dismissed Hisoka-X’s stale review via ef2f3ec June 11, 2024 14:49

github-actions bot removed the approved label Jun 11, 2024

TaoZex added 4 commits June 12, 2024 18:57

support sql projection

631d445

fix doc

85e552d

fix

f7b8e04

fix

37864f5

TaoZex added 2 commits June 13, 2024 01:01

fix

ca615e0

fix

34af45b

Hisoka-X previously approved these changes Jun 14, 2024

View reviewed changes

github-actions bot added the approved label Jun 14, 2024

typo

ac94258

TaoZex dismissed Hisoka-X’s stale review via ac94258 June 14, 2024 06:48

github-actions bot removed the approved label Jun 14, 2024

dailai approved these changes Jun 14, 2024

View reviewed changes

fix by comment

8e5a1d3

TaoZex requested a review from hailin0 June 14, 2024 11:44

hailin0 approved these changes Jun 14, 2024

View reviewed changes

github-actions bot added the approved label Jun 14, 2024

Hisoka-X approved these changes Jun 14, 2024

View reviewed changes

hailin0 merged commit 6c15772 into apache:dev Jun 14, 2024
6 checks passed

chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024

[Paimon]support projection for paimon source (apache#6343)

5c70fd8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Paimon]support projection for paimon source #6343

[Paimon]support projection for paimon source #6343

TaoZex commented Feb 7, 2024

TaoZex commented Apr 11, 2024

Hisoka-X commented Apr 11, 2024

dailai commented Apr 11, 2024

TaoZex commented Apr 11, 2024 •

edited

Loading

dailai commented Apr 12, 2024

Hisoka-X Apr 13, 2024

TaoZex Jun 5, 2024

Hisoka-X commented Jun 4, 2024

TaoZex commented Jun 4, 2024

Hisoka-X Jun 5, 2024

TaoZex Jun 5, 2024

Carl-Zhou-CN commented Jun 6, 2024

TaoZex commented Jun 6, 2024

Carl-Zhou-CN commented Jun 6, 2024

TaoZex commented Jun 7, 2024 •

edited

Loading

hailin0 commented Jun 12, 2024

TaoZex commented Jun 13, 2024

hailin0 commented Jun 13, 2024

Hisoka-X left a comment

dailai left a comment

TaoZex commented Jun 14, 2024

hailin0 left a comment

		@@ -40,6 +41,10 @@ The table you want to access

		The file path of `hdfs-site.xml`

		### projection [string]

[Paimon]support projection for paimon source #6343

[Paimon]support projection for paimon source #6343

Conversation

TaoZex commented Feb 7, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

TaoZex commented Apr 11, 2024

Hisoka-X commented Apr 11, 2024

dailai commented Apr 11, 2024

TaoZex commented Apr 11, 2024 • edited Loading

dailai commented Apr 12, 2024

Hisoka-X Apr 13, 2024

Choose a reason for hiding this comment

TaoZex Jun 5, 2024

Choose a reason for hiding this comment

Hisoka-X commented Jun 4, 2024

TaoZex commented Jun 4, 2024

Hisoka-X Jun 5, 2024

Choose a reason for hiding this comment

TaoZex Jun 5, 2024

Choose a reason for hiding this comment

Carl-Zhou-CN commented Jun 6, 2024

TaoZex commented Jun 6, 2024

Carl-Zhou-CN commented Jun 6, 2024

TaoZex commented Jun 7, 2024 • edited Loading

hailin0 commented Jun 12, 2024

TaoZex commented Jun 13, 2024

hailin0 commented Jun 13, 2024

Hisoka-X left a comment

Choose a reason for hiding this comment

dailai left a comment

Choose a reason for hiding this comment

TaoZex commented Jun 14, 2024

hailin0 left a comment

Choose a reason for hiding this comment

TaoZex commented Apr 11, 2024 •

edited

Loading

TaoZex commented Jun 7, 2024 •

edited

Loading