Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improve][File-Connector] Support write fixed/timestamp as int96 of parquet #6971

Merged
merged 1 commit into from
Jun 13, 2024
Merged

[Improve][File-Connector] Support write fixed/timestamp as int96 of parquet #6971

merged 1 commit into from
Jun 13, 2024

Conversation

hailin0
Copy link
Member

@hailin0 hailin0 commented Jun 11, 2024

Purpose of this pull request

[File-Connector] Support write fixed/timestamp as int96 of parquet

Does this PR introduce any user-facing change?

Add new config items of sink

sink {
    LocalFile {
        ...
        parquet_avro_write_timestamp_as_int96 = true
        parquet_avro_write_fixed_as_int96 = ["f1", "f2", "f3"]
   }
}

How was this patch tested?

Added

Check list

@hailin0 hailin0 force-pushed the dev-support-parquet-int96 branch 2 times, most recently from bfeb060 to c8dd70f Compare June 12, 2024 01:53
Hisoka-X
Hisoka-X previously approved these changes Jun 12, 2024
View reviewed changes
wuchunfu
wuchunfu previously approved these changes Jun 12, 2024
View reviewed changes
TyrantLucifer
TyrantLucifer reviewed Jun 12, 2024
View reviewed changes
...n/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/ParquetReadStrategy.java Outdated
@@ -100,9 +98,6 @@ public void read(String path, String tableId, Collector<SeaTunnelRow> output)
HadoopInputFile.fromPath(filePath, configuration));
int fieldsCount = seaTunnelRowType.getTotalFields();
GenericData dataModel = new GenericData();
dataModel.addLogicalTypeConversion(new Conversions.DecimalConversion());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why delete this? This will cause abnormal data to be read out.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this was a mistake.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverted

TyrantLucifer
TyrantLucifer previously requested changes Jun 12, 2024
View reviewed changes
Copy link
Member

@TyrantLucifer TyrantLucifer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix CI.

image image

@hailin0 hailin0 dismissed stale reviews from wuchunfu and Hisoka-X via d99e48e June 12, 2024 02:45
@hailin0 hailin0 force-pushed the dev-support-parquet-int96 branch from c8dd70f to d99e48e Compare June 12, 2024 02:45
@hailin0 hailin0 closed this Jun 12, 2024
@hailin0 hailin0 reopened this Jun 12, 2024
@hailin0 hailin0 force-pushed the dev-support-parquet-int96 branch 2 times, most recently from fae64ec to c11f51a Compare June 12, 2024 05:59
@hailin0 hailin0 force-pushed the dev-support-parquet-int96 branch from c11f51a to e4affb9 Compare June 12, 2024 06:27
@hailin0
Copy link
Member Author

hailin0 commented Jun 12, 2024

cc @Hisoka-X @TyrantLucifer @wuchunfu ci is passed

@Hisoka-X Hisoka-X requested a review from TyrantLucifer June 12, 2024 08:50
@EricJoy2048 EricJoy2048 merged commit 1a48a9c into apache:dev Jun 13, 2024
7 checks passed
chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024
@hailin0
[Improve][Files] Support write fixed/timestamp as int96 of parquet (a…
1bb0179 
…pache#6971)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hisoka-X Hisoka-X Hisoka-X approved these changes

@EricJoy2048 EricJoy2048 EricJoy2048 approved these changes

@wuchunfu wuchunfu wuchunfu left review comments

@TyrantLucifer TyrantLucifer Awaiting requested review from TyrantLucifer

Assignees
No one assigned
Labels
approved file improve reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@hailin0 @wuchunfu @EricJoy2048 @Hisoka-X @TyrantLucifer