Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Apache Parquet 1.14.0 RC1 #21802

Closed
wants to merge 1 commit into from

Conversation

Fokko
Copy link
Contributor

@Fokko Fokko commented May 2, 2024

Description

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label May 2, 2024
@wendigo
Copy link
Contributor

wendigo commented May 2, 2024

Hudi seems related:

Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.lang.UnsupportedOperationException: Override ParquetWriter$Builder#getWriteSupport(ParquetConfiguration)
	at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:71)
	at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:37)
	at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
	... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.lang.UnsupportedOperationException: Override ParquetWriter$Builder#getWriteSupport(ParquetConfiguration)
	at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:75)
	at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:67)
	... 24 more
Caused by: java.lang.UnsupportedOperationException: Override ParquetWriter$Builder#getWriteSupport(ParquetConfiguration)
	at org.apache.parquet.hadoop.ParquetWriter$Builder.getWriteSupport(ParquetWriter.java:506)
	at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:930)
	at org.apache.hudi.io.storage.HoodieBaseParquetWriter.<init>(HoodieBaseParquetWriter.java:80)
	at org.apache.hudi.io.storage.HoodieAvroParquetWriter.<init>(HoodieAvroParquetWriter.java:54)
	at org.apache.hudi.io.storage.HoodieAvroFileWriterFactory.newParquetFileWriter(HoodieAvroFileWriterFactory.java:69)
	at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriterByFormat(HoodieFileWriterFactory.java:80)
	at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:67)
	at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:104)
	at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:76)
	at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:45)
	at org.apache.hudi.execution.CopyOnWriteInsertHandler.consume(CopyOnWriteInsertHandler.java:101)
	at org.apache.hudi.execution.CopyOnWriteInsertHandler.consume(CopyOnWriteInsertHandler.java:44)
	at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:69)
	... 25 more

@Fokko
Copy link
Contributor Author

Fokko commented May 2, 2024

@wendigo That's indeed a problem. I also raised this at the Parquet devlist: https://lists.apache.org/thread/nlfqqnc4jzdkzp49k7pj1fddxxo0rfrl

Iceberg is not affected in this PR since it shadows Parquet, but has the same issue: apache/iceberg#10209

@findinpath findinpath requested a review from raunaqmorarka May 3, 2024 09:06
@Fokko Fokko changed the title Test Apache Parquet 1.14.0 RC0 Test Apache Parquet 1.14.0 RC1 May 4, 2024
@Fokko
Copy link
Contributor Author

Fokko commented May 4, 2024

Testing out RC1 which should fix the issue 👍

@Fokko Fokko force-pushed the fd-test-parquet-1-14 branch from d3d2cfc to 5fcc532 Compare May 4, 2024 14:22
@wendigo
Copy link
Contributor

wendigo commented May 8, 2024

Superseded by #21853

@wendigo wendigo closed this May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

2 participants