Change native parquet writer to write v1 parquet files

[joshthoward]

This PR instead adding a toggle as in #9497, this PR has the native parquet writer write v1 files by default.

It's a longer discussion to determine if we want to support both v1 and v2; this just fixes the known bugs.

fixes #6377

[findepi]

i rerun CI with tests:hive.

[alexjo2144]

There are V2 headers and statistics written in https://github.com/trinodb/trino/blob/master/lib/trino-parquet/src/main/java/io/trino/parquet/writer/PrimitiveColumnWriter.java

I'd assume those also need to be changed?

[joshthoward]

@alexjo2144 If you mean specifically this, I mentioned in #9497 that Spark fails to read a V1 header. Statistics are the same between V1 and V2 according to https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift.

[alexjo2144]

Spark is probably failing to read the V1 header because something else needs to be updated. There's a comment here implying that this change is not enough from @anjalinorwood #7953 (comment)

Maybe she or @rdblue can point out the spec differences?

There's also the EncodingStats I added this week which uses DATA_PAGE_V2 to indicate v2 pages: https://github.com/trinodb/trino/blob/master/lib/trino-parquet/src/main/java/io/trino/parquet/writer/PrimitiveColumnWriter.java#L187

V1 also included the rlEncoding and dlEncoding in the ColumnChunkMetaData encodings set, may be why Spark didn't read the data correctly.

[alexjo2144]

Comparing the differences between writeDataPage and writeDataPageV2 in the parquet-mr implementation shows that change with the encodings, and potentially more. I haven't taken that close of a look yet: https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L508

per Alex's comments

[losipiuk]

Looks valid. Though I did not follow deeply discussion which led it this stage, so I not comfortable ✅ ing it

[findepi]

skimmed. lgtm but i'm not a Parquet expert.

[findepi]

@alexjo2144 @martint @rdblue would you like to take a look?

[joshthoward]

I'm adding the release blocker label so that this gets actually merged prior to the release.

[findepi]

Thanks @alexjo2144 @martint for the review.

@joshthoward it cannot be considered a release blocker.