Access metadata of flushed row groups on write (#1691)

[tustvold]

Which issue does this PR close?

Closes #1691

@pacman82 please let me know if this provides the necessary functionality

Rationale for this change

Provides information on data that has already been written, to allow for adaptive writing policies

What changes are included in this PR?

Adds the ability to fetch metadata about flushed row groups

Are there any user-facing changes?

No

[codecov-commenter]

Codecov Report

Merging #1774 (a3701af) into master (c1a91dc) will decrease coverage by 0.00%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##           master    #1774      +/-   ##
==========================================
- Coverage   83.39%   83.39%   -0.01%     
==========================================
  Files         198      198              
  Lines       56142    56212      +70     
==========================================
+ Hits        46821    46876      +55     
- Misses       9321     9336      +15     

Impacted Files	Coverage Δ
parquet/src/arrow/arrow_writer.rs	97.53% <0.00%> (-0.24%)	⬇️
parquet/src/file/metadata.rs	95.12% <ø> (ø)
parquet/src/schema/types.rs	83.73% <ø> (ø)
parquet/src/file/writer.rs	92.92% <100.00%> (+0.06%)	⬆️
parquet/src/util/cursor.rs	62.18% <0.00%> (-1.69%)	⬇️
parquet/src/file/serialized_reader.rs	94.46% <0.00%> (-1.17%)	⬇️
parquet_derive/src/parquet_field.rs	65.98% <0.00%> (-0.23%)	⬇️
parquet/src/encodings/encoding.rs	93.46% <0.00%> (-0.20%)	⬇️
arrow/src/ipc/reader.rs	90.73% <0.00%> (-0.11%)	⬇️
... and 16 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c1a91dc...a3701af. Read the comment docs.

[alamb]

I also recommend some sort of test that uses this API (maybe also demonstrating its use for adaptive writing)

[pacman82]

@tustvold For now getting the compressed_size of each row group after I've written it, did the trick for me. End to end my usecase is about creating files of roughly the same size, while streaming data from a database. My current solution is like this:

1. Accumulate the compressed size of each row group written
2. If sum of compressed size goes over a threshold, reset to zero and start writing the next row group into a new file.

Maybe this interface would help me simplify things? Or is there a way to be more "precise" in the resulting file size. Anyhow I do not see it at the moment.

Cheers, Markus

[tustvold]

Maybe this interface would help me simplify things

It would theoretically allow for more complex heuristics, but ultimately it is just a different way to access the data returned by row_group_writer.close()

Or is there a way to be more "precise" in the resulting file size

Not easily, it is very hard to predict the encoded size accurately, especially with block compression. The way this is handled internally for pages is writing in small batches, and tracking the size as they go. This is much the same as your approach for row groups. TLDR I think your approach sounds sensible

[alamb]

🎉