feat: add to_parquet_row_groups

[zbilodea]

No description provided.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f2a2340) 81.87% compared to head (14087c8) 81.89%.

Additional details and impacted files

Files	Coverage Δ
src/awkward/operations/__init__.py	100.00% <100.00%> (ø)
src/awkward/operations/ak_to_parquet.py	62.01% <100.00%> (+7.38%)	⬆️
src/awkward/operations/ak_to_parquet_row_groups.py	100.00% <100.00%> (ø)

[jpivarski]

You might not need to define the batch_iterator function. (Is it a copy of what was in Awkward 1.x? This is a different context.) You're already iterating over batches in ak_to_parquet.py lines 408‒435. I haven't looked closely enough at it to point out whether something is superfluous, but it looks like it might be. It's worth focusing on that a bit more.

I wish there was a way to write a test that would check to see that all the batches are not loaded into memory at once. Maybe you could try it on a manual test, outside of CI, with a dataset that is too large to fit in memory at once. Putting (temporary!) print statements in some places would also let you check the temporal order of things.

[jpivarski]

This is good in that it works, but the code is a bit repetitive and the control flow is hard to follow. I'll just make a few edits after this review: don't merge until I'm done.

[jpivarski]

This refactoring reduces three repetitions of layout = ... followed by table = ... into one by defining a temporary function, and it simplifies the while ... try ... except StopIteration logic with a for loop. It also adds guards to prevent users from making some likely mistakes.

The for loop takes advantage of the distinction between iterables (such as lists, array, and iterators) and iterators: when the first items in an iterator have been consumed with next, a for loop over the iterator does not include the already-consumed items—Python does not go back to the beginning.

>>> some_list = ["one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten"]
>>> next(some_list)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'list' object is not an iterator

>>> iterator = iter(some_list)
>>> next(iterator)
'one'
>>> next(iterator)
'two'
>>> next(iterator)
'three'
>>> for item in iterator:
...     print(item)
... 
four
five
six
seven
eight
nine
ten

but

>>> for item in some_list:
...     print(item)
... 
one
two
three
four
five
six
seven
eight
nine
ten

I think this PR is ready for merging, after doing an "Update branch." Look it over and decide if you want to make any counter-edits. If so, ping me again for another review. If not, go ahead and merge it.