fix: support conversion to arrow and back with non-option Unknown type

[tcawlfield]

When we convert an Awkward array of type Unknown, this becomes
an arrow array (even prior to this) with zero length, nullable-null type,
but with Awkward metadata indicating no mask (non-option-type).
I believe in this case we can be sure that the original Awkward array was an EmptyArray.

[jpivarski]

The only really important issue is on emptyarray.py new line 376: you're setting is_nonnullable_nulltype=True for every EmptyArray.

If it's originally a non-optional unknowntype, you want to wrap it in UnmaskedArray and set is_nonnullable_nulltype=True so that when it gets read back from Arrow, the fake UnmaskedArray gets taken off again.

If it's originally a ?unknowntype (the ? is option-type), you want to not wrap it in UnmaskedArray (it's already in an option-tyep) and set is_nonnullable_nulltype=True so that when it gets read back from Arrow, nothing gets taken off again.

To ensure that this behavior gets tested, you should have some originally ?unknowntype examples. You're currently testing

• [] of unknowntype
• [[], [], []] of var * unknowntype

Add to that

• [] of ?unknowntype: you can make it with ak.Array([None])[0:0]
• [[], [], []] of var * ?unknowntype: you can make it with ak.Array([[], [], [], [None]])[0:3].

The rest of my comments are minor coding style preferences.

[jpivarski]

This is done! I checked the code carefully and I tested a few manually constructed arrays:

>>> import numpy as np
>>> import awkward as ak
>>> from awkward.contents import EmptyArray, UnmaskedArray, ListOffsetArray
>>> from awkward.index import Index64
>>>
>>> ak.from_arrow(ak.to_arrow(EmptyArray(), extensionarray=False), highlevel=False)
<IndexedOptionArray len='0'>
    <index><Index dtype='int64' len='0'>[]</Index></index>
    <content><EmptyArray len='0'/></content>
</IndexedOptionArray>
>>>
>>> ak.from_arrow(ak.to_arrow(EmptyArray(), extensionarray=True), highlevel=False)
<EmptyArray len='0'/>

That's good: with extensionarray=False, it can't round-trip: the EmptyArray comes back as an option-type (IndexedOptionArray) EmptyArray, but that's because it has nowhere to store the metadata. Meanwhile, the extensionarray=True case does round-trip: it's using that metadata.

Same thing inside of a list:

>>> ak.from_arrow(ak.to_arrow(ListOffsetArray(Index64(np.array([0])), EmptyArray()), extensionarray=False), highlevel=False)
<ListOffsetArray len='0'>
    <offsets><Index dtype='int64' len='1'>[0]</Index></offsets>
    <content><IndexedOptionArray len='0'>
        <index><Index dtype='int64' len='0'>
            []
        </Index></index>
        <content><EmptyArray len='0'/></content>
    </IndexedOptionArray></content>
</ListOffsetArray>
>>>
>>> ak.from_arrow(ak.to_arrow(ListOffsetArray(Index64(np.array([0])), EmptyArray()), extensionarray=True), highlevel=False)
<ListOffsetArray len='0'>
    <offsets><Index dtype='int64' len='1'>[0]</Index></offsets>
    <content><EmptyArray len='0'/></content>
</ListOffsetArray>

Without an extensionarray, it can't store the metadata to know that the EmptyArray originally wasn't wrapped in any option-type node, but with an extensionarray, it knows that and reproduces the original type.

Now if we wrap the EmptyArray in an option-type (UnmaskedArray), it always comes back with the option-type, regardless of the extensionarray setting.

>>> ak.from_arrow(ak.to_arrow(UnmaskedArray(EmptyArray()), extensionarray=False), highlevel=False)
<IndexedOptionArray len='0'>
    <index><Index dtype='int64' len='0'>[]</Index></index>
    <content><EmptyArray len='0'/></content>
</IndexedOptionArray>
>>>
>>> ak.from_arrow(ak.to_arrow(UnmaskedArray(EmptyArray()), extensionarray=True), highlevel=False)
<IndexedOptionArray len='0'>
    <index><Index dtype='int64' len='0'>[]</Index></index>
    <content><EmptyArray len='0'/></content>
</IndexedOptionArray>
>>>
>>> ak.from_arrow(ak.to_arrow(ListOffsetArray(Index64(np.array([0])), UnmaskedArray(EmptyArray())), extensionarray=False), highlevel=False)
<ListOffsetArray len='0'>
    <offsets><Index dtype='int64' len='1'>[0]</Index></offsets>
    <content><IndexedOptionArray len='0'>
        <index><Index dtype='int64' len='0'>
            []
        </Index></index>
        <content><EmptyArray len='0'/></content>
    </IndexedOptionArray></content>
</ListOffsetArray>
>>>
>>> ak.from_arrow(ak.to_arrow(ListOffsetArray(Index64(np.array([0])), UnmaskedArray(EmptyArray())), extensionarray=True), highlevel=False)
<ListOffsetArray len='0'>
    <offsets><Index dtype='int64' len='1'>[0]</Index></offsets>
    <content><IndexedOptionArray len='0'>
        <index><Index dtype='int64' len='0'>
            []
        </Index></index>
        <content><EmptyArray len='0'/></content>
    </IndexedOptionArray></content>
</ListOffsetArray>

The unit tests test these cases as well, but not explicitly with UnmaskedArray. (Constructing it from Python lists makes the option-type with IndexedOptionArray, but the code isn't hard-coded to any particular node type, just the option-ness.)

So I think this is ready to merge!