feat: extend to_layout options

[agoose77]

At present, to_layout eagerly converts anything it can recognise into an array. Yet, there are several places where we want to be stricter about what is supported and to be able to identify scalar inputs.

This PR extends the configurability of to_layout, to permit finer control of scalar input handling. It adds new parameters to ak.to_layout:

• allow_none — whether None values are permitted as inputs
• use_from_iter — whether iterable values are permitted as inputs
• primitive_policy — the policy governing handling of scalars, e.g. promotion, pass-through, error
• string_policy — the policy governing handling of strings, e.g. promotion, pass-through, as-characters, error
  I also removed the __cast__ default that casts str to length-1 arrays of strings.

[codecov]

Codecov Report

Merging #2763 (175c52f) into main (b333fe4) will decrease coverage by 0.04%.
Report is 2 commits behind head on main.
The diff coverage is 91.66%.

Additional details and impacted files

Files	Coverage Δ
src/awkward/_connect/numba/arrayview.py	93.37% <100.00%> (ø)
src/awkward/_connect/numexpr.py	90.66% <ø> (ø)
src/awkward/_nplikes/typetracer.py	76.58% <100.00%> (+0.12%)	⬆️
src/awkward/_slicing.py	86.86% <100.00%> (+0.26%)	⬆️
src/awkward/contents/content.py	76.25% <100.00%> (+0.09%)	⬆️
src/awkward/highlevel.py	76.53% <ø> (ø)
src/awkward/operations/ak_all.py	96.00% <100.00%> (ø)
src/awkward/operations/ak_any.py	96.00% <100.00%> (ø)
src/awkward/operations/ak_argcartesian.py	88.57% <ø> (ø)
src/awkward/operations/ak_argcombinations.py	86.95% <ø> (ø)
... and 67 more

[agoose77]

Annoyingly, I've run into the problem of specifying scalar-promotion rules again. We kicked the bucket down the road somewhat, but I think it's time to robustly solve this. At this stage, I think we can just replicate NumPy's rules for Python scalars, because Python scalars are expected to be non-kernel outputs (i.e. static).

However, if we have places where this doesn't hold, we might need to think about implementing NEP-50 (https://numpy.org/neps/nep-0050-scalar-promotion.html#motivation-and-scope) which proposes to remove the value-dependence that we're worried about in Awkward

[jpivarski]

I also removed the __cast__ default that casts str to length-1 arrays of strings.

This might be needed for some commonsense broadcasting cases. For instance,

some_array["new_field"] = "a single string"

may need to be presented as ak.Array(["a single string"]) so that broadcasting sees this as one atom to duplicate len(some_array) times.

[agoose77]

@jpivarski the failing test in 3.12 on windows is due to the change in scalar promotion; Awkward promotes scalars to the ArrayBuilder default type (int64) rather than the NumPy default for the platform. I don't think that is necessarily a problem, though I wonder whether ArrayBuilder should default to the NumPy default dtype for the current platform rather than hard-coding int64. Any strong feelings here? The real question, though, is are you OK with changing this behavior such that untyped scalars go via ArrayBuilder rather than using asarray?

[jpivarski]

We should be interpreting Python int as int64. That's a widespread convention; only NumPy on Windows doesn't do it and I think that's a mistake.

If the scalars are really untyped (i.e. Python int or float), they can go via ArrayBuilder, but NumPy-typed scalars (i.e. np.int32 and np.float64) would lose their types if they went through ArrayBuilder. ArrayBuilder takes any integer as int64 and any floating point number as float64; it just doesn't have other input types compiled into it (which would dramatically increase its surface area, since its number of methods has to be equal to the number of classes, so handling $n$ input types means $n^2$ method-class pairs).

[agoose77]

We should be interpreting Python int as int64. That's a widespread convention; only NumPy on Windows doesn't do it and I think that's a mistake.

Fine! Although, now that's also a list that includes Emscripten whose default is 32 bit.

If the scalars are really untyped (i.e. Python int or float), they can go via ArrayBuilder, but NumPy-typed scalars (i.e. np.int32 and np.float64) would lose their types if they went through ArrayBuilder.

Yes, the np.generic scalars are handled via np.asarray :)

[jpivarski]

Emscripten whose default is 32 bit

I ran into JavaScript being limited to 32-bit in the Kaitai web tool. That was quite a surprise.

Okay, there are two exceptions: NumPy in Windows and JavaScript/Emscripten.

[agoose77]

I just changed strings_as_characters to a new literal string_policy, and renamed scalar_policy to primitive_policy.

This allows finer control of strings, and brings some symmetry to the API

[jpivarski]

More granularity in the options is good.

Although a Python string is now by default turned into an array of characters, which is not good for broadcasting, the functions that use it set the argument to turn the Python string into a length-1 array of strings, so that nothing breaks.

We talked about how Python numbers are not always promoted to int64 and float64, but are promoted to whatever they need to be to not change the array they're combined with, which is what NumPy does, and since NumPy does it, we'll do it, too.

This allow_other has changed its meaning, so it should also change its name: it will become allow_unknown (Angus and I are on Zoom now). Then, if anybody is using allow_other, they'll be alerted to the change by a TypeError. When we scanned for usage in the Awkward-dependent repos, none of them are using ak.to_layout, so we don't need a deprecation cycle.

So one change is needed before merging: rename allow_other → allow_unknown. Other than that, it's done.