fix: Make output dtype known for list.to_struct when fields are passed

[nameexhaustion]

If we receive a list of names in the fields parameter, we will be able to return a dtype in to_field (or if we have a non-zero upper_bound)

Also refactors to define list.to_struct to be a ListFunction in the IR.

#                  Output for Polars 1.11.0 | Output for this branch
#                                           |
print(lf.select(pl.first().list.to_struct(fields=["a", "b"])).collect_schema())
#                 Schema({'a': Struct({})}) | Schema({'a': Struct({'a': Int64, 'b': Int64})})
#                                           |
print(lf.select(pl.first().list.to_struct(fields=str, upper_bound=1)).collect_schema())
# Schema({'a': Struct({'field_0': Int64})}) | Schema({'a': Struct({'0': Int64})})
#                                           |
print(lf.select(pl.first().list.to_struct(upper_bound=1)).collect_schema())
# Schema({'a': Struct({'field_0': Int64})}) | Schema({'a': Struct({'field_0': Int64})})
#                                           |
# Schema changed Unknown instead of Struct({}) when ambiguous:
print(lf.select(pl.first().list.to_struct(fields=str)).collect_schema())
#                 Schema({'a': Struct({})}) | Schema({'a': Unknown})
#                                           |
print(lf.select(pl.first().list.to_struct()).collect_schema())
#                 Schema({'a': Struct({})}) | Schema({'a': Unknown})
# Bugfix - upper_bound was ignored in execution
print(lf.select(pl.first().list.to_struct(upper_bound=1)).collect())
#                            shape: (2, 1)  | shape: (2, 1)
#                            ┌────────────┐ | ┌───────────┐
#                            │ a          │ | │ a         │
#                            │ ---        │ | │ ---       │
#                            │ struct[3]  │ | │ struct[1] │
#                            ╞════════════╡ | ╞═══════════╡
#                            │ {1,2,3}    │ | │ {1}       │
#                            │ {4,5,null} │ | │ {4}       │
#                            └────────────┘ | └───────────┘

[nameexhaustion]

upper_bound was previously ignored when width was being inferred during execution - it was only used during IR resolving for getting the output names

[nameexhaustion]

drive-by - outdated todo

[nameexhaustion]

We have this functionality where you can pass a Python function that gets called with a field index to generate names, but I'm not sure it's needed since you can just pass a list of field names. Maybe we should remove it?

[ritchie46]

It allows you to create fields at runtime. Which is a bit dangerous as it will be dependent of the data. But I think we should keep for now.

I want to add a return_dtype: DataType argument as well that allows you to specify the datatype immediately.

[ritchie46]

Can we add a Notes warning that it is a bug in a users query if they don't set fields or return_dtype? Then we can make it required in 2.0.

[nameexhaustion]

I want to add a return_dtype: DataType argument as well that allows you to specify the datatype immediately.

I think we can already do this by passing a list of names to the fields parameter. For the function case, we also get the output schema if the upper_bound is set - I can add a warning if we get a function without upper_bound being set?

[nameexhaustion]

Actually, I think I want to add the warnings/notes in a separate PR

[ritchie46]

Nice! If a user doesn't provide any fields/dtype. We must set the output type to Unknown, not a Struct({}).

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 30 lines in your changes missing coverage. Please review.

Project coverage is 79.95%. Comparing base (7a2f547) to head (eed6f03).
Report is 19 commits behind head on main.

Files with missing lines	Patch %	Lines
...tes/polars-ops/src/chunked_array/list/to_struct.rs	75.83%	29 Missing ⚠️
crates/polars-plan/src/dsl/function_expr/list.rs	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #19439      +/-   ##
==========================================
+ Coverage   79.94%   79.95%   +0.01%     
==========================================
  Files        1534     1534              
  Lines      211059   211121      +62     
  Branches     2444     2444              
==========================================
+ Hits       168727   168799      +72     
+ Misses      41777    41767      -10     
  Partials      555      555              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

Nice one Simon. Thanks, this will fix a lot of hairy bugs.