feat(rust): eliminate ProjectionExprs and handle CSE by stacking extra columns

[wence-]

Rather than special-casing CSE expressions, treat them as "just another HStack". That way they are more transparent to the executor. The single nit is that CSE might introduce columns that are not the same length as the dataframe we are hstacking against, so pass through (via ProjectionOptions) that the executor should not broadcast mismatching length-1 columns if they have appeared via CSE.

[ritchie46]

Nice, this cleans it up and exposes CSE to the streaming engine. 👍

We don't have to create a dag. For HStack we should do something like this (in pseudo code):

input_q = ...

schema_input = input_q.schema
schema_out = schema_input.with_columns(hstack_operation)

q = input_q.with_columns(cse_expressions) # adds cse expressions add the end

q = q.with_columns(hstack_operation)  # does with columns operation

q.simple_project(schema_out) # selects the proper columns and thuse prunes the cse columns

[ritchie46]

For the streaming engine, the operators HStackOperator and ProjectionOperator should also be updated and stop expanding the columns.

The default engine's projection should also stop expanding. As we can have cse's with different lengths in a single select.

[codspeed-hq]

CodSpeed Performance Report

Merging #16682 will not alter performance

_{Comparing wence-:wence/fea/remove-projectionexprs (35d23cd) with main (9912af0)}

Summary

✅ 37 untouched benchmarks

[codecov]

Codecov Report

Attention: Patch coverage is 90.90909% with 13 lines in your changes missing coverage. Please review.

Project coverage is 81.46%. Comparing base (534b655) to head (35d23cd).
Report is 35 commits behind head on main.

Files	Patch %	Lines
...s/polars-lazy/src/physical_plan/executors/stack.rs	62.50%	6 Missing ⚠️
py-polars/src/lazyframe/visitor/nodes.rs	0.00%	2 Missing ⚠️
...ars-lazy/src/physical_plan/executors/projection.rs	75.00%	1 Missing ⚠️
...zy/src/physical_plan/executors/projection_utils.rs	87.50%	1 Missing ⚠️
crates/polars-lazy/src/physical_plan/planner/lp.rs	80.00%	1 Missing ⚠️
.../polars-pipe/src/executors/operators/projection.rs	90.00%	1 Missing ⚠️
py-polars/src/lazyframe/visit.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16682      +/-   ##
==========================================
- Coverage   81.50%   81.46%   -0.05%     
==========================================
  Files        1415     1415              
  Lines      186600   186767     +167     
  Branches     3014     3023       +9     
==========================================
+ Hits       152097   152158      +61     
- Misses      33973    34078     +105     
- Partials      530      531       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wence-]

For the streaming engine, the operators HStackOperator and ProjectionOperator should also be updated and stop expanding the columns.

The default engine's projection should also stop expanding. As we can have cse's with different lengths in a single select.

Thanks, I think I handled them all.

[wence-]

I'm not happy about leaving should_broadcast as a public field in ProjectionOptions, but I am a little unsure of the appropriate idiom to hide it (even though it is used across crates).

[wence-]

It appears this one works now?

[wence-]

We're no longer checking that cse is performed, because it's not reported in the executor any more.

[ritchie46]

Looks great @wence-. Thanks a lot.