2024-02-01 kernel meeting notes

[zachschuermann]

summary

Smattering of discussions around various design decisions, opening repo soon (TM)

action items

two PRs in flight:

• Switch to new data passing API #109
• Leverage more engine capabilities in data skipping 2/n #83

attendees

@tdas @roeap @zachschuermann @nicklan @ryan-johnson-databricks @dennyglee @hntd187

notes stream

• some discussion on opening the repository: in the works

nick: scan result

• we used to return a Vec of record batches
• then would concat_batches and filter_record_batch both of which are arrow compute functions
• to remove this we can: (1) do a contains() method in a ScanResult type or (2) just return a selection vector along with the data (probably still as a ScanResult). Likely pursuing 2, just doing a boolean array.

How can we flatten roaring bitmap? Maybe build the dumb thing now (for loop) and take this on as an optimization later? Roaring bitmap stores indicies but we would want a sparse array.
Aside (zach): does allocating for a selection vector cause memory management issues?

Ryan: 3 things

1. FFI and error result types
   FFI version of EngineInterface
2. traits/vtable implementation: what to do if the engine already has rust trait version of something? we want to allow connectors to use normal (default) expression eval but bring their own custom parquet. How?
3. we have lots of iterators. can we unify them? Then just one way of iterating.

From nick/robert: unsafe in kernel used in one place to do Box::into_raw can we work around this?