2025-01-30 kernel meeting notes

[OussamaSaoudi]

summary

• Using the discussion as a way to detect blockers in community PRs.
• Discussing where vacuum/optimize belong (whether in the kernel, or outside of it).
• The new arrow feature flags that are coming will help integrations

pull requests/ongoing threads

• @nicklan Part 2: propagate transform in visit_scan_files #612
• @timsaucer partition skipping filter #624
• @zachschuermann feat!: new create_one ExpressionHandler API #662
• @OussamaSaoudi refactor: Make CDF use TableConfiguration and refactor log replay #645
• @OussamaSaoudi feat: Enable simple schema evolution for Change Data Feed #625

attendees

@nicklan @hntd187 @zachschuermann @OussamaSaoudi-db @roeap @mightyshazam @dennyglee @sebastiantia

leftovers

• Change the discussion format, particularly the pull requests/ongoing threads section.
• CI to build nightly with external integrations and make sure FFI works
• Implementing partial_cmp/eq for datetime, decimal, and other types

notes stream

• Should allow external contributors to put down comments for the meeting. TODO (Oussama)
• Feature flag for arrow is still WIP. There's more discussion to be done.
• Expression transform is moving along.
• Oussama: feature parity between kernel and delta-rs
• Robert: Working on pr to move delta-rs to kernel. Can put up differences between kernel/delta-rs. There was a change that reads sql and parses it. What should sql look like in kernel?
• Robert: The change was generated columns. This doesn't fit in with the expression transform work nick's on.
• nick: This is a blocker for full adoption right?
• Robert: We can still leverage a bunch of parts of kernel. We're kicking the can down the road by adding more of these features.
• Ryan: This can benefit from an opaque expressions. The exiysting usecases are arbitrary expressions that customers are using. We can't enumerate the set of all expressions. engines don't want to iml generated columns because it may introduce too much. Pitch: Introduce table feature for constraints, generated columns, and default values that incorporates that into delta. This gives clarity on what delta actually supports. Opens the door for people to meaningfully use features in delta.
• Stephen: Whatever set we choose, we're going to miss something. iS there a way to iterate on it as new cases come up? this is more RFC wrangling.
• Ryan: The feature specifies the set of functions that are going to be supported. But the point of RFC is that the RFC can be accepted with no obligation that it stay in that form. We can iterate it and eventually stabilize it to be part of the delta spec.
• Robert: There's a concept of a preview feature. We can use this.
• Ryan: As long as it's an RFC and not in spec, we can revise using v1, v2, v3...
• Ryan: If someone implements a certain version, you trivially implement the full spec version when it's finally stablized. We can specify an upgrade process when it _is_stabilitze.
• Ryan: There needs to be a reference implementation for a feature.r
• Robert: We're mostly interested in data skipping for these expressions. Engines are going to have to figure out these expressions, not kernel, so the opaque type is a good idea.
• Ryan: Expression eval today is in good place to support opaque expressions
• Kyjah: On engine FFI: Could implement the ffi for dot net engine. Are there any immediate pitfalls?
• Ryan: You would be the first to do it. FFI/impedence mismatch with c# would be an issue. There's also a lot of boilerplate.
• Nick: Most untested part of that is around the engine data passing. It's meant to allow you to maintain your own engine data, but no one has done that significantly yet. there is a way to opt into kernel's engine data representation if you're using Arrow. This is an easy shortcut.
• nick: Engine data has a lot of surface area already modulo engine api. Ideally use arrow to just need to implement engine api.
• Stephen: Is arrow you're using based on the CPP library?
• Kyjah: Probably not going to use it.
• Stephen: There's a bug in cpp arrow that breaks kernel.
• Kyjah: Encountered this issue in delta-rs. Prefer arrow all the way.
• nick: Please ping if anything comes up. we're here to help!
• Robert: 2 things that could change things: 1) With datafusion, you need to track what file you read from. Engine data can be more than arrow + metadata. This is especially relevant with expressions that correlate with files. 2) Catalog integration will impact this significantly.
• nick: 1) We need to think more about the file tracking. 2) Engine interface doesn't care about catalog, but your engine may need to be aware of a catalog. You don't need to worry that the engine interface will be impacted by catalogs.
• Ryan: 2 notes: 1) Arrow cpp isn't coded up in kernel yet. It could be done, but PR isn't up. May not need if arrow is broken. 2) As you deal with FFI. If you see opportunities to simplify the surface area, we are very interested in making this simpler.
• Nick: Arrow crate in rust lets you get raw arrow memory. This should be able to pass to any arrow implementation in any implementation. Nick used the GLibC arrow implementation.
• Tim; Working on the partition columns, hoping to get it in over the next couple weeks
• Robert: There's interest in getting delta into datafusion beyond just table providers. Reach out if there's interest from maintainers in bringing delta in.
• nick: We're happy to help datafusion push this forward. Get in touch on the delta-users slack channel to push this forward more.
• Stephen: For ffi, we expose minimal snapshot info. Do we want to expose table properties, root and version, etc? Are we ammenable to expose more stuff?
• nick: Tableconfiguration should help with that. The goal of FFI should be to have parity with the rust interface.
• Ryan: For table properties: 1) We want ffi to have that information. 2) We exposed too much on the rust side and we're trying to claw it back. Ex: catalog and kernel definitions of table properties do not match up. We want user-visible versions of everything. High level "is cdf enabled" type apis instead of the raw delta propreties.
• Ryan: We're probably not going to get dev-visibility into the FFI. This is a much larger API surface.
• Robert: In the in list Pr, is there a reference for how to implement this? Partial_cmp is failing because we don't support decimals. Need to impl that, and we may need some reference implementation/library.
• Ryan: I suspect that delta-spark uses spark's decimal machinery. Let's try to use library. Decimal is defined by the parquet format, so the semantics should be consistent for everyone. Just supporting partial_cm for decimals, we don't need to implement it for adds/sub/other operations
• nick: datetime is going to be a pain
• Nick: we are missing date/timestamp scalar types in the FFI expression visitor.
• Robert: Arrow always keeps it in UTC. So we could convert everything into the timestamp type. Life's easier if your data is all in UTC.
• Nick: There's still an issue of 96 bit/inclusion of microseconds.