feat: add encoder utilities for pushdown

[westonpace]

This adds a new field encoder (ZoneMapFieldEncoder) that calculates pushdown statistics and places them in the metadata.

It also changes the encoder so that it the choice of encoding is configurable. This makes it possible for extensions to register custom encodings. The zone maps encoder is an example of this as it is placed in a special crate for "encodings that rely on datafusion".

It also adds some utilities for converting an EncodedBatch to Bytes according to the lance file format. This makes it possible to go from RecordBatch to Bytes using the lance file format.

There is not much testing for the zone maps encoder. More will come when we add support for reading zone maps but I want to keep this PR simple for now.

[westonpace]

@wjones127 I used the same MinAccumulator / MaxAccumulator approach that the btree indices use for finding min/max. I wasn't sure if there was any good reason to prefer the StatisticsCollector over this approach? If not, we can simplify once we are done with v1.

[wjones127]

I wasn't sure if there was any good reason to prefer the StatisticsCollector over this approach?

Two things that I think StatisticsCollector handles well:

• Large binary/string values are truncated correctly. The truncation happens before comparison, so it also makes the comparisons cheaper.
• We calculate min, max, and null count in one scan over an array, rather than in independent scans.

Though would be happy to simplify if we can.

[wjones127]

One type that always has special considerations are floats. Will want to document how we want to handle +/-0, Infinity, NaN. For V1, we wrote down the semantics here: https://lancedb.github.io/lance/format.html#statistic-values

[wjones127]

Truncating binary/string values seems like a good idea. There are special considerations for the max to make sure you actually get a greater value after truncation.

[westonpace]

@wjones127 alright, between the truncation and the float ordering I'm convinced I'll switch to using the statistics collector in a future PR. This way users don't have to have breaking changes and I think your logic is simpler when it comes to NaN (ignore).

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 75.93985% with 96 lines in your changes are missing coverage. Please review.

Project coverage is 79.77%. Comparing base (68b45c3) to head (980534b).
Report is 2 commits behind head on main.

Files	Patch %	Lines
rust/lance-encoding-datafusion/src/lib.rs	0.00%	36 Missing ⚠️
rust/lance-encoding-datafusion/src/zone.rs	81.29%	9 Missing and 17 partials ⚠️
rust/lance-file/src/v2/writer.rs	80.85%	14 Missing and 4 partials ⚠️
rust/lance-encoding/src/encoder.rs	89.15%	4 Missing and 5 partials ⚠️
...ust/lance-encoding/src/encodings/physical/value.rs	55.55%	4 Missing ⚠️
.../lance-encoding/src/encodings/logical/primitive.rs	62.50%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2388      +/-   ##
==========================================
- Coverage   79.98%   79.77%   -0.22%     
==========================================
  Files         200      202       +2     
  Lines       54713    55219     +506     
  Branches    54713    55219     +506     
==========================================
+ Hits        43764    44051     +287     
- Misses       8402     8605     +203     
- Partials     2547     2563      +16     

Flag	Coverage Δ
unittests	79.77% <75.93%> (-0.22%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[albertlockett]

Out of curiosity, why try_to_self_described_lance/try_to_mini_lance vs something more descriptive like try_to_self_with_schema/try_to_lance_no_schema? Are mini/self-described common in data format parlance, or is there eventually things other than schema we'll emit from mini vs. self-described?

[westonpace]

Out of curiosity, why try_to_self_described_lance/try_to_mini_lance vs something more descriptive like

I could be argued either way. My original goal was to describe the "why" and not the "what". As a potential API user if I see "to_lance_with_schema" and "to_lance_no_schema" I might not know why I would pick one or the other. Still, there are definitely advantages to being more explicit with what is happening.

"self described" -> the resulting buffer can be deserialized without any extra information.
"mini" -> the smallest possible serialization.

Are mini/self-described common in data format parlance

Not that I'm aware of.

there eventually things other than schema we'll emit from mini vs. self-described?

I don't know of anything at the moment.