Releases · lancedb/lance

05 Apr 21:30

v0.4.1

ecc1d18

The vector search in Lance now supports live updates. Previously, when you added new vectors to the dataset, you would be required to rebuild the index. Now, the index is "inherited" and the vector search results are the combination of ANN search on the indexed data and KNN on the new Appended data. So there's a small latency increase and the recall should be the same or better.

This provides a smooth performance curve until you have inserted enough new data that re-indexing is warranted.

What's Changed

Adding secret to publish task by @gsilvestrin in #742
[Rust] make distance function to take slice instead of Float32Array by @eddyxu in #748
Vector search should support appending new rows by @changhiskhan in #593
windows lapack support by @gsilvestrin in #743
Fix LanceDataset.to_batches by @changhiskhan in #751

Full Changelog: v0.4.0...v0.4.1

Contributors

eddyxu, gsilvestrin, and changhiskhan

Assets 2

30 Mar 22:22

changhiskhan

v0.4.0

2922f54

v0.4.0 Windows support

A warm welcome to @gsajko ! Thanks for making our tutorial notebook easier to use and understand!

Note: OPQ is disabled in windows for the vector index. This will be addressed once LAPACK support is added.

What's Changed

small fixes by @gsajko in #725
Windows support by @gsilvestrin in #724

New Contributors

@gsajko made their first contribution in #725

Full Changelog: v0.3.19...v0.4.0

Contributors

gsilvestrin and gsajko

Assets 2

27 Mar 17:58

changhiskhan

v0.3.19

8aa5345

v0.3.19 Bug fix for filter predicates on large-utf8 type

Also fix publishing to crates.io

What's Changed

Make contract clear for KNN nodes by @eddyxu in #729
Refactor Scan I/O plan by @eddyxu in #731
[Rust] Use folked sqlparser to unblock rust crate release by @eddyxu in #732
[Rust] Fix filter on large UTF8 columns by @eddyxu in #733

Full Changelog: v0.3.18...v0.3.19

Contributors

eddyxu

Assets 2

24 Mar 07:45

changhiskhan

v0.3.18

369850d

v0.3.18 Bug fix release for binary offsets

Fix for incorrect offset for string/variable list columns as reported in #720 (comment)

Thanks @lucazanna for the feedback!

What's Changed

Train OPQ and write rotation matrix to index file by @eddyxu in #713
removing warnings by @gsilvestrin in #721
[Bug] Fix IVF merge sort when refine factor is presented. by @eddyxu in #722
Add input / output schema contract to Global Take by @eddyxu in #728
Fix offsets for Binary/Lists/LargeLists by @gsilvestrin in #727

Full Changelog: v0.3.17...v0.3.18

Contributors

eddyxu, gsilvestrin, and lucazanna

Assets 2

22 Mar 02:05

changhiskhan

v0.3.17

ec02352

v0.3.17 Support for nested dict columns

A warm welcome to @haoxins , a new contributor who has helped improve Lance documentation.

This release adds support for list-of-dict columns (thanks @lucazanna for reporting the bug in #715).

Also included in this release are various vector index improvements for scalability and more progress towards OPQ implementation.

What's Changed

docs: fix the links by @haoxins in #701
repair macos build for duckdb extension by @changhiskhan in #705
filter evaluation with flat search by @changhiskhan in #704
fix flaky test by @changhiskhan in #706
[Bug] Fix transpose in MatrixView.data() by @eddyxu in #711
Refactored variable length encoders by @gsilvestrin in #710
add notebook for q&a bot by @changhiskhan in #707
Allow iteratively train PQ by @eddyxu in #712
Use relative eq and fix a compiling warning by @eddyxu in #714
docs: fix the mod path by @haoxins in #718
Composable vector search pipeline by @eddyxu in #716
Fix CI failure by increasing epsilon for test_train_pq_iteratively by @eddyxu in #719
Implement support for list of Dictionaries by @gsilvestrin in #664

New Contributors

@haoxins made their first contribution in #701

Full Changelog: v0.3.16...v0.3.17

Contributors

eddyxu, gsilvestrin, and 3 other contributors

Assets 2

18 Mar 06:48

changhiskhan

v0.3.16

27d36e8

v0.3.16 Filte pushdown improvements

Welcome @wangfenjin to lance contributors. Thanks for submitting a bug fix for the Lance DuckDB extensions 🔥

This release contains 2 workarounds for arrow limitations:

Lance datasets now support <field> LIKE '%' and <field> IN (<values>) filters to be passed in as string. Generic SQL syntax supported by datafusion is now accepted. This is a break from standard pyarrow Dataset behavior which only accepts arrow compute Expression, which is not present in rust and also does not support introspection in python for developers to build custom adapter.
When concatenating arrow dictionary arrays, the dict values are duplicated. There is currently no concrete plans to change this behavior in Arrow. Instead, we fix that at write time in Lance.