v0.22.1-beta.3

What's Changed

Bug Fixes 🐛

• fix: full text search index may be corrupted after remapping by @BubbleCal in #3388

Full Changelog: v0.22.1-beta.2...v0.22.1-beta.3

v0.22.1-beta.2

What's Changed

New Features 🎉

• feat: add drop_index by @westonpace in #3382

Bug Fixes 🐛

• fix: updating schema/field metadata now retains fragments by @albertlockett in #3384

Full Changelog: v0.22.1-beta.1...v0.22.1-beta.2

v0.22.1-beta.1

What's Changed

New Features 🎉

• feat: enable all datafusion functions by @westonpace in #3381
• feat: support float16/float64 for multivector by @BubbleCal in #3387

Bug Fixes 🐛

• fix: json schema serializes field metadata by @albertlockett in #3379
• fix: flat FTS would return all unindexed rows by @BubbleCal in #3386

Full Changelog: v0.22.0...v0.22.1-beta.1

v0.22.0

What's Changed

Breaking Changes 🛠

• feat!: support multivector type by @BubbleCal in #3190

New Features 🎉

• feat: cache btree sub-index pages by @westonpace in #3309
• feat(java): support spark in predict push down to lance scan by @SaintBacchus in #3314
• feat(py): support count rows with filter in a fragment by @eddyxu in #3318
• feat(java): support take api for java module by @yanghua in #3316
• feat(java): support overwrite for spark connector by @SaintBacchus in #3313
• feat: add global counters for bytes_read & iops for benchmarking utility by @westonpace in #3321
• feat: vector search with distance range by @BubbleCal in #3326
• feat: add utility for reporting data stats by @westonpace in #3328
• feat: cache miniblock metadata by @westonpace in #3323
• feat(java): support statistics row num for lance scan by @SaintBacchus in #3304
• feat: support with_rowaddr for spark by @chenkovsky in #3336
• feat(java): support get real data size for lance spark statistics interface by @SaintBacchus in #3337
• feat(java): support add columns via sql expressions by @yanghua in #3287
• feat: move fsl handling to structural encodings and add support for miniblock by @westonpace in #3324
• feat: support lindera for japanese and korea tokenization by @chenkovsky in #3218
• feat: add support for repetition index to the full zip structural encoding by @westonpace in #3335
• feat: support IVF_FLAT and hamming in pylance by @BubbleCal in #3301
• feat: allow blob in write_fragments by @fecet in #3235
• feat: make it possible to build lance without protoc (except on Windows) by @westonpace in #3363
• feat: log the number of rows we were able to sample by @westonpace in #3367
• feat: upgrade datafusion to 44.0 by @westonpace in #3341
• feat: execute_uncommitted for merge insert by @wjones127 in #3233

Bug Fixes 🐛

• fix: fix pyproject.toml by @chenkovsky in #3299
• fix: is not false crash by @chenkovsky in #3298
• fix: default value is overwritten by @chenkovsky in #3319
• fix: lance ray sink crash when fields contain none by @Jay-ju in #3322
• fix: allow empty scalar indices and don't drop nulls on update by @westonpace in #3329
• fix: coerce scalar for between by @chenkovsky in #3327
• fix(java): replace org.json with gson to resolve the jar conflict with spark 3.5.1 by @SaintBacchus in #3340
• fix: avoid double-take in some scenarios by @westonpace in #3357
• fix: handle deletions in take by @wjones127 in #3360
• fix: fix ray lance sink error by @Jay-ju in #3230
• fix: scan out of range by @chenkovsky in #3339
• fix: cast null arrays to the appropriate type when coercing to a table by @andrijazz in #3362
• fix(python): correct type hint for write_fragments() by @chenkovsky in #3373

Performance Improvements 🚀

• perf: parallelize indexing partitions by @BubbleCal in #3303

Other Changes

• refactor(java): simpilfy fragment by @chenkovsky in #3307

New Contributors

• @fecet made their first contribution in #3235
• @andrijazz made their first contribution in #3362
• @kemingy made their first contribution in #3370

Full Changelog: v0.21.0...v0.22.0

v0.22.0-beta.1

What's Changed

Breaking Changes 🛠

• feat!: support multivector type by @BubbleCal in #3190

New Features 🎉

• feat: add utility for reporting data stats by @westonpace in #3328
• feat: cache miniblock metadata by @westonpace in #3323
• feat(java): support statistics row num for lance scan by @SaintBacchus in #3304
• feat: support with_rowaddr for spark by @chenkovsky in #3336
• feat(java): support get real data size for lance spark statistics interface by @SaintBacchus in #3337
• feat(java): support add columns via sql expressions by @yanghua in #3287
• feat: move fsl handling to structural encodings and add support for miniblock by @westonpace in #3324
• feat: support lindera for japanese and korea tokenization by @chenkovsky in #3218
• feat: add support for repetition index to the full zip structural encoding by @westonpace in #3335
• feat: support IVF_FLAT and hamming in pylance by @BubbleCal in #3301

Bug Fixes 🐛

• fix: coerce scalar for between by @chenkovsky in #3327
• fix(java): replace org.json with gson to resolve the jar conflict with spark 3.5.1 by @SaintBacchus in #3340
• fix: avoid double-take in some scenarios by @westonpace in #3357

Full Changelog: v0.21.1-beta.2...v0.22.0-beta.1

v0.21.1-beta.2

What's Changed

New Features 🎉

• feat(java): support overwrite for spark connector by @SaintBacchus in #3313
• feat: vector search with distance range by @BubbleCal in #3326

Bug Fixes 🐛

• fix: lance ray sink crash when fields contain none by @Jay-ju in #3322
• fix: allow empty scalar indices and don't drop nulls on update by @westonpace in #3329

Performance Improvements 🚀

• perf: parallelize indexing partitions by @BubbleCal in #3303

Full Changelog: v0.21.1-beta.1...v0.21.1-beta.2

v0.21.1-beta.1

What's Changed

New Features 🎉

• feat: cache btree sub-index pages by @westonpace in #3309
• feat(java): support spark in predict push down to lance scan by @SaintBacchus in #3314
• feat(py): support count rows with filter in a fragment by @eddyxu in #3318
• feat(java): support take api for java module by @yanghua in #3316

Bug Fixes 🐛

• fix: fix pyproject.toml by @chenkovsky in #3299
• fix: is not false crash by @chenkovsky in #3298
• fix: default value is overwritten by @chenkovsky in #3319

Full Changelog: v0.21.0...v0.21.1-beta.1

v0.21.0

What's Changed

Breaking Changes 🛠

• fix!: correctly handle nulls in btree and bitmap indices by @westonpace in #3211
• feat!: support hamming distance & binary vector by @BubbleCal in #3198
• refactor(python)!: simplify marshalling of Fragment, DataFile, Operation, Transaction by @wjones127 in #3240

New Features 🎉

• feat: enhance repdef utilities to handle empty / null lists by @westonpace in #3200
• feat: support _rowid meta column for spark connector in java by @SaintBacchus in #3194
• feat: support blob api in pytorch loader by @eddyxu in #3217
• feat(python): add experimental parameter enable_move_stable_row_ids for pylance by @SaintBacchus in #3216
• feat: add the repetition index to the miniblock write path by @westonpace in #3208
• feat: packed struct encoding by @broccoliSpicy in #3186
• feat: support between sql clauses by @connellPortrait in #3225
• feat(java): support drop columns for dataset by @yanghua in #3237
• feat(java): expose uri method for Dataset instance by @yanghua in #3231
• feat: add file statistics by @broccoliSpicy in #3232
• feat: enable tracing for object storage by @wjones127 in #3244
• feat(java): support limit and offset interface for spark connector by @SaintBacchus in #3253
• feat: adds list decode support for mini-block encoded data by @westonpace in #3241
• feat(java): support topn pushdown in spark connector by @SaintBacchus in #3261
• feat: add replace_schema_metadata and replace_field_metadata by @westonpace in #3263
• feat: merge-insert supports inserting subset of columns by @wjones127 in #3100
• feat: support merge by row_id, row_addr by @chenkovsky in #3254
• feat: add the s3 retry config options for storage option by @SaintBacchus in #3268
• feat(java): support alter columns for dataset by @yanghua in #3259
• feat: support remapping for IVF_FLAT, IVF_PQ and IVF_SQ by @BubbleCal in #2708
• feat: change MSRV from 1.78 to 1.80.1 by @westonpace in #3279
• feat: support merge fragment with dataset by @chenkovsky in #3256

Bug Fixes 🐛

• fix: test failure in test_fsl_packed_struct by @broccoliSpicy in #3227
• fix: remove overzealous warning by @westonpace in #3239
• fix: correctly copy null buffer when making deep copy by @westonpace in #3238
• fix: allow LANCE_LOG to be set to trace by @westonpace in #3246
• fix: list indices always shows vector index type is IVF_PQ even it's not by @BubbleCal in #3258
• fix: panic when get stats from index over binary vectors by @BubbleCal in #3267
• fix(rust): adjust scan range to avoid unnecessary warnings by @takaebato in #3248
• fix: when taking struct fields they should be merged into the output in the correct order by @westonpace in #3277
• fix: full text search with limit may return an incorrect results by @BubbleCal in #3284
• fix: refine type annotation by @chenkovsky in #3278

Documentation 📚

• docs: add the documentation about how to install packages for tests by @yanghua in #3213
• docs: add doc and test for 4bit PQ by @BubbleCal in #3212
• docs: blob api documents by @eddyxu in #3247
• docs: add java module into directory structure by @yanghua in #3273

Performance Improvements 🚀

• perf: in-register lookup table & SIMD for 4bit PQ by @BubbleCal in #3178

New Contributors

• @connellPortrait made their first contribution in #3225
• @takaebato made their first contribution in #3248

Full Changelog: v0.20.0...v0.21.0

v0.21.0-beta.5

What's Changed

New Features 🎉

• feat(java): support limit and offset interface for spark connector by @SaintBacchus in #3253
• feat: adds list decode support for mini-block encoded data by @westonpace in #3241
• feat(java): support topn pushdown in spark connector by @SaintBacchus in #3261
• feat: add replace_schema_metadata and replace_field_metadata by @westonpace in #3263
• feat: merge-insert supports inserting subset of columns by @wjones127 in #3100
• feat: support merge by row_id, row_addr by @chenkovsky in #3254
• feat: add the s3 retry config options for storage option by @SaintBacchus in #3268

Bug Fixes 🐛

• fix: allow LANCE_LOG to be set to trace by @westonpace in #3246
• fix: list indices always shows vector index type is IVF_PQ even it's not by @BubbleCal in #3258
• fix: panic when get stats from index over binary vectors by @BubbleCal in #3267
• fix(rust): adjust scan range to avoid unnecessary warnings by @takaebato in #3248

Documentation 📚

• docs: blob api documents by @eddyxu in #3247

New Contributors

• @takaebato made their first contribution in #3248

Full Changelog: v0.21.0-beta.4...v0.21.0-beta.5

v0.21.0-beta.4

What's Changed

New Features 🎉

• feat: add file statistics by @broccoliSpicy in #3232
• feat: enable tracing for object storage by @wjones127 in #3244

Full Changelog: v0.21.0-beta.3...v0.21.0-beta.4