Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure to create index after upserting with subschema #3400

Closed
wjones127 opened this issue Jan 21, 2025 · 1 comment · Fixed by #3404
Closed

Failure to create index after upserting with subschema #3400

wjones127 opened this issue Jan 21, 2025 · 1 comment · Fixed by #3404
Assignees
@wjones127
Labels
bug Something isn't working

Comments

@wjones127
Copy link
Contributor

wjones127 commented Jan 21, 2025 •
edited
Loading

Details TBD

It appears the underlying error is actually: #3402
@wjones127 wjones127 added the bug Something isn't working label Jan 21, 2025
@wjones127 wjones127 self-assigned this Jan 21, 2025
@wjones127 wjones127 mentioned this issue Jan 21, 2025
Merged
@wjones127
Copy link
Contributor Author

Underlying error turned out to be #3406

We are still looking into the null issue though.

wjones127 added a commit that referenced this issue Jan 28, 2025
@wjones127
fix: filter out null values when sampling for index training (#3404)
bfacd7c 
We were not filtering out null values when sampling. Because we often
call `array.values()` on Arrow arrays, which ignores the null bitmap, we
are often silently treating the nulls as zeros (or possibly undefined
values). Only thing that caught these nulls is an assertion. However,
residualization occurring with L2 and Cosine often meant that these
values were transformed and null information was lost before the
assertion, which is why it got past previous unit tests.

This PR adds more assertions validating there aren't nulls, and makes
sure the sampling code handles null vectors.

Closes #3402
Closes #3400
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wjones127 wjones127

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: filter out null values when sampling for index training wjones127/lance
1 participant
@wjones127