Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabled the efficient filtering support for Faiss Engine #906

Closed
wants to merge 1 commit into from

Conversation

navneet1v
Copy link
Collaborator

Description

This is an initial PR for enabling the pre-filtering support for Faiss Engine.

Changes include:

  1. Upgrading the Faiss to commit id: facebookresearch/faiss@3219e3d
  2. Doing an optimization to select when to use IDArraySelector and IDBitMapSelector.
  3. Don't run Faiss search if no docIds are returned from filters.

Next Steps:

  1. Optimize the JNI interfaces to have single query interface.
  2. Write Unit tests for both java and c++ code.
  3. Write ITs for filtering.
  4. Do perf testing, compare with lucene and share the results on the github issue.
  5. Figure out better way to select IDSelector Map.

Issues Resolved

#903

###Testing
Tested the code using the example mentioned here: https://opensearch.org/docs/latest/search-plugins/knn/filter-search-knn/#using-a-lucene-k-nn-filter for lucene

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov
Copy link

codecov bot commented May 22, 2023

Codecov Report

Merging #906 (e105f0f) into 2.x (485a748) will decrease coverage by 1.20%.
The diff coverage is 31.11%.

@@             Coverage Diff              @@
##                2.x     #906      +/-   ##
============================================
- Coverage     85.13%   83.94%   -1.20%     
- Complexity     1088     1094       +6     
============================================
  Files           152      152              
  Lines          4414     4485      +71     
  Branches        392      401       +9     
============================================
+ Hits           3758     3765       +7     
- Misses          479      536      +57     
- Partials        177      184       +7     
Impacted Files Coverage Δ
...rg/opensearch/knn/index/query/KNNQueryBuilder.java 83.55% <0.00%> (-0.66%) ⬇️
...java/org/opensearch/knn/index/query/KNNScorer.java 45.00% <0.00%> (-24.24%) ⬇️
...main/java/org/opensearch/knn/jni/FaissService.java 85.71% <ø> (ø)
...c/main/java/org/opensearch/knn/jni/JNIService.java 81.08% <0.00%> (-4.64%) ⬇️
...pensearch/knn/plugin/rest/RestGetModelHandler.java 77.77% <0.00%> (ø)
...earch/knn/plugin/transport/DeleteModelRequest.java 84.61% <0.00%> (ø)
.../java/org/opensearch/knn/index/query/KNNQuery.java 48.57% <11.76%> (-34.77%) ⬇️
...java/org/opensearch/knn/index/query/KNNWeight.java 63.15% <23.25%> (-26.32%) ⬇️
...rg/opensearch/knn/index/query/KNNQueryFactory.java 86.48% <83.33%> (-3.52%) ⬇️
...java/org/opensearch/knn/index/util/KNNLibrary.java 100.00% <100.00%> (ø)
... and 4 more

... and 1 file with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant