Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat #1950

shatejas · 2024-08-12T20:48:12Z

Description

The commit merges feature/iterative-index-build branch
It integrates with KNNVectorValues and NativeEngines990KnnVectorsFormat
There are changes to reuse offheap vector buffer during iterative vector transfer

Related Issues

Resolves #1853

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

navneet1v · 2024-08-12T21:42:32Z

@shatejas lets fix the CIs first

navneet1v

Added initial comments, will be reviewing the code further in next few hours.

src/main/java/org/opensearch/knn/index/codec/BasePerFieldKnnVectorsFormat.java

src/main/java/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriter.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexWriter.java

src/main/java/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriter.java

jni/src/faiss_index_service.cpp

Vikasht34

I have covered 4 files , will continue with remaining file tomoroww.

src/main/java/org/opensearch/knn/common/KNNVectorUtil.java

src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80DocValuesConsumer.java

src/main/java/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriter.java

...in/java/org/opensearch/knn/index/codec/nativeindex/BulkVectorTransferIndexBuildStrategy.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexWriter.java

jni/src/org_opensearch_knn_jni_FaissService.cpp

jni/src/faiss_index_service.cpp

src/main/java/org/opensearch/knn/common/KNNVectorUtil.java

...in/java/org/opensearch/knn/index/codec/nativeindex/BulkVectorTransferIndexBuildStrategy.java

...in/java/org/opensearch/knn/index/codec/nativeindex/MemOptimizedNativeIndexBuildStrategy.java

src/main/java/org/opensearch/knn/index/codec/transfer/VectorTransfer.java

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapQuantizedVectorTransfer.java

src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexWriter.java

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapFloatVectorTransfer.java

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapByteVectorTransfer.java

navneet1v · 2024-08-13T21:14:52Z

Some of the comments can be old as there were updates in the PR. Please ignore comments if they are already answered.

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapQuantizedVectorTransfer.java

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapFloatVectorTransfer.java

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapQuantizedVectorTransfer.java

navneet1v · 2024-08-20T06:29:32Z

CHANGELOG.md

+* Fix graph merge stats size calculation [#1844](https://github.com/opensearch-project/k-NN/pull/1844)
+* Integrate Lucene Vector field with native engines to use KNNVectorFormat during segment creation [#1945](https://github.com/opensearch-project/k-NN/pull/1945)
+* Disallow a vector field to have an invalid character for a physical file name. [#1936] (https://github.com/opensearch-project/k-NN/pull/1936)


Few of these changelogs were moved to separate sections. Please check the latest changelog. Like Integrate Lucene Vector field with native engines to use KNNVectorFormat during segment creation is not a bug fix.

navneet1v · 2024-08-20T06:49:31Z

Overall code looks good to me. CIs are failing due to some backward incompatible changes in main branch of Opensearch. That needs to be fixed first and then CIs needs to be run again. Approving this PR assuming CIs will pass. But lets not merge the PR till CIs are passing.

Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]>

ryanbogan

LGTM! CIs are passing now

opensearch-trigger-bot · 2024-08-20T18:32:45Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1950-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fd59b9adf42b07aa2b2058c4badff6dacf8306a8
# Push it to GitHub
git push --set-upstream origin backport/backport-1950-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1950-to-2.x.

…at (opensearch-project#1950) * Iterative Vector Insertion (opensearch-project#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (opensearch-project#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]> (cherry picked from commit fd59b9a)

…at (#1950) (#1992) * Iterative Vector Insertion (#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]> (cherry picked from commit fd59b9a) Signed-off-by: Tejas Shah <[email protected]>

…at (opensearch-project#1950) * Iterative Vector Insertion (opensearch-project#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (opensearch-project#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]> Signed-off-by: Akash Shankaran <[email protected]>

shatejas requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, ryanbogan and luyuncheng as code owners August 12, 2024 20:48

shatejas force-pushed the iterative-index-integration branch from 71c5f0e to bbc8bb2 Compare August 12, 2024 20:49

shatejas force-pushed the iterative-index-integration branch from bbc8bb2 to af635b3 Compare August 12, 2024 21:58

navneet1v requested changes Aug 12, 2024

View reviewed changes

shatejas force-pushed the iterative-index-integration branch 5 times, most recently from fa51043 to fe9c992 Compare August 13, 2024 01:43

Vikasht34 reviewed Aug 13, 2024

View reviewed changes

navneet1v mentioned this pull request Aug 13, 2024

Integrate KNNVectorValues with vector ANN Search flow #1952

Merged

3 tasks

jmazanec15 reviewed Aug 13, 2024

View reviewed changes

navneet1v reviewed Aug 13, 2024

View reviewed changes

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapQuantizedVectorTransfer.java Outdated Show resolved Hide resolved

navneet1v reviewed Aug 13, 2024

View reviewed changes

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapQuantizedVectorTransfer.java Outdated Show resolved Hide resolved

navneet1v reviewed Aug 13, 2024

View reviewed changes

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapFloatVectorTransfer.java Outdated Show resolved Hide resolved

navneet1v reviewed Aug 13, 2024

View reviewed changes

src/main/java/org/opensearch/knn/index/codec/transfer/OffHeapQuantizedVectorTransfer.java Outdated Show resolved Hide resolved

shatejas force-pushed the iterative-index-integration branch 2 times, most recently from b317c86 to 2528ea6 Compare August 13, 2024 23:23

shatejas dismissed jmazanec15’s stale review via 2256603 August 19, 2024 22:57

shatejas force-pushed the iterative-index-integration branch 2 times, most recently from d56d07f to ad2746f Compare August 20, 2024 00:59

shatejas requested review from navneet1v and jmazanec15 August 20, 2024 01:04

shatejas force-pushed the iterative-index-integration branch 3 times, most recently from 1536f17 to 452fe05 Compare August 20, 2024 04:06

navneet1v reviewed Aug 20, 2024

View reviewed changes

navneet1v previously approved these changes Aug 20, 2024

View reviewed changes

shatejas dismissed navneet1v’s stale review via 54fd143 August 20, 2024 14:33

shatejas force-pushed the iterative-index-integration branch from 452fe05 to 54fd143 Compare August 20, 2024 14:33

jmazanec15 previously approved these changes Aug 20, 2024

View reviewed changes

Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat

bdced71

Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]>

shatejas dismissed jmazanec15’s stale review via bdced71 August 20, 2024 16:55

shatejas force-pushed the iterative-index-integration branch from 54fd143 to bdced71 Compare August 20, 2024 16:55

navneet1v approved these changes Aug 20, 2024

View reviewed changes

ryanbogan approved these changes Aug 20, 2024

View reviewed changes

ryanbogan added backport 2.x v2.17.0 labels Aug 20, 2024

ryanbogan merged commit fd59b9a into opensearch-project:main Aug 20, 2024
34 of 35 checks passed

This was referenced Aug 20, 2024

[Backport 2.x] Integrates FAISS iterative builds with NativeEngines990KnnVectorsForm… #1992

Merged

Adds integration tests for NativeEngineKnnVectorsFormat code path #1995

Closed

navneet1v mentioned this pull request Aug 21, 2024

[Enhancement] [Build-Time V1] [FEATURE] Native Index Memory Footprint Reduction during Indexing #1938

Closed

shatejas deleted the iterative-index-integration branch August 29, 2024 00:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat #1950

Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat #1950

shatejas commented Aug 12, 2024 •

edited

Loading

navneet1v commented Aug 12, 2024

navneet1v left a comment

Vikasht34 left a comment

navneet1v commented Aug 13, 2024

navneet1v Aug 20, 2024

navneet1v commented Aug 20, 2024

ryanbogan left a comment •

edited

Loading

opensearch-trigger-bot bot commented Aug 20, 2024

Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat #1950

Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat #1950

Conversation

shatejas commented Aug 12, 2024 • edited Loading

Description

Related Issues

Check List

navneet1v commented Aug 12, 2024

navneet1v left a comment

Choose a reason for hiding this comment

Vikasht34 left a comment

Choose a reason for hiding this comment

navneet1v commented Aug 13, 2024

navneet1v Aug 20, 2024

Choose a reason for hiding this comment

navneet1v commented Aug 20, 2024

ryanbogan left a comment • edited Loading

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Aug 20, 2024

shatejas commented Aug 12, 2024 •

edited

Loading

ryanbogan left a comment •

edited

Loading