Initial cut for a cuVS Java API #450

chatman · 2024-11-08T14:05:47Z

A Java API for cuVS for easy integration into Apache Lucene or other Java based projects.

Try:

./build.sh libcuvs
./build.sh java

For generating docs, mvn javadoc:javadoc

Prerequisites:

JDK 22
Maven 3.9.6+

Todo:

Generate project panama classes using jextract on every build
Algorithms other than Cagra
Prefiltering in cagra

Co-authored-by: Vivek Narang <[email protected]>

copy-pr-bot · 2024-11-08T14:05:51Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

chatman · 2024-11-08T14:07:30Z

FYI @cjnolet ^
An ExampleApp.java is added as a starting point for the review.

cjnolet · 2024-11-08T16:06:56Z

/ok to test

chatman · 2024-11-18T02:56:53Z

@naramgvivek10 Let's move CuVSResources to the cuvs package instead of common? That way we can abstract out the internals of Panama out of sight of the users.

… etc. (#6) * Ability to configure CAGRA compression parameters * Enabling indexing threads * Enabling RMM pool resource configuration * Bug fixes for wrong values passed in index and search parameters * Deallocation of resources using Autoclosable * Including a randomized test Co-authored-by: Vivek Narang <[email protected]> Co-authored-by: Ishan Chattopadhyaya <[email protected]> Co-authored-by: Puneet Ahuja <[email protected]>

* Bruteforce API implementation Co-authored-by: Vivek Narang <[email protected]>

* Bruteforce serialize and deserialize API implementation Co-authored-by: Vivek Narang <[email protected]>

* code updates for the added index parameter Co-authored-by: Vivek Narang <[email protected]>

Co-authored-by: Vivek Narang <[email protected]>

* move prefilter from index to query Co-authored-by: Vivek Narang <[email protected]>

Co-authored-by: Vivek Narang <[email protected]>

…alues from the cuvs layer

Co-authored-by: Vivek Narang <[email protected]>

* review updates --------- Co-authored-by: Vivek Narang <[email protected]>

cjnolet · 2025-01-08T20:08:20Z

/ok to test

cjnolet · 2025-01-15T15:48:16Z

/ok to test

cjnolet · 2025-01-16T01:26:27Z

java/build.sh

@@ -0,0 +1,10 @@
+VERSION="25.02"


Make sure this version gets updated when ci/release/update_versions.sh is called.

cjnolet · 2025-01-16T01:30:04Z

java/cuvs-java/src/main/java/com/nvidia/cuvs/panama/CuVSCagraIndexParams.java

@@ -0,0 +1,354 @@
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION.


Just a small nitpick: I notice this file is prefixed with "CuVS" while most of the other files are prefixed with "Cuvs". I personally prefer "CuVS" (or even "cuVS" if that doesn't break convention too badly) but I would at least make it consistent.

Thanks for this note. I will fix this soon.

ChrisHegarty · 2025-01-20T11:57:09Z

java/cuvs-java/src/main/java/com/nvidia/cuvs/BruteForceIndex.java

+  private MethodHandle searchMethodHandle;
+  private MethodHandle destroyIndexMethodHandle;
+  private MethodHandle serializeMethodHandle;
+  private MethodHandle deserializeMethodHandle;


Can we make these MethodHandles static final - since they optimise much nicer by the JVM? I'm happy to rework some of the code flow here, if possible (and it that would be helpful) - though I'm not overly familiar with the development environment.

* update ci release script to update the version in build.sh * class name updates --------- Co-authored-by: Vivek Narang <[email protected]>

Co-authored-by: Vivek Narang <[email protected]>

Initial cut for a cuVS Java API

c777924

Co-authored-by: Vivek Narang <[email protected]>

chatman requested review from a team as code owners November 8, 2024 14:05

chatman requested a review from msarahan November 8, 2024 14:05

github-actions bot added the CMake label Nov 8, 2024

cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Nov 8, 2024

cjnolet assigned chatman Nov 8, 2024

cjnolet changed the base branch from branch-24.10 to branch-24.12 November 8, 2024 16:06

Vivek Narang added 2 commits November 9, 2024 07:11

API changes, serialize optional tmpFile param + other changes

f9c2df7

make topK configurable

1597158

narangvivek10 force-pushed the java-api branch from 2cfab29 to 1597158 Compare November 11, 2024 15:07

Vivek Narang and others added 10 commits November 14, 2024 05:39

multiple fixes

6f405e9

pom file update

8531be2

sample test

c9b8891

sample test update + other changes

9bb865a

sample test update + use logging

d40b3d0

variable name updates + other

1e03585

Javadoc draft 1

3222536

Fixes for Javadocs, reducing visibility of internals, other refactorings

c5536b6

Loading the .so file from the jar instead of a hardcoded path

2d79580

package renaming and formatting

84c6121

chatman changed the title ~~[WIP] Initial cut for a cuVS Java API~~ Initial cut for a cuVS Java API Nov 18, 2024

Vivek Narang added 3 commits November 18, 2024 00:14

moving CuVSResources + other updates

20562ee

code refactoring

ad11369

javadoc maven config update and bug fixes

6a851ef

narangvivek10 and others added 13 commits December 13, 2024 13:28

Merge branch 'branch-25.02' into java-api

53dfa23

Upgrading to 25.02

6f708eb

Bruteforce API implementation (#8)

b98d2bb

* Bruteforce API implementation Co-authored-by: Vivek Narang <[email protected]>

Bruteforce serialize and deserialize API implementation (#9)

81b60f1

* Bruteforce serialize and deserialize API implementation Co-authored-by: Vivek Narang <[email protected]>

Merge branch 'rapidsai:branch-25.02' into java-api

dca835d

metric parameter addition to the CAGRA index parameters (#10)

e3757c0

* code updates for the added index parameter Co-authored-by: Vivek Narang <[email protected]>

update year

0aa2665

Co-authored-by: Vivek Narang <[email protected]>

API update - move prefilter from BruteForceIndex to BruteForceQuery

a89fe5d

* move prefilter from index to query Co-authored-by: Vivek Narang <[email protected]>

HNSW API implementation

b44baa9

Co-authored-by: Vivek Narang <[email protected]>

Mapping is a single integer list, added an error logging for absurd v…

98c3c93

…alues from the cuvs layer

Compilation error fix (enum needs to be mentioned as per C standards)

feed385

If dataset is smaller than topK, then adjust topK to be the dataset size

1d80b41

chatman requested a review from a team as a code owner January 6, 2025 15:16

github-actions bot added the cpp label Jan 6, 2025

narangvivek10 and others added 3 commits January 7, 2025 09:29

Add randomized tests for HNSW and Bruteforce

2f0f143

Co-authored-by: Vivek Narang <[email protected]>

Adding module-info.java

2c0e4cd

Review updates

72bc6e3

* review updates --------- Co-authored-by: Vivek Narang <[email protected]>

Removing dependency on slf4j and commons-io

b79a17e

chatman mentioned this pull request Jan 10, 2025

Integrating GPU based Vector Search using cuVS apache/lucene#14131

Open

Plugin configuration update in pom.xml and javadoc fix

6819bdf

cjnolet reviewed Jan 16, 2025

View reviewed changes

ChrisHegarty reviewed Jan 20, 2025

View reviewed changes

Review feedback updates

bc280e4

* update ci release script to update the version in build.sh * class name updates --------- Co-authored-by: Vivek Narang <[email protected]>

narangvivek10 requested a review from a team as a code owner January 22, 2025 22:29

github-actions bot added the ci label Jan 22, 2025

Feature: get GPU information

392d0ff

Co-authored-by: Vivek Narang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial cut for a cuVS Java API #450

Initial cut for a cuVS Java API #450

chatman commented Nov 8, 2024 •

edited

Loading

copy-pr-bot bot commented Nov 8, 2024

chatman commented Nov 8, 2024

cjnolet commented Nov 8, 2024

chatman commented Nov 18, 2024

cjnolet commented Jan 8, 2025

cjnolet commented Jan 15, 2025

cjnolet Jan 16, 2025

cjnolet Jan 16, 2025

narangvivek10 Jan 16, 2025

ChrisHegarty Jan 20, 2025

		@@ -0,0 +1,354 @@
		/*
		* Copyright (c) 2025, NVIDIA CORPORATION.

Initial cut for a cuVS Java API #450

Are you sure you want to change the base?

Initial cut for a cuVS Java API #450

Conversation

chatman commented Nov 8, 2024 • edited Loading

copy-pr-bot bot commented Nov 8, 2024

chatman commented Nov 8, 2024

cjnolet commented Nov 8, 2024

chatman commented Nov 18, 2024

cjnolet commented Jan 8, 2025

cjnolet commented Jan 15, 2025

cjnolet Jan 16, 2025

Choose a reason for hiding this comment

cjnolet Jan 16, 2025

Choose a reason for hiding this comment

narangvivek10 Jan 16, 2025

Choose a reason for hiding this comment

ChrisHegarty Jan 20, 2025

Choose a reason for hiding this comment

chatman commented Nov 8, 2024 •

edited

Loading