Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PineconeDataPlaneClient Interface #76

Merged
merged 2 commits into from
Mar 12, 2024

Conversation

austin-denoble
Copy link
Contributor

@austin-denoble austin-denoble commented Mar 8, 2024

Problem

Currently, we have two flavors of classes for dealing with blocking and future behavior for gRPC data plane calls:

  • PineconeBlockingDataPlaneClient
  • PineconeFutureDataPlaneClient

Ultimately, both of these classes conform to the same interface for wrapping data plane calls, and both classes have internal logic for validating the arguments passed to these requests that's duplicated across both classes:

PineconeBlockingDataPlaneClient PineconeFutureDataPlaneClient
upsert(String id, List values) upsert(String id, List values)
upsert(String id, List values, String namespace) upsert(String id, List values, String namespace)
upsert(String id, List values, List sparseIndices, List sparseValues, com.google.protobuf.Struct metadata, String namespace) upsert(String id, List values, List sparseIndices, List sparseValues, com.google.protobuf.Struct metadata, String namespace)
queryByVectorId(int topK, String id, String namespace, Struct filter, boolean includeValues, boolean includeMetadata) queryByVectorId(int topK, String id, String namespace, Struct filter, boolean includeValues, boolean includeMetadata)
queryByVectorId(int topK, String id, String namespace, Struct filter) queryByVectorId(int topK, String id, String namespace, Struct filter)
queryByVectorId(int topK, String id, String namespace) queryByVectorId(int topK, String id, String namespace)
queryByVectorId(int topK, String id) queryByVectorId(int topK, String id)
query(int topK, List vector, List sparseIndices, List sparseValues, String id, String namespace, Struct filter, boolean includeValues, boolean includeMetadata) query(int topK, List vector, List sparseIndices, List sparseValues, String id, String namespace, Struct filter, boolean includeValues, boolean includeMetadata)
fetch(List ids) fetch(List ids)
fetch(List ids, String namespace) fetch(List ids, String namespace)
update(String id, List values) update(String id, List values)
update(String id, List values, String namespace) update(String id, List values, String namespace)
update(String id, List values, Struct metadata, String namespace, List sparseIndices, List sparseValues) update(String id, List values, Struct metadata, String namespace, List sparseIndices, List sparseValues)
deleteByIds(List ids, String namespace) deleteByIds(List ids, String namespace)
deleteByIds(List ids) deleteByIds(List ids)
deleteByFilter(Struct filter, String namespace) deleteByFilter(Struct filter, String namespace)
deleteByFilter(Struct filter) deleteByFilter(Struct filter)
deleteAll(String namespace) deleteAll(String namespace)
delete(List ids, boolean deleteAll, String namespace, Struct filter) delete(List ids, boolean deleteAll, String namespace, Struct filter)
describeIndexStats(Struct filter) describeIndexStats(Struct filter)

Solution

Unify the two data plane wrapper classes with a PineconeDataPlaneClient interface.

This interface has default methods for dealing with validation steps for each core dataplane operation:

  • upsert
  • query
  • fetch
  • update
  • delete
  • describeIndexStats

The interface also has stubs for the various overloaded functions that are used to improve the development experience for consumers since Java 8 doesn't support optional arguments. Keeping this contract in an interface allows us to define the expected shape in one place, and avoid needing to validate consistency across the two files.

The interface takes in 6 generic types to represent the return types for each "group" of data plane operations. The reason for this is PineconeBlockingDataPlaneClient and PineconeFutureDataPlaneClient have different return types where the blocking class returns the response objects directly, and the future class returns ListenableFuture<T>. Using generics, this gives us the flexibility to define our own return types for each group of operations:

  • upsert - T
  • query - U
  • fetch - V
  • update - W
  • delete - X
  • describeIndexStats - Z

Type of Change

  • None of the above: refactoring / abstraction

Test Plan

The classes remain the same, so integration tests and builds should pass as normal.

@austin-denoble austin-denoble marked this pull request as ready for review March 8, 2024 18:15
@rohanshah18 rohanshah18 requested a review from aulorbe March 8, 2024 19:00
@rohanshah18
Copy link
Contributor

Thanks for creating the interface, I have added one small comment but otherwise it looks great!!

@austin-denoble austin-denoble force-pushed the adenoble/add-data-plane-interface branch from 7f7f321 to eed989e Compare March 12, 2024 00:53
@austin-denoble austin-denoble merged commit 7daa964 into main Mar 12, 2024
8 checks passed
@austin-denoble austin-denoble deleted the adenoble/add-data-plane-interface branch March 12, 2024 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants