Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Qdrant vectorstore support #4689

Merged
merged 3 commits into from
Oct 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,14 @@ test-python-universal-singlestore-online:
not test_snowflake" \
sdk/python/tests

test-python-universal-qdrant-online:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.qdrant_repo_configuration \
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.qdrant \
python -m pytest -n 8 --integration \
-k "test_retrieve_online_documents" \
sdk/python/tests/integration/online_store/test_universal_online.py

test-python-universal:
python -m pytest -n 8 --integration sdk/python/tests

Expand Down
22 changes: 19 additions & 3 deletions docs/reference/alpha-vector-database.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,9 @@ Below are supported vector databases and implemented features:
| Milvus | [ ] | [ ] |
| Faiss | [ ] | [ ] |
| SQLite | [x] | [ ] |
| Qdrant | [x] | [x] |

Note: SQLite is in limited access and only working on Python 3.10. It will be updated as [sqlite_vec](https://github.com/asg017/sqlite-vec/) progresses.
Note: SQLite is in limited access and only working on Python 3.10. It will be updated as [sqlite_vec](https://github.com/asg017/sqlite-vec/) progresses.

## Example

Expand Down Expand Up @@ -113,9 +114,11 @@ print_online_features(features)
```

### Configuration
We offer two Online Store options for Vector Databases. PGVector and SQLite.

We offer [PGVector](https://github.com/pgvector/pgvector), [SQLite](https://github.com/asg017/sqlite-vec), [Elasticsearch](https://www.elastic.co) and [Qdrant](https://qdrant.tech/) as Online Store options for Vector Databases.

#### Installation with SQLite

If you are using `pyenv` to manage your Python versions, you can install the SQLite extension with the following command:
```bash
PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" \
Expand All @@ -124,6 +127,19 @@ PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" \
pyenv install 3.10.14
```
And you can the Feast install package via:

```bash
pip install feast[sqlite_vec]
```
```

#### Installation with Elasticsearch

```bash
pip install feast[elasticsearch]
```

#### Installation with Qdrant

```bash
pip install feast[qdrant]
```
81 changes: 81 additions & 0 deletions docs/reference/online-stores/qdrant.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Qdrant online store (contrib)

## Description

[Qdrant](http://qdrant.tech) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage vectors with additional payload and extended filtering support. It makes it useful for all sorts of neural network or semantic-based matching, faceted search, and other applications.

## Getting started

In order to use this online store, you'll need to run `pip install 'feast[qdrant]'`.

## Example

{% code title="feature_store.yaml" %}

```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: qdrant
host: localhost
port: 6333
vector_len: 384
write_batch_size: 100
```

{% endcode %}

The full set of configuration options is available in [QdrantOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.qdrant.QdrantOnlineStoreConfig).

## Functionality Matrix

| | Qdrant |
| :-------------------------------------------------------- | :------- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

## Retrieving online document vectors

The Qdrant online store supports retrieving document vectors for a given list of entity keys. The document vectors are returned as a dictionary where the key is the entity key and the value is the document vector. The document vector is a dense vector of floats.

{% code title="python" %}

```python
from feast import FeatureStore

feature_store = FeatureStore(repo_path="feature_store.yaml")

query_vector = [1.0, 2.0, 3.0, 4.0, 5.0]
top_k = 5

# Retrieve the top k closest features to the query vector
# Since Qdrant supports multiple vectors per entry,
# the vector to use can be specified in the repo config.
# Reference: https://qdrant.tech/documentation/concepts/vectors/#named-vectors
feature_values = feature_store.retrieve_online_documents(
feature="my_feature",
query=query_vector,
top_k=top_k
)
```

{% endcode %}

These APIs are subject to change in future versions of Feast to improve performance and usability.
16 changes: 16 additions & 0 deletions sdk/python/docs/source/feast.infra.online_stores.contrib.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,22 @@ feast.infra.online\_stores.contrib.elasticsearch\_repo\_configuration module
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.contrib.qdrant module
-------------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.qdrant
:members:
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.contrib.qdrant\_repo\_configuration module
----------------------------------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.qdrant_repo_configuration
:members:
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.contrib.hazelcast\_repo\_configuration module
------------------------------------------------------------------------

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
"thrift",
"tpcds",
"tpch",
"qdrant",
}
CONNECTORS_WITHOUT_WITH_STATEMENTS: Set[str] = {
"bigquery",
Expand Down
Loading
Loading