From 5f9c1f90ade9eccd1a1104806d34a1337853ac90 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 23 Oct 2024 16:45:22 +0800 Subject: [PATCH] move vector search docs to the tidb-cloud folder --- tidb-cloud/tidb-cloud-release-notes.md | 12 +- .../vector-search-data-types.md | 20 +- .../vector-search-functions-and-operators.md | 12 +- .../vector-search-get-started-using-python.md | 10 +- .../vector-search-get-started-using-sql.md | 12 +- .../vector-search-improve-performance.md | 4 +- tidb-cloud/vector-search-index.md | 181 ++++++------ ...vector-search-integrate-with-django-orm.md | 6 +- ...-search-integrate-with-jinaai-embedding.md | 4 +- .../vector-search-integrate-with-langchain.md | 6 +- ...vector-search-integrate-with-llamaindex.md | 6 +- .../vector-search-integrate-with-peewee.md | 6 +- ...vector-search-integrate-with-sqlalchemy.md | 6 +- .../vector-search-integration-overview.md | 6 +- .../vector-search-limitations.md | 4 +- .../vector-search-overview.md | 8 +- vector-search-index.md | 263 ------------------ 17 files changed, 146 insertions(+), 420 deletions(-) rename vector-search-data-types.md => tidb-cloud/vector-search-data-types.md (90%) rename vector-search-functions-and-operators.md => tidb-cloud/vector-search-functions-and-operators.md (96%) rename vector-search-get-started-using-python.md => tidb-cloud/vector-search-get-started-using-python.md (92%) rename vector-search-get-started-using-sql.md => tidb-cloud/vector-search-get-started-using-sql.md (88%) rename vector-search-improve-performance.md => tidb-cloud/vector-search-improve-performance.md (92%) rename vector-search-integrate-with-django-orm.md => tidb-cloud/vector-search-integrate-with-django-orm.md (97%) rename vector-search-integrate-with-jinaai-embedding.md => tidb-cloud/vector-search-integrate-with-jinaai-embedding.md (98%) rename vector-search-integrate-with-langchain.md => tidb-cloud/vector-search-integrate-with-langchain.md (99%) rename vector-search-integrate-with-llamaindex.md => tidb-cloud/vector-search-integrate-with-llamaindex.md (98%) rename vector-search-integrate-with-peewee.md => tidb-cloud/vector-search-integrate-with-peewee.md (96%) rename vector-search-integrate-with-sqlalchemy.md => tidb-cloud/vector-search-integrate-with-sqlalchemy.md (96%) rename vector-search-integration-overview.md => tidb-cloud/vector-search-integration-overview.md (91%) rename vector-search-limitations.md => tidb-cloud/vector-search-limitations.md (93%) rename vector-search-overview.md => tidb-cloud/vector-search-overview.md (86%) delete mode 100644 vector-search-index.md diff --git a/tidb-cloud/tidb-cloud-release-notes.md b/tidb-cloud/tidb-cloud-release-notes.md index 24787e0bd4304..56bb54a1d6f0e 100644 --- a/tidb-cloud/tidb-cloud-release-notes.md +++ b/tidb-cloud/tidb-cloud-release-notes.md @@ -120,7 +120,7 @@ This page lists the release notes of [TiDB Cloud](https://www.pingcap.com/tidb-c - [Data Service (beta)](https://tidbcloud.com/console/data-service) supports automatically generating vector search endpoints. - If your table contains [vector data types](/vector-search-data-types.md), you can automatically generate a vector search endpoint that calculates vector distances based on your selected distance function. + If your table contains [vector data types](/tidb-cloud/vector-data-types.md), you can automatically generate a vector search endpoint that calculates vector distances based on your selected distance function. This feature enables seamless integration with AI platforms such as [Dify](https://docs.dify.ai/guides/tools) and [GPTs](https://openai.com/blog/introducing-gpts), enhancing your applications with advanced natural language processing and AI capabilities for more complex tasks and intelligent solutions. @@ -166,12 +166,12 @@ This page lists the release notes of [TiDB Cloud](https://www.pingcap.com/tidb-c The vector search (beta) feature provides an advanced search solution for performing semantic similarity searches across various data types, including documents, images, audio, and video. This feature enables developers to easily build scalable applications with generative artificial intelligence (AI) capabilities using familiar MySQL skills. Key features include: - - [Vector data types](/vector-search-data-types.md), [vector index](/vector-search-index.md), and [vector functions and operators](/vector-search-functions-and-operators.md). - - Ecosystem integrations with [LangChain](/vector-search-integrate-with-langchain.md), [LlamaIndex](/vector-search-integrate-with-llamaindex.md), and [JinaAI](/vector-search-integrate-with-jinaai-embedding.md). - - Programming language support for Python: [SQLAlchemy](/vector-search-integrate-with-sqlalchemy.md), [Peewee](/vector-search-integrate-with-peewee.md), and [Django ORM](/vector-search-integrate-with-django-orm.md). - - Sample applications and tutorials: perform semantic searches for documents using [Python](/vector-search-get-started-using-python.md) or [SQL](/vector-search-get-started-using-sql.md). + - [Vector data types](/tidb-cloud/vector-data-types.md), [vector index](/tidb-cloud/vector-index.md), and [vector functions and operators](/tidb-cloud/vector-functions-and-operators.md). + - Ecosystem integrations with [LangChain](/tidb-cloud/vector-integrate-with-langchain.md), [LlamaIndex](/tidb-cloud/vector-integrate-with-llamaindex.md), and [JinaAI](/tidb-cloud/vector-integrate-with-jinaai-embedding.md). + - Programming language support for Python: [SQLAlchemy](/tidb-cloud/vector-integrate-with-sqlalchemy.md), [Peewee](/tidb-cloud/vector-integrate-with-peewee.md), and [Django ORM](/tidb-cloud/vector-integrate-with-django-orm.md). + - Sample applications and tutorials: perform semantic searches for documents using [Python](/tidb-cloud/vector-get-started-using-python.md) or [SQL](/tidb-cloud/vector-get-started-using-sql.md). - For more information, see [Vector search (beta) overview](/vector-search-overview.md). + For more information, see [Vector search (beta) overview](/tidb-cloud/vector-overview.md). - [TiDB Cloud Serverless](/tidb-cloud/select-cluster-tier.md#tidb-cloud-serverless) now offers weekly email reports for organization owners. diff --git a/vector-search-data-types.md b/tidb-cloud/vector-search-data-types.md similarity index 90% rename from vector-search-data-types.md rename to tidb-cloud/vector-search-data-types.md index 62031f506629d..6c86acb5d5fd6 100644 --- a/vector-search-data-types.md +++ b/tidb-cloud/vector-search-data-types.md @@ -26,7 +26,7 @@ The following Vector data types are currently available: Using vector data types provides the following advantages over using the [`JSON`](/data-type-json.md) type: -- Vector index support: You can build a [vector search index](/vector-search-index.md) to speed up vector searching. +- Vector index support: You can build a [vector search index](/tidb-cloud/vector-search-index.md) to speed up vector searching. - Dimension enforcement: You can specify a dimension to forbid inserting vectors with different dimensions. - Optimized storage format: Vector data types are optimized for handling vector data, offering better space efficiency and performance compared to `JSON` types. @@ -65,9 +65,9 @@ In the following example, because dimension `3` is enforced for the `embedding` ERROR 1105 (HY000): vector has 2 dimensions, does not fit VECTOR(3) ``` -For available functions and operators over the vector data types, see [Vector Functions and Operators](/vector-search-functions-and-operators.md). +For available functions and operators over the vector data types, see [Vector Functions and Operators](/tidb-cloud/vector-search-functions-and-operators.md). -For more information about building and using a vector search index, see [Vector Search Index](/vector-search-index.md). +For more information about building and using a vector search index, see [Vector Search Index](/tidb-cloud/vector-search-index.md). ## Store vectors with different dimensions @@ -83,11 +83,11 @@ INSERT INTO vector_table VALUES (1, '[0.3, 0.5, -0.1]'); -- 3 dimensions vector, INSERT INTO vector_table VALUES (2, '[0.3, 0.5]'); -- 2 dimensions vector, OK ``` -However, note that you cannot build a [vector search index](/vector-search-index.md) for this column, as vector distances can be only calculated between vectors with the same dimensions. +However, note that you cannot build a [vector search index](/tidb-cloud/vector-search-index.md) for this column, as vector distances can be only calculated between vectors with the same dimensions. ## Comparison -You can compare vector data types using [comparison operators](/functions-and-operators/operators.md) such as `=`, `!=`, `<`, `>`, `<=`, and `>=`. For a complete list of comparison operators and functions for vector data types, see [Vector Functions and Operators](/vector-search-functions-and-operators.md). +You can compare vector data types using [comparison operators](/functions-and-operators/operators.md) such as `=`, `!=`, `<`, `>`, `<=`, and `>=`. For a complete list of comparison operators and functions for vector data types, see [Vector Functions and Operators](/tidb-cloud/vector-search-functions-and-operators.md). Vector data types are compared element-wise numerically. For example: @@ -231,7 +231,7 @@ You can also explicitly cast a vector to its string representation. Take using t 1 row in set (0.01 sec) ``` -For additional cast functions, see [Vector Functions and Operators](/vector-search-functions-and-operators.md). +For additional cast functions, see [Vector Functions and Operators](/tidb-cloud/vector-search-functions-and-operators.md). ### Cast between Vector ⇔ other data types @@ -241,7 +241,7 @@ Note that vector data type columns stored in a table cannot be converted to othe ## Restrictions -For restrictions on vector data types, see [Vector search limitations](/vector-search-limitations.md) and [Vector index restrictions](/vector-search-index.md#restrictions). +For restrictions on vector data types, see [Vector search limitations](/tidb-cloud/vector-search-limitations.md) and [Vector index restrictions](/tidb-cloud/vector-search-index.md#restrictions). ## MySQL compatibility @@ -249,6 +249,6 @@ Vector data types are TiDB specific, and are not supported in MySQL. ## See also -- [Vector Functions and Operators](/vector-search-functions-and-operators.md) -- [Vector Search Index](/vector-search-index.md) -- [Improve Vector Search Performance](/vector-search-improve-performance.md) \ No newline at end of file +- [Vector Functions and Operators](/tidb-cloud/vector-search-functions-and-operators.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) +- [Improve Vector Search Performance](/tidb-cloud/vector-search-improve-performance.md) \ No newline at end of file diff --git a/vector-search-functions-and-operators.md b/tidb-cloud/vector-search-functions-and-operators.md similarity index 96% rename from vector-search-functions-and-operators.md rename to tidb-cloud/vector-search-functions-and-operators.md index f6ed6449e9567..9477d3dab4e18 100644 --- a/vector-search-functions-and-operators.md +++ b/tidb-cloud/vector-search-functions-and-operators.md @@ -21,7 +21,7 @@ This document lists the functions and operators available for Vector data types. ## Vector functions -The following functions are designed specifically for [Vector data types](/vector-search-data-types.md). +The following functions are designed specifically for [Vector data types](/tidb-cloud/vector-search-data-types.md). **Vector distance functions:** @@ -43,7 +43,7 @@ The following functions are designed specifically for [Vector data types](/vecto ## Extended built-in functions and operators -The following built-in functions and operators are extended to support operations on [Vector data types](/vector-search-data-types.md). +The following built-in functions and operators are extended to support operations on [Vector data types](/tidb-cloud/vector-search-data-types.md). **Arithmetic operators:** @@ -52,7 +52,7 @@ The following built-in functions and operators are extended to support operation | [`+`](https://dev.mysql.com/doc/refman/8.0/en/arithmetic-functions.html#operator_plus) | Vector element-wise addition operator | | [`-`](https://dev.mysql.com/doc/refman/8.0/en/arithmetic-functions.html#operator_minus) | Vector element-wise subtraction operator | -For more information about how vector arithmetic works, see [Vector Data Type | Arithmetic](/vector-search-data-types.md#arithmetic). +For more information about how vector arithmetic works, see [Vector Data Type | Arithmetic](/tidb-cloud/vector-search-data-types.md#arithmetic). **Aggregate (GROUP BY) functions:** @@ -84,7 +84,7 @@ For more information about how vector arithmetic works, see [Vector Data Type | | [`!=`, `<>`](https://dev.mysql.com/doc/refman/8.0/en/comparison-operators.html#operator_not-equal) | Not equal operator | | [`NOT IN()`](https://dev.mysql.com/doc/refman/8.0/en/comparison-operators.html#operator_not-in) | Check whether a value is not within a set of values | -For more information about how vectors are compared, see [Vector Data Type | Comparison](/vector-search-data-types.md#comparison). +For more information about how vectors are compared, see [Vector Data Type | Comparison](/tidb-cloud/vector-search-data-types.md#comparison). **Control flow functions:** @@ -102,7 +102,7 @@ For more information about how vectors are compared, see [Vector Data Type | Com | [`CAST()`](https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html#function_cast) | Cast a value as a string or vector | | [`CONVERT()`](https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html#function_convert) | Cast a value as a string | -For more information about how to use `CAST()`, see [Vector Data Type | Cast](/vector-search-data-types.md#cast). +For more information about how to use `CAST()`, see [Vector Data Type | Cast](/tidb-cloud/vector-search-data-types.md#cast). ## Full references @@ -289,4 +289,4 @@ The vector functions and the extended usage of built-in functions and operators ## See also -- [Vector Data Types](/vector-search-data-types.md) +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) diff --git a/vector-search-get-started-using-python.md b/tidb-cloud/vector-search-get-started-using-python.md similarity index 92% rename from vector-search-get-started-using-python.md rename to tidb-cloud/vector-search-get-started-using-python.md index 0a39d65a28fc1..e962fa020a66f 100644 --- a/vector-search-get-started-using-python.md +++ b/tidb-cloud/vector-search-get-started-using-python.md @@ -7,7 +7,7 @@ summary: Learn how to quickly develop an AI application that performs semantic s This tutorial demonstrates how to develop a simple AI application that provides **semantic search** features. Unlike traditional keyword search, semantic search intelligently understands the meaning behind your query and returns the most relevant result. For example, if you have documents titled "dog", "fish", and "tree", and you search for "a swimming animal", the application would identify "fish" as the most relevant result. -Throughout this tutorial, you will develop this AI application using [TiDB Vector Search](/vector-search-overview.md), Python, [TiDB Vector SDK for Python](https://github.com/pingcap/tidb-vector-python), and AI models. +Throughout this tutorial, you will develop this AI application using [TiDB Vector Search](/tidb-cloud/vector-search-overview.md), Python, [TiDB Vector SDK for Python](https://github.com/pingcap/tidb-vector-python), and AI models. @@ -69,7 +69,7 @@ pip install sqlalchemy pymysql sentence-transformers tidb-vector python-dotenv ``` - `tidb-vector`: the Python client for interacting with TiDB vector search. -- [`sentence-transformers`](https://sbert.net): a Python library that provides pre-trained models for generating [vector embeddings](/vector-search-overview.md#vector-embedding) from text. +- [`sentence-transformers`](https://sbert.net): a Python library that provides pre-trained models for generating [vector embeddings](/tidb-cloud/vector-search-overview.md#vector-embedding) from text. ### Step 3. Configure the connection string to the TiDB cluster @@ -135,7 +135,7 @@ The following are descriptions for each parameter: ### Step 4. Initialize the embedding model -An [embedding model](/vector-search-overview.md#embedding-model) transforms data into [vector embeddings](/vector-search-overview.md#vector-embedding). This example uses the pre-trained model [**msmarco-MiniLM-L12-cos-v5**](https://huggingface.co/sentence-transformers/msmarco-MiniLM-L12-cos-v5) for text embedding. This lightweight model, provided by the `sentence-transformers` library, transforms text data into 384-dimensional vector embeddings. +An [embedding model](/tidb-cloud/vector-search-overview.md#embedding-model) transforms data into [vector embeddings](/tidb-cloud/vector-search-overview.md#vector-embedding). This example uses the pre-trained model [**msmarco-MiniLM-L12-cos-v5**](https://huggingface.co/sentence-transformers/msmarco-MiniLM-L12-cos-v5) for text embedding. This lightweight model, provided by the `sentence-transformers` library, transforms text data into 384-dimensional vector embeddings. To set up the model, copy the following code into the `example.py` file. This code initializes a `SentenceTransformer` instance and defines a `text_to_embedding()` function for later use. @@ -247,5 +247,5 @@ Therefore, according to the output, the swimming animal is most likely a fish, o ## See also -- [Vector Data Types](/vector-search-data-types.md) -- [Vector Search Index](/vector-search-index.md) \ No newline at end of file +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) \ No newline at end of file diff --git a/vector-search-get-started-using-sql.md b/tidb-cloud/vector-search-get-started-using-sql.md similarity index 88% rename from vector-search-get-started-using-sql.md rename to tidb-cloud/vector-search-get-started-using-sql.md index 9c3d493647daa..37d1e206a2701 100644 --- a/vector-search-get-started-using-sql.md +++ b/tidb-cloud/vector-search-get-started-using-sql.md @@ -5,7 +5,7 @@ summary: Learn how to quickly get started with Vector Search in TiDB using SQL s # Get Started with Vector Search via SQL -TiDB extends MySQL syntax to support [Vector Search](/vector-search-overview.md) and introduce new [Vector data types](/vector-search-data-types.md) and several [vector functions](/vector-search-functions-and-operators.md). +TiDB extends MySQL syntax to support [Vector Search](/tidb-cloud/vector-search-overview.md) and introduce new [Vector data types](/tidb-cloud/vector-search-data-types.md) and several [vector functions](/tidb-cloud/vector-search-functions-and-operators.md). This tutorial demonstrates how to get started with TiDB Vector Search just using SQL statements. You will learn how to use the [MySQL command-line client](https://dev.mysql.com/doc/refman/8.4/en/mysql.html) to complete the following operations: @@ -90,7 +90,7 @@ mysql --comments --host 127.0.0.1 --port 4000 -u root ### Step 2. Create a vector table -When creating a table, you can define a column as a [vector](/vector-search-overview.md#vector-embedding) column by specifying the `VECTOR` data type. +When creating a table, you can define a column as a [vector](/tidb-cloud/vector-search-overview.md#vector-embedding) column by specifying the `VECTOR` data type. For example, to create a table `embedded_documents` with a three-dimensional `VECTOR` column, execute the following SQL statements using your MySQL CLI: @@ -113,7 +113,7 @@ Query OK, 0 rows affected (0.27 sec) ### Step 3. Insert vector embeddings to the table -Insert three documents with their [vector embeddings](/vector-search-overview.md#vector-embedding) into the `embedded_documents` table: +Insert three documents with their [vector embeddings](/tidb-cloud/vector-search-overview.md#vector-embedding) into the `embedded_documents` table: ```sql INSERT INTO embedded_documents @@ -134,7 +134,7 @@ Records: 3 Duplicates: 0 Warnings: 0 > > This example simplifies the dimensions of the vector embeddings and uses only 3-dimensional vectors for demonstration purposes. > -> In real-world applications, [embedding models](/vector-search-overview.md#embedding-model) often produce vector embeddings with hundreds or thousands of dimensions. +> In real-world applications, [embedding models](/tidb-cloud/vector-search-overview.md#embedding-model) often produce vector embeddings with hundreds or thousands of dimensions. ### Step 4. Query the vector table @@ -191,5 +191,5 @@ Therefore, according to the output, the swimming animal is most likely a fish, o ## See also -- [Vector Data Types](/vector-search-data-types.md) -- [Vector Search Index](/vector-search-index.md) +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) diff --git a/vector-search-improve-performance.md b/tidb-cloud/vector-search-improve-performance.md similarity index 92% rename from vector-search-improve-performance.md rename to tidb-cloud/vector-search-improve-performance.md index a723a4af95927..bf7ebdf98b873 100644 --- a/vector-search-improve-performance.md +++ b/tidb-cloud/vector-search-improve-performance.md @@ -21,11 +21,11 @@ TiDB Vector Search enables you to perform Approximate Nearest Neighbor (ANN) que ## Add vector search index for vector columns -The [vector search index](/vector-search-index.md) dramatically improves the performance of vector search queries, usually by 10x or more, with a trade-off of only a small decrease of recall rate. +The [vector search index](/tidb-cloud/vector-search-index.md) dramatically improves the performance of vector search queries, usually by 10x or more, with a trade-off of only a small decrease of recall rate. ## Ensure vector indexes are fully built -After you insert a large volume of vector data, some of it might be in the Delta layer waiting for persistence. The vector index for such data will be built after the data is persisted. Until all vector data is indexed, vector search performance is suboptimal. To check the index build progress, see [View index build progress](/vector-search-index.md#view-index-build-progress). +After you insert a large volume of vector data, some of it might be in the Delta layer waiting for persistence. The vector index for such data will be built after the data is persisted. Until all vector data is indexed, vector search performance is suboptimal. To check the index build progress, see [View index build progress](/tidb-cloud/vector-search-index.md#view-index-build-progress). ## Reduce vector dimensions or shorten embeddings diff --git a/tidb-cloud/vector-search-index.md b/tidb-cloud/vector-search-index.md index 0b0eea2a3bdec..67e8709ec61f4 100644 --- a/tidb-cloud/vector-search-index.md +++ b/tidb-cloud/vector-search-index.md @@ -5,155 +5,142 @@ summary: Learn how to build and use the vector search index to accelerate K-Near # Vector Search Index -K-nearest neighbors (KNN) search is the problem of finding the K closest points for a given point in a vector space. The most straightforward approach to solving this problem is a brute force search, where the distance between all points in the vector space and the reference point is computed. This method guarantees perfect accuracy, but it is usually too slow for practical applications. Thus, nearest neighbors search problems are often solved with approximate algorithms. +K-nearest neighbors (KNN) search is the method for finding the K closest points to a given point in a vector space. The most straightforward approach to perform KNN search is a brute force search, which calculates the distance between the given vector and all other vectors in the space. This approach guarantees perfect accuracy, but it is usually too slow for real-world use. Therefore, approximate algorithms are commonly used in KNN search to enhance speed and efficiency. -In TiDB, you can create and utilize vector search indexes for such approximate nearest neighbor (ANN) searches over columns with [vector data types](/tidb-cloud/vector-search-data-types.md). By using vector search indexes, vector search queries could be finished in milliseconds. +In TiDB, you can create and use vector search indexes for such approximate nearest neighbor (ANN) searches over columns with [vector data types](/tidb-cloud/vector-search-data-types.md). By using vector search indexes, vector search queries could be finished in milliseconds. -TiDB currently supports the following vector search index algorithms: + -- HNSW +> **Warning:** +> +> The vector search feature is experimental. It is not recommended that you use it in the production environment. This feature might be changed without prior notice. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + + > **Note:** > -> Vector search index is only available for [TiDB Cloud Serverless](/tidb-cloud/select-cluster-tier.md#tidb-cloud-serverless) clusters. +> The vector search feature is only available for TiDB Self-Managed clusters and [TiDB Cloud Serverless](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-cloud-serverless) clusters. + +Currently, TiDB supports the [HNSW (Hierarchical Navigable Small World)](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) vector search index algorithm. + +## Restrictions + +- TiFlash nodes must be deployed in your cluster in advance. +- Vector search indexes cannot be used as primary keys or unique indexes. +- Vector search indexes can only be created on a single vector column and cannot be combined with other columns (such as integers or strings) to form composite indexes. +- A distance function must be specified when creating and using vector search indexes. Currently, only cosine distance `VEC_COSINE_DISTANCE()` and L2 distance `VEC_L2_DISTANCE()` functions are supported. +- For the same column, creating multiple vector search indexes using the same distance function is not supported. +- Directly dropping columns with vector search indexes is not supported. You can drop such a column by first dropping the vector search index on that column and then dropping the column itself. +- Modifying the type of a column with a vector index is not supported. +- Setting vector search indexes as [invisible](/sql-statements/sql-statement-alter-index.md) is not supported. +- Building vector search indexes on TiFlash nodes with [encryption at rest](https://docs.pingcap.com/tidb/stable/encryption-at-rest) enabled is not supported. ## Create the HNSW vector index -[HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) is one of the most popular vector indexing algorithms. The HNSW index provides good performance with relatively high accuracy (> 98% in typical cases). +[HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) is one of the most popular vector indexing algorithms. The HNSW index provides good performance with relatively high accuracy, up to 98% in specific cases. -To create an HNSW vector index, specify the index definition in the comment of a column with a [vector data type](/tidb-cloud/vector-search-data-types.md) when creating the table: +In TiDB, you can create an HNSW index for a column with a [vector data type](/tidb-cloud/vector-search-data-types.md) in either of the following ways: -```sql -CREATE TABLE vector_table_with_index ( - id INT PRIMARY KEY, doc TEXT, - embedding VECTOR(3) COMMENT "hnsw(distance=cosine)" -); -``` +- When creating a table, use the following syntax to specify the vector column for the HNSW index: -> **Note:** -> -> The syntax to create a vector index might change in future releases. + ```sql + CREATE TABLE foo ( + id INT PRIMARY KEY, + embedding VECTOR(5), + VECTOR INDEX idx_embedding ((VEC_COSINE_DISTANCE(embedding))) + ); + ``` -You must specify the distance metric via the `distance=` configuration when creating the vector index: +- For an existing table that already contains a vector column, use the following syntax to create an HNSW index for the vector column: -- Cosine Distance: `COMMENT "hnsw(distance=cosine)"` -- L2 Distance: `COMMENT "hnsw(distance=l2)"` + ```sql + CREATE VECTOR INDEX idx_embedding ON foo ((VEC_COSINE_DISTANCE(embedding))); + ALTER TABLE foo ADD VECTOR INDEX idx_embedding ((VEC_COSINE_DISTANCE(embedding))); -The vector index can only be created for fixed-dimensional vector columns like `VECTOR(3)`. It cannot be created for mixed-dimensional vector columns like `VECTOR` because vector distances can only be calculated between vectors with the same dimensions. + -- You can also explicitly specify "USING HNSW" to build the vector search index. + CREATE VECTOR INDEX idx_embedding ON foo ((VEC_COSINE_DISTANCE(embedding))) USING HNSW; + ALTER TABLE foo ADD VECTOR INDEX idx_embedding ((VEC_COSINE_DISTANCE(embedding))) USING HNSW; + ``` -If you are using programming language SDKs or ORMs, refer to the following documentation for creating vector indexes: +> **Note:** +> +> The vector search index feature relies on TiFlash replicas for tables. +> +> - If a vector search index is defined when a table is created, TiDB automatically creates a TiFlash replica for the table. +> - If no vector search index is defined when a table is created, and the table currently does not have a TiFlash replica, you need to manually create a TiFlash replica before adding a vector search index to the table. For example: `ALTER TABLE 'table_name' SET TIFLASH REPLICA 1;`. -- Python: [TiDB Vector SDK for Python](https://github.com/pingcap/tidb-vector-python) -- Python: [SQLAlchemy](/tidb-cloud/vector-search-integrate-with-sqlalchemy.md) -- Python: [Peewee](/tidb-cloud/vector-search-integrate-with-peewee.md) -- Python: [Django](/tidb-cloud/vector-search-integrate-with-django-orm.md) +When creating an HNSW vector index, you need to specify the distance function for the vector: -Be aware of the following limitations when creating the vector index. These limitations might be removed in future releases: +- Cosine Distance: `((VEC_COSINE_DISTANCE(embedding)))` +- L2 Distance: `((VEC_L2_DISTANCE(embedding)))` -- L1 distance and inner product are not supported for the vector index yet. +The vector index can only be created for fixed-dimensional vector columns, such as a column defined as `VECTOR(3)`. It cannot be created for non-fixed-dimensional vector columns (such as a column defined as `VECTOR`) because vector distances can only be calculated between vectors with the same dimension. -- You can only define and create a vector index when the table is created. You cannot create the vector index on demand using DDL statements after the table is created. You cannot drop the vector index using DDL statements as well. +For restrictions and limitations of vector search indexes, see [Restrictions](#restrictions). ## Use the vector index -The vector search index can be used in K-nearest neighbor search queries by using the `ORDER BY ... LIMIT` form like below: +The vector search index can be used in K-nearest neighbor search queries by using the `ORDER BY ... LIMIT` clause as follows: ```sql SELECT * -FROM vector_table_with_index -ORDER BY Vec_Cosine_Distance(embedding, '[1, 2, 3]') +FROM foo +ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3, 4, 5]') LIMIT 10 ``` -You must use the same distance metric as you have defined when creating the vector index if you want to utilize the index in vector search. +To use an index in a vector search, make sure that the `ORDER BY ... LIMIT` clause uses the same distance function as the one specified when creating the vector index. ## Use the vector index with filters Queries that contain a pre-filter (using the `WHERE` clause) cannot utilize the vector index because they are not querying for K-Nearest neighbors according to the SQL semantics. For example: ```sql --- Filter is performed before kNN, so Vector Index cannot be used: +-- For the following query, the `WHERE` filter is performed before KNN, so the vector index cannot be used: SELECT * FROM vec_table WHERE category = "document" -ORDER BY Vec_Cosine_distance(embedding, '[1, 2, 3]') +ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') LIMIT 5; ``` -Several workarounds are as follows: - -**Post-Filter after Vector Search:** Query for the K-Nearest neighbors first, then filter out unwanted results: +To use the vector index with filters, query for the K-Nearest neighbors first using vector search, and then filter out unwanted results: ```sql --- The filter is performed after kNN for these queries, so Vector Index can be used: +-- For the following query, the `WHERE` filter is performed after KNN, so the vector index cannot be used: SELECT * FROM ( SELECT * FROM vec_table - ORDER BY Vec_Cosine_distance(embedding, '[1, 2, 3]') + ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') LIMIT 5 ) t WHERE category = "document"; --- Note that this query may return less than 5 results if some are filtered out. -``` - -**Use Table Partitioning**: Queries within the [table partition](/partitioned-table.md) can fully utilize the vector index. This can be useful if you want to perform equality filters, as equality filters can be turned into accessing specified partitions. - -Example: Suppose you want to find the closest documentation for a specific product version. - -```sql --- Filter is performed before kNN, so Vector Index cannot be used: -SELECT * FROM docs -WHERE ver = "v2.0" -ORDER BY Vec_Cosine_distance(embedding, '[1, 2, 3]') -LIMIT 5; +-- Note that this query might return fewer than 5 results if some are filtered out. ``` -Instead of writing a query using the `WHERE` clause, you can partition the table and then query within the partition using the [`PARTITION` keyword](/partitioned-table.md#partition-selection): - -```sql -CREATE TABLE docs ( - id INT, - ver VARCHAR(10), - doc TEXT, - embedding VECTOR(3) COMMENT "hnsw(distance=cosine)" -) PARTITION BY LIST COLUMNS (ver) ( - PARTITION p_v1_0 VALUES IN ('v1.0'), - PARTITION p_v1_1 VALUES IN ('v1.1'), - PARTITION p_v1_2 VALUES IN ('v1.2'), - PARTITION p_v2_0 VALUES IN ('v2.0') -); - -SELECT * FROM docs -PARTITION (p_v2_0) -ORDER BY Vec_Cosine_distance(embedding, '[1, 2, 3]') -LIMIT 5; -``` - -See [Table Partitioning](/partitioned-table.md) for more information. - ## View index build progress -Unlike other indexes, vector indexes are built asynchronously. Therefore, vector indexes might not be immediately available after bulk data insertion. This does not affect data correctness or consistency, and you can perform vector searches at any time and get complete results. However, performance will be suboptimal until vector indexes are fully built. +After you insert a large volume of data, some of it might not be instantly persisted to TiFlash. For vector data that has already been persisted, the vector search index is built synchronously. For data that has not yet been persisted, the index will be built once the data is persisted. This process does not affect the accuracy and consistency of the data. You can still perform vector searches at any time and get complete results. However, performance will be suboptimal until vector indexes are fully built. To view the index build progress, you can query the `INFORMATION_SCHEMA.TIFLASH_INDEXES` table as follows: ```sql SELECT * FROM INFORMATION_SCHEMA.TIFLASH_INDEXES; -+---------------+------------+----------------+----------+--------------------+-------------+-----------+------------+---------------------+-------------------------+--------------------+------------------------+------------------+ -| TIDB_DATABASE | TIDB_TABLE | TIDB_PARTITION | TABLE_ID | BELONGING_TABLE_ID | COLUMN_NAME | COLUMN_ID | INDEX_KIND | ROWS_STABLE_INDEXED | ROWS_STABLE_NOT_INDEXED | ROWS_DELTA_INDEXED | ROWS_DELTA_NOT_INDEXED | TIFLASH_INSTANCE | -+---------------+------------+----------------+----------+--------------------+-------------+-----------+------------+---------------------+-------------------------+--------------------+------------------------+------------------+ -| test | sample | NULL | 106 | -1 | vec | 2 | HNSW | 0 | 13000 | 0 | 2000 | store-6ba728d2 | -| test | sample | NULL | 106 | -1 | vec | 2 | HNSW | 10500 | 0 | 0 | 4500 | store-7000164f | -+---------------+------------+----------------+----------+--------------------+-------------+-----------+------------+---------------------+-------------------------+--------------------+------------------------+------------------+ ++---------------+------------+----------+-------------+---------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+ +| TIDB_DATABASE | TIDB_TABLE | TABLE_ID | COLUMN_NAME | INDEX_NAME | COLUMN_ID | INDEX_ID | INDEX_KIND | ROWS_STABLE_INDEXED | ROWS_STABLE_NOT_INDEXED | ROWS_DELTA_INDEXED | ROWS_DELTA_NOT_INDEXED | ERROR_MESSAGE | TIFLASH_INSTANCE | ++---------------+------------+----------+-------------+---------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+ +| test | tcff1d827 | 219 | col1fff | 0a452311 | 7 | 1 | HNSW | 29646 | 0 | 0 | 0 | | 127.0.0.1:3930 | +| test | foo | 717 | embedding | idx_embedding | 2 | 1 | HNSW | 0 | 0 | 0 | 3 | | 127.0.0.1:3930 | ++---------------+------------+----------+-------------+---------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+ ``` -- The `ROWS_STABLE_INDEXED` and `ROWS_STABLE_NOT_INDEXED` columns show the index build progress. When `ROWS_STABLE_NOT_INDEXED` becomes 0, the index build is complete. +- You can check the `ROWS_STABLE_INDEXED` and `ROWS_STABLE_NOT_INDEXED` columns for the index build progress. When `ROWS_STABLE_NOT_INDEXED` becomes 0, the index build is complete. - As a reference, indexing a 500 MiB vector dataset might take up to 20 minutes. The indexer can run in parallel for multiple tables. Currently, adjusting the indexer priority or speed is not supported. + As a reference, indexing a 500 MiB vector dataset with 768 dimensions might take up to 20 minutes. The indexer can run in parallel for multiple tables. Currently, adjusting the indexer priority or speed is not supported. -- The `ROWS_DELTA_NOT_INDEXED` column shows the number of rows in the Delta layer. The Delta layer stores _recently_ inserted or updated rows and is periodically merged into the Stable layer according to the write workload. This merge process is called Compaction. +- You can check the `ROWS_DELTA_NOT_INDEXED` column for the number of rows in the Delta layer. Data in the storage layer of TiFlash is stored in two layers: Delta layer and Stable layer. The Delta layer stores recently inserted or updated rows and is periodically merged into the Stable layer according to the write workload. This merge process is called Compaction. The Delta layer is always not indexed. To achieve optimal performance, you can force the merge of the Delta layer into the Stable layer so that all data can be indexed: @@ -163,15 +150,17 @@ SELECT * FROM INFORMATION_SCHEMA.TIFLASH_INDEXES; For more information, see [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md). +In addition, you can monitor the execution progress of the DDL job by executing `ADMIN SHOW DDL JOBS;` and checking the `row count`. However, this method is not fully accurate, because the `row count` value is obtained from the `rows_stable_indexed` field in `TIFLASH_INDEXES`. You can use this approach as a reference for tracking the progress of indexing. + ## Check whether the vector index is used -Use the [`EXPLAIN`](/sql-statements/sql-statement-explain.md) or [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) statement to check whether this query is using the vector index. When `annIndex:` is presented in the `operator info` column for the `TableFullScan` executor, it means this table scan is utilizing the vector index. +Use the [`EXPLAIN`](/sql-statements/sql-statement-explain.md) or [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) statement to check whether a query is using the vector index. When `annIndex:` is presented in the `operator info` column for the `TableFullScan` executor, it means this table scan is utilizing the vector index. **Example: the vector index is used** ```sql [tidb]> EXPLAIN SELECT * FROM vector_table_with_index -ORDER BY Vec_Cosine_Distance(embedding, '[1, 2, 3]') +ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') LIMIT 10; +-----+-------------------------------------------------------------------------------------+ | ... | operator info | @@ -193,7 +182,7 @@ LIMIT 10; ```sql [tidb]> EXPLAIN SELECT * FROM vector_table_with_index - -> ORDER BY Vec_Cosine_Distance(embedding, '[1, 2, 3]'); + -> ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]'); +--------------------------------+-----+--------------------------------------------------+ | id | ... | operator info | +--------------------------------+-----+--------------------------------------------------+ @@ -210,9 +199,9 @@ LIMIT 10; When the vector index cannot be used, a warning occurs in some cases to help you learn the cause: ```sql --- Using a wrong distance metric: +-- Using a wrong distance function: [tidb]> EXPLAIN SELECT * FROM vector_table_with_index -ORDER BY Vec_l2_Distance(embedding, '[1, 2, 3]') +ORDER BY VEC_L2_DISTANCE(embedding, '[1, 2, 3]') LIMIT 10; [tidb]> SHOW WARNINGS; @@ -220,7 +209,7 @@ ANN index not used: not ordering by COSINE distance -- Using a wrong order: [tidb]> EXPLAIN SELECT * FROM vector_table_with_index -ORDER BY Vec_Cosine_Distance(embedding, '[1, 2, 3]') DESC +ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') DESC LIMIT 10; [tidb]> SHOW WARNINGS; @@ -229,11 +218,11 @@ ANN index not used: index can be used only when ordering by vec_cosine_distance( ## Analyze vector search performance -The [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) statement contains detailed information about how the vector index is used in the `execution info` column: +To learn detailed information about how a vector index is used, you can execute the [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) statement and check the `execution info` column in the output: ```sql [tidb]> EXPLAIN ANALYZE SELECT * FROM vector_table_with_index -ORDER BY Vec_Cosine_Distance(embedding, '[1, 2, 3]') +ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') LIMIT 10; +-----+--------------------------------------------------------+-----+ | | execution info | | @@ -259,12 +248,12 @@ LIMIT 10; Explanation of some important fields: -- `vector_index.load.total`: The total duration of loading index. This field could be larger than actual query time because multiple vector indexes may be loaded in parallel. +- `vector_index.load.total`: The total duration of loading index. This field might be larger than the actual query time because multiple vector indexes might be loaded in parallel. - `vector_index.load.from_s3`: Number of indexes loaded from S3. - `vector_index.load.from_disk`: Number of indexes loaded from disk. The index was already downloaded from S3 previously. - `vector_index.load.from_cache`: Number of indexes loaded from cache. The index was already downloaded from S3 previously. -- `vector_index.search.total`: The total duration of searching in the index. Large latency usually means the index is cold (never accessed before, or accessed long ago) so that there is heavy IO when searching through the index. This field could be larger than actual query time because multiple vector indexes may be searched in parallel. -- `vector_index.search.discarded_nodes`: Number of vector rows visited but discarded during the search. These discarded vectors are not considered in the search result. Large values usually indicate that there are many stale rows caused by UPDATE or DELETE statements. +- `vector_index.search.total`: The total duration of searching in the index. Large latency usually means the index is cold (never accessed before, or accessed long ago) so that there are heavy I/O operations when searching through the index. This field might be larger than the actual query time because multiple vector indexes might be searched in parallel. +- `vector_index.search.discarded_nodes`: Number of vector rows visited but discarded during the search. These discarded vectors are not considered in the search result. Large values usually indicate that there are many stale rows caused by `UPDATE` or `DELETE` statements. See [`EXPLAIN`](/sql-statements/sql-statement-explain.md), [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md), and [EXPLAIN Walkthrough](/explain-walkthrough.md) for interpreting the output. diff --git a/vector-search-integrate-with-django-orm.md b/tidb-cloud/vector-search-integrate-with-django-orm.md similarity index 97% rename from vector-search-integrate-with-django-orm.md rename to tidb-cloud/vector-search-integrate-with-django-orm.md index 5cbdaea4f893b..6deb74b3ee29a 100644 --- a/vector-search-integrate-with-django-orm.md +++ b/tidb-cloud/vector-search-integrate-with-django-orm.md @@ -5,7 +5,7 @@ summary: Learn how to integrate TiDB Vector Search with Django ORM to store embe # Integrate TiDB Vector Search with Django ORM -This tutorial walks you through how to use [Django](https://www.djangoproject.com/) ORM to interact with the [TiDB Vector Search](/vector-search-overview.md), store embeddings, and perform vector search queries. +This tutorial walks you through how to use [Django](https://www.djangoproject.com/) ORM to interact with the [TiDB Vector Search](/tidb-cloud/vector-search-overview.md), store embeddings, and perform vector search queries. @@ -276,5 +276,5 @@ results = Document.objects.annotate( ## See also -- [Vector Data Types](/vector-search-data-types.md) -- [Vector Search Index](/vector-search-index.md) +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) diff --git a/vector-search-integrate-with-jinaai-embedding.md b/tidb-cloud/vector-search-integrate-with-jinaai-embedding.md similarity index 98% rename from vector-search-integrate-with-jinaai-embedding.md rename to tidb-cloud/vector-search-integrate-with-jinaai-embedding.md index 5a3b1abd4d96e..cb83e8f985762 100644 --- a/vector-search-integrate-with-jinaai-embedding.md +++ b/tidb-cloud/vector-search-integrate-with-jinaai-embedding.md @@ -291,5 +291,5 @@ with Session(engine) as session: ## See also -- [Vector Data Types](/vector-search-data-types.md) -- [Vector Search Index](/vector-search-index.md) +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) diff --git a/vector-search-integrate-with-langchain.md b/tidb-cloud/vector-search-integrate-with-langchain.md similarity index 99% rename from vector-search-integrate-with-langchain.md rename to tidb-cloud/vector-search-integrate-with-langchain.md index f710e80ec7d6e..cf1124631bcb0 100644 --- a/vector-search-integrate-with-langchain.md +++ b/tidb-cloud/vector-search-integrate-with-langchain.md @@ -5,7 +5,7 @@ summary: Learn how to integrate Vector Search in TiDB Cloud with LangChain. # Integrate Vector Search with LangChain -This tutorial demonstrates how to integrate the [vector search](/vector-search-overview.md) feature in TiDB Cloud with [LangChain](https://python.langchain.com/). +This tutorial demonstrates how to integrate the [vector search](/tidb-cloud/vector-search-overview.md) feature in TiDB Cloud with [LangChain](https://python.langchain.com/). @@ -650,5 +650,5 @@ The expected output is as follows: ## See also -- [Vector Data Types](/vector-search-data-types.md) -- [Vector Search Index](/vector-search-index.md) +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) diff --git a/vector-search-integrate-with-llamaindex.md b/tidb-cloud/vector-search-integrate-with-llamaindex.md similarity index 98% rename from vector-search-integrate-with-llamaindex.md rename to tidb-cloud/vector-search-integrate-with-llamaindex.md index 1aaff4c6a4f2d..0172716194c10 100644 --- a/vector-search-integrate-with-llamaindex.md +++ b/tidb-cloud/vector-search-integrate-with-llamaindex.md @@ -5,7 +5,7 @@ summary: Learn how to integrate TiDB Vector Search with LlamaIndex. # Integrate Vector Search with LlamaIndex -This tutorial demonstrates how to integrate the [vector search](/vector-search-overview.md) feature of TiDB with [LlamaIndex](https://www.llamaindex.ai). +This tutorial demonstrates how to integrate the [vector search](/tidb-cloud/vector-search-overview.md) feature of TiDB with [LlamaIndex](https://www.llamaindex.ai). @@ -330,5 +330,5 @@ Empty Response ## See also -- [Vector Data Types](/vector-search-data-types.md) -- [Vector Search Index](/vector-search-index.md) +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) diff --git a/vector-search-integrate-with-peewee.md b/tidb-cloud/vector-search-integrate-with-peewee.md similarity index 96% rename from vector-search-integrate-with-peewee.md rename to tidb-cloud/vector-search-integrate-with-peewee.md index 8842ca2e68269..21ac6fc1abd02 100644 --- a/vector-search-integrate-with-peewee.md +++ b/tidb-cloud/vector-search-integrate-with-peewee.md @@ -5,7 +5,7 @@ summary: Learn how to integrate TiDB Vector Search with peewee to store embeddin # Integrate TiDB Vector Search with peewee -This tutorial walks you through how to use [peewee](https://docs.peewee-orm.com/) to interact with the [TiDB Vector Search](/vector-search-overview.md), store embeddings, and perform vector search queries. +This tutorial walks you through how to use [peewee](https://docs.peewee-orm.com/) to interact with the [TiDB Vector Search](/tidb-cloud/vector-search-overview.md), store embeddings, and perform vector search queries. @@ -266,5 +266,5 @@ results = Document.select(Document, distance).where(distance_expression < 0.2).o ## See also -- [Vector Data Types](/vector-search-data-types.md) -- [Vector Search Index](/vector-search-index.md) +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) diff --git a/vector-search-integrate-with-sqlalchemy.md b/tidb-cloud/vector-search-integrate-with-sqlalchemy.md similarity index 96% rename from vector-search-integrate-with-sqlalchemy.md rename to tidb-cloud/vector-search-integrate-with-sqlalchemy.md index 93965e454c6d7..ded83f8765672 100644 --- a/vector-search-integrate-with-sqlalchemy.md +++ b/tidb-cloud/vector-search-integrate-with-sqlalchemy.md @@ -5,7 +5,7 @@ summary: Learn how to integrate TiDB Vector Search with SQLAlchemy to store embe # Integrate TiDB Vector Search with SQLAlchemy -This tutorial walks you through how to use [SQLAlchemy](https://www.sqlalchemy.org/) to interact with [TiDB Vector Search](/vector-search-overview.md), store embeddings, and perform vector search queries. +This tutorial walks you through how to use [SQLAlchemy](https://www.sqlalchemy.org/) to interact with [TiDB Vector Search](/tidb-cloud/vector-search-overview.md), store embeddings, and perform vector search queries. @@ -237,5 +237,5 @@ with Session(engine) as session: ## See also -- [Vector Data Types](/vector-search-data-types.md) -- [Vector Search Index](/vector-search-index.md) +- [Vector Data Types](/tidb-cloud/vector-search-data-types.md) +- [Vector Search Index](/tidb-cloud/vector-search-index.md) diff --git a/vector-search-integration-overview.md b/tidb-cloud/vector-search-integration-overview.md similarity index 91% rename from vector-search-integration-overview.md rename to tidb-cloud/vector-search-integration-overview.md index d0f4e51c9cff1..7848959935fe7 100644 --- a/vector-search-integration-overview.md +++ b/tidb-cloud/vector-search-integration-overview.md @@ -25,8 +25,8 @@ TiDB provides official support for the following AI frameworks, enabling you to | AI frameworks | Tutorial | |---------------|---------------------------------------------------------------------------------------------------| -| Langchain | [Integrate Vector Search with LangChain](/vector-search-integrate-with-langchain.md) | -| LlamaIndex | [Integrate Vector Search with LlamaIndex](/vector-search-integrate-with-llamaindex.md) | +| Langchain | [Integrate Vector Search with LangChain](/tidb-cloud/vector-search-integrate-with-langchain.md) | +| LlamaIndex | [Integrate Vector Search with LlamaIndex](/tidb-cloud/vector-search-integrate-with-llamaindex.md) | Moreover, you can also use TiDB for various purposes, such as document storage and knowledge graph storage for AI applications. @@ -40,7 +40,7 @@ The following table lists some mainstream embedding service providers and the co | Embedding service providers | Tutorial | |-----------------------------|---------------------------------------------------------------------------------------------------------------------| -| Jina AI | [Integrate Vector Search with Jina AI Embeddings API](/vector-search-integrate-with-jinaai-embedding.md) | +| Jina AI | [Integrate Vector Search with Jina AI Embeddings API](/tidb-cloud/vector-search-integrate-with-jinaai-embedding.md) | ## Object Relational Mapping (ORM) libraries diff --git a/vector-search-limitations.md b/tidb-cloud/vector-search-limitations.md similarity index 93% rename from vector-search-limitations.md rename to tidb-cloud/vector-search-limitations.md index 063ddcdd4186c..1f3ae1722318c 100644 --- a/vector-search-limitations.md +++ b/tidb-cloud/vector-search-limitations.md @@ -21,7 +21,7 @@ This document describes the known limitations of TiDB vector search. ## Vector data type limitations -- Each [vector](/vector-search-data-types.md) supports up to 16383 dimensions. +- Each [vector](/tidb-cloud/vector-search-data-types.md) supports up to 16383 dimensions. - Vector data types cannot store `NaN`, `Infinity`, or `-Infinity` values. - Vector data types cannot store double-precision floating-point numbers. If you insert or store double-precision floating-point numbers in vector columns, TiDB converts them to single-precision floating-point numbers. - Vector columns cannot be used as primary keys or as part of a primary key. @@ -31,7 +31,7 @@ This document describes the known limitations of TiDB vector search. ## Vector index limitations -See [Vector search restrictions](/vector-search-index.md#restrictions). +See [Vector search restrictions](/tidb-cloud/vector-search-index.md#restrictions). ## Compatibility with TiDB tools diff --git a/vector-search-overview.md b/tidb-cloud/vector-search-overview.md similarity index 86% rename from vector-search-overview.md rename to tidb-cloud/vector-search-overview.md index 9d149fbb159ff..56e5ed63c365c 100644 --- a/vector-search-overview.md +++ b/tidb-cloud/vector-search-overview.md @@ -43,7 +43,7 @@ A vector embedding, also known as an embedding, is a sequence of numbers that re Vector embeddings are essential in machine learning and serve as the foundation for semantic similarity searches. -TiDB introduces [Vector data types](/vector-search-data-types.md) and [Vector search index](/vector-search-index.md) designed to optimize the storage and retrieval of vector embeddings, enhancing their use in AI applications. You can store vector embeddings in TiDB and perform vector search queries to find the most relevant data using these data types. +TiDB introduces [Vector data types](/tidb-cloud/vector-search-data-types.md) and [Vector search index](/tidb-cloud/vector-search-index.md) designed to optimize the storage and retrieval of vector embeddings, enhancing their use in AI applications. You can store vector embeddings in TiDB and perform vector search queries to find the most relevant data using these data types. ### Embedding model @@ -57,7 +57,7 @@ To learn how to generate vector embeddings for your specific data types, refer t After converting raw data into vector embeddings and storing them in TiDB, your application can execute vector search queries to find the data most semantically or contextually relevant to a user's query. -TiDB vector search identifies the top-k nearest neighbor (KNN) vectors by using a [distance function](/vector-search-functions-and-operators.md) to calculate the distance between the given vector and vectors stored in the database. The vectors closest to the given vector in the query represent the most similar data in meaning. +TiDB vector search identifies the top-k nearest neighbor (KNN) vectors by using a [distance function](/tidb-cloud/vector-search-functions-and-operators.md) to calculate the distance between the given vector and vectors stored in the database. The vectors closest to the given vector in the query represent the most similar data in meaning. ![The Schematic TiDB Vector Search](/media/vector-search/embedding-search.png) @@ -84,5 +84,5 @@ A recommendation engine is a system that proactively suggests content, products, To get started with TiDB Vector Search, see the following documents: -- [Get started with vector search using Python](/vector-search-get-started-using-python.md) -- [Get started with vector search using SQL](/vector-search-get-started-using-sql.md) +- [Get started with vector search using Python](/tidb-cloud/vector-search-get-started-using-python.md) +- [Get started with vector search using SQL](/tidb-cloud/vector-search-get-started-using-sql.md) diff --git a/vector-search-index.md b/vector-search-index.md deleted file mode 100644 index 828cd2accf3d2..0000000000000 --- a/vector-search-index.md +++ /dev/null @@ -1,263 +0,0 @@ ---- -title: Vector Search Index -summary: Learn how to build and use the vector search index to accelerate K-Nearest neighbors (KNN) queries in TiDB. ---- - -# Vector Search Index - -K-nearest neighbors (KNN) search is the method for finding the K closest points to a given point in a vector space. The most straightforward approach to perform KNN search is a brute force search, which calculates the distance between the given vector and all other vectors in the space. This approach guarantees perfect accuracy, but it is usually too slow for real-world use. Therefore, approximate algorithms are commonly used in KNN search to enhance speed and efficiency. - -In TiDB, you can create and use vector search indexes for such approximate nearest neighbor (ANN) searches over columns with [vector data types](/vector-search-data-types.md). By using vector search indexes, vector search queries could be finished in milliseconds. - - - -> **Warning:** -> -> The vector search feature is experimental. It is not recommended that you use it in the production environment. This feature might be changed without prior notice. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. - - - -> **Note:** -> -> The vector search feature is only available for TiDB Self-Managed clusters and [TiDB Cloud Serverless](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-cloud-serverless) clusters. - -Currently, TiDB supports the [HNSW (Hierarchical Navigable Small World)](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) vector search index algorithm. - -## Restrictions - -- TiFlash nodes must be deployed in your cluster in advance. -- Vector search indexes cannot be used as primary keys or unique indexes. -- Vector search indexes can only be created on a single vector column and cannot be combined with other columns (such as integers or strings) to form composite indexes. -- A distance function must be specified when creating and using vector search indexes. Currently, only cosine distance `VEC_COSINE_DISTANCE()` and L2 distance `VEC_L2_DISTANCE()` functions are supported. -- For the same column, creating multiple vector search indexes using the same distance function is not supported. -- Directly dropping columns with vector search indexes is not supported. You can drop such a column by first dropping the vector search index on that column and then dropping the column itself. -- Modifying the type of a column with a vector index is not supported. -- Setting vector search indexes as [invisible](/sql-statements/sql-statement-alter-index.md) is not supported. -- Building vector search indexes on TiFlash nodes with [encryption at rest](https://docs.pingcap.com/tidb/stable/encryption-at-rest) enabled is not supported. - -## Create the HNSW vector index - -[HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) is one of the most popular vector indexing algorithms. The HNSW index provides good performance with relatively high accuracy, up to 98% in specific cases. - -In TiDB, you can create an HNSW index for a column with a [vector data type](/vector-search-data-types.md) in either of the following ways: - -- When creating a table, use the following syntax to specify the vector column for the HNSW index: - - ```sql - CREATE TABLE foo ( - id INT PRIMARY KEY, - embedding VECTOR(5), - VECTOR INDEX idx_embedding ((VEC_COSINE_DISTANCE(embedding))) - ); - ``` - -- For an existing table that already contains a vector column, use the following syntax to create an HNSW index for the vector column: - - ```sql - CREATE VECTOR INDEX idx_embedding ON foo ((VEC_COSINE_DISTANCE(embedding))); - ALTER TABLE foo ADD VECTOR INDEX idx_embedding ((VEC_COSINE_DISTANCE(embedding))); - - -- You can also explicitly specify "USING HNSW" to build the vector search index. - CREATE VECTOR INDEX idx_embedding ON foo ((VEC_COSINE_DISTANCE(embedding))) USING HNSW; - ALTER TABLE foo ADD VECTOR INDEX idx_embedding ((VEC_COSINE_DISTANCE(embedding))) USING HNSW; - ``` - -> **Note:** -> -> The vector search index feature relies on TiFlash replicas for tables. -> -> - If a vector search index is defined when a table is created, TiDB automatically creates a TiFlash replica for the table. -> - If no vector search index is defined when a table is created, and the table currently does not have a TiFlash replica, you need to manually create a TiFlash replica before adding a vector search index to the table. For example: `ALTER TABLE 'table_name' SET TIFLASH REPLICA 1;`. - -When creating an HNSW vector index, you need to specify the distance function for the vector: - -- Cosine Distance: `((VEC_COSINE_DISTANCE(embedding)))` -- L2 Distance: `((VEC_L2_DISTANCE(embedding)))` - -The vector index can only be created for fixed-dimensional vector columns, such as a column defined as `VECTOR(3)`. It cannot be created for non-fixed-dimensional vector columns (such as a column defined as `VECTOR`) because vector distances can only be calculated between vectors with the same dimension. - -For restrictions and limitations of vector search indexes, see [Restrictions](#restrictions). - -## Use the vector index - -The vector search index can be used in K-nearest neighbor search queries by using the `ORDER BY ... LIMIT` clause as follows: - -```sql -SELECT * -FROM foo -ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3, 4, 5]') -LIMIT 10 -``` - -To use an index in a vector search, make sure that the `ORDER BY ... LIMIT` clause uses the same distance function as the one specified when creating the vector index. - -## Use the vector index with filters - -Queries that contain a pre-filter (using the `WHERE` clause) cannot utilize the vector index because they are not querying for K-Nearest neighbors according to the SQL semantics. For example: - -```sql --- For the following query, the `WHERE` filter is performed before KNN, so the vector index cannot be used: - -SELECT * FROM vec_table -WHERE category = "document" -ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') -LIMIT 5; -``` - -To use the vector index with filters, query for the K-Nearest neighbors first using vector search, and then filter out unwanted results: - -```sql --- For the following query, the `WHERE` filter is performed after KNN, so the vector index cannot be used: - -SELECT * FROM -( - SELECT * FROM vec_table - ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') - LIMIT 5 -) t -WHERE category = "document"; - --- Note that this query might return fewer than 5 results if some are filtered out. -``` - -## View index build progress - -After you insert a large volume of data, some of it might not be instantly persisted to TiFlash. For vector data that has already been persisted, the vector search index is built synchronously. For data that has not yet been persisted, the index will be built once the data is persisted. This process does not affect the accuracy and consistency of the data. You can still perform vector searches at any time and get complete results. However, performance will be suboptimal until vector indexes are fully built. - -To view the index build progress, you can query the `INFORMATION_SCHEMA.TIFLASH_INDEXES` table as follows: - -```sql -SELECT * FROM INFORMATION_SCHEMA.TIFLASH_INDEXES; -+---------------+------------+----------+-------------+---------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+ -| TIDB_DATABASE | TIDB_TABLE | TABLE_ID | COLUMN_NAME | INDEX_NAME | COLUMN_ID | INDEX_ID | INDEX_KIND | ROWS_STABLE_INDEXED | ROWS_STABLE_NOT_INDEXED | ROWS_DELTA_INDEXED | ROWS_DELTA_NOT_INDEXED | ERROR_MESSAGE | TIFLASH_INSTANCE | -+---------------+------------+----------+-------------+---------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+ -| test | tcff1d827 | 219 | col1fff | 0a452311 | 7 | 1 | HNSW | 29646 | 0 | 0 | 0 | | 127.0.0.1:3930 | -| test | foo | 717 | embedding | idx_embedding | 2 | 1 | HNSW | 0 | 0 | 0 | 3 | | 127.0.0.1:3930 | -+---------------+------------+----------+-------------+---------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+ -``` - -- You can check the `ROWS_STABLE_INDEXED` and `ROWS_STABLE_NOT_INDEXED` columns for the index build progress. When `ROWS_STABLE_NOT_INDEXED` becomes 0, the index build is complete. - - As a reference, indexing a 500 MiB vector dataset with 768 dimensions might take up to 20 minutes. The indexer can run in parallel for multiple tables. Currently, adjusting the indexer priority or speed is not supported. - -- You can check the `ROWS_DELTA_NOT_INDEXED` column for the number of rows in the Delta layer. Data in the storage layer of TiFlash is stored in two layers: Delta layer and Stable layer. The Delta layer stores recently inserted or updated rows and is periodically merged into the Stable layer according to the write workload. This merge process is called Compaction. - - The Delta layer is always not indexed. To achieve optimal performance, you can force the merge of the Delta layer into the Stable layer so that all data can be indexed: - - ```sql - ALTER TABLE COMPACT; - ``` - - For more information, see [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md). - -In addition, you can monitor the execution progress of the DDL job by executing `ADMIN SHOW DDL JOBS;` and checking the `row count`. However, this method is not fully accurate, because the `row count` value is obtained from the `rows_stable_indexed` field in `TIFLASH_INDEXES`. You can use this approach as a reference for tracking the progress of indexing. - -## Check whether the vector index is used - -Use the [`EXPLAIN`](/sql-statements/sql-statement-explain.md) or [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) statement to check whether a query is using the vector index. When `annIndex:` is presented in the `operator info` column for the `TableFullScan` executor, it means this table scan is utilizing the vector index. - -**Example: the vector index is used** - -```sql -[tidb]> EXPLAIN SELECT * FROM vector_table_with_index -ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') -LIMIT 10; -+-----+-------------------------------------------------------------------------------------+ -| ... | operator info | -+-----+-------------------------------------------------------------------------------------+ -| ... | ... | -| ... | Column#5, offset:0, count:10 | -| ... | ..., vec_cosine_distance(test.vector_table_with_index.embedding, [1,2,3])->Column#5 | -| ... | MppVersion: 1, data:ExchangeSender_16 | -| ... | ExchangeType: PassThrough | -| ... | ... | -| ... | Column#4, offset:0, count:10 | -| ... | ..., vec_cosine_distance(test.vector_table_with_index.embedding, [1,2,3])->Column#4 | -| ... | annIndex:COSINE(test.vector_table_with_index.embedding..[1,2,3], limit:10), ... | -+-----+-------------------------------------------------------------------------------------+ -9 rows in set (0.01 sec) -``` - -**Example: The vector index is not used because of not specifying a Top K** - -```sql -[tidb]> EXPLAIN SELECT * FROM vector_table_with_index - -> ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]'); -+--------------------------------+-----+--------------------------------------------------+ -| id | ... | operator info | -+--------------------------------+-----+--------------------------------------------------+ -| Projection_15 | ... | ... | -| └─Sort_4 | ... | Column#4 | -| └─Projection_16 | ... | ..., vec_cosine_distance(..., [1,2,3])->Column#4 | -| └─TableReader_14 | ... | MppVersion: 1, data:ExchangeSender_13 | -| └─ExchangeSender_13 | ... | ExchangeType: PassThrough | -| └─TableFullScan_12 | ... | keep order:false, stats:pseudo | -+--------------------------------+-----+--------------------------------------------------+ -6 rows in set, 1 warning (0.01 sec) -``` - -When the vector index cannot be used, a warning occurs in some cases to help you learn the cause: - -```sql --- Using a wrong distance function: -[tidb]> EXPLAIN SELECT * FROM vector_table_with_index -ORDER BY VEC_L2_DISTANCE(embedding, '[1, 2, 3]') -LIMIT 10; - -[tidb]> SHOW WARNINGS; -ANN index not used: not ordering by COSINE distance - --- Using a wrong order: -[tidb]> EXPLAIN SELECT * FROM vector_table_with_index -ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') DESC -LIMIT 10; - -[tidb]> SHOW WARNINGS; -ANN index not used: index can be used only when ordering by vec_cosine_distance() in ASC order -``` - -## Analyze vector search performance - -To learn detailed information about how a vector index is used, you can execute the [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) statement and check the `execution info` column in the output: - -```sql -[tidb]> EXPLAIN ANALYZE SELECT * FROM vector_table_with_index -ORDER BY VEC_COSINE_DISTANCE(embedding, '[1, 2, 3]') -LIMIT 10; -+-----+--------------------------------------------------------+-----+ -| | execution info | | -+-----+--------------------------------------------------------+-----+ -| ... | time:339.1ms, loops:2, RU:0.000000, Concurrency:OFF | ... | -| ... | time:339ms, loops:2 | ... | -| ... | time:339ms, loops:3, Concurrency:OFF | ... | -| ... | time:339ms, loops:3, cop_task: {...} | ... | -| ... | tiflash_task:{time:327.5ms, loops:1, threads:4} | ... | -| ... | tiflash_task:{time:327.5ms, loops:1, threads:4} | ... | -| ... | tiflash_task:{time:327.5ms, loops:1, threads:4} | ... | -| ... | tiflash_task:{time:327.5ms, loops:1, threads:4} | ... | -| ... | tiflash_task:{...}, vector_idx:{ | ... | -| | load:{total:68ms,from_s3:1,from_disk:0,from_cache:0},| | -| | search:{total:0ms,visited_nodes:2,discarded_nodes:0},| | -| | read:{vec_total:0ms,others_total:0ms}},...} | | -+-----+--------------------------------------------------------+-----+ -``` - -> **Note:** -> -> The execution information is internal. Fields and formats are subject to change without any notification. Do not rely on them. - -Explanation of some important fields: - -- `vector_index.load.total`: The total duration of loading index. This field might be larger than the actual query time because multiple vector indexes might be loaded in parallel. -- `vector_index.load.from_s3`: Number of indexes loaded from S3. -- `vector_index.load.from_disk`: Number of indexes loaded from disk. The index was already downloaded from S3 previously. -- `vector_index.load.from_cache`: Number of indexes loaded from cache. The index was already downloaded from S3 previously. -- `vector_index.search.total`: The total duration of searching in the index. Large latency usually means the index is cold (never accessed before, or accessed long ago) so that there are heavy I/O operations when searching through the index. This field might be larger than the actual query time because multiple vector indexes might be searched in parallel. -- `vector_index.search.discarded_nodes`: Number of vector rows visited but discarded during the search. These discarded vectors are not considered in the search result. Large values usually indicate that there are many stale rows caused by `UPDATE` or `DELETE` statements. - -See [`EXPLAIN`](/sql-statements/sql-statement-explain.md), [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md), and [EXPLAIN Walkthrough](/explain-walkthrough.md) for interpreting the output. - -## See also - -- [Improve Vector Search Performance](/vector-search-improve-performance.md) -- [Vector Data Types](/vector-search-data-types.md)