No matter what index type I specify, it is always listed as IVF_PQ.

[benmayersohn]

I've tried creating a IVF_HNSW_SQ index on a dataset that has a vector column called embedding, but it seems the index type is always listed as IVF_PQ no matter what.

Here's a reproducible example. I use lancedb here but I extract the underlying lance dataset right away and create the index on that, so I don't think it's a lancedb issue:

import polars as pl
import numpy as np
from lancedb.db import LanceDBConnection

df = pl.DataFrame({
    'product_id': np.arange(1000),
    'embedding': np.random.rand(1000, 512).astype(np.float32)
    },
    schema_overrides={
        "embedding": pl.Array(inner=pl.Float32, shape=(512,)),
    },
)

db_test = LanceDBConnection('test.lancedb')

db_test.create_table('test_table', df, mode='overwrite')
table_test = db_test['test_table']
dataset_test = table_test.to_lance()

dataset_test.create_index(
    'embedding',
    'IVF_HNSW_SQ',
    metric='cosine',
    num_partitions=16,
    replace=True,
    ef_construction=100,
    M=48,
    max_levels=4
)

dataset_test.list_indices()

# This is my output
# [{'name': 'embedding_idx', 'type': 'IVF_PQ', 'uuid': '32acea8a-b94b-4578-a698-0950facdcf84', 'fields': ['embedding'], 'version': 10, 'fragment_ids': {0}}]

Versions of packages I'm using in this example with Python 3.10:

lancedb==0.17.0
pylance==0.20.0
polars==1.17.0
numpy==1.26.4

Any suggestions would be appreciated. Thanks!

[BubbleCal]

hi @benmayersohn
it's a bug, that the list_indices would always report the vector index type is IVF_PQ even it's not.
just filed a fix #3258

[benmayersohn]

Thanks so much @BubbleCal!