Add new int8 scalar quantization to HNSW codec

[benwtrent]

As with most codec changes, this is an eye popping number of LoC and the design isn't finished yet.

Initial benchmarking (utilizing non-normalized cohere embeddings + max-inner product, which is a particularly difficult case for naive quantization), I get 10-20% faster search and ~4x smaller storage used for the search (I am keeping the raw vectors around...we can debate if we want to do that).

Recall@10 with 100 fanout = 0.804
Recall@100 with 200 fanout = 0.9.

I am reaching the point where the design needs to be finalized and I wanted to reachout for feedback.

Some design discussion points that I am unsure about are:

• Do we want to have a new "flat" vector codec that HNSW (or other complicated vector indexing methods), can use? Detractor here is that now HNSW codec relies on another pluggable thing that is a "flat" vector index (just provides mechanisms for reading, writing, merging vectors in a flat index).
• Should "quantization" just be a thing that is provided to vector codecs? The main detractor here is future scalar quantization could easily be added (like int4 or even binary).
• Should the "quantizer" keep the raw vectors around itself? Or rely on some external party to provide them (in this case, I an relying on the HNSW codec)?

[benwtrent]

@uschindler I switch it around. Lucene95Codec is back, moved the previous HNSW format into backwards_codec and switched Lucene95Codec to use the Lucene99Hnsw format.

[mayya-sharipova]

@benwtrent Amazing work!!! Thanks Ben, Jim and Tom for such a great contribution!
I left some small comments, but otherwise this PR LGTM (I am still fuzzy about all the math in the quantizer, so I trust you and @tveasey that you got it right).

[naveentatikonda]

I get 10-20% faster search and ~4x smaller storage used for the search (I am keeping the raw vectors around...we can debate if we want to do that)

@benwtrent Based on your comments, does 4x smaller storage means optimization in terms of disk space or the memory savings(RAM). Can you please confirm and how did you run these benchmarkings (the tool used) and how did you conclude that we are saving x% of memory savings. The reason being when I tried to integrate lucene with my service I was not able to see the same amount of memory savings during search.

[benwtrent]

@naveentatikonda , I used lucene-util.

It's an off-heap memory requirement reduction. Instead of having to load float32 values in memory, it will load byte values. HNSW does many random reads, this read behavior really needs to be in memory for it to be fast.

It uses about 25% more disk as the raw float32 vectors are kept along with the quantized ones.

[navneet1v]

@naveentatikonda , I used lucene-util.

It's an off-heap memory requirement reduction. Instead of having to load float32 values in memory, it will load byte values. HNSW does many random reads, this read behavior really needs to be in memory for it to be fast.

It uses about 25% more disk as the raw float32 vectors are kept along with the quantized ones.

can you share some code reference and dataset you used for testing? It will be very helpful.

[benwtrent]

https://github.com/mikemccand/luceneutil

It can be a little frustrating at first to set up, but it's nice for quick integration and performance tests.

The python script knnPerfTest.py and I downloaded a bunch of CohereV2 parquet files from huggingface.

CohereV3 is released now and they have many GB of vectors available for testing on huggingface. So, you could use those instead.

Love those folks over at Cohere :)

[navneet1v]

Thanks @benwtrent will look into this.