Streaming PQ

[eddyxu]

Train IVF_PQ on sample dataset, and use streaming to compute PQ code without loading full dataset in memory.

[changhiskhan]

Given how much has changed, could you re-run the sift benchmarks just to make sure the latency / recall profile looks roughly the same still?

[changhiskhan]

Should this be an error?

[changhiskhan]

are these values from FAISS?

[changhiskhan]

do we have to change much to support configurable num_bits? is it just exposing a new API parameter? or is the underlying index creation hard coded to 8 bits?

[eddyxu]

num_nbits is not configurable via user-facing API yet. 8 bits is just 1 byte. other than 8 bits, the other configuration needs some work .

[changhiskhan]

Should we move hard coded constants into one place so we know what needs to change in the future?

[eddyxu]

oh, yea, lemme just make this a parameter passed from user-facing api

[changhiskhan]

i thought this doesn't use OPQ yet?

[eddyxu]

Running on /benchmark/sift/run_all.sh

Running with ivf512
nprobes: 1, refine=None, recall@10=0.546, mean(s)=0.00124342679977417
nprobes: 1, refine=1, recall@10=0.573, mean(s)=0.001298525333404541
nprobes: 1, refine=5, recall@10=0.780, mean(s)=0.0016234064102172853
nprobes: 1, refine=10, recall@10=0.810, mean(s)=0.0021506214141845705
nprobes: 10, refine=None, recall@10=0.643, mean(s)=0.001989481449127197
nprobes: 10, refine=1, recall@10=0.643, mean(s)=0.002009923458099365
nprobes: 10, refine=5, recall@10=0.968, mean(s)=0.002357909679412842
nprobes: 10, refine=10, recall@10=0.975, mean(s)=0.0028011083602905275
nprobes: 50, refine=None, recall@10=0.627, mean(s)=0.005469377040863037
nprobes: 50, refine=1, recall@10=0.647, mean(s)=0.00498615026473999
nprobes: 50, refine=5, recall@10=0.972, mean(s)=0.005275309085845947
nprobes: 50, refine=10, recall@10=0.995, mean(s)=0.006533384323120117
nprobes: 100, refine=None, recall@10=0.650, mean(s)=0.008666768074035644
nprobes: 100, refine=1, recall@10=0.683, mean(s)=0.009069747924804687
nprobes: 100, refine=5, recall@10=0.973, mean(s)=0.009081656932830811
nprobes: 100, refine=10, recall@10=0.992, mean(s)=0.009473004341125489