Improve indexing performance

[eddyxu]

No description provided.

[eddyxu]

Run sift benchmark

Running with ivf512
nprobes: 1, refine=None, recall@10=0.554, mean(s)=0.002198166847229004
nprobes: 1, refine=1, recall@10=0.565, mean(s)=0.0022933125495910645
nprobes: 1, refine=5, recall@10=0.790, mean(s)=0.002348010540008545
nprobes: 1, refine=10, recall@10=0.788, mean(s)=0.002333817481994629
nprobes: 10, refine=None, recall@10=0.667, mean(s)=0.0036499619483947752
nprobes: 10, refine=1, recall@10=0.657, mean(s)=0.003946800231933594
nprobes: 10, refine=5, recall@10=0.954, mean(s)=0.004477522373199463
nprobes: 10, refine=10, recall@10=0.975, mean(s)=0.0038238167762756348
nprobes: 50, refine=None, recall@10=0.656, mean(s)=0.0102131986618042
nprobes: 50, refine=1, recall@10=0.649, mean(s)=0.010562391281127929
nprobes: 50, refine=5, recall@10=0.959, mean(s)=0.011186907291412354
nprobes: 50, refine=10, recall@10=0.991, mean(s)=0.011933960914611817

[changhiskhan]

Not a blocker but wondering if this shareable with computing distortion.