faiss.index_cpu_to_gpu slow on Conda build

[eddienewton]

Summary

Using docker, I've build a conda build (see attached). I'm using faiss to match sift descriptors (128bit). I'm able to query correctly using the CPU version. However, when I use the GPU, the machine takes a few minutes on the following:

res = faiss.StandardGpuResources()  # use a single GPU
gpu_index = faiss.index_cpu_to_gpu(res, 0, index_flat)

I'm running this on an ubuntu machine (18.04) with a T4 GPU.

I'm installing faiss using:

conda install -y faiss-gpu cudatoolkit=10.0 -c pytorch

Are there any other setup instructions I need to do? Since I'm just creating the index on the gpu, this shouldn't take more than a second correct?

Thanks!
Dockerfile.txt

Platform

Ubuntu 18.04

Faiss version:

Faiss compilation options:

Running on:

• CPU
• GPU

Interface:

• C++
• Python

[eddienewton]

I had the same issue with the test gpu python tests, so it looks like my problem is similar to this one:

#34

Basically, the conda builds are unable to correctly select the correct nvidia configuration?

As others have probably stated, a docker build will fix this issue. Closing for now.

Thanks!

[aruslantsev]

I have the same issue on Ubuntu 18.04. I use faiss to find neighbors for 128-elements vectors.

I'm installing faiss using conda install -y faiss-gpu cudatoolkit=10.0 -c pytorch
When I run gpu_index = faiss.index_cpu_to_gpu(res, 0, index_flat) without docker, it takes approximately 3 seconds to finish. In docker it lasts for more than 950 seconds.

[terminalh2t3]

I had the same issue.

• Ran on local virtual env with faiss==1.6.3 and cudatoolkil=10.2 tooks 10s
• Ran on docker which was built from nvidia/cuda:10.2-base took ~300s
  But, if I remote to the docker container and start it manually, the first run takes long (~300s) but the second was very fast (~11s). Any idea?

[CholoTook]

I'm seeing the same, very slow moving the index to the GPU:

$ pip freeze | grep faiss
faiss-gpu==1.7.2

$ nvidia-smi # reports
CUDA Version: 12.3

$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  545.23.08  Mon Nov  6 23:49:37 UTC 2023
GCC version:  gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04) 

$ /usr/local/cuda-12.3/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:17:15_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0

import numpy as np
import faiss

import timeit


# Define critical index parameters...
d = 1536  # dimensions
nb = 1000    # number of rows
nq = 10  # number of queries to test

# Initalise random vectors for testing the search:
np.random.seed(1234)
xb = np.random.random((nb, d)).astype("float32")
xq = np.random.random((nq, d)).astype("float32")

# Not actually sure why the example does this...
xb[:, 0] += np.arange(nb) / 1000.0
xq[:, 0] += np.arange(nq) / 1000.0

# Build the index
index = faiss.IndexFlatL2(d)
print(index.is_trained)
index.add(xb)  # add vectors to the index
print(index.is_trained)
print(index.ntotal)

res = faiss.StandardGpuResources()  # use a single GPU
gpu_index_flat = faiss.index_cpu_to_gpu(res, 0, index) # Stuck here
gpu_index_flat.add(xb)         # add vectors to the index

Any suggestions appreciated!

[CholoTook]

Heh... I've seen what others have reported... Once it's 'warmed up' it's now acceptably fast :-)