Add linux-gpu profile to enable GPU support

[matthieuml]

This is probably the solution of this issue: #62.

I added some deploy resources to allow GPU access of service llm-gpu. For now, I have created a new profile linux-gpu associated with llm-gpu service as if there is no GPU, the stack can't start.

[oskarhane]

Nice! I can't test this myself so pinging @slimslenderslacks and @mchiang0610 to have a look.

[jli113]

Not working on Ubuntu 22.04

(base) midea@G4:~/genai-stack$ docker compose --profile linux-gpu up
WARN[0000] The "OPENAI_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_PROJECT" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_ACCESS_KEY_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_SECRET_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_DEFAULT_REGION" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_PROJECT" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_ACCESS_KEY_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_SECRET_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_DEFAULT_REGION" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_PROJECT" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_ACCESS_KEY_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_SECRET_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_DEFAULT_REGION" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_PROJECT" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_ACCESS_KEY_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_SECRET_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_DEFAULT_REGION" variable is not set. Defaulting to a blank string. 
[+] Running 7/7
 ✔ Container genai-stack-database-1    Running                                                                                                                                             0.0s 
 ✔ Container genai-stack-pull-model-1  Recreated                                                                                                                                           0.1s 
 ✔ Container genai-stack-bot-1         Recreated                                                                                                                                           0.6s 
 ✔ Container genai-stack-loader-1      Recreated                                                                                                                                           1.0s 
 ✔ Container genai-stack-api-1         Recreated                                                                                                                                           1.2s 
 ✔ Container genai-stack-pdf_bot-1     Recreated                                                                                                                                           0.8s 
 ✔ Container genai-stack-front-end-1   Recreated                                                                                                                                          10.4s 
Attaching to genai-stack-api-1, genai-stack-bot-1, genai-stack-database-1, genai-stack-front-end-1, genai-stack-llm-gpu-1, genai-stack-loader-1, genai-stack-pdf_bot-1, genai-stack-pull-model-1
Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

GPU and cuda

(base) midea@G4:~/genai-stack$ nvidia-smi
Tue Nov  7 15:01:19 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01             Driver Version: 535.113.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P620                    Off | 00000000:01:00.0  On |                  N/A |
| 41%   55C    P0              N/A /  N/A |    392MiB /  2048MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2002      G   /usr/lib/xorg/Xorg                          116MiB |
|    0   N/A  N/A      2142      G   /usr/bin/gnome-shell                        100MiB |
|    0   N/A  N/A   1462607      G   ...irefox/3290/usr/lib/firefox/firefox      146MiB |
|    0   N/A  N/A   1464832      G   gnome-control-center                          1MiB |
+---------------------------------------------------------------------------------------+

[matthieuml]

To use GPU capabilities inside Docker, you need to have the nvidia-container-toolkit package installed on your machine: https://docs.docker.com/config/containers/resource_constraints/#gpu

I forgot to add it in the readme.md, adding "ensure that you have nvidia-container-toolkit package installed" would do it.

[jli113]

I have installed nvidia-container-toolkit, still cannot select.

(base) midea@G4:~/genai-stack$ which nvidia-container-runtime-hook
/usr/bin/nvidia-container-runtime-hook
(base) midea@G4:~/genai-stack$ docker compose --profile linux-gpu up
WARN[0000] The "LANGCHAIN_PROJECT" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_ACCESS_KEY_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_SECRET_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_DEFAULT_REGION" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_PROJECT" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_ACCESS_KEY_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_SECRET_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_DEFAULT_REGION" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_PROJECT" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_ACCESS_KEY_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_SECRET_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_DEFAULT_REGION" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPENAI_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_PROJECT" variable is not set. Defaulting to a blank string. 
WARN[0000] The "LANGCHAIN_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_ACCESS_KEY_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_SECRET_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "AWS_DEFAULT_REGION" variable is not set. Defaulting to a blank string. 
[+] Running 2/0
 ✔ Container genai-stack-pull-model-1  Created                                                                                                                             0.0s 
 ✔ Container genai-stack-database-1    Running                                                                                                                             0.0s 
Attaching to genai-stack-api-1, genai-stack-bot-1, genai-stack-database-1, genai-stack-front-end-1, genai-stack-llm-gpu-1, genai-stack-loader-1, genai-stack-pdf_bot-1, genai-stack-pull-model-1
Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

[matthieuml]

Does this command return the nvidia-smi output sucessfully?

docker run -it --rm --gpus all ubuntu nvidia-smi

[jli113]

Does this command return the nvidia-smi output sucessfully?

docker run -it --rm --gpus all ubuntu nvidia-smi

Thanks, worked after systemctl restart docker

[jli113]

Returning 200 with no answer

[matthieuml]

From the results of nvidia-smi on your computer: 392MiB / 2048MiB, I suppose that you don't have enough room for the model to be loaded on the GPU. Thus, the answer will be quite slow.

I don't see where the bot service is outputting a 200 answer, don't confuse the api health check output with the output of bot.

My guess is that if you wait long enough, you will have a result.