How do I build this in windows (cublas)

[kevkid]

I want to try out the cublast (#1412) (master) build to offload some of the layers to the gpu. trying to build this in windows is proving to be a bit difficult for me. I have installed cmake and have installed the nvidia cuda toolkit and I even installed Build Tools for Visual Studio 2022. every time I try to run:

mkdir build
cd build
cmake .. -DLLAMA_CUBLAS=ON
cmake --build . --config Release

I get:

cmake .. -DLLAMA_CUBLAS=ON
CMake Error at CMakeLists.txt:2 (project):
  Running

   'nmake' '-?'

  failed with:

   The system cannot find the file specified


CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!

I have also tried with make:

g++: warning: Files/NVIDIA: linker input file unused because linking not done
g++: error: Files/NVIDIA: linker input file not found: No such file or directory
g++: warning: GPU: linker input file unused because linking not done
g++: error: GPU: linker input file not found: No such file or directory
g++: warning: Computing: linker input file unused because linking not done
g++: error: Computing: linker input file not found: No such file or directory
g++: warning: Toolkit/CUDA/v12.1/targets/x86_64-linux/include: linker input file unused because linking not done
g++: error: Toolkit/CUDA/v12.1/targets/x86_64-linux/include: linker input file not found: No such file or directory
make: *** [Makefile:188: llama.o] Error 1

nvcc:

C:\Users\kevin>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:36:15_Pacific_Daylight_Time_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0

OK, I think I got it to build by using powershell -> and dropping the files from the cuda toolkit. since I have version 12.1 the path is a little different. I copied the files from here:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\extras\visual_studio_integration\MSBuildExtensions to here
C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\BuildCustomizations
as per:
https://stackoverflow.com/questions/56636714/cuda-compile-problems-on-windows-cmake-error-no-cuda-toolset-found

then ran the regular cmake build procedure in powershell
If anyone can list how they got this built in windows that would really help.

[toncho11]

I have made progress:

1. You need to start VS 2019 prompt so that cl.exe is in your path. I think VS 2017-2019 are only supported. Then run:

w64devkit\w64devkit.exe

and execute:

make LLAMA_CUBLAS=1

after you have done the modifications below to Makefile.

2. For my NVIDIA T600 I had to put "compute_75" on line 376:

else ifndef CUDA_POWER_ARCH
	MK_NVCCFLAGS += -arch=compute_75 

3. I had to put the correct DOS path to CUDA on Windows:

ifdef LLAMA_CUBLAS
	MK_CPPFLAGS  += -DGGML_USE_CUBLAS -IC:/PROGRA~1/NVIDIA~2/CUDA/v11.5/include
	MK_LDFLAGS   += -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -LC:/PROGRA~1/NVIDIA~2/CUDA/v11.5/lib/x64

My CUDA folder is: C:/PROGRA~1/NVIDIA~2/CUDA/v11.5. I am using DOS names. Adapt the /lib and /include folders accordingly.

4. Remove all warnings. cl.exe does not like any of the specified warnings.

• at lines 204-206

# warnings
# removed
# removed
# removed

• around line 420 like that (removing -Wno-pedantic -Xcompiler "$(CUDA_CXXFLAGS)"):

ifdef LLAMA_CUDA_CCBIN
	MK_NVCCFLAGS += -ccbin $(LLAMA_CUDA_CCBIN)
endif
ggml-cuda.o: ggml-cuda.cu ggml-cuda.h
	$(NVCC) $(BASE_CXXFLAGS) $(NVCCFLAGS) -c $< -o $@
endif # LLAMA_CUBLAS

5. You will run into my current problem where the current line around 360:

MK_LDFLAGS += -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -LC:/PROGRA~1/NVIDIA~2/CUDA/v11.5/lib/x64

requires lib files that are not available in the CUDA lib's x64 folder: culibos, dl, rt

My Makefile is available here: Makefile.csv. Just remove the .csv extension.

[toncho11]

I got the response from NVIDIA. These lib files are only available on Linux. For a moment I thought I would crack it :)

[sorasoras]

cmake .. -DLLAMA_CUBLAS=ON -DLLAMA_CUDA_FORCE_DMMV=TRUE -DLLAMA_CUDA_DMMV_X=64 -DLLAMA_CUDA_MMV_Y=4 -DLLAMA_CUDA_F16=TRUE -DGGML_CUDA_FORCE_MMQ=YES
That's how I built it in windows.
by the way ,you need to add path to the env in windows.

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\bin add the path in env

[bullno1]

This works on windows both locally and on CI: https://github.com/bullno1/hey/blob/master/msvc.bat

CI build log: https://github.com/bullno1/hey/actions/runs/7325404244/job/19949954554

Github actions: https://github.com/bullno1/hey/blob/master/.github/workflows/windows-build.yml

I generate a MSVC solution because that's how one would develop on Windows.