cuda : improve cuda pool efficiency using virtual memory (ggerganov#4606

) * cuda : improve cuda pool efficiency using virtual memory * fix mixtral * fix cmake build * check for vmm support, disable for hip ggml-ci * fix hip build * clarify granularity * move all caps to g_device_caps * refactor error checking * add cuda_pool_alloc, refactor most pool allocations ggml-ci * fix hip build * CUBLAS_TF32_TENSOR_OP_MATH is not a macro * more hip crap * llama : fix msvc warnings * ggml : fix msvc warnings * minor * minor * cuda : fallback to CPU on host buffer alloc fail * Update ggml-cuda.cu Co-authored-by: Johannes Gäßler <[email protected]> * Update ggml-cuda.cu Co-authored-by: Johannes Gäßler <[email protected]> * ensure allocations are always aligned * act_size -> actual_size --------- Co-authored-by: Johannes Gäßler <[email protected]>
jordankanter · Feb 3, 2024 · 1fc2dcc · 1fc2dcc
1 parent aff07f0
commit 1fc2dcc
Show file tree

Hide file tree

Showing 8 changed files with 1,046 additions and 1,542 deletions.
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -302,6 +302,8 @@ if (LLAMA_CUBLAS)
             set(LLAMA_EXTRA_LIBS ${LLAMA_EXTRA_LIBS} CUDA::cudart CUDA::cublas CUDA::cublasLt)
         endif()
 
+        set(LLAMA_EXTRA_LIBS ${LLAMA_EXTRA_LIBS} CUDA::cuda_driver)
+
     if (NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
         # 52 == lowest CUDA 12 standard
         # 60 == f16 CUDA intrinsics